Notebook

BlackHoles@Home Tutorial: Creating a `BOINC` app using the `WrapperApp`¶

Author: Leo Werneck¶

This tutorial notebook demonstrates how to write a program that runs in the `BOINC` infrastructure using the `WrapperApp`¶

WARNING: this tutorial notebook is currently incompatible with Windows

Introduction:¶

The BlackHoles@Home project allows users to volunteer CPU time so a large number of binary black holes simulations can be performed. The objective is to create a large catalog of gravitational waveforms, which can be used by observatories such as LIGO, VIRGO, and, in the future, LISA in order to infer what was the source of a detected gravitational wave.

BlackHoles@Home is destined to run on the BOINC infrastructure (alongside Einstein@Home and many other great projects), enabling anyone with a computer to contribute to the construction of the largest numerical relativity gravitational wave catalogs ever produced.

Additional Reading Material:¶

Table of Contents¶

$$\label{toc}$$

This tutorial explains how to use the BOINC wrapper application to run a simple program. The structture of this notebook is as follows:

Step 1: Introduction
Step 2: Compiling the BOINC wrapper app for your platform
Step 3: Using the BOINC wrapper app
1. Step 3.a: The main application
2. Step 3.b: Compiling the main application
3. Step 3.c: The job.xml file
  1. Step 3.c.i: A very simple job.xml file
  2. Step 3.c.ii: Redirecting and zipping output files
Step 4: Output this notebook to $\LaTeX$-formatted PDF file

Step 1: Introduction [Back to top]¶

$$\label{introduction}$$

The WrapperApp is the simplest way of converting an existing program into a BOINC compatible application. The program that will be actually running is the WrapperApp and it will take care of:

Interfacing with the BOINC libraries
Running the original program
Handling input/output files

Let us assume a simple BOINC application, which is made out of only one program, bhah_test_app. The directory of this application should then contain the following files:

The application file bhah_test_app with the name format appname_version_platform.
The WrapperApp file with the name format WrapperAppname_version_platform.
The WrapperApp configuration file, which we will typically call appname_version_job.xml.
The appication version file, which is called version.xml.

We note that the application we will create in this tutorial notebook is analogous to the native BOINC application we create in this tutorial notebook, and thus reading that tutorial notebook is also recommended.

Step 2: Compiling the `BOINC` wrapper app for your platform [Back to top]¶

$$\label{compiling_wrapper_app}$$

In [ ]:

# Step 2: Compiling the BOINC wrapper app
# Step 2.a: Load needed Python modules
import os,sys

# Step 2.b: Add NRPy's root directory to the sys.path()
sys.path.append("..")

# Step 2.c: Load NRPy+'s command line helper module
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface

# Step 2.d: Set the path to the BOINC source code
path_to_boinc = "~/bhah/boinc"
current_path  = os.getcwd()

# Step 2.e: Check the platform and adjust the compilation command accordingly
if sys.platform == "linux":
    wrapper_compile = "make"

elif sys.platform == "darwin":
    wrapper_compile = "source BuildMacWrapper.sh"
    
else:
    print("Unsupported platform: "+sys.platform)
    sys.exit(1)

# Step 2.f: Compile the wrapper app
!cd $path_to_boinc/samples/wrapper && $wrapper_compile

# Step 2.g: Copy the wrapper app to the current working directory
!cp $path_to_boinc/samples/wrapper/wrapper $current_path

Step 3: Using the `BOINC` wrapper app [Back to top]¶

$$\label{using_wrapper_app}$$

Once we have everything set up, using the wrapper app is as simple as running

$: ./wrapper

The following steps will describe how to set the configuration files so that the wrapper app works as you expect it to.

Step 3.a: The main application [Back to top]¶

$$\label{the_main_application}$$

Let us start by writing a simple application which we will run using the BOINC wrapper app. In order for us to be able to see some additional configuration features of the wrapper app, we will make our main application slightly more complicated than a simple "Hello World!" program.

This application takes any number of command line arguments and then prints them to stdout, stderr, and an output text file.

In [ ]:

%%writefile simple_app.c
// Step 0: Load all the necessary C header files
#include <stdio.h>
#include <stdlib.h>

// Program description: this program is just a slightly
//                      more complicated version of the
//                      "Hello World!" program, where
//                      we will be taking some command
//                      line inputs and printing them to
//                      stdout, stderr, and an output file.
int main( int argc, char** argv ) {

  // Step 1: Check correct usage
  if( argc == 1 ) {
    fprintf(stderr,"(ERROR) Correct usage is ./simple_app <command_line_arguments>\n");
    exit(1);
  }

  // Step 2: Print all command line arguments to
  //         stdout, stderr, and an output file
  //
  // Step 2.a: Open the output file
  // Step 2.a.i: Set the output file name
  char filename[100] = "output_file.txt";
  // Step 2.a.ii: Open the file
  FILE* filept = fopen(filename,"w");
  // Step 2.a.iii: Check everything is OK
  if( !filept ) {
    fprintf(stderr,"(ERROR) Could not open file %s\n.",filename);
    exit(1);
  }

  // Step 2.b: Print an information line
  fprintf(stdout,"(INFO) Got the following command line arguments:");
  fprintf(stderr,"(INFO) Got the following command line arguments:");
  fprintf(filept,"(INFO) Got the following command line arguments:");
  
  // Step 2.c: Loop over the command line arguments, printing
  //           them to stdout, stderr, and our output file
  for(int i=1;i<argc;i++) {
    fprintf(stdout," %s",argv[i]);
    fprintf(stderr," %s",argv[i]);
    fprintf(filept," %s",argv[i]);
  }

  // Step 2.d: Add a line break to the output
  fprintf(stdout,"\n");
  fprintf(stderr,"\n");
  fprintf(filept,"\n");

  // Step 2.d: Close the output file
  fclose(filept);

  // All done!
  return 0;
}

Step 3.b: Compiling the main application [Back to top]¶

$$\label{compiling_the_main_application}$$

We now compile the main application using NRPy+'s cmdline_helper module.

In [ ]:

cmd.C_compile("simple_app.c","simple_app")

Step 3.c: The `job.xml` file [Back to top]¶

$$\label{job_xml}$$

Let's see what happens if we try running the wrapper app:

In [ ]:

!rm -f job.xml
cmd.Execute("wrapper")

As can be seen above, the BOINC wrapper application requests an input file, job.xml, to be present in the current working directory. We will now set up a job.xml file for the wrapper app in a way that it works correctly with our simple_app. A job.xml has the following syntax:

<job_desc>
    <task>
        ...task_options...
    </task>
    ...additional_options...
</job_desc>

All the configurations for the wrapper application are enclosed by the job_desc environment. To configure the wrapper to work with our specific application, we provide the task_options, while additional_options can be provided for additional configuration, as we will see.

Step 3.c.i: A very simple `job.xml` file [Back to top]¶

$$\label{simple_job_xml}$$

First, let us start with a very basic configuration: let us ask the wrapper to run our simple application using the command line arguments 1 2 3 4 testing hello world 4 3 2 1. This is achieved with the following job.xml file:

In [ ]:

%%writefile job.xml
<job_desc>
    <task>
      <application>simple_app</application>
      <command_line>1 2 3 4 testing hello world 4 3 2 1</command_line>
    </task>
</job_desc>

Let us now copy everything into a new, fresh directory and run our wrapper application.

In [ ]:

!rm -rf wrapper_app_test
cmd.mkdir("wrapper_app_test")
!cp wrapper simple_app job.xml wrapper_app_test && cd wrapper_app_test && ./wrapper && ls

Note that after execution, we see the output (INFO) Got the following command line arguments: 1 2 3 4 testing hello world 4 3 2 1 printed to stdout. If we examine the file output_file.txt, we will see the same output:

In [ ]:

!cat output_file.txt

The stderr.txt file is automatically generated by the BOINC wrapper app, and it contains all the output which was sent to stderr. We see that while we also have the expected output in it, there is also some additional information which was generated by the wrapper app:

In [ ]:

!cat stderr.txt

Aditionally, we see that the wrapper application has created two additional files: the boinc_finish_called and the wrapper_checkpoint.txt. For our purposes, the wrapper_checkpoint.txt file is irrelevant, so we will ignore it for now. The boinc_finish_called file contains the numerical value returned by our program, simple_app. As is usual in C, if the return value is 0, then the execution was successful, while a non-zero value indicates an error:

In [ ]:

!cat boinc_finish_called

Step 3.c.ii: Redirecting and zipping output files [Back to top]¶

$$\label{job_xml_output_redirect_and_zip}$$

Now that we have seen the simplest possible case, let us look at something slightly more complicated. The following job.xml file asks the wrapper app to perform the following tasks:

Run the simple_app application with command line arguments 1 2 3 4 testing hello world 4 3 2 1
Redirect all stdout output to the file simple_app.out
Redirect all stderr output to the file simple_app.err
Zip all the output files we have seen before into a single file: output.zip

In [ ]:

%%writefile job.xml
<job_desc>
    <task>
      <application>simple_app</application>
      <command_line>1 2 3 4 testing hello world 4 3 2 1</command_line>
      <stdout_filename>simple_app.out</stdout_filename>
      <stderr_filename>simple_app.err</stderr_filename>
    </task>
    <zip_output>
      <zipfilename>output.zip</zipfilename>
      <filename>simple_app.out</filename>
      <filename>simple_app.err</filename>
      <filename>output_file.txt</filename>
      <filename>boinc_finish_called</filename>
      <filename>wrapper_checkpoint.txt</filename>
    </zip_output>
</job_desc>

Now let us see what happens when we run the wrapper app:

In [ ]:

!rm -rf wrapper_app_test
cmd.mkdir("wrapper_app_test")
!cp wrapper simple_app job.xml wrapper_app_test && cd wrapper_app_test && ./wrapper && ls

Notice that we now have the output files simple_app.out and simple_app.err, as expected. The file stderr.txt is still present, by default. We also have all our output files neatly collected into a single zip file, output.zip. Note that zipping the output is not done with the goal of reducing the overall size of the output, but because it easier to communicate the output files back to the BOINC server.

Step 4: Output this notebook to $\LaTeX$-formatted PDF file [Back to top]¶

$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp.pdf (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

In [ ]:

!cp ../latex_nrpy_style.tplx .
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-BlackHolesAtHome-BOINC_applications-Using_the_WrapperApp")
!rm -f latex_nrpy_style.tplx

BlackHoles@Home Tutorial: Creating a BOINC app using the WrapperApp¶

Author: Leo Werneck¶

This tutorial notebook demonstrates how to write a program that runs in the BOINC infrastructure using the WrapperApp¶

WARNING: this tutorial notebook is currently incompatible with Windows

Introduction:¶

Additional Reading Material:¶

Table of Contents¶

Step 1: Introduction [Back to top]¶

Step 2: Compiling the BOINC wrapper app for your platform [Back to top]¶

Step 3: Using the BOINC wrapper app [Back to top]¶

Step 3.a: The main application [Back to top]¶

Step 3.b: Compiling the main application [Back to top]¶

Step 3.c: The job.xml file [Back to top]¶

Step 3.c.i: A very simple job.xml file [Back to top]¶

Step 3.c.ii: Redirecting and zipping output files [Back to top]¶

Step 4: Output this notebook to $\LaTeX$-formatted PDF file [Back to top]¶

BlackHoles@Home Tutorial: Creating a `BOINC` app using the `WrapperApp`¶

This tutorial notebook demonstrates how to write a program that runs in the `BOINC` infrastructure using the `WrapperApp`¶

Step 2: Compiling the `BOINC` wrapper app for your platform [Back to top]¶

Step 3: Using the `BOINC` wrapper app [Back to top]¶

Step 3.c: The `job.xml` file [Back to top]¶

Step 3.c.i: A very simple `job.xml` file [Back to top]¶