BOINC native applications¶BOINC infrastructure, as well as how to convert NRPy+ code into a BOINC application.¶The BlackHoles@Home project allows users to volunteer CPU time so a large number of binary black hole simulations can be performed. The objective is to create an extensive catalog of gravitational waveforms, which can be used by observatories such as LIGO, VIRGO, and, in the future, LISA in order to infer what was the source of a detected gravitational wave.
BlackHoles@Home is destined to run on the BOINC infrastructure (alongside Einstein@Home and many other great projects), enabling anyone with a computer to contribute to the construction of the largest numerical relativity gravitational wave catalogs ever produced.
This tutorial explains how to use the BOINC wrapper application to run a simple program. The structure of this notebook is as follows:
A native BOINC application is a program that directly interfaces with the BOINC API. During compilation, we link the executable with the BOINC libraries, thus creating an executable that can run in the BOINC infrastructure. If you have not yet compiled the BOINC libraries, please read the tutorial notebook on how to do so.
This tutorial notebook aims at teaching you two key concepts:
BOINC applications by handNRPy+ code into a BOINC applicationWe will be using the NRPy+ code generated by the Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb NRPy+ tutorial notebook as an example.
# Step 2: Load Python/NRPy+ modules and perform basic setup
# Step 2.a: Load needed Python modules
import os,sys
# Step 2.b: Add NRPy's root directory to the sys.path()
sys.path.append("..")
# Step 2.c: Load NRPy+'s command line helper module
import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface
# Step 2.d: Set the path to the BOINC source code
path_to_boinc = "/Users/werneck/bhah/boinc"
boinc_api_dir = os.path.join(path_to_boinc,"api")
boinc_lib_dir = os.path.join(path_to_boinc,"lib")
boinc_zip_dir = os.path.join(path_to_boinc,"zip")
current_path = os.getcwd()
# Step 2.e: Adjust the compiler and compilation flags based on the system
# Step 2.e.i: Set the C++ compiler flags
global CXX_compiler,CXXFLAGS,LDFLAGS
CXXFLAGS = "-fopenmp -march=native -Ofast -funroll-loops "
CXXFLAGS += "-I%s -I%s -I%s "%(boinc_api_dir,boinc_lib_dir,boinc_zip_dir)
LDFLAGS = "-L%s -L%s -L%s -lboinc_api -lboinc -lboinc_zip "%(boinc_api_dir,boinc_lib_dir,boinc_zip_dir)
# Step 2.e.ii: Set the C++ compiler
if sys.platform == 'linux':
CXX_compiler = "g++ "
elif sys.platform == 'darwin':
# Set path to Clang compiler installed with homebrew
path_to_llvm = "/usr/local/opt/llvm/"
path_to_clangpp = os.path.join(path_to_llvm,"bin","clang++")
path_to_clang_include = os.path.join(path_to_llvm,"include")
path_to_clang_library = os.path.join(path_to_llvm,"lib")
CXX_compiler = path_to_clangpp+" "
CXXFLAGS += "-I%s "%(path_to_clang_include)
LDFLAGS += "-L%s "%(path_to_clang_library)
else:
print("Error: platform %s is currently not supported."%sys.platform)
sys.exit(1)
BOINC native application [Back to top]¶A native BOINC application can be created by:
BOINC API header file by adding #include "boinc_api.h" to your codeboinc_init() function at the beginning of the main functionboinc_finish(0) instead of return 0 at the end of the main functionThe boinc_finish(err_code) function should also be used instead of the exit(err_code) function in case you need the program to stop running and return an error code.
BOINC native application [Back to top]¶We now provide one of the simplest possible examples of a BOINC application, with minimal error handling included. This application:
BOINC environmentBOINC environment was initialized correctlyBOINC environment and terminates%%writefile simplest_boinc_app.cpp
// Step 0: Basic includes
// Step 0.a: Basic C++ header files
#include <iostream>
// Step 0.b: BOINC api header file
#include "boinc_api.h"
// Program description: this is one of the simplest BOINC
// applications that can be written.
// We start the BOINC environment
// by calling the boinc_init() function,
// check everything is OK (erroring out
// if it isn't), print a message to the
// user, and terminate using a call to
// the boinc_finish() function.
int main() {
// Step 1: Initialize the BOINC environment with boinc_init()
int status = boinc_init();
// Step 2: Check everything is OK, error out if not
if( status != 0 ) {
fprintf(stderr,"ERROR: boinc_init() returned a non-zero value: %d\n",status);
boinc_finish(status);
}
// Step 3: Print a message to the user
printf("Hello BOINC!\n");
// Step 4: Terminate the program with boinc_finish()
boinc_finish(0);
}
Let us now compile and run the application:
compile_string = CXX_compiler+CXXFLAGS+"simplest_boinc_app.cpp -o simplest_boinc_app "+LDFLAGS
!rm -rf simplest_boinc_app_test_dir
cmd.mkdir("simplest_boinc_app_test_dir")
!mv simplest_boinc_app.cpp simplest_boinc_app_test_dir
!cd simplest_boinc_app_test_dir && $compile_string && ./simplest_boinc_app && ls
Note that similar to using the BOINC WrapperApp, we have produced the output files boinc_finish_called and stderr.txt, even though we did not explicitly generate them in our program. This is because the BOINC API generates these files automatically for us. If we take a look at the contents of the files, we see that the boinc_finish_called simply contains the integer argument of the boinc_finish() function, while the stderr.txt contains some basic information stating that we are running the application outside of the BOINC infrastructure and that the boinc_finish() function was called:
!cd simplest_boinc_app_test_dir && cat boinc_finish_called stderr.txt
NRPy+ code into a BOINC native app [Back to top]¶We now provide a script for converting an existing NRPy+ code into a BOINC application. Note that it is relatively easy to convert an existing C or C++ application into a native BOINC application. Unless you want to manually create a wrapper function that calls your C code, it is recommended to compile your code using a C++ compiler instead. In the case of NRPy+ applications, this can be achieved by simply adding:
#ifdef __cplusplus
# define restrict __restrict__
#endif
to the very top of the main application source code file, changing the file extension from .c to .cpp/.cc/.C, and then compiling the code using the flag -std=c++11. We also need to replace all calls to the exit() function with calls to the boinc_finish() function.
The following script takes care of that:
# Converting NRPy+ code into a BOINC app
# Description: This function reads a NRPy+ source code
# one line at a time and copies them into
# a new file which is compatible with the
# BOINC infrastructure.
def NRPy_to_BOINC(input_file,output_file):
# Step 1: Open the NRPy+ input file
with open(input_file,"r") as file:
# Step 2: Create the BOINC application
# Step 2.a: Print a message to the user describing
# some basic changes. Add the "restrict"
# keyword so that it is compatible with
# C++, which is required by BOINC.
output_string = """
//****************************************************************
// This NRPy+ code has been converted to work with the
// BOINC infrastructure. Please compile it with a C++
// compiler. Don't forget to add the -std=c++11 flag.
#ifdef __cplusplus
# define restrict __restrict__
#endif
// .--------------------.
// | BOINC HEADER FILES |
// .--------------------.
// Note: You can comment out (or remove) the boinc_zip.h header
// if you do not plan on using the BOINC zip functions.
#include \"boinc_api.h\"
#include \"boinc_zip.h\"
//****************************************************************
"""
# Step 2.b: Loop over the file, adding calls to
# the BOINC API functions as needed.
indent = " "
for line in file:
# Step 2.b.i: After the main() function, add a call to the boinc_init() function
if "int main" in line:
output_string += "\n"+line+"\n"+indent+"boinc_init();\n"
# Step 2.b.ii: Replace return 0; with boinc_finish(0);
elif "return 0" in line:
output_string += indent+"boinc_finish(0);\n"
# Step 2.b.iii: Replace exit(err_code) function calls with boinc_finish(err_code)
elif "exit(" in line:
output_string += line.replace("exit","boinc_finish")
else:
# Step 2.b.iv: Otherwise, just copy the original source code
output_string += line
# Step 3: Write the output file
with open(output_file,"w") as file:
file.write(output_string)
Now let's convert a NRPy+ generated code into a BOINC code, compile it, and run it. We will take as an example the files obtained after running the Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb. Running the cell below will perform the following tasks:
nrpytutorial/BHAH)BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.c file, into a BOINC compatible applicationBOINC librariesWARNING: because this step involves generating the source code for the BSSN equations, running the cell below will take a few minutes.
# Run the Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb tutorial notebook
!pip install runipy > /dev/null
!rm -rf BSSN_Two_BHs_Collide_Ccodes out96*.txt out96*.png
!cd .. && runipy Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb && mv BSSN_Two_BHs_Collide_Ccodes BHAH
# Compute
NRPy_to_BOINC("BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.c","BSSN_Two_BHs_Collide_Ccodes/BrillLindquist_Playground.cpp")
compile_string = CXX_compiler+CXXFLAGS+"BrillLindquist_Playground.cpp -o ../BrillLindquist_Playground "+LDFLAGS
!cd BSSN_Two_BHs_Collide_Ccodes/ && $compile_string
!./BrillLindquist_Playground 96 16 2
We can now visualize the solution, just like with the regular NRPy+ code (the cell below contains code that was extracted from the Tutorial-Start_to_Finish-BSSNCurvilinear-Two_BHs_Collide.ipynb NRPy+ tutorial notebook):
## VISUALIZATION ANIMATION, PART 1: Generate PNGs, one per frame of movie ##
import numpy as np
from scipy.interpolate import griddata
import matplotlib.pyplot as plt
from matplotlib.pyplot import savefig
from IPython.display import HTML
import matplotlib.image as mgimg
import glob
import sys
from matplotlib import animation
outdir = "./"
globby = glob.glob(os.path.join(outdir,'out96-00*.txt'))
file_list = []
for x in sorted(globby):
file_list.append(x)
bound=1.4
pl_xmin = -bound
pl_xmax = +bound
pl_ymin = -bound
pl_ymax = +bound
for filename in file_list:
fig = plt.figure()
x,y,cf,Ham = np.loadtxt(filename).T #Transposed for easier unpacking
plotquantity = cf
plotdescription = "Numerical Soln."
plt.title("Black Hole Head-on Collision (conf factor)")
plt.xlabel("y/M")
plt.ylabel("z/M")
grid_x, grid_y = np.mgrid[pl_xmin:pl_xmax:300j, pl_ymin:pl_ymax:300j]
points = np.zeros((len(x), 2))
for i in range(len(x)):
# Zach says: No idea why x and y get flipped...
points[i][0] = y[i]
points[i][1] = x[i]
grid = griddata(points, plotquantity, (grid_x, grid_y), method='nearest')
gridcub = griddata(points, plotquantity, (grid_x, grid_y), method='cubic')
im = plt.imshow(gridcub, extent=(pl_xmin,pl_xmax, pl_ymin,pl_ymax))
ax = plt.colorbar()
ax.set_label(plotdescription)
savefig(os.path.join(filename+".png"),dpi=150)
plt.close(fig)
sys.stdout.write("%c[2K" % 27)
sys.stdout.write("Processing file "+filename+"\r")
sys.stdout.flush()
## VISUALIZATION ANIMATION, PART 2: Combine PNGs to generate movie ##
# https://stackoverflow.com/questions/14908576/how-to-remove-frame-from-matplotlib-pyplot-figure-vs-matplotlib-figure-frame
# https://stackoverflow.com/questions/23176161/animating-pngs-in-matplotlib-using-artistanimation
fig = plt.figure(frameon=False)
ax = fig.add_axes([0, 0, 1, 1])
ax.axis('off')
myimages = []
for i in range(len(file_list)):
img = mgimg.imread(file_list[i]+".png")
imgplot = plt.imshow(img)
myimages.append([imgplot])
ani = animation.ArtistAnimation(fig, myimages, interval=100, repeat_delay=1000)
ani.save(os.path.join(outdir,'BH_Head-on_Collision.mp4'), fps=5,dpi=150)
plt.close()
# Embed video based on suggestion:
# https://stackoverflow.com/questions/39900173/jupyter-notebook-html-cell-magic-with-python-variable
HTML("""
<video width="480" height="360" controls>
<source src=\""""+os.path.join(outdir,"BH_Head-on_Collision.mp4")+"""\" type="video/mp4">
</video>
""")
The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications.pdf (Note that clicking on this link may not work; you may need to open the PDF file through another means.)
!cp ../latex_nrpy_style.tplx .
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-BlackHolesAtHome-BOINC_applications-Native_applications")
!rm -f latex_nrpy_style.tplx