#!/usr/bin/env python # coding: utf-8 # # # # # `cmdline_helper.py`: Multi-platform command-line helper functions # ## Authors: Brandon Clark & Zach Etienne # # ## The notebook presents [cmdline_helper.py](../edit/cmdline_helper.py), a multi-platform Python module for interacting with the command-line within the NRPy+ tutorial framework. It provides functions for checking if an executable exists, compiling and executing C code, executing input strings, and file manipulations such as deletion or directory creation. Its functionality ensures compatibility across Linux, Windows, and Mac OS command-line interfaces. # # **Notebook Status:** Self-Validated # # **Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented. (TODO)** # # ### NRPy+ Python Module associated with this notebook: [cmdline_helper.py](../edit/cmdline_helper.py) # # ## Introduction: # Throughout the NRPy+ tutorial, there are a handful of modules that require interaction with the command line, to compile C code, manipulate files, execute code, etc. This module serves as a reference for Python functions that exist in [cmdline_helper](../edit/cmdline_helper.py), which is designed to be compatible with Linux, Windows, and Mac OS command line interfaces. # # # # Table of Contents # $$\label{toc}$$ # # This notebook is organized as follows # # 1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules # 1. [Step 2](#functions): The Functions # 1. [Step 2.a](#checkexec): **`check_executable_exists()`** # 1. [Step 2.b](#compile): **`C_compile()`** and **`new_C_compile()`** # 1. [Step 2.c](#execute): **`Execute()`** # 1. [Step 2.d](#output): **`Execute_input_string()`** # 1. [Step 2.e](#delete): **`delete_existing_files()`** & **`mkdir()`** # 1. [Step 3](#code_validation): Code Validation against `cmdline_helper.py`NRPy+ module # 1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file # # # # Step 1: Initialize core NRPy+ modules \[Back to [top](#toc)\] # $$\label{initializenrpy}$$ # # Let's start by importing the necessary Python modules. # In[1]: get_ipython().run_cell_magic('writefile', 'cmdline_helper-validation.py', '# As documented in the NRPy+ tutorial notebook\n# Tutorial-cmdline_helper.ipynb, this Python script\n# provides a multi-platform means to run executables,\n# remove files, and compile code.\n\n# Basic functions:\n# check_executable_exists(): Check to see whether an executable exists.\n# Error out or return False if it does not exist;\n# return True if executable exists in PATH.\n# C_compile(): Compile C code using gcc.\n# Execute(): Execute generated executable file, using taskset\n# if available. Calls Execute_input_string() to\n# redirect output from stdout & stderr to desired\n# destinations.\n# Execute_input_string(): Executes an input string and redirects\n# output from stdout & stderr to desired destinations.\n# delete_existing_files(file_or_wildcard):\n# Runs del file_or_wildcard in Windows, or\n# rm file_or_wildcard in Linux/MacOS\n\n# Authors: Brandon Clark\n# Zach Etienne\n# zachetie **at** gmail **dot* com\n# Kevin Lituchy\n\nimport io, os, shlex, subprocess, sys, time, multiprocessing, getpass, platform, glob\n') # # # # Step 2: The Functions \[Back to [top](#toc)\] # $$\label{functions}$$ # # # # ## Step 2.a: `check_executable_exists()` \[Back to [top](#toc)\] # $$\label{checkexec}$$ # # `check_executable_exists()` takes the required string `exec_name` (i.e., the name of the executable) as its first input. Its second input is the optional boolean `error_if_not_found`, which defaults to `True` (so that it exits with an error if the executable is not found). # # `check_executable_exists()` returns `True` if the executable exists, `False` if the executable does not exist, and `error_if_not_found` is set to `False`. # In[2]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n# check_executable_exists(): Check to see whether an executable exists.\n# Error out or return False if it does not exist;\n# return True if executable exists in PATH.\ndef check_executable_exists(exec_name, error_if_not_found=True):\n cmd = "where" if os.name == "nt" else "which"\n try:\n subprocess.check_output([cmd, exec_name])\n except subprocess.CalledProcessError:\n if error_if_not_found:\n print("Sorry, cannot execute the command: " + exec_name)\n sys.exit(1)\n else:\n return False\n return True\n') # # # ## Step 2.b: **`C_compile()`** and **`new_C_compile()`** \[Back to [top](#toc)\] # $$\label{compile}$$ # # ### `C_compile()` # # The `C_compile()` function takes the following inputs as **strings** # * Path name to the generated C_file, `"main_C_output_path"`, and # * Name of the executable playground file, `"main_C_output_file"`. # # The `C_compile()` function first checks for a ***gcc compiler***, which is a must when compiling C code within the NRPy+ tutorial. The function then removes any existing executable file. After that, the function constructs a script `compile_string` to run the compilation based on the function inputs and the operating system (OS) in use. # # Finally, it runs the actual compilation, by passing the compilation script `compile_string` on to the `Execute_input_string()` function, see [Step 2.d](#output). # In[3]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n# C_compile(): Write a function to compile the Main C code into an executable file\ndef C_compile(main_C_output_path, main_C_output_file, compile_mode="optimized", custom_compile_string="", additional_libraries=""):\n print("Compiling executable...")\n # Step 1: Check for gcc compiler\n check_executable_exists("gcc")\n\n if additional_libraries != "":\n additional_libraries = " " + additional_libraries\n\n # Step 2: Delete existing version of executable\n if os.name == "nt":\n main_C_output_file += ".exe"\n delete_existing_files(main_C_output_file)\n\n # Step 3: Compile the executable\n if compile_mode=="safe":\n compile_string = "gcc -std=gnu99 -O2 -g -fopenmp "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n if not os.path.isfile(main_C_output_file):\n # Step 3.A: Maybe gcc is actually clang in disguise (as in MacOS)?!\n # https://stackoverflow.com/questions/33357029/using-openmp-with-clang\n print("Most safe failed. Probably on MacOS. Replacing -fopenmp with -fopenmp=libomp:")\n compile_string = "gcc -std=gnu99 -O2 -fopenmp=libomp "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n if not os.path.isfile(main_C_output_file):\n print("Sorry, compilation failed")\n sys.exit(1)\n elif compile_mode=="icc":\n check_executable_exists("icc")\n compile_string = "icc -std=gnu99 -O2 -xHost -qopenmp -unroll "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n if not os.path.isfile(main_C_output_file):\n print("Sorry, compilation failed")\n sys.exit(1)\n elif compile_mode=="custom":\n Execute_input_string(custom_compile_string, os.devnull)\n # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n if not os.path.isfile(main_C_output_file):\n print("Sorry, compilation failed")\n sys.exit(1)\n elif compile_mode=="optimized":\n compile_string = "gcc -std=gnu99 -Ofast -fopenmp -march=native -funroll-loops "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n if not os.path.isfile(main_C_output_file):\n # Step 3.A: Revert to more compatible gcc compile option\n print("Most optimized compilation failed. Removing -march=native:")\n compile_string = "gcc -std=gnu99 -Ofast -fopenmp -funroll-loops "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n if not os.path.isfile(main_C_output_file):\n # Step 3.B: Revert to maximally compatible gcc compile option\n print("Next-to-most optimized compilation failed. Moving to maximally-compatible gcc compile option:")\n compile_string = "gcc -std=gnu99 -O2 "+str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm"+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n # Step 3.C: If there are still missing components within the compiler, say compilation failed\n if not os.path.isfile(main_C_output_file):\n print("Sorry, compilation failed")\n sys.exit(1)\n elif compile_mode=="emscripten":\n compile_string = "emcc -std=gnu99 -s -O3 -march=native -funroll-loops -s ALLOW_MEMORY_GROWTH=1 "\\\n +str(main_C_output_path)+" -o "+str(main_C_output_file)+" -lm "+additional_libraries\n Execute_input_string(compile_string, os.devnull)\n # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n # If there are still missing components within the compiler, say compilation failed\n if not os.path.isfile(main_C_output_file):\n print("Sorry, compilation failed.")\n sys.exit(1)\n else:\n print("Sorry, compile_mode = \\""+compile_mode+"\\" unsupported.")\n sys.exit(1)\n\n print("Finished compilation.")\n') # ### `new_C_compile()` # # The `new_C_compile()` function first constructs a `Makefile` from all functions registered to `outputC`'s `outC_function_dict`, and then attempts to compile the code using a parallel `make`. If that fails (e.g., due to the `make` command not being found or optimizations not being supported), it instead constructs a script that builds the code in serial with most compiler optimizations disabled. # # `new_C_compile()` takes the following inputs: # * `Ccodesrootdir`: Path to the generated C code, `Makefile`, and executable. # * `exec_name`: File name of executable. # * `uses_free_parameters_h=False` (optional): Will cause the compilation of `main.c` to depend on `free_parameters.h`, such that if the latter is updated `main.c` will be recompiled due to `make` being run. # * `compiler_opt_option="fast"` (optional): Optimization option for compilation. Choose from `fast`, `fastdebug`, or `debug`. The latter two will enable the `-g` flag. # * `addl_CFLAGS=None` (optional): A list of additional compiler flags. E.g., `[-funroll-loops,-ffast-math]`. # * `addl_libraries=None` (optional): A list of additional libraries against which to link. # * `mkdir_Ccodesrootdir=True` (optional): Attempt to make the `Ccodesrootdir` directory if it doesn't exist. If it does exist, this has no effect. # * `CC="gcc"` (optional): Choose the compiler. `"gcc"` should be used for the C compiler and `"g++"` for C++. FIXME: Other compilers may throw warnings due to default compilation flags being incompatible. # * `attempt=1` (optional, **do not touch**): An internal flag used to recursively call this function again in case the `Makefile` build fails (runs a shell script in serial with debug options disabled, as a backup). # In[4]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n\nfrom outputC import construct_Makefile_from_outC_function_dict\ndef new_C_compile(Ccodesrootdir, exec_name, uses_free_parameters_h=False,\n compiler_opt_option="fast", addl_CFLAGS=None,\n addl_libraries=None, mkdir_Ccodesrootdir=True, CC="gcc", attempt=1):\n check_executable_exists("gcc")\n use_make = check_executable_exists("make", error_if_not_found=False)\n\n construct_Makefile_from_outC_function_dict(Ccodesrootdir, exec_name, uses_free_parameters_h,\n compiler_opt_option, addl_CFLAGS,\n addl_libraries, mkdir_Ccodesrootdir, use_make, CC=CC)\n orig_working_directory = os.getcwd()\n os.chdir(Ccodesrootdir)\n if use_make:\n Execute_input_string("make -j" + str(int(multiprocessing.cpu_count()) + 2), os.devnull)\n else:\n Execute_input_string(os.path.join("./", "backup_script_nomake.sh"))\n os.chdir(orig_working_directory)\n\n if not os.path.isfile(os.path.join(Ccodesrootdir, exec_name)) and attempt == 1:\n print("Optimized compilation FAILED. Removing optimizations (including OpenMP) and retrying with debug enabled...")\n # First clean up object files.\n filelist = glob.glob(os.path.join(Ccodesrootdir, "*.o"))\n for file in filelist:\n os.remove(file)\n # Then retry compilation (recursion)\n new_C_compile(Ccodesrootdir, exec_name, uses_free_parameters_h,\n compiler_opt_option="debug", addl_CFLAGS=addl_CFLAGS,\n addl_libraries=addl_libraries, mkdir_Ccodesrootdir=mkdir_Ccodesrootdir, CC=CC, attempt=2)\n if not os.path.isfile(os.path.join(Ccodesrootdir, exec_name)) and attempt == 2:\n print("Sorry, compilation failed")\n sys.exit(1)\n print("Finished compilation.")\n') # # # ## Step 2.c: `Execute()` \[Back to [top](#toc)\] # $$\label{execute}$$ # # The `Execute()` function takes the following inputs as **strings** # * Name of the executable file, `main_C_output_file`, # * **(Optional):** Any necessary arguments associated with the executable file output, `executable_output_arguments`, and # * **(Optional):** Name of a file to store output during execution, `executable_output_file_name`. # # The `Execute()` function first removes any existing output files. It then begins to construct the script `execute_string` in order to execute the executable file that has been generated by the `C_compile()` function. `execute_string` is built based on the function inputs and the operating system (OS) in use. # # Finally, it runs the actual execution, by passing the execution script `execute_string` on to the `Execute_input_string()` function, see [Step 2.d](#output). # In[5]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n\n# Execute(): Execute generated executable file, using taskset\n# if available. Calls Execute_input_string() to\n# redirect output from stdout & stderr to desired\n# destinations.\ndef Execute(executable, executable_output_arguments="", file_to_redirect_stdout=os.devnull, verbose=True):\n # Step 1: Delete old version of executable file\n if file_to_redirect_stdout != os.devnull:\n delete_existing_files(file_to_redirect_stdout)\n\n # Step 2: Build the script for executing the desired executable\n execute_string = ""\n # When in Windows...\n # https://stackoverflow.com/questions/1325581/how-do-i-check-if-im-running-on-windows-in-python\n if os.name == "nt":\n # ... do as the Windows do\n # https://stackoverflow.com/questions/49018413/filenotfounderror-subprocess-popendir-windows-7\n execute_prefix = "cmd /c " # Run with cmd /c executable [options] on Windows\n else:\n execute_prefix = "./" # Run with ./executable [options] on Linux & Mac\n taskset_exists = check_executable_exists("taskset", error_if_not_found=False)\n if taskset_exists:\n execute_string += "taskset -c 0"\n on_4900hs = False\n if platform.system() == "Linux" and \\\n "AMD Ryzen 9 4900HS" in str(subprocess.check_output("cat /proc/cpuinfo", shell=True)):\n on_4900hs = True\n if not on_4900hs and getpass.getuser() != "jovyan": # on mybinder, username is jovyan, and taskset -c 0 is the fastest option.\n # If not on mybinder and taskset exists:\n has_HT_cores = False # Does CPU have hyperthreading cores?\n if platform.processor() != \'\': # If processor string returns null, then assume CPU does not support hyperthreading.\n # This will yield correct behavior on ARM (e.g., cell phone) CPUs.\n has_HT_cores=True\n if has_HT_cores == True:\n # NOTE: You will observe a speed-up by using only *PHYSICAL* (as opposed to logical/hyperthreading) cores:\n N_cores_to_use = int(multiprocessing.cpu_count()/2) # To account for hyperthreading cores\n else:\n N_cores_to_use = int(multiprocessing.cpu_count()) # Use all cores if none are hyperthreading cores.\n # This will happen on ARM (e.g., cellphone) CPUs\n for i in range(N_cores_to_use-1):\n execute_string += ","+str(i+1)\n if on_4900hs:\n execute_string = "taskset -c 1,3,5,7,9,11,13,15"\n execute_string += " "\n execute_string += execute_prefix+executable+" "+executable_output_arguments\n\n # Step 3: Execute the desired executable\n Execute_input_string(execute_string, file_to_redirect_stdout, verbose)\n') # # # ## Step 2.d: `Execute_input_string()` \[Back to [top](#toc)\] # $$\label{output}$$ # # The `Execute_input_string()` function takes the following inputs as strings # * The script to be executed, `input_string`, and # * An output file name for any needed redirects, `executable_output_file_name`. # # The `Execute_input_string()` executes a script, outputting `stderr` to the screen and redirecting any additional outputs from the executable to the specified `executable_output_file_name`. # # In[6]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n# Execute_input_string(): Executes an input string and redirects\n# output from stdout & stderr to desired destinations.\ndef Execute_input_string(input_string, file_to_redirect_stdout=os.devnull, verbose=True):\n\n if verbose:\n print("(EXEC): Executing `"+input_string+"`...")\n start = time.time()\n # https://docs.python.org/3/library/subprocess.html\n if os.name != \'nt\':\n args = shlex.split(input_string)\n else:\n args = input_string\n\n # https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command\n filename = "tmp.txt"\n with io.open(filename, \'w\') as writer, io.open(filename, \'rb\', buffering=-1) as reader, io.open(file_to_redirect_stdout, \'wb\') as rdirect:\n process = subprocess.Popen(args, stdout=rdirect, stderr=writer)\n while process.poll() is None:\n # https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write/21689447\n sys.stdout.write(reader.read().decode(\'utf-8\'))\n time.sleep(0.2)\n # Read the remaining\n sys.stdout.write(reader.read().decode(\'utf-8\'))\n delete_existing_files(filename)\n end = time.time()\n if verbose:\n print("(BENCH): Finished executing in "+\'{:#.2f}\'.format(round(end-start, 2))+" seconds.")\n') # # # ## Step 2.e: `delete_existing_files()` & `mkdir()` \[Back to [top](#toc)\] # $$\label{delete}$$ # # The `delete_existing_files()` function takes a string, `file_or_wildcard`, as input. # # `delete_existing_files()` deletes any existing files that match the pattern given by `file_or_wildcard`. Deleting files is important when running the same code multiple times, ensuring that you're not reusing old data from a previous run, or seeing the same plot from a previous output. # # The `mkdir()` function makes a directory if it does not yet exist. It passes the input string "newpath" through `os.path.join()` to ensure that forward slashes are replaced by backslashes in Windows environments. # In[7]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n# delete_existing_files(file_or_wildcard):\n# Runs del file_or_wildcard in Windows, or\n# rm file_or_wildcard in Linux/MacOS\ndef delete_existing_files(file_or_wildcard):\n delete_string = ""\n if os.name == "nt":\n delete_string += "del " + file_or_wildcard\n else:\n delete_string += "rm -f " + file_or_wildcard\n os.system(delete_string)\n\n# https://stackoverflow.com/questions/1274405/how-to-create-new-folder\ndef mkdir(newpath):\n if not os.path.exists(os.path.join(newpath)):\n os.makedirs(os.path.join(newpath))\n') # In[8]: get_ipython().run_cell_magic('writefile', '-a cmdline_helper-validation.py', '\n# TO BE RUN ONLY FROM nrpytutorial or nrpytutorial/subdir/\ndef output_Jupyter_notebook_to_LaTeXed_PDF(notebookname, verbose=True):\n in_nrpytutorial_rootdir = os.getcwd().split("/")[-1] == "nrpytutorial"\n if sys.version_info[0] == 3:\n location_of_template_file = "."\n if not in_nrpytutorial_rootdir:\n location_of_template_file = ".."\n Execute_input_string(r"jupyter nbconvert --to latex --template="\n +os.path.join(location_of_template_file, "nbconvert_latex_settings")\n +r" --log-level=\'WARN\' "+notebookname+".ipynb",verbose=False)\n else:\n Execute_input_string(r"jupyter nbconvert --to latex --log-level=\'WARN\' "+notebookname+".ipynb",verbose=False)\n for _i in range(3): # _i is an unused variable.\n Execute_input_string(r"pdflatex -interaction=batchmode "+notebookname+".tex",verbose=False)\n delete_existing_files(notebookname+".out "+notebookname+".aux "+notebookname+".log")\n if verbose:\n import textwrap\n wrapper = textwrap.TextWrapper(initial_indent="",subsequent_indent=" ",width=75)\n print(wrapper.fill("Created "+notebookname+".tex, and compiled LaTeX file to PDF file "+notebookname+".pdf"))\n') # # # # Step 3: Code Validation against `cmdline_helper.py` NRPy+ module \[Back to [top](#toc)\] # $$\label{code_validation}$$ # # To validate the code in this tutorial we check for agreement between the files # # 1. `cmdline_helper-validation.py` (written in this tutorial) and # 1. the NRPy+ [cmdline_helper.py](../edit/cmdline_helper.py) module # # In[1]: import difflib import sys def trim_lines(lines): new_lines=[] for line in lines: x = line.rstrip() if x != '': new_lines += [x] return new_lines print("Printing difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.") # Open the files to compare with open("cmdline_helper.py") as file1, open("cmdline_helper-validation.py") as file2: # Read the lines of each file file1_lines = trim_lines(file1.readlines()) file2_lines = trim_lines(file2.readlines()) num_diffs = 0 for line in difflib.unified_diff(file1_lines, file2_lines, fromfile="cmdline_helper.py", tofile="cmdline_helper-validation.py"): sys.stdout.writelines(line) num_diffs = num_diffs + 1 if num_diffs == 0: print("No difference. TEST PASSED!") else: print("ERROR: Disagreement found with .py file. See differences above.") sys.exit(1) # # # # Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\] # $$\label{latex_pdf_output}$$ # # The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename # [Tutorial-cmdline_helper.pdf](Tutorial-cmdline_helper.pdf). (Note that clicking on this link may not work; you may need to open the PDF file through another means.) # In[1]: import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-cmdline_helper")