Notebook

Performing the azimutal integration from shell scripts¶

There are three main tools for performing azimuthal integration from the command line. Two of them specicalized for SAXS and WAXS and the last which is common to the GUI but we will see how to use it in command line mode (CLI).

They all require 3 mandatory parameters:

The PONI-file containing the geometry of the experimental setup
The number of bins used for the histogram
The list of input files

The three tools differ only by the default options selected to fit the best the needs of a given user community.

Azimuthal integration for powder diffraction¶

The pyFAI-waxs which stands for wide angle scattering produces integrated patterns in $2\theta$ units by default.

Note: while this cookbook is performed in a Jupyter notebook, all commands are run in the command line interpreter (Bash), not in Python.

In [1]:

pyFAI-waxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif

Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m	PONI= 5.394843e-02, 5.551600e-02m	rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
DirectBeamDist= 118.224mm	Center: x=1066.839, y=1036.336 pix	Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
Mask: None	Method: splitpixel
Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.xy,	 reading: 0.022s	 1D integration: 1.789s.

In [2]:

#Let's have a look at the header of the produces ascii file:
head -n 50 LaB6_29.4keV.xy

# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
# 
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
# 
# Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
#    Detector has a mask: False 
#    Detector has a dark current: False 
#    detector has a flat field: False 
# 
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
# 
# Headers of the input frame:
# {
# "info": {},
# "photometricInterpretation": 1,
# "rowsPerStrip": 2048,
# "nColumns": 2048,
# "compression": false,
# "sampleFormat": 3,
# "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
# "nRows": 2048,
# "colormap": null,
# "nBits": 32,
# "date": "Fri Mar  2 10:02:31 2018",
# "software": "fabio.tifimage",
# "compression_type": 1,
# "stripOffsets": [
# 1143
# ],
# "stripByteCounts": [
# 16777216
# ]
# }
# --> LaB6_29.4keV.xy
#       2th_deg             I 
  1.668855e-02    2.425633e+00
  5.006564e-02    2.804243e+00
  8.344272e-02    2.121904e+00
  1.168198e-01    2.734025e+00

In [3]:

pyFAI-waxs -h

usage: pyFAI-waxs [options] -p ponifile file1.edf file2.edf ...

Azimuthal integration for powder diffraction.

positional arguments:
  FILE                  Image files to integrate

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -p PONIFILE           PyFAI parameter file (.poni)
  -n NPT, --npt NPT     Number of points in radial dimension
  -w WAVELENGTH, --wavelength WAVELENGTH
                        wavelength of the X-Ray beam in Angstrom
  -e ENERGY, --energy ENERGY
                        energy of the X-Ray beam in keV
                        (hc=12.3984197396keV.A)
  -u DUMMY, --dummy DUMMY
                        dummy value for dead pixels
  -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
                        delta dummy value
  -m MASK, --mask MASK  name of the file containing the mask image
  -d DARK, --dark DARK  name of the file containing the dark current
  -f FLAT, --flat FLAT  name of the file containing the flat field
  -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
                        Polarization factor, from -1 (vertical) to +1
                        (horizontal), default is None for no correction,
                        synchrotrons are around 0.95
  --error-model ERROR_MODEL
                        Error model to use. Currently on 'poisson' is
                        implemented
  --unit UNIT           unit for the radial dimension: can be q_nm^-1, q_A^-1,
                        2th_deg, 2th_rad or r_mm
  --ext EXT             extension of the regrouped filename (.xy)
  --method METHOD       Integration method
  --multi               Average out all frame in a file before integrating
                        extracting variance, otherwise treat every single
                        frame
  --average AVERAGE     Method for averaging out: can be 'mean' (default),
                        'min', 'max' or 'median
  --do-2D               Perform 2D integration in addition to 1D

pyFAI-waxs is the script of pyFAI that allows data reduction (azimuthal
integration) for Wide Angle Scattering to produce X-Ray Powder Diffraction
Pattern with output axis in 2-theta space.

One options which may be of some interest is the --do-2D which allow to perform caking.

Azimuthal integration for small angle data¶

Small angle scattering usually prefer integration in a radial unit independant of the wavelength, typically $q = \frac{4 \pi sin(2\theta/2)}{\lambda}$ in inverse nanometers ($nm^{-1}$).

In [4]:

pyFAI-saxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif

Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m	PONI= 5.394843e-02, 5.551600e-02m	rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
DirectBeamDist= 118.224mm	Center: x=1066.839, y=1036.336 pix	Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
Mask: None	Method: splitpixel
Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.dat,	 reading: 0.022s	 1D integration: 1.792s.

In [5]:

head -n 50 LaB6_29.4keV.dat

# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
# 
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
# 
# Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
#    Detector has a mask: False 
#    Detector has a dark current: False 
#    detector has a flat field: False 
# 
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
# 
# Headers of the input frame:
# {
# "info": {},
# "photometricInterpretation": 1,
# "rowsPerStrip": 2048,
# "nColumns": 2048,
# "compression": false,
# "sampleFormat": 3,
# "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
# "nRows": 2048,
# "colormap": null,
# "nBits": 32,
# "date": "Fri Mar  2 10:02:31 2018",
# "software": "fabio.tifimage",
# "compression_type": 1,
# "stripOffsets": [
# 1143
# ],
# "stripByteCounts": [
# 16777216
# ]
# }
# --> LaB6_29.4keV.dat
#       q_nm^-1             I 
  4.278568e-02    2.427221e+00
  1.283570e-01    2.821874e+00
  2.139284e-01    2.101271e+00
  2.994998e-01    2.708885e+00

In [6]:

pyFAI-saxs -h

usage: pyFAI-saxs [options] -n 1000 -p ponifile file1.edf file2.edf ...

Azimuthal integration for SAXS users.

positional arguments:
  FILE                  Image files to integrate

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -p PONIFILE           PyFAI parameter file (.poni)
  -n NPT, --npt NPT     Number of points in radial dimension
  -w WAVELENGTH, --wavelength WAVELENGTH
                        wavelength of the X-Ray beam in Angstrom
  -e ENERGY, --energy ENERGY
                        energy of the X-Ray beam in keV
                        (hc=12.3984197396keV.A)
  -u DUMMY, --dummy DUMMY
                        dummy value for dead pixels
  -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
                        delta dummy value
  -m MASK, --mask MASK  name of the file containing the mask image
  -d DARK, --dark DARK  name of the file containing the dark current
  -f FLAT, --flat FLAT  name of the file containing the flat field
  -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
                        Polarization factor, from -1 (vertical) to +1
                        (horizontal), default is None for no correction,
                        synchrotrons are around 0.95
  --error-model ERROR_MODEL
                        Error model to use. Currently on 'poisson' is
                        implemented
  --unit UNIT           unit for the radial dimension: can be q_nm^-1, q_A^-1,
                        2th_deg, 2th_rad or r_mm
  --ext EXT             extension of the regrouped filename (.dat)
  --method METHOD       Integration method

pyFAI-saxs is the SAXS script of pyFAI that allows data reduction (azimuthal
integration) for Small Angle Scattering with output axis in q space.

Common tool for azimuthal intration¶

pyFAI-integrate offers a graphical user interface for integration which exposes most options from pyFAI. Once all options have been configured in the the windows, one can either launch the processing or save the configuration into a JSON file to be able to create script with all the options.

pyFAI-integrate GUI

To run pyFAI-integrate as a script, i.e. without the graphical interface, the --no-gui option is required:

In [7]:

pyFAI-integrate --no-gui -j azimint.json -o LaB6_29.4keV.cli  LaB6_29.4keV.tif

INFO:root:Enter, port=54321.
INFO:root:Enter.
start config ...
INFO:pyFAI.worker:Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m	PONI= 5.394843e-02, 5.551600e-02m	rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
DirectBeamDist= 118.224mm	Center: x=1066.839, y=1036.336 pix	Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
INFO:pyFAI.azimuthalIntegrator:AI.integrate1d: Resetting integrator because init
INFO:pyFAI.opencl.processing:234.570MB are needed on device which has 12644.188MB
INFO:pyFAI.opencl.processing:Compiling file ['kahan.cl', 'preprocess.cl', 'memset.cl', 'ocl_azim_CSR.cl'] with options -D NBINS=1000  -D NIMAGE=4194304 -D WORKGROUP_SIZE=32
INFO:pyFAI:Processing done in 2.433s !

In [8]:

pyFAI-integrate --help

INFO:root:Enter, port=54321.
INFO:root:Enter.
usage: pyFAI-integrate [options] file1.edf file2.edf ...

PyFAI-integrate is a graphical interface (based on Python/Qt4) to perform
azimuthal integration on a set of files. It exposes most of the important
options available within pyFAI and allows you to select a GPU (or an openCL
platform) to perform the calculation on.

positional arguments:
  FILE                  Files to be integrated

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit
  -v, --verbose         switch to verbose/debug mode
  -o OUTPUT, --output OUTPUT
                        Directory or file where to store the output data
  -f FORMAT, --format FORMAT
                        output data format (can be HDF5)
  -s SLOW, --slow-motor SLOW
                        Dimension of the scan on the slow direction (makes
                        sense only with HDF5)
  -r RAPID, --fast-motor RAPID
                        Dimension of the scan on the fast direction (makes
                        sense only with HDF5)
  --no-gui              Process the dataset without showing the user
                        interface.
  -j JSON, --json JSON  Configuration file containing the processing to be
                        done
  --monitor-name MONITOR_KEY
                        Name of the monitor in the header of each input files.
                        If defined the contribution of each input file is
                        divided by the monitor. If the header does not contain
                        or contains a wrong value, the contribution of the
                        input file is ignored. On EDF files, values from
                        'counter_pos' can accessed by using the expected
                        mnemonic. For example 'counter/bmon'.

PyFAI-integrate saves all parameters in a .azimint.json (hidden) file. This
JSON file is an ascii file which can be edited and used to configure online
data analysis using the LImA plugin of pyFAI. Nota: there is bug in debian6
making the GUI crash (to be fixed inside pyqt) http://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=697348

In [9]:

head -n 50 LaB6_29.4keV.cli

# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
# 
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
# 
# Detector Detector	 Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline	 PixelSize= 5.168e-05, 5.126e-05 m
#    Detector has a mask: False 
#    Detector has a dark current: False 
#    detector has a flat field: False 
# 
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
# --> /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/LaB6_29.4keV.cli
#          r_mm             I 
  3.913197e-02    2.562033e+00
  1.173959e-01    2.491645e+00
  1.956599e-01    2.354185e+00
  2.739238e-01    2.858854e+00
  3.521877e-01    2.796831e+00
  4.304517e-01    2.496695e+00
  5.087156e-01    2.796695e+00
  5.869795e-01    2.742193e+00
  6.652435e-01    2.834705e+00
  7.435074e-01    2.855904e+00
  8.217714e-01    2.906582e+00
  9.000353e-01    3.021892e+00
  9.782992e-01    3.171136e+00
  1.056563e+00    3.010916e+00
  1.134827e+00    3.027250e+00
  1.213091e+00    3.020955e+00
  1.291355e+00    2.935973e+00
  1.369619e+00    3.207142e+00
  1.447883e+00    2.983296e+00
  1.526147e+00    3.007866e+00
  1.604411e+00    3.052688e+00
  1.682675e+00    2.898143e+00
  1.760939e+00    3.027955e+00
  1.839203e+00    3.147613e+00
  1.917466e+00    3.243755e+00
  1.995730e+00    3.213144e+00
  2.073994e+00    2.948822e+00

Conclusion¶

This cookbook exposed the way to perfrom azimuthal integration in scripts with pyFAI using three tools: pyFAI-waxs, pyFAI-saxs and pyFAI-integrate. Using shell scripts is probably not as efficient as writing Python scripts but they may be easier to interface with batch queueing system managing compute-clusters.