Notebook

Integration of Python with compiled languages¶

In [ ]:

import addutils.toc ; addutils.toc.js(ipy_notebook=True)

In [ ]:

from addutils import css_notebook
css_notebook()

1 Introduction¶

Python users and developers can use many tools for integrating Python with other languages. For example it's possible to link Python code to external libraries, such as C or Fortran ones, usually with a more efficient processing and a general speed up. These tools are tipically multi-platform, so that you can develop in Linux, Windows or MacOS, at your will. Here we will consider only the most quoted tools, i.e. Cython, Weave and ctypes. Other tools, such as F2PY (Fortran, C) and SWIG, are available. SWIG (Simplified Wrapper Interface Generator) automates the generation of wrapper code for interfacing C/C++ with many languages (Guile, Java, Ocaml, Perl, Pike, PHP, Python, Ruby, and Tcl). A good primer on F2PY and SWIG is in Langtangen - A Primer on Scientific Programming with Python. CXX, Boost.Python, and SCXX are C++ tools that simplify programming with the Python C API.

2 Speeding up Python code with Cython¶

Cython is (almost) a superset of Python. With Cython you can assign C types to Python variables, coding in something similar to Python but getting the speed advantages of C. The resulting run time can be much shorter, even with improvements of 200-300x.

Cython code is translated in C code using Python setup scripts based on the distutils library. An example is below: it represents the setup script for the three compiled extensions that we will consider in this paragraph. Setup scripts are run from the command lines.

NB: the following examples are meant to be run in MacOSX. Before running them on other Operating Systems it is necessary to install a C compiler and to modify them accordingly.

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
  cmdclass = {'build_ext': build_ext},
  ext_modules = [
    Extension("integrate_compiled",
              ["integrate.py"],
             ),
    Extension("integrate_hints",
              ["integrate_hints.py"],
             ),
    Extension("integrate_cy",
              ["integrate_cy.pyx"],
             ),
  ])

The C code is compiled as a Python library, in pyd format, that can be imported in Python scripts as a normal Python module (such as Numpy and Pandas).

We consider as a specific example a case of integration of a power function, deriving from a presentation by van der Walt at the 2010 Summer School in Trento. Execution times for Python and Cython versions have been measured on a normal laptop (Windows Vista, 4 GB RAM).

We try to compute the approximate integral of $$\int_{a}^{b} f(x)dx$$ by using rectangular discretisation.

In [ ]:

# integrate.py
from __future__ import division

def f(x):
    return x**4 - 3 * x

def integrate_f(a, b, N):
    """Rectangle integration of a function.

    Parameters
    ----------
    a, b : ints
        Interval over which to integrate.
    N : int
        Number of intervals to use in the discretisation.

    """
    s = 0
    dx = (b - a) / N
    for i in range(N):
        s += f(a + i * dx)
    return s * dx

After running the setup script on the integrate.py script, we obtain a integrate.pyd module that we can import and test.

In [ ]:

import timeit

setup_statement="import utilities.integrate as i0"

s = """\
i0.integrate_f(0,10,100000)
"""
t = timeit.Timer( s, setup_statement )

print("Execution time: {:.2f} msec/pass".format( 1000 * t.timeit(number=100)/100 ))

If we modify the code with Cython decorators ('@cython.locals'), we can attribute a type to the function arguments and to local variables, as is the norm for static languages. This code is still compatible with Python.

In [ ]:

from __future__ import division
import cython

@cython.locals(x=cython.double)
def f(x):
    # Note: x**4 translates to slow code in Cython
    # Use x*x*x*x to avoid calling 'pow'
    return x**4 - 3 * x

@cython.locals(a=cython.double, b=cython.double,
               N=cython.int, s=cython.double,
               dx=cython.double, i=cython.int)
def integrate_f(a, b, N):

    s = 0
    dx = (b - a) / N
    for i in range(N):
        s += f(a + i * dx)
    return s * dx

If we compile the code with the setup scripts, we obtain a pyd dll that we can import and use.

In [ ]:

setup_statement="import utilities.integrate_hints as i1"

s = """i1.integrate_f(0,10,100000)"""
t = timeit.Timer( s, setup_statement )

print("Execution time: {:.2f} msec/pass".format( 1000 * t.timeit(number=100)/100))

The execution time should get a 3x improvement.

We can further optimize the code, using cdef (c definitions), in a way that this code is no longer compatible with Python, but, when compiled, it run much faster than the original code.

# cython: cdivision=True

# ^^^ Could also use @cython.cdivision(True) decorator

cdef double f(double x):
    return x*x*x*x - 3 * x

def integrate_f(double a, double b, int N):
    cdef double s = 0
    cdef double dx = (b - a) / N
    cdef int i
    for i in range(N):
        s += f(a + i * dx)
    return s * dx

In [ ]:

setup_statement="import utilities.integrate_cy as i2"

s = """\
i2.integrate_f(0,10,100000)
"""
t = timeit.Timer( s, setup_statement )

print("Execution time: {:.2f} msec/pass".format( 1000 * t.timeit(number=100)/100 ))

We obtain an improvement of about 150x, and the code is still quite readable, at the (small) price of having the cython code automatically translated into C by the setup scripts.

2.1 References¶

Cython for NumPy users - http://docs.cython.org/src/userguide/numpy_tutorial.html#numpy-tutorial

A good introduction is: van der Walt, Summer School 2010 Trento - https://python.g-node.org/python-autumnschool-2010/start

3 Integrating C/C++ code into Python with Weave¶

Can we insert C code into Python scripts? With Weave it is possible. Thanks to the 'inline' function, it is possible to integrate C code lines as text into a Python script. At the first execution of the script, the C code is compiled, so that the total run time can get longer (generally a few seconds), but the compiled code is saved in cache in the computer and is then directly executed the next times. Usually the compiled code is tens of times faster than the interpreted version.

There is also another Weave functionality, 'blitz', that allows to directly compile a single Python expression into C++. For instance, the following Python expression (from http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html):

In [ ]:

from scipy import *  # or from NumPy import *
import numpy as np
a = np.zeros((512,512), float)
b = np.ones((512,512), float)

In [ ]:

a[1:-1,1:-1] =  (b[1:-1,1:-1] + b[2:,1:-1] + 3*b[:-2,1:-1] \
               + b[1:-1,2:] + b[1:-1,:-2]) / 6.0
print a

can be compiled with blitz in this way:

In [ ]:

import scipy.weave
from scipy import *  # or from NumPy import *

a = np.zeros((512,512), float)
b = np.ones((512,512), float)

expr = "a[1:-1,1:-1] =  (b[1:-1,1:-1] + b[2:,1:-1] + 3*b[:-2,1:-1]" \
                      "+ b[1:-1,2:] + b[1:-1,:-2]) / 5.0"
scipy.weave.blitz(expr)
print a

Generally the increase in run speed with 'blitz' is lower to the one that we obtain with 'inline'.

An example of stochastic simulation of stocks with 'inline' is here exemplified. The C code is inserted as a text string. It do FOR loops on a Numpy array (named S), passed by reference. The function arguments are passed as list elements in the weave.inline arguments.

Original version:

In [ ]:

import numpy as np
import pandas as pd
from pandas.io.data import read_csv

AAPL = read_csv('example_data/p03_AAPL.txt',
                index_col='Date', parse_dates=True)

AAPL['Ret'] = np.log(AAPL.Close/AAPL.Close.shift(1)) 
vol = np.std(AAPL['Ret'])*np.sqrt(252)               
r = 0.025                                            # Constant Short Rate
S0 = AAPL['Close'][-1]                               # End Value                                                                                      
K = S0 * 1.1                                         # OTM Call Option                                                                                
T = 1.0                                              # Maturity Year
M = 100; dt = T / M                                  # Time Steps                                                                                     
I = 10000                                            # Simulation Paths
S = np.zeros((M + 1, I))
S[0,:] = S0
for t in range(1, M + 1):
    ran = np.random.standard_normal(I)
    S[t, :] = S[t - 1, :] * np.exp((r - vol**2 / 2) * dt + vol * np.sqrt(dt) * ran)

Modified version:

In [ ]:

from scipy import weave
from scipy.weave import converters
# changes for C compatibility
vol_c = float(vol) # vol is a np array, would give error in C code
ran = np.random.normal(0, 1, (M+1)*I)
ran = np.reshape(ran, (M+1,I)) 

S = np.zeros((M+1,I))

code = r"""

for ( int i = 0; i < I; i++ ) S(0,i) = S0;

for ( int t = 1; t < M+1; t++ )
{
  for ( int i = 0; i < I; i++ ) S(t,i) = S(t-1,i) * exp( (r - vol_c * vol_c/2.0) * dt + vol_c * sqrt(dt) * ran(t,i) );
}

"""
 
weave.inline(code,['S', 'M', 'dt', 'I', 'vol_c','r', 'S0', 'ran'], type_converters=converters.blitz, compiler='gcc')

3.1 References¶

Weave (scipy.weave) - http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html

Weave (Sage) - http://www.sagemath.org/doc/numerical_sage/weave.html

4 Access to external libraries with ctypes¶

Numerical libraries such as Blas, Lapack or Atlas are routinely available in Python. We can access other libraries from Python with the ctypes module. ctypes presents many functionalities: it has wrappers for C data, it allows to create pointers and to return the referenced values, it can call functions from the C library (libc).

As an example, we consider the creation of a simple function in a shared library, created and tested in C in Linux (Ubuntu) (from http://stackoverflow.com/questions/5862915/passing-numpy-arrays-to-a-c-function-for-input-and-output). The C source code is the following:

/* double_array.c */ #include <stdio.h> void cdouble(const double *indata, size_t size, double *outdata) { size_t i; for (i = 0; i < size; ++i) outdata[i] = indata[i] * 2.0; }

We can compile this file as shared library in Linux:

gcc -fPIC -shared -o double_array.so double_array.c

and use this new library from Python:

import numpy as np import ctypes from numpy.ctypeslib import ndpointer lib = ctypes.cdll.LoadLibrary('./double_array.so') cdouble = lib.cdouble cdouble.restype = None cdouble.argtypes = [ndpointer(ctypes.c_double), ctypes.c_size_t, ndpointer(ctypes.c_double)] indata = np.ones((10,10)) outdata = np.empty((10,10)) cdouble(indata, indata.size, outdata) print 'array originale: %s' % indata print 'nuovo array: %s' % outdata

A tested compilation setting for Windows with gcc that allows to create dll recognised by ctypes is:

gcc -c -DBUILD_DLL sourcename.c gcc -shared -o sourcename.dll sourcename.o

See: http://codespeak.net/pipermail/cython-dev/2009-May/005517.html

4.1 References¶

ctypes — A foreign function library for Python - http://docs.python.org/library/ctypes.html

ctypes (Sage) - http://www.sagemath.org/doc/numerical_sage/ctypes.html

5 Embedding Python in C¶

Incorporating Python in C presents a greater complexity and more freedom degrees than incorporating C in Python. It also requires the knowledge of the C APIs in Python. The first step for Python embedding is the inclusion of the Python header file 'Python.h' in our C program. 'Py_Initialize()' and 'Py_Finalize()' initialize and close the Python interpreter, respectively. There are many possible ways to interface C and Python:

insert in C static (or dinamically constructed) string of Python code (in a way similar to Weave.inline) and execute them with 'PyRun_SimpleString()'. Since the compilation is a slow process, it is better to precompile them as Python bytecode;
call Python objects associating a list of arguments;
call and execute entire Python scripts with 'PyRun_SimpleFile()'.

When a Python object is no longer used, the memory can be set free with 'Py_DECREF'. The incorporated Python code can be in turn extended with C, using the previously described techniques. Considering the difficulties of embedding in C, there are no largely used tools that can simplify the work of integrating of Python in C.

5.1 References¶

Supercharging C++ Code With Embedded Python – EuroPython 2012 Talk - http://realmike.org/blog/2012/07/05/supercharging-c-code-with-embedded-python/

Embedding Python in Another Application - http://docs.python.org/extending/embedding.html

Python/C API Reference Manual - http://docs.python.org/c-api/index.html#c-api-index

Visit www.add-for.com for more tutorials and updates.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.