This notebook contains a demonstration of new features present in the 0.43.0 release of Numba. Whilst release notes are produced as part of the CHANGE_LOG
, there's nothing like seeing code in action!
Included are demonstrations of:
hash()
support replicating Python 3heapq
module support.split()
and .join()
and for developers of Numba extensions, the following are demonstrated:
@overload
@overload
safetyFirst, import the necessary from Numba and NumPy...
from numba import njit, config, __version__
from numba.extending import overload
import numpy as np
assert tuple(int(x) for x in __version__.split('.')[:2]) >= (0, 43)
Initial support for dictionaries has been implemented for all Python versions. This is the first round of implementation work so improvments to design and usability can be expected in future releases. Numba's dictionary implementation is a specialized dictionary for use in Numba's nopython
mode, it behaves like a standard dictionary but dictionary operations outside of nopython
mode are inherently slower than the equivalent on a standard dictionary. Most dictionary operations are supported and are demonstrated below...
Numba dictionaries are "typed" (they are of a specified type that may not be altered once instantiated). First, let's create a Numba dictionary with int32
type keys and float32
type values:
from numba.typed import Dict
from numba import int32, float32
d = Dict.empty(int32, float32)
Now let's perform some standard operations on the dictionary from Python:
# len
assert len(d) == 0
# setitems
d[1] = 1
d[2] = 2.3
d[3] = 3.4
# print the values, note the float32 representation of the values
print(*d.values())
# popitem
print(d.popitem())
# recheck len
assert len(d) == 2
and then pass this dictonary to a nopython
mode function which mutates it. Note that Numba's Dict
follows the behaviour of Python 3.7 dictionaries in that they are by default ordered.
@njit
def mutate_dict(di):
di[10] = 100.
di[2] += 7.
default_val = 3.14159
for i in range(8, 12):
di.setdefault(i, default_val)
del di[1]
mutate_dict(d)
for k, v in d.items():
print(k, v)
This function does some work on a dictionary:
from numba.types import unicode_type, int64
@njit
def snake_graph():
"""
This function creates a dictionary (str->int) and then randomly increases the integer
values and finally prints the results as a snake-y graph.
"""
x = Dict.empty(unicode_type, int64)
# set up
bins = ['n','u','m','b','a']
for v in bins:
x[v] = 0
# add values to bins at random
it = 100
for i in range(it):
key = bins[np.random.randint(len(bins))]
x[key] += 1
# iterate the key space and print a "bar" of snakes
for k in x.keys():
print(k + ':' + ''.join(['🐍' for _ in range(x[k])]))
return x
r = snake_graph()
r
A necessary side effect of dictionary support was the requirement for __hash__()
to work for the hashable Numba types. The implementations present in Numba replicate the hashing behaviour found in Python 3 (even when running with Python 2!) and also acknowledge the PYTHONHASHSEED
as needed. A few examples:
def hash_things():
things = [100, 2**61 + 2, 1.7, 2j, -1]
print(100, hash(100))
print(2**61 + 2, hash(2**61 + 2)) # wrap around in hashing!
print(3.14159, hash(3.14159))
print(2j, hash(2j))
print(-1, hash(-1)) # magic -1
unicode_things = ['Numba', 'is', '🐍⚡']
for val in unicode_things:
print(val, hash(val))
hash_things()
njit(hash_things)()
For those developing Numba extensions, documentation on hashing can be found here. It is possible to override the default hashing implementations for a type and also implement a hash algorithm for custom types by simply overloading the type's __hash__
method.
heapq
¶Numba 0.43 has support for the built in heapq
module (everything except heapq.merge
). A quick demonstration based on the example code from the CPython documentation.
from heapq import heappush, heappop
@njit
def heapsort(iterable):
# method to seed the type of the list
ty = iterable[0]
h = [ty for _ in range(0)]
for value in iterable:
heappush(h, value)
return [heappop(h) for i in range(len(h))]
heapsort([1, 3, 5, 7, 9, 2, 4, 6, 8, 0])
Numba's string processing support now includes the .split()
and .join()
methods, including the use of sep
and maxsplit
.
@njit
def split_and_join(string):
celebrate = '🎉'.join("🎆".join("🎇🎇🎇"))
return ('⚡'.join(string.split('🐍')) + " .split() and .join()").join([celebrate, celebrate])
split_and_join("n🐍u🐍m🐍b🐍a")
User defined exceptions with arguments are now supported in nopython
mode, for example:
class UDE(Exception):
def __init__(self, arg_for_super, value):
super(UDE, self).__init__(arg_for_super)
self.value = value
def __str__(self):
return "%s. Custom value: %s" % (super(UDE, self).__str__(), self.value)
@njit
def raise_ude():
raise UDE('Some error message', 10)
try:
raise_ude()
except Exception as e:
print(e)
C structures can now be mapped directly to NumPy record types (structured dtype
). The following demonstrates (it is a slightly modified version of this example, interested users should take a look at both):
from cffi import FFI
from numba import cffi_support
from numba import cfunc, carray
src = """
/* Define the C struct */
typedef struct my_struct {
int i1;
double d[4];
} my_struct;
/* Define a callback function */
typedef double (*my_func)(my_struct*, size_t);
"""
ffi = FFI()
ffi.cdef(src)
d_len = 4 # there are 4 values in the my_struct.d
# Make an array of 2 my_struct
mydata = ffi.new('my_struct[2]')
ptr = ffi.cast('my_struct*', mydata)
for i in range(2):
ptr[i].i1 = 123 + i
for j in range(d_len):
ptr[i].d[j] = i + 100 * (1 + j)
# map the type to a Numba record type and use it in a `@cfunc`
sig = cffi_support.map_type(ffi.typeof('my_func'), use_record_dtype=True)
@cfunc(sig)
def foo(ptr, n):
base = carray(ptr, n) # view pointer as an array of my_struct
tmp = 0
for i in range(n):
acc = 0
for j in range(d_len):
acc += base[i].d[j]
tmp += acc / (base[i].i1 + 1)
return tmp
# Test using the .ctypes callable
addr = int(ffi.cast('size_t', ptr)) # get the address of the `mydata` array of structs
result = foo.ctypes(addr, 2) # array of structs is 2 long
print(result)
This release contains a number of newly supported NumPy functions:
asarray
ptp
roll
extract
trapz
, interp
and broadcasting has been added to the in np.where
implementation.
@njit
def numpy_new():
# construct an array from a list, tuple, scalar with np.asarray
from_list = np.asarray([1, 2, 3])
from_tuple = np.asarray((1j, 2j, 3j))
from_scalar = np.asarray(11.4)
from_bool = np.asarray(False)
print("asarray:", from_list / from_scalar + from_tuple + from_bool)
# np.ptp
print("\nptp one cycle of cosine", np.ptp(np.cos(np.linspace(0, 2 * np.pi, 1000))))
# np.roll
print("\nroll forwards 2", np.roll(np.arange(7), 2))
print("roll backwards 3", np.roll(np.arange(7), -3))
# np.extract
print("\nextract odd indexes", np.extract(np.arange(8) % 2, np.arange(8)))
# np.trapz
x = np.linspace(0, np.pi, 1000)
print("\nintegral of a half cycle of sine", np.trapz(np.sin(x), x))
# np.interp
x = np.linspace(0, 2 * np.pi, 1000)
y = np.cos(x)
x_i = np.random.uniform(0, 2 * np.pi, (4,))
print("\ninterpolate along one cycle of cosine", np.interp(x_i, x, y))
print(" direct result", np.cos(x_i))
# np.where
tmp = np.arange(-16, 16).reshape(4, 8)
mask = np.where(tmp > 0, 1, 0)
print("\nbroadcast condition:\n", mask)
numpy_new()
For developers using Numba to accelerate their libraries, two new features have been added to the @overload
decorator. The first makes it such that it is considerably harder to have a mismatch between the declared typing and implementing signatures. For example:
from numba.extending import overload
from numba import errors
def myfoo(a, b, k=7):
pass
@overload(myfoo)
def _myfoo(a, b, k=7):
def impl(a, b, k=12): # oops, different default value in the implementation detail
pass
return impl
@njit
def use_foo():
print(myfoo(1, 2, 3))
try:
use_foo()
except errors.TypingError as e:
print("Showing error for demonstration purposes:\n")
print('-' * 80)
print(e)
print('-' * 80)
The second addition is a new compiler pass that prunes (removes) branches that can be computed as dead at compile time. This has been added to make it possible to avoid the, somewhat awkward but necessary, pattern required to handle None
-like arguments. For example:
from numba import types, config
# 1. want to write an overload for this...
# def python_mybar(a, b=None):
# if b is None:
# return a
# else:
# return a + b
def python_mybar(a, b=None):
pass
# 2. used to have to write this... to handle `None`
# @overload(python_mybar)
# def _mybar(a, b=None):
# if b is None or isinstance(b, types.NoneType) or getattr(b, 'value', False) is None:
# def impl(a, b=None):
# return a
# else:
# def impl(a, b=None):
# return a + b
# return impl
# 3. but now this just works...
@overload(python_mybar)
def _new_mybar(a, b=None):
def impl(a, b=None):
if b is None:
return a
else:
return a + b
return impl
@njit
def use_bar():
print(python_mybar(1, None))
print(python_mybar(1, 2))
use_bar()