import numpy as np
from math import sin, cos
import matplotlib.pyplot as plt
from time import sleep
%matplotlib inline
%%sh
mamba install memory_profiler line_profiler
Example: Given a large 2D array, we will explore different ways to create the array and to calculate its mean. Determine which one is fastest, using the %timeit notebook function.
We'll make some dummy test data that looks like:
$M_{ij} = \sin(i)\cos(0.1 j)$
and we will construct this array in multiple ways.
def create_array_loop(N,M):
arr = []
for y in range(M):
row = []
for x in range(N):
row.append(sin(x)*cos(0.1*y))
arr.append(row)
return arr
def create_array_list(N,M):
"""a 2D array using a list-comprehension"""
return [[sin(x)*cos(0.1*y) for x in range(N)] for y in range(M)]
def create_array_np(N,M):
""" a 2D array using numpy"""
X,Y = np.meshgrid(np.arange(N), np.arange(M))
return np.sin(X)*np.cos(0.1*Y)
Let's first just plot the arrays, to see if they are the same:
N=30; M=10 # our array dimensions
plt.figure(figsize=(12,5))
plt.subplot(1,3,1)
plt.imshow( create_array_loop(N,M))
plt.subplot(1,3,2)
plt.imshow( create_array_list(N,M))
plt.subplot(1,3,3)
plt.imshow( create_array_np(N,M))
And make a plot of the speed of each! Does the result change much when the array size becomes larger? Try much larger sizes for N and M
Hint: use the %timeit -o magic function to have %timeit return results (see the timeit help)
Note: to minimize the uncertainty, try increasing the problem size!
N=30; M=10
Useful tip: timeit has a -o option that lets you save the results for later comparison!
t = {} # a place to record out time statistics
t['loop'] = %timeit -o create_array_loop(N,M)
t['list'] = %timeit -o create_array_list(N,M)
t['numpy'] = %timeit -o create_array_np(N,M)
def plot_performance(time_dict):
mean = [t.average for t in time_dict.values()]
std = [t.stdev for t in time_dict.values()]
x = range(len(time_dict))
plt.errorbar(x, mean, yerr=std, lw=3, fmt="o")
plt.xticks( np.arange(len(time_dict)), time_dict.keys())
plt.ylabel("average execution time (s)")
plt.grid()
plot_performance(t)
note that create_array_list() and create_array_loop both return a list-of-lists, while create_array_np returns a 2D numpy array. There are multiple ways to compute the mean of these arrays. See again which is fastest!
try at least:
sum function and either a for-loop or list-comprehensionarray.mean())N=100; M=100
a_list = create_array_list(N,M)
a_np = create_array_np(N,M)
sum([sum(x) for x in a_list])/(N*M)
a_np.mean()
t2 = {}
t2["numpy a.mean"] = %timeit -o a_np.mean()
t2["numpy mean(a)"] = %timeit -o np.mean(a_np)
t2["sum(a)"] = %timeit -o sum([sum(x) for x in a_list])/(N*M)
plot_performance(t2)
A profiler gives you speed measurements of all functions in your code at once (and all their dependencies)
def slow_function(x):
sleep(1)
return x**2
def faster_function(x):
return x**2
%%prun -r
for ii in range(5):
x = slow_function(ii)
y = faster_function(ii)
stats = _
stats.sort_stats("call").print_stats()
The syntax is:
%lprun -f <function(s)> <python statement that uses the function>
%load_ext line_profiler
def run_all():
for ii in range(5):
x = slow_function(ii)
y = faster_function(ii)
%lprun -f run_all run_all()
Try adding also -f slow_function to see inside that function as well!
%lprun -f create_array_loop create_array_loop(100,100)
import numpy as np
from math import sin, cos
%load_ext memory_profiler
%memit np.sum(np.sin(np.arange(1000000)))
%memit sum(sin(x) for x in range(1000000))
One problem: in a notebook, this is measuring total memory, not just for the function being run, and is affected by garbage collection. Try instead making modules:
%%writefile tmp_sum.py
import numpy as np
import math
def do_some_sums(n=100_000):
sum1 = np.sum(np.sin(np.arange(n)))
sum2 = sum(math.sin(x) for x in range(n))
return sum1+sum2
from tmp_sum import do_some_sums
%mprun -f do_some_sums do_some_sums(500_000)