Python NumPy library
NumPy is the fundamental package for scientific computing with Python. :¶
- a powerful N-dimensional array object
- sophisticated (broadcasting) functions
- tools for integrating C/C++ and Fortran code
- useful linear algebra, Fourier transform, and random number capabilities
NumPy is licensed under the BSD license, enabling reuse with few restrictions.
In [1]:
import numpy as np # extremely common naming for Numpy to be used as np
In [2]:
a = np.arange(20).reshape(4,5)
a
Out[2]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [3]:
b = np.arange(5,500,10)
b
Out[3]:
array([ 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, 105, 115, 125,
135, 145, 155, 165, 175, 185, 195, 205, 215, 225, 235, 245, 255,
265, 275, 285, 295, 305, 315, 325, 335, 345, 355, 365, 375, 385,
395, 405, 415, 425, 435, 445, 455, 465, 475, 485, 495])
In [4]:
type(a),type(b),a.dtype,b.dtype
Out[4]:
(numpy.ndarray, numpy.ndarray, dtype('int32'), dtype('int32'))
Main NumPy object¶
NumPy’s main object is the homogeneous multidimensional array. It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In NumPy dimensions are called axes.
In other words think of matrix (2-D, 3-D, etc) with elements given their coordinates¶
NumPy’s array class is called ndarray. It is also known by the alias array. Note that numpy.array is not the same as the Standard Python Library class array.array, which only handles one-dimensional arrays and offers less functionality.
When Should you use ndarray instad of normal Python data structures such as list?¶
NumPy's arrays are more compact than Python lists (about 3-5x difference)
Faster access for reading/writing items / less Python "bookkeeping"
Convencience operations matrix operations/Fourier Transforms/histograms/statistics/filtering/etc
Spatial locality in memory access patterns results in performance gains notably due to the CPU cache
Items stored contiguously in memory allow NumPy to take advantage of vectorized instructions of modern CPUs, such as Intel's SSE and AVX, AMD's XOP
Restriction(uniform data types)
In [7]:
d=a*2
d
Out[7]:
array([[ 0, 2, 4, 6, 8],
[10, 12, 14, 16, 18],
[20, 22, 24, 26, 28],
[30, 32, 34, 36, 38]])
In [6]:
c=a**2
c
Out[6]:
array([[ 0, 1, 4, 9, 16],
[ 25, 36, 49, 64, 81],
[100, 121, 144, 169, 196],
[225, 256, 289, 324, 361]], dtype=int32)
In [8]:
# individual elements
c[1,1]
Out[8]:
36
In [9]:
c[1]
Out[9]:
array([25, 36, 49, 64, 81], dtype=int32)
In [46]:
c[:2]
Out[46]:
array([[ 0, 1, 4, 9, 16],
[25, 36, 49, 64, 81]], dtype=int32)
In [11]:
c[:,1]
Out[11]:
array([ 1, 36, 121, 256], dtype=int32)
In [12]:
c[::-1]
Out[12]:
array([[225, 256, 289, 324, 361],
[100, 121, 144, 169, 196],
[ 25, 36, 49, 64, 81],
[ 0, 1, 4, 9, 16]], dtype=int32)
In [8]:
c[::-1,::-1]
Out[8]:
array([[361, 324, 289, 256, 225],
[196, 169, 144, 121, 100],
[ 81, 64, 49, 36, 25],
[ 16, 9, 4, 1, 0]], dtype=int32)
In [13]:
c
Out[13]:
array([[ 0, 1, 4, 9, 16],
[ 25, 36, 49, 64, 81],
[100, 121, 144, 169, 196],
[225, 256, 289, 324, 361]], dtype=int32)
In [14]:
c[:, 1], c[0:2, 1]
Out[14]:
(array([ 1, 36, 121, 256], dtype=int32), array([ 1, 36], dtype=int32))
In [ ]:
# Loading files
In [17]:
my_csv = np.genfromtxt('./resources/random4x9.csv', delimiter=',')
my_csv
Out[17]:
array([[0.72809242, 0.43954956, 0.96609444, 0.95103408],
[0.43789427, 0.56524801, 0.26548715, 0.87307645],
[0.68820884, 0.41444608, 0.33806432, 0.56520271],
[0.14000647, 0.955224 , 0.07058813, 0.09764705],
[0.03317341, 0.77677129, 0.08200467, 0.19588284],
[0.32389879, 0.60277379, 0.77265652, 0.53549964],
[0.78382241, 0.44545018, 0.09123232, 0.24865343],
[0.07532197, 0.13528559, 0.59399663, 0.31171427],
[0.31212006, 0.70050059, 0.44611673, 0.85198123]])
In [18]:
list(my_csv.reshape(1,36))
Out[18]:
[array([0.72809242, 0.43954956, 0.96609444, 0.95103408, 0.43789427,
0.56524801, 0.26548715, 0.87307645, 0.68820884, 0.41444608,
0.33806432, 0.56520271, 0.14000647, 0.955224 , 0.07058813,
0.09764705, 0.03317341, 0.77677129, 0.08200467, 0.19588284,
0.32389879, 0.60277379, 0.77265652, 0.53549964, 0.78382241,
0.44545018, 0.09123232, 0.24865343, 0.07532197, 0.13528559,
0.59399663, 0.31171427, 0.31212006, 0.70050059, 0.44611673,
0.85198123])]
In [ ]:
# going back to Python lists....
In [19]:
myl=list(my_csv)
myl
Out[19]:
[array([0.72809242, 0.43954956, 0.96609444, 0.95103408]), array([0.43789427, 0.56524801, 0.26548715, 0.87307645]), array([0.68820884, 0.41444608, 0.33806432, 0.56520271]), array([0.14000647, 0.955224 , 0.07058813, 0.09764705]), array([0.03317341, 0.77677129, 0.08200467, 0.19588284]), array([0.32389879, 0.60277379, 0.77265652, 0.53549964]), array([0.78382241, 0.44545018, 0.09123232, 0.24865343]), array([0.07532197, 0.13528559, 0.59399663, 0.31171427]), array([0.31212006, 0.70050059, 0.44611673, 0.85198123])]
In [18]:
# Converting individual ndarrays in our regular list to regular inner lists
In [17]:
myll = [list(el) for el in myl]
myll
Out[17]:
[[0.728092422, 0.439549556, 0.966094444, 0.951034076], [0.437894272, 0.565248006, 0.265487155, 0.873076449], [0.688208844, 0.414446083, 0.338064325, 0.565202708], [0.140006466, 0.955223997, 0.070588134, 0.097647052], [0.033173413, 0.776771295, 0.082004671, 0.195882839], [0.323898787, 0.602773787, 0.77265652, 0.535499641], [0.783822409, 0.44545018, 0.091232322, 0.24865343], [0.075321971, 0.13528559, 0.59399663, 0.311714268], [0.312120058, 0.700500595, 0.446116732, 0.851981232]]
In [19]:
## Litmus test: for group operations NumPy, if you have many individual operations then regular list might suffice
In [ ]:
## NOTE: For multiple data types better to use Pandas library(another lecture)
In [20]:
dir(a)
Out[20]:
['T', '__abs__', '__add__', '__and__', '__array__', '__array_finalize__', '__array_interface__', '__array_prepare__', '__array_priority__', '__array_struct__', '__array_ufunc__', '__array_wrap__', '__bool__', '__class__', '__complex__', '__contains__', '__copy__', '__deepcopy__', '__delattr__', '__delitem__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__iand__', '__ifloordiv__', '__ilshift__', '__imatmul__', '__imod__', '__imul__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__ior__', '__ipow__', '__irshift__', '__isub__', '__iter__', '__itruediv__', '__ixor__', '__le__', '__len__', '__lshift__', '__lt__', '__matmul__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmatmul__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__setitem__', '__setstate__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__xor__', 'all', 'any', 'argmax', 'argmin', 'argpartition', 'argsort', 'astype', 'base', 'byteswap', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'ctypes', 'cumprod', 'cumsum', 'data', 'diagonal', 'dot', 'dtype', 'dump', 'dumps', 'fill', 'flags', 'flat', 'flatten', 'getfield', 'imag', 'item', 'itemset', 'itemsize', 'max', 'mean', 'min', 'nbytes', 'ndim', 'newbyteorder', 'nonzero', 'partition', 'prod', 'ptp', 'put', 'ravel', 'real', 'repeat', 'reshape', 'resize', 'round', 'searchsorted', 'setfield', 'setflags', 'shape', 'size', 'sort', 'squeeze', 'std', 'strides', 'sum', 'swapaxes', 'take', 'tobytes', 'tofile', 'tolist', 'tostring', 'trace', 'transpose', 'var', 'view']
In [21]:
my_csv.ndim
Out[21]:
2
In [23]:
my_csv
Out[23]:
array([[0.72809242, 0.43954956, 0.96609444, 0.95103408],
[0.43789427, 0.56524801, 0.26548715, 0.87307645],
[0.68820884, 0.41444608, 0.33806432, 0.56520271],
[0.14000647, 0.955224 , 0.07058813, 0.09764705],
[0.03317341, 0.77677129, 0.08200467, 0.19588284],
[0.32389879, 0.60277379, 0.77265652, 0.53549964],
[0.78382241, 0.44545018, 0.09123232, 0.24865343],
[0.07532197, 0.13528559, 0.59399663, 0.31171427],
[0.31212006, 0.70050059, 0.44611673, 0.85198123]])
In [24]:
?my_csv.T
In [25]:
my_csv.shape
Out[25]:
(9, 4)
In [26]:
my_csv.dtype
Out[26]:
dtype('float64')
In [27]:
my_csv.itemsize
Out[27]:
8
In [28]:
my_csv.data
Out[28]:
<memory at 0x00000000056D48B8>
Most important array attributes¶
- ndarray.ndim
the number of axes (dimensions) of the array.
- ndarray.shape
the dimensions of the array. This is a tuple of integers indicating the size of the array in each dimension. For a matrix with n rows and m columns, shape will be (n,m). The length of the shape tuple is therefore the number of axes, ndim.
- ndarray.size
the total number of elements of the array. This is equal to the product of the elements of shape.
- ndarray.dtype
an object describing the type of the elements in the array. One can create or specify dtype’s using standard Python types. Additionally NumPy provides types of its own. numpy.int32, numpy.int16, and numpy.float64 are some examples.
- ndarray.itemsize
the size in bytes of each element of the array. For example, an array of elements of type float64 has itemsize 8 (=64/8), while one of type complex32 has itemsize 4 (=32/8). It is equivalent to ndarray.dtype.itemsize.
- ndarray.data
the buffer containing the actual elements of the array. Normally, we won’t need to use this attribute because we will access the elements in an array using indexing facilities.
In [29]:
a.ndim
Out[29]:
2
In [30]:
a.shape
Out[30]:
(4, 5)
In [31]:
a.size
Out[31]:
20
In [32]:
a.dtype
Out[32]:
dtype('int32')
In [33]:
a.itemsize
Out[33]:
4
In [43]:
a.data
Out[43]:
<memory at 0x00000000081FDC18>
In [35]:
a.data
Out[35]:
<memory at 0x00000000081FD558>
In [36]:
a.data
Out[36]:
<memory at 0x00000000081FD630>
In [ ]:
## NumPy ndarray advantage over normal Python data structures:
* efficient and fast
In [44]:
my_csv.mean()
Out[44]:
0.46707556552777774
In [45]:
my_csv.median()
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-45-9875fbb1cc5e> in <module>() ----> 1 my_csv.median() AttributeError: 'numpy.ndarray' object has no attribute 'median'
In [46]:
a
Out[46]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [48]:
a[2,0]=20
a
Out[48]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[20, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [49]:
np.median(a, axis=1)
Out[49]:
array([ 2., 7., 13., 17.])
In [50]:
np.median([32,3,7,10,20])
Out[50]:
10.0
In [51]:
np.median([5,6,3,8,111,90])
Out[51]:
7.0
In [52]:
sorted([5,6,3,8,111,90])
Out[52]:
[3, 5, 6, 8, 90, 111]
In [54]:
b
Out[54]:
array([ 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, 105, 115, 125,
135, 145, 155, 165, 175, 185, 195, 205, 215, 225, 235, 245, 255,
265, 275, 285, 295, 305, 315, 325, 335, 345, 355, 365, 375, 385,
395, 405, 415, 425, 435, 445, 455, 465, 475, 485, 495])
In [53]:
b.mean()
Out[53]:
250.0
In [55]:
np.median(b)
Out[55]:
250.0
In [56]:
b.std() ## standard deviation
Out[56]:
144.30869689661813
In [57]:
a
Out[57]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[20, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [58]:
np.median(a, axis=0)
Out[58]:
array([10. , 8.5, 9.5, 10.5, 11.5])
In [60]:
np.median(a, axis=1)
Out[60]:
array([ 2., 7., 13., 17.])
In [59]:
np.mean(a, axis=0)
Out[59]:
array([10. , 8.5, 9.5, 10.5, 11.5])
In [61]:
np.mean(a, axis=1)
Out[61]:
array([ 2., 7., 14., 17.])
In [29]:
a.mean(axis=0) ## same as above
Out[29]:
array([ 7.5, 8.5, 9.5, 10.5, 11.5])
In [30]:
np.median(a, axis=1)
Out[30]:
array([ 2., 7., 12., 17.])
In [62]:
np.median(a, axis=2)
--------------------------------------------------------------------------- AxisError Traceback (most recent call last) <ipython-input-62-3bab11b7d1c6> in <module>() ----> 1 np.median(a, axis=2) C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\function_base.py in median(a, axis, out, overwrite_input, keepdims) 4117 """ 4118 r, k = _ureduce(a, func=_median, axis=axis, out=out, -> 4119 overwrite_input=overwrite_input) 4120 if keepdims: 4121 return r.reshape(k) C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\function_base.py in _ureduce(a, func, **kwargs) 4011 keepdim = list(a.shape) 4012 nd = a.ndim -> 4013 axis = _nx.normalize_axis_tuple(axis, nd) 4014 4015 for ax in axis: C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\numeric.py in normalize_axis_tuple(axis, ndim, argname, allow_duplicate) 1504 except TypeError: 1505 axis = tuple(axis) -> 1506 axis = tuple(normalize_axis_index(ax, ndim, argname) for ax in axis) 1507 if not allow_duplicate and len(set(axis)) != len(axis): 1508 if argname: C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\numeric.py in <genexpr>(.0) 1504 except TypeError: 1505 axis = tuple(axis) -> 1506 axis = tuple(normalize_axis_index(ax, ndim, argname) for ax in axis) 1507 if not allow_duplicate and len(set(axis)) != len(axis): 1508 if argname: AxisError: axis 2 is out of bounds for array of dimension 2
In [63]:
d=np.arange(90).reshape(3,3,10)
d
Out[63]:
array([[[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]],
[[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]],
[[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]]])
In [64]:
d[2,1,5]
Out[64]:
75
In [65]:
d.reshape(3,3,2,5)
Out[65]:
array([[[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9]],
[[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29]]],
[[[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49]],
[[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]],
[[[60, 61, 62, 63, 64],
[65, 66, 67, 68, 69]],
[[70, 71, 72, 73, 74],
[75, 76, 77, 78, 79]],
[[80, 81, 82, 83, 84],
[85, 86, 87, 88, 89]]]])
In [32]:
# of course, as we have 2-D data and we attempted to use 3rd dimension ! :)
A frequent error consists in calling array with multiple numeric arguments, rather than providing a single list of numbers as an argument.
In [66]:
c = np.array(2,3,4,5,6,7) # WRONG
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-66-6548e3003d99> in <module>() ----> 1 c = np.array(2,3,4,5,6,7) # WRONG ValueError: only 2 non-keyword arguments accepted
In [67]:
# CORRECT would be
c = np.array([2,3,4,6,7,9,44])
c
Out[67]:
array([ 2, 3, 4, 6, 7, 9, 44])
In [89]:
t = np.array([4000000000000,"R"])
t
Out[89]:
array(['4000000000000', 'R'], dtype='<U21')
In [73]:
t = np.array((5,6,7,9,1111111111112,0.4))
t
Out[73]:
array([5.00000000e+00, 6.00000000e+00, 7.00000000e+00, 9.00000000e+00,
1.11111111e+12, 4.00000000e-01])
In [75]:
s = np.array(["Hello",'World','''Aha something strange'''])
s
Out[75]:
array(['Hello', 'World', 'Aha something strange'], dtype='<U21')
In [77]:
s.itemsize
Out[77]:
84
In [78]:
s.dtype
Out[78]:
dtype('<U21')
In [ ]:
## Can specify array type at creation
In [90]:
c = np.array([[3,4,5], [3,1,6]], dtype=complex)
c
Out[90]:
array([[3.+0.j, 4.+0.j, 5.+0.j],
[3.+0.j, 1.+0.j, 6.+0.j]])
In [91]:
c[0,0].real
Out[91]:
3.0
In [92]:
c[0,0].imag
Out[92]:
0.0
In [93]:
c[1,1].imag = 5
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-93-11ea9d4ba07d> in <module>() ----> 1 c[1,1].imag = 5 AttributeError: attribute 'imag' of 'numpy.generic' objects is not writable
In [ ]:
# above does not work as imag only returns the value ie getter not setter
In [94]:
c[1,1] = 2 + 3j #so use j for imaginary part
c
Out[94]:
array([[3.+0.j, 4.+0.j, 5.+0.j],
[3.+0.j, 2.+3.j, 6.+0.j]])
In [95]:
c.dtype,c.size,c.shape
Out[95]:
(dtype('complex128'), 6, (2, 3))
In [96]:
d = np.array([[3+4j,4,5+2.1j], [3,1-3j,6]], dtype=complex)
d
Out[96]:
array([[3.+4. j, 4.+0. j, 5.+2.1j],
[3.+0. j, 1.-3. j, 6.+0. j]])
In [97]:
d[0,0].real,d[0,0].imag
Out[97]:
(3.0, 4.0)
In [98]:
z = np.zeros((5,6))
z[3,3] = 42
print(type(z))
print(z.dtype)
z
<class 'numpy.ndarray'> float64
Out[98]:
array([[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 42., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]])
In [58]:
## Data Type conversion IN PLACE!
In [ ]:
In [99]:
z = z.astype(np.float32, copy=True)
In [65]:
print(z.dtype)
float32
In [100]:
z
Out[100]:
array([[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 42., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]], dtype=float32)
In [101]:
z = z.astype(np.int32, copy=True)
print(z.dtype)
z
int32
Out[101]:
array([[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 42, 0, 0],
[ 0, 0, 0, 0, 0, 0]])
In [102]:
zz = z.astype(np.float32, copy=True)
zz
Out[102]:
array([[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 42., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]], dtype=float32)
In [111]:
zz[1,1]=5000000000
zz
Out[111]:
array([[0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00],
[0.0e+00, 5.0e+09, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00],
[0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00],
[0.0e+00, 0.0e+00, 0.0e+00, 4.2e+01, 0.0e+00, 0.0e+00],
[0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00, 0.0e+00]],
dtype=float32)
In [114]:
2**64
Out[114]:
18446744073709551616
In [115]:
2**128
Out[115]:
340282366920938463463374607431768211456
In [109]:
2**32
Out[109]:
4294967296
In [110]:
2**31
Out[110]:
2147483648
In [112]:
zzz = zz.astype(np.int64, copy=True)
zzz
Out[112]:
array([[ 0, 0, 0, 0, 0,
0],
[ 0, 5000000000, 0, 0, 0,
0],
[ 0, 0, 0, 0, 0,
0],
[ 0, 0, 0, 42, 0,
0],
[ 0, 0, 0, 0, 0,
0]], dtype=int64)
In [116]:
ones = np.full((5,6), 1, dtype=np.int32)
ones
Out[116]:
array([[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1]])
In [117]:
ones=ones+z
ones
Out[117]:
array([[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 43, 1, 1],
[ 1, 1, 1, 1, 1, 1]])
In [69]:
ones = ones.astype(np.int32, copy=True)
ones
Out[69]:
array([[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 43, 1, 1],
[ 1, 1, 1, 1, 1, 1]])
In [118]:
ones*z
Out[118]:
array([[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 1806, 0, 0],
[ 0, 0, 0, 0, 0, 0]])
In [119]:
twos = np.full((5,6), 2, dtype=np.int32)
twos
Out[119]:
array([[2, 2, 2, 2, 2, 2],
[2, 2, 2, 2, 2, 2],
[2, 2, 2, 2, 2, 2],
[2, 2, 2, 2, 2, 2],
[2, 2, 2, 2, 2, 2]])
In [120]:
ones*twos
Out[120]:
array([[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 86, 2, 2],
[ 2, 2, 2, 2, 2, 2]])
In [121]:
np.matmul(ones, twos)
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-121-c47a4b300c6e> in <module>() ----> 1 np.matmul(ones, twos) ValueError: shapes (5,6) and (5,6) not aligned: 6 (dim 1) != 5 (dim 0)
In [ ]:
## We mismatching dimensions for matrix multiplication
In [122]:
twos.transpose()
Out[122]:
array([[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2],
[2, 2, 2, 2, 2]])
In [123]:
mult = np.matmul(ones, twos.transpose())
mult
Out[123]:
array([[12, 12, 12, 12, 12],
[12, 12, 12, 12, 12],
[12, 12, 12, 12, 12],
[96, 96, 96, 96, 96],
[12, 12, 12, 12, 12]])
In [125]:
mult1 = np.matmul(ones.transpose(), twos)
mult1
Out[125]:
array([[10, 10, 10, 10, 10, 10],
[10, 10, 10, 10, 10, 10],
[10, 10, 10, 10, 10, 10],
[94, 94, 94, 94, 94, 94],
[10, 10, 10, 10, 10, 10],
[10, 10, 10, 10, 10, 10]])
In [126]:
mult.sum(),mult1.sum()
Out[126]:
(720, 864)
In [ ]:
# Anyone remembers the algorithm for Matrix Multiplication, I don't but numpy lets us not worry about it :)
In [127]:
np.sqrt(mult)
Out[127]:
array([[3.46410162, 3.46410162, 3.46410162, 3.46410162, 3.46410162],
[3.46410162, 3.46410162, 3.46410162, 3.46410162, 3.46410162],
[3.46410162, 3.46410162, 3.46410162, 3.46410162, 3.46410162],
[9.79795897, 9.79795897, 9.79795897, 9.79795897, 9.79795897],
[3.46410162, 3.46410162, 3.46410162, 3.46410162, 3.46410162]])
In [128]:
mult
Out[128]:
array([[12, 12, 12, 12, 12],
[12, 12, 12, 12, 12],
[12, 12, 12, 12, 12],
[96, 96, 96, 96, 96],
[12, 12, 12, 12, 12]])
In [129]:
mult**2
Out[129]:
array([[ 144, 144, 144, 144, 144],
[ 144, 144, 144, 144, 144],
[ 144, 144, 144, 144, 144],
[9216, 9216, 9216, 9216, 9216],
[ 144, 144, 144, 144, 144]], dtype=int32)
In [ ]:
# np.sin, np.cos, np.exp(power), and so on
In [132]:
[1,3,65]+[4,6,7]
Out[132]:
[1, 3, 65, 4, 6, 7]
In [130]:
vstack = np.vstack((ones,twos,ones)) # notice the double parenthesis, ie we supply a tuple as an argument!
vstack
Out[130]:
array([[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 43, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 2, 2, 2, 2, 2, 2],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 43, 1, 1],
[ 1, 1, 1, 1, 1, 1]])
In [ ]:
In [131]:
hstack = np.hstack((twos,ones, twos))
hstack
Out[131]:
array([[ 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
2, 2],
[ 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
2, 2],
[ 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
2, 2],
[ 2, 2, 2, 2, 2, 2, 1, 1, 1, 43, 1, 1, 2, 2, 2, 2,
2, 2],
[ 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2,
2, 2]])
In [82]:
hsplit = np.hsplit(hstack, 3)
print(type(hsplit))
hsplit
<class 'list'>
Out[82]:
[array([[ 1, 1, 1, 1],
[ 1, 1, 1, 1],
[ 1, 1, 1, 1],
[ 1, 1, 1, 43],
[ 1, 1, 1, 1]]), array([[1, 1, 2, 2],
[1, 1, 2, 2],
[1, 1, 2, 2],
[1, 1, 2, 2],
[1, 1, 2, 2]]), array([[2, 2, 2, 2],
[2, 2, 2, 2],
[2, 2, 2, 2],
[2, 2, 2, 2],
[2, 2, 2, 2]])]
In [133]:
h2 = np.hsplit(hstack, 6)
print(type(h2))
h2
<class 'list'>
Out[133]:
[array([[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]]), array([[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]]), array([[1, 1, 1],
[1, 1, 1],
[1, 1, 1],
[1, 1, 1],
[1, 1, 1]]), array([[ 1, 1, 1],
[ 1, 1, 1],
[ 1, 1, 1],
[43, 1, 1],
[ 1, 1, 1]]), array([[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]]), array([[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]])]
In [ ]:
## vsplit is similar along vertical axis and array_split lets you specify axis
View or Shallow Copy¶
In [134]:
v = a.view()
In [135]:
v
Out[135]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[20, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
In [137]:
v is a
Out[137]:
False
In [136]:
v == a
Out[136]:
array([[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True],
[ True, True, True, True, True]])
In [138]:
v.base is a
Out[138]:
False
In [117]:
## Turns out tutorial is outdated, new comparision for bases should be
In [139]:
v.base is a.base
Out[139]:
True
In [140]:
v[2,2]=999
In [141]:
v
Out[141]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[ 20, 11, 999, 13, 14],
[ 15, 16, 17, 18, 19]])
In [142]:
a
Out[142]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[ 20, 11, 999, 13, 14],
[ 15, 16, 17, 18, 19]])
In [143]:
s = a[:,1:3]
s
Out[143]:
array([[ 1, 2],
[ 6, 7],
[ 11, 999],
[ 16, 17]])
In [144]:
s.base is a.base
Out[144]:
True
In [145]:
s[:] = 55
s
Out[145]:
array([[55, 55],
[55, 55],
[55, 55],
[55, 55]])
In [146]:
a
Out[146]:
array([[ 0, 55, 55, 3, 4],
[ 5, 55, 55, 8, 9],
[20, 55, 55, 13, 14],
[15, 55, 55, 18, 19]])
In [148]:
b = a
In [149]:
b.base is a.base
Out[149]:
True
Remember Slice is a shallow copy to original(ie reference)¶
Deep Copy¶
In [150]:
d = a.copy()
d
Out[150]:
array([[ 0, 55, 55, 3, 4],
[ 5, 55, 55, 8, 9],
[20, 55, 55, 13, 14],
[15, 55, 55, 18, 19]])
In [151]:
d.base is a.base
Out[151]:
False
In [152]:
d[1,1]=777
d
Out[152]:
array([[ 0, 55, 55, 3, 4],
[ 5, 777, 55, 8, 9],
[ 20, 55, 55, 13, 14],
[ 15, 55, 55, 18, 19]])
In [153]:
a
Out[153]:
array([[ 0, 55, 55, 3, 4],
[ 5, 55, 55, 8, 9],
[20, 55, 55, 13, 14],
[15, 55, 55, 18, 19]])
In [108]:
a[:1]
Out[108]:
array([[ 0, 55, 55, 3, 4]])
In [156]:
a[0,1]
Out[156]:
55
In [158]:
f = d == 55
f
Out[158]:
array([[False, True, True, False, False],
[False, False, True, False, False],
[False, True, True, False, False],
[False, True, True, False, False]])
In [154]:
e = d == a[0,1]
e
Out[154]:
array([[False, True, True, False, False],
[False, False, True, False, False],
[False, True, True, False, False],
[False, True, True, False, False]])
In [155]:
e.dtype
Out[155]:
dtype('bool')
Now d is completely separate object from a !!¶
In [ ]:
## More on copy vs sliceing: https://stackoverflow.com/questions/47181092/numpy-views-vs-copy-by-slicing
# key depends on how you slice
# slice by rows -> view (shallow copy)
# slice by column -> deep copy because not contigous
In [167]:
f=np.ones((5,6), dtype='int32')
f[2,3]=0
f
Out[167]:
array([[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 0, 1, 1],
[1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1]])
In [168]:
f.nonzero()
Out[168]:
(array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3,
3, 4, 4, 4, 4, 4, 4], dtype=int64),
array([0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5, 0, 1, 2, 4, 5, 0, 1, 2, 3, 4,
5, 0, 1, 2, 3, 4, 5], dtype=int64))
In [159]:
d.nonzero()
Out[159]:
(array([0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3],
dtype=int64),
array([1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4],
dtype=int64))
Useful NumPY Functions and Methods (there are more!)¶
- Array Creation
arange, array, copy, empty, empty_like, eye, fromfile, fromfunction, identity, linspace, logspace, mgrid, ogrid, ones, ones_like, r, zeros, zeros_like
- Conversions
ndarray.astype, atleast_1d, atleast_2d, atleast_3d, mat
- Manipulations
array_split, column_stack, concatenate, diagonal, dsplit, dstack, hsplit, hstack, ndarray.item, newaxis, ravel, repeat, reshape, resize, squeeze, swapaxes, take, transpose, vsplit, vstack
- Questions
all, any, nonzero, where
- Ordering
argmax, argmin, argsort, max, min, ptp, searchsorted, sort
- Operations
choose, compress, cumprod, cumsum, inner, ndarray.fill, imag, prod, put, putmask, real, sum
- Basic Statistics
cov, mean, std, var
- Basic Linear Algebra
cross, dot, outer, linalg.svd, vdot
In [ ]:
Boolean Arrays¶
In [169]:
a
Out[169]:
array([[ 0, 55, 55, 3, 4],
[ 5, 55, 55, 8, 9],
[20, 55, 55, 13, 14],
[15, 55, 55, 18, 19]])
In [170]:
b = a > 17
b
Out[170]:
array([[False, True, True, False, False],
[False, True, True, False, False],
[ True, True, True, False, False],
[False, True, True, True, True]])
In [136]:
## USeful in assignments
In [171]:
a[b] = 500
a
Out[171]:
array([[ 0, 500, 500, 3, 4],
[ 5, 500, 500, 8, 9],
[500, 500, 500, 13, 14],
[ 15, 500, 500, 500, 500]])
In [123]:
a[b]
Out[123]:
array([500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500])
Automatic Reshaping¶
In [172]:
a = np.arange(24)
a
Out[172]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23])
To change the dimensions of an array, you can omit one of the sizes which will then be deduced automatically:¶
In [177]:
a.shape = 6, -1
a
Out[177]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]])
In [179]:
a.shape = -1, 2
a
Out[179]:
array([[ 0, 1],
[ 2, 3],
[ 4, 5],
[ 6, 7],
[ 8, 9],
[10, 11],
[12, 13],
[14, 15],
[16, 17],
[18, 19],
[20, 21],
[22, 23]])
Broadcasting¶
The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes
usually leads to efficient algorithm implementations.
In [180]:
e = np.arange(6)
f = np.arange(3)*10 #same as np.arange(0,30,10)
e,f
Out[180]:
(array([0, 1, 2, 3, 4, 5]), array([ 0, 10, 20]))
In [181]:
np.arange(0,30,10)
Out[181]:
array([ 0, 10, 20])
In [182]:
e + f
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-182-079705e5a008> in <module>() ----> 1 e + f ValueError: operands could not be broadcast together with shapes (6,) (3,)
In [183]:
e.reshape(6,1) + f.reshape(1,3)
Out[183]:
array([[ 0, 10, 20],
[ 1, 11, 21],
[ 2, 12, 22],
[ 3, 13, 23],
[ 4, 14, 24],
[ 5, 15, 25]])
In [184]:
e.reshape(6,1) * f.reshape(1,3)
Out[184]:
array([[ 0, 0, 0],
[ 0, 10, 20],
[ 0, 20, 40],
[ 0, 30, 60],
[ 0, 40, 80],
[ 0, 50, 100]])
In [185]:
mtab = np.arange(10).reshape(10,1) * np.arange(10).reshape(1,10)
mtab
Out[185]:
array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18],
[ 0, 3, 6, 9, 12, 15, 18, 21, 24, 27],
[ 0, 4, 8, 12, 16, 20, 24, 28, 32, 36],
[ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45],
[ 0, 6, 12, 18, 24, 30, 36, 42, 48, 54],
[ 0, 7, 14, 21, 28, 35, 42, 49, 56, 63],
[ 0, 8, 16, 24, 32, 40, 48, 56, 64, 72],
[ 0, 9, 18, 27, 36, 45, 54, 63, 72, 81]])
In [186]:
g=np.arange(12)
h=np.arange(4)*100
g,h
Out[186]:
(array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]), array([ 0, 100, 200, 300]))
In [187]:
g.reshape(4,3)
Out[187]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]])
In [189]:
g.reshape(4,3)+h.reshape(4,1)
Out[189]:
array([[ 0, 1, 2],
[103, 104, 105],
[206, 207, 208],
[309, 310, 311]])
In [190]:
g.reshape(4,3)+50
Out[190]:
array([[50, 51, 52],
[53, 54, 55],
[56, 57, 58],
[59, 60, 61]])
In [40]:
## We can do this with more dimensions as well! But our 3D thinking heads will hurt!
In [ ]:
## Basic image processing with NumPy (PIL is another option!)
In [191]:
url='https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png'
save_file='lena.png'
In [ ]:
#Original: November 1972, photographed by Dwight Hooker.
#This 512x512 electronic/mechanical scan of a section of the full portrait: Alexander Sawchuk (1972) and two others
#Permission
#Use of this 512x512 scan is "overlooked" and by implication permitted by Playboy
In [192]:
import urllib.request
import shutil
In [193]:
# Current recommended way of reading a file from url and then saving it immediately and closing the file
with urllib.request.urlopen(url) as response, open(save_file, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
Imageio is a Python library that provides an easy interface to read and write a wide range of image data.¶
Including support for animated images, volumetric data, and scientific formats. It is cross-platform, runs on Python 2.7 and 3.4+, and is easy to install.
Main website: http://imageio.github.io
In [194]:
import imageio
In [195]:
lena = imageio.imread('lena.png')
In [196]:
l2 = imageio.imread('https://upload.wikimedia.org/wikipedia/en/7/7d/Lenna_%28test_image%29.png')
In [197]:
type(lena)
Out[197]:
imageio.core.util.Image
What is imageio.core.util.Image ?¶
We can find its Parent class with myclass.bases
In [198]:
imageio.core.util.Image.__bases__
Out[198]:
(numpy.ndarray,)
In [199]:
import matplotlib.pyplot as plt
In [200]:
plt.imshow(lena)
Out[200]:
<matplotlib.image.AxesImage at 0x8a40da0>
In [201]:
lena.shape
Out[201]:
(512, 512, 3)
In [203]:
lena.ndim
Out[203]:
3
In [202]:
lena.size
Out[202]:
786432
In [204]:
lena = imageio.imread('lena.png')
In [208]:
#lena[60:160,60:160,]=90
lena[30:60,30:60,0]= 10
lena[60:160,60:160,1]=180
lena[160:260,160:260,2]=180
plt.imshow(lena)
Out[208]:
<matplotlib.image.AxesImage at 0x8aeac88>
In [161]:
plt.imshow(lena)
Out[161]:
<matplotlib.image.AxesImage at 0x8b6a7f0>
In [ ]:
# working with URL files directly
In [209]:
import requests
from io import BytesIO
import matplotlib.image as mpimg
In [210]:
lena2 = mpimg.imread('lena.png')
type(lena2)
Out[210]:
numpy.ndarray
In [211]:
plt.imshow(lena2)
Out[211]:
<matplotlib.image.AxesImage at 0x904eac8>
In [32]:
response = requests.get(url)
lena3 = mpimg.imread(BytesIO(response.content))
type(lena3)
# this method copies the image directly from the stream into our img
Out[32]:
numpy.ndarray
In [33]:
plt.imshow(lena3)
Out[33]:
<matplotlib.image.AxesImage at 0xc0f51d0>
In [213]:
from PIL import Image
In [214]:
response = requests.get(url)
img = Image.open(BytesIO(response.content))
# this method does not copies the image directly from the stream into our img
In [67]:
# img.show() # in external viewer not in Jupyter!
In [215]:
plt.imshow(img)
Out[215]:
<matplotlib.image.AxesImage at 0x99c79e8>
In [216]:
img.size,img.format_description
Out[216]:
((512, 512), 'Portable network graphics')
In [72]:
dir(img)
Out[72]:
['_Image__transformer', '_PngImageFile__idat', '__array_interface__', '__class__', '__copy__', '__del__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_close_exclusive_fp_after_loading', '_copy', '_crop', '_dump', '_ensure_mutable', '_exclusive_fp', '_expand', '_min_frame', '_new', '_open', '_repr_png_', '_seek_check', 'alpha_composite', 'category', 'close', 'convert', 'copy', 'crop', 'decoderconfig', 'decodermaxblock', 'draft', 'effect_spread', 'encoderconfig', 'encoderinfo', 'filename', 'filter', 'format', 'format_description', 'fp', 'frombytes', 'fromstring', 'getbands', 'getbbox', 'getchannel', 'getcolors', 'getdata', 'getextrema', 'getim', 'getpalette', 'getpixel', 'getprojection', 'height', 'histogram', 'im', 'info', 'load', 'load_end', 'load_prepare', 'load_read', 'map', 'mode', 'offset', 'palette', 'paste', 'png', 'point', 'putalpha', 'putdata', 'putpalette', 'putpixel', 'pyaccess', 'quantize', 'readonly', 'remap_palette', 'resize', 'rotate', 'save', 'seek', 'show', 'size', 'split', 'tell', 'text', 'thumbnail', 'tile', 'tobitmap', 'tobytes', 'toqimage', 'toqpixmap', 'tostring', 'transform', 'transpose', 'verify', 'width']
In [178]:
type(img)
Out[178]:
PIL.PngImagePlugin.PngImageFile
In [217]:
import matplotlib.pyplot as plt
In [221]:
# Build a vector of 10000 normal deviates with variance 0.5^2 and mean 2
mu, sigma = 2, 0.5
v = np.random.normal(mu,sigma,20000)
In [222]:
v[:20], v.mean(),v.max(),v.min()
Out[222]:
(array([2.31632739, 2.219109 , 1.53796432, 2.46312243, 1.90145972,
1.47151823, 2.21456619, 2.05749909, 1.91774699, 1.94418668,
2.51501025, 2.43988414, 2.47336132, 2.54512566, 0.95851055,
0.96754106, 3.31049021, 1.72561352, 1.58489223, 1.71679022]),
1.9930709054847882,
3.9389775779957246,
0.2489672552480482)
In [223]:
# Plot a normalized histogram with 50 bins
plt.hist(v, bins=500, normed=1) # matplotlib version (plot)
plt.show()
In [224]:
# Compute the histogram with numpy and then plot it
(n, bins) = np.histogram(v, bins=500, normed=True) # NumPy version (no plot)
In [225]:
plt.plot(.5*(bins[1:]+bins[:-1]), n)
plt.show()
Resources / Books¶
Python for Data Analysis # excellent, by the creator of Pandas
NumPy Cookbook # some good recipes but uneven, this author has many books on NumPy
From Python to Numpy # nice free resource