Section I: (Overview of Linear Algebra: Vectors)
Section II: (Overview of Linear Alagebra: Matrices)
Section III: (Solving System of Linear Equations)
2
variablesm
variablesSection IV: (Linear Transformation and Matrices)
Section V: (Eigen Decomposition and its Applications)
Section VI: (Singular Value Decomposition and its Applications)
# Unlike the other modules, we have been working so far, you have to download and install...
# To install this library in Jupyter notebook
import sys
!{sys.executable} -m pip install -q --upgrade pip
import numpy as np
import numpy.linalg
import math
import scipy
from matplotlib import pyplot as plt
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
Some of the codes of this notebook are adapted from:
We come accross the concept of vectors in the domains of physics, engineering, mathematics, computer science and more. Each field's interpretation of what a vector is a bit different:
Vectors can be 2-dimensional, 3-dimensional and so on to N-dimensional. The two and three dimensional vecgtors are pretty easy to visualize. If we are able to understand and visualize the vector operations in 2-dimensions, we can map the concepts to larger dimensions as well. For example, to model the age, weight, daily hours of sleep, weekly hours of exercise, and blood pressure of an individual, we need a five dimensional vector.
order/rank/degree/dimension of the vector
.$\hspace{2 cm}\overrightarrow{\rm v} = (a, b) = \begin{bmatrix} a \\ b \end{bmatrix} \hspace{2 cm}\overrightarrow{\rm v} = (2, 5) = \begin{bmatrix} 2 \\ 5 \end{bmatrix}$
$\hspace{2 cm}\overrightarrow{\rm v} = a\hat{i} + b\hat{j} \hspace{3 cm} \overrightarrow{\rm v} = 2\hat{i} + 5\hat{j}$
$\hspace{2 cm}\overrightarrow{\rm v} = a\begin{bmatrix} 1 \\ 0 \end{bmatrix} + b\begin{bmatrix} 0 \\ 1 \end{bmatrix} \hspace{1.5 cm} \overrightarrow{\rm v} = 2\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 5\begin{bmatrix} 0 \\ 1 \end{bmatrix} $
Algebraically, vectors of rank 3 consists of three scalar values written as comma separated list of numbers horizontally or may be from top to bottom.
The length of the vector is the number of scalar values in the vector, and is also called the order/rank/degree/dimension of the vector
, in this case it is three.
Algebraically, in $\mathbb{R}3$ space, a vector $\overrightarrow{\rm v}$ can be written as:
$\hspace{2 cm}\overrightarrow{\rm v} = (a, b, c) = \begin{bmatrix} a \\ b \\ c \end{bmatrix} \hspace{5 cm}\overrightarrow{\rm v} = (2, 5, 3) = \begin{bmatrix} 2 \\ 5 \\ 3 \end{bmatrix}$
$\hspace{2 cm}\overrightarrow{\rm v} = a\hat{i} + b\hat{j} + c\hat{k} \hspace{5 cm} \overrightarrow{\rm v} = 2\hat{i} + 5\hat{j} + 3\hat{k}$
$\hspace{2 cm}\overrightarrow{\rm v} = a\begin{bmatrix} 1 \\ 0 \\0 \end{bmatrix} + b\begin{bmatrix} 0 \\ 1 \\0 \end{bmatrix} + c\begin{bmatrix} 0 \\ 0 \\1 \end{bmatrix} \hspace{3 cm} \overrightarrow{\rm v} = 2\begin{bmatrix} 1 \\ 0 \\0 \end{bmatrix} + 5\begin{bmatrix} 0 \\ 1 \\0 \end{bmatrix} + 3\begin{bmatrix} 0 \\ 0 \\1 \end{bmatrix}$
Example 1: Creating Vector $[3,2]$ having tail at origin
# To better visualize and plot, you need to import following module/script having three helper functions
# plot_vector(), plot_linear_transformation, plot_linear_transformations
from plot_helper import *
v = [(3,2)] # A list having a single tuple of two elements representing x and y component of vector
plot_vector(v)
Example 2: Creating four vectors, one in each quadrant having their tails at origin
v = [(3, 3), (-3, 1), (-3,-2),(3,-2)] # A list having four tuples of two elements each
plot_vector(v)
Example 3: Creating a vector $[4,3]$ having its tail at $[2,2]$
v = [(4,3)] # A list having a single tuple of two elements representing x and y component of vector
tail = [(2,2)] # A list having a single tuple of two elements representing tail of vector
plot_vector(v, tail)
Example 4: Three vectors with their tails at $[2,2]$
v = [(1, 3), (3, 3), (4, 6)] # A list having three tuples of two elements each
tail = [(2, 2)] # A list having a single tuple of two elements
plot_vector(v, tail)
Example 5: Three vectors with different tails
v = [(1, 3), (4, 4), (4, 6)]
tails = [(3, 2), (-6, -6), (-1, 2)]
plot_vector(v, tails)
Example 6: Translate above three vectors to origin
v = [(1, 3), (4, 4), (4, 6)]
tails = [(0, 0), (0, 0), (0, 0)]
plot_vector(v, tails)
non-negative
number that describes the extent of the vector in space, and is sometimes referred to as the vector’s magnitude or the norm.Euclidean norm
as it is calculated as the Euclidean distance from the origin
. The result is a positive
distance value.square root
of the sum of the squared vector values.taxicab norm
or the Manhattan norm
.maximum value
of the vector, hence the name.Determine the magnitude of following vectors:
Example: Determine the magnitude of vector $\overrightarrow{\rm AB}$ whose starting point is at $A = (-2, 2)$ and ending point is $B = (2, 8)$
import numpy as np
# Translating the vector to origin
a = np.array([-2, 2])
b = np.array([2, 8])
v = np.array([b[0]-a[0], b[1]-a[1]]) # v = (4,6)
# Plotting both the vectors
vectors = [v, v ] # A list having two vectors
tails = [[-2,2], [0,0]]
plot_vector(vectors, tails)
print("v = ", v)
# calculating L1 norm
l1 = numpy.linalg.norm(v, ord=1)
l1 = np.abs(v[0]) + np.abs(v[1])
print("L1 Norm = ",l1)
# calculating L2 norm
l2 = numpy.linalg.norm(v, ord=2)
l2 = (v[0]**2 + v[1]**2 )**(1/2)
print("L2 Norm = ",l2)
# calculating Squared L2 norm
sq_l2 = (numpy.linalg.norm(v, ord=2))**2
sq_l2 = (v[0]**2 + v[1]**2 )
print("Squared L2 Norm = ",sq_l2)
# calculating Max norm
maxnorm = numpy.linalg.norm(v, ord=np.inf)
maxnorm = np.max([np.abs(v[0]), np.abs(v[1])])
print("L∞ = ", maxnorm)
v = [4 6] L1 Norm = 10 L2 Norm = 7.211102550927978 Squared L2 Norm = 52 L∞ = 6
A vector having its tail at origin is called position vector
Geometrically Measuring the Direction of a Vector:
Mathematically Measuring the Direction of a Vector:
Example 1: Find the direction of a vector whose coordinates are $(4, 6)$
v = np.array([4, 6])
print("v = ", v)
theta_rad = math.atan(v[1]/v[0]) # acos, asin, and atan take a ratio as input and return an angle in radians.
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
print("Angle in radians: ", theta_rad)
print("Shortest angle from x-axis: ", theta_deg)
# Since, both the coordinates are positive, that means the angle exists in the first quadrant
# So the angle is already computed from positive x-axis
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
plot_vector([v])
v = [4 6] Angle in radians: 0.982793723247329 Shortest angle from x-axis: 56.309932474020215 Counter-Clockwise angle in degrees from positive x-axis: 56.309932474020215
Example 2: Find the direction of a vector whose coordinates are $(-4, 6)$
v = np.array([-4, 6])
print("v = ", v)
theta_rad = math.atan(v[1]/v[0]) # acos, asin, and atan take a ratio as input and return an angle in radians.
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
print("Angle in radians: ", theta_rad)
print("Shortest angle from x-axis: ", theta_deg)
# Since, x-coordinate is negative and y-coordinate is positive, that means, the angle exists in the second
# quadrant. Since, the angle is negative, that means, it is measured from negative x-axis in clock-wise direction
# So we have to add 180 degree to get the angle from positive x-axis in counter-clockwise direction
theta_deg = theta_deg + 180
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
plot_vector([v])
v = [-4 6] Angle in radians: -0.982793723247329 Shortest angle from x-axis: -56.309932474020215 Counter-Clockwise angle in degrees from positive x-axis: 123.69006752597979
Example 3: Find the direction of a vector whose coordinates are $(-4, -6)$
v = np.array([-4, -6])
print("v = ", v)
theta_rad = math.atan(v[1]/v[0]) # acos, asin, and atan take a ratio as input and return an angle in radians.
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
print("Angle in radians: ", theta_rad)
print("Angle in degrees: ", theta_deg)
# Since, both the coordinates are negative, that means, the angle exists in the third quadrant
# Since, the angle is positive, that means, it is measured from negative x-axis in counter-clockwise direction
# So we have to add 180 degree to get the angle from positive x-axis in counter-clockwise direction
theta_deg = theta_deg + 180
print("Angle in degrees from x-axis: ", theta_deg)
plot_vector([v])
v = [-4 -6] Angle in radians: 0.982793723247329 Angle in degrees: 56.309932474020215 Angle in degrees from x-axis: 236.30993247402023
Example 4: Find the direction of a vector whose coordinates are $(4, -6)$
v = np.array([4, -6])
print("v = ", v)
theta_rad = math.atan(v[1]/v[0]) # acos, asin, and atan take a ratio as input and return an angle in radians.
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
print("Angle in radians: ", theta_rad)
print("Shortest angle from x-axis: ", theta_deg)
#Since, x-coordinate is positive and y-coordinate is negative, that means, the angle exists in the fourth quadrant
#Since, the angle is negative, that means, it is measured from positive x-axis in clockwise direction
#So we have to add 360 degree to get the angle from positive x-axis in counter-clockwise direction
theta_deg = theta_deg + 360
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
plot_vector([v])
v = [ 4 -6] Angle in radians: -0.982793723247329 Shortest angle from x-axis: -56.309932474020215 Counter-Clockwise angle in degrees from positive x-axis: 303.69006752597977
$ cosθ = \frac{F_x}{F} \implies F_x = F.cosθ $
$ sinθ = \frac{F_y}{F} \implies F_y = F.sinθ $
Example 1: A force $\overrightarrow{\rm F}$ of 10 N is applied at an angle of 30º along the horizontal surface. Resolve the vector into its components. Verify your answer by calculating its magnitude and direction from its components.
# calculating x and y components of vector F
fx = 10*math.cos(30 * math.pi/180) # sin, cos and tan take input angle in radians and return a real value.
fy = 10*math.sin(30 * math.pi/180) # sin, cos and tan take input angle in radians and return a real value.
print("Fx: %.2f" %fx, "N")
print("Fy: %.2f" %fy, "N")
# calculating magnitude for verification
f = numpy.linalg.norm([fx, fy], ord=2)
print("|F| = ",f)
# calculating angle for verification
theta_rad = math.atan(fy/fx) # returned angle is in radians.
theta_deg = theta_rad*(180/math.pi) # convert it into degrees
# Since, x and y-components are both positive, that means the vector is in first quadrant
print("Angle: %.2f" % theta_deg, "degrees")
# Plot the vector
plot_vector([(fx,fy)])
Fx: 8.66 N Fy: 5.00 N |F| = 10.0 Angle: 30.00 degrees
Example 2: Given a vector $\mathbf{v}$ having magnitude of 4 and direction of 45 degrees in $\mathbb{R}^2$. Find out its x and y-components.
Verify your answer by calculating its magnitude and direction from its components.
# calculating x and y components of vector F
vx = 4*math.cos(45 * math.pi/180) # sin, cos and tan take input angle in radians and return a real value.
vy = 4*math.sin(45 * math.pi/180) # sin, cos and tan take input angle in radians and return a real value.
print("vx: %.2f" %vx)
print("vy: %.2f" %vy)
# calculating magnitude for verification
mag = numpy.linalg.norm([vx, vy], ord=2)
print("|v| = ", mag)
# calculating angle for verification
theta_rad = math.atan(vy/vx) # returned angle is in radians.
theta_deg = theta_rad*(180/math.pi) # convert it into degrees
# Since, x and y-components are both positive, that means the vector is in first quadrant
print("Angle: %.2f" % theta_deg, "degrees")
# Plot the vector
plot_vector([(vx,vy)])
vx: 2.83 vy: 2.83 |v| = 4.0 Angle: 45.00 degrees
In contrast to a vector in 2-D space having an x-component and a y-component with one angle in between, a vector in 3-D space will have an x-component, a y-component and a z-component with three angles in between.
Given the three angles and the magnitude of a vector, we can calculate its components using following formulae:
Consider a vector: $\overrightarrow{\rm v} = (4,6,4) = 4\hat{i} + 6\hat{j} + 4\hat{k}$
Example 1: Given a vector having magnitude of 8.2463 making angles of 60.98, 43.3, and 60.98 with x,y, and z-axis respectively in $\mathbb{R}^3$. Find out its x, y and z-components. Verify your answer by calculating the vector's magnitude from components.
# calculating x and y components of vector F
fx = 8.2463*math.cos(60.98 * math.pi/180)
fy = 8.2463*math.cos(43.3 * math.pi/180)
fz = 8.2463*math.cos(60.98 * math.pi/180)
print("fx: %.2f" %fx)
print("fy: %.2f" %fy)
print("fx: %.2f" %fz)
# calculating magnitude for verification
mag = numpy.linalg.norm([fx, fy, fz], ord=2)
print("|f| = %.3f" %mag)
fx: 4.00 fy: 6.00 fx: 4.00 |f| = 8.248
$ \hspace{5.0cm} \overrightarrow{\rm c} = \overrightarrow{\rm a} + \overrightarrow{\rm b} $
Vector addition can be performed graphically, using the head-to-tail as shown in this figure. (First, the two vectors a
and b
are placed together such that the head of vector a
connects the tail of vector b
. Next, to find the sum, a resultant vector c
is drawn such that it connects the tail of a
to the head of b
.)
Vection addition can also be done by simply performing an element by elementt addition.
$ \hspace{3.0cm} \overrightarrow{\rm c} = (a1 + b1, a2 + b2, a3 + b3) $
Example 1: Consider a vector $\overrightarrow{\rm a}$, having its tail at point $(-1, 3)$ and head at point $(5,2)$. Consider another vector $\overrightarrow{\rm b}$, having its tail at point $(1, -2)$ and head at point $(-2,2)$. Determine the resultant sum vector \overrightarrow{\rm c}. Also, give the magnitude and angle of the resultant vector.
# Translating the two vectors to origin
u = np.array([5-(-1), 2-3])
v = np.array([-2-1, 2-(-2)])
print("U = ", u)
print("V = ", v)
# calculate the resultant vector R
r = u + v
print("R = ", r)
# calculating magnitude of R from its components using L2 norm
mag = numpy.linalg.norm([r[0], r[1]], ord=2)
mag = (r[0]**2 + r[1]**2)**(1/2)
print("|R| = ", mag)
# calculating angle of resultant vector R
theta_rad = math.atan(r[1]/r[0]) # acos, asin, and atan take a ratio as input and return an angle in radians
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
# Since, both x and y coordinates are positive, that means the angle exists in the first quadrant
# So no need to add anything
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
vectors = [u, v, r]
tails = [(0,0), u, (0,0)]
plot_vector(vectors, tails)
U = [ 6 -1] V = [-3 4] R = [3 3] |R| = 4.242640687119285 Counter-Clockwise angle in degrees from positive x-axis: 45.0
Example 2: Given two vectors, U = 10 m, Φ = 30 degrees and V = 20m, Φ = 60 degrees, determine their sum. Then, calculate the magnitude and the angle of the resultant vector using the component method
import numpy as np
import math as m
# calculating x and y components of vector U
ux = 10*math.cos(30*math.pi/180) # sin, cos and tan take input angle in radians and return the ratio.
uy = 10*math.sin(30*math.pi/180)
print("Ux: %.2f" %ux, "m")
print("Uy: %.2f" %uy, "m")
# calculating x and y components of vector V
vx = 20*m.cos(60*math.pi/180)
vy = 20*m.sin(60*math.pi/180)
print("Vx: %.2f" %vx, "m")
print("Vy: %.2f" %vy, "m")
# calculate the sum of the two vectors R = U + V
rx = ux + vx
ry = uy + vy
r = np.array([rx, ry])
print("Resultant vector R: ", r)
# calculating magnitude of R from its components using L2 norm
mag = numpy.linalg.norm([rx, ry], ord=2)
mag = (rx**2 + ry**2)**(1/2)
print("|R| = ", mag)
# calculating angle of resultant vector R
theta_rad = math.atan(r[1]/r[0]) # acos, asin, and atan take a ratio as input and return an angle in radians
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
# Since, both the coordinates are positive, that means the angle exists in the first quadrant
# So the angle is already computed from positive x-axis
theta_deg = math.degrees(theta_rad)
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
vectors = [(ux,uy), (vx,vy), (rx,ry)]
tails = [(0,0), (ux,uy), (0,0)]
plot_vector(vectors, tails)
Ux: 8.66 m Uy: 5.00 m Vx: 10.00 m Vy: 17.32 m Resultant vector R: [18.66025404 22.32050808] |R| = 29.093129111764092 Counter-Clockwise angle in degrees from positive x-axis: 50.103909361017095
Multiplication by a scalar is a way of changing the magnitude and/or direction of a vector.
The multiplication of a scalar value with a vector A will yield another vector.
Example 1: Given the vector a = (-4, -6). If you multiply this vector by $-1/2$, its length (magnitude) halves and direction is reversed.
a = np.array([-4, -6])
print ("Vector A: ", a)
mag = numpy.linalg.norm(a, ord=2) # calculate magnitude using L2 norm
print("|A| = ",mag)
# calculating angle of vector A
theta_rad = math.atan(a[1]/a[0])
theta_deg = theta_rad*(180/math.pi)
# Since, both the coordinates are negative, that means, the angle exists in the third quadrant
# Since, the angle is positive, that means, it is measured from negative x-axis in counter-clockwise direction
# So we have to add 180 degree to get the angle from positive x-axis in counter-clockwise direction
theta_deg = theta_deg + 180
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
# Multiplying the vector A by -0.5 will give a new vector B
b = a * -0.5
print ("\nVector B: ", b)
# To calculate magnitude using L2 norm
mag = numpy.linalg.norm(b, ord=2)
mag = (b[0]**2 + b[1]**2) **(1/2)
print("|-A| = ",mag)
# calculating angle of vector B
theta_rad = m.atan(b[1]/b[0])
theta_deg = theta_rad*(180/m.pi)
# Since, both the coordinates are positive, that means, the angle exists in the first quadrant
# Since, the angle is positive, that means, it is measured from negative x-axis in counter-clockwise direction
# So need not to add anything
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
vectors = [a, b]
plot_vector(vectors)
Vector A: [-4 -6] |A| = 7.211102550927978 Counter-Clockwise angle in degrees from positive x-axis: 236.30993247402023 Vector B: [2. 3.] |-A| = 3.605551275463989 Counter-Clockwise angle in degrees from positive x-axis: 56.309932474020215
Example 2: Given the vector a = (1, 1). If you multiply this vector by $3$, its length (magnitude) is trippled and direction remains the same.
$$ \mathbf{3a} = 3\left[ \begin{array}{c} 1 \\ 1 \end{array} \right] = \left[ \begin{array}{c} 3 \\ 3 \end{array} \right] $$# Given vector A
a = np.array([1, 1])
print ("Vector A: ", a)
# calculate magnitude using L2 norm
mag = numpy.linalg.norm(a, ord=2)
mag = (a[0]**2 + a[1]**2) **(1/2)
print("|A| = ",mag)
# calculating angle of vector A
theta_rad = math.atan(a[1]/a[0]) # acos, asin, and atan take a ratio as input and return an angle in radians
theta_deg = theta_rad*(180/math.pi) # so we need to convert it into degrees
# Since, both the coordinates are positive, that means, the angle exists in the first quadrant
# So we need not to add anything
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
# Multiplying the vector A by 3 will give a new vector
b = a * 3
print ("\nVector B: ", b)
# To calculate magnitude using L2 norm
mag = numpy.linalg.norm(b, ord=2)
mag = (b[0]**2 + b[1]**2) **(1/2)
print("|-A| = ",mag)
# calculating angle of vector B
theta_rad = m.atan(b[1]/b[0]) # acos, asin, and atan take a ratio as input and return an angle in radians
theta_deg = theta_rad*(180/m.pi) # so we need to convert it into degrees
# Since, both the coordinates are positive, that means, the angle exists in the first quadrant
# So we need not to add anything
theta_deg = theta_deg + 0
print("Counter-Clockwise angle in degrees from positive x-axis: ", theta_deg)
vectors = [a, b]
plot_vector(vectors)
Vector A: [1 1] |A| = 1.4142135623730951 Counter-Clockwise angle in degrees from positive x-axis: 45.0 Vector B: [3 3] |-A| = 4.242640687119285 Counter-Clockwise angle in degrees from positive x-axis: 45.0
Example 1: Find the unit vector of $\overrightarrow{\rm v} =\begin{bmatrix} 3 \\2 \end{bmatrix}$
$$\hat{\rm v_1} = \frac{1}{3.6}\begin{bmatrix} 3 \\2 \end{bmatrix}=\begin{bmatrix} 0.832 \\0.554 \end{bmatrix}$$v = np.array((3,2))
mag_v = numpy.linalg.norm(v, ord=2)
vhat = (v[0]/mag_v, v[1]/mag_v)
mag_vhat = numpy.linalg.norm(vhat,ord=2)
print("vector v = ", v, " magnitude =", mag_v)
print("Unit vector of v = ", vhat, " magnitude =", mag_vhat)
vectors = [v, vhat]
plot_vector(vectors)
vector v = [3 2] magnitude = 3.605551275463989 Unit vector of v = (0.8320502943378437, 0.5547001962252291) magnitude = 1.0
Note: The unit vector $\overrightarrow{\rm u}$ of the given vector $\overrightarrow{\rm v} =(3,2)$ is one unit long, and sits right on top of $\overrightarrow{\rm v}$, pointing in the same direction as $\overrightarrow{\rm v}$.
The smaller triangle formed by the unit vector $\overrightarrow{\rm u}$ is similar to the larger triangle formed by the vector $\overrightarrow{\rm v}$
Example 2: Find the unit vector of $\overrightarrow{\rm v} =\begin{bmatrix} 12 \\3 \\-4 \end{bmatrix}$
$$\hat{\rm v_1} = \frac{1}{13}\begin{bmatrix} 12 \\3\\-4 \end{bmatrix}=\begin{bmatrix} 0.92 \\0.23 \\-0.3\end{bmatrix}$$v = np.array((12, 3, -4))
mag_v = numpy.linalg.norm(v, ord=2)
vhat = (v[0]/mag_v, v[1]/mag_v, v[2]/mag_v)
mag_vhat = numpy.linalg.norm(vhat,ord=2)
print("vector v = ", v, " magnitude =", mag_v)
print("Unit vector of v = ", vhat, " magnitude =", mag_vhat)
vector v = [12 3 -4] magnitude = 13.0 Unit vector of v = (0.9230769230769231, 0.23076923076923078, -0.3076923076923077) magnitude = 1.0
Example 1: Write down the vector $\overrightarrow{\rm v} =(3,2)$, as linear combination of its unit vectors and visualize.
# create two basis vectors
i = numpy.array((1,0))
j = numpy.array((0,1))
# create a new vector, that is a scaled version of these two basis vectors
vec = 3*i + 2*j
vectors = [i, j, 3*i, 2*j, vec]
plot_vector(vectors)
Example 1: Generate hundred random vectors from linear combinations of basis vectors, $ \hat{i}=(1,0)$ and $\hat{j}=(0,1)$. The scalar multiples $m$ and $n$ range from random values $-8$ to $+8$.
from numpy.random import randint
i = numpy.array((1,0))
j = numpy.array((0,1))
vectors = []
for _ in range(10000):
m = randint(-8,8)
n = randint(-8,8)
vectors.append(m*i + n*j)
plot_vector(vectors)
pyplot.title("Hundred random vectors, created from the basis vectors");
You can imagine that we can fill up the whole plane with infinite linear combinations, eventually filling up the entire 2D plane. Indeed, the span of the basis vectors is the whole 2D space. Remember, we are not forced to use the unit vectors $\mathbf{i}$ and $\mathbf{j}$ as our basis vectors: other pairs of vectors could form a basis.
- Question: Can we use another pair of vectors as basis and generates linear combinations?
Example 2: Generate hundred random vectors from linear combinations of two vectors, $a=(-2,1)$ and $b=(1,-3)$. The scalar multiples $m$ and $n$ range from random values $-8$ to $+8$.
a = numpy.array((-2,1))
b = numpy.array((1,-3))
vectors = []
for _ in range(100):
m = randint(-8,8)
n = randint(-8,8)
vectors.append(m*a + n*b)
plot_vector(vectors)
pyplot.title("Hundred random vectors, created from the basis vectors");
Example 3: Generate fifty random vectors from linear combinations of two vectors, $ \hat{c}=(-2,-1)$ and $\hat{d}=(1,0.5)$. The scalar multiples $m$ and $n$ range from random values $-8$ to $+8$.
c = numpy.array((-2,-1))
d = numpy.array((1,0.5))
vectors = []
for _ in range(50):
m = randint(-8,8)
n = randint(-8,8)
vectors.append(m*c + n*d)
plot_vector(vectors)
pyplot.title("Fifty linear combinations of the vectors $\mathbf{c}$ and $\mathbf{d}$.");
What's going on?
Example 1: Find the dot product of the two vectors A(2, 2, -1) and B(5, -3, 2), and also calculate the angle between them. $\hspace{8 cm}u \cdot v = \sum_{i=1}^{n} u_i v_i\hspace{2 cm}$ OR $\hspace{2 cm} u.v \hspace{.3 cm}=\hspace{.3 cm} |u| \hspace{.3 cm} |v| \hspace{.3 cm} Cos \theta $
$$ \theta \hspace{.3 cm}=\hspace{.3 cm} cos^{-1} \hspace{0.1 cm} \frac{u.v}{|u| |v|} $$import math
a = np.array([2, 2, -1])
b = np.array([5, -3, 2])
print("a = ", a)
print("b = ", b)
#calculate magnitude of vector A and B
mag1 = numpy.linalg.norm(a, ord=2)
mag2 = numpy.linalg.norm(b, ord=2)
print("|a| = ", mag1)
print("|b| = ", mag2)
# Calculate Dot Product mathematically
ab = np.dot(a,b)
ab = a[0]*b[0] + a[1]*b[1]+a[2]*b[2]
print("\na.dot(b) = ", ab)
theta_rad = math.acos(ab/(mag1*mag2))
theta_deg = theta_rad*(180/math.pi)
print("Angle: ", theta_deg)
# Calculate Dot Product geometrically
print("a.b = |a| |b| cos(𝜃) = ", mag1*mag2*math.cos(theta_rad))
a = [ 2 2 -1] b = [ 5 -3 2] |a| = 3.0 |b| = 6.164414002968976 a.dot(b) = 2 Angle: 83.79145537381414 a.b = |a| |b| cos(𝜃) = 2.0000000000000027
Example 2: Find out the angle between the given two vectors using dot product:
$a = 2i + 2j + 3k$
$b = 6i + 3j + 1k$
$\hspace{8 cm}u \cdot v = \sum_{i=1}^{n} u_i v_i\hspace{2 cm}$ OR $\hspace{2 cm} u.v \hspace{.3 cm}=\hspace{.3 cm} |u| \hspace{.3 cm} |v| \hspace{.3 cm} Cos \theta $
$$ \theta \hspace{.3 cm}=\hspace{.3 cm} cos^{-1} \hspace{0.1 cm} \frac{u.v}{|u| |v|} $$a = np.array([2, 2, 3])
b = np.array([6, 3, 1])
# printing original vectors
print("a = ", a)
print("b = ", b)
#calculate magnitude of vector A and B
mag1 = numpy.linalg.norm(a, ord=2)
mag2 = numpy.linalg.norm(b, ord=2)
print("|a| = ", mag1)
print("|b| = ", mag2)
# Calculate Dot Product mathematically
ab = np.dot(a,b)
ab = a[0]*b[0] + a[1]*b[1]+a[2]*b[2]
print("\na.dot(b) = ", ab)
theta_rad = math.acos(ab/(mag1*mag2))
theta_deg = theta_rad*(180/math.pi)
print("Angle: ", theta_deg)
# Calculate Dot Product geometrically
print("a.b = |a| |b| cos(𝜃) = ", mag1*mag2*math.cos(theta_rad))
a = [2 2 3] b = [6 3 1] |a| = 4.123105625617661 |b| = 6.782329983125268 a.dot(b) = 21 Angle: 41.32652841215388 a.b = |a| |b| cos(𝜃) = 21.0
Unlike dot product, the cross product of two vectors produce a new vector and the direction of the resultant vector is given by the right-hand rule.
Mathematically:
Where $\hat{n}$ is the unit vector perpendicular to the plane containing the given two vectors, in the direction given by the right-hand rule.
The formula to compute the angle between two vectors using Cross Product is:
Example 1: Find the cross product of two vectors A(3,5,-7) and B(2,-6,4). Later prove that the resultant vector is perpendicular to both A and B.
$$ \vec{a}\times\vec{b} = \begin{pmatrix} \hat{i} & \hat{j} & \hat{k}\\ 3 & 5 & -7 \\ 2 & -6 & 4 \end{pmatrix} $$$$ = (20-42)\hat{i} - (12+14)\hat{j} +(-18-10)\hat{k} $$$$ = -22\hat{i}-26\hat{j}-28\hat{k} $$a = np.array([3, 5, -7])
b = np.array([2, -6, 4])
# printing original vectors
print("a = ", a)
print("b = ", b)
# Calculate Cross Product mathematically
ab = np.cross(a,b)
print("a x b = ", ab)
#calculate magnitude of vector a and b
mag1 = numpy.linalg.norm(a, ord=2)
mag2 = numpy.linalg.norm(b, ord=2)
mag3 = numpy.linalg.norm(ab, ord=2)
print("\n|a| = ", mag1)
print("|b| = ", mag2)
print("|axb| = ", mag2)
# Calculate the angle between two vectors
mag3 = numpy.linalg.norm(ab, ord=2)
theta_rad = math.asin(mag3/(mag1*mag2))
theta_deg = math.degrees(theta_rad) #theta_rad*(180/m.pi)
print("Angle: ", theta_deg)
a = [ 3 5 -7] b = [ 2 -6 4] a x b = [-22 -26 -28] |a| = 9.1104335791443 |b| = 7.483314773547883 |axb| = 7.483314773547883 Angle: 40.29462137837708
Orthogonal Vector:
$\hspace{3 cm}\vec{a}.\vec{b} \hspace{.3 cm}=\hspace{.3 cm} |a| \hspace{.1 cm} |b| \hspace{.3 cm} cos(90) = 0$
$\hspace{3 cm} \vec{a}$ x $\vec{b} \hspace{.3 cm}=\hspace{.3 cm} |a| \hspace{.1 cm} |b| \hspace{.3 cm} sin (90) \hspace{.3 cm} = |a| \hspace{.1 cm} |b|$
Orthonormal Vector:
Example 1: Determine if the two vectors A(6, -2, -1) and B(2, 5, 2) are perpendicular to eachother. (If the vectors are perpendicular to each other then their dot product is zero) $$ u \cdot v = \sum_{i=1}^{n} u_i v_i $$
a = np.array([6, -2, -1])
b = np.array([2, 5, 2])
# printing original vectors
print("a = ", a)
print("b = ", b)
# Calculating Dot Product of Two Vectors
ab = a.dot(b)
ab = a[0]*b[0] + a[1]*b[1]+a[2]*b[2]
print("a.b = ", ab)
mag1 = numpy.linalg.norm(a, ord=2)
mag2 = numpy.linalg.norm(b, ord=2)
theta_rad = math.acos(ab/(mag1*mag2))
theta_deg = math.degrees(theta_rad) #theta_rad*(180/m.pi)
print("Angle: ", theta_deg)
lhs = mag1*mag2*math.sin(90*(math.pi/180))
rhs = mag1*mag2
print ("|a| |b| sin(90) = ", lhs)
print ("|a| |b| = ", rhs)
a = [ 6 -2 -1] b = [2 5 2] a.b = 0 Angle: 90.0 |a| |b| sin(90) = 36.78314831549904 |a| |b| = 36.78314831549904
import numpy as np
import numpy.linalg
import math
import scipy
from matplotlib import pyplot as plt
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
m × n
for the number of rows and the number of columns respectively.
Example: Defining a 3x3 matrix using NumPy, $A =$ $\begin{bmatrix} -2 & 1 & 3\\ 1 & -3 & 5\\ -3 & 2 & 1 \end{bmatrix} $
A = np.array([[-2, 1, 3], [1, -3, 2], [-3, 2, 1]])
print("Matrix A = \n", A)
print("A.ndim: ", A.ndim)
print("A.shape: ", A.shape)
print("A.size: ", A.size)
Matrix A = [[-2 1 3] [ 1 -3 2] [-3 2 1]] A.ndim: 2 A.shape: (3, 3) A.size: 9
$\begin{bmatrix} b_1 & b_2 & b_3 & \cdots & b_n\\ \end{bmatrix} $
B = np.array([-2, 1, 3])
print("Matrix B = \n", B)
print("A.ndim: ", B.ndim)
print("A.shape: ", B.shape)
# Taking the transpose of a matrix means to interchange the rows with columns.
# The rows become columns and the columns become rows.
Bt = B.T
print("\nTranspose of Matrix B = \n", Bt)
print("A.ndim: ", Bt.ndim)
print("A.shape: ", Bt.shape)
Matrix B = [-2 1 3] A.ndim: 1 A.shape: (3,) Transpose of Matrix B = [-2 1 3] A.ndim: 1 A.shape: (3,)
$\begin{bmatrix} b_1 \\ b_2 \\ b_3 \\ \vdots \\ b_n\\ \end{bmatrix} $
B = np.array([[-2], [1], [3]])
print("Matrix B = \n", B)
print("A.ndim: ", B.ndim)
print("A.shape: ", B.shape)
# Taking the transpose of a matrix means to interchange the rows with columns.
# The rows become columns and the columns become rows.
Bt = B.T
print("\nTranspose of Matrix B = \n", Bt)
print("A.ndim: ", Bt.ndim)
print("A.shape: ", Bt.shape)
Matrix B = [[-2] [ 1] [ 3]] A.ndim: 2 A.shape: (3, 1) Transpose of Matrix B = [[-2 1 3]] A.ndim: 2 A.shape: (1, 3)
$\begin{bmatrix} 0 & 0 & 0\\ 0 & 0 & 0 \end{bmatrix} $
numpy.zeros()
method as shown below:A = np.zeros((2,3), dtype=np.int16)
print("Matrix A = \n", A)
Matrix A = [[0 0 0] [0 0 0]]
$\begin{bmatrix} 1 & 1\\ 1 & 1\\ 1 & 1 \end{bmatrix} $
numpy.ones()
method as shown below:A = np.ones((2, 3), dtype=int)
print("Matrix A = \n", A)
Matrix A = [[1 1 1] [1 1 1]]
A = np.random.randint(0, 10, size =(5, 3))
print ("Matrix A = \n", A)
Matrix A = [[2 2 3] [0 2 4] [1 1 6] [7 0 3] [1 3 4]]
n × n
matrix is said to be a square matrix of order n
.$A =$ $\begin{bmatrix} 6 & 3 & 8\\ 2 & 1 & 9\\ 8 & 2 & 7 \end{bmatrix} $
A = np.random.randint(1, 10, size =(3, 3))
np.random.seed(54)
print ("Matrix A = \n", A)
Matrix A = [[6 3 8] [2 1 9] [8 2 7]]
top-right triangle
is the same as the bottom-left
triangle.$A =$ $\begin{bmatrix} 4 & 1 & 7\\ 1 & -3 & 5\\ 7 & 5 & 2 \end{bmatrix} $
A = np.array([[4, 1, 7], [1, -3, 5], [7, 5, 2]])
print("Matrix A = \n", A)
# taking transpose of the matrix
print('\nTranspose of A = \n', A.T)
np.transpose(A)
Matrix A = [[ 4 1 7] [ 1 -3 5] [ 7 5 2]] Transpose of A = [[ 4 1 7] [ 1 -3 5] [ 7 5 2]]
array([[ 4, 1, 7], [ 1, -3, 5], [ 7, 5, 2]])
A.T
array([[ 4, 1, 7], [ 1, -3, 5], [ 7, 5, 2]])
m
rows and n
columns.Upper triangular matrix
of matrix $A$ has values only above the main diagonal, while the remaining elements are filled with zeros. $\begin{pmatrix} a_{1,1} & a_{1,2} & a_{1,3}& \cdots & a_{1,n} \\ 0 & a_{2,2} & a_{2,3}& \cdots & a_{2,n} \\ 0 & 0 & a_{3,3}&\cdots & a_{3,n} \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0& \cdots & a_{m,n} \end{pmatrix} $
Lower triangular matrix
of matrix $A$ has values only below the main diagonal, while the remaining elements are filled with zeros. $ \begin{pmatrix} a_{1,1} & 0 & 0& \cdots & 0 \\ a_{2,1} & a_{2,2} & 0& \cdots & 0 \\ a_{3,1} & a_{3,2} & a_{3,3}&\cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ a_{m,1} & a_{m,2} & a_{m,3}& \cdots & a_{m,n} \end{pmatrix} $
A = np.array([[6, 3, 8, 5], [2, 1, 9, 7], [8, 2, 4, 7], [3, 1, 5, 8]])
print("Matrix A: \n", A)
# Lower triangular matrix of matrix A
lower = np.tril(A)
print("Lower Triangular Matrix of A:\n", lower)
# Upper triangular matrix of matrix A
upper = np.triu(A)
print("Upper Triangular Matrix of A: \n", upper)
Matrix A: [[6 3 8 5] [2 1 9 7] [8 2 4 7] [3 1 5 8]] Lower Triangular Matrix of A: [[6 0 0 0] [2 1 0 0] [8 2 4 0] [3 1 5 8]] Upper Triangular Matrix of A: [[6 3 8 5] [0 1 9 7] [0 0 4 7] [0 0 0 8]]
D
There are two ways to represent a diagonal matrix:
NumPy provides the function np.diag()
that can create a diagonal matrix from an existing matrix, or transform a vector into a diagonal matrix.
A = np.random.randint(1, 10, size =(4, 4))
print("Matrix A : \n", A)
# Extract a diagonal vector from a matrix
d = np.diag(A)
print("\nDiagonal Vector = ", d)
# create diagonal matrix from diognal vector
D = np.diag(d)
print("\nDiagonal Matrix : \n", D)
Matrix A : [[7 9 8 4] [3 2 3 7] [9 2 8 8] [3 8 8 9]] Diagonal Vector = [7 2 8 9] Diagonal Matrix : [[7 0 0 0] [0 2 0 0] [0 0 8 0] [0 0 0 9]]
Properties of Identity Matrix:
n
remains unchanged when multiplied with $I_n$.trace
(sum of elements in principal diagonal) of an identity matrix is equal to identity matrix’s order.The numpy.eye(rows, cols, k, dtype)
function is used to create an identigy matrix.
k
is zero, means main diagonal, a positive value refers to an upper diagonal, and a negative value refers to a lower diagonal.dtype
is floatI3 = np.eye(4, 4)
print("3x3 Identiy Matrix:\n", I3)
3x3 Identiy Matrix: [[1. 0. 0. 0.] [0. 1. 0. 0.] [0. 0. 1. 0.] [0. 0. 0. 1.]]
I3 = np.eye(4, 4, 1,dtype=np.uint8)
print("3x3 Identiy Matrix:\n", I3)
3x3 Identiy Matrix: [[0 1 0 0] [0 0 1 0] [0 0 0 1] [0 0 0 0]]
I3 = np.eye(4, 4, -1,dtype=np.uint8)
print("3x3 Identiy Matrix:\n", I3)
3x3 Identiy Matrix: [[0 0 0 0] [1 0 0 0] [0 1 0 0] [0 0 1 0]]
$\begin{bmatrix} 2 & 0\\ 0 & 2 \end{bmatrix} \hspace{2 cm}\begin{bmatrix} -3 & 0\\ 0 & -3 \end{bmatrix} \hspace{2 cm}\begin{bmatrix} 4 & 0\\ 0 & 4 \end{bmatrix} $
$\begin{bmatrix} 4 & 0 & 0\\ 0 & 4 & 0\\ 0 & 0 & 4 \end{bmatrix} \hspace{2 cm}\begin{bmatrix} -3 & 0 & 0\\ 0 & -3 & 0\\ 0 & 0 & -3 \end{bmatrix} \hspace{2 cm}\begin{bmatrix} 2 & 0 & 0\\ 0 & 2 & 0\\ 0 & 0 & 2 \end{bmatrix} $
A = np.eye(3, 3, dtype=np.uint8)
print("Scalar Matrix A:\n", 2*A)
B = np.eye(4, 4, dtype=np.uint8)
print("Scalar Matrix B:\n", -3*B)
Scalar Matrix A: [[2 0 0] [0 2 0] [0 0 2]] Scalar Matrix B: [[-3 0 0 0] [ 0 -3 0 0] [ 0 0 -3 0] [ 0 0 0 -3]]
$\hspace{3 cm} Q_{2,2} = \begin{bmatrix} 1 & 0\\ 0 & -1 \end{bmatrix}$ $\hspace{3 cm}Q_{3,3} = \begin{bmatrix} 1 & 0 & 0\\ 0 & -1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$
linear transformations
, such as reflections
and permutations
.Q1 = np.array([ [1, 0, 0],[0, -1, 0], [0,0,1]])
Q2 = np.array([ [-1, 0, 0],[0, 1, 0], [0,0,1]])
print("Q1: \n", Q1)
print("Q2: \n", Q1)
# Product of two orthogonal matrices will also be an orthogonal matrix
print("\nQ1.Q2 = \n", np.dot(Q1,Q2))
# Transpose of the orthogonal matrix will also be an orthogonal matrix
print("\nQ1.T = \n", Q1.T)
# When an Orthogonal matrix is multiplied with its transpose, it will return an identity matrix
print("\nnp.dot(Q1, Q1.T) = \n", np.dot(Q1,(Q1.T)))
Q1: [[ 1 0 0] [ 0 -1 0] [ 0 0 1]] Q2: [[ 1 0 0] [ 0 -1 0] [ 0 0 1]] Q1.Q2 = [[-1 0 0] [ 0 -1 0] [ 0 0 1]] Q1.T = [[ 1 0 0] [ 0 -1 0] [ 0 0 1]] np.dot(Q1, Q1.T) = [[1 0 0] [0 1 0] [0 0 1]]
A = np.random.randint(-5, 6, size =(3, 3))
B = np.random.randint(-5, 6, size =(3, 3))
print("Matrix A : \n", A)
print("Matrix B : \n", B)
# adding two matrices
C = A + B
print("\nA + B = \n", C)
Matrix A : [[-2 -2 0] [ 2 3 -2] [ 0 3 -4]] Matrix B : [[-2 -3 -3] [ 3 -3 -2] [ 4 4 4]] A + B = [[-4 -5 -3] [ 5 0 -4] [ 4 7 0]]
A = np.random.randint(1, 6, size =(3, 3))
print("Matrix A : \n", A)
b = 2
C = b*A
print("\n b*A: \n", C)
Matrix A : [[2 4 5] [5 4 5] [5 5 1]] b*A: [[ 4 8 10] [10 8 10] [10 10 2]]
element-wise matrix multiplication
or the Hadamard product
.asterik *
operator directly on the two NumPy arrays.A = np.random.randint(1, 6, size =(2, 3))
B = np.random.randint(1, 6, size =(2, 3))
print("Matrix A : \n", A)
print("Matrix B : \n", B)
# Matrices Hadamard product
C = A * B
# printing product of two matrices
print("\n A * B = \n", C)
Matrix A : [[4 4 4] [1 5 3]] Matrix B : [[1 2 5] [5 4 3]] A * B = [[ 4 8 20] [ 5 20 9]]
matrix dot product
is more complicated than the previous operations and involves a rule as not all matrices can be multiplied together.columns
in the first matrix must equal the number of rows
in the second matrix". The result is a new matrix with m rows and k columns.numpy.dot()
function or using the newer @ operator
, since Python version 3.5.A = np.array([ [1, 2],[3, 4], [5, 6]])
B = np.array([ [1, 2],[3, 4]])
print("Matrix A = \n", A)
print("\nMatrix B = \n", B)
# multiply matrices using dot function
C = np.dot(A,B)
# print dot product matrix
print("\n A.dot(B) = \n", C)
# multiply matrices with @ operator
D = A @ B
# print dot product matrix
print("\nA @ B = \n", D)
Matrix A = [[1 2] [3 4] [5 6]] Matrix B = [[1 2] [3 4]] A.dot(B) = [[ 7 10] [15 22] [23 34]] A @ B = [[ 7 10] [15 22] [23 34]]
numpy.dot()
function or using the newer @ operator
, since Python version 3.5.A = np.array([ [1, 2],[3, 4], [5, 6]])
v = np.array([2, 3])
print("Matrix A = \n", A)
print("\nMatrix v = \n", v)
C = A.dot(v)
print("\n A.dot(v) = \n", C)
D = A @ v
# print dot product matrix
print("\nA @ v = \n", D)
Matrix A = [[1 2] [3 4] [5 6]] Matrix v = [2 3] A.dot(v) = [ 8 18 28] A @ v = [ 8 18 28]
A = np.array([ [0, 2, 1],[2, 0, 0]])
print("Matrix A = \n", A)
f = np.linalg.norm(A, ord=None)
print("\n Frobenius Norm =", f)
Matrix A = [[0 2 1] [2 0 0]] Frobenius Norm = 3.0
columns and rows flipped
. This is denoted by the superscript T next to the matrix AT.A = np.array([ [1, 2],
[3, 4],
[5, 6]])
print("Original Matrix A: \n", A)
# computing transpose of Matrix A using T attribute
C = A.T
C = np.transpose(A)
print("\nTranspose Matrix: \n", C)
Original Matrix A: [[1 2] [3 4] [5 6]] Transpose Matrix: [[1 3 5] [2 4 6]]
a
by the determinant of the “submatrix” formed by ignoring a's
row and column. In this case, this submatrix is the 1×1 matrix consisting of d
, and its determinant is just d
. So the first term of the determinant is ad
. Next, we proceed to the second component of the first row, which is the upper right component b
. We multiply b
by the determinant of the submatrix formed by ignoring b's
row and column, which is c
. So, the next term of the determinant is bc
. The total determinant is simply the first term ad minus the second term. :$\hspace{2 cm}A = \begin{bmatrix} a & b\\ c & d \end{bmatrix} $
$\hspace{2 cm}det(A) = |A| = ad - cb$
$\hspace{2 cm}B =$ $\begin{bmatrix} a & b & c\\ d & e & f\\ g & h & i \end{bmatrix} $
$\hspace{2 cm}det(B) = |B| = a\begin{vmatrix} e & f\\h & i \end{vmatrix} - b\begin{vmatrix} d & f\\g & i \end{vmatrix} +c\begin{vmatrix} d & e\\g & h \end{vmatrix} = a(ef-hf) - b(di-gf) + c(dh-ge)$
det()
function.A = np.array([[-3, 1],[6, -4]])
print("Matrix A: \n", A)
det1 = numpy.linalg.det(A)
print("det(A): ", det1)
B = np.array([[1, 2, 4],[2, -1, 3],[0, 5, 1]])
print("\nMatrix B: \n", B)
det2 = numpy.linalg.det(B)
print("det(B): ", det2)
Matrix A: [[-3 1] [ 6 -4]] det(A): 6.0 Matrix B: [[ 1 2 4] [ 2 -1 3] [ 0 5 1]] det(B): 19.999999999999996
A = np.array([[6, -3],[4, -2]])
print("Matrix A: \n", A)
det1 = numpy.linalg.det(A)
print("det(A): ", det1)
B = np.array([[1, 1, 1],[2, 3, 1],[0, -1, 1]])
print("\nMatrix B: \n", B)
det2 = numpy.linalg.det(B)
print("det(B): ", det2)
C = np.array([[2, 1, 2],[1, 0, 1],[4, 1, 4]])
print("\nMatrix C: \n", C)
det3 = numpy.linalg.det(C)
print("det(C): ", det3)
Matrix A: [[ 6 -3] [ 4 -2]] det(A): 0.0 Matrix B: [[ 1 1 1] [ 2 3 1] [ 0 -1 1]] det(B): 0.0 Matrix C: [[2 1 2] [1 0 1] [4 1 4]] det(C): 0.0
Note the determinant of matrix A is zero, that means matrix A is not invertable. This can be observed that the two columns of matrix A are not independent, or we can say that the two columns of matrix A are dependent. So the first column is a multiple of the second column, i.e., you can multiply the second column by -2 to get the first column. So that means the matrix A represent two parallel lines and it is impossible to solve this matrix for unknowns.
$\hspace{2 cm}AA^{-1} = A^{-1}A = I_n$
a
and d
).b
and c
).:
$\hspace{2 cm}A = \begin{bmatrix}
a & b\\
c & d
\end{bmatrix} $
$\hspace{2 cm}A = \frac{1}{det(A)}\begin{bmatrix} d & -b\\ -c & a \end{bmatrix} $
matrix of minors
for the given matrix.matrix of cofactors
.adjoint of matrix
.determinant
of given matrix.
Example: Perform step by step calculations to calculate the Inverse of $\hspace{1 cm}A_{3,3} = \begin{bmatrix} 3 & 0 & 2\\ 2 & 0 & -2\\0 & 1 & 1\end{bmatrix} $
Step 1: Find the matrix of minors
for the given matrix.
matrix of minors
, as shown below:Step 2: Turn that matrix into the matrix of cofactors
.
Step 3: Find the adjugate or adjoint of matrix
.
Step 4: Divide adjugate matrix by determinant
of given matrix.
- To rescue us from all this labour, Python gives us
numpy.linalg.inv()
method to compute the inverse of a non-singular matrix (a square matrix having non-zero dterminant)
A = np.array([ [3,0,2],[2,0,-2],[0,1,1]])
print("Matrix A: \n", A)
print("det(A): ", numpy.linalg.det(A))
AI = numpy.linalg.inv(A)
print("\nInverse of Matrix A: \n", AI)
# Verify
I = np.dot(A,AI)
print("\nnp.dot(A,AI): \n", I.astype(int))
Matrix A: [[ 3 0 2] [ 2 0 -2] [ 0 1 1]] det(A): 10.000000000000002 Inverse of Matrix A: [[ 0.2 0.2 0. ] [-0.2 0.3 1. ] [ 0.2 -0.3 -0. ]] np.dot(A,AI): [[1 0 0] [0 1 0] [0 0 1]]
trace()
method.A = np.array([ [1, 2, 3], [4, 5, 6], [7, 8, 9]])
print("Original Matrix = \n", A)
print("\nTrace value = ", np.trace(A))
Original Matrix = [[1 2 3] [4 5 6] [7 8 9]] Trace value = 15
A = np.array([ [0, 2, 1],[2, 0, 0]])
print("Matrix A = \n", A)
f1 = np.linalg.norm(A, ord=None)
print("\n Frobenius Norm =", f1)
AT = A.T
AAT = np.dot(A, AT)
tr_AAT = np.trace(AAT)
f2 = (tr_AAT)**.5
print("\nnp.trace(np.dot(A,A.T))**.5 = ", f2)
Matrix A = [[0 2 1] [2 0 0]] Frobenius Norm = 3.0 np.trace(np.dot(A,A.T))**.5 = 3.0
linearly independent
columns (or rows ) of a matrix is called its rank. (Two columns are linearly dependent if you can add+scale one column to make the other)np.linalg.matrix_rank()
method.mat = np.array([])
print("Rank of matrix having zero elements = ", np.linalg.matrix_rank(mat))
A = np.array([[1,2,3]])
print("\nA:\n", A)
print("Rank of A = ", np.linalg.matrix_rank(A))
B = np.array([[1,2,3], [3, 2, 0]])
print("\nB:\n",B)
print("Rank of B = ", np.linalg.matrix_rank(B))
C = np.array([[1,2,3], [5,6,9], [3, 2, 0]])
print("\nC:\n",C)
print("Rank of C = ", np.linalg.matrix_rank(C))
C = np.array([[1,2, 3, 4], [9, 6, 2, 4]])
print("\nC:\n", C)
print("Rank of C = ", np.linalg.matrix_rank(C))
Rank of matrix having zero elements = 0 A: [[1 2 3]] Rank of A = 1 B: [[1 2 3] [3 2 0]] Rank of B = 2 C: [[1 2 3] [5 6 9] [3 2 0]] Rank of C = 3 C: [[1 2 3 4] [9 6 2 4]] Rank of C = 2
# all columns are linearly dependent on each other
# 2nd = 3 times 1st col and 3rd col = 2 times 1st column
A = np.array([[1,3, 2], [2, 6, 4],[3, 9, 6]]) # all columns are dependent
print("A:\n", A)
print("Rank of A = ", np.linalg.matrix_rank(A))
# 1st and 2nd col are independent, but 3rd column = 1st col + 2nd col
B = np.array([[1,0, 1], [2, 2, 4],[3, 2, 5]])
print("B:\n", B)
print("Rank of B = ", np.linalg.matrix_rank(B))
A: [[1 3 2] [2 6 4] [3 9 6]] Rank of A = 1 B: [[1 0 1] [2 2 4] [3 2 5]] Rank of B = 2
import numpy as np
import numpy.linalg
import math
import scipy
from matplotlib import pyplot as plt
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
n
variables. A linear equation having two variables can be written as:
Example 1: Let us first see how we can draw a line from a linear equation of two variables using Matplotlib. $$ 2x - y = -4 $$ $$ y = 4 + 2x $$
We need to calculate at least two (x,y) pair of points that statisfies this equation, and then we can draw a line connecting those points.
%matplotlib inline
x = np.array([0, 5])
y = 4 + 2 * x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
plt.title("Line representing equation: y = 4 + 2x")
ax.set_xlim([0, 5])
ax.set_ylim([1, 14])
ax.plot(x, y)
plt.grid(True)
$$y=4+2x$$$$y=9+2x$$$$y=-2+2x$$Let us change the y-intercept, without changing the slope, and draw these three lines:
%matplotlib inline
x = np.linspace(-10, 20, 5) # start, finish, n points
y1 = -2 + 2*x
y2 = 4 + 2*x
y3 = 9 + 2*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.set_xlim([0, 6])
ax.set_ylim([-2, 16])
plt.title("Line representing three equations")
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
ax.plot(x, y3, c='purple')
plt.grid(True)
$$y=4+2x$$$$y=4+0x$$$$y=4-2x$$Let us change the slope, without changing the y-intercept, and draw these three lines:
%matplotlib inline
x = np.linspace(-10, 20, 5) # start, finish, n points
y1 = 4 + 2*x
y2 = 4 + 0*x
y3 = 4 - 2*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.set_xlim([0, 6])
ax.set_ylim([-4, 12])
plt.title("Line representing three equations")
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='purple')
ax.plot(x, y3, c='red')
plt.grid(True)
Example 2:
rent price
and independent variable time
rent
.time
.%matplotlib inline
x = np.linspace(0, 10, 5) # start, finish, n points
y = 5 + 2*x # values are in thousands
fig, ax = plt.subplots()
plt.xlabel('Time in Hours')
plt.ylabel('Rent in thousands')
ax.plot(x, y, c='purple')
plt.grid(True)
Example 3:
salary in thousands
.education in years
.%matplotlib inline
x = np.linspace(0, 20, 5) # start, finish, n points
y = 20 + 5*x
fig, ax = plt.subplots()
plt.xlabel('Education in Years')
plt.ylabel('Salary in thousands')
ax.plot(x, y, c='purple')
plt.grid(True)
Example 4:
forgetness level
.drug dosage
.%matplotlib inline
x = np.linspace(0, 20, 5) # start, finish, n points
y = 10 - 1.5*x
fig, ax = plt.subplots()
ax.set_xlim([0, 9])
ax.set_ylim([-5, 15])
plt.xlabel('Drug Dosage in ml')
plt.ylabel('Forgetness Level')
ax.plot(x, y, c='purple')
plt.grid(True)
two equations
and two variables
is shown below:three equations
and three variables
is shown below:Solution(s) to a System of Linear Equations:
lines
intersect.planes
intersect.
$\hspace{1 cm}y = 3x$ and $-5x + 2y = 2$
$\hspace{1 cm}4x - 3y = 35$ and $-3x + 8y = 10$
Example 5:
Solve the following set of two linear equations with two unknowns using substitution strategy, or elimination strategy (at your own). Use Matplotlib and see if two lines representing the two equations intersect.
$$ y = 1 + 2x$$
$$ y = 3 -\frac{1}{2}x$$
%matplotlib inline
x = np.linspace(-10, 10, 1000) # start, finish, n points
y1 = 1 + 2*x
y2 = 3 - 0.5*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.set_xlim([-2, 3])
ax.set_ylim([-1, 5])
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
plt.grid(True)
- The above graph represents the two equations as straight lines and the solution of these two set of linear equations with two unknowns is the point where both the equations are intersecting with each other, and that is $(\frac{4}{5},\frac{13}{5})$ or $(0.8, 2.6)$
Example 6:
Solve the following set of two linear equations with two unknowns using substitution strategy, or elimination strategy (at your own). Use Matplotlib and see if two lines representing the two equations intersect.
$$ y = 1 + 2x$$
$$ y = 3 + 2x$$
%matplotlib inline
x = np.linspace(-10, 10, 1000) # start, finish, n points
y1 = 1 + 2*x
y2 = 3 + 2*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.set_xlim([-2, 3])
ax.set_ylim([-1, 5])
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
plt.grid(True)
- No Solution (Parallel lines having same slope, but different y-intercept
Example 7:
Solve the following set of two linear equations with two unknowns using substitution strategy, or elimination strategy (at your own). Use Matplotlib and see if two lines representing the two equations intersect.
%matplotlib inline
x = np.linspace(-10, 10, 1000) # start, finish, n points
y1 = 1 + 2*x
y2 = 2/2 + (4/2)*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.set_xlim([-2, 3])
ax.set_ylim([-1, 5])
ax.plot(x, y1, 'r:')
ax.plot(x, y2, 'g-.')
plt.grid(True)
- Infinite Solutions (Parallel lines having same slope and same intercept)
Example 8:
%matplotlib inline
t = np.linspace(0, 100, 100) # start, finish, n points
# to manage units on graph divide both equations by 60
d1 = 2.5*t
d2 = 3*(t-5)
fig, ax = plt.subplots()
plt.xlabel('Time')
plt.ylabel('Distance')
ax.set_xlim([0, 40])
ax.set_ylim([0, 100])
ax.plot(t, d1, c='green',linestyle= '-.')
ax.plot(t, d2, c='r', linestyle=':')
plt.grid(True)
The above line graph shows that the two lines intersect at point $(30, 75)$. So we can infer that the sheriff has caught the robber after 30 minutes drive and after both has travelled a distnce of 75 Km.
- Replot the line graph, if the speed of the sheriff's car and robber's car is same
%matplotlib inline
t = np.linspace(0, 100, 100) # start, finish, n points
d1 = 0 + 2.5*t
d2 = -15 + 2.5*t
fig, ax = plt.subplots()
plt.xlabel('Time')
plt.ylabel('Distance')
ax.set_xlim([0, 40])
ax.set_ylim([0, 100])
ax.plot(t, d1, c='green',linestyle= '-.')
ax.plot(t, d2, c='r', linestyle=':')
plt.grid(True)
The two lines are parallel and never intersect. So we can infer that the sheriff will never be able to catch the theif.
- Replot the line graph, if the speed of the sheriff's car and robber's car is same as well as they leave at the same time
%matplotlib inline
t = np.linspace(0, 100, 100) # start, finish, n points
d1 = 2.5*t
d2 = 2.5*t
fig, ax = plt.subplots()
plt.xlabel('Time')
plt.ylabel('Distance')
ax.set_xlim([0, 40])
ax.set_ylim([0, 100])
#ax.plot(t, d1, c='green')
#ax.plot(t, d2, c='red')
ax.plot(t, d1, c='green',linestyle= '-.')
ax.plot(t, d2, c='r', linestyle=':')
plt.grid(True)
The two lines are having same slope and same y-intercept. So there are infinite solutions, the sheriff can catch the theif any time.
Consistant Independent System of Linear Equations:
$ \hspace{2 cm}x +y +z = 3$
$ \hspace{2 cm}x +2y +3z = 0$
$ \hspace{2 cm}x +3y +2z = 3$
Consistant Dependent System of Linear Equations:
$ \hspace{2 cm}2x + 2y +2z = -2$
$ \hspace{2 cm}2x + 3y +2z = 4$
$ \hspace{2 cm}x + y +z = -1$
Inconsistant System of Linear Equations:
- **A system of linear equations is consistent if it has one or more solutions and inconsistent if no solutions exist. Furthermore, a consistent system is said to be independent if it has exactly one solution (often referred to as the unique solution) and dependent if it has more than one solution. Two systems of equations are
equivalent if they have the same solution set.**
Problem 1: If 2 adult tickets and 1 child ticket cost 32, and if 1 adult ticket and 3 child tickets cost 36, what is the price of each?
$$2x + y = 32$$$$x + 3y = 36$$Problem 2: Jasmine wants to use milk and orange juice to increase the amount of calcium and vitamin A in her daily diet. An ounce of milk contains 37 milligrams of calcium and 57 micrograms* of vitamin A. An ounce of orange juice contains 5 milligrams of calcium and 65 micrograms of vitamin A. How many ounces of milk and orange juice should Jasmine drink each day to provide exactly 500 milligrams of calcium and 1,200 micrograms of vitamin A? $$37x + 5y = 500$$ $$57x + 65y = 1200$$
System of Two Linear Equations having two unknowns ($x_1$, and $x_2$), with four coefficients:
$$ a_{1,1}x_1 + a_{1,2}x_2 = b_1 $$
$$ a_{2,1}x_1 + a_{2,2}x_2 = b_2 $$
System of Three Linear Equations having three unknowns($x_1, x_2$, and $x_3$), with nine coefficients:
known matrix
)unknown vector
)known vector
)Matrix
$A$
, when applied on vector$x$
of unknowns, transforms it into vector$b$
of known values.
augmented matrix
form, which contains all the essential information about the linear system — everything but the names of the variables. The augmented matrix for above system of linear equations is shown below:upper triangular marix
(Row echelon form is a diagonal matrix where all entries below a leading coefficient are zero. Some textbooks also state that the leading coefficient must equal one):The matrix operations performed for triangularization are:
Step 3:Once in this form, we can say that $x_3 = b_3$ and use back substitution
to solve for $x_2$ and $x_1$.
Example 9: Given the system of three linear equations with three unknowns, solve for unknowns by using Gauss Elimination method:
Write the System of Linear Equation in Augmented Matrix Form: An augmented matrix is a rectangular array of numbers that represents a system of equations. $$𝑥 + 𝑦 + 𝑧 = 3$$ $$𝑥 + 2𝑦 +3z= 0$$ $$𝑥 + 3𝑦 + 2𝑧 = 3$$
Step 1: Augmented Matrix of system of equations $$
𝑥 + 𝑦 + 𝑧 = 3$$ $$ 2𝑦 +4z= -6$$ $$ 3𝑧 = -6$$ Perform back substitution, you get $z=-2, y=1, x=4$
Practice Problems: Use paper pencil to solve following system of linear equations:
(i) $\hspace{.5 cm}x +3𝑦 + 4𝑧 = 4$ $\hspace{2 cm}−𝑥 + 3𝑦 +2z = 2$ $\hspace{2 cm}3𝑥 + 9𝑦 + 6𝑧 = -6$
(ii) $\hspace{.5 cm}x +4𝑦 + 3𝑧 = 1$ $\hspace{2 cm}𝑥 + 2𝑦 + 9z = 1$ $\hspace{2 cm}𝑥 + 6𝑦 + 6𝑧 = 1$
(iii) $\hspace{.5 cm}x +3𝑦 + 3𝑧 = 2$ $\hspace{2 cm}3𝑥 + 9𝑦 +3z = 3$ $\hspace{2 cm}3𝑥 + 6𝑦 + 6𝑧 = 4$
Verify your answers using this online calculator https://www.handymath.com/cgi-bin/matrix3d.cgi
continuation of Gaussian elimination. However, it works for non-square matrices as well.
augmented matrix
form. An augmentation matrix for a system of three linear equations with three unknowns is of the form:diagonal matrix
(non-zero elements along main diagonal and zeros every where else)):
Example 10: Given the system of three linear equations with three unknowns, solve for unknowns by using Gauss Elimination method:
Write the System of Linear Equation in Augmented Matrix Form: An augmented matrix is a rectangular array of numbers that represents a system of equations. $$5𝑥 + 2𝑦 = 2$$ $$2𝑥 + 𝑦 -z= 0$$ $$2𝑥 + 3𝑦 - 𝑧 = 3$$
Step 1: Augmented Matrix of system of equations $$
$$
$\hspace{3 cm} R_1 = \frac{1}{5}R_1:\hspace{5 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 2 & 1 & -1 \bigm| & 0 \\ 2 & 3 & -1\bigm| & 3 \end{bmatrix}$
$\hspace{3 cm} R_2 = R_2 - 2R_1:\hspace{4 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 0 & 1/5 & -1 \bigm| & -4/5 \\ 2 & 3 & -1\bigm| & 3 \end{bmatrix}$
$\hspace{3 cm} R_3 = R_3 - 2R_1:\hspace{4 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 0 & 1/5 & -1 \bigm| & -4/5 \\ 0 & 11/5 & -1\bigm| & 11/5 \end{bmatrix}$
$\hspace{3 cm} R_2 = 5R_2$ and $R_3=5R_3:\hspace{2.6 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 0 & 1 & -5 \bigm| & -4 \\ 0 & 11 & -5\bigm| & 11 \end{bmatrix}$
$\hspace{3 cm} R_2 = R_2 - R_3:\hspace{4.55 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 0 & -10 & 0 \bigm| & -15 \\ 0 & 11 & -5\bigm| & 11 \end{bmatrix}$
$\hspace{3 cm} R_2 = \frac{-1}{10}R_2$ and $R_3=\frac{1}{11}R_3:\hspace{2 cm} \begin{bmatrix}1 & 2/5 & 0 \bigm| & 2/5 \\ 0 & 1 & 0 \bigm| & 3/2 \\ 0 & 1 & -5/11\bigm| & 1 \end{bmatrix}$
$\hspace{3 cm} R_1 = R_1 - \frac{2}{5}R_2$ and $R_3=R_3 - R_2:\hspace{1 cm} \begin{bmatrix}1 & 0 & 0 \bigm| & -1/5 \\ 0 & 1 & 0 \bigm| & 3/2 \\ 0 & 0 & -5/11\bigm| & -1/2 \end{bmatrix}$
$\hspace{3 cm} R_3 = \frac{-11}{5}R_3:\hspace{5 cm} \begin{bmatrix}1 & 0 & 0 \bigm| & -1/5 \\ 0 & 1 & 0 \bigm| & 3/2 \\ 0 & 0 & 1\bigm| & 11/10 \end{bmatrix}$
$x=-1/5, y=3/2, z=11/10$
Where,
D is the determinant of matrix $A = \begin{bmatrix} a_{1,1} & a_{1,2} & a_{1,3} \\ a_{2,1} & a_{2,2} & a_{2,3} \\a_{3,1} & a_{3,2} & a_{3,3}\end{bmatrix}$
$D_x$ is the determinant of matrix $A$ by replacing $x_1$-column by the vector of right hand side of the matrix equation: $\begin{vmatrix} b_1 & a_{1,2} & a_{1,3} \\ b_2 & a_{2,2} & a_{2,3} \\b_3 & a_{3,2} & a_{3,3}\end{vmatrix}$
$D_y$ is the determinant of matrix $A$ by replacing $x_2$-column by the vector of right hand side of the matrix equation: $\begin{vmatrix} a_{1,1} & b_1 & a_{1,3} \\ a_{2,1} & b_2 & a_{2,3} \\a_{3,1} & b_2 & a_{3,3}\end{vmatrix}$
$D_z$ is the determinant of matrix $A$ by replacing $x_3$-column by the vector of right hand side of the matrix equation: $\begin{vmatrix} a_{1,1} & a_{1,2} & b_1 \\ a_{2,1} & a_{2,2} & b_2 \\a_{3,1} & a_{3,2} & b_3\end{vmatrix}$
Example 11: Given the system of two linear equations with two unknowns, solve for unknowns by using Cramer's Rule $$𝑥 + 2𝑦 = 3$$ $$4𝑥 + 5𝑦 = 6$$
$$ \begin{bmatrix} 1 & 2 \bigm| & 3 \\ 4 & 5 \bigm| & 6 \end{bmatrix} $$$\hspace{4 cm}D = \begin{vmatrix} 1 & 2 \\ 4 & 5 \end{vmatrix} = 5 - 8 = -3$
$\hspace{4 cm}D_x = \begin{vmatrix} 3 & 2 \\ 6 & 5 \end{vmatrix} = 15 - 12 = 3$
$\hspace{4 cm}D_y = \begin{vmatrix} 1 & 3 \\ 4 & 6 \end{vmatrix} = 6 - 12 = -6$
$\hspace{2 cm}x = \frac{D_x}{D} = \frac{3}{-3} = -1$
$\hspace{2 cm}y = \frac{D_y}{D} = \frac{-6}{-3} = 2$
Example 12: Given the system of three linear equations with three unknowns, solve by using Cramer's Rule: $$𝑥 + 4𝑦 +3z = 1$$ $$𝑥 + 2𝑦 +9z= 1$$ $$𝑥 + 6𝑦 + 6𝑧 = 1$$
$$ \begin{bmatrix} 1 & 4 & 3 \bigm| & 1 \\ 1 & 2 & 9 \bigm| & 1 \\ 1 & 6 & 6\bigm| & 1 \end{bmatrix} $$$\hspace{4 cm}D = \begin{vmatrix} 1 & 4 & 3 \\ 1 & 2 & 9 \\ 1 & 6 & 6 \end{vmatrix} = 1(12-54) - 4(6-9)+3(6-2) = -18$
$\hspace{4 cm}D_x = \begin{vmatrix} 1 & 4 & 3 \\ 1 & 2 & 9 \\ 1 & 6 & 6 \end{vmatrix} = 1(12-54) - 4(6-9)+3(6-2) = -18$
$\hspace{4 cm}D_y = \begin{vmatrix} 1 & 1 & 3 \\ 1 & 1 & 9 \\ 1 & 1 & 6 \end{vmatrix} = 1(6-9) - 1(6-9)+3(1-1) = 0$
$\hspace{4 cm}D_z = \begin{vmatrix} 1 & 4 & 1 \\ 1 & 2 & 1 \\ 1 & 6 & 1 \end{vmatrix} = 1(2-6) - 4(1-1)+1(6-2) = 0$
$\hspace{2 cm}x = \frac{D_x}{D} = \frac{-18}{-18} = 1$
$\hspace{2 cm}y = \frac{D_y}{D} = \frac{0}{-18} = 0$
$\hspace{2 cm}z = \frac{D_z}{D} = \frac{0}{-18} = 0$
Note: This is possible if and only if:
>- Matrix $A$ is a square matrix
>- Matrix $A$ is non-singular, i.e., its determinant is non-zero
Example 13: Solve the set of two linear equations having two unknowns, by Matrix Inversion technique using paper pencil:
$$ 4x + 2y = 4 $$
$$ -5x - 3y = -7 $$
We can write these two equations in Matrix form as:
$$
\begin{bmatrix} 4 & 2 \\ -5 & -3 \end{bmatrix}
\begin{bmatrix} x \\ y \end{bmatrix} =
\begin{bmatrix} 4 \\ -7 \end{bmatrix}
$$
- So the solution is $x=-1$ and $y=4$
- Let us perfrom the above steps in Python
A = np.array([[4, 2], [-5, -3]])
b = np.array([4,-7])
print("Matrix A:\n", A)
print("Vector b: ", b)
Matrix A: [[ 4 2] [-5 -3]] Vector b: [ 4 -7]
Ainv = np.linalg.inv(A)
print("Inverse of A:\n", Ainv)
Inverse of A: [[ 1.5 1. ] [-2.5 -2. ]]
np.dot(Ainv, b)
array([-1., 4.])
np.linalg.solve(A, b)
array([-1., 4.])
Example 14: Solve the set of two linear equations having two unknowns, by Matrix Inversion technique using Python: $$ x - \frac{1}{2}y = 1 $$ $$ -\frac{1}{2}x + y = -1 $$
$$Ax = b$$
$$x = A^{-1}b$$
$$x = \begin{bmatrix} 4/3 & 2/3 \\2/3 & 4/3\end{bmatrix} \begin{bmatrix} 1 \\ -1 \end{bmatrix}$$
A = np.array([[1, -1/2], [-1/2, 1]])
b = np.array([1,-1])
Ainv = np.linalg.inv(A)
x = np.dot(Ainv, b)
x = np.linalg.solve(A, b)
x
array([ 0.66666667, -0.66666667])
Example 15: Solve the set of three linear equations having three unknowns, by Matrix Inversion technique using Python: $$ x + 2y + 3z = -7 $$ $$ 2x - 3y - 5z = 9 $$ $$ -6x - 8y + z = -22$$
$$
Ax = b
$$
A = np.array([[1, 2, 3], [2, -3, -5], [-6, -8, 1]])
b = np.array([-7,9, -22])
Ainv = np.linalg.inv(A)
x = np.dot(Ainv, b)
x = np.linalg.solve(A, b)
x
array([-1., 3., -4.])
Example 16: Solve the set of four linear equations having four unknowns, by Matrix Inversion technique using Python: $$ w + x + y = 75 $$ $$ w + x + z = 80 $$ $$ w + y + z = 75$$ $$ x + y + z = 70 $$
$$
Ax = b
$$
A = np.array([[1, 1, 1, 0], [1, 1, 0, 1], [1, 0, 1, 1], [0, 1, 1, 1]])
b = np.array([75,80, 75,70])
Ainv = np.linalg.inv(A)
x = np.dot(Ainv, b)
x = np.linalg.solve(A, b)
x
array([30., 25., 20., 25.])
Limitation 1: ($A$ must be non-singular)
Limitation 2: ($A$ must Square)
Possible Solutions:
Unique
SolutionInfinite
SolutionNo
Solution(i) Consistent Overdetermined System having Unique
Solution:
$$\hspace{3 cm}x+y=3$$
$$\hspace{3 cm}x-y=1$$
$$\hspace{3 cm}3x-y=5$$
%matplotlib inline
x = np.linspace(-5, 5, 5) # start, finish, n points
y1 = 3 - x
y2 = -1 + x
y3 = -5 + 3*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
ax.plot(x, y3, c='purple')
plt.grid(True)
(ii) Consistent Overdetermined System having Infinite
Solution:
$$x+y=1$$
$$2x+2y=2$$
$$ 3x+3y=3$$
%matplotlib inline
x = np.linspace(-10, 20, 5) # start, finish, n points
y1 = 1 - x
y2 = 2/2 - (2/2)*x
y3 = 3/3 - (3/3)*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
ax.plot(x, y3, c='purple')
plt.grid(True)
(iii) InConsistent Overdetermined System: $$x+y=3$$ $$x-y=1$$ $$ 2x-5y=10$$
%matplotlib inline
x = np.linspace(-5, 5, 10) # start, finish, n points
y1 = 3 - x
y2 = -1 + x
y3 = -2 + (2/5)*x
fig, ax = plt.subplots()
plt.xlabel('x')
plt.ylabel('y')
ax.plot(x, y1, c='green')
ax.plot(x, y2, c='red')
ax.plot(x, y3, c='purple')
plt.grid(True)
Consistent Underdetermined System having Infinite
Solution:
$$x+y=3$$
InConsistent Underdetermined System: $$x+2y+2z=2$$ $$x+2y+2z=4$$
Dependent Variable ($y$) | Independent Variable ($x$) |
---|---|
GPA of a student | Number of hours studied daily |
Forgetness level | Drug dosage in ml |
Salary of a person | Number of Education years |
House price | Covered area of the house |
Electricity Bill | Amount of electricity consumed |
Distance travelled | Time |
Sales | Advertising Expenditures |
m
Variables¶Dependent Variable ($y$) | Independent Variable ($x_1$) | Independent Variable ($x_2$) | Independent Variable ($x_3$) |
---|---|---|---|
GPA of a student | Number of hours studied daily | Teaching Aids | Instructor Qualification |
Forgetness level | Drug dosage in ml | Social bindings | Family Environment |
Salary of a person | Number of Education years | Managerial skills | Communication skills |
House price | Covered area of the house | Number of bed rooms | distance from office |
Electricity Bill | Amount of electricity consumed | Loadshedding | Billing slabs |
Distance travelled | Time | type of car | traffic conditions |
Sales of AC | Advertising Expenditures | Marketing | Season |
Simple Linear Regression Model
is a mathematical equation that allows us to predict the output/response/dependent variable for a given input/predictor/independent variable. $\hspace{7 cm}\epsilon_i = \hat{y_i} - y_i = 0$
Linear Least Squares
is a statistical procedure to approximate the solution of Inconsistent overdetermined systems by minimizing the sum of the offsets of points from the plotted curve.
Example 17:
GPA
and study hours
is positive or negative.study hours
has on the independent variable GPA
daily study hours
and acquired GPA
):$$(1,\hspace{.2 cm}1.4), \hspace{.5 cm}(2,\hspace{.2 cm}1.6),\hspace{.5 cm}(3,\hspace{.2 cm}2.5),\hspace{.5 cm}(4,\hspace{.2 cm}2.6), \hspace{.5 cm}(5,\hspace{.2 cm}3.5),\hspace{.5 cm}(6,\hspace{.2 cm}3.7),\hspace{.5 cm}(7,\hspace{.2 cm}4.0) $$
Step 1:
Plot the seven observations using a scatter chart:
sh = np.array([1, 2, 3, 4, 5, 6, 7.]) # study hours
gpa = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0]) # gpa
fig, ax = plt.subplots()
plt.title("Study Hours vs GPA of Students")
plt.xlabel("Study Hours")
plt.ylabel("GPA")
ax.scatter(sh,gpa)
plt.grid(True)
$$(1,1.4), \hspace{.5 cm} (2,1.6), \hspace{.5 cm}(3,2.5), \hspace{.5 cm}(4,2.6), \hspace{.5 cm}(5,3.5), \hspace{.5 cm}(6,3.7), \hspace{.5 cm}(7,4.0) $$$$y=c+mx$$
Step 2:
Write down seven linear equations, one for each observation/point:
$$y = \beta_0 + \beta_1x + \epsilon$$
Where,
Together, $\beta_0$ and $\beta_1$ are called the model coefficients. Our objective is to find the parameters $\beta_0$ and $\beta_1$ that minimize $\epsilon$ across all the available data points.
Mention the seven equations for the seven points, having two unknowns: y-intercept ($\beta_0$) and slope ($\beta_1$)
- Step 3: Create matrix equation from above seven equations: $Ax=b$
Inconsistent Overdetermined System
of seven linear equations having two unknowns in matrix form.Since the y-intercept is constant, so we have set it equal to 1
across the board
So the above equations can be written as:
Step 4:
Solve the system of linear equations using Linear Least Squares Method:
Option 1: Use Pearson Correlation coefficient, which is used to quantify the strength and direction of linear relationship between two quantitative variables
x = np.array([1, 2, 3, 4, 5, 6, 7])
y = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
N = 7
xbar = sum(x)/N
ybar = sum(y)/N
b1 = (sum((x - xbar)*(y - ybar)))/(sum((x - xbar)**2))
b0 = ybar - b1*xbar
print("Slope: ", b1)
print("Y-intercept: ", b0)
Slope: 0.4642857142857143 Y-intercept: 0.8999999999999999
Option 2: Use np.cov(x,y)
to calculate variance-covariance matrix. The diagornal values of the matrix show variances while the other values are the covariances. For two variables the covariance matrix is shown below:
$ b_1 = \frac{\sigma(x,y)}{\sigma(x,x)}\hspace{2 cm}$ and $ \hspace{2 cm}b_0 = \bar{y} - b_1\bar{x}$
# OPTION 2
x = np.array([1, 2, 3, 4, 5, 6, 7])
y = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
cov_mat = np.cov(x, y)
print("Covariance matrix: \n",cov_mat)
b1 = cov_mat[0,1]/cov_mat[0,0] #slope = cov(x,y)/var(x)
b0 = sum(y)/N - b1 * sum(x)/N
print("\nSlope: ", b1)
print("Y-intercept: ", b0)
Covariance matrix: [[4.66666667 2.16666667] [2.16666667 1.04285714]] Slope: 0.4642857142857143 Y-intercept: 0.8999999999999999
$$y = 0.9 + 0.464*x $$
Step 5:
Draw the regression line using the computed slope and y-intercept, which is the best fit line
x = np.array([1, 2, 3, 4, 5, 6, 7])
y = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
fig, ax = plt.subplots()
plt.title("Study Hours vs GPA of Students")
plt.xlabel("Study Hours")
plt.ylabel("GPA")
ax.scatter(sh,gpa);
x2 = np.linspace(0,8, 10)
b0 = 0.9
b1 = 0.464
y2 = b0 + b1*x2
ax.plot(x2,y2)
plt.grid(True)
Step 6:
Perform prediction
Step 7:
Finally calculate the least square errors to determine how well the regression line actually fits the data:
x = np.array([1, 2, 3, 4, 5, 6, 7])
y = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
yhat = b0 + b1*x
sse = sum((yhat - y)**2)
ssr = sum((yhat- np.mean(y))**2)
sst = ssr + sse
r2 = ssr/sst
r2
0.9645681098359337
This tells us that 96.4% of the variation in GPA can be explained by the number of hours studied.
Given four observations of some random experiment, use paper pencil and calculate equation for the best fit linear regression line and calculate R-squared error to to determine how well the regression line fits the dataset. Finally write Python code to verify your answers. $$(-1, \hspace{.2 cm}0),\hspace{.5 cm} (0,\hspace{.2 cm} 1),\hspace{.5 cm}(1, \hspace{.2 cm}2), \hspace{.5 cm}(2,\hspace{.2 cm} 1) $$
import scipy.stats
x = np.array([-1, 0, 1, 2])
y = np.array([0, 1, 2, 1])
result = scipy.stats.linregress(x,y)
b1 = result.slope
b0 = result.intercept
print("Slope: ", b1)
print("Y-intercept: ", b0)
yhat = b0 + b1*x
sse = sum((yhat - y)**2)
ssr = sum((yhat- np.mean(y))**2)
sst = ssr + sse
r2 = ssr/sst
print("R-Squared Error: ", r2)
fig, ax = plt.subplots()
ax.scatter(x, y);
x2 = np.linspace(-3,6, 10)
y2 = b0 + b1*x2
ax.plot(x2,y2)
plt.grid(True)
Slope: 0.4 Y-intercept: 0.8 R-Squared Error: 0.4000000000000001
forgetness level
and drug dosage
is positive or negative.drug dosage
has on the independent variable forgetness level
$$(0,\hspace{.2 cm}1.86),\hspace{.5 cm} (1,\hspace{.2 cm}1.31),\hspace{.5 cm}(2, \hspace{.2 cm}0.62),\hspace{.5 cm}(3, \hspace{.2 cm}0.33), \hspace{.5 cm}(4,\hspace{.2 cm}0.09),\hspace{.5 cm}(5,\hspace{.2 cm}-0.67),\hspace{.5 cm}(6, \hspace{.2 cm}-1.23), \hspace{.5 cm}(7,\hspace{.2 cm} -1.37) $$
x = np.array([0, 1, 2, 3, 4, 5, 6, 7.]) # Drug dosage in ml
y = np.array([1.86, 1.31, 0.62, 0.33, 0.09, -0.67, -1.23, -1.37]) # Level of forgetfullness
result = scipy.stats.linregress(x,y)
b1 = result.slope
b0 = result.intercept
print("Slope: ", b1)
print("Y-intercept: ", b0)
yhat = b0 + b1*x
sse = sum((yhat - y)**2)
ssr = sum((yhat- np.mean(y))**2)
sst = ssr + sse
r2 = ssr/sst
print("R-Squared Error: ", r2)
fig, ax = plt.subplots()
plt.title("Clinical Trial")
plt.xlabel("Drug dosage (mL)")
plt.ylabel("Forgetfulness")
ax.scatter(x, y)
x2 = np.linspace(0, 8, 10)
y2 = result.slope*x2 + result.intercept
ax.plot(x2,y2)
plt.grid(True)
Slope: -0.4692857142857143 Y-intercept: 1.76 R-Squared Error: 0.9834425543516645
n
Equations and m
Variables¶n
observations, or n
data points one for each student. So this system of of n
linear equations involving m
variables can be written as:a
, which is the minimum gpa that a student can get
Example 20:
daily study hours
, instructor teaching skills
and acquired GPA
):
- Write down seven linear equations, one for each observation/point:
GPA
based on two feature or input or predictor variables, study_hours
and instr_teaching_skills
.In this example of multiple regression, $\beta_0$, $\beta_1$, and $\beta_2$ are called the model coefficients.
The seven equations for the seven data points, makes up a system of overdetermined system as shown below:
- Create matrix equation from above seven equations: $Ax=b$
Inconsistent Overdetermined System
of seven linear equations having two unknowns in matrix form.Since the y-intercept is constant, so we have set it equal to 1
across the board
So the above equations can be written as:
- Solve the system of linear equations using Ordinary Least Squares Method:
import numpy as np
sh = np.array([1, 2, 3, 4, 5, 6, 7.]) # study hours
skills = np.array([4, 1, 3, 4, 3, 1, 2]) # instructor's teaching skills
gpa = np.array([2.4, 2.2, 2.9, 3.8, 3.0, 2.0, 3.0]) # gpa
X = np.array(list(zip(sh,skills)))
y = gpa
print("Input Variables (sh, skills):\n", X)
print("\nOutput Variable (gpa):", y)
Input Variables (sh, skills): [[1. 4.] [2. 1.] [3. 3.] [4. 4.] [5. 3.] [6. 1.] [7. 2.]] Output Variable (gpa): [2.4 2.2 2.9 3.8 3. 2. 3. ]
LinearRegression()
method of scikit-learn Libraryfrom sklearn.linear_model import LinearRegression
#initiate linear regression model
model = LinearRegression()
#fit regression model
model.fit(X, y)
#display regression coefficients and R-squared value of model
intercept = model.intercept_
sh_coef = model.coef_[0]
skill_coef = model.coef_[1]
r2 = model.score(X,y)
print('Intercept: ', intercept)
print('Coeffient of Study hour: ', sh_coef)
print('Coeffient of Instructor Teaching Skills: ', skill_coef)
print('R-Squared Errors: ', r2)
Intercept: 1.1627118644067802 Coeffient of Study hour: 0.13983050847457615 Coeffient of Instructor Teaching Skills: 0.40254237288135586 R-Squared Errors: 0.6464164339675734
$\hspace{1 cm}y \hspace{.3 cm}=\hspace{.3 cm} \hspace{.3 cm}1.16 \hspace{.3 cm}+\hspace{.3 cm} 0.1398 \hspace{.1 cm}$(study_hours) $\hspace{.3 cm} + 0.402 \hspace{.1 cm}$(instr_skills)
# Carry out the prediction with new study hours and instructor's teaching skills
gpa = intercept + sh_coef*1 + skill_coef*2
gpa = model.predict([[1, 1]])
print("GPA: ", gpa)
GPA: [1.70508475]
Example 21:
datasets/stockdata.csv
, containing year, month, interest_rate, unemployment_rate, and stock_index_prices.stock_index_price
based on two feature variables, interest_rate
, and unemployment_rate
.! cat datasets/stockdata.csv
$$\beta_0 + 2.75\beta_1 + 5.3\beta_2 = 1464$$$$\beta_0 + 2.5\beta_1 + 5.3\beta_2 = 1394$$$$\beta_0 + 2.5\beta_1 + 5.3\beta_2 = 1357$$$$\cdots \hspace{1 cm} \cdots \hspace{1 cm} \cdots $$$$\cdots \hspace{1 cm} \cdots \hspace{1 cm} \cdots $$$$\beta_0 + 1.75\beta_1 + 6.1\beta_2 = 719$$
- Write down 24 linear equations, one for each observation/point:
$$ \begin{bmatrix} 1 & 2.75 & 5.3 \\ 1 & 2.5 & 5.3 \\ 1 & 2.5 & 5.3 \\ \vdots & \vdots & \vdots \\ 1 & 1.75 & 6.1 \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \end{bmatrix} = \begin{bmatrix} y_1 \\ y_2 \\ y_3 \\ \vdots \\ y_{24} \end{bmatrix} $$$$ Ax = b$$
- Create matrix equation from above seven equations: $Ax=b$
- Solve the system of linear equations using Ordinary Least Squares Method:
dataset = np.genfromtxt("datasets/stockdata.txt", delimiter=',')
# Two input or dependent or predictor variables are interest_rate and unemployment_rate
X = dataset[:,2:4]
X
# The only independent or response variable is stock_index_price
Y = dataset[:,4]
Y
# Use sklearn to determine the y-intercept and two model coefficients
import sklearn.linear_model
regr = sklearn.linear_model.LinearRegression()
regr.fit(X, Y)
intercept = regr.intercept_ # beta0
interest_rate_coef = regr.coef_[0] # beta1
unemployment_rate_coef = regr.coef_[1] # beta2
print('Intercept: ', intercept)
print('Coeffient of Interest Rate: ', interest_rate_coef)
print('Coeffient of Unemployment Rate: ', unemployment_rate_coef)
# Carry out the prediction with new interest_rate and unemployment_rate
stock_index_price = intercept + interest_rate_coef*2.75 + unemployment_rate_coef*5.3
stock_index_price = regr.predict([[2.75, 5.3]])
stock_index_price