Section I: (Overview of Linear Algebra: Vectors)
1. Overview of Vectors - Scalars vs Vectors - Mathematical and Graphical Representation of Vectors in $\mathbb{R}^2$ - Mathematical and Graphical Representation of Vectors in $\mathbb{R}^3$ - Hands on Implementation in Python 2. Magnitude of a Vector (Vector Norms) 3. Direction of a Vector 4. Components of a Vector - Components of a Vector in $\mathbb{R}^2$ - Components of a Vector in $\mathbb{R}^3$ 5. Two Fundamental Vector Operations - Vector Addition - Multiplying a Vector with Scalar value 6. Basis and Unit Vectors 7. Linear combination and Span of Vectors 8. Vector to Vector Multiplication - Vector Dot Product - Vector Cross Product **Section II: (Overview of Linear Alagebra: Matrices)** 1. Overview of Matrices 2. Matrices and its Types - Row Vector - Column Vector - Zero Matrix - Ones Matrix - Random Integer Matrix - Square Matrix - Symmetric Matrix - Triangular Matrix - Diagonal Matrix - Identity Matrix - Scalar Matrix - Orthogonal Matrix 3. Matrix Operations - Matrix Addition - Matrix-Scalar Multiplication - Matrix Multiplication (Hadamard Product) - Matrix Multiplication (Dot Product) - Matrix-Vector Multiplication - Frobenius Norms - Transpose of a Matrix - Determinant of a Matrix - Inverse of a Matrix - Trace of a Matrix - Rank of a Matrix **Section III: (Solving System of Linear Equations)** 1. An overview of Linear Equations - What is a Linear Equation? - What is a system of Linear Equations? 2. How to solve a system of Linear Equations? - Substitution strategy - Elimination strategy - Graphing strategy - Consistent vs Inconsistent System of Linear Equations - Plotting a Linear Equation with Three Variables - Solving set of Three Linear Equations with Three variables 3. Solving System of Linear Equations using Matrix Algebra - Writing a system of Linear Equations in Matrix form - Solving system of Linear Equations using Gaussian Elimination Method - Solving system of Linear Equations using Gauss Jordan Method - Solving system of Linear Equations using Cramer's Rule - Solving system of Linear Equations using Matrix Inverse Method - Limitations of Matrix Inversion Method 4. Categories of System of Linear Equations - Standard systems - Overdetermined systems - Underdetermined systems 5. Solving Inconsistent Overdetermined System of Linear Equations using Least Squares Method - Modeling Linear Equations in Machine Learning with `2` variables - Modeling Linear Equations in Machine Learning with `m` variables - Simple Linear Regression using Least Squares Method - Multiple Linear Regression using Ordinary Least Squares (OLS) Method **Section IV: (Matrix as a Linear Transformation)** 1. What do you mean by applying a Matrix to a Vector? 2. Algebra Behind Linear Transformations in 2-D Space - Applying an Identity Matrix - Applying a Scaling Matrix - Applying a Reflection Matrix - Applying a Shear Matrix - Applying a Rotation Matrix - Apply Multiple Transformations Simultaneously 3. Scaling, Translation and Rotation in Three Dimensional Space 4. Hierarchy of Transformations - Class-I: Isometries - Class-II: Similarity - Class-III: Affine - Class-IV: ProjectiveSection V: (Eigen Decomposition and SVD)
Section VI: (Applications of SVD & Eigendecomposition)
import numpy as np
import pandas as pd
import math
import scipy
from matplotlib import pyplot as plt
from matplotlib import image
import sklearn
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
Some of the codes of this notebook are adapted from:
$\hspace{2 cm}i' = Ai = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ -1 \end{bmatrix}$
$\hspace{2 cm}j' = Aj = \begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix}\begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$
Example:
A = numpy.array([[1,1], [-1,1]])
print(A)
plot_linear_transformation(A)
[[ 1 1] [-1 1]]
$\hspace{7 cm}\overrightarrow{v_2} = A\overrightarrow{v_1}$
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} a_1 & a_2 \\ a_3 & a_4 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} \hspace{3 cm} \begin{bmatrix} x_2 \\ y_2 \\ 1\end{bmatrix} = \begin{bmatrix} a_1 & a_2 & 0 \\ a_3 & a_4 & 0 \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ 1\end{bmatrix} $
$\hspace{6 cm}x_2 = a_1x_1 +a_2y_1$
$\hspace{6 cm}y_2 = a_3x_1 +a_4y_1$
The matrix $A$ represents the linear transformation that takes vector $\overrightarrow{v_1} \hspace{.2 cm}(x_1, y_1)$ and transforms it into $\overrightarrow{v_2}\hspace{.2 cm} (x_2, y_2)$.
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{2 cm} x_2 = x_1 + 0 = x_1$
$\hspace{2 cm}y_2 = 0 + y_1 = y_1$
Example: Apply identity matrix, $A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}_{2x2}$ on a vector $\overrightarrow{v_1} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}$. Calculate and graphically show the vector before and after the transformation.
%matplotlib inline
v1 = np.array([3, 1])
A = np.array([[1, 0], [0, 1]])
v2 = np.dot(A, v1)
print("v1 = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation")
plot_vector([v2])
pyplot.title("After Transformation");
v1 = [3 1] v2 = [3 1]
# Apply identiy matrix on all points in a 2-D plane
plot_linear_transformation(A)
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} S_x & 0 \\ 0 & S_y \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = x_1S_x$
$\hspace{3 cm}y_2 = y_1S_y$
Example: Apply scaling matrix, $A = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix}_{2x2}$ on a vector $\overrightarrow{v_1} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}$. Calculate and graphically show the vector before and after the transformation.
v1 = np.array([3, 1])
A = np.array([[2, 0], [0, 4]])
v2 = np.dot(A, v1)
print("v1 = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation")
plot_vector([v2])
pyplot.title("After Transformation");
v1 = [3 1] v2 = [6 4]
# Apply scaling matrix 'A' on all points in a 2-D plane
plot_linear_transformation(A)
$$A^{-1}v_2 = \begin{bmatrix} 0.5 & 0 \\ 0 & 0.25 \end{bmatrix}\begin{bmatrix} 6 \\ 4 \end{bmatrix}= \begin{bmatrix} 3 \\ 1 \end{bmatrix} = \overrightarrow{v_1}$$Applying inverse of scaling matrix, $A = \begin{bmatrix} 2 & 0 \\ 0 & 4 \end{bmatrix}$ on the transformed vector $\overrightarrow{v_2} = \begin{bmatrix} 6 \\ 4 \end{bmatrix}$, will undo the effect of transformation and give you $\overrightarrow{v_1}$.
A_inv = numpy.linalg.inv(A)
A_inv
array([[0.5 , 0. ], [0. , 0.25]])
# Apply matrix A and then inverse of matrix A on all points in a 2-D plane
v2 = np.array([6, 4])
A_inv = np.linalg.inv(A)
v1 = np.dot(A_inv, v2)
print("v2 = ", v2)
print("v1 = ", v1)
v2 = [6 4] v1 = [3. 1.]
# Apply matrix A and then inverse of matrix A on all points in a 2-D plane
plot_linear_transformations(A, numpy.linalg.inv(A))
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = x_1$
$\hspace{3 cm}y_2 = -y_1$
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} -1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = -x_1$
$\hspace{3 cm}y_2 = y_1$
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} -1 & 0 \\ 0 & -1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = -x_1$
$\hspace{3 cm}y_2 = -y_1$
Example: Apply reflection matrix, $A = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}_{2x2}$ on a vector $\overrightarrow{v_1} = \begin{bmatrix} 3 \\ 2 \end{bmatrix}$. Calculate and graphically show the vector before and after the transformation.
v1 = np.array([3, 2])
A = np.array([[1, 0], [0, -1]])
v2 = np.dot(A, v1)
print("v = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation");
plot_vector([v2])
pyplot.title("After Transformation");
v = [3 2] v2 = [ 3 -2]
# Apply reflection matrix on all points in a 2-D plane
A = np.array([[1, 0], [0, -1]])
print(A)
plot_linear_transformation(A)
[[ 1 0] [ 0 -1]]
# Apply matrix A and then inverse of matrix A on all points in a 2-D plane
plot_linear_transformations(A, numpy.linalg.inv(A))
$\hspace{2 cm}\begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} 1 & s_x \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = x_1 + s_x y_1$
$\hspace{3 cm}y_2 = y_1$
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ s_y & 1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = x_1$
$\hspace{3 cm}y_2 = S_y x_1 + y_1$
Example: Apply x-shear matrix, $A = \begin{bmatrix} 1 & 3 \\ 0 & 1 \end{bmatrix}_{2x2}$ on a vector $\overrightarrow{v_1} = \begin{bmatrix} 2 \\ 2 \end{bmatrix}$. Calculate and graphically show the vector before and after the transformation.
v1 = np.array([2, 2])
A = np.array([[1, 3], [0, 1]])
v2 = np.dot(A, v1)
print("v1 = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation");
plot_vector([v2])
pyplot.title("After Transformation");
v1 = [2 2] v2 = [8 2]
Since this is x-shear, so it maintains the y-coordinate but changes the x-coordinate, which causes the horizontal lines to tilt up (Give a negative value to check the horizontal lines tilting down)
# Apply x-shear matrix on all points in a 2-D plane
A = np.array([[1,3], [3, 1]])
print(A)
plot_linear_transformation(A)
[[1 3] [3 1]]
# Apply matrix A and then inverse of matrix A on all points in a 2-D plane
A = np.array([[1, 3], [0, 1]])
plot_linear_transformations(A, numpy.linalg.inv(A))
$\hspace{2 cm} R = \begin{bmatrix} cos\theta & -sin\theta \\ sin\theta & cos\theta \end{bmatrix} $
$\hspace{2 cm} R^T = \begin{bmatrix} cos(-\theta) & -sin(-\theta) \\ sin(-\theta) & cos(-\theta) \end{bmatrix} = \begin{bmatrix} cos\theta & sin\theta \\ -sin\theta & cos\theta \end{bmatrix} $
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \end{bmatrix} = \begin{bmatrix} cos\theta & -sin\theta \\ sin\theta & cos\theta \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \end{bmatrix} $
$\hspace{3 cm}x_2 = x_1cos\theta - y_1sin\theta$
$\hspace{3 cm}y_2 = x_1sin\theta + y_1cos\theta$
Example: If $\overrightarrow{v_1} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$ is rotated in the counter clockwise direction by 90 degrees, what are the coordinate values of new vector $\overrightarrow{v_2}$?
v1 = np.array([2, 3])
theta = 90*math.pi/180 # the input to sin() and cos() should be in radians
A = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]
])
v2 = np.dot(A, v1)
print("Rotation matrix A:\n", A)
print("v1 = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation");
plot_vector([v2])
pyplot.title("After Transformation");
Rotation matrix A: [[ 6.123234e-17 -1.000000e+00] [ 1.000000e+00 6.123234e-17]] v1 = [2 3] v2 = [-3. 2.]
# Apply rotation matrix (90 degrees), in counter clockwise direction on all points in a 2-D plane
theta = 90*math.pi/180 # the input to sin() and cos() should be in radians
A = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]
])
print(A)
plot_linear_transformation(A)
[[ 6.123234e-17 -1.000000e+00] [ 1.000000e+00 6.123234e-17]]
Example: If $\overrightarrow{v_1} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$ is rotated in the counter clockwise direction by 45 degrees, what are the coordinate values of new vector $\overrightarrow{v_2}$?
v1 = np.array([2, 3])
theta = 45*math.pi/180 # the input to sin() and cos() should be in radians
A = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]
])
v2 = np.dot(A, v1)
print("Rotation matrix A:\n", A)
print("v1 = ", v1)
print("v2 = ", v2)
plot_vector([v1])
pyplot.title("Before Transformation");
plot_vector([v2])
pyplot.title("After Transformation");
Rotation matrix A: [[ 0.70710678 -0.70710678] [ 0.70710678 0.70710678]] v1 = [2 3] v2 = [-0.70710678 3.53553391]
# Apply rotation matrix (45 degrees), in counter clockwise direction on all points in a 2-D plane
theta = 45*math.pi/180 # the input to sin() and cos() should be in radians
A = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]
])
plot_linear_transformations(A, numpy.linalg.inv(A))
$\hspace{10 cm}x_2 = x_1 + t_x$
$\hspace{10 cm}y_2 = x_1 + t_y$
Example: Apply two transformations: Scaling and then rotation
S = np.array([[2, 0], [0, 4]])
theta = 90*math.pi/180
R = np.array([[math.cos(theta), -math.sin(theta)],
[math.sin(theta), math.cos(theta)]
])
print("S: \n",S)
print("R: \n",R)
plot_linear_transformations(S,R)
S: [[2 0] [0 4]] R: [[ 6.123234e-17 -1.000000e+00] [ 1.000000e+00 6.123234e-17]]
plot_linear_transformations(R,S)
$\hspace{3 cm} A\hspace{.5 cm}=\hspace{.5 cm} \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \hspace{.5 cm}=\hspace{.5 cm} \begin{bmatrix} a_{11} & a_{12} & a_{13} & 0\\ a_{21} & a_{22} & a_{23} & 0 \\ a_{31} & a_{32} & a_{33} & 0 \\ 0&0&0&1 \end{bmatrix} $
$\hspace{4 cm}\overrightarrow{v_2} \hspace{.5 cm}=\hspace{.5 cm} A\overrightarrow{v_1}$
$\hspace{3 cm} \begin{bmatrix} x_2 \\ y_2 \\z_2\end{bmatrix} \hspace{.5 cm}= \hspace{.5 cm} \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ z_1\end{bmatrix} \hspace{3 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\ 1\end{bmatrix} \hspace{.5 cm}= \hspace{.5 cm} \begin{bmatrix} a_{11} & a_{12} & a_{13} & 0\\ a_{21} & a_{22} & a_{23} & 0 \\ a_{31} & a_{32} & a_{33} & 0 \\ 0&0&0&1 \end{bmatrix} \begin{bmatrix} x_1 \\ y_1 \\ z_1\\ 1\end{bmatrix} $
$\hspace{2 cm}x_2 = a_{11}x_1 + a_{12}y_1 + a_{13}z_1$
$\hspace{2 cm}y_2 = a_{21}x_1 + a_{22}y_1 + a_{23}z_1$
$\hspace{2 cm}z_2 = a_{31}x_1 + a_{32}y_1 + a_{33}z_1$
The matrix $A$ represents the linear transformation that takes vector $\overrightarrow{v_1} \hspace{.2 cm}(x_1, y_1,z_1)$ and transforms it into $\overrightarrow{v_2}\hspace{.2 cm} (x_2, y_2, z_2)$.
3-D Scaling:
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\1\end{bmatrix} =
\begin{bmatrix} S_x & 0 & 0 & 0\\ 0 & S_y & 0 & 0 \\ 0 & 0 & S_z & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}
\begin{bmatrix} x_1 \\ y_1 \\ z_1 \\1\end{bmatrix}
$
$\hspace{2 cm}x_2 = a_{11}x_1 + a_{12}y_1 + a_{13}z_1$
$\hspace{2 cm}y_2 = a_{21}x_1 + a_{22}y_1 + a_{23}z_1$
$\hspace{2 cm}z_2 = a_{31}x_1 + a_{32}y_1 + a_{33}z_1$
3-D Translation:
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\1\end{bmatrix} =
\begin{bmatrix} 1 & 0 & 0 & t_x\\ 0 & 1 & 0 & t_y \\ 0 & 0 & 1 & t_z \\ 0 & 0 & 0 & 1 \end{bmatrix}
\begin{bmatrix} x_1 \\ y_1 \\ z_1 \\1\end{bmatrix}
$
$\hspace{2 cm}x_2 = x_1 + t_x$
$\hspace{2 cm}y_2 = y_1 + t_y$
$\hspace{2 cm}z_2 = z_1 + t_z$
3-D Counter Clockwise Rotation about Z-axis by an angle $\theta$:
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\1\end{bmatrix} =
\begin{bmatrix} cos\theta & -sin\theta & 0 & 0\\ sin\theta & cos\theta & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}
\begin{bmatrix} x_1 \\ y_1 \\ z_1 \\1\end{bmatrix}
$
$\hspace{2 cm}x_2 = x_1cos\theta - y_1sin\theta$
$\hspace{2 cm}y_2 = x_1sin\theta + y_1cos\theta$
$\hspace{2 cm}z_2 = z_1$
3-D Counter Clockwise Rotation about X-axis by an angle $\gamma$:
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\1\end{bmatrix} =
\begin{bmatrix} 1 & 0 &0 & 0 \\ 0 & cos\gamma & -sin\gamma & 0 \\ 0 & sin\gamma & cos\gamma & 0 \\ 0 & 0 & 0& 1 \end{bmatrix}
\begin{bmatrix} x_1 \\ y_1 \\ z_1 \\1\end{bmatrix}
$
$\hspace{2 cm}x_2 = x_1$
$\hspace{2 cm}y_2 = y_1cos\gamma - z_1sin\gamma$
$\hspace{2 cm}z_2 = y_1sin\gamma + z_1cos\gamma$
3-D Counter Clockwise Rotation about Y-axis by an angle $\beta$:
$\hspace{2 cm} \begin{bmatrix} x_2 \\ y_2 \\ z_2 \\1\end{bmatrix} =
\begin{bmatrix} cos\beta & 0 & sin\beta & 0 \\ 0 & 1 & 0 &1 \\ -sin\beta & 0 & cos\beta & 0 \\ 0&0&0&1\end{bmatrix}
\begin{bmatrix} x_1 \\ y_1 \\ z_1 \\1\end{bmatrix}
$
$\hspace{2 cm}x_2 = x_1cos\beta + z_1sin\beta$
$\hspace{2 cm}y_2 = y_1$
$\hspace{2 cm}z_2 = -x_1sin\beta + z_1cos\beta$
import numpy as np
import pandas as pd
import math
import scipy
from matplotlib import pyplot as plt
from matplotlib import image
import sklearn
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
We all have done factorization or prime factorization of integers:
Similar to factorizing an integer into its prime factors, Matrix decomposition is the process of reducing a matrix to its constituent parts in order to make certain subsequent matrix calculations simpler.
There are different types of matrix decompositions like:
Eigendecomposition
Singular Value Decomposition
LU Decomposition
LR Decomposition
QR Decomposition
Cholesky Decomposition
So the transformation matrix $A = \begin{bmatrix}1 & 2 \\ 3 & -4 \end{bmatrix}$ has two eigen vectors:
We can normalize the eigenvectors by dividing the elements by $L^2$ norm.:
These two vectors are called eigenvectors, each having an associated eigenvalue, which is the factor by which it is stretched or squished.
Example 1-A: Use paper pencil to determine the Eigenvalues and Eigenvectors of a transformation matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}_{2x2}$, and later verify that the eigenvalues and eigenvectors satisfies the equation $A\overrightarrow{\rm v} = \lambda \overrightarrow{\rm v}$
$$ (A-\lambda I)\hspace{.5 cm}=\hspace{.5 cm} \begin{bmatrix} 1 & 2 \\ 3 & -4 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm} \begin{bmatrix} 1 & 2 \\ 3 & -4 \end{bmatrix} - \begin{bmatrix} \lambda & 0 \\ 0 & \lambda \end{bmatrix} \hspace{.5 cm}=\hspace{.5 cm} \begin{bmatrix} 1-\lambda & 2 \\ 3 & -4-\lambda \end{bmatrix} $$
Step 1:
Calculate eigenvalues by solving $det(A-\lambda I)=0$
$\hspace{2 cm} det(A-\lambda I)\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{vmatrix} 1-\lambda & 2 \\ 3 & -4-\lambda \end{vmatrix} \hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} (1-\lambda)(-4 - \lambda)- (2)(3) \hspace{.5 cm}=\hspace{.5 cm}0$
$\hspace{2 cm} -4 - \lambda + 4\lambda + \lambda^2 -6 \hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} \lambda^2 + 3\lambda - 10 \hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} \lambda^2 + 5\lambda - 2\lambda - 10 \hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} \lambda(\lambda + 5) - 2(\lambda + 5) \hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} (\lambda + 5) (\lambda - 2) \hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} \lambda_1 = 2,\hspace{2 cm} \lambda_2 = -5$
Step 2:
Calculate $1^{st}$ eigenvector by solving $(A-\lambda I)\overrightarrow{\rm v} = 0$ for eigenvalue $\lambda_1 = 2$
$\hspace{2 cm} (A-\lambda I)\overrightarrow{\rm v}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} 1-\lambda_1 & 2 \\ 3 & -4-\lambda_1 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} 1-2 & 2 \\ 3 & -4-2 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} -1 & 2 \\ 3 & -6 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} -v_1 +2v_2\hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} 3v_1 -6v_2\hspace{.5 cm}=\hspace{.5 cm} 0$
Both the equations are same, so you can use any eigenvector such that $v_1 = 2v_2$.
So there is a whole family of eigen vectors that satisfies this, and some are: $\begin{bmatrix} 2 \\ 1 \end{bmatrix}, \begin{bmatrix} 4 \\ 2 \end{bmatrix},\begin{bmatrix} 6 \\ 3 \end{bmatrix}\cdots$
You can verify by calculating $A\overrightarrow{\rm v} = \lambda \overrightarrow{\rm v}$, by plugging $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}$, $\lambda=2$, and using $\overrightarrow{\rm v}$ as any of the family of above eigenvectors.
Step 3:
Calculate $2^{nd}$ eigenvector by solving $(A-\lambda I)\overrightarrow{\rm v} = 0$ for eigenvalue $\lambda_1 = -5$
$\hspace{2 cm} (A-\lambda I)\overrightarrow{\rm v}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} 1-\lambda_2 & 2 \\ 3 & -4-\lambda_2 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} 1+5 & 2 \\ 3 & -4+5 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} \begin{bmatrix} 6 & 2 \\ 3 & 1 \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \end{bmatrix}\hspace{.5 cm}=\hspace{.5 cm}\begin{bmatrix} 0 \\ 0 \end{bmatrix}$
$\hspace{2 cm} 6v_1 +2v_2\hspace{.5 cm}=\hspace{.5 cm} 0$
$\hspace{2 cm} 3v_1 +v_2\hspace{.5 cm}=\hspace{.5 cm} 0$
Both the equations are same, so you can use any eigenvector such that $v_2 = -3v_1$.
So there is a whole family of eigen vectors that satisfies this, and some are: $\begin{bmatrix} 1 \\ -3 \end{bmatrix}, \begin{bmatrix} 2 \\ -6 \end{bmatrix},\begin{bmatrix} 3 \\ -9 \end{bmatrix}\cdots$
You can verify by calculating $A\overrightarrow{\rm v} = \lambda \overrightarrow{\rm v}$, by plugging $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}$, $\lambda=-5$, and using $\overrightarrow{\rm v}$ as any of the family of above eigenvectors.
Step 4:
Normalize the eigenvectors by dividing the elements by $L^2$ norm
- $\begin{bmatrix}\frac{2}{\sqrt5} \\ \frac{1}{\sqrt5} \end{bmatrix}$, with eigenvalue of 2.<br><br>
- $\begin{bmatrix}\frac{-1}{\sqrt10} \\ \frac{3}{\sqrt10} \end{bmatrix}$, with eigenvalue of -5.
Example 1-B: Write Python code to determine the Eigenvalues and Eigenvectors of a transformation matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}_{2x2}$
A = np.array([[1, 2], [3, -4]])
lambdas, V = np.linalg.eig(A)
print("Columns of matrix are eigenvectors:\n", V)
print("\nEigenvalues corresponding to each Eigenvector: ", lambdas)
Columns of matrix are eigenvectors: [[ 0.89442719 -0.31622777] [ 0.4472136 0.9486833 ]] Eigenvalues corresponding to each Eigenvector: [ 2. -5.]
Example 1-C: Apply the transformation matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}$ on any of the family of $1^{st}$ eigenvector you have computed above and plot the vector before and after transformation. $$\begin{bmatrix} 0.894 \\ 0.447 \end{bmatrix}, \begin{bmatrix} 2 \\ 1 \end{bmatrix}, \begin{bmatrix} 4 \\ 2 \end{bmatrix},\begin{bmatrix} 6 \\ 3 \end{bmatrix}\cdots$$
first_ev = V[:,0] #slice the first column from matrix of eigenvectors
first_ev = np.array([2,1])
first_ev = np.array([4,2])
new = np.dot(A, first_ev)
print("First Eigenvector = ", first_ev)
print("After Transformation = ", new)
vectors = [first_ev, new]
plot_vector(vectors)
pyplot.title("$\lambda = 2$");
First Eigenvector = [4 2] After Transformation = [8 4]
Note that once we apply the transformation matrix to eigenvector $v$, its direction remains the same. Moreover, it's length has increased two times, and that corresponds exactly to its eigenvalue and that is 2.
Example 1-D: Apply the transformation matrix $A = \begin{bmatrix} 1 & 2 \\ 3 & -4\end{bmatrix}$ on any of the family of $2^{nd}$ eigenvector you have computed above and plot the vector before and after transformation. $$\begin{bmatrix} 0.316 \\ -0.948 \end{bmatrix}, \begin{bmatrix} 1 \\ -3 \end{bmatrix}, \begin{bmatrix} 2 \\ -6 \end{bmatrix},\begin{bmatrix} 3 \\ -9 \end{bmatrix}\cdots$$
second_ev = V[:,1] #slice the second column from matrix of eigenvectors
second_ev = np.array([2, -6]) # second_ev = np.array([-2, 6])
second_ev = np.array([3, -9]) # second_ev = np.array([-3, 9])
second_ev = np.array([1, -3]) # second_ev = np.array([-1, 3])
second_ev = V[:,1]
new = np.dot(A, second_ev)
print("Second Eigenvector = ", second_ev)
print("After Transformation = ", new)
vectors = [second_ev, new]
plot_vector(vectors)
pyplot.title("$\lambda = -5$");
Second Eigenvector = [-0.31622777 0.9486833 ] After Transformation = [ 1.58113883 -4.74341649]
Note that once we apply the transformation matrix to vector $v$, its direction becomes exactly opposite. Moreover, it's length has increased five times, and that corresponds exactly to its eigenvalue and that is -5.
Example 1-E: Confirm that $Av = \lambda v$ for the first eigenvector
first_ev = V[:,0] # slice the first column from matrix of eigenvectors
lamb1 = lambdas[0] # get the first scalar value from the vector of eigenvalues
#lhs = np.dot(A, first_ev)
lhs = A @ first_ev
rhs = lamb1 * first_ev
print("First Eigenvector: ", first_ev)
print("First Eigenvalue: ", lamb1)
print("Av = ", lhs)
print("𝜆𝑣 = ", rhs)
First Eigenvector: [0.89442719 0.4472136 ] First Eigenvalue: 2.0 Av = [1.78885438 0.89442719] 𝜆𝑣 = [1.78885438 0.89442719]
Example 1-F: Confirm that $Av = \lambda v$ for the second eigenvector
second_ev = V[:,1] # slice the first column from matrix of eigenvectors
lamb2 = lambdas[1] # get the first scalar value from the vector of eigenvalues
#lhs = np.dot(A, second_ev)
lhs = A @ second_ev
rhs = lamb2 * second_ev
print("Second Eigenvector: ", second_ev)
print("Second Eigenvalue: ", lamb2)
print("Av = ", lhs)
print("𝜆𝑣 = ", rhs)
Second Eigenvector: [-0.31622777 0.9486833 ] Second Eigenvalue: -5.0 Av = [ 1.58113883 -4.74341649] 𝜆𝑣 = [ 1.58113883 -4.74341649]
A = np.array([[1, 2, 4], [2, -1, 3], [0, 5, 1]])
det = np.linalg.det(A)
print("Matrix A :\n", A)
print("\ndet(A) = ", det)
lambdas, V = np.linalg.eig(A)
print("\nEigenvalues of matrix A:", lambdas)
rhs = np.product(lambdas)
print("\nProduct of eigenvalues of matrix A: ", rhs)
Matrix A : [[ 1 2 4] [ 2 -1 3] [ 0 5 1]] det(A) = 19.999999999999996 Eigenvalues of matrix A: [-3.25599251 -1.13863631 5.39462882] Product of eigenvalues of matrix A: 19.99999999999999
A = np.array([[1, 2], [3, -4]])
det = np.linalg.det(A)
print("Matrix A :\n", A)
lambdas, V = np.linalg.eig(A)
print("\nEigenvalues of matrix A:", lambdas)
print("\nSum of eigenvalues of matrix A: ", np.sum(lambdas))
print("\nTrace if matrix A: ", np.trace(A))
Matrix A : [[ 1 2] [ 3 -4]] Eigenvalues of matrix A: [ 2. -5.] Sum of eigenvalues of matrix A: -3.0 Trace if matrix A: -3
Example: Find the Eigenvalues and Eigenvectors of transformation matrix $A = \begin{bmatrix} 0 & -1 \\ 1 & 0\end{bmatrix}$. Verify that this matrix has no eigenvalues in the real domain, however, in the complex field has two eigenvalues: ±𝑖
A = np.array([[0, -1], [1, 0]])
lambdas, V = np.linalg.eig(A)
print("Eigenvectors:\n", V)
print("\nEigenvalues: ", lambdas)
print("Product of eigenvalues = ", np.product(lambdas), " and det(A)= ", np.linalg.det(A))
print("\nSum of eigenvalues = ", np.sum(lambdas), " and trace(A) = ", np.trace(A))
Eigenvectors: [[0.70710678+0.j 0.70710678-0.j ] [0. -0.70710678j 0. +0.70710678j]] Eigenvalues: [0.+1.j 0.-1.j] Product of eigenvalues = (1+0j) and det(A)= 1.0 Sum of eigenvalues = 0j and trace(A) = 0
Example : Find the Eigenvalues and Eigenvectors of a diagonal transformation matrix $A = \begin{bmatrix} 2 & 0 \\ 0 & 2\end{bmatrix}$. Verify that the eigenvectors of a diagonal transformation matrix are the basis vectors and the values along the diagonal are their eigenvalues.
A = np.array([[2, 0], [0, 2]])
lambdas, V = np.linalg.eig(A)
print("Columns of matrix are eigenvectors:\n", V)
print("\nEigenvalues corresponding to each Eigenvector: ", lambdas)
Columns of matrix are eigenvectors: [[1. 0.] [0. 1.]] Eigenvalues corresponding to each Eigenvector: [2. 2.]
So every vector in the plane is an eigenvector with an eigenvalue of 2
Example: Find the Eigenvalues and Eigenvectors of a random transformation matrix $A = \begin{bmatrix} -1 & 0 \\ 0 & 2\end{bmatrix}$. Verify that the eigenvectors of a diagonal transformation matrix are the basis vectors and the values along the diagonal are their eigenvalues.
A = np.array([[-1, 0], [0, 2]])
lambdas, V = np.linalg.eig(A)
print("Columns of matrix are eigenvectors:\n", V)
print("\nEigenvalues corresponding to each Eigenvector: ", lambdas)
Columns of matrix are eigenvectors: [[1. 0.] [0. 1.]] Eigenvalues corresponding to each Eigenvector: [-1. 2.]
Example: Find the Eigenvalues and Eigenvectors of a random transformation matrix $A = \begin{bmatrix} -5 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & -4\end{bmatrix}$. Verify that the eigenvectors of a diagonal transformation matrix are the basis vectors and the values along the diagonal are their eigenvalues.
A = np.array([[-5, 0, 0], [0, 2, 0],[0, 0, -4]])
lambdas, V = np.linalg.eig(A)
print("Columns of matrix are eigenvectors:\n", V)
print("\nEigenvalues corresponding to each Eigenvector: ", lambdas)
Columns of matrix are eigenvectors: [[1. 0. 0.] [0. 1. 0.] [0. 0. 1.]] Eigenvalues corresponding to each Eigenvector: [-5. 2. -4.]
square matrix
$A$, we first need to calculate its eigenvalues
and eigenvectors
. Then we write the matrix as a product of three matrices as shown below:eigenvectors
of matrix $A$,eigenvalues
of matrix $A$
Example: Given the matrix $A = \begin{bmatrix} 4 & 2\\ -5 & -3\end{bmatrix}$, decompose it using eigendecomposition, and later prove that the equation $A = V \Lambda V^{-1}$ holds true.
A = np.array([[4, 2], [-5, -3]])
lambdas, V = np.linalg.eig(A) # Calculate eigenvalues and eigenvectors of matrix A
Lambda = np.diag(lambdas) # Calculate the matrix Λ, which is the diagonal matrix of eigenvalues
Vinv = np.linalg.inv(V) # Calculate inverse of matrix V
print("Matrix A: \n", A)
print("\nMatrix 𝑉:\n", V, end=' ')
print("\nMatrix Λ:\n", Lambda, end=' ')
print("\nMatrix 𝑉inv:\n", Vinv)
# Now we have all the components of R.H.S of the equation. So we can confirm 𝐴=𝑉Λ𝑉−1:
print("\nMatrix 𝑉Λ𝑉−1: \n", V @ Lambda @ Vinv) #np.dot(V, np.dot(Lambda, Vinv))
Matrix A: [[ 4 2] [-5 -3]] Matrix 𝑉: [[ 0.70710678 -0.37139068] [-0.70710678 0.92847669]] Matrix Λ: [[ 2. 0.] [ 0. -1.]] Matrix 𝑉inv: [[2.3570226 0.94280904] [1.79505494 1.79505494]] Matrix 𝑉Λ𝑉−1: [[ 4. 2.] [-5. -3.]]
This is the original matrix $A$, and this is what eigendecomposition is :)
Example: A very common operation that we perform in data science is taking the power of a matrix, i.e., $A^n$. Given the matrix $A = \begin{bmatrix} 4 & 2\\ -5 & -3\end{bmatrix}$. Compute the power of this matrix $A^{16}$
A = np.array([[4, 2], [-5, -3]])
a2 = np.dot(A,A)
a4 = np.dot(a2,a2)
a8 = np.dot(a4,a4)
a16 = np.dot(a8,a8)
a16
array([[ 109226, 43690], [-109225, -43689]])
Example: Given the matrix $A = \begin{bmatrix} 4 & 2\\ -5 & -3\end{bmatrix}$. Compute the power of this matrix $A^{16}$, by using eigendecomposition
\Lambda^{16}=\begin{bmatrix} \lambda_1^{16} & 0\\ 0 & \lambda_2^{16} \end{bmatrix} $
A = np.array([[4, 2], [-5, -3]])
lambdas, V = np.linalg.eig(A)
lambdas16 = lambdas**16
Lambdas16 = np.diag(lambdas16)
Vinv = np.linalg.inv(V)
result = V @ Lambdas16 @ Vinv # np.dot(V,np.dot(Lambdas16,Vinv))
print("\nlambdas: ", lambdas)
print("Lambdas16: \n", Lambdas16)
print("Result: \n", result)
lambdas: [ 2. -1.] Lambdas16: [[6.5536e+04 0.0000e+00] [0.0000e+00 1.0000e+00]] Result: [[ 109226. 43690.] [-109225. -43689.]]
If matrix $A$ is a real symmetric matrix (a matrix having $A = A^T$), then you can use the tranpose instead of inverse.
To perform eigendecomposition using $A = V \Lambda V^{-1}$, we have to calculate the inverse of a matrix $V$, which is a complex operation. (However for matrices involving complex numbers that is the only option)
In machine learning, most of the times, we are typically working with real symmetric matrices, which can be conveniently and efficiently decomposed into real-only eigenvectors and real-only eigenvalues using the following formula, that use transpose instead of inverse:
Where,
eigenvectors
of matrix $A$,eigenvalues
of matrix $A$transpose
of the matrix comprised of the eigenvectorsThe only requirement is that the matrix $A$ need to be a real symmetric matrix (a matrix having $A = A^T$).
For a symmetric matrix $A$:
Example: Given a symmetric matrix $A = \begin{bmatrix} 2 & 1\\ 1 & 2\end{bmatrix}$, prove that the equation $A = Q \Lambda Q^T$ holds true. Finally, also prove that Q is an orthogonal matrix.
A = np.array([[2, 1], [1, 2]])
lambdas, Q = np.linalg.eig(A) # Calculate eigenvalues and eigenvectors of matrix A
Lambda = np.diag(lambdas) # Calculate the matrix Λ, which is the diagonal matrix of eigenvalues
Qt = Q.T # Calculate transpose of matrix Q
print("Matrix A: \n", A)
print("\nMatrix Q:\n", Q, end=' ')
print("\nMatrix Λ:\n", Lambda, end=' ')
print("\nMatrix QT:\n", Qt)
# Now we have all the components of R.H.S of the equation. So we can confirm 𝐴=QΛQt:
print("\nMatrix QΛQT: \n", Q @ Lambda @ Qt)
Matrix A: [[2 1] [1 2]] Matrix Q: [[ 0.70710678 -0.70710678] [ 0.70710678 0.70710678]] Matrix Λ: [[3. 0.] [0. 1.]] Matrix QT: [[ 0.70710678 0.70710678] [-0.70710678 0.70710678]] Matrix QΛQT: [[2. 1.] [1. 2.]]
A = np.array([[1, 5, 3], [4, 2, 1], [9,7,6], [3,4,8], [7,5,2]])
U, s, VT = np.linalg.svd(A)# V is already transposed and s is a vector of singular values
S = np.diag(s)
print("A:\n", A)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
print("A:\n",U[:, 0:3] @ S @ VT)
A: [[1 5 3] [4 2 1] [9 7 6] [3 4 8] [7 5 2]] U: [[-0.27100758 -0.30553854 0.89172505 0.07283989 -0.18089188] [-0.22574406 0.30341591 -0.15540972 0.57451812 -0.7090523 ] [-0.68888455 0.17905033 -0.13626746 -0.65730639 -0.2067814 ] [-0.44753102 -0.73558093 -0.38129625 0.29931827 0.15381294] [-0.44804975 0.49137222 0.12930042 0.37812719 0.63095641]] S: [[18.63328108 0. 0. ] [ 0. 5.81591983 0. ] [ 0. 0. 2.82416584]] VT: [[-0.63611374 -0.57204583 -0.51780584] [ 0.6452016 -0.02630216 -0.76355949] [-0.42317161 0.81979983 -0.38581604]] A: [[1. 5. 3.] [4. 2. 1.] [9. 7. 6.] [3. 4. 8.] [7. 5. 2.]]
A = np.array([[1, 5, 3], [4, 2, 1], [9,7,6], [3,4,8], [7,5,2]])
U, s, VT = np.linalg.svd(A, full_matrices=False)
S = np.diag(s)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
print("U @ S @ VT:\n", U @ S @ VT)
U: [[-0.27100758 -0.30553854 0.89172505] [-0.22574406 0.30341591 -0.15540972] [-0.68888455 0.17905033 -0.13626746] [-0.44753102 -0.73558093 -0.38129625] [-0.44804975 0.49137222 0.12930042]] S: [[18.63328108 0. 0. ] [ 0. 5.81591983 0. ] [ 0. 0. 2.82416584]] VT: [[-0.63611374 -0.57204583 -0.51780584] [ 0.6452016 -0.02630216 -0.76355949] [-0.42317161 0.81979983 -0.38581604]] U @ S @ VT: [[1. 5. 3.] [4. 2. 1.] [9. 7. 6.] [3. 4. 8.] [7. 5. 2.]]
import scipy
A = np.array([[1, 5, 3], [4, 2, 1], [9,7,6], [3,4,8], [7,5,2]])
U, s, VT = scipy.linalg.svd(A, full_matrices=False)
S = np.diag(s)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
print("U @ S @ VT:\n", U @ S @ VT)
U: [[-0.27100758 -0.30553854 0.89172505] [-0.22574406 0.30341591 -0.15540972] [-0.68888455 0.17905033 -0.13626746] [-0.44753102 -0.73558093 -0.38129625] [-0.44804975 0.49137222 0.12930042]] S: [[18.63328108 0. 0. ] [ 0. 5.81591983 0. ] [ 0. 0. 2.82416584]] VT: [[-0.63611374 -0.57204583 -0.51780584] [ 0.6452016 -0.02630216 -0.76355949] [-0.42317161 0.81979983 -0.38581604]] U @ S @ VT: [[1. 5. 3.] [4. 2. 1.] [9. 7. 6.] [3. 4. 8.] [7. 5. 2.]]
from sklearn.utils.extmath import randomized_svd
A = np.array([[1, 5, 3], [4, 2, 1], [9,7,6], [3,4,8], [7,5,2]])
U, s, VT = randomized_svd(A, n_components = 3)
S = np.diag(s)
print("A:\n", A)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
print("U @ S @ VT:\n", U @ S @ VT)
A: [[1 5 3] [4 2 1] [9 7 6] [3 4 8] [7 5 2]] U: [[ 0.27100758 0.30553854 0.89172505] [ 0.22574406 -0.30341591 -0.15540972] [ 0.68888455 -0.17905033 -0.13626746] [ 0.44753102 0.73558093 -0.38129625] [ 0.44804975 -0.49137222 0.12930042]] S: [[18.63328108 0. 0. ] [ 0. 5.81591983 0. ] [ 0. 0. 2.82416584]] VT: [[ 0.63611374 0.57204583 0.51780584] [-0.6452016 0.02630216 0.76355949] [-0.42317161 0.81979983 -0.38581604]] U @ S @ VT: [[1. 5. 3.] [4. 2. 1.] [9. 7. 6.] [3. 4. 8.] [7. 5. 2.]]
import numpy as np
import pandas as pd
import math
import scipy
from matplotlib import pyplot as plt
from matplotlib import image
import sklearn
from plot_helper import * # Helper functions: plot_vector, plot_linear_transformation, plot_linear_transformations
np.set_printoptions(suppress=True, precision=2)
Example: Given the matrix $A$, decompose it using SVD, and later prove that the equation $A = U\sum V^T$ holds true by reconstructing the matrix $A$ from its SVD constituent parts (step by step).
Decompose matrix $A$ to its component matrices using SVD:
A = np.array([[3,1,4,1], [5,9,2,6], [5,3,5,8], [9,7,9,3]])
U, s, VT = np.linalg.svd(A, full_matrices=False)# V is already transposed and s is a vector of singular values
S = np.diag(s)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
U: [[-0.21 0.37 -0.13 -0.89] [-0.52 -0.7 0.43 -0.23] [-0.48 -0.21 -0.84 0.15] [-0.67 0.57 0.31 0.36]] S: [[21.23 0. 0. 0. ] [ 0. 6.43 0. 0. ] [ 0. 0. 4.88 0. ] [ 0. 0. 0. 0.15]] VT: [[-0.55 -0.52 -0.49 -0.43] [ 0.26 -0.4 0.65 -0.59] [ 0.07 0.7 -0.22 -0.68] [ 0.79 -0.29 -0.54 -0.04]]
Generate matrix $A$ from its component matrices: $A = USV^T$
U @ S @ VT
array([[3., 1., 4., 1.], [5., 9., 2., 6.], [5., 3., 5., 8.], [9., 7., 9., 3.]])
Create matrix $rv1$
from sklearn.utils.extmath import randomized_svd
A = np.array([[3,1,4,1], [5,9,2,6], [5,3,5,8], [9,7,9,3]])
U, s, VT = np.linalg.svd(A, full_matrices=False)
S = np.diag(s)
rv1 = S[0,0] * U[:,0].reshape(4,1) @ VT[0,:].reshape(1,4)
print("First component:\n", rv1)
U, s, VT = randomized_svd(A, n_components = 1)
S = np.diag(s)
print("First component:\n", U @ S @ VT)
First component: [[2.51 2.37 2.22 1.97] [6.07 5.72 5.37 4.77] [5.63 5.31 4.99 4.43] [7.88 7.43 6.98 6.19]] First component: [[2.51 2.37 2.22 1.97] [6.07 5.72 5.37 4.77] [5.63 5.31 4.99 4.43] [7.88 7.43 6.98 6.19]]
Create matrix $rv1 + rv2$
A = np.array([[3,1,4,1], [5,9,2,6], [5,3,5,8], [9,7,9,3]])
U, s, VT = np.linalg.svd(A, full_matrices=False)
S = np.diag(s)
rv2 = S[1,1] * U[:,1].reshape(4,1) @ VT[1,:].reshape(1,4)
print("First two components:\n", rv1 + rv2)
U, s, VT = randomized_svd(A, n_components = 2)
S = np.diag(s)
print("First two component:\n", U @ S @ VT)
First two components: [[3.15 1.41 3.79 0.56] [4.87 7.53 2.44 7.43] [5.28 5.85 4.12 5.22] [8.85 5.96 9.36 4.03]] First two component: [[3.15 1.41 3.79 0.56] [4.87 7.53 2.44 7.43] [5.28 5.85 4.12 5.22] [8.85 5.96 9.36 4.03]]
Create matrix $rv1 + rv2 + rv3$
A = np.array([[3,1,4,1], [5,9,2,6], [5,3,5,8], [9,7,9,3]])
U, s, VT = np.linalg.svd(A, full_matrices=False)
S = np.diag(s)
rv3 = S[2,2] * U[:,2].reshape(4,1) @ VT[2,:].reshape(1,4)
print("First three components:\n", rv1 + rv2 + rv3)
U, s, VT = randomized_svd(A, n_components = 3)
S = np.diag(s)
print("First three component:\n", U @ S @ VT)
First three components: [[3.1 0.96 3.93 0.99] [5.03 8.99 1.98 6. ] [4.98 3.01 5.01 8. ] [8.96 7.02 9.03 3. ]] First three component: [[3.1 0.96 3.93 0.99] [5.03 8.99 1.98 6. ] [4.98 3.01 5.01 8. ] [8.96 7.02 9.03 3. ]]
Create matrix $rv1 + rv2 + rv3 + rv4$
A = np.array([[3,1,4,1], [5,9,2,6], [5,3,5,8], [9,7,9,3]])
U, s, VT = np.linalg.svd(A, full_matrices=False)# V is already transposed and s is a vector of singular values
S = np.diag(s)
rv4 = S[3,3] * U[:,3].reshape(4,1) @ VT[3,:].reshape(1,4)
print("First four components:\n", rv1 + rv2 + rv3 + rv4)
U, s, VT = randomized_svd(A, n_components = 4)
S = np.diag(s)
print("First four component:\n", U @ S @ VT)
First four components: [[3. 1. 4. 1.] [5. 9. 2. 6.] [5. 3. 5. 8.] [9. 7. 9. 3.]] First four component: [[3. 1. 4. 1.] [5. 9. 2. 6.] [5. 3. 5. 8.] [9. 7. 9. 3.]]
Example (Image Compression using SVD):
Read a color image into a NumPy 3-D array
'''
imread(), reads an image from a file into an array. The returned array has shape
- (M, N) for grayscale images.
- (M, N, 3) for RGB images.
- (M, N, 4) for RGBA images.
'''
rgb_img = image.imread("datasets/img.jpg")
fig = plt.figure(figsize=(10, 10))
plt.imshow(rgb_img)
print(rgb_img.shape)
(1786, 2880, 3)
Convert to gray scale and store in a NumPy 2-D array
gray_img = np.mean(rgb_img, -1)
fig = plt.figure(figsize=(10, 10))
plt.imshow(gray_img, cmap='gray')
print(gray_img.shape)
(1786, 2880)
Decompose using SVD:
U, s, VT = np.linalg.svd(gray_img, full_matrices=False)
S = np.diag(s)
print("U:\n", U)
print("S:\n", S)
print("VT:\n", VT)
U: [[-0.03 -0.01 0.02 ... 0.02 0.02 -0.04] [-0.03 -0.01 0.02 ... -0.01 -0.02 0.05] [-0.03 -0.01 0.02 ... -0.02 0. -0.02] ... [-0.02 0.04 0.03 ... 0. 0. 0. ] [-0.02 0.04 0.03 ... -0. -0. -0. ] [-0.02 0.04 0.03 ... 0. 0. -0. ]] S: [[358995.35 0. 0. ... 0. 0. 0. ] [ 0. 61144.51 0. ... 0. 0. 0. ] [ 0. 0. 44711.25 ... 0. 0. 0. ] ... [ 0. 0. 0. ... 3.99 0. 0. ] [ 0. 0. 0. ... 0. 3.93 0. ] [ 0. 0. 0. ... 0. 0. 3.53]] VT: [[-0.02 -0.02 -0.02 ... -0.02 -0.02 -0.02] [ 0.04 0.04 0.04 ... -0.01 -0.01 -0.01] [-0.03 -0.03 -0.03 ... -0.04 -0.04 -0.04] ... [ 0.02 0.01 -0.01 ... -0.06 -0. 0.06] [ 0.01 -0.01 -0.02 ... -0.03 -0.03 0.01] [ 0.02 -0.04 -0.03 ... 0.01 -0.03 0.01]]
Compress by taking the first
k
components:
k = 100
compressed_img = U[:, :k] @ S[0:k, 0:k] @ VT[0:k, :]
fig = plt.figure(figsize=(10, 10))
plt.imshow(compressed_img, cmap='gray');
With 100 singular vectors, the image is reconstructed quite well, however the data footprint is much smaller than the original image
$$= (m × k) + k + (k × n)$$$$= (1786 × 100) + 100 + (100 × 2880)$$$$= 466700$$original_size = 1786*2880
original_size/(2**20)
4.9053955078125
compressed_size = 1786*100 + 100 + 100*2880
compressed_size/(2**20)
0.4450798034667969
Curse of Dimensionality:
dimension
of a dataset corresponds to the number of attributes/features that exist in a dataset.Curse of Dimensionality
Techniques to Reduce the Dimensionality of your dataset:
What is Principal Component Analysis (PCA):
Note:
- PCA and SVD are closely related approaches and can be both applied to decompose any rectangular matrix, and thus both can beused for dimensionality reduction.
- The main difference between The Singular value decomposition and principal component analysis is that the SVD is a data-driven Fourier transform generalization, whereas PCA allows us to represent statistical variations in our data sets using a hierarchical coordinate system based on data.
- One caveat of using the covariance matrix to calculate the PCA is that it can be hard to compute when there are many features. For this reason, it is usually preferred to use the Singular Value Decomposition (SVD) to calculate the PCA.
eigenvectors of the dataset
. This process of PCA is described in the following five steps:k
eigenvectors corresponding to k
highest eigenvalues, and create Feature/Projection Matrix containing top k
Principal Components.
df = pd.read_csv("datasets/pca_housingdata.csv")
df
Size | CoveredArea | Bedrooms | TreesNearby | Bathrooms | SchoolsNearby | Price | |
---|---|---|---|---|---|---|---|
0 | 2000 | 3500 | 3 | 1 | 3 | 1 | 20500000 |
1 | 2600 | 3000 | 2 | 0 | 4 | 1 | 1800000 |
2 | 1800 | 2000 | 3 | 1 | 3 | 2 | 2000000 |
3 | 1600 | 2600 | 1 | 2 | 0 | 0 | 500000 |
4 | 2600 | 2000 | 3 | 3 | 1 | 1 | 400000 |
5 | 3000 | 1000 | 2 | 2 | 1 | 0 | 600000 |
6 | 2000 | 3600 | 4 | 4 | 3 | 3 | 3000000 |
7 | 1500 | 2800 | 2 | 2 | 1 | 0 | 600000 |
8 | 1000 | 2000 | 3 | 0 | 1 | 2 | 1000000 |
A = df.to_numpy()
A = A[:, 0:6]
A
array([[2000, 3500, 3, 1, 3, 1], [2600, 3000, 2, 0, 4, 1], [1800, 2000, 3, 1, 3, 2], [1600, 2600, 1, 2, 0, 0], [2600, 2000, 3, 3, 1, 1], [3000, 1000, 2, 2, 1, 0], [2000, 3600, 4, 4, 3, 3], [1500, 2800, 2, 2, 1, 0], [1000, 2000, 3, 0, 1, 2]])
$$ z = \frac{value - mean}{stdev}$$Step 1: Normalize/Standardize your data to avoid biasness towards specific features
- Compute mean and stndard deviation of every column.
- The standardized data value can be computed by using following formula:
Standard deviation
tells us how much a (single) quantity varies w.r.t. its mean 'OR' how spread out the data is around the center of the distribution (the mean).- Standard deviation of a sample can be calculated using the formula $\sqrt{\frac{\sum(x_i-\bar{x})^2}{n}}$, or by using
np.std(data)
- Once the standardization is done, all the variables will be transformed to the same scale.
mean_vector = np.mean(A, axis=0)
std_vector = np.std(A, axis=0)
N = (A - mean_vector)/std_vector
print("Mean of each column is: ", mean_vector)
print("Standard Deviation of each column is: ", std_vector)
print("Normalize data N:\n", N)
Mean of each column is: [2011.11 2500. 2.56 1.67 1.89 1.11] Standard Deviation of each column is: [593.38 785.99 0.83 1.25 1.29 0.99] Normalize data N: [[-0.02 1.27 0.53 -0.53 0.86 -0.11] [ 0.99 0.64 -0.67 -1.34 1.64 -0.11] [-0.36 -0.64 0.53 -0.53 0.86 0.89] [-0.69 0.13 -1.87 0.27 -1.47 -1.12] [ 0.99 -0.64 0.53 1.07 -0.69 -0.11] [ 1.67 -1.91 -0.67 0.27 -0.69 -1.12] [-0.02 1.4 1.74 1.87 0.86 1.9 ] [-0.86 0.38 -0.67 0.27 -0.69 -1.12] [-1.7 -0.64 0.53 -1.34 -0.69 0.89]]
Step 2: Determine Covariance Matrix to identify the relationship among features.
- The aim of this step is to understand how the variables of the input data set are varying from the mean with respect to each other, or in other words, to see if there is any relationship between them. Because sometimes, variables are highly correlated in such a way that they contain redundant information. So, in order to identify these correlations, we compute the covariance matrix.
- Covariance is a statistical measure that is used to measure as to how the mean values of two random variables move together. It measures the direction of the relationship between two features or variables.
- `Positive covariance`: Indicates that two variables tend to move in the same direction.
- `Negative covariance`: Reveals that two variables tend to move in inverse directions.
- `Zero covarince`: Indicates that two variables have no relationship between each other.
- For multi-dimensional data, we use covariance matrix also called variance-covariance matrix, as its diagonal values show variances and the other values are the covariances.
- Covariance can be calculated using the formula $ \text{cov}(x, y) = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) }{n} $, or by using
np.cov(data)
V = np.cov(N.T)
V
array([[ 1.12, -0.29, -0.06, 0.28, 0.28, -0.28], [-0.29, 1.13, 0.31, 0.15, 0.57, 0.37], [-0.06, 0.31, 1.12, 0.32, 0.53, 0.97], [ 0.28, 0.15, 0.32, 1.12, -0.26, 0.13], [ 0.28, 0.57, 0.53, -0.26, 1.12, 0.6 ], [-0.28, 0.37, 0.97, 0.13, 0.6 , 1.12]])
Step 3: Perform Eigendecomposition of Covariance Matrix to get eigenvectors and eigenvalues.
- In this example, you get a 6x6 matrix, because we have six features.
- The six columns of the resulting matrix are the six Principal Components.
- The eigenvector having the maximum corresponding eigenvalue is called the Principal Component 1, that holds the maximum possible information.
- Computing the eigenvectors and ordering them by their eigenvalues in descending order, allow us to find the principal components in order of significance
- An important thing to realize here is that, the principal components are less interpretable and don’t have any real meaning since they are constructed as linear combinations of the initial variables.
lambdas, vectors = np.linalg.eig(V)
print("Eigenvectors = \n", vectors)
print("\nEigenvalues = ", lambdas)
Eigenvectors = [[-0.09 0.45 0.06 -0.68 -0.56 -0.01] [ 0.39 0.36 0.17 0.2 0.01 -0.8 ] [ 0.55 -0.06 0.72 -0.19 0.17 0.34] [ 0.1 -0.35 -0.25 -0.67 0.5 -0.33] [ 0.46 -0.6 -0.2 0.04 -0.61 -0.09] [ 0.57 0.42 -0.59 0.03 0.16 0.35]] Eigenvalues = [2.87 0.05 0.13 1.46 1.29 0.96]
Step 4: Select
k
eigenvectors corresponding tok
highest eigenvalues, and create Feature Matrix
- Featue Vectors is a matrix that has eigenvectors of the components that we decide to keep as columns.
- For this we sort the eigenvalues and select the first
n
corresponding eigenvectors.- For example, if you want to decrease the dimension to two from six, then take the first two eigenvectors which are corresponding to the two highest eigenvalues, i.e., 2.87 and 1.46
- If all eigenvalues have a similar value, that means the existing representation is already reasonably compressed or dense and that the projection may offer little.
- If there are eigenvalues close to zero, they represent components or axes that may be discarded.
FM = vectors[:,0:5:3]
FM
array([[-0.09, -0.68], [ 0.39, 0.2 ], [ 0.55, -0.19], [ 0.1 , -0.67], [ 0.46, 0.04], [ 0.57, 0.03]])
Step 5: Create the Final Dataset with reduced dimensions by multiplying normalized matrix with feature matrix
- Note the shape of the normalized data matrix is 9x6, while the shape of the Feature matrix having two Principal Components is 6x2, and ofcourse we can take their dot product.
print("N.shape: ", N.shape)
print("FM.shape: ", FM.shape)
N.shape: (9, 6) FM.shape: (6, 2)
PCs = N @ FM
print("\nProjected Data = \n", PCs)
Projected Data = [[ 1.07 0.56] [ 0.36 0.53] [ 0.93 0.42] [-2.2 0.6 ] [-0.32 -1.66] [-2.18 -1.64] [ 3.15 -1.21] [-1.07 0.56] [ 0.25 1.83]]
PCA is in fact a method for doing feature extraction. In PCA, derived features are also called composite features or principal components. Moreover, these principal components are linearly independent from one another.
principal_df = pd.DataFrame(data = PCs, columns = ['PC1', 'PC2'])
principal_df
PC1 | PC2 | |
---|---|---|
0 | 1.071032 | 0.555674 |
1 | 0.358702 | 0.529988 |
2 | 0.928666 | 0.424145 |
3 | -2.197137 | 0.599276 |
4 | -0.319193 | -1.656922 |
5 | -2.177729 | -1.635425 |
6 | 3.151058 | -1.210562 |
7 | -1.068286 | 0.562874 |
8 | 0.252886 | 1.830952 |
Now we can always create a new data set having just two feature columns and the output column and perform predictions on it . But this will be done in our ML module where we will be covering lot more InshaAllah.
final_df = pd.concat([principal_df, df[['Price']]], axis = 1)
final_df
PC1 | PC2 | Price | |
---|---|---|---|
0 | 1.071032 | 0.555674 | 20500000 |
1 | 0.358702 | 0.529988 | 1800000 |
2 | 0.928666 | 0.424145 | 2000000 |
3 | -2.197137 | 0.599276 | 500000 |
4 | -0.319193 | -1.656922 | 400000 |
5 | -2.177729 | -1.635425 | 600000 |
6 | 3.151058 | -1.210562 | 3000000 |
7 | -1.068286 | 0.562874 | 600000 |
8 | 0.252886 | 1.830952 | 1000000 |
Now you can use this new dataframe having just two features instead of six to run the Linear Regression or whatever machine learning model we want to apply
scikit-learn
¶Step 1: Load Dataset
from sklearn import datasets
digits_dict = sklearn.datasets.load_digits()
digits_dict.keys()
dict_keys(['data', 'target', 'frame', 'feature_names', 'target_names', 'images', 'DESCR'])
digits_dict.data
array([[ 0., 0., 5., ..., 0., 0., 0.], [ 0., 0., 0., ..., 10., 0., 0.], [ 0., 0., 0., ..., 16., 9., 0.], ..., [ 0., 0., 1., ..., 6., 0., 0.], [ 0., 0., 2., ..., 12., 0., 0.], [ 0., 0., 10., ..., 12., 1., 0.]])
digits_dict.data.shape
(1797, 64)
digits_dict.target
array([0, 1, 2, ..., 8, 9, 8])
digits_dict.target.shape
(1797,)
digits_dict.data[100]
array([ 0., 0., 0., 2., 13., 0., 0., 0., 0., 0., 0., 8., 15., 0., 0., 0., 0., 0., 5., 16., 5., 2., 0., 0., 0., 0., 15., 12., 1., 16., 4., 0., 0., 4., 16., 2., 9., 16., 8., 0., 0., 0., 10., 14., 16., 16., 4., 0., 0., 0., 0., 0., 13., 8., 0., 0., 0., 0., 0., 0., 13., 6., 0., 0.])
digits_dict.target[100]
4
digits_dict.data[100].reshape(8,8)
array([[ 0., 0., 0., 2., 13., 0., 0., 0.], [ 0., 0., 0., 8., 15., 0., 0., 0.], [ 0., 0., 5., 16., 5., 2., 0., 0.], [ 0., 0., 15., 12., 1., 16., 4., 0.], [ 0., 4., 16., 2., 9., 16., 8., 0.], [ 0., 0., 10., 14., 16., 16., 4., 0.], [ 0., 0., 0., 0., 13., 8., 0., 0.], [ 0., 0., 0., 0., 13., 6., 0., 0.]])
plt.imshow(digits_dict.data[100].reshape(8,8),cmap='gray');
df1 = pd.DataFrame(digits_dict.data, columns=digits_dict.feature_names)
df1
pixel_0_0 | pixel_0_1 | pixel_0_2 | pixel_0_3 | pixel_0_4 | pixel_0_5 | pixel_0_6 | pixel_0_7 | pixel_1_0 | pixel_1_1 | ... | pixel_6_6 | pixel_6_7 | pixel_7_0 | pixel_7_1 | pixel_7_2 | pixel_7_3 | pixel_7_4 | pixel_7_5 | pixel_7_6 | pixel_7_7 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.0 | 0.0 | 5.0 | 13.0 | 9.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 6.0 | 13.0 | 10.0 | 0.0 | 0.0 | 0.0 |
1 | 0.0 | 0.0 | 0.0 | 12.0 | 13.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 11.0 | 16.0 | 10.0 | 0.0 | 0.0 |
2 | 0.0 | 0.0 | 0.0 | 4.0 | 15.0 | 12.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.0 | 11.0 | 16.0 | 9.0 | 0.0 |
3 | 0.0 | 0.0 | 7.0 | 15.0 | 13.0 | 1.0 | 0.0 | 0.0 | 0.0 | 8.0 | ... | 9.0 | 0.0 | 0.0 | 0.0 | 7.0 | 13.0 | 13.0 | 9.0 | 0.0 | 0.0 |
4 | 0.0 | 0.0 | 0.0 | 1.0 | 11.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 16.0 | 4.0 | 0.0 | 0.0 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
1792 | 0.0 | 0.0 | 4.0 | 10.0 | 13.0 | 6.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 4.0 | 0.0 | 0.0 | 0.0 | 2.0 | 14.0 | 15.0 | 9.0 | 0.0 | 0.0 |
1793 | 0.0 | 0.0 | 6.0 | 16.0 | 13.0 | 11.0 | 1.0 | 0.0 | 0.0 | 0.0 | ... | 1.0 | 0.0 | 0.0 | 0.0 | 6.0 | 16.0 | 14.0 | 6.0 | 0.0 | 0.0 |
1794 | 0.0 | 0.0 | 1.0 | 11.0 | 15.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 9.0 | 13.0 | 6.0 | 0.0 | 0.0 |
1795 | 0.0 | 0.0 | 2.0 | 10.0 | 7.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 2.0 | 0.0 | 0.0 | 0.0 | 5.0 | 12.0 | 16.0 | 12.0 | 0.0 | 0.0 |
1796 | 0.0 | 0.0 | 10.0 | 14.0 | 8.0 | 1.0 | 0.0 | 0.0 | 0.0 | 2.0 | ... | 8.0 | 0.0 | 0.0 | 1.0 | 8.0 | 12.0 | 14.0 | 12.0 | 1.0 | 0.0 |
1797 rows × 64 columns
df2 = pd.DataFrame(digits_dict.target, columns=['Digit'])
df2
Digit | |
---|---|
0 | 0 |
1 | 1 |
2 | 2 |
3 | 3 |
4 | 4 |
... | ... |
1792 | 9 |
1793 | 0 |
1794 | 8 |
1795 | 9 |
1796 | 8 |
1797 rows × 1 columns
df = pd.concat([df1,df2], axis=1)
df
pixel_0_0 | pixel_0_1 | pixel_0_2 | pixel_0_3 | pixel_0_4 | pixel_0_5 | pixel_0_6 | pixel_0_7 | pixel_1_0 | pixel_1_1 | ... | pixel_6_7 | pixel_7_0 | pixel_7_1 | pixel_7_2 | pixel_7_3 | pixel_7_4 | pixel_7_5 | pixel_7_6 | pixel_7_7 | Digit | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0.0 | 0.0 | 5.0 | 13.0 | 9.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 6.0 | 13.0 | 10.0 | 0.0 | 0.0 | 0.0 | 0 |
1 | 0.0 | 0.0 | 0.0 | 12.0 | 13.0 | 5.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 11.0 | 16.0 | 10.0 | 0.0 | 0.0 | 1 |
2 | 0.0 | 0.0 | 0.0 | 4.0 | 15.0 | 12.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 3.0 | 11.0 | 16.0 | 9.0 | 0.0 | 2 |
3 | 0.0 | 0.0 | 7.0 | 15.0 | 13.0 | 1.0 | 0.0 | 0.0 | 0.0 | 8.0 | ... | 0.0 | 0.0 | 0.0 | 7.0 | 13.0 | 13.0 | 9.0 | 0.0 | 0.0 | 3 |
4 | 0.0 | 0.0 | 0.0 | 1.0 | 11.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 2.0 | 16.0 | 4.0 | 0.0 | 0.0 | 4 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
1792 | 0.0 | 0.0 | 4.0 | 10.0 | 13.0 | 6.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 2.0 | 14.0 | 15.0 | 9.0 | 0.0 | 0.0 | 9 |
1793 | 0.0 | 0.0 | 6.0 | 16.0 | 13.0 | 11.0 | 1.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 6.0 | 16.0 | 14.0 | 6.0 | 0.0 | 0.0 | 0 |
1794 | 0.0 | 0.0 | 1.0 | 11.0 | 15.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 2.0 | 9.0 | 13.0 | 6.0 | 0.0 | 0.0 | 8 |
1795 | 0.0 | 0.0 | 2.0 | 10.0 | 7.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 5.0 | 12.0 | 16.0 | 12.0 | 0.0 | 0.0 | 9 |
1796 | 0.0 | 0.0 | 10.0 | 14.0 | 8.0 | 1.0 | 0.0 | 0.0 | 0.0 | 2.0 | ... | 0.0 | 0.0 | 1.0 | 8.0 | 12.0 | 14.0 | 12.0 | 1.0 | 0.0 | 8 |
1797 rows × 65 columns
A = df1.to_numpy()
A = digits_dict.data
A
array([[ 0., 0., 5., ..., 0., 0., 0.], [ 0., 0., 0., ..., 10., 0., 0.], [ 0., 0., 0., ..., 16., 9., 0.], ..., [ 0., 0., 1., ..., 6., 0., 0.], [ 0., 0., 2., ..., 12., 0., 0.], [ 0., 0., 10., ..., 12., 1., 0.]])
A.shape
(1797, 64)
Step 2: Standardize Dataset
from sklearn.preprocessing import StandardScaler
# Create the object
scalar = StandardScaler()
# Calculate the mean and standard deviation
scalar.fit(A)
# Transform the values
N = scalar.transform(A)
#N = scalar.fit_transform(A)
print("Normalize data N:\n", N)
print("N.shape: ", N.shape)
Normalize data N: [[ 0. -0.34 -0.04 ... -1.15 -0.51 -0.2 ] [ 0. -0.34 -1.09 ... 0.55 -0.51 -0.2 ] [ 0. -0.34 -1.09 ... 1.57 1.7 -0.2 ] ... [ 0. -0.34 -0.88 ... -0.13 -0.51 -0.2 ] [ 0. -0.34 -0.67 ... 0.89 -0.51 -0.2 ] [ 0. -0.34 1.01 ... 0.89 -0.26 -0.2 ]] N.shape: (1797, 64)
Step 3: Conduct PCA
from sklearn.decomposition import PCA
#pca = sklearn.decomposition.PCA()
#pca = sklearn.decomposition.PCA(n_components=20)
pca = sklearn.decomposition.PCA(n_components=0.9)
#pca.fit(N)
#PCs = pca.transform(N)
PCs = pca.fit_transform(N)
print("PCs:\n", PCs)
print("PCs.shape: ", PCs.shape)
print("A.shape: ", A.shape)
PCs: [[ 1.91 -0.95 -3.95 ... 0.4 -0.32 0.53] [ 0.59 0.92 3.92 ... -0.86 -0.28 1.1 ] [ 1.3 -0.32 3.02 ... -0.36 1.18 -0.75] ... [ 1.02 -0.15 2.47 ... 0.52 -0.27 -0.95] [ 1.08 -0.38 -2.46 ... 0.29 -0.71 0.18] [-1.26 -2.23 0.28 ... 0.63 0.33 0.6 ]] PCs.shape: (1797, 31) A.shape: (1797, 64)
pca.explained_variance_ratio_*100
array([12.03, 9.56, 8.44, 6.5 , 4.86, 4.21, 3.94, 3.39, 3. , 2.93, 2.78, 2.58, 2.28, 2.23, 2.17, 1.91, 1.78, 1.64, 1.6 , 1.49, 1.35, 1.27, 1.17, 1.06, 0.98, 0.94, 0.86, 0.84, 0.8 , 0.75, 0.73])
print("Variance explained by first PC: ", np.cumsum(pca.explained_variance_ratio_*100)[0])
Variance explained by first PC: 12.0339160977349
print("Variance explained by first ten PCs: ", np.cumsum(pca.explained_variance_ratio_*100)[9])
Variance explained by first ten PCs: 58.873755337302974
print("Variance explained by first twenty PCs: ", np.cumsum(pca.explained_variance_ratio_*100)[19])
Variance explained by first twenty PCs: 79.31376270240806
print("Variance explained by all the 31 PCs: ", np.cumsum(pca.explained_variance_ratio_*100)[30])
Variance explained by all the 31 PCs: 90.04642597586596
Step 4: Create new Dataframe having Principal Components and the output variable
principal_df = pd.DataFrame(data = PCs)
principal_df
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1.914214 | -0.954502 | -3.946035 | 2.028723 | -0.267173 | 0.530327 | -1.415321 | 1.496062 | 0.124914 | -0.822246 | ... | 0.558703 | 0.508826 | -0.758896 | -0.464500 | 0.732176 | 0.327318 | 0.157022 | 0.404918 | -0.324594 | 0.534683 |
1 | 0.588980 | 0.924636 | 3.924755 | -1.779850 | -0.993430 | -0.675652 | 1.878565 | 0.556336 | 1.079877 | 0.087451 | ... | -0.242434 | -0.478472 | 0.491158 | 0.790504 | 0.067788 | -1.525555 | -0.742482 | -0.855856 | -0.277084 | 1.098196 |
2 | 1.302039 | -0.317189 | 3.023333 | -2.043376 | -2.081155 | 0.935121 | -1.296200 | 1.156160 | 0.785606 | -1.099206 | ... | 0.069262 | -1.276142 | -0.389437 | -0.407580 | -0.121614 | 0.373418 | 0.205099 | -0.362138 | 1.181186 | -0.749792 |
3 | -3.020770 | -0.868772 | -0.801744 | -2.187039 | -0.556813 | 0.727124 | 0.959766 | -1.382638 | 0.259075 | 0.744555 | ... | -0.167976 | 0.573332 | -0.532997 | -0.084318 | -0.466793 | 0.154232 | -0.403244 | 0.291934 | -1.244539 | 0.268599 |
4 | 4.528949 | -1.093480 | 0.973121 | -1.419510 | -1.715106 | 1.431592 | 1.073649 | -0.968240 | -1.660216 | 1.174593 | ... | -0.185698 | 0.960864 | -0.184642 | 0.563999 | -2.142481 | 0.064456 | 0.746668 | 1.076306 | 0.297665 | -0.660454 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
1792 | 0.104331 | 0.255024 | -3.765861 | -1.947006 | -0.190094 | -0.555760 | 0.531222 | 0.476475 | 1.152430 | -0.473054 | ... | -0.783514 | -0.676671 | -0.042743 | -0.359572 | -0.066849 | 0.203868 | 0.058615 | 0.979933 | -0.122796 | -0.430137 |
1793 | 2.423234 | -1.429611 | -3.045245 | 2.632089 | -0.822902 | 0.004061 | -1.106872 | 2.330903 | 0.569455 | -1.654173 | ... | -0.041950 | 0.379564 | 0.082078 | 0.213136 | -0.864603 | 0.272784 | -0.567033 | -0.769823 | -0.008326 | -0.047497 |
1794 | 1.022596 | -0.147911 | 2.469974 | -0.620307 | -0.972043 | -0.007377 | 0.381115 | 0.529064 | 2.054709 | -2.036838 | ... | -0.237439 | 0.041549 | 0.235181 | -0.261485 | -1.340361 | -0.054225 | -0.352066 | 0.521824 | -0.271170 | -0.947936 |
1795 | 1.076055 | -0.380906 | -2.455487 | -1.312013 | 0.253533 | -0.638322 | 1.034470 | 0.763325 | 1.077474 | -0.334527 | ... | -0.433929 | 1.111662 | 0.312411 | -0.940159 | 1.152000 | -0.662983 | 0.979041 | 0.293561 | -0.711736 | 0.176904 |
1796 | -1.257702 | -2.227591 | 0.283628 | -0.127073 | -1.570173 | 2.342953 | -0.383075 | -1.200566 | 0.816831 | -1.825428 | ... | -0.291210 | -1.073341 | -0.274700 | -0.909278 | -0.450495 | -0.322974 | 0.916469 | 0.629528 | 0.327032 | 0.600322 |
1797 rows × 31 columns
new_df = pd.concat([principal_df,df2], axis=1)
new_df
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | Digit | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1.914214 | -0.954502 | -3.946035 | 2.028723 | -0.267173 | 0.530327 | -1.415321 | 1.496062 | 0.124914 | -0.822246 | ... | 0.508826 | -0.758896 | -0.464500 | 0.732176 | 0.327318 | 0.157022 | 0.404918 | -0.324594 | 0.534683 | 0 |
1 | 0.588980 | 0.924636 | 3.924755 | -1.779850 | -0.993430 | -0.675652 | 1.878565 | 0.556336 | 1.079877 | 0.087451 | ... | -0.478472 | 0.491158 | 0.790504 | 0.067788 | -1.525555 | -0.742482 | -0.855856 | -0.277084 | 1.098196 | 1 |
2 | 1.302039 | -0.317189 | 3.023333 | -2.043376 | -2.081155 | 0.935121 | -1.296200 | 1.156160 | 0.785606 | -1.099206 | ... | -1.276142 | -0.389437 | -0.407580 | -0.121614 | 0.373418 | 0.205099 | -0.362138 | 1.181186 | -0.749792 | 2 |
3 | -3.020770 | -0.868772 | -0.801744 | -2.187039 | -0.556813 | 0.727124 | 0.959766 | -1.382638 | 0.259075 | 0.744555 | ... | 0.573332 | -0.532997 | -0.084318 | -0.466793 | 0.154232 | -0.403244 | 0.291934 | -1.244539 | 0.268599 | 3 |
4 | 4.528949 | -1.093480 | 0.973121 | -1.419510 | -1.715106 | 1.431592 | 1.073649 | -0.968240 | -1.660216 | 1.174593 | ... | 0.960864 | -0.184642 | 0.563999 | -2.142481 | 0.064456 | 0.746668 | 1.076306 | 0.297665 | -0.660454 | 4 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
1792 | 0.104331 | 0.255024 | -3.765861 | -1.947006 | -0.190094 | -0.555760 | 0.531222 | 0.476475 | 1.152430 | -0.473054 | ... | -0.676671 | -0.042743 | -0.359572 | -0.066849 | 0.203868 | 0.058615 | 0.979933 | -0.122796 | -0.430137 | 9 |
1793 | 2.423234 | -1.429611 | -3.045245 | 2.632089 | -0.822902 | 0.004061 | -1.106872 | 2.330903 | 0.569455 | -1.654173 | ... | 0.379564 | 0.082078 | 0.213136 | -0.864603 | 0.272784 | -0.567033 | -0.769823 | -0.008326 | -0.047497 | 0 |
1794 | 1.022596 | -0.147911 | 2.469974 | -0.620307 | -0.972043 | -0.007377 | 0.381115 | 0.529064 | 2.054709 | -2.036838 | ... | 0.041549 | 0.235181 | -0.261485 | -1.340361 | -0.054225 | -0.352066 | 0.521824 | -0.271170 | -0.947936 | 8 |
1795 | 1.076055 | -0.380906 | -2.455487 | -1.312013 | 0.253533 | -0.638322 | 1.034470 | 0.763325 | 1.077474 | -0.334527 | ... | 1.111662 | 0.312411 | -0.940159 | 1.152000 | -0.662983 | 0.979041 | 0.293561 | -0.711736 | 0.176904 | 9 |
1796 | -1.257702 | -2.227591 | 0.283628 | -0.127073 | -1.570173 | 2.342953 | -0.383075 | -1.200566 | 0.816831 | -1.825428 | ... | -1.073341 | -0.274700 | -0.909278 | -0.450495 | -0.322974 | 0.916469 | 0.629528 | 0.327032 | 0.600322 | 8 |
1797 rows × 32 columns
Now you can use this new dataframe having just 31 features instead of 64 to run whatever classification machine learning model we want to apply
Step 1: Load Dataset
from sklearn import datasets
iris_dict = sklearn.datasets.load_iris()
iris_dict.keys()
dict_keys(['data', 'target', 'frame', 'target_names', 'DESCR', 'feature_names', 'filename'])
# contains 150 rows and 4 columns
iris_dict.data
array([[5.1, 3.5, 1.4, 0.2], [4.9, 3. , 1.4, 0.2], [4.7, 3.2, 1.3, 0.2], [4.6, 3.1, 1.5, 0.2], [5. , 3.6, 1.4, 0.2], [5.4, 3.9, 1.7, 0.4], [4.6, 3.4, 1.4, 0.3], [5. , 3.4, 1.5, 0.2], [4.4, 2.9, 1.4, 0.2], [4.9, 3.1, 1.5, 0.1], [5.4, 3.7, 1.5, 0.2], [4.8, 3.4, 1.6, 0.2], [4.8, 3. , 1.4, 0.1], [4.3, 3. , 1.1, 0.1], [5.8, 4. , 1.2, 0.2], [5.7, 4.4, 1.5, 0.4], [5.4, 3.9, 1.3, 0.4], [5.1, 3.5, 1.4, 0.3], [5.7, 3.8, 1.7, 0.3], [5.1, 3.8, 1.5, 0.3], [5.4, 3.4, 1.7, 0.2], [5.1, 3.7, 1.5, 0.4], [4.6, 3.6, 1. , 0.2], [5.1, 3.3, 1.7, 0.5], [4.8, 3.4, 1.9, 0.2], [5. , 3. , 1.6, 0.2], [5. , 3.4, 1.6, 0.4], [5.2, 3.5, 1.5, 0.2], [5.2, 3.4, 1.4, 0.2], [4.7, 3.2, 1.6, 0.2], [4.8, 3.1, 1.6, 0.2], [5.4, 3.4, 1.5, 0.4], [5.2, 4.1, 1.5, 0.1], [5.5, 4.2, 1.4, 0.2], [4.9, 3.1, 1.5, 0.2], [5. , 3.2, 1.2, 0.2], [5.5, 3.5, 1.3, 0.2], [4.9, 3.6, 1.4, 0.1], [4.4, 3. , 1.3, 0.2], [5.1, 3.4, 1.5, 0.2], [5. , 3.5, 1.3, 0.3], [4.5, 2.3, 1.3, 0.3], [4.4, 3.2, 1.3, 0.2], [5. , 3.5, 1.6, 0.6], [5.1, 3.8, 1.9, 0.4], [4.8, 3. , 1.4, 0.3], [5.1, 3.8, 1.6, 0.2], [4.6, 3.2, 1.4, 0.2], [5.3, 3.7, 1.5, 0.2], [5. , 3.3, 1.4, 0.2], [7. , 3.2, 4.7, 1.4], [6.4, 3.2, 4.5, 1.5], [6.9, 3.1, 4.9, 1.5], [5.5, 2.3, 4. , 1.3], [6.5, 2.8, 4.6, 1.5], [5.7, 2.8, 4.5, 1.3], [6.3, 3.3, 4.7, 1.6], [4.9, 2.4, 3.3, 1. ], [6.6, 2.9, 4.6, 1.3], [5.2, 2.7, 3.9, 1.4], [5. , 2. , 3.5, 1. ], [5.9, 3. , 4.2, 1.5], [6. , 2.2, 4. , 1. ], [6.1, 2.9, 4.7, 1.4], [5.6, 2.9, 3.6, 1.3], [6.7, 3.1, 4.4, 1.4], [5.6, 3. , 4.5, 1.5], [5.8, 2.7, 4.1, 1. ], [6.2, 2.2, 4.5, 1.5], [5.6, 2.5, 3.9, 1.1], [5.9, 3.2, 4.8, 1.8], [6.1, 2.8, 4. , 1.3], [6.3, 2.5, 4.9, 1.5], [6.1, 2.8, 4.7, 1.2], [6.4, 2.9, 4.3, 1.3], [6.6, 3. , 4.4, 1.4], [6.8, 2.8, 4.8, 1.4], [6.7, 3. , 5. , 1.7], [6. , 2.9, 4.5, 1.5], [5.7, 2.6, 3.5, 1. ], [5.5, 2.4, 3.8, 1.1], [5.5, 2.4, 3.7, 1. ], [5.8, 2.7, 3.9, 1.2], [6. , 2.7, 5.1, 1.6], [5.4, 3. , 4.5, 1.5], [6. , 3.4, 4.5, 1.6], [6.7, 3.1, 4.7, 1.5], [6.3, 2.3, 4.4, 1.3], [5.6, 3. , 4.1, 1.3], [5.5, 2.5, 4. , 1.3], [5.5, 2.6, 4.4, 1.2], [6.1, 3. , 4.6, 1.4], [5.8, 2.6, 4. , 1.2], [5. , 2.3, 3.3, 1. ], [5.6, 2.7, 4.2, 1.3], [5.7, 3. , 4.2, 1.2], [5.7, 2.9, 4.2, 1.3], [6.2, 2.9, 4.3, 1.3], [5.1, 2.5, 3. , 1.1], [5.7, 2.8, 4.1, 1.3], [6.3, 3.3, 6. , 2.5], [5.8, 2.7, 5.1, 1.9], [7.1, 3. , 5.9, 2.1], [6.3, 2.9, 5.6, 1.8], [6.5, 3. , 5.8, 2.2], [7.6, 3. , 6.6, 2.1], [4.9, 2.5, 4.5, 1.7], [7.3, 2.9, 6.3, 1.8], [6.7, 2.5, 5.8, 1.8], [7.2, 3.6, 6.1, 2.5], [6.5, 3.2, 5.1, 2. ], [6.4, 2.7, 5.3, 1.9], [6.8, 3. , 5.5, 2.1], [5.7, 2.5, 5. , 2. ], [5.8, 2.8, 5.1, 2.4], [6.4, 3.2, 5.3, 2.3], [6.5, 3. , 5.5, 1.8], [7.7, 3.8, 6.7, 2.2], [7.7, 2.6, 6.9, 2.3], [6. , 2.2, 5. , 1.5], [6.9, 3.2, 5.7, 2.3], [5.6, 2.8, 4.9, 2. ], [7.7, 2.8, 6.7, 2. ], [6.3, 2.7, 4.9, 1.8], [6.7, 3.3, 5.7, 2.1], [7.2, 3.2, 6. , 1.8], [6.2, 2.8, 4.8, 1.8], [6.1, 3. , 4.9, 1.8], [6.4, 2.8, 5.6, 2.1], [7.2, 3. , 5.8, 1.6], [7.4, 2.8, 6.1, 1.9], [7.9, 3.8, 6.4, 2. ], [6.4, 2.8, 5.6, 2.2], [6.3, 2.8, 5.1, 1.5], [6.1, 2.6, 5.6, 1.4], [7.7, 3. , 6.1, 2.3], [6.3, 3.4, 5.6, 2.4], [6.4, 3.1, 5.5, 1.8], [6. , 3. , 4.8, 1.8], [6.9, 3.1, 5.4, 2.1], [6.7, 3.1, 5.6, 2.4], [6.9, 3.1, 5.1, 2.3], [5.8, 2.7, 5.1, 1.9], [6.8, 3.2, 5.9, 2.3], [6.7, 3.3, 5.7, 2.5], [6.7, 3. , 5.2, 2.3], [6.3, 2.5, 5. , 1.9], [6.5, 3. , 5.2, 2. ], [6.2, 3.4, 5.4, 2.3], [5.9, 3. , 5.1, 1.8]])
iris_dict.data.shape
(150, 4)
iris_dict.target
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
iris_dict.target.shape
(150,)
iris_dict.target_names
array(['setosa', 'versicolor', 'virginica'], dtype='<U10')
iris_dict.feature_names
['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
iris_dict.DESCR
'.. _iris_dataset:\n\nIris plants dataset\n--------------------\n\n**Data Set Characteristics:**\n\n :Number of Instances: 150 (50 in each of three classes)\n :Number of Attributes: 4 numeric, predictive attributes and the class\n :Attribute Information:\n - sepal length in cm\n - sepal width in cm\n - petal length in cm\n - petal width in cm\n - class:\n - Iris-Setosa\n - Iris-Versicolour\n - Iris-Virginica\n \n :Summary Statistics:\n\n ============== ==== ==== ======= ===== ====================\n Min Max Mean SD Class Correlation\n ============== ==== ==== ======= ===== ====================\n sepal length: 4.3 7.9 5.84 0.83 0.7826\n sepal width: 2.0 4.4 3.05 0.43 -0.4194\n petal length: 1.0 6.9 3.76 1.76 0.9490 (high!)\n petal width: 0.1 2.5 1.20 0.76 0.9565 (high!)\n ============== ==== ==== ======= ===== ====================\n\n :Missing Attribute Values: None\n :Class Distribution: 33.3% for each of 3 classes.\n :Creator: R.A. Fisher\n :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n :Date: July, 1988\n\nThe famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\nfrom Fisher\'s paper. Note that it\'s the same as in R, but not as in the UCI\nMachine Learning Repository, which has two wrong data points.\n\nThis is perhaps the best known database to be found in the\npattern recognition literature. Fisher\'s paper is a classic in the field and\nis referenced frequently to this day. (See Duda & Hart, for example.) The\ndata set contains 3 classes of 50 instances each, where each class refers to a\ntype of iris plant. One class is linearly separable from the other 2; the\nlatter are NOT linearly separable from each other.\n\n.. topic:: References\n\n - Fisher, R.A. "The use of multiple measurements in taxonomic problems"\n Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to\n Mathematical Statistics" (John Wiley, NY, 1950).\n - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n (Q327.D83) John Wiley & Sons. ISBN 0-471-22361-1. See page 218.\n - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System\n Structure and Classification Rule for Recognition in Partially Exposed\n Environments". IEEE Transactions on Pattern Analysis and Machine\n Intelligence, Vol. PAMI-2, No. 1, 67-71.\n - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule". IEEE Transactions\n on Information Theory, May 1972, 431-433.\n - See also: 1988 MLC Proceedings, 54-64. Cheeseman et al"s AUTOCLASS II\n conceptual clustering system finds 3 classes in the data.\n - Many, many more ...'
A = iris_dict.data
A
array([[5.1, 3.5, 1.4, 0.2], [4.9, 3. , 1.4, 0.2], [4.7, 3.2, 1.3, 0.2], [4.6, 3.1, 1.5, 0.2], [5. , 3.6, 1.4, 0.2], [5.4, 3.9, 1.7, 0.4], [4.6, 3.4, 1.4, 0.3], [5. , 3.4, 1.5, 0.2], [4.4, 2.9, 1.4, 0.2], [4.9, 3.1, 1.5, 0.1], [5.4, 3.7, 1.5, 0.2], [4.8, 3.4, 1.6, 0.2], [4.8, 3. , 1.4, 0.1], [4.3, 3. , 1.1, 0.1], [5.8, 4. , 1.2, 0.2], [5.7, 4.4, 1.5, 0.4], [5.4, 3.9, 1.3, 0.4], [5.1, 3.5, 1.4, 0.3], [5.7, 3.8, 1.7, 0.3], [5.1, 3.8, 1.5, 0.3], [5.4, 3.4, 1.7, 0.2], [5.1, 3.7, 1.5, 0.4], [4.6, 3.6, 1. , 0.2], [5.1, 3.3, 1.7, 0.5], [4.8, 3.4, 1.9, 0.2], [5. , 3. , 1.6, 0.2], [5. , 3.4, 1.6, 0.4], [5.2, 3.5, 1.5, 0.2], [5.2, 3.4, 1.4, 0.2], [4.7, 3.2, 1.6, 0.2], [4.8, 3.1, 1.6, 0.2], [5.4, 3.4, 1.5, 0.4], [5.2, 4.1, 1.5, 0.1], [5.5, 4.2, 1.4, 0.2], [4.9, 3.1, 1.5, 0.2], [5. , 3.2, 1.2, 0.2], [5.5, 3.5, 1.3, 0.2], [4.9, 3.6, 1.4, 0.1], [4.4, 3. , 1.3, 0.2], [5.1, 3.4, 1.5, 0.2], [5. , 3.5, 1.3, 0.3], [4.5, 2.3, 1.3, 0.3], [4.4, 3.2, 1.3, 0.2], [5. , 3.5, 1.6, 0.6], [5.1, 3.8, 1.9, 0.4], [4.8, 3. , 1.4, 0.3], [5.1, 3.8, 1.6, 0.2], [4.6, 3.2, 1.4, 0.2], [5.3, 3.7, 1.5, 0.2], [5. , 3.3, 1.4, 0.2], [7. , 3.2, 4.7, 1.4], [6.4, 3.2, 4.5, 1.5], [6.9, 3.1, 4.9, 1.5], [5.5, 2.3, 4. , 1.3], [6.5, 2.8, 4.6, 1.5], [5.7, 2.8, 4.5, 1.3], [6.3, 3.3, 4.7, 1.6], [4.9, 2.4, 3.3, 1. ], [6.6, 2.9, 4.6, 1.3], [5.2, 2.7, 3.9, 1.4], [5. , 2. , 3.5, 1. ], [5.9, 3. , 4.2, 1.5], [6. , 2.2, 4. , 1. ], [6.1, 2.9, 4.7, 1.4], [5.6, 2.9, 3.6, 1.3], [6.7, 3.1, 4.4, 1.4], [5.6, 3. , 4.5, 1.5], [5.8, 2.7, 4.1, 1. ], [6.2, 2.2, 4.5, 1.5], [5.6, 2.5, 3.9, 1.1], [5.9, 3.2, 4.8, 1.8], [6.1, 2.8, 4. , 1.3], [6.3, 2.5, 4.9, 1.5], [6.1, 2.8, 4.7, 1.2], [6.4, 2.9, 4.3, 1.3], [6.6, 3. , 4.4, 1.4], [6.8, 2.8, 4.8, 1.4], [6.7, 3. , 5. , 1.7], [6. , 2.9, 4.5, 1.5], [5.7, 2.6, 3.5, 1. ], [5.5, 2.4, 3.8, 1.1], [5.5, 2.4, 3.7, 1. ], [5.8, 2.7, 3.9, 1.2], [6. , 2.7, 5.1, 1.6], [5.4, 3. , 4.5, 1.5], [6. , 3.4, 4.5, 1.6], [6.7, 3.1, 4.7, 1.5], [6.3, 2.3, 4.4, 1.3], [5.6, 3. , 4.1, 1.3], [5.5, 2.5, 4. , 1.3], [5.5, 2.6, 4.4, 1.2], [6.1, 3. , 4.6, 1.4], [5.8, 2.6, 4. , 1.2], [5. , 2.3, 3.3, 1. ], [5.6, 2.7, 4.2, 1.3], [5.7, 3. , 4.2, 1.2], [5.7, 2.9, 4.2, 1.3], [6.2, 2.9, 4.3, 1.3], [5.1, 2.5, 3. , 1.1], [5.7, 2.8, 4.1, 1.3], [6.3, 3.3, 6. , 2.5], [5.8, 2.7, 5.1, 1.9], [7.1, 3. , 5.9, 2.1], [6.3, 2.9, 5.6, 1.8], [6.5, 3. , 5.8, 2.2], [7.6, 3. , 6.6, 2.1], [4.9, 2.5, 4.5, 1.7], [7.3, 2.9, 6.3, 1.8], [6.7, 2.5, 5.8, 1.8], [7.2, 3.6, 6.1, 2.5], [6.5, 3.2, 5.1, 2. ], [6.4, 2.7, 5.3, 1.9], [6.8, 3. , 5.5, 2.1], [5.7, 2.5, 5. , 2. ], [5.8, 2.8, 5.1, 2.4], [6.4, 3.2, 5.3, 2.3], [6.5, 3. , 5.5, 1.8], [7.7, 3.8, 6.7, 2.2], [7.7, 2.6, 6.9, 2.3], [6. , 2.2, 5. , 1.5], [6.9, 3.2, 5.7, 2.3], [5.6, 2.8, 4.9, 2. ], [7.7, 2.8, 6.7, 2. ], [6.3, 2.7, 4.9, 1.8], [6.7, 3.3, 5.7, 2.1], [7.2, 3.2, 6. , 1.8], [6.2, 2.8, 4.8, 1.8], [6.1, 3. , 4.9, 1.8], [6.4, 2.8, 5.6, 2.1], [7.2, 3. , 5.8, 1.6], [7.4, 2.8, 6.1, 1.9], [7.9, 3.8, 6.4, 2. ], [6.4, 2.8, 5.6, 2.2], [6.3, 2.8, 5.1, 1.5], [6.1, 2.6, 5.6, 1.4], [7.7, 3. , 6.1, 2.3], [6.3, 3.4, 5.6, 2.4], [6.4, 3.1, 5.5, 1.8], [6. , 3. , 4.8, 1.8], [6.9, 3.1, 5.4, 2.1], [6.7, 3.1, 5.6, 2.4], [6.9, 3.1, 5.1, 2.3], [5.8, 2.7, 5.1, 1.9], [6.8, 3.2, 5.9, 2.3], [6.7, 3.3, 5.7, 2.5], [6.7, 3. , 5.2, 2.3], [6.3, 2.5, 5. , 1.9], [6.5, 3. , 5.2, 2. ], [6.2, 3.4, 5.4, 2.3], [5.9, 3. , 5.1, 1.8]])
Step 2: Standardize Dataset
Step 3: Conduct PCA
from sklearn.decomposition import PCA
'''
we are asking for two principal component because we want to plot this. Compress four features of the dataset
to just two features so that we can visualize our dataset using a 2-D plot
four column worth of information down to two principal components
'''
#pca = sklearn.decomposition.PCA()
#pca = sklearn.decomposition.PCA(n_components=0.9)
pca = sklearn.decomposition.PCA(n_components=2)
#pca.fit(A)
#PCs = pca.transform(A)
PCs = pca.fit_transform(A)
print("First ten rows:\n", PCs[0:9, :])
print("PCs.shape: ", PCs.shape)
print("A.shape: ", A.shape)
print("A.shape: ", A.shape)
print("Variance explained by first PC: ", (pca.explained_variance_ratio_*100)[0])
print("Variance explained by second PC: ",(pca.explained_variance_ratio_*100)[1])
First ten rows: [[-2.68 0.32] [-2.71 -0.18] [-2.89 -0.14] [-2.75 -0.32] [-2.73 0.33] [-2.28 0.74] [-2.82 -0.09] [-2.63 0.16] [-2.89 -0.58]] PCs.shape: (150, 2) A.shape: (150, 4) A.shape: (150, 4) Variance explained by first PC: 92.46187232017272 Variance explained by second PC: 5.306648311706782
Now we have two principal components instead for four different features for each flower.
Step 4: Data Visualization
plt.scatter(PCs[:, 0], PCs[:, 1]);
plt.scatter(PCs[:, 0], PCs[:, 1], c=iris_dict.target);
Unique
SolutionInfinite
SolutionNo
Solution
Option 1: If $A$ is a small rectangular matrix, you can make it square by multiplying it with its Transpose
$\hspace{3 cm}Ax = b$
$\hspace{3 cm}A^TAx = A^Tb$
$\hspace{3 cm}(A^TA)^{-1}(A^TA)x = (A^TA)^{-1}A^Tb$
$\hspace{3 cm}x = (A^TA)^{-1}A^Tb$
$\hspace{3 cm} A^\dagger = (A^TA)^{-1}A^T$
Option 2:
$\hspace{3 cm}Ax = b$
$\hspace{3 cm}USV^Tx = b$
$\hspace{3 cm}U^TUSV^Tx = U^Tb$
$\hspace{3 cm}SV^Tx = U^Tb$
$\hspace{3 cm}S^+SV^Tx = S^+U^Tb$
$\hspace{3 cm}V^Tx = S^+U^Tb$
$\hspace{3 cm}VV^Tx = VS^{-1}U^Tb$
$\hspace{3 cm}x = VS^{-1}U^Tb$
$\hspace{3 cm} A^\dagger = VS^+U^T $
Example: Given a non-square matrix $A = \begin{bmatrix} -1 & 2\\ 3 & -2\\5 & 7\end{bmatrix}$. See if you can calculate its inverse. If not why not? Then compute its Pseudoinverse:
pinv()
methodA = np.array([[-1, 2], [3, -2], [5, 7]])
#np.linalg.det(A) # Error: Cannot calculate determinant of a non-square matrix
#np.linalg.inv(A) # Error: Cannot calculate inverse of a non-square matrix
Option A: $\hspace{2 cm}A^\dagger = VS^+U^T $
U, s, VT = np.linalg.svd(A, full_matrices=False) # NumPy SVD method returns U, s, and transpose of V
V = VT.T
Splus = np.diag(1/s,0)
Aplus = V @ Splus @ U.T
Aplus
array([[-0.09, 0.18, 0.08], [ 0.08, -0.12, 0.09]])
Option B: $\hspace{2 cm}A^\dagger = (A^TA)^{-1}A^T $
Aplus = (np.linalg.inv(A.T @ A)) @ A.T
Aplus
array([[-0.09, 0.18, 0.08], [ 0.08, -0.12, 0.09]])
Option C: $\hspace{2 cm}A^\dagger = np.linalg.pinv() $
Aplus = np.linalg.pinv(A)
Aplus
array([[-0.09, 0.18, 0.08], [ 0.08, -0.12, 0.09]])
Example: Given a square matrix $A = \begin{bmatrix} 2 & 4\\ 6 & 12\end{bmatrix}$. See if you can calculate its inverse. If not why not? Then compute its Pseudoinverse:
pinv()
methodA = np.array([[2, 4], [6, 12]])
np.linalg.det(A)
#Ainv = np.linalg.inv(A) # Error: Cannot calculate inverse of a singular matrix having det(A) = 0
# We can get the second column by multiplying the first column by 2
0.0
Option A: $\hspace{2 cm}A^\dagger = VS^+U^T $
U, s, VT = np.linalg.svd(A, full_matrices=False) # NumPy SVD method returns U, s, and transpose of V
V = VT.T
Splus = np.diag(1/s,0)
Aplus = V @ Splus @ U.T
Aplus
array([[ 1.07e+15, -3.56e+14], [-5.34e+14, 1.78e+14]])
Option B: $\hspace{2 cm}A^\dagger = (A^TA)^{-1}A^T $
# This option does not work, since inverse of A.T does not exist
#Aplus = (np.linalg.inv(A.T @ A)) @ A.T
Option C: $\hspace{2 cm}A^\dagger = np.linalg.pinv() $
Aplus = np.linalg.pinv(A)
Aplus
array([[0.01, 0.03], [0.02, 0.06]])
Example:
GPA
and study hours
is positive or negative.study hours
has on the independent variable GPA
daily study hours
and acquired GPA
):$$(1,\hspace{.2 cm}1.4), \hspace{.5 cm}(2,\hspace{.2 cm}1.6),\hspace{.5 cm}(3,\hspace{.2 cm}2.5),\hspace{.5 cm}(4,\hspace{.2 cm}2.6), \hspace{.5 cm}(5,\hspace{.2 cm}3.5),\hspace{.5 cm}(6,\hspace{.2 cm}3.7),\hspace{.5 cm}(7,\hspace{.2 cm}4.0) $$
Step 1:
Plot the seven observations using a scatter chart:
sh = np.array([1, 2, 3, 4, 5, 6, 7.]) # study hours
gpa = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0]) # gpa
fig, ax = plt.subplots()
plt.title("Study Hours vs GPA of Students")
plt.xlabel("Study Hours")
plt.ylabel("GPA")
ax.scatter(sh,gpa)
plt.grid(True)
$$(1,1.4), \hspace{.5 cm} (2,1.6), \hspace{.5 cm}(3,2.5), \hspace{.5 cm}(4,2.6), \hspace{.5 cm}(5,3.5), \hspace{.5 cm}(6,3.7), \hspace{.5 cm}(7,4.0) $$$$y=c+mx$$
Step 2:
Write down seven linear equations, one for each observation/point:
$$y = \beta_0 + \beta_1x + \epsilon$$
Where,
Together, $\beta_0$ and $\beta_1$ are called the model coefficients. Our objective is to find the parameters $\beta_0$ and $\beta_1$ that minimize $\epsilon$ across all the available data points.
Mention the seven equations for the seven points, having two unknowns: y-intercept ($\beta_0$) and slope ($\beta_1$)
- Step 3: Create matrix equation from above seven equations: $Ax=b$
Inconsistent Overdetermined System
of seven linear equations having two unknowns in matrix form.Since the y-intercept is constant, so we have set it equal to 1
across the board
So the above equations can be written as:
ones = np.ones((sh.size,1))
ones
array([[1.], [1.], [1.], [1.], [1.], [1.], [1.]])
sh = np.array([1, 2, 3, 4, 5, 6, 7.]) # is a row vector having 7 elements
sh.shape= (sh.size, 1) # make sh a matrix of size 7x1
sh
array([[1.], [2.], [3.], [4.], [5.], [6.], [7.]])
A = np.hstack((ones, sh)) # stack arrays horizontally or column-wise
A
array([[1., 1.], [1., 2.], [1., 3.], [1., 4.], [1., 5.], [1., 6.], [1., 7.]])
b = gpa
b
array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4. ])
Step 4:
Solve the system of linear equations using Moore Penrose Pseudoinverse:
Option A: $\hspace{.5 cm}A^\dagger = VS^+U^T $
U, s, VT = np.linalg.svd(A, full_matrices=False) # NumPy SVD method returns U, s, and transpose of V
V = VT.T
Splus = np.diag(1/s,0)
Aplus = V @ Splus @ U.T
Aplus
array([[ 0.57, 0.43, 0.29, 0.14, 0. , -0.14, -0.29], [-0.11, -0.07, -0.04, 0. , 0.04, 0.07, 0.11]])
Option B: $\hspace{.5 cm}A^\dagger = (A^TA)^{-1}A^T $
Aplus = (np.linalg.inv(A.T @ A)) @ A.T
Aplus
array([[ 0.57, 0.43, 0.29, 0.14, -0. , -0.14, -0.29], [-0.11, -0.07, -0.04, 0. , 0.04, 0.07, 0.11]])
Option C: $\hspace{.5 cm}A^\dagger = np.linalg.pinv() $
Aplus = np.linalg.pinv(A)
Aplus
array([[ 0.57, 0.43, 0.29, 0.14, 0. , -0.14, -0.29], [-0.11, -0.07, -0.04, 0. , 0.04, 0.07, 0.11]])
Solve$\hspace{1 cm}x \approx A^\dagger b \hspace{2 cm}$
result = Aplus @ b
print("Y-intercept (𝛽0): ", result[0])
print("Slope (𝛽1): ", result[1])
Y-intercept (𝛽0): 0.9000000000000004 Slope (𝛽1): 0.4642857142857145
$$y = \beta_0 + \beta_1 x $$$$gpa = 0.9 + 0.464*sh $$
Step 5:
Draw the regression line using the computed slope and y-intercept, which is the best fit line
sh = np.array([1, 2, 3, 4, 5, 6, 7])
gpa = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
fig, ax = plt.subplots()
plt.title("Study Hours vs GPA of Students")
plt.xlabel("Study Hours")
plt.ylabel("GPA")
ax.scatter(sh,gpa);
x2 = np.linspace(0,8, 10)
b0 = 0.9
b1 = 0.464
y2 = b0 + b1*x2
ax.plot(x2, y2)
plt.grid(True)
Step 6:
Perform prediction
Step 7:
Finally calculate the least square errors to determine how well the regression line actually fits the data:
x = np.array([1, 2, 3, 4, 5, 6, 7])
y = np.array([1.4, 1.6, 2.5, 2.6, 3.5, 3.7, 4.0])
yhat = b0 + b1*x
sse = sum((yhat - y)**2)
ssr = sum((yhat- np.mean(y))**2)
sst = ssr + sse
r2 = ssr/sst
r2
0.9645681098359337
This tells us that 96.4% of the variation in GPA can be explained by the number of hours studied.