Dimensionality Reduction-Master Template¶

In this notebook, we will learn about fundamental dimensionality reduction techniques that will help us to summarize the information content of a dataset by transforming it onto a new feature subspace of lower dimensionality than the original one.

The details of the dimensionality Reduction Methodology from scikit Learn can be found under: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition

Content¶

• 1. Introduction
• 2. Getting Started - Load Libraries and Dataset
  • 2.1. Load Libraries
  • 2.2. Load Dataset
• 3. Exploratory Data Analysis
  • 3.1 Descriptive Statistics
  • 3.2. Data Visualisation
• 4. Data Preparation
  • 4.1 Data Cleaning
  • 4.3.Data Transformation
• 5.Evaluate Algorithms and Models
  • 5.1. Principal Component Analysis
  • 5.2. Singular Value Decomposition-SVD
  • 5.3. Kernel Principal Component Analysis
  • 5.4. t-SNE

1. Introduction¶

We will look at the following models and the related concepts

1. Principal Component Analysis (PCA)
2. Kernel PCA (KPCA)
3. t-distributed Stochastic Neighbor Embedding (t-SNE)

2. Getting Started- Loading the data and python packages¶

2.1. Loading the python packages¶

2.2. Loading the Data¶

3. Exploratory Data Analysis¶

3.1. Descriptive Statistics¶

3.2. Data Visualization¶

Taking a look at the correlation. More detailed look at the data will be performed after implementing the Dimensionality Reduction Models.

4. Data Preparation¶

4.1. Data Cleaning¶

Check for the NAs in the rows, either drop them or fill them with the mean of the column

Getting rid of the columns with more than 30% missing values.

Given that there are null values drop the rown contianing the null values.

4.2. Data Transformation¶

All the variables should be on the same scale before applying PCA, otherwise a feature with large values will dominate the result. Below I use StandardScaler in scikit-learn to standardize the dataset’s features onto unit scale (mean = 0 and variance = 1).

5. Evaluate Algorithms and Models¶

We will look at the following Models:

1. Principal Component Analysis (PCA)
2. Kernel PCA (KPCA)
3. t-distributed Stochastic Neighbor Embedding (t-SNE)

5.1. Principal Component Analysis (PCA)¶

The idea of principal component analysis (PCA) is to reduce the dimensionality of a dataset consisting of a large number of related variables, while retaining as much variance in the data as possible. PCA finds a set of new variables that the original variables are just their linear combinations. The new variables are called Principal Components (PCs). These principal components are orthogonal: In a 3-D case, the principal components are perpendicular to each other. X can not be represented by Y or Y cannot be presented by Z.

The cumulative plot shows a typical 'elbow' pattern that can help identify a suitable target dimensionality because it indicates that additional components add less explanatory value.

We find that the most important factor explains around 30% of the daily return variation. The dominant factor is usually interpreted as ‘the market’, whereas the remaining factors can be interpreted as industry or style factors in line with our discussion in chapters 5 and 7, depending on the results of closer inspection (see next example).

The plot on the right shows the cumulative explained variance and indicates that around 10 factors explain 60% of the returns of this large cross-section of stocks.

First Principal Component /Eigenvector¶

Eigenvalues¶

Explained Variance¶

Factor Loading¶

Eigenvectors are unit-scaled loadings; and they are the coefficients (the cosines) of orthogonal transformation (rotation) of variables into principal components or back. Therefore it is easy to compute the components' values (not standardized) with them. Besides that their usage is limited. Eigenvector value squared has the meaning of the contribution of a variable into a pr. component; if it is high (close to 1) the component is well defined by that variable alone.

Here orthonormal eigen vectors (i.e. the term Orthonormal Eigenvectors ) provides a direction and the term Square root of (Absolute Eigen values) provide the value.

Although eigenvectors and loadings are simply two different ways to normalize coordinates of the same points representing columns (variables) of the data on a biplot, it is not a good idea to mix the two terms.

Factor loadings of the First two components¶

PCA to Reduce Dimension¶

5.2. Singular Value Decomposition (SVD)¶

This transformer performs linear dimensionality reduction by means of truncated singular value decomposition (SVD). Contrary to PCA, this estimator does not center the data before computing the singular value decomposition.

We are using the TruncatedSVD method in the scikit-learn package (Truncated-SVD is a quicker calculation, and using scikit-learn is more convenient and consistent with our usage elsewhere) to transform the full dataset into a representation using the top 300 components, thus preserving variance in the data but using fewer dimensions/features to do so. This has a similar effect to Principal Component Analysis (PCA) where we represent the original data using an orthogonal set of axes rotated and aligned to the variance in the dataset.

5.3. Kernel PCA (KPCA)¶

PCA applies linear transformation, which is just its limitation. Kernel PCA extends PCA to non-linearity. It first maps the original data to some nonlinear feature space (usually higher dimension), then applies PCA to extract the principal components in that space. But if all the dots are projected onto a 3D space, the result becomes linearly separable! We then apply PCA to separate the components.

5.4. t-distributed Stochastic Neighbor Embedding (t-SNE)¶

t-SNE models the similarities among points. How does it define similarities? First, it is defined by the Euclidean distance between point Xi and Xj. Second, it is defined as the conditional probability that “the similarity of data point i to point j is the conditional probability p that point i would pick data j as its neighbor if other neighbors were picked according to their probabilities under a Gaussian distribution.” In the following conditional expression, if point j is closer to point i than other points, it has a higher probability (notice the negative sign) to be chosen.

Pairs-plots are a simple representation using a set of 2D scatterplots, plotting each component against another component, and coloring the datapoints according to their classification