Pairs Trading Enhancements¶

By Sam Galita and James Zhang | Smith Investment Fund | Spring 2023

Introduction¶

Pairs trading is a type of mean reversion trading strategy first introduced by technical analyst researchers at Morgan Stanley in the mid-1980s. On a high level, pairs trading involves matching a long position with a short position in two highly cointegrated stocks. A statistical tool often used in timeseries analysis, cointegration measures the degree to which two time series' means trend together. In pairs trading, we apply cointegration on the difference of prices between two stocks, otherwise known as the spread, to detect assets that tend to trend similarly.

One of the main advantages of this strategy is market neutrality. Since traders are trading the spread between two assets and not individual assets, a pairs trader can find statistical arbitrage opportunities regardless of market direction. In other words, we should not expect better or worse performance during economic downswings. Furthermore, traders remain delta neutral when using this strategy, making pairs trading relatively low risk, and theoretically less volatile than a market-following strategy. As a result of this risk advantage, many hedge funds have historically leveraged pairs trading, despite it's low relative returns. However, this strategy is widely known in the modern market and would likely not provide high returns in a real market setting. Despite this, pairs trading sets the groundwork for further research into statistical arbitrage and remains an interesting topic for research.

After two cointegrated stocks are identified, if a cointegration discrepancy occurs - when the spread between two securities diverges from their historical rolling mean and beyond some threshold - the trader invests a dollar-matched long position in the underperforming security and short position in the overperforming security. If the spread between these assets does indeed reconverge, which is expected due to the identified relationship between pairs, traders will make a profit.

One of the keys to successful pairs trading is being able to efficiently detect pairs. In a notebook published a few months ago, Abhi designed and applied a basic pais trading strategy to the equity and crypto markets. In our notebook, we propose enhancements to last semester's pairs trading work done by Abhi. First, we will apply Principal Component Analysis (PCA) and clustering to the pricing timeseries, exploring a variety of clustering methods in the process. Within each cluster, we will then perform cointegration tests to identify pairs. Because of the use of clustering and PCA, we expect to find a reduction in needed computation time, thus allowing us to use a larger universe size when backtesting, which theoretically will allow us to compute more profitable pairs.

Furthermore, we will also explore a variety of parameter optimizations for the strategy, specifically with respect to lookback, reset frequency of pair selection, and zscore cutoffs. Lastly, we will also explore the idea of using the other validation techniques as an additional criteria for selecting profitable pairs.

Required Imports¶

Below is some more information about some of the more fundamental libraries if you want to learn more.

• NumPy
• Pandas
• Matplotlib
• Sklearn
• Statsmodels

We initialize multiple backtesters from the SIF Infrastructure that will allow us to test our pairs trading strategies in sample from $2000-2015$ and out of sample from $2016-2020$. Note that we can save them locally to save time each time we want to test our alphas.

Previous Work and Novelties¶

SIF Researcher Abhinav Modugula worked on pairs trading for much of last semester. Our work in this notebook builds on top of his initial research. You can read his work here: https://www.smithinvestmentfund.com/blog/crypto-pairs-trading. His initial work on pairs trading should be read before the rest of this notebook.

Selecting Pairs and Weights¶

As a baseline metric, we present brute force pairs trading alphas without dimensionality reduction, machine learning enhancements, additional trading criteria, or parameter optimization. These alphas are based off of Abhi's original alphas, with notable fixes to certain bugs.

For both of the following alphas, we test for cointegration between the price time series of two securities using the Statsmodels implementation of the Engle-Granger Test, which outputs a pvalue that can be used to either accept or reject the null hypothesis that the two secrities are cointegrated. The first alpha takes all of the statistically significant pairs between the $50$ securities in the universe and randomly selects $10$ to trade off of. Extracting random pairs out of all the generated pairs can possibly be improved, however, by choosing the lowest pvalue pairs. The second alpha implements this idea.

See the implementation below of the Random pairs trading strategy explained in the introduction. The Lowest Pvalue strategy - and all strategies testing in this blog post - can be found in the alphas.py file.

Following all tests, we also created the alphas_v2.py file, which leverages inheritance to maximize alpha customizability. Note that while all alphas in this blog post can be implemented with the PairsTrader class in alphas_v2.py, we decided to include alphas.py as they include the alphas that we actually used in this notebook.

Parameters¶

For this inital strategy, we rechoose pairs every $90$ days and select $10$ pairs to trade on. The universe size is $500$, and we enter trades when the magnitude of the zscore of the difference of prices is above $1.0$ and exit when it is below $0.5$.

Brute force Random Alphas¶

Method and Approach¶

Before we build the alphas for our improvements, we will apply the alterations that we wish to implement to a dataset of pricing data from the SIF database. This will help us build the framework for implementing our enhanced strategy in the alpha environment.

Data Collection and Preprocessing¶

We use Beautiful Soup to scrape Wikipedia for a list of tickers, company names, and industries for companies in the S&P500, a stock market index tracking stock performance of 500 large, public companies traded in America.

Now use the SIF Infrastructure to get close price data for all available tickers from January 1, 2015 to December 31, 2015.

Optimizing Pair Selection¶

• PCA

• OPTICS, KMeans, and DBSCAN (Clustering)

Principal Component Analysis (PCA)¶

Principal component analysis is a mathematical procedure used to reduce the number of features in a dataset into a smaller number of uncorrelated variables known as "principal components," while preserving as much information as possible within the original dataset. This process takes us one step closer to easing computation and efficiently finding profitable pairs, as it reduces the amount of time it takes to perform clustering while retaining most of the information contained in the original dataset.

Primarily, PCA asks the question, how can we position all of these points (data) on new axes such that the maximum amount of information is retained? In other words, what direction axes can we choose to retain the the most information? Note that projecting points onto a pre-defined axis like the x-axis or y-axis (in a 2D example) is wasteful, since we lose all information from other potential axes. Projecting of a point $x$ onto a unit vector $u$, we obtain that the magnitude of the new point $x'$ is the inner product $x' = x^Tu$. Now suppose that the inner product squared $(x^Tu)^2$ represents the amount of information preserved after projection.

Back to our problem, suppose $X$ is the set of all points and $x_i \in X$. Maximizing the preserved information means we want to obtain $\max \sum (x_i^Tu)^2$, and we can use the Lagrange Multipliers Method to find this maximum. Suppose we want to maximize the function $f(u) = \sum(x_i^Tu)^2$ given the constraint $g(u) = u^Tu = 1$. Note that this constraint forces $u$ to be a unit vector because we are trying to find the direction of the axis that preserves the most information. Therefore, the magnitude is irrelevant. $$f(u) = \sum(x_i^Tu)^2$$ $$f(u) = \sum(x_i^Tu)(x_i^Tu)$$ $$f(u) = u^T\sum(x_i)(x_i^Tu)$$ $$f(u) = u^TC u \implies C = \frac{1}{n}\sum_i x_ix_i^T$$ where $C$ is the covariance matrix of the Function. Now let us use Lagrange Multipliers. $$X - \lambda I = 0$$ $$u^TCu - \lambda(u^Tu-1)$$ Now take the gradient with respect to $u$. $$2Cu - 2\lambda u \implies Cu = \lambda u$$

$u$ is the eigenvector and $\lambda$ is the eigenvalue. It becomes evident that the first principal component is the eigenvector of the covariance matrix with the largest eigenvalue. As for the second component, we want to find a direction that contains information that the first line doesn't. Thus, the second component should be orthogonal to the first, and since we are maximizing the same function with the same constraint, we find that the second component is simply the eigenvector with the second largest component. Generally, $N$ directions means $N$ covariance matrix eigenvectors and eigenvectors with each corresponding eigenvalue determining how important that component is.

For those less familiar with linear algebra, but have a basic understanding of its uses, one can understand this to be similar to performing a change of bases on our data, with the intention of selecting bases that allow us to use less datapoints to represent the same information. As an example, if we have two days that for each stock in our universe, have nearly identical/scaled returns as the other day, we can use a single axis to record that information, reducing the dimensionality of our data (there is no point in keeping 2 axes if that information can be stored on a single axis)

Clustering Methods¶

For our use purposes, clustering securities based on their percent returns gives us an insight into which companies are affected by similar factors ie. which securites are likely to be cointegrated with one another. This is critical for reducing the computation time of our strategy, as rather than computing, for example, $\binom{500}{2} = 124,750$ cointegration tests for a universe of size $500$ stocks, if we can instead identify $20$ clusters, each with $25$ stocks, we reduce the number of cointegration tests nesseccary to $20 \times \binom{25}{2} = 3000$, which is roughly $2.4$% of the brute forcee computation. Note that this can be optimized even further with algorithms that identify outliers. We will investigate three clustering algorithms - K-Means, DBSCAN, and OPTICS, to see which is most useful for this implementation.

K-Means¶

K-Means clustering aims to partition data into $k$ clusters such that Euclidean distances within a cluster are minimized and distances between clusters are maximized. Traditionally, the elbow method is employed to determine the optimal number of $k$ clusters, where we determine the optimal number for $k$ based on the incremental reduction in distortion attained from further increasing k.

The K-Means algorithm is as follows:

Suppose we have $n$ input data points $x_1, \cdots, x_n$ and our predetermined value of k = 11 industries.

1. Randomly select k of the $n$ data points, and let them be the k centroids of the k clusters.
2. For each point in the data, determine the Euclidean distance from the point to the k centroids.

Note that the Euclidean Distance between two points $a, b \in \mathbb{R}^n$ is defined as the following: $$\text{distance}(\textbf{a}, \textbf{b}) = \sqrt{(a_1-b_1)^2 + \cdots + (a_n-b_n)^2}$$ 3. Let each cluster centroid be denoted as $c_i \in C$ where $C$ is the set of all centroids, then each data point $x_i$ is assigned to the closest cluster defined by the above Euclidean distance, or $$\text{arg min}_{c_i\in C} \ \text{distance}(c_i, x)$$ 4. Define the new centroid of the current cluster by taking the average of the points in the cluster. Let $S_i$ be the ith cluster. Then the new centroid is defined as $$c_i = \dfrac{1}{|S_i|}\sum_{x_i \in S_i} x_i$$ such that $S_i$ is the size of set $S_i$. 5. Continue to iterate through the input data points until the k centroids don't change.

Elbow Method vs. Industries¶

While the elbow method is widely used in research, we realized there were practical limitations when trying to determine when additional distortion is small enough to warrant selecting a value for $k$. Thus, we decided to instead use the number of industries $(11)$ as our $k$.

DBSCAN (Density Based Spatial Clustering of Applications)¶

DBSCAN is a density-based clustering algorithm, which is best suited to deal with arbitrary shapes or outliers. Given that companies are unique entities, their returns will all be affected by factors differently, so we expect there to be many outliers. DBSCAN and OPTICS are very similar. Let us define parameters and subsequent definitions to algorithmically understand both methods.

1. eps ($\epsilon$): the distance that specifies neighbors

2. minPts: minimum number of data points to define a cluster. We will set this value to 2 because all we need is 2 stocks in a cluster to potentially have a cointegrated and therefore highly profitable pair.

These two parameters allow us to define the following 3 definitions.

1. core point: a point x such that there at least minPts number of points (inclusive of the point itself) in the closed ball $B_\epsilon(x)$.
2. border point: a point x such that it is reachable from a core point and there are less than minPts number of points within $B_\epsilon(x)$.
3. outlier: a point such that it is not a core point and it is not reachable from any core points.

The DBSCAN algorithm is as follows:

1. minPts and eps are predetermined. Suppose there exist $n$ data points $x_1, \cdots, x_n \in X$. Select a starting point $x_i$ at random.
2. Let $S$ be the set of all points in $B_\epsilon (x_i)$. If $|S| >= \ $ minPts, then $x_i$ is marked as a core point and cluster formation starts. Otherwise, the point is marked as noise. All other points $x_j \in B_{\epsilon}(x_i)$ become a part of the cluster.
3. If for the other $x_j$, if there exists at least minPts in $B_\epsilon(x_j)$, then $x_j$ is also determined to be core points, then all $x_k \in B_\epsilon(x_j)$ are also added to the cluster and evaluated if they are core points.
4. Once that cluster is formed, choose another random point that has not been visited yet and follow the same procedure. Note that points that are marked as noise can be revisited.
5. The algorithm is done when all points classified as part of a cluster or a noise point.

OPTICS (Ordering Points to Identify Cluster Structure)¶

We observe firsthand that the Elbow Technique to determine optimal eps is unreliable and cumbersome. Such a small difference in the value of eps can drastically transform final results. This required level of precision for eps limits DBSCAN predominantly to the detection of clusters of similar density. OPTICS solves this issue

Testing KMeans vs. DBSCAN vs. OPTICS in Alphas¶

Clustering Conclusion¶

Of the three methods, we select OPTICS to use in our final alpha moving forward. Although the OPTICS and K-Means clustering methods give similar performance, OPTICS runs much quicker, and K-Means is further limited by the need to manually select $k$.

Thus, we will continue to build on our alpha using OPTICS as our clustering method for the rest of the notebook.

Critically, notice that the performance of these alphas is similar/better than the original brute force lowest pvalue alpha. This is extremely important as it means we are losing minimal value by using clustering to reduce the number of pairs of securities we check cointegration between, while gaining massive computational advantage.

Time Complexity Analysis - Clustering vs. Original¶

The objective of principal component analysis and clustering methods is to identify cointegrated pairs faster and more efficiently. To test the success of this new implementation, we conduct a time complexity analysis of a bare bones pairs trading strategy versus an enhanced strategy when both are tested in a $500$ stock universe.

To accomplish this, We create a shorter $6$ month backtester, as running the original pairs trading alpha on a $500$ stock universe is very slow.

Time Complexity Results¶

As we can see, the pairs trading alpha with clustering runs over $350$ times faster than the original alpha, even though it gives similar performance. We ran the comparison a second time to ensure the results were accurate, as it was already surprising to see such a large performance change.

We consider this implementation successful at reducing our computational speeds to allow us to adequately test and use pairs trading alphas with larger universes.

Pairs Trading Parameters¶

There are many parameters that we can tune for pairs trading to build the optimal strategy. In this portion of the notebook, we will explore them.

Thresholds¶

We define opening and closing of trade thresholds for our trades by cutoff values of the z-score of the spread between the price timeseries of the two securities. Z-score is defined as $$z = \dfrac{x-\mu}{\sigma}$$ where $x$ is some value, $\mu$ is the mean, and $\sigma$ is the standard deviation, and this value represents the number of standard deviations away from the mean our current value is.

Open Position Criteria¶

A trade will be initialized (ie. we take a long position for the underperforming security and a short position on the overperforming security) when the absolute value of the zscore of the spread is greater than our open threshold, currently $1.0$.

Close Position Criteria¶

The trade will be closed when the asbolute value of the zscore of the spread becomes less than our close (take profit) threshold, which is currently $0.5$. If this threshold is reached on an open position, it means the difference between close prices have revered back toward their mean and a profit should be made.

Stop Loss Criteria¶

In the circumstance where two securities seem to continue to diverge beyond what we consider reasonable, we propose a mechanism for cutting our losses by exiting the trade. If the absolute of the zscore of the spread ever becomes greater than our stop loss threshold, we reevaluate the cointegration levels again using the Engle-Granger test. If the resulting pvalue is greater than $0.05$ (no longer cointegrated), we choose to suspect that the pair is no longer cointegrated, and we will exit the trade and take our losses before the securities further diverge and result in even greater losses.

See the eda_traced.py file (linked) for the results. Now we will illustrate the trade status in a visual graph.

Information about the OPTICS Pairs¶

• Number of pairs trades opened.
• Average length of trade opened.
• Generate a graph of the zscore distributions.