Predicting Customer Churn with Machine Learning 🏦💰¶

Table of Contents

• Introduction
• Objective
• Libraries
• Default Parameters
• Functions
• A Quick Look at our Data
• Creating a Test Set
• Exploratory Data Analysis
  • Target Variable: Exited
  • Continuous Variables
  • Categorical Variables
• Data Pre-processing
  • Feature Selection
    
  • Encoding Categorical Features
    
  • Scaling
    
  • Addressing Class Imbalance
    
  • Pre-processing the Test Set
• Building Machine Learning Models
  • Baseline Models
    
  • Model Tuning
    
    • Logistic Regression
      
    • Support Vector Classifier
      
    • Random Forest
      
    • Random Forest With Random Undersampling
      
    • Gradient Boosting Classifier
      
    • XGBoost Classifier
      
    • LGBMClassifier
      
    • Ensemble Learning
      
• Results
  • Feature Importance
    
  • Performance Comparison
    
• Evaluating the Test Set
• Calculating Profit
• References
• Conclusions

Introduction¶

Customer churn (also known as customer attrition) occurs when a customer stops using a company's products or services.

Customer churn affects profitability, especially in industries where revenues are heavily dependent on subscriptions (e.g. banks, telephone and internet service providers, pay-TV companies, insurance firms, etc.). It is estimated that acquiring a new customer can cost up to five times more than retaining an existing one.

Therefore, customer churn analysis is essential as it can help a business:

• Identify problems in its services (e.g. poor quality product/service, poor customer support, wrong target audience, etc.), and
• Make correct strategic decisions that would lead to higher customer satisfaction and consequently higher customer retention.

If you want to know more about this topic, please refer to Refs. [10-13].

Objective¶

This notebook aims to understand and predict customer churn for a bank. Specifically, we will initially perform Exploratory Data Analysis (EDA) to identify and visualise the factors contributing to customer churn. This analysis will later help us pre-process our data and build Machine Learning (ML) models to predict whether a customer will churn or not.

This problem is a typical binary classification task. The predictive power of our models will be assessed based on the area under the ROC curve (AUC). The same metric will be used during hyperparameter tuning to find the optimal parameters for the ML models.

Skills: Exploratory Data Analysis, Data Visualisation, Data Pre-processing (Feature Selection, Encoding Categorical Features, Feature Scaling), Addressing Class Imbalance, Model Tuning, Ensemble Learning.

Models Used: Logistic Regression, Support Vector Machines, Random Forests, Gradient Boosting, XGBoost, and Light Gradient Boosting Machine.

Libraries¶

Default Parameters¶

Functions¶

plot_continuous()¶

plot_categorical()¶

plot_conf_mx()¶

clf_performance()¶

A Quick Look at our Data¶

We start by importing the dataset as a Pandas DataFrame. We can also take a look at the top five rows using the head() method:

Our DataFrame has 14 features and 10K instances. The last feature, 'Exited', is the target variable and indicates whether the customer has churned (0 = No, 1 = Yes). The meaning of the rest of the features can be easily inferred from their name.

Features 'RowNumber', 'CustomerId', and 'Surname' are specific to each customer and can be dropped:

The info() method can give us valuable information such as the number of non-null values and the type of each feature:

Thankfully, there are no missing values in our DataFrame. The describe() method gives us a statistical summary of the numerical features:

The most important things to note are:

• The age of customers ranges from 18 to 92, with a mean value approximately equal to 40.
• The mean (and median) tenure is five years, so the majority of customers are loyal (tenure > 3).
• Approximately 50% of customers are active.

EDA will help us understand our dataset better. However, before we look at the data further, we need to create a test set, put it aside, and use it only to evaluate our ML models. This practice protects our models from data snooping bias (you can read more on page 51 of Ref. [1]) and ensures that evaluation will be performed using unseen data.

Creating a Test Set¶

We will split our dataset into a train and a test set using scikit-learn's StratifiedShuffleSplit() function, which implements stratified sampling. In this method, the population is divided into homogeneous subgroups called strata, and the right number of instances are sampled from each stratum to guaranτee that the test set is representative of the overall population (Ref. [1]).

Exploratory Data Analysis¶

Target Variable: Exited¶

As we mentioned earlier, the target variable is already encoded and can take two possible values:

• Zero (0) for a customer that has not churned, and
• One (1) for a customer that has churned.

The bank kept 80% of its clientele.

Notice that our dataset is skewed/imbalanced since the number of instances in the 'Retained' class outnumbers the number of instances in the 'Churned' class by a factor of 4. Therefore, we need to factor in class imbalance before building any models.

Different visualisation techniques apply to different types of variables, so it is helpful to differentiate between continuous and categorical variables and look at them separately.

Continuous Variables¶

By calling the hist() method, we can plot a histogram for each of the four continuous numeric features:

• 'Age' is slightly tail-heavy, i.e. it extends more further to the right of the median than to the left,
• Most values for 'CreditScore' are above 600,
• If we ignore the first bin, 'Balance' follows a relatively normal distribution, and
• The distribution of 'EstimatedSalary' is more or less uniform and provides little information.

Looking for Correlations¶

We can compute the standard correlation coefficient between every pair of (continuous) features using the pandas' corr() method and plot it as a matrix:

There is no significant intercorrelation between our features, so we do not have to worry about multicollinearity.

Let's look at these features in greater detail.

Age¶

Interestingly, there is a clear difference between age groups, with older customers more likely to churn. This observation could potentially indicate that preferences change with age, and the bank hasn't adapted its strategy to meet the requirements of older customers.

Credit Score¶

There is no significant difference between retained and churned customers regarding their credit scores.

Balance¶

Again, the two distributions are pretty similar. There is a significant percentage of non-churned customers with a low account balance.

Estimated Salary¶

Both churned and retained customers display a similar uniform distribution in their salaries. Consequently, we can conclude that salary does not have a significant effect on the likelihood to churn.

Categorical Variables¶

Let's plot a seaborn.countplot for each categorical feature:

Important points:

• The bank has customers in three countries (France, Spain, and Germany). Most customers are in France.
• There are more male customers than females,
• Only a small percentage leaves within the first year. The count of customers in tenure years between 1 and 9 is almost the same,
• Most of the customers have purchased 1 or 2 products, while a small portion has purchased 3 and 4,
• A significant majority of customers has a credit card, and
• Almost 50% of customers are not active.

Again, we will look at these features in greater detail.

Geography¶

Customers in Germany are more likely to churn than customers in the other two countries (the churn rate is almost double compared to Spain and France). Many reasons could explain this finding, such as higher competition or different preferences for German customers.

Gender¶

Female customers are more likely to churn.

Tenure¶

The number of years (tenure) does not affect the churn rate.

Number of Products¶

Interestingly, having three or four products can significantly increase the likelihood of churn. I am not sure how to interpret this result. It could potentially mean that the bank cannot properly support customers with more products, thus increasing customer dissatisfaction.

Card Holders¶

Having a credit card does not seem to affect the churn rate.

Active Members¶

It is not a surprise that inactive customers are more likely to churn. A significant portion of the clientele is inactive; therefore, the bank will benefit from changing its policy so that more customers become active.

Data Pre-processing¶

Data pre-processing is the process of converting raw data into a well-readable format that is suitable for building and training ML models.

Let's complete this process step-by-step.

Feature Selection¶

We have already performed feature selection by dropping columns 'RowNumber', 'CustomerId', and 'Surname' at the beginning of our notebook. EDA revealed several more features that can be dropped as they do not provide any value in predicting our target variable:

• 'EstimatedSalary' displays a uniform distribution for both types of customers and can be dropped.
• The categories in 'Tenure' and 'HasCrCard' have a similar churn rate and are deemed redundant. This observation can be confirmed from a chi-square test (Ref. [2]):

'Tenure' and 'HasCrCard' have a small chi-square and a p-value greater than 0.05 (the standard cut-off value), confirming our initial hypothesis that these two features do not convey any useful information.

We can use the drop() method to remove these three features from the train set:

Encoding Categorical Features¶

ML algorithms usually require all input (and output) features to be numeric. Consequently, categorical features must be converted (encoded) to numbers before building models.

Our dataset contains two features that require encoding.

• For 'Gender', we will use scikit-learn's LabelEncoder() which maps each unique label to an integer (Male --> 1 and Female --> 0).
• For 'Geography', we will manually map values so that customers in Germany have the value of one (1) and all other customers (France and Spain) have zero (0). We chose this method since the churn rate for customers in the other two countries is almost equal and considerably lower than in Germany. Therefore, it makes sense to encode this feature to differentiate between German and non-German customers. Additionally, I tried one-hot encoding (get_dummies()) this feature, and the two new features for France and Spain had small feature importance.

Scaling¶

Feature scaling is a technique used to normalise the range of features. Some algorithms are sensitive to feature scaling (e.g. SVMs), while others are invariant (e.g. Random Forests).

I decided to use scikit-learn's StandardScaler(), which standardises features by subtracting the mean and dividing by the standard deviation. This transformation results in features with zero mean and unit variance.

Finally, we will split the train set into 'X_train' and 'y_train' sets:

Addressing Class Imbalance¶

As we have seen previously, there is an imbalance in the classes to be predicted, with one class (0 – retained) much more prevalent than the other (1 - churned):

Class imbalance is usually a problem and occurs in many real-world tasks. Classification using imbalanced data favours the majority class, meaning that ML algorithms will likely result in models that do little more than predict the most common class. Also, common metrics can be misleading when handling class-imbalanced data (e.g. if a dataset contains 99.9% 0s and 0.1% 1s, a classifier that always predicts 0 will have 99.9% accuracy).

Thankfully, some methods can address this problem. Resampling is a widely adopted technique designed to alter the class distribution in the training dataset. It consists of removing instances of the majority class (Undersampling) and/or adding copies of instances from the minority class (Oversampling). Additionally, there are algorithms for generating synthetic instances of the minority class. For example, as we read in Ref. [2], the SMOTE ('Synthetic Minority Oversampling Technique') algorithm 'finds a record that is similar to the record being upsampled and creates a synthetic record that is a randomly weighted average of the original record and the neighbouring record, where the weight is generated separately for each predictor'.

In this project, we will leave our dataset untouched and use the same algorithms but give them a different perspective on the problem. Penalised classification imposes an additional cost on the model for making classification mistakes on the minority class during training. These penalties can bias the model to pay more attention to the minority class (Ref. [9]). For most algorithms, penalised classification is implemented by including an extra hyperparameter called class_weight and setting it equal to balanced.

Pre-processing the Test Set¶

We need to perform the same pre-processing steps for the test set as we did for the training set.

Building Machine Learning Models¶

Baseline Models¶

In this section, we will create two simple models to estimate the baseline performance on the training set. Specifically, we will use the Gaussian Naïve Bayes and Logistic Regression algorithms.

We will use their default parameters and evaluate their (mean) AUC score by performing k-fold cross-validation. The idea behind k-fold cross-validation, illustrated in this figure, is simple: it splits the (training) set into k subsets/folds, trains the models using k-1 folds, and evaluates the model on the remaining one fold. This process is repeated until every fold is tested once. We will define the number of folds as a separate variable, cv, equal to 5. By setting this variable as an integer, stratified sampling will be used to create the folds.

Model Tuning¶

We are now ready to start building ML models. The seven classifiers we will use are:

1. Logistic Regression,
   
2. Support Vector Classifier,
   
3. Random Forest Classifier,
   
4. Balanced Random Forest Classifier,
   
5. Gradient Boosting Classifier,
   
6. Xtreme Gradient Boosting Classifier, and
   
7. Light Gradient Boosting Machine.

We will not go into detail about how these algorithms work. More information can be found in Ref. [1] or the corresponding documentation.

Using default hyperparameters usually results in non-optimised models that overfit or underfit the dataset. Hyperparameter tuning is the process of finding the set of hyperparameter values that achieves optimal performance. For this purpose, we will first define which hyperparameters we want to experiment with and what values to try out. We will pass this information to Scikit-Learn’s GridSearchCV, which evaluates all the possible combinations of hyperparameter values. As mentioned earlier, roc_auc will be the scoring metric for optimising the models. Note that GridSearchCV evaluates the performance by performing k-fold cross-validation; therefore, the number of folds must be provided. We will set this parameter equal to 5, as we did earlier for the baseline performance.

Logistic Regression¶

Support Vector Classifier¶

Random Forest¶

There are two variations of the standard Random Forest classifier that are suited for imbalanced classification (Ref. [8]):

1. Random Forest With Class Weighting: This variation is implemented by setting the class_weight parameter to balanced, which changes each class's weight when calculating the impurity score of a chosen split point.

2. Random Forest With Bootstrap Class Weighting: This variation is implemented by setting the class_weight parameter to balanced_subsample. This variation is different from the first, as it changes the class weighting based on the class distribution in each bootstrap sample instead of the entire training dataset.

Random Forest With Random Undersampling¶

A third variation of the Random Forest classifier performs data resampling on the bootstrap sample to explicitly change the class distribution. The BalancedRandomForestClassifier algorithm is provided by the imbalanced-learn package.

Gradient Boosting Classifier¶

The number of estimators after early stopping is:

XGBoost Classifier¶

LGBMClassifier¶

Ensemble Learning¶

We can combine the predictions of all these classifiers to determine if we get better predictive performance compared to each individual constituent classifier. This practice is the main motivation behind Ensemble Learning.

Specifically, we will use Soft Voting. In this case, every individual classifier provides a probability value that a specific data point belongs to a particular target class. The predictions are weighted by the classifier's importance and summed up. Then the target label with the greatest sum of weighted probabilities wins the vote.

Results¶

Feature Importance¶

Some classifiers allow us to visualise feature importance, i.e. how useful a feature is in predicting the target variable.

The first three classifiers ('RF', 'GBC', and 'XGB') display similar behaviour: 'Age' and 'NumOfProducts' seem like the most useful features, followed by 'IsActiveMember'. On the other hand, 'CreditScore' is the least important feature with a small value close to zero.

In contrast, 'Balance' is the most important feature for the LGBM model, followed by 'Age'. For the same classifier, 'CreditScore' is placed in the middle of the leaderboard.

Performance Comparison¶

Initially, we can compare the performance of our classifiers in terms of three individual metrics (Precision, recall, and area under the ROC curve):

Apart from Logistic Regression, all other classifiers have an AUC score higher than 0.80 (baseline performance). The LGBM model has the highest AUC score (0.865). However, apart from the GBC model, almost all classifiers display a similar performance with an AUC score around 0.860 (and similar scores for the other three metrics).

Using single metrics is not the only way of comparing the predictive performance of classification models. The ROC curve (Receiver Operating Characteristic curve) is a graph showing the performance of a classifier at different classification thresholds. It plots the true positive rate (another name for recall) against the false positive rate.

The dashed diagonal line represents a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner). In our case, all classifiers, apart from Logistic Regression, perform similarly.

Recently, I came across another tool for assessing the performance of a classifier model. Simply put, a Cumulative Gain shows the percentage of targets reached when considering a certain percentage of the population with the highest probability to be targeted according to the model (see here and here). The scikitplot library offers an easy way of plotting this chart:

This chart shows that if we target 50% of the customers most likely to churn (according to the model), the model will pick 89% of customers who will actually churn, while the random pick would pick only 50% of the targets.

Evaluating the Test Set¶

An integral part of any ML project is testing our models on unseen data. For this purpose, we will use the AUC score and compare the training and test set performance for all tuned models.

The performance on the test set for all models is fairly similar to the training set, which proves that we do not overfit the training set. Therefore, we can predict customer churn with an AUC approximately equal to 86%.

Calculating Profit¶

We assume that the bank intends to invest £1,000 in services and offers for each customer likely to churn. If the customer remains, the bank expects a return approximately five times the initial investment, hence £5,000. We can write a function that calculates the total profit for our predictions (Ref. [7]).

The LGBM classifier, the classifier with the highest AUC score, yields the highest profit, around £900,000! This amount is almost double the profit we would get with the baseline Gaussian Naive Bayes classifier.

References¶

The main resources I used are the following two books:

[1] Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition, by Aurélien Géron (2019)

[2] Practical Statistics for Data Scientists, 2nd Edition, by Peter Bruce, Andrew Bruce, and Peter Gedeck (2020)

The following resources also helped me in my analysis:

[3] Bank Customer Churn, by Zicheng Shi (same dataset but analysis in R) (Accessed: 21 December 2020)

[4] Metrics and scoring: quantifying the quality of predictions on scikit-learn.org (Accessed: 20 December 2020)

[5] Easy Guide To Data Preprocessing In Python, by Ahmad Anis (Accessed: 20 December 2020)

[6] Meaningful Metrics: Cumulative Gains and Lyft Charts, by Raffi Sahakyan (Accessed: 20 December 2020)

[7] Predict Customer Churn (the right way) using PyCaret, by Moez Ali (Accessed: 08 December 2021)

[8] Bagging and Random Forest for Imbalanced Classification by Jason Brownlee (Accessed: 06 December 2021)

[9] 8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset by Jason Brownlee (Accessed: 12 December 2021)

If you would like to read more about customer churn:

[10] Churn Rate, by Jake Frankenfield on Investopedia (Accessed: 15 December 2020)

[11] Customer attrition on Wikipedia (Accessed: 15 December 2020)

[12] A Survey on Churn Prediction Techniques in Communication Sector, by N.Kamalraj and A.Malathi (Accessed: 15 December 2020)

[13] Customer Attrition on optimove (Accessed: 15 December 2020)

Conclusions¶

Our notebook just came to an end! Our final report to the bank should be based on two main points:

• EDA can help us identify which features contribute to customer churn. Additionally, feature importance analysis can quantify the importance of each feature in predicting the likelihood of churn. Our results reveal that the most significant feature is age (older customers are more likely to churn), followed by the number of products (having more products increases a customer’s likelihood to churn). The bank could use our findings to adapt and improve its services to increase satisfaction for those customers more likely to churn.

• We can build several ML models using the pre-processed dataset. The best-performing model with an AUC score of ~0.865 is the optimised model based on the LGBM classifier. The model was tested on unseen data. For a test set consisting of 2,000 customers, this model yields a total profit of £900,000! As a comparison, our baseline model (based on the Gaussian Naïve Bayes algorithm) produces a total profit of £456,000.