Table of Contents

click here to view notebook on nbviewer.

Import modules¶

Import all modules required for this notebook. This is the only place where all modules are imported.

Function definitions¶

This code segment aligns all plots at the center.

Inroduction¶

This case study is about demonstrating data science skills to build and evaluate mathematical models for a real-world dataset while minimizing dollar amount cost. Models shall be developed for the given dataset and compared with respect to dollar amount based on appropriate evaluation metric. Evaluation metric shall be determined after exploring features provided in the data and distribution of class labels shall be studied to decide on metric to be used for model evaluation.

Business Understanding¶

A Business client has reached out to us with an unknown dataset with many features and would like us to build a model that balances minimizing cost to the business, with respect to the model being accurate enough to apply to their day-to-day business.

Here is how the decision cost breaks down:

• True Positive - $0

• True Negative - $0

• False Positive - $10

• False Negative - $500

Incorrectly classifying the data can get costly as you can see above. Correctly classified data does not cost the client any money. False Positive or classifying the data as true when its false costs the client \$10. False Negative or classifying the data as false when it is true costs the client $500. We will need to factor in the cost of each of the features and their associated values during the model building to minimize the cost. For this analysis, we will compare recall or the true positive rate for each model analyzed. Recall or the true positive rate is the number of positive samples that are correctly classified as ‘positive’. If all of them are identified correctly, then recall will be 1. If all of them were classified incorrectly, then recall will be 0. With some positive samples classified as negative, recall with be in between 0 and 1. The client requires us to provide a detailed cost benefit analysis of the models we are proposing and provide comparative analysis of models if more than one models are proposed.

Data Evaluation / Engineering¶

This section provides insights about datasets as follows

• Indentifies categorical and numerical features
• Data type conversion as required
• Missing value analysis
• Makes assumptions and limitations if any

Load Data¶

Dataset was downloaded from the SMU cloud. The dataset contains 160K observations, 50 independent variables and 1 target variable. Target variable contains value 0s and 1s only this is going to be binary classification problem.

Summary is as follows:

• 50 indepedent variables
• 1 target variables
• 160K observations
• 3 categorical variables 47 continuous variables
• It contains few missing values that needs to be investigated for elimination or imputation

Further analysis is conducted in next sections.

Dataframe structure¶

Information about fields is not available. df.info() provides us following information.

• x0 to x23, x25 to x28, x31, x33 to x36 and x38 to x49 are numeric features.
• x24, x29-x30, x32 and x37 are categorical or non-numeric features
• y is target variable which is binary.

Data Summary¶

Below is descriptive statisics of all features. It indicates that features most features are on different scale and required normalization before they can be used for the modeling.

Non numeric features¶

Here are non-numeric features which has content other than numbers. following are the fields.

• x24 - Region names
• x29 - Month
• x30 - Day of the week
• x32 - Looks like rate of something.so can be converted to float.
• x37 - has $ sign assigned to numbers. Should be numeric field. can be converted to float.

Basically x24, x29 and x30 are categorical variables in the dataset.All other fields are numeric fields.

Change Data Types¶

• x32 remove % sign and convert to float
• x37 remove $ sign and convert to float
• Indices of non-numeric features are stored in indices_obj_features
• Indices of numeric features are stored in indices_num_features

Missing Values¶

Inspection of missing values begins with plotting missing data. missingno is the python module that enables us to visually inspect missing values as shown below. This is quick check to see if there are any missing values or NAs int the dataset. If dataset contains missing values they need to be analyzed further.

The dataset that we have got has less than 1% of missing values and they don't cause any major change in distribution of data. So we have removed missing values in later section of this notebook shown here step by step.

Missingno Plot¶

Featurewise missing percentage¶

Missing value treatment¶

Since missing values are very nominal in size (less than 0.5%) compared to size of the dataset rows with missing values are removed and final dataset was obtained as below.

Below plot shows there are no missing values left in the df_final Dataframe.

Target distribution¶

Overall target distribution¶

Below plot indicates that this is going to be imbalanced classification problem.

Distribution of the target

Target - Region, Month and Daywise¶

Here are observations from the plots

• Asia is main contributor to postive class
• Summer period is the main period for positive class
• Middle of the week i.e. Wednesday class is seen most occurrences of postive class.

Correlogram¶

Below correleogram shows there are not many correlated features and we will rely on automatic recursive feature elimination technique to reduce number of feature. This approach will keep optimal number of features.

• x2 and x6 are highly correlated
• x38 and x41 are highly correlated

Assumptions¶

• Details of the featurs are not available so analysis shall be done without taking specific domain into consideration.
• Since false negative is costing higher focus shall be on minimizing false negative as much as possible.
• Not knowing domain may not provide best model but it shall be competitve enough.

Limitations¶

• The major limitation is we don't know what the dataset is about
• cannot apply domain knowledge to the data

Sampling Techniques¶

• From the plots that have examined above it is clear that class labels are not evenly distributed.
• This is not highly imbalanced but moderately imbalanced data
• Default stratified sampling shall be used for train/test split.
• Stratified sampling will maintain the class distribution in train/test split as in original data
• Data will be split into Train(80%) and Test(20%)

Modeling Preparations¶

Train/Test Split¶

One Hot Encoding¶

Normalize data¶

Store dataframes into csv. These spreadsheets shall be used to run different models on different machines.

Principal Component Analysis¶

Since we have 67 features in total after one hot encoding here we are checking if Principal component analysis is helpful in reducing features are not. Running PCA we found that at least 47 components which contributes to 90% variance in the data are required which is not significant reduction in the features. So we won't be using feature reduction using PCA. we will rather use recursive feature elimination techqniue for logistic regression.

• 47 components which contributes 90% variance are required.
• PCA is not much helpful in reducing features dramatically
• We only explored option and found to be not useful for our modeling.

Choice of Metric¶

Choice of metric for model development

The requirement is to minimize the cost function, which penalizes false negatives (FN) 50x more than false positives (FP).

$ cost penalty for FN = $500

$ cost penalty for FP = $10

$ cost penalty for TP, TN = 0

Effectively, above implies that for the same total cost, count of FN = 1/50th of FP. Or one needs to prioritize minimizing FN.

Given above, Recall is the metric of choice to search for best model. Recall definition is given below:

Recall = TP / TP +FN

Ideal model should have very low or zero FN, hence requires high recall score.

Model Building & Evaluation¶

Logistic Regression

Sklearn comes with get_param function which returns list of hyperparameters for the given estimators. All it requires is to create object of the estimator.

Following hyperparameter can be tuned for Logistic Regression with different solver (lbfgs,sag,saga,newton-cg,liblinear) and regularization (l1 an l2) :

• C - also known as inverse regularization parameter. It controls penalty strength.

KNN - K Nearest Neighbors

For KNN n_neighbors is the hyperparameter that can be tuned. Different values of n_neighbor are tried and value that gives best model can be used for predictions.

Random Forest

Random forest is non-parametric classifier. There are many hyperparamters tha can be tuned to build random forest listed as below

• n_estimators - The number of trees in the forest.
• max_depth - maximum depth of the tree
• min_samples_split - Minimum number of samples required to split internal nodes
• min_samples_leafint - Minimum number of samples required to be at leaf nodes.
• max_features - The number of features to be considered when looking at best split
• min_weight_fraction_leaf - Minimum weighted fraction of the sum total weights required to be at leaf node.

These are important hyperparamerters which can be tuned to build models. For our dataset are using only two hyperparameters:

• n_estimators
• max_features

max features is number of features to consider when looking for the best split:

If int, then consider max_features features at each split.

If float, then max_features is a fraction and int(max_features * n_features) features are considered at each split.

If “auto”, then max_features=sqrt(n_features).

If “sqrt”, then max_features=sqrt(n_features) (same as “auto”).

If “log2”, then max_features=log2(n_features).

If None, then max_features=n_features. [scikitlearn-doc]

Logistic Regression¶

Logistic Regression Hyperparameters¶

Following are the hyperparameters returned by get_params function of sklearn library.

Recursive Feature Elimination¶

Our dataset has 67 features after one hot encoding and not all features might be important. Recursive feature elimination finds optimal number of features and identifies features that should be kept in the model. Below code runs Recursive feature elimination with cross validation and finds optimal number of features and same shall be used for building model.

Plot below is self-explanatory. It shows how recall/accuracy score varies with number of features increases. After running full cross validation number of optimal features are found to be - 60

Hyperparameter Tuning¶

Grid search has been executed for logistic regression with following:

• C with different values - [0.1,0.01,0.001,0.0001,0.5,0.6,10]
• Two solvers ['lbfgs','sag']
• Both optimizers supports L2 regularization. So only L2 has been used in Grid Search.
• Random seed - 1999 for reproducibility
• max_iter set to 500 for sag solver. sag is stochastic gradient descent.

With these variables set Grid search runs for all possible combinations of the grid parameters. Result of grid search is stored in the file as shown in below code.

Best Parameters¶

Grid search runs for all grid parameters and finds best from all combinations in runs in best_score_ and best_params_.

Grid Search Result

Grid search result obained above was built with all evaluation metric against hyperparameter C:

• recall
• precision
• f1
• roc_auc

C=0.6 is the optimal value found for the logistic regression. It is quite evident from the plot that precision,recall, f1 score and accuracy remains constant if we increase value of C. It cannot be tuned beyond 0.6

Build Logistic Regression¶

Feature Importance¶

Model Performance¶

Build Cost Matrix¶

Precision Recall Curve¶

Cost Curve¶

K-Nearest Neighbor¶

KNN Classifier Hyperparameters¶

Hyperparameter Tuning¶

Best Parameters¶

Grid Search Result

Grid search found n_neighbor=12 with better recall and selected for the predictions.

Build KNN classifier¶

Model Performance¶

Build Cost Matrix¶

Precision-Recall Curve¶

Cost Curve¶

Feature Importance¶

Random Forest¶

Random Forest Hyperparameters¶

Hyperparameter tuning¶

Best Parameters¶

This is the best model returned by Grid Search hyperparameter tuning.

3D Plot (estimators X max_features X Recall Score)¶

This is 3D plot after tunining following hyperparameters

• Number of estimators
• Max features
• Mean Recall score

Build Random Forest Model¶

Feature Importance¶

Model Performance¶

Build Cost Matrix¶

Precision Recall Curve¶

Cost Curve¶

Analysis of model performance¶

Table 7.4.1 - Performance Vs Dollar Cost

Random forest model provides the best performance by minimizing the false negatives, and provides best (highest) recall. That also is evident from lost $cost for Random Forest.

Discussion on model performance

Random Forest is an ensemble learning method that consists of multiple decision trees. An ensemble learning method ends up providing better metric, as it’s possible to train trees addressing a specific feature space in data set that other trees don’t.

Logistic regression performs the poorest here. Logistic regression has assumption of linear dependence between dependent variable and independent features. It is possible the available data set is not linearly separable, providing poorest model metric.

KNN performs between Random forest and logistic regression. Given the number of feature set and final 'y' variable, it is possible the data set lends itself well into determining output based on how close the features are.

It is possible that this data could be for financial fraud detection, where y=1 implies fraud detected. If this is true, then it is likely that majority of frauds have a signature in feature set that is common. This common feature set would likely drive better performance for KNN as commonality in feature ties to nearest neighbor approach for determining a fraud. Hence it is better than logistic regression for this problem.

Discussion on Dollar cost vs. model metric

Dollar cost and model metric tradeoffs are further analyzed by varying the binary classification threshold. Table 7.4.2 below captures binary classification threshold that minimizes Dollar cost, for a model originally obtained through grid search and hyper parameter tuning, as summarized in Table 7.4.1.

Table 7.4.2 - Threshold selection

Observations

For each of the models, the \$ cost is significantly reduced by using a much lower classification threshold. Compare with default results in Table 7.4.1.

Using a very low threshold is converting TN and FN to FP (false positives). Since the cost of FP is smaller by 50x than of FN, it is helping lower the actual \$ cost to company. Above also implies, one need not build any sophisticated model, and classify all new records as TRUE, yet not be too worse off the minimum \$ cost. This is happening as the $ cost of FN = 50x \$ cost of FP, i.e. there is a very high skew.

Experiments with setting \$ cost of FN = 500, and $ cost of FP = 100, (i.e. much less skew),gave a much higher Accuracy, and higher threshold. In such a scenario, spending time and effort to build a model will likely provide higher return on investment of building a model

Model Interpretability & Explainability¶

Logistic Regression¶

Random Forest¶

Feature importance refers to a class of techniques that measure how useful input features are for predicting a target variable using a given model. The score assigned alludes to the relative importance of a feature in a particular model. It can also used to better understanding a model’s logic or reduce the number of input variables. Feature importance is not defined for KNN algorithms, but it is available for tree models.

The feature importance result for logistic regression model prioritizes x46, x38, x35, x37 variables whereas in the random forest listed the best/importance variables are x23, x20, x49, x48, x42, x28, x12, x40, x37, x27, x7, x41, x38, x46, x6, x2, x32. Even though there is difference list hierarchy and magnitude both models list x46, x38, x37 as the top important variables. That is both tree models identified similar features as most important for predicting product formulation. This indicates that model performance is consistent and that the appropriate variables are being included in our model for prediction.

As depicted in table 7.4.1 random forest is the best model with cost of \$1,049,500

Case Conclusions¶

The analysis involved building a binary classification model, of unknown dataset, with primary ask to minimize the \$ cost.

The \$ cost penalized False Negatives (FN) much higher than False Positives (FP), with numbers being \$500 and $10 respectively.

Logistic Regression, KNN and Random Forest models were tuned for best recall score, and compared for \$ cost performance. It is clear that Random Forest does the best, by providing lowest $ cost, while providing a high Recall value.

However, a careful analysis of choice of binary classification threshold, indicated that one can minimize the \$ cost further, and achieve False Negative Rate to be ‘0’ for Logistic regression, and just 0.8% for Random Forest. Interestingly, Logistic regression result indicates that one doesn’t need a sophisticated model to minimize \$ cost. The cost impact of FN vs FP is so skewed, that a very low threshold or effectively assuming all records are TRUE still gives minimal cost. Further, Random Forest model does better than logistic regression by minimizes cost further by ~ \$40,000.

Above brings to fore a tradeoff between optimizing for minimal \$ cost, vs a good model with high accuracy. One can have a good model, but may not have lowest \$ cost. On the other hand, if \$ cost reduction is the ONLY criterion, we may not need a complicated model, or detailed modeling activity. The \$ cost penalty for FN and FP have a significant bearing on development, and deployment of the right model. It is very important that we have right estimate of cost of FN and FP. It will be also important to ask if cost of FN and FP can vary, depending on some of the features in the data set. E.g. the cost may be different based on geographical location (note that data set does appear to have a variable encoding geographical area). All of these needs to be considered before considering to productize and use this methodology.

Reference¶

• Scikit learn documentation
• StackOverflow

Appendix-I - Source code¶

Logistic Regression Grid Search¶

This code was used to run grid search for logisitc regression after applying recursive feature elimination. Grid search takes time to run therefore it was executed separately as background process and results were stored in csv file.

Logistic regression cost matrix¶

KNN - Grid Search¶

KNN cost matrix¶

Random Forest Cost matrix¶