BreezeGen - Imbalanced Classification¶

Analyze an imbalanced dataset. Train models using the imbalanced-learn library for over- and under-sampling, and compare model performance.

BreezeGen is a maintenance company specializing in wind turbines. They have supplied ciphered sensor data. Generator maintenance is costly, so preventing a full replacement is paramount.

• An inspection costs $5,000 regardless of whether any repairs end up being necessary.

• A repair costs $15,000.

• A complete device replacement costs $40,000. We obviously want to minimize the frequency of replacement.

The target variable encodes the failure state, with 0 indicating no failure and 1 signalling failure.

Importing libraries¶

First update scikit-learn and imbalanced-learn libraries.

Then we can import the necessary libraries.

Loading Data¶

EDA¶

• The ciphered data consists of floating point numbers with values near 0. There are both positive and negative values.

• The target variable is categorical, Boolean in fact. There seems, at first glance, to be imbalance in the target classes.

Indeed, around 95% of the target observations are 0. From the data dictionary, we learn this indicates 'No Failure', which is ideal. Thus, only around 5% of observations are 'Failure' cases.

Just to double-check the integrity of our data, we confirm that the forty ciphered features are all stored as floats. This indicates to me that there aren't any errant entries, such as '?', which would force the column to be string data type.

We see some columns have missing entries.

Columns 'V1' and 'V2' have missing values.

Looking at the table of statistics, it is challenging to glean much insight. From the statistics, it appears that most of the distributions are fairly symmetric.

We find that cases requiring the replacement of equipment are fairly rare.

In the next section, we will process the data for modeling. This includes scaling the features so they have approximately mean 0 and standard deviation 1.

Data Pre-processing¶

To start, we change the target variable type to categorical.

We separate our data set. First, we break off the target variable. Then we split X and y into training and validation sets. Recall that we have a separate CSV file with test data for final model production, so we need not apportion a test set here.

We next make a simple pre-processing pipeline that outputs pandas DataFrames. We will use this to scale our data and impute missing values:

• We will scale columns so they have mean 0 and standard deviation 1.
• We impute missing data using the K Nearest Neighbors method. By default, we use 5 neighbors.

We fit the pipeline on the training data and transform it.

By fitting the pipeline on the training data and then transforming the validation data using this pipeline, we avoid data leakage: The data in the validation set does not influence the means, standard deviations, or nearest neighbors calculation.

Model Building¶

Functions¶

The following functions will assist with model building and assessment. Some are adapted from my previous projects.

The function above prints a confusion matrix of model performance on validation data. It also prints a table with validation accuracy, precision, recall, and F1 scores.

The function above returns the mean cross-validated recall for a given model.

The funciton above collects various metrics for evaluating model performance into a comparison table.

Model Building with original data¶

Decision Tree¶

A plain decision tree classifier yields a cross-validated recall of 0.72, decent performance for the first attempt.

After fitting the model to the whole training set, we find high accuracy (around 97%), while precision and recall are closer to 75%.

Note: All confusion matrices in this project are compiled using the validation data.

Moreover, we see clear evidence of overfitting when we compare training and validation set results.

Logistic Regression¶

The plain logistic regression model performs far worse than the decision tree, with a cross-validated mean recall of under 50%. In other words, this model performs worse than randomly guessing for the positive class (1).

As the negative class (0) is by far the majority class, this logistic regression scores better on precision because it predicts few false positives.

While this model's performance is poor, at least it isn't overfit.

Bagging Classifier¶

A CV recall of 71% is much better than the logistlic regression, similar to the decision tree score.

In addition to good recall, this model produces high accuracy and precision!

Unfortunately, when comparing training and validation scores, we find that this model is also overfitting (note recall scores).

Random Forest¶

Our second bagging model, random forest, scores a bit better, with a CV recall of around 76%.

Again, precision and accuracy are stellar on the validation data. Note in the confusion matrix above that there are SIX false positives in a data set of 10,000! This translates to a specificity (true negative rate) of over 99.9%.

Our model unfortunately suffers from overfitting.

AdaBoost¶

While AdaBoost yields a lower CV recall score, at only around 61%, we are hopeful that boosting methods will be less susceptible to overfitting.

An accuracy of 97% and good precision is promising, but the surprising result in the table above is the recall, which is higher than the CV recall calculated above. We find 65% recall on validation data with this model.

While there is not much evidence of overfitting here, the model performance is appreciably lower than other models.

Gradient Boosting¶

Gradient boosting has done much better as compared to AdaBoost, with a CV recall of 72%.

As with many other models, recall is actually the lowest score of the four metrics above, with accuracy and precision being much higher.

So far, the gradient boosting model is one of the better models. The only issue is evidence of overfitting (note the recall).

XGBoost¶

A score of 74% is comparable to the better performing models above.

Recall at 77% is one of the best we've seen so far, and both precision and accuracy scores are quite high.

Note how much lower the CV recall score is from the training and validation recall scores. The latter two are comparable, so I do not fear overfitting here.

Model Building with Oversampled data¶

Now we will oversample the minority class of the target variable.

Decision Tree [Oversampled]¶

With such a high CV recall score (97%!), I am skeptical already of this model.

Unlike most of the previous models, recall is higher here than precision. This is certainly due to the oversampling.

Unsurprisingly though, this model is exceptionally overfit.

Logistic Regression [Oversampled]¶

A CV recall score of 87% is one of the highest we've seen so far, but isn't so high as to immediately suggest overfitting.

Both accuracy and recall are quite good with this logistic regression model. The cost, however, is disasterous precision: 28%! The regression's prediction of the positive class is far worse than randomly guessing.

There is no real evidence of overfitting, so this model looks solid. Great accuracy and recall in both training and validation. It's only downside is terrible precision.

Bagging Classifier [Oversampled]¶

As with the last decision tree, this CV recall is high enough (97%) to raise suspicion of overfitting.

Great scores across the board, while promising, could be symptomatic of overfitting.

Sure enough, this model is overfit (see recall).

Random Forest [Oversampled]¶

Another questionably high score here: 98%.

Again, it would be lovely if these scores were beliveable, but we should first look at the table below to assess whether this model is overfit.

Indeed, the difference in training and validation recall is enough to confirm overfitting.

AdaBoost [Oversampled]¶

We have seen good results with boosting before, so a score of 88% is promising.

Here we see that the cost of good recall is poor precision. Indeed, the percentage of false positives is nearly double that of true positives predicted by this AdaBoost model (see confusion matrix).

This boosting model offers one of the higher recall scores we have seen without overfitting.

Gradient Boosting [Oversampled]¶

As with the boosting models trained on the original data, the gradient boosting model here has a better CV recall score than the previous AdaBoost model, at around 91%.

Here too we have good performance on all four metrics. In contrast to the AdaBoost model above, this model offers much better precision, in particular, a score greater than 50%, or random chance.

As we have seen with boosting models, this model does not have much issue with overfitting.

XGBoost [Oversampled]¶

With comparable performance to gradient boosting, a score of about 91% for XGBoost is unsurprising.

Good or great performance on all metrics, and more true positives than both false positive and false negatives.

Looking back over this table so far, the gradient boosting and XGBoost models trained on oversampled data have performed best. Random forest boasts promising precision if only we could curtail overfitting.

Model Building with Undersampled data¶

Now we undersample the majority class of the target variable. This is another method to balanced the class weights in the target.

Decision Tree [Undersampled]¶

A score of 85% is good for this estimator, but since previous decision trees had a tendency to overfit, I will wait to assess performance until I see the other metrics.

This decision tree is the worst on precision (see the concerning rate of false positives). Recall and accuracy, however, are good.

As with the other decision trees, this model overfits disasterously.

Logistic Regression [Undersampled]¶

A comparable score to the last model, 84% is promising!

Again, the false positive rate really cuts into precision (and thus F1) score. Accuracy and recall are good though.

Thankfully, this model doesn't overfit. Decent recall would make this a top contender, were it not for the worse-than-guessing precision, which would end up costing BreezeGen greatly in inspection costs.

Bagging Classifier [Undersampled]¶

Bagging has been overfitting, so I'm curious to see if the model trained on undersampled data avoids this issue. The cross-validated recall score is 87%.

With about as many false positives as true positives (see confusion matrix), precision lands at 50%, or as good as random guessing. That being said, accuracy and recall and both great.

Unfortunately overfitting plagues this model too. Note especially the disparity in recall.

Random Forest [Undersampled]¶

This model scores 98%, just like the random forest trained on oversampled data.

As with the random forest model trained on oversampled data, the precision is good. Accuracy and recall are high.

While this model is surely overfit, tuning might just curtail the issue. And with a decent precision score, this model might be worth tuning to hang onto the cost savings good precision affords.

AdaBoost [Undersampled]¶

The cross-validated recall score for this AdaBoost classifier is 86%.

While both accuracy and recall are good on the validation set, precision is really quite poor.

This model performs well for recall, but insufficient precision would cost BreezeGen in the long run.

Gradient Boosting [Undersampled]¶

A CV recall score of 88% is slightly less than that of the Gradient Boosting classifier trained on oversampled data.

Accuracy and recall are both great, and as this is a boosting model, I do not fear overfitting. Precision is about as good as guessing: compare the true positives and false positives in the confusion matrix above.

This is one of the better models we've seen. Great recall and accuracy, at around 91% and 95% respectively.

XGBoost [Undersampled]¶

A CV recall score of 88% is good, comparable with the previous gradient boosting model.

This model scores comparably on accuracy and recall to the previous model, but improves on precision! (Compare 53% to 59%.)

Another promising model. Great recall and accuracy, without excessive detriment to precision.

HyperparameterTuning¶

Model Finalists¶

We'll assemble some candidate finalists and compare them in a table. The following function computes many performance metrics for a given model.