4 Pre-Processing and Training Data¶

4.1 Contents¶

• 4 Pre-Processing and Training Data
  • 4.1 Contents
  • 4.2 Introduction
  • 4.3 Imports
  • 4.4 Load Data
  • 4.5 Extract Big Mountain Data
  • 4.6 Train/Test Split
  • 4.7 Initial Not-Even-A-Model
    • 4.7.1 Metrics
      • 4.7.1.1 R-squared, or coefficient of determination
      • 4.7.1.2 Mean Absolute Error
      • 4.7.1.3 Mean Squared Error
    • 4.7.2 sklearn metrics
      • 4.7.2.0.1 R-squared
      • 4.7.2.0.2 Mean absolute error
      • 4.7.2.0.3 Mean squared error
    • 4.7.3 Note On Calculating Metrics
  • 4.8 Initial Models
    • 4.8.1 Imputing missing feature (predictor) values
      • 4.8.1.1 Impute missing values with median
        • 4.8.1.1.1 Learn the values to impute from the train set
        • 4.8.1.1.2 Apply the imputation to both train and test splits
        • 4.8.1.1.3 Scale the data
        • 4.8.1.1.4 Train the model on the train split
        • 4.8.1.1.5 Make predictions using the model on both train and test splits
        • 4.8.1.1.6 Assess model performance
      • 4.8.1.2 Impute missing values with the mean
        • 4.8.1.2.1 Learn the values to impute from the train set
        • 4.8.1.2.2 Apply the imputation to both train and test splits
        • 4.8.1.2.3 Scale the data
        • 4.8.1.2.4 Train the model on the train split
        • 4.8.1.2.5 Make predictions using the model on both train and test splits
        • 4.8.1.2.6 Assess model performance
    • 4.8.2 Pipelines
      • 4.8.2.1 Define the pipeline
      • 4.8.2.2 Fit the pipeline
      • 4.8.2.3 Make predictions on the train and test sets
      • 4.8.2.4 Assess performance
  • 4.9 Refining The Linear Model
    • 4.9.1 Define the pipeline
    • 4.9.2 Fit the pipeline
    • 4.9.3 Assess performance on the train and test set
    • 4.9.4 Define a new pipeline to select a different number of features
    • 4.9.5 Fit the pipeline
    • 4.9.6 Assess performance on train and test data
    • 4.9.7 Assessing performance using cross-validation
    • 4.9.8 Hyperparameter search using GridSearchCV
  • 4.10 Random Forest Model
    • 4.10.1 Define the pipeline
    • 4.10.2 Fit and assess performance using cross-validation
    • 4.10.3 Hyperparameter search using GridSearchCV
  • 4.11 Final Model Selection
    • 4.11.1 Linear regression model performance
    • 4.11.2 Random forest regression model performance
    • 4.11.3 Conclusion
  • 4.12 Data quantity assessment
  • 4.13 Save best model object from pipeline
  • 4.14 Summary

4.2 Introduction¶

In preceding notebooks, we performed preliminary assessments of data quality and refined the question to be answered. We found a small number of observations that clearly indicated whether to replace values or drop a whole row. We determined that predicting the adult weekend ticket price was our primary aim. We threw away records with missing price data, but not before making the most of the other available data to look for any patterns among the states. We didn't see any and decided to treat all states equally; the state label didn't seem to be particularly useful.

In this notebook, we'll start to build machine learning models. Before diving into a machine learning model, however, we'll start by considering how useful the mean value is as a predictor. We never want to go to stakeholders with a machine learning model only to have the CEO point out that it performs worse than just guessing the average! Our first model is always a baseline performance comparitor for any subsequent model. Next, we'll build up the process of efficiently creating robust models to compare to our baseline forecast. We can validate steps with our own functions for checking expected equivalences between, say, pandas and sklearn implementations.

4.3 Imports¶

4.4 Load Data¶

4.5 Extract Big Mountain Data¶

Big Mountain is our resort. Separate it from the rest of the data to use later.

4.6 Train/Test Split¶

So far, we've treated ski resort data as a single entity. In machine learning, when we train our model on all of our data, we end up with no data set aside to evaluate model performance. We could keep making more and more complex models that fit the data better and better and not realise we are overfitting the model. By partitioning the data into training and testing splits, without letting a model (or missing-value imputation) learn anything about the test split, we have a somewhat independent assessment of how our model might perform in the future. An often overlooked subtlety here is that people all too frequently use the test set to assess model performance and then compare multiple models to pick the best. This means their overall model selection process is flawed: The engineer picks the model sans help from the test set. Instead we use held-out data and/or k-fold cross-validation to simulate additional test sets and assess model performance. The formal test set is very useful as a final check on expected future performance.

What partition sizes would we have with a 70/30 train/test split?

We have only numeric features in X now!

4.7 Initial Not-Even-A-Model¶

We'll begin by determining how good the mean is as a predictor. In other words, what if we simply say our best guess is the average price?

sklearn's DummyRegressor easily does this:

Having established the grand mean, we need to determine how closely it matches, or explains, the actual values. There are many ways of assessing how good one set of values agrees with another, which brings us to the subject of metrics.

4.7.1 Metrics¶

4.7.1.1 R-squared, or coefficient of determination¶

One measure is $R^2$, the coefficient of determination. This is a measure of the proportion of variance in the dependent variable (our ticket price) that is predicted by our "model". The linked Wikipedia articles gives a nice explanation of how negative values can arise. This is frequently a cause of confusion for newcomers who, reasonably, ask how can a squared value be negative?

Recall the mean can be denoted by $\bar{y}$, where

and where $y_i$ are the individual values of the dependent variable.

The total sum of squares (error), can be expressed as

The above formula should be familiar as it's simply the variance without the denominator to scale (divide) by the sample size.

The residual sum of squares is similarly defined to be

where $\hat{y}$ are our predicted values for the depended variable.

The coefficient of determination, $R^2$, here is given by

Putting it into words, it's one minus the ratio of the residual variance to the original variance. Thus, the baseline model here, which always predicts $\bar{y}$, should give $R^2=0$. A model that perfectly predicts the observed values would have no residual error and so give $R^2=1$. Models that do worse than predicting the mean will have increased the sum of squares of residuals and so produce a negative $R^2$.

We make our predictions by creating an array of length the size of the training set with the single value of the mean.

Remember the sklearn dummy regressor?

We can see that DummyRegressor produces exactly the same results and saves us from having to broadcast the mean (or whichever other statistic we used - check out the documentation to see what's available) to an array of the appropriate length. It also gives us an object with fit() and predict() methods as well, so we can use them as conveniently as any other sklearn estimator.

Exactly as expected, if we use the average value as our prediction, we get an $R^2$ of zero on our training set. What if we use this "model" to predict unseen values from the test set? Remember, of course, that our "model" is trained on the training set; we still use the training set mean as our prediction.

Make predictions by creating an array of length the size of the test set with the single value of the (training) mean.

Generally, you can expect performance on a test set to be slightly worse than on the training set. As you are getting an $R^2$ of zero on the training set, there's nowhere to go but negative!

$R^2$ is a common metric, and interpretable in terms of the amount of variance explained, it's less appealing if we want an idea of how "close" our predictions are to the true values. Metrics that summarise the difference between predicted and actual values are mean absolute error and mean squared error.

4.7.1.2 Mean Absolute Error¶

This is very simply the average of the absolute errors:

Mean absolute error is arguably the most intuitive of all the metrics, this essentially tells you that, on average, you might expect to be off by around \$19 if you guessed ticket price based on an average of known values.

4.7.1.3 Mean Squared Error¶

Another common metric (and an important one internally for optimizing machine learning models) is the mean squared error. This is simply the average of the square of the errors:

So here, we get a slightly better MSE on the test set than we did on the train set. And what does a squared error mean anyway? To convert this back to our measurement space, we often take the square root, to form the root mean square error thus:

4.7.2 sklearn metrics¶

Functions are good, but you don't want to have to define functions every time we want to assess performance. sklearn.metrics provides many commonly used metrics, included the ones above.

4.7.2.0.1 R-squared¶

4.7.2.0.2 Mean absolute error¶

4.7.2.0.3 Mean squared error¶

4.7.3 Note On Calculating Metrics¶

In a Jupyter code cell, running r2_score? will bring up the docstring for the function, and r2_score?? will bring up the actual code of the function! Here we try it and compare the source for sklearn's function with ours.

We can get very different results swapping the argument order. It's worth highlighting this because data scientists do this too much in the real world! Frequently the argument order doesn't matter, but it will bite when we do it with a function that does care. It's sloppy, bad practice and if we don't make a habit of putting arguments in the right order, we stand to forget!

Remember:

• argument order matters,
• check function syntax with func? in a code cell

4.8 Initial Models¶

4.8.1 Imputing missing feature (predictor) values¶

Recall when performing EDA, we imputed (filled in) some missing values in Pandas. We can impute missing values using scikit-learn, but we will prioritize imputation from a train split and apply that to the test split to then assess how well our imputation worked.

4.8.1.1 Impute missing values with median¶

We have missing values. Recall from our data exploration that many distributions were skewed. Our first thought might be to impute missing values using the median.

4.8.1.1.1 Learn the values to impute from the train set¶

4.8.1.1.2 Apply the imputation to both train and test splits¶

4.8.1.1.3 Scale the data¶

As we have features measured in many different units, with numbers that vary by orders of magnitude, start off by scaling them to put them all on a consistent scale. The StandardScaler scales each feature to zero mean and unit variance.

4.8.1.1.4 Train the model on the train split¶

4.8.1.1.5 Make predictions using the model on both train and test splits¶

4.8.1.1.6 Assess model performance¶

Recall that we estimated ticket prices by simply using a known average. As expected, this produced an $R^2$ of zero for both the training and test set, because $R^2$ tells us how much of the variance we've explaining beyond that of using just the mean. Here, we see that our simple linear regression model explains over 80% of the variance on the train set and over 70% on the test set. Clearly, we are onto something, although the much lower value for the test set is indicative of overfitting. This isn't a surprise as we've made no effort to select a parsimonious set of features or deal with multicollinearity in our data.

Using this model, then, on average we'd expect to estimate a ticket price within \$9 or so of the real price. This is much, much better than the \$19 from just guessing using the average. There may be something to this machine learning lark after all!

4.8.1.2 Impute missing values with the mean¶

We chose to use the median for filling missing values because of the skew of many of our predictor feature distributions, let's try the mean.

4.8.1.2.1 Learn the values to impute from the train set¶

By eye, we can immediately tell that our replacement values are much higher than those from using the median.

4.8.1.2.2 Apply the imputation to both train and test splits¶

4.8.1.2.3 Scale the data¶

4.8.1.2.4 Train the model on the train split¶

4.8.1.2.5 Make predictions using the model on both train and test splits¶

4.8.1.2.6 Assess model performance¶

These results don't seem very different to when the one's we used with the median for imputing missing values. Perhaps it doesn't make much difference here. Maybe our overtraining is worse than we thought. Maybe other feature transformations, such as taking the log, would help. We could try with just a subset of features rather than using all of them as inputs.

To perform the median/mean comparison, we copied and pasted a lot of code just to change the function for imputing missing values. It would make more sense to write a function that performed the sequence of steps:

1. impute missing values
2. scale the features
3. train a model
4. calculate model performance

These are common steps, and sklearn provides something much better than writing custom functions.

4.8.2 Pipelines¶

One of the most important and useful components of sklearn is the pipeline. In place of Pandas's fillna DataFrame method, there is sklearn's SimpleImputer. Remember the first linear model above performed the steps:

1. replace missing values with the median for each feature
2. scale the data to zero mean and unit variance
3. train a linear regression model

and all these steps were trained on the train split and then applied to the test split for assessment.

The pipeline below defines exactly those same steps. Crucially, the resultant Pipeline object has a fit() method and a predict() method, just like the LinearRegression() object itself. Just as we might create a linear regression model and train it with .fit() and predict with .predict(), we can wrap the entire process of imputing and feature scaling and regression in a single object you can train with .fit() and predict with .predict(). And that's basically a pipeline: a model on steroids.

4.8.2.1 Define the pipeline¶

4.8.2.2 Fit the pipeline¶

Here, a single call to the pipeline's fit() method combines the steps of learning the imputation (determining what values to use to fill the missing ones), the scaling (determining the mean to subtract and the variance to divide by), and then training the model. It does this all in the one call with the training data as arguments.

4.8.2.3 Make predictions on the train and test sets¶

4.8.2.4 Assess performance¶

And compare with our earlier (non-pipeline) result:

Compare with our earlier result:

Compare with our earlier result:

These results confirm the pipeline is doing exactly what's expected, and results are identical to our earlier steps. This allows we to move faster but with confidence.

4.9 Refining The Linear Model¶

We suspected the model was overfitting. This is no real surprise given the number of features we blindly used. It's likely a judicious subset of features would generalize better. sklearn has a number of feature selection functions available. The one we'll use here is SelectKBest which, as we might guess, selects the k best features. We can read about SelectKBest here. f_regression is just the score function We're using because we're performing regression. It's important to choose an appropriate one for our machine learning task.

4.9.1 Define the pipeline¶

Redefine our pipeline to include this feature selection step:

4.9.2 Fit the pipeline¶

4.9.3 Assess performance on the train and test set¶

This has made things worse! Clearly selecting a subset of features has an impact on performance. SelectKBest defaults to k=10. Let's create a new pipeline with a different value of k:

4.9.4 Define a new pipeline to select a different number of features¶

4.9.5 Fit the pipeline¶

4.9.6 Assess performance on train and test data¶

We could keep going, trying different values of k, training a model, measuring performance on the test set, and then picking the model with the best test set performance. There's a fundamental problem with this approach: we're tuning the model to the arbitrary test set! If we continue this way we'll end up with a model works well on the particular quirks of our test set but fails to generalize to new data. The whole point of keeping a test set is for it to be a set of unseen data on which to test performance.

The way around this is a technique called cross-validation. We partition the training set into k folds, train our model on k-1 of those folds, and calculate performance on the fold not used in training. This procedure then cycles through k times with a different fold held back each time. Thus we end up building k models on k sets of data with k estimates of how the model performs on unseen data but without having to touch the test set.

4.9.7 Assessing performance using cross-validation¶

Without using the same random state for initializing the CV folds, our actual numbers will be different.

These results highlight that assessing model performance in inherently open to variability. We'll get different results depending on the quirks of which points are in which fold. An advantage of this is that you can also obtain an estimate of the variability, or uncertainty, in our performance estimate.

4.9.8 Hyperparameter search using GridSearchCV¶

Pulling the above together, we have:

• a pipeline that
  • imputes missing values
  • scales the data
  • selects the k best features
  • trains a linear regression model
• a technique (cross-validation) for estimating model performance

Now we will use cross-validation for multiple values of k, and then use cross-validation to pick the value of k that gives the best performance. make_pipeline automatically names each step in lowercase. Parameters of each step are then accessed by appending a double underscore followed by the parameter name. We know the name of the step will be 'selectkbest', and we know the parameter is 'k'.

We can also list the names of all the parameters in a pipeline as follows:

The above can be particularly useful as our pipelines becomes more complex (we can even nest pipelines within pipelines).

Now we have a range of k to investigate. Is 1 feature best? 2? 3? 4? All of them? We could write a for loop and iterate over each possible value, doing all the housekeeping ourselves to track the best value of k. But this is a common task, so there's a built in function in sklearn. This is GridSearchCV.

This takes the pipeline object, in fact it takes anything with a .fit() and .predict() method. In simple cases with no feature selection or imputation or feature scaling etc. we may see the classifier or regressor object itself directly passed into GridSearchCV. The other key input is the set of parameters and values to search over. Optional parameters include the cross-validation strategy and number of CPUs to use.

The above suggests a good value for k is 8. There was an initial rapid increase with k, followed by a slow decline. Also noticeable is the variance of the results greatly increase above k=8. As we increasingly overfit, expect greater swings in performance as different points move in and out of the train/test folds.

Which features were most useful? Step into our best model, shown below. Starting with the fitted grid search object, you get the best estimator, then the named step 'selectkbest', for which you can its get_support() method for a logical mask of the features selected.

Similarly, instead of using the 'selectkbest' named step, we access the named step for the linear regression model and, from that, grab the model coefficients via its coef_ attribute:

These results suggest that vertical drop is our biggest positive feature. This makes intuitive sense and is consistent with what you saw during the EDA work. Also, we see the area covered by snow making equipment is a strong positive as well. People like guaranteed skiing! The skiable terrain area is negatively associated with ticket price! This seems odd. People will pay less for larger resorts? There could be all manner of reasons for this. It could be an effect whereby larger resorts can host more visitors at any one time and so can charge less per ticket.

As has been mentioned previously, the data are missing information about visitor numbers. Bear in mind, the coefficient for skiable terrain is negative for this model. For example, if you kept the total number of chairs and fastQuads constant, but increased the skiable terrain extent, you might imagine the resort is worse off because the chairlift capacity is stretched thinner.

4.10 Random Forest Model¶

A model that can work very well in a lot of cases is the random forest. For regression, this is provided by sklearn's RandomForestRegressor class.

Time to stop the bad practice of repeatedly checking performance on the test split. Instead, go straight from defining the pipeline to assessing performance using cross-validation. cross_validate will perform the fitting as part of the process. This uses the default settings for the random forest so you'll then proceed to investigate some different hyperparameters.

4.10.1 Define the pipeline¶

4.10.2 Fit and assess performance using cross-validation¶

4.10.3 Hyperparameter search using GridSearchCV¶

Random forest has a number of hyperparameters that can be explored, however, here we'll limit ourselves to exploring some different values for the number of trees. We'll try it with and without feature scaling, and try both the mean and median as strategies for imputing missing values.

It looks like imputing with the median helps, but scaling the features doesn't.

We've marginally improved upon the default CV results. Random forest has many more hyperparameters you could tune, but we won't dive into that here.

Encouragingly, the dominant top four features are in common with our linear model:

• fastQuads
• Runs
• Snow Making_ac
• vertical_drop

4.11 Final Model Selection¶

Time to select our final model to use for further business modeling! It would be good to revisit the above model selection; there is undoubtedly more that could be done to explore possible hyperparameters. It would also be worthwhile to investigate removing the least useful features. Gathering or calculating, and storing, features adds business cost and dependencies, so if features genuinely are not needed they should be removed. Building a simpler model with fewer features can also have the advantage of being easier to sell (and/or explain) to stakeholders. Certainly there seem to be four strong features here and so a model using only those would probably work well. However, we want to explore some different scenarios where other features vary so keep the fuller model for now. The business is waiting for this model and we have something that we have confidence in to be much better than guessing with the average price.

Or, rather, we have two "somethings". We built a best linear model and a best random forest model. We need to finally choose between them. We can calculate the mean absolute error using cross-validation. Although cross-validate defaults to the $R^2$ metric for scoring regression, we can specify the mean absolute error as an alternative via the scoring parameter.

4.11.1 Linear regression model performance¶

4.11.2 Random forest regression model performance¶

4.11.3 Conclusion¶

The random forest model has a lower cross-validation mean absolute error by almost ~$1. It also exhibits less variability. Verifying performance on the test set produces performance consistent with the cross-validation results.

4.12 Data quantity assessment¶

Finally, we need to advise the business whether it needs to undertake further data collection. Would more data be useful? We're often led to believe more data is always good, but gathering data invariably has a cost associated with it. Assess this trade off by seeing how performance varies with differing data set sizes. The learning_curve function does this conveniently.

This shows that you seem to have plenty of data. There's an initial rapid improvement in model scores as one would expect, but it's essentially levelled off by around a sample size of 40-50.

4.13 Save best model object from pipeline¶