The first three chapters focused on model validation techniques. In chapter 4 we apply these techniques, specifically cross-validation, while learning about hyperparameter tuning. After all, model validation makes tuning possible and helps us select the overall best model. This is the Summary of lecture "Model Validation in Python", via datacamp.
import pandas as pd
import numpy as np
For a school assignment, your professor has asked your class to create a random forest model to predict the average test score for the final exam.
After developing an initial random forest model, you are unsatisfied with the overall accuracy. You realize that there are too many hyperparameters to choose from, and each one has a lot of possible values. You have decided to make a list of possible ranges for the hyperparameters you might use in your next model.
Your professor has provided de-identified data for the last ten quizzes to act as the training data. There are 30 students in your class.
from sklearn.ensemble import RandomForestRegressor
rfr = RandomForestRegressor(n_estimators='warn', max_features='auto', random_state=1111)
rfr.get_params()
{'bootstrap': True, 'ccp_alpha': 0.0, 'criterion': 'mse', 'max_depth': None, 'max_features': 'auto', 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 2, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 'warn', 'n_jobs': None, 'oob_score': False, 'random_state': 1111, 'verbose': 0, 'warm_start': False}
# Maximum Depth
max_depth = [4, 8, 12]
# Minimum samples for a split
min_samples_split = [2, 5, 10]
# Max features
max_features = [4, 6, 8, 10]
Hyperparameter tuning requires selecting parameters to tune, as well the possible values these parameters can be set to.
You have just finished creating a list of hyperparameters and ranges to use when tuning a predictive model for an assignment.
import random
# Fill in rfr using your variables
rfr = RandomForestRegressor(
n_estimators=100,
max_depth=random.sample(max_depth, 1)[0],
min_samples_split=random.sample(min_samples_split, 1)[0],
max_features = random.sample(max_features, 1)[0]
)
# Print out the parameters
print(rfr.get_params())
{'bootstrap': True, 'ccp_alpha': 0.0, 'criterion': 'mse', 'max_depth': 8, 'max_features': 6, 'max_leaf_nodes': None, 'max_samples': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 5, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 100, 'n_jobs': None, 'oob_score': False, 'random_state': None, 'verbose': 0, 'warm_start': False}
Last semester your professor challenged your class to build a predictive model to predict final exam test scores. You tried running a few different models by randomly selecting hyperparameters. However, running each model required you to code it individually.
After learning about RandomizedSearchCV()
, you're revisiting your professors challenge to build the best model. In this exercise, you will prepare the three necessary inputs for completing a random search.
from sklearn.metrics import make_scorer, mean_squared_error
# Finish the dictionary by adding the max_depth parameter
param_dist = {
"max_depth": [2, 4, 6, 8],
"max_features": [2, 4, 6, 8, 10],
"min_samples_split": [2, 4, 8, 16]
}
# Create a random forest regression model
rfr = RandomForestRegressor(n_estimators=10, random_state=1111)
# Create a scorer to use (use the mean squared error)
scorer = make_scorer(mean_squared_error)
To use RandomizedSearchCV()
, you need a distribution dictionary, an estimator, and a scorer—once you've got these, you can run a random search to find the best parameters for your model.
You are hoping that using a random search algorithm will help you improve predictions for a class assignment. You professor has challenged your class to predict the overall final exam average score.
In preparation for completing a random search, you have created:
param_dist
: the hyperparameter distributionsrfr
: a random forest regression modelscorer
: a scoring method to usefrom sklearn.model_selection import RandomizedSearchCV
# Build a random search using param_dist, rfr, and scorer
random_search = RandomizedSearchCV(estimator=rfr,
param_distributions=param_dist,
n_iter=10,
cv=5,
scoring=scorer
)
Although it takes a lot of steps, hyperparameter tuning with random search is well worth it and can improve the accuracy of your models. Plus, you are already using cross-validation to validate your best model.
You are in a competition at work to build the best model for predicting the winner of a Tic-Tac-Toe game. You already ran a random search and saved the results of the most accurate model to rs
.
tic_tac_toe = pd.read_csv('./dataset/tic-tac-toe.csv')
# Create dummy variables using pandas
X = pd.get_dummies(tic_tac_toe.iloc[:, 0:9])
y = tic_tac_toe.iloc[:, 9]
y = tic_tac_toe['Class'].apply(lambda x: 1 if x == 'positive' else 0)
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(max_features='auto')
param_dist = {
'max_depth': [2, 4, 8, 12],
'min_samples_split': [2, 4, 6, 8],
'n_estimators':[10, 20, 30]
}
rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist, n_iter=10,
cv=5, scoring=None, random_state=1111)
rs.fit(X, y)
RandomizedSearchCV(cv=5, error_score=nan, estimator=RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion='gini', max_depth=None, max_features='auto', max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=None, oob_score=False, random_state=None, verbose=0, warm_start=False), iid='deprecated', n_iter=10, n_jobs=None, param_distributions={'max_depth': [2, 4, 8, 12], 'min_samples_split': [2, 4, 6, 8], 'n_estimators': [10, 20, 30]}, pre_dispatch='2*n_jobs', random_state=1111, refit=True, return_train_score=False, scoring=None, verbose=0)
rs.best_params_
{'n_estimators': 20, 'min_samples_split': 4, 'max_depth': 12}
From the result of random search, we can find that the optimal parameter is:
{'n_estimators': 20, 'min_samples_split': 4, 'max_depth': 12}
Your boss has offered to pay for you to see three sports games this year. Of the 41 home games your favorite team plays, you want to ensure you go to three home games that they will definitely win. You build a model to decide which games your team will win.
To do this, you will build a random search algorithm and focus on model precision (to ensure your team wins). You also want to keep track of your best model and best parameters, so that you can use them again next year (if the model does well, of course).
sports = pd.read_csv('./dataset/sports.csv')
X = sports.drop('win', axis=1)
y = sports['win']
rfc = RandomForestClassifier()
param_dist = {
'max_depth': range(2, 12, 2),
'min_samples_split': range(2, 12, 2),
'n_estimators': [10, 25, 50]
}
from sklearn.metrics import precision_score
# Create a precision scorer
precision = make_scorer(precision_score)
# Finalize the random search
rs = RandomizedSearchCV(estimator=rfc, param_distributions=param_dist,
scoring=precision, cv=5, n_iter=10,
random_state=1111)
rs.fit(X, y)
# Print the mean test scores:
print('The accuracy for each run was: {}.'.format(rs.cv_results_['mean_test_score']))
# Print the best model scores:
print('The best accuracy for a single model was: {}'.format(rs.best_score_))
The accuracy for each run was: [0.91107742 0.75556059 0.68148435 0.88886879 0.88136354 0.88578829 0.68821403 0.81709152 0.87505491 0.90675415]. The best accuracy for a single model was: 0.9110774179109917
Your model's precision was 91%! The best model accurately predicts a winning game 91% of the time. If you look at the mean test scores, you can tell some of the other parameter sets did really poorly. Also, since you used cross-validation, you can be confident in your predictions. Well done!