H2O GBM Tuning Tutorial for Python¶

Arno Candel, PhD, Chief Architect, H2O.ai¶

Ported to Python by Navdeep Gill, M.S., Hacker/Data Scientist, H2O.ai¶

In this tutorial, we show how to build a well-tuned H2O GBM model for a supervised classification task. We specifically don't focus on feature engineering and use a small dataset to allow you to reproduce these results in a few minutes on a laptop. This script can be directly transferred to datasets that are hundreds of GBs large and H2O clusters with dozens of compute nodes.

You can download the source from H2O's github repository.

Ports to R Markdown and Flow UI (now part of Example Flows) are available as well.

Installation of the H2O Python Package¶

Either download H2O from H2O.ai's website or install the latest version of H2O into Python with the following set of commands:

Install dependencies from command line (prepending with sudo if needed):

[sudo] pip install -U requests
[sudo] pip install -U tabulate
[sudo] pip install -U future
[sudo] pip install -U six

The following command removes the H2O module for Python.

[sudo] pip uninstall h2o

Next, use pip to install this version of the H2O Python module.

[sudo] pip install http://h2o-release.s3.amazonaws.com/h2o/rel-zahradnik/3/Python/h2o-3.30.0.3-py2.py3-none-any.whl

Launch an H2O cluster on localhost¶

Import the data into H2O¶

Everything is scalable and distributed from now on. All processing is done on the fully multi-threaded and distributed H2O Java-based backend and can be scaled to large datasets on large compute clusters. Here, we use a small public dataset (Titanic), but you can use datasets that are hundreds of GBs large.

From now on, everything is generic and directly applies to most datasets. We assume that all feature engineering is done at this stage and focus on model tuning. For multi-class problems, you can use h2o.logloss() or h2o.confusion_matrix() instead of h2o.auc() and for regression problems, you can use h2o.mean_residual_deviance() or h2o.mse().

Split the data for Machine Learning¶

We split the data into three pieces: 60% for training, 20% for validation, 20% for final testing. Here, we use random splitting, but this assumes i.i.d. data. If this is not the case (e.g., when events span across multiple rows or data has a time structure), you'll have to sample your data non-randomly.

Establish baseline performance¶

As the first step, we'll build some default models to see what accuracy we can expect. Let's use the AUC metric for this demo, but you can use h2o.logloss() and stopping_metric="logloss" as well. It ranges from 0.5 for random models to 1 for perfect models.

The first model is a default GBM, trained on the 60% training split

The AUC is 95%, so this model is highly predictive!

The second model is another default GBM, but trained on 80% of the data (here, we combine the training and validation splits to get more training data), and cross-validated using 4 folds. Note that cross-validation takes longer and is not usually done for really large datasets.

We see that the cross-validated performance is similar to the validation set performance:

Next, we train a GBM with "I feel lucky" parameters. We'll use early stopping to automatically tune the number of trees using the validation AUC. We'll use a lower learning rate (lower is always better, just takes more trees to converge). We'll also use stochastic sampling of rows and columns to (hopefully) improve generalization.

This model doesn't seem to be better than the previous models:

For this small dataset, dropping 20% of observations per tree seems too aggressive in terms of adding regularization. For larger datasets, this is usually not a bad idea. But we'll let this parameter tune freshly below, so no worries.

Hyper-Parameter Search¶

Next, we'll do real hyper-parameter optimization to see if we can beat the best AUC so far (around 94%).

The key here is to start tuning some key parameters first (i.e., those that we expect to have the biggest impact on the results). From experience with gradient boosted trees across many datasets, we can state the following "rules":

1. Build as many trees (ntrees) as it takes until the validation set error starts increasing.
2. A lower learning rate (learn_rate) is generally better, but will require more trees. Using learn_rate=0.02and learn_rate_annealing=0.995 (reduction of learning rate with each additional tree) can help speed up convergence without sacrificing accuracy too much, and is great to hyper-parameter searches. For faster scans, use values of 0.05 and 0.99 instead.
3. The optimum maximum allowed depth for the trees (max_depth) is data dependent, deeper trees take longer to train, especially at depths greater than 10.
4. Row and column sampling (sample_rate and col_sample_rate) can improve generalization and lead to lower validation and test set errors. Good general values for large datasets are around 0.7 to 0.8 (sampling 70-80 percent of the data) for both parameters. Column sampling per tree (col_sample_rate_per_tree) can also be tuned. Note that it is multiplicative with col_sample_rate, so setting both parameters to 0.8 results in 64% of columns being considered at any given node to split.
5. For highly imbalanced classification datasets (e.g., fewer buyers than non-buyers), stratified row sampling based on response class membership can help improve predictive accuracy. It is configured with sample_rate_per_class (array of ratios, one per response class in lexicographic order).
6. Most other options only have a small impact on the model performance, but are worth tuning with a Random hyper-parameter search nonetheless, if highest performance is critical.

First we want to know what value of max_depth to use because it has a big impact on the model training time and optimal values depend strongly on the dataset. We'll do a quick Cartesian grid search to get a rough idea of good candidate max_depth values. Each model in the grid search will use early stopping to tune the number of trees using the validation set AUC, as before. We'll use learning rate annealing to speed up convergence without sacrificing too much accuracy.

It appears that max_depth values of 5 to 13 are best suited for this dataset, which is unusally deep!

Now that we know a good range for max_depth, we can tune all other parameters in more detail. Since we don't know what combinations of hyper-parameters will result in the best model, we'll use random hyper-parameter search to "let the machine get luckier than a best guess of any human".

We can see that the best models have even better validation AUCs than our previous best models, so the random grid search was successful!

You can also see the results of the grid search in Flow:

Model Inspection and Final Test Set Scoring¶

Let's see how well the best model of the grid search (as judged by validation set AUC) does on the held out test set:

Good news. It does as well on the test set as on the validation set, so it looks like our best GBM model generalizes well to the unseen test set:

We can inspect the winning model's parameters:

Now we can confirm that these parameters are generally sound, by building a GBM model on the whole dataset (instead of the 60%) and using internal 5-fold cross-validation (re-using all other parameters including the seed):

It looks like the winning model performs slightly better on the validation and test sets than during cross-validation on the training set as the mean AUC on the 5 folds is estimated to be only 97.4%, but with a fairly large standard deviation of 0.9%. For small datasets, such a large variance is not unusual. To get a better estimate of model performance, the Random hyper-parameter search could have used nfolds = 5 (or 10, or similar) in combination with 80% of the data for training (i.e., not holding out a validation set, but only the final test set). However, this would take more time, as nfolds+1 models will be built for every set of parameters.

Instead, to save time, let's just scan through the top 5 models and cross-validate their parameters with nfolds=5 on the entire dataset:

The avid reader might have noticed that we just implicitly did further parameter tuning using the "final" test set (which is part of the entire dataset df), which is not good practice - one is not supposed to use the "final" test set more than once. Hence, we're not going to pick a different "best" model, but we're just learning about the variance in AUCs. It turns out, for this tiny dataset, that the variance is rather large, which is not surprising.

Keeping the same "best" model, we can make test set predictions as follows:

Note that the label (survived or not) is predicted as well (in the first predict column), and it uses the threshold with the highest F1 score (here: 0.528098) to make labels from the probabilities for survival (p1). The probability for death (p0) is given for convenience, as it is just 1-p1.

You can also see the "best" model in more detail in Flow:

The model and the predictions can be saved to file as follows:

The model can also be exported as a plain old Java object (POJO) for H2O-independent (standalone/Storm/Kafka/UDF) scoring in any Java environment.

/*
 Licensed under the Apache License, Version 2.0
    http://www.apache.org/licenses/LICENSE-2.0.html

  AUTOGENERATED BY H2O at 2016-07-17T18:38:50.337-07:00
  3.8.3.3

  Standalone prediction code with sample test data for GBMModel named final_grid_model_45

  How to download, compile and execute:
      mkdir tmpdir
      cd tmpdir
      curl http://127.0.0.1:54321/3/h2o-genmodel.jar > h2o-genmodel.jar
      curl http://127.0.0.1:54321/3/Models.java/final_grid_model_45 > final_grid_model_45.java
      javac -cp h2o-genmodel.jar -J-Xmx2g -J-XX:MaxPermSize=128m final_grid_model_45.java

     (Note:  Try java argument -XX:+PrintCompilation to show runtime JIT compiler behavior.)
*/
import java.util.Map;
import hex.genmodel.GenModel;
import hex.genmodel.annotations.ModelPojo;

...
class final_grid_model_45_Tree_0_class_0 {
  static final double score0(double[] data) {
    double pred =      (Double.isNaN(data[1]) || !GenModel.bitSetContains(GRPSPLIT0, 0, data[1 /* sex */]) ? 
         (Double.isNaN(data[7]) || !GenModel.bitSetContains(GRPSPLIT1, 13, data[7 /* cabin */]) ? 
             (Double.isNaN(data[7]) || !GenModel.bitSetContains(GRPSPLIT2, 9, data[7 /* cabin */]) ? 
                 (Double.isNaN(data[7]) || !GenModel.bitSetContains(GRPSPLIT3, 9, data[7 /* cabin */]) ? 
                     (data[2 /* age */] <1.4174492f ? 
                        0.13087687f : 
                         (Double.isNaN(data[7]) || !GenModel.bitSetContains(GRPSPLIT4, 9, data[7 /* cabin */]) ? 
                             (Double.isNaN(data[3]) || data[3 /* sibsp */] <1.000313f ? 
                                 (data[6 /* fare */] <7.91251f ? 
                                     (Double.isNaN(data[5]) || data[5 /* ticket */] <368744.5f ? 
                                        -0.08224204f : 
                                         (Double.isNaN(data[2]) || data[2 /* age */] <13.0f ? 
                                            -0.028962314f : 
                                            -0.08224204f)) : 
                                     (Double.isNaN(data[7]) || !GenModel.bitSetContains(GRPSPLIT5, 9, data[7 /* cabin */]) ? 
                                         (data[6 /* fare */] <7.989957f ? 
                                             (Double.isNaN(data[3]) || data[3 /* sibsp */] <0.0017434144f ? 
                                                0.07759714f : 
                                                0.13087687f) : 
                                             (data[6 /* fare */] <12.546303f ? 
                                                -0.07371729f : 
                                                 (Double.isNaN(data[4]) || data[4 /* parch */] <1.0020853f ? 
                                                    -0.037374903f : 
                                                    -0.08224204f))) : 
                                        0.0f)) : 
                                -0.08224204f) : 
                            0.0f)) : 
                    0.0f) : 
                -0.08224204f) : 
            -0.08224204f) :  
...

Ensembling Techniques¶

After learning above that the variance of the test set AUC of the top few models was rather large, we might be able to turn this into our advantage by using ensembling techniques. The simplest one is taking the average of the predictions (survival probabilities) of the top k grid search model predictions (here, we use k=10):

We now have a blended probability of survival for each person on the Titanic.

We can bring those ensemble predictions to our Python session's memory space and use other Python packages.

This simple blended ensemble test set prediction has an even higher AUC than the best single model, but we need to do more validation studies, ideally using cross-validation. We leave this as an exercise for the reader - take the parameters of the top 10 models, retrain them with nfolds=5 on the full dataset, set keep_holdout_predictions=True and sum up their predicted probabilities, then score that with sklearn's roc_auc_score as shown above.

For more sophisticated ensembling approaches, such as stacking via a superlearner, we refer to the H2O Ensemble github page.

Summary¶

We learned how to build H2O GBM models for a binary classification task on a small but realistic dataset with numerical and categorical variables, with the goal to maximize the AUC (ranges from 0.5 to 1). We first established a baseline with the default model, then carefully tuned the remaining hyper-parameters without "too much" human guess-work. We used both Cartesian and Random hyper-parameter searches to find good models. We were able to get the AUC on a holdout test set from 95% range with the default model to 97% range after tuning, and to above 98% with some simple ensembling technique known as blending. We performed simple cross-validation variance analysis to learn that results were slightly "lucky" due to the specific train/valid/test set splits, and settled to expect 97% AUCs instead.

Note that this script and the findings therein are directly transferrable to large datasets on distributed clusters including Spark/Hadoop environments.

More information can be found here http://www.h2o.ai/docs/.