H2O Tutorial¶

Author: Spencer Aiello

Contact: spencer@h2oai.com

This tutorial steps through a quick introduction to H2O's Python API. The goal of this tutorial is to introduce through a complete example H2O's capabilities from Python. Also, to help those that are accustomed to Scikit Learn and Pandas, the demo will be specific call outs for differences between H2O and those packages; this is intended to help anyone that needs to do machine learning on really Big Data make the transition. It is not meant to be a tutorial on machine learning or algorithms.

Detailed documentation about H2O's and the Python API is available at http://docs.h2o.ai.

Setting up your system for this demo¶

The following code creates two csv files using data from the Boston Housing dataset which is built into scikit-learn and adds them to the local directory

Enable inline plotting in the Jupyter Notebook

Intro to H2O Data Munging¶

Read csv data into H2O. This loads the data into the H2O column compressed, in-memory, key-value store.

View the top of the H2O frame.

View the bottom of the H2O Frame

Select a column

fr["VAR_NAME"]

Select a few columns

Select a subset of rows

Unlike in Pandas, columns may be identified by index or column name. Therefore, when subsetting by rows, you must also pass the column selection.

Key attributes: * columns, names, col_names * len, shape, dim, nrow, ncol * types

Note:

Since the data is not in local python memory there is no "values" attribute. If you want to pull all of the data into the local python memory then do so explicitly with h2o.export_file and reading the data into python memory from disk.

Select rows based on value

Boolean masks can be used to subselect rows based on a criteria.

Get summary statistics of the data and additional data distribution information.

Set up the predictor and response column names

Using H2O algorithms, it's easier to reference predictor and response columns by name in a single frame (i.e., don't split up X and y)

Machine Learning With H2O¶

H2O is a machine learning library built in Java with interfaces in Python, R, Scala, and Javascript. It is open source and well-documented.

Unlike Scikit-learn, H2O allows for categorical and missing data.

The basic work flow is as follows:

• Fit the training data with a machine learning algorithm
• Predict on the testing data

Simple model¶

The performance of the model can be checked using the holdout dataset

Train-Test Split¶

Instead of taking the first 400 observations for training, we can use H2O to create a random test train split of the data.

There was a massive jump in the R^2 value. This is because the original data is not shuffled.

Cross validation¶

H2O's machine learning algorithms take an optional parameter nfolds to specify the number of cross-validation folds to build. H2O's cross-validation uses an internal weight vector to build the folds in an efficient manner (instead of physically building the splits).

In conjunction with the nfolds parameter, a user may specify the way in which observations are assigned to each fold with the fold_assignment parameter, which can be set to either: * AUTO: Perform random assignment * Random: Each row has a equal (1/nfolds) chance of being in any fold. * Modulo: Observations are in/out of the fold based by modding on nfolds

However, you can still make use of the cross_val_score from Scikit-Learn

Cross validation: H2O and Scikit-Learn¶

You still must use H2O to make the folds. Currently, there is no H2OStratifiedKFold. Additionally, the H2ORandomForestEstimator is similar to the scikit-learn RandomForestRegressor object with its own train method.

There isn't much difference in the R^2 value since the fold strategy is exactly the same. However, there was a major difference in terms of computation time and memory usage.

Since the progress bar print out gets annoying let's disable that

Grid Search¶

Grid search in H2O is still under active development and it will be available very soon. However, it is possible to make use of Scikit's grid search infrastructure (with some performance penalties)

Randomized grid search: H2O and Scikit-Learn¶

If you have 0.16.1, then your system can't handle complex randomized grid searches (it works in every other version of sklearn, including the soon to be released 0.16.2 and the older versions).

The steps to perform a randomized grid search:

1. Import model and RandomizedSearchCV
2. Define model
3. Specify parameters to test
4. Define grid search object
5. Fit data to grid search object
6. Collect scores

All the steps will be repeated from above.

Because 0.16.1 is installed, we use scipy to define specific distributions

ADVANCED TIP:

Turn off reference counting for spawning jobs in parallel (n_jobs=-1, or n_jobs > 1). We'll turn it back on again in the aftermath of a Parallel job.

If you don't want to run jobs in parallel, don't turn off the reference counting.

Pattern is: >>> h2o.turn_off_ref_cnts() >>> .... parallel job .... >>> h2o.turn_on_ref_cnts()

We might be tempted to think that we just had a large improvement; however we must be cautious. The function below creates a more detailed report.

Based on the grid search report, we can narrow the parameters to search and rerun the analysis. The parameters below were chosen after a few runs:

Transformations¶

Rule of machine learning: Don't use your testing data to inform your training data. Unfortunately, this happens all the time when preparing a dataset for the final model. But on smaller datasets, you must be especially careful.

At the moment, there are no classes for managing data transformations. On the one hand, this requires the user to tote around some extra state, but on the other, it allows the user to be more explicit about transforming H2OFrames.

Basic steps:

0. Remove the response variable from transformations.
1. Import transformer
2. Define transformer
3. Fit train data to transformer
4. Transform test and train data
5. Re-attach the response variable.

First let's normalize the data using the means and standard deviations of the training data. Then let's perform a principal component analysis on the training data and select the top 5 components. Using these components, let's use them to reduce the train and test design matrices.

Normalize Data: Use the means and standard deviations from the training data.¶

Then, we can apply PCA and keep the top 5 components. A user warning is expected here.

Although this is MUCH simpler than keeping track of all of these transformations manually, it gets to be somewhat of a burden when you want to chain together multiple transformers.

Pipelines¶

"Tranformers unite!"

If your raw data is a mess and you have to perform several transformations before using it, use a pipeline to keep things simple.

Steps:

1. Import Pipeline, transformers, and model
2. Define pipeline. The first and only argument is a list of tuples where the first element of each tuple is a name you give the step and the second element is a defined transformer. The last step is optionally an estimator class (like a RandomForest).
3. Fit the training data to pipeline
4. Either transform or predict the testing data

This is so much easier!!!

But, wait a second, we did worse after applying these transformations! We might wonder how different hyperparameters for the transformations impact the final score.

Combining randomized grid search and pipelines¶

"Yo dawg, I heard you like models, so I put models in your models to model models."

Steps:

1. Import Pipeline, grid search, transformers, and estimators
2. Define pipeline
3. Define parameters to test in the form: "(Step name)__(argument name)" A double underscore separates the two words.
4. Define grid search
5. Fit to grid search

Currently Under Development (drop-in scikit-learn pieces): * Richer set of transforms (only PCA and Scale are implemented) * Richer set of estimators (only RandomForest is available) * Full H2O Grid Search

Other Tips: Model Save/Load¶

It is useful to save constructed models to disk and reload them between H2O sessions. Here's how: