In [5]:
import talos as ta
import pandas as pd

%matplotlib inline

Table of Contents

1. Data Preparation

For this experiment, we're going to use the famous Iris dataset.

In [6]:
x, y = ta.datasets.iris()

2. Model Preparation

Talos works with any Keras model, without changing the structure of the model in anyway, or without introducing any new syntax. The below example shows clearly how this works.

For this example, we have to import two helper functions from Talos, one for early stopping callout, and the other for using normalized learning rate values. Because we might want to work on trying out several optimizers in a single scan, without normalization, inputting of the values would become cumbersome.

In [7]:
from talos.model.normalizers import lr_normalizer

Note that the only difference in the model below is how instead of using a label or value to define a given model parameter, we do it using a dictionary label. Also for optimizer we are using a learning rate parameter, which involves the use of two dictionary labels.

In [8]:
from keras.models import Sequential
from keras.layers import Dropout, Dense

def iris_model(x_train, y_train, x_val, y_val, params):
    model = Sequential()                            

    model.compile(optimizer=params['optimizer'](lr=lr_normalizer(params['lr'], params['optimizer'])),

    out =, y_train,
                    validation_data=[x_val, y_val])
    return out, model

3. Setting the Parameter Space Boundaries

In the last and final step, we're going to create the dictionary, which will then be passed on to Talos together with the model above. Here we have three different ways to input values:

  • as stepped ranges (min, max, steps)
  • as multiple values [in a list]
  • as a single value [in a list]

For values we don't want to use, it's ok to set it as None.

NOTE: at this point you have to import from Keras the optimizer, activations, and losses you want to scan for.

In [9]:
from keras.optimizers import Adam, Nadam
from keras.activations import softmax
from keras.losses import categorical_crossentropy, logcosh

p = {'lr': (0.1, 10, 10),
     'first_neuron':[4, 8, 16, 32, 64, 128],
     'batch_size': [2, 3, 4],
     'epochs': [200],
     'dropout': (0, 0.40, 10),
     'optimizer': [Adam, Nadam],
     'loss': ['categorical_crossentropy'],
     'last_activation': ['softmax'],
     'weight_regulizer': [None]}

4. Run the Hyperparameter Scan()

Now we are ready to run the model based on the parameters and the layer configuration above. The exact same process would apply with any other model, just make sure to pass the model function name in the Scan() command as in the below example. To get started quickly, we're going to invoke the 'grid_downsample' parameter to 1/100 of the entire permutations.

In [ ]:
h = ta.Scan(x, y, params=p,
  0%|          | 0/36 [00:00<?, ?it/s]

5. Access the results through the Scan object

In [ ]:
# accessing the results data frame

# accessing epoch entropy values for each round

# access the summary details

In addition to statistics and meta-data related with the Scan, the used data (x and y) together with the saved model and model weights for each hyperparameter permutation is stored in the Scan object.

In [ ]:
# accessing the saved models

# accessing the saved weights for models

The Scan object can be further used, and is required, as input for Predict(), Evaluate(), and Deploy(). More about this in the corresponding sections below.

6. Analysing the Scan results with Reporting()

In the Scan process, the results are stored round-by-round in the corresponding experiment log which is a .csv file stored in the present working directory. The Reporting() accepts as its source either a file name, or the Scan object.

In [ ]:
# use Scan object as input
r = ta.Reporting(h)

# use filename as input
r = ta.Reporting('iris_1.csv')
In [ ]:
# access the dataframe with the results
In [ ]:
# get the number of rounds in the Scan

# get the highest result ('val_acc' by default)

# get the highest result for any metric

# get the round with the best result

# get the best paramaters

# get correlation for hyperparameters against a metric

In addition to the key obsevations, several useful plots are available for analysis of the results.

In [ ]:
# a regression plot for two dimensions 

# line plot

# up to two dimensional kernel density estimator

# a simple histogram

# heatmap correlation

# a four dimensional bar grid
r.plot_bars('batch_size', 'val_acc', 'first_neuron', 'lr')

7. Evaluating Models with Evaluate()

Models can be evaluated with Evaluate() against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the Scan process and only exposed into Evaluate once one or more candidate models have been identified.

In [ ]:
e = ta.Evaluate(h)
e.evaluate(x, y, folds=10, average='macro')

Once a sufficiently performing model have been found, a deployment package can be easily created.

8. Deploying Models with Deploy()

Once the right model or models have been found, you can create a deployment package with Deploy() which is then easy to transfer to a production or other environment, send via email, or upload to shared remote location. Best model is automatically chosen based on a given metric ('val_acc' by default).

The Deploy package is a zip file that consist of:

  • details of the scan
  • model weights
  • model json
  • results of the experiment
  • sample of x data
  • sample of y data

The Deploy package can be easily restored with Restore() which is covered in the next section.

In [ ]:
ta.Deploy(h, 'iris');

9. Restoring Models with Restore()

Models can be evaluated with Evaluate() against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the Scan process and only exposed into Evaluate once one or more candidate models have been identified.

In [ ]:
iris = ta.Restore('')

The Restore object now consists of the assets from the Scan object originally associated with the experiment, together with the model that had been picked as 'best'. The model can be immediately used for making prediction, or use in any other other way Keras model objects can be used.

In [ ]:
# make predictions with the model

In addition, for book keeping purpose, and for simplicity of sharing models with team members and other stakeholders, various attributes are included in the Restore object:

In [ ]:
# get the meta-data for the experiment
In [ ]:
# get the hyperparameter space boundary
In [ ]:
# sample of x and y data
In [ ]:
# the results dataframe

Back to the repository page >>