import talos
import pandas as pd
%matplotlib inline
Scan()
Scan
objectReporting()
Evaluate()
Deploy()
Restore()
For this experiment, we're going to use the famous Iris dataset.
x, y = talos.templates.datasets.iris()
Talos works with any Keras model, without changing the structure of the model in anyway, or without introducing any new syntax. The below example shows clearly how this works.
For this example, we have to import two helper functions from Talos, one for early stopping callout, and the other for using normalized learning rate values. Because we might want to work on trying out several optimizers in a single scan, without normalization, inputting of the values would become cumbersome.
from talos.utils import lr_normalizer
Note that the only difference in the model below is how instead of using a label or value to define a given model parameter, we do it using a dictionary label. Also for optimizer we are using a learning rate parameter, which involves the use of two dictionary labels.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dropout, Dense
def iris_model(x_train, y_train, x_val, y_val, params):
model = Sequential()
model.add(Dense(params['first_neuron'],
input_dim=x_train.shape[1],
activation='relu'))
model.add(Dropout(params['dropout']))
model.add(Dense(y_train.shape[1],
activation=params['last_activation']))
model.compile(optimizer=params['optimizer'](lr=lr_normalizer(params['lr'], params['optimizer'])),
loss=params['loss'],
metrics=['acc'])
out = model.fit(x_train, y_train,
batch_size=params['batch_size'],
epochs=params['epochs'],
verbose=0,
validation_data=[x_val, y_val])
return out, model
In the last and final step, we're going to create the dictionary, which will then be passed on to Talos together with the model above. Here we have three different ways to input values:
For values we don't want to use, it's ok to set it as None.
NOTE: at this point you have to import from tensorflow.keras the optimizer, activations, and losses you want to scan for.
from tensorflow.keras.optimizers.legacy import Adam, Adagrad
from tensorflow.keras.activations import softmax
from tensorflow.keras.losses import categorical_crossentropy, LogCosh
p = {'lr': (0.1, 10, 10),
'first_neuron':[4, 8, 16, 32, 64, 128],
'batch_size': [2, 3, 4],
'epochs': [200],
'dropout': (0, 0.40, 10),
'optimizer': [Adam, Adagrad],
'loss': ['categorical_crossentropy'],
'last_activation': ['softmax'],
'weight_regulizer': [None]}
Scan()
¶Now we are ready to run the model based on the parameters and the layer configuration above. The exact same process would apply with any other model, just make sure to pass the model function name in the Scan() command as in the below example. To get started quickly, we're going to invoke the 'grid_downsample' parameter to 1/100 of the entire permutations.
scan_object = talos.Scan(x,
y,
params=p,
model=iris_model,
experiment_name='iris',
fraction_limit=.001)
Scan
object ¶# accessing the results data frame
scan_object.data.head()
# accessing epoch entropy values for each round
scan_object.learning_entropy
# access the summary details
scan_object.details
In addition to statistics and meta-data related with the Scan, the used data (x and y) together with the saved model and model weights for each hyperparameter permutation is stored in the Scan object.
# accessing the saved models
scan_object.saved_models
# accessing the saved weights for models
scan_object.saved_weights
The Scan object can be further used, and is required, as input for Predict(), Evaluate(), and Deploy(). More about this in the corresponding sections below.
Reporting()
¶In the Scan process, the results are stored round-by-round in the corresponding experiment log which is a .csv file stored in the present working directory. The Reporting() accepts as its source either a file name, or the Scan object.
# use Scan object as input
analyze_object = talos.Analyze(scan_object)
# access the dataframe with the results
analyze_object.data
# get the number of rounds in the Scan
analyze_object.rounds()
# get the highest result for any metric
analyze_object.high('val_acc')
# get the round with the best result
analyze_object.rounds2high('val_acc')
# get the best paramaters
analyze_object.best_params('val_acc', ['acc', 'loss', 'val_loss'])
# get correlation for hyperparameters against a metric
analyze_object.correlate('val_loss', ['acc', 'loss', 'val_loss'])
In addition to the key obsevations, several useful plots are available for analysis of the results.
# a regression plot for two dimensions
analyze_object.plot_regs('val_acc', 'val_loss')
# line plot
analyze_object.plot_line('val_acc')
# up to two dimensional kernel density estimator
analyze_object.plot_kde('val_acc')
# a simple histogram
analyze_object.plot_hist('val_acc', bins=50)
# heatmap correlation
analyze_object.plot_corr('val_loss', ['acc', 'loss', 'val_loss'])
# a four dimensional bar grid
analyze_object.plot_bars('batch_size', 'val_acc', 'first_neuron', 'lr')
Evaluate()
¶Models can be evaluated with Evaluate()
against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the Scan
process and only exposed into Evaluate once one or more candidate models have been identified.
evaluate_object = talos.Evaluate(scan_object)
evaluate_object.evaluate(x, y, folds=10, metric='val_acc', task='multi_label')
Once a sufficiently performing model have been found, a deployment package can be easily created.
Deploy()
¶Once the right model or models have been found, you can create a deployment package with Deploy()
which is then easy to transfer to a production or other environment, send via email, or upload to shared remote location. Best model is automatically chosen based on a given metric ('val_acc' by default).
The Deploy package is a zip file that consist of:
The Deploy
package can be easily restored with Restore()
which is covered in the next section.
talos.Deploy(scan_object=scan_object, model_name='iris_deploy', metric='val_acc');
Restore()
¶Models can be evaluated with Evaluate()
against a k-fold cross-validation. Ideally at least 50% of the data, or more if possible, is kept completely out of the Scan
process and only exposed into Evaluate once one or more candidate models have been identified.
iris = talos.Restore('iris_deploy.zip')
The Restore
object now consists of the assets from the Scan object originally associated with the experiment, together with the model that had been picked as 'best'. The model can be immediately used for making prediction, or use in any other other way Keras model objects can be used.
# make predictions with the model
iris.model.predict(x)
In addition, for book keeping purpose, and for simplicity of sharing models with team members and other stakeholders, various attributes are included in the Restore
object:
# get the meta-data for the experiment
iris.details
# get the hyperparameter space boundary
iris.params
# sample of x and y data
iris.x
iris.y
# the results dataframe
iris.results
Back to the repository page >> http://github.com/autonomio/talos