In this tutorial we will use the EasyVVUQ GridSampler
to perform a grid search on the hyperparameters of a simple Keras neural network model, trained to recognize hand-written digits. This is the famous MNIST data set, of which 4 input features (of size 28 x 28) are show below. These are fed into a standard feed-forward neural network, which will predict the label 0-9.
Note: This tutorial always runs on the localhost. One possibility of performing the grid search on a remote supercomputer involves the use of FabSim, see the hyperparameter_tuning_tutorial_with_fabsim.ipynb
notebook tutorial.
The (Keras) neural network script is located in mnist/keras_mnist.template
, which will form the input template for the EasyVVUQ encoder. We will assume you are familiar with the basic EasyVVUQ building blocks. If not, you can look at the basic tutorial.
We need EasyVVUQ, TensorFlow and the TensorFlow data sets to execute this tutorial. If you need to install these, uncomment the corresponding line below.
# !pip install easyvvuq
# !pip install tensorflow
# !pip install tensorflow_datasets
import easyvvuq as uq
import os
import numpy as np
from easyvvuq.actions import CreateRunDirectory, Encode, Decode, ExecuteLocal, Actions
We now set some flags:
# Work directory, where the EasyVVUQ directory will be placed
WORK_DIR = '/tmp'
# target output filename generated by the code
TARGET_FILENAME = 'output.csv'
# EasyVVUQ campaign name
CAMPAIGN_NAME = 'grid_test'
As is standard in EasyVVUQ, we now define the parameter space. In this case these are 4 hyperparameters. There is one hidden layer with n_neurons
neurons, a Dropout layer after the input and hidden layer, with dropout probability dropout_prob_in
and dropout_prob_hidden
respectively. We made the learning_rate
tuneable as well.
params = {}
params["n_neurons"] = {"type":"integer", "default": 32}
params["dropout_prob_in"] = {"type":"float", "default": 0.0}
params["dropout_prob_hidden"] = {"type":"float", "default": 0.0}
params["learning_rate"] = {"type":"float", "default": 0.001}
These 4 hyperparameter appear as flags in the input template mnist/keras_mnist.template
. Typically this is generated from an input file used by some simualtion code. In this case however, mnist/keras_mnist.template
is directly our Python script, with the hyperparameters replaced by flags. For instance:
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dropout($dropout_prob_in),
tf.keras.layers.Dense($n_neurons, activation='relu'),
tf.keras.layers.Dropout($dropout_prob_hidden),
tf.keras.layers.Dense(10)
])
is simply the neural network construction part with flags for the dropout probabilities and the number of neurons in the hidden layer. The encoder reads the flags and replaces them with numeric values, and it subsequently writes the corresponding target_filename=hyper_param_tune.py
:
encoder = uq.encoders.GenericEncoder('./mnist/keras_mnist.template', target_filename='hyper_param_tune.py')
What follows are standard steps in setting up an EasyVVUQ Campaign
# execute the runs locally
execute = ExecuteLocal('python3 hyper_param_tune.py')
# decode the output CSV files, with stored training and test accuracy values
output_columns = ["accuracy_train", "accuracy_test"]
decoder = uq.decoders.SimpleCSV(target_filename=TARGET_FILENAME, output_columns=output_columns)
# actions are 1) create run dirs, 2) encode input template, 3) execute runs, 4) decode output files
actions = Actions(CreateRunDirectory(root='/tmp', flatten=True), Encode(encoder), execute, Decode(decoder))
# create the EasyVVUQ main campaign object
campaign = uq.Campaign(
name=CAMPAIGN_NAME,
work_dir=WORK_DIR,
)
# add the param definitions and actions to the campaign
campaign.add_app(
name=CAMPAIGN_NAME,
params=params,
actions=actions
)
As with the uncertainty-quantification (UQ) samplers, the vary
is used to select which of the params
we actually vary. Unlike the UQ samplers we do not specify an input probability distribution. This being a grid search, we simply specify a list of values for each hyperparameter. Parameters not in vary
, but with a flag in the template, will be given the default value specified in params
.
vary = {"n_neurons": [64, 128], "learning_rate": [0.005, 0.01, 0.015]}
Note: we are mixing integer and floats in the vary
dict. Other data types (string, boolean) can also be used.
The vary
dict is passed to the Grid_Sampler
. As can be seen, it created a tensor product of all 1D points specified in vary
. If a single tensor product is not useful (e.g. because it creates combinations of parameters that do not makes sense), you can also pass a list of different vary
dicts. For even more flexibility you can also write the required parameter combinations to a CSV file, and pass it to the CSV_Sampler
instead.
# create an instance of the Grid Sampler
sampler = uq.sampling.Grid_Sampler(vary)
# Associate the sampler with the campaign
campaign.set_sampler(sampler)
# print the points
print("There are %d points:" % (sampler.n_samples()))
sampler.points
There are 6 points:
[array([[64, 0.005], [64, 0.01], [64, 0.015], [128, 0.005], [128, 0.01], [128, 0.015]], dtype=object)]
Run the actions
(create directories with hyper_param_tune.py
files in it)
###############################
# execute the defined actions #
###############################
campaign.execute().collate()
2023-03-02 11:34:46.627885: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:46.628054: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:46.630651: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:46.632733: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:46.635383: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:46.637862: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:47.056973: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.056973: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.057003: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:47.057002: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:47.057004: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.057004: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.057011: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.057011: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2023-03-02 11:34:47.057026: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:47.057026: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:47.057032: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:47.057032: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2023-03-02 11:34:48.728698: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.728836: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.728853: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:48.735720: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.735854: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.735872: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:48.747918: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.747999: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.748009: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:48.760968: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.761089: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.761108: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:48.803344: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.803468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.803486: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:48.828321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.828443: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory 2023-03-02 11:34:48.828462: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. 2023-03-02 11:34:52.172878: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.173245: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.173292: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.175002: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:52.177382: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.177406: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.177426: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.177712: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:52.179983: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.179983: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.180013: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.180013: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.180044: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.180049: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.180416: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:52.180415: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:52.188145: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.188183: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.188216: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.188602: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-03-02 11:34:52.193807: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-03-02 11:34:52.193845: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) 2023-03-02 11:34:52.193878: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (wouter-XPS-13-7390): /proc/driver/nvidia/version does not exist 2023-03-02 11:34:52.194312: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Epoch 1/6 Epoch 1/6 Epoch 1/6 Epoch 1/6 Epoch 1/6 Epoch 1/6 469/469 [==============================] - 15s 13ms/step - loss: 0.4033 - sparse_categorical_accuracy: 0.8792 - val_loss: 0.2163 - val_sparse_categorical_accuracy: 0.9375 Epoch 2/6 469/469 [==============================] - 15s 9ms/step - loss: 0.4464 - sparse_categorical_accuracy: 0.8710 - val_loss: 0.2590 - val_sparse_categorical_accuracy: 0.9264 Epoch 2/6 469/469 [==============================] - 15s 13ms/step - loss: 0.5436 - sparse_categorical_accuracy: 0.8448 - val_loss: 0.2959 - val_sparse_categorical_accuracy: 0.9169 469/469 [==============================] - 16s 14ms/step - loss: 0.4242 - sparse_categorical_accuracy: 0.8773 - val_loss: 0.2251 - val_sparse_categorical_accuracy: 0.9334 Epoch 2/6 469/469 [==============================] - 16s 14ms/step - loss: 0.3825 - sparse_categorical_accuracy: 0.8884 - val_loss: 0.2041 - val_sparse_categorical_accuracy: 0.9407 Epoch 2/6 469/469 [==============================] - 16s 14ms/step - loss: 0.5121 - sparse_categorical_accuracy: 0.8543 - val_loss: 0.2868 - val_sparse_categorical_accuracy: 0.9208 296/469 [=================>............] - ETA: 0s - loss: 0.2019 - sparse_categorical_accuracy: 0.9404Epoch 2/6 469/469 [==============================] - 2s 5ms/step - loss: 0.2190 - sparse_categorical_accuracy: 0.9385 - val_loss: 0.1947 - val_sparse_categorical_accuracy: 0.9430 243/469 [==============>...............] - ETA: 1s - loss: 0.2207 - sparse_categorical_accuracy: 0.9359Epoch 3/6 469/469 [==============================] - 3s 6ms/step - loss: 0.1926 - sparse_categorical_accuracy: 0.9432 - val_loss: 0.1692 - val_sparse_categorical_accuracy: 0.9502 Epoch 3/6 469/469 [==============================] - 4s 8ms/step - loss: 0.2056 - sparse_categorical_accuracy: 0.9409 - val_loss: 0.1634 - val_sparse_categorical_accuracy: 0.9507 Epoch 3/6 469/469 [==============================] - 4s 8ms/step - loss: 0.1765 - sparse_categorical_accuracy: 0.9489 - val_loss: 0.1405 - val_sparse_categorical_accuracy: 0.9589 469/469 [==============================] - 4s 9ms/step - loss: 0.2690 - sparse_categorical_accuracy: 0.9233 - val_loss: 0.2342 - val_sparse_categorical_accuracy: 0.9324 Epoch 3/6 469/469 [==============================] - 3s 7ms/step - loss: 0.1725 - sparse_categorical_accuracy: 0.9514 - val_loss: 0.1574 - val_sparse_categorical_accuracy: 0.9550 84/469 [====>.........................] - ETA: 2s - loss: 0.2292 - sparse_categorical_accuracy: 0.9323Epoch 4/6 469/469 [==============================] - 3s 7ms/step - loss: 0.1440 - sparse_categorical_accuracy: 0.9579 - val_loss: 0.1253 - val_sparse_categorical_accuracy: 0.9631 Epoch 4/6 221/469 [=============>................] - ETA: 1s - loss: 0.1635 - sparse_categorical_accuracy: 0.9525Epoch 3/6 113/469 [======>.......................] - ETA: 1s - loss: 0.1428 - sparse_categorical_accuracy: 0.9583Epoch 2/6 469/469 [==============================] - 4s 8ms/step - loss: 0.1552 - sparse_categorical_accuracy: 0.9557 - val_loss: 0.1408 - val_sparse_categorical_accuracy: 0.9578 455/469 [============================>.] - ETA: 0s - loss: 0.1411 - sparse_categorical_accuracy: 0.9602Epoch 4/6 469/469 [==============================] - 3s 7ms/step - loss: 0.2198 - sparse_categorical_accuracy: 0.9381 - val_loss: 0.1971 - val_sparse_categorical_accuracy: 0.9432 469/469 [==============================] - 3s 6ms/step - loss: 0.1407 - sparse_categorical_accuracy: 0.9603 - val_loss: 0.1350 - val_sparse_categorical_accuracy: 0.9623 469/469 [==============================] - 3s 6ms/step - loss: 0.1177 - sparse_categorical_accuracy: 0.9659 - val_loss: 0.1115 - val_sparse_categorical_accuracy: 0.9675 437/469 [==========================>...] - ETA: 0s - loss: 0.2782 - sparse_categorical_accuracy: 0.9216Epoch 5/6 469/469 [==============================] - 3s 7ms/step - loss: 0.1296 - sparse_categorical_accuracy: 0.9632 - val_loss: 0.1169 - val_sparse_categorical_accuracy: 0.9652 58/469 [==>...........................] - ETA: 1s - loss: 0.1079 - sparse_categorical_accuracy: 0.9674Epoch 4/6 469/469 [==============================] - 3s 6ms/step - loss: 0.2760 - sparse_categorical_accuracy: 0.9222 - val_loss: 0.2362 - val_sparse_categorical_accuracy: 0.9323 361/469 [======================>.......] - ETA: 0s - loss: 0.1000 - sparse_categorical_accuracy: 0.9704Epoch 4/6 469/469 [==============================] - 2s 5ms/step - loss: 0.1241 - sparse_categorical_accuracy: 0.9643 - val_loss: 0.1146 - val_sparse_categorical_accuracy: 0.9661 Epoch 5/6 109/469 [=====>........................] - ETA: 1s - loss: 0.2055 - sparse_categorical_accuracy: 0.9414Epoch 5/6 469/469 [==============================] - 2s 4ms/step - loss: 0.1003 - sparse_categorical_accuracy: 0.9705 - val_loss: 0.1062 - val_sparse_categorical_accuracy: 0.9665 147/469 [========>.....................] - ETA: 1s - loss: 0.1988 - sparse_categorical_accuracy: 0.9439Epoch 6/6 119/469 [======>.......................] - ETA: 1s - loss: 0.1241 - sparse_categorical_accuracy: 0.9641Epoch 3/6 469/469 [==============================] - 3s 6ms/step - loss: 0.1030 - sparse_categorical_accuracy: 0.9703 - val_loss: 0.1046 - val_sparse_categorical_accuracy: 0.9687 67/469 [===>..........................] - ETA: 2s - loss: 0.2337 - sparse_categorical_accuracy: 0.9356Epoch 5/6 469/469 [==============================] - 3s 7ms/step - loss: 0.1873 - sparse_categorical_accuracy: 0.9469 - val_loss: 0.1769 - val_sparse_categorical_accuracy: 0.9504 248/469 [==============>...............] - ETA: 1s - loss: 0.0810 - sparse_categorical_accuracy: 0.9773Epoch 5/6 469/469 [==============================] - 3s 6ms/step - loss: 0.1213 - sparse_categorical_accuracy: 0.9653 - val_loss: 0.1245 - val_sparse_categorical_accuracy: 0.9628 Epoch 6/6 469/469 [==============================] - 3s 6ms/step - loss: 0.0867 - sparse_categorical_accuracy: 0.9755 - val_loss: 0.0991 - val_sparse_categorical_accuracy: 0.9702 469/469 [==============================] - 3s 7ms/step - loss: 0.1038 - sparse_categorical_accuracy: 0.9709 - val_loss: 0.1101 - val_sparse_categorical_accuracy: 0.9670 469/469 [==============================] - 3s 6ms/step - loss: 0.2256 - sparse_categorical_accuracy: 0.9364 - val_loss: 0.2024 - val_sparse_categorical_accuracy: 0.9425 469/469 [==============================] - 3s 6ms/step - loss: 0.0848 - sparse_categorical_accuracy: 0.9758 - val_loss: 0.0951 - val_sparse_categorical_accuracy: 0.9703 469/469 [==============================] - 2s 3ms/step - loss: 0.0748 - sparse_categorical_accuracy: 0.9791 469/469 [==============================] - 2s 4ms/step - loss: 0.1070 - sparse_categorical_accuracy: 0.9691 - val_loss: 0.1089 - val_sparse_categorical_accuracy: 0.9682 Epoch 6/6 469/469 [==============================] - 2s 5ms/step - loss: 0.1628 - sparse_categorical_accuracy: 0.9542 - val_loss: 0.1559 - val_sparse_categorical_accuracy: 0.9545 Epoch 6/6 311/469 [==================>...........] - ETA: 0s - loss: 0.0950 - sparse_categorical_accuracy: 0.9729Epoch 4/6 79/79 [==============================] - 0s 3ms/step - loss: 0.0991 - sparse_categorical_accuracy: 0.9702 84/469 [====>.........................] - ETA: 1s - loss: 0.2051 - sparse_categorical_accuracy: 0.9418Epoch 6/6 469/469 [==============================] - 1s 3ms/step - loss: 0.0933 - sparse_categorical_accuracy: 0.9735 469/469 [==============================] - 3s 5ms/step - loss: 0.0900 - sparse_categorical_accuracy: 0.9743 - val_loss: 0.0938 - val_sparse_categorical_accuracy: 0.9720 469/469 [==============================] - 3s 5ms/step - loss: 0.1441 - sparse_categorical_accuracy: 0.9596 - val_loss: 0.1390 - val_sparse_categorical_accuracy: 0.9595 79/79 [==============================] - 0s 3ms/step - loss: 0.1089 - sparse_categorical_accuracy: 0.9682 469/469 [==============================] - 2s 5ms/step - loss: 0.1951 - sparse_categorical_accuracy: 0.9449 - val_loss: 0.1784 - val_sparse_categorical_accuracy: 0.9480 Epoch 5/6 469/469 [==============================] - 3s 5ms/step - loss: 0.0724 - sparse_categorical_accuracy: 0.9797 - val_loss: 0.0847 - val_sparse_categorical_accuracy: 0.9726 469/469 [==============================] - 2s 3ms/step - loss: 0.1309 - sparse_categorical_accuracy: 0.9635 79/79 [==============================] - 0s 3ms/step - loss: 0.1390 - sparse_categorical_accuracy: 0.9595 460/469 [============================>.] - ETA: 0s - loss: 0.1721 - sparse_categorical_accuracy: 0.9512
data_frame = campaign.get_collation_result()
data_frame
Display the hyperparameters with the maximum test accuracy
print("Best hyperparameters with %.2f%% test accuracy:" % (data_frame['accuracy_test'].max().values * 100,))
data_frame.loc[data_frame['accuracy_test'].idxmax()][vary.keys()]