Classification of Malicious URL using Deep Learning with Keras¶
University of Toronto School of Continuing Studies
SCS 3546 Deep Learning
Name:
Arjie CristobalStudent Number:
X261973
Introduction¶
Artificial Intelligence (AI) and cybersecurity are two of the most rapidly growing sectors in the technology industry.
The global AI in cybersecurity market was valued at $19.2 billion in 2022 , and is projected to reach $154.8 billion by 2032, growing at a CAGR of 23.6% from 2023 to 2032.
The future growth of both AI and cybersecurity is promising and will be critical in the future.
Objective¶
This study will explore a lightweight approach to identify and classify malicious URL using deep learning via Keras.
Dataset¶
University of New Brunswick
Canadian Institute for Cybersecurity
Loading and Preparing the dataset¶
This study reused the UrlDatasetLoader from the Machine Learning (ML) project Detection and categorization of malicious URLs for data cleaning and preparation. It is responsible on handling Null and NaN values, feature selections and anomaly detection.
The prepared dataset is then exported to CSV files and uploaded to Deep Learning Git repository for use in training.
Types of Malicious URLs
- Defacement
- Benign
- Phishing
- Malware
- Spam
Prepare the Environment and Library¶
Install Scikeras libraries
In [ ]:
!pip install scikeras
Collecting scikeras Downloading scikeras-0.12.0-py3-none-any.whl (27 kB) Requirement already satisfied: packaging>=0.21 in /usr/local/lib/python3.10/dist-packages (from scikeras) (23.2) Requirement already satisfied: scikit-learn>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from scikeras) (1.2.2) Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.0.0->scikeras) (1.23.5) Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.0.0->scikeras) (1.11.3) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.0.0->scikeras) (1.3.2) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.0.0->scikeras) (3.2.0) Installing collected packages: scikeras Successfully installed scikeras-0.12.0
Import Required Python libraries
In [ ]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import warnings
import tensorflow as tf
from tensorflow import keras
from keras import layers, models
from keras.callbacks import ModelCheckpoint, EarlyStopping
from keras.layers import Dense
from keras.metrics import Accuracy, F1Score
from keras.models import Sequential
from keras.utils import to_categorical
from keras import backend as K
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from scikeras.wrappers import KerasClassifier, KerasRegressor
warnings.filterwarnings('ignore')
tf.get_logger().setLevel('ERROR')
np.random.seed(42)
tf.random.set_seed(42)
In [ ]:
# mount the colab
from google.colab import drive
drive.mount('/content/drive')
# For saving and retrieving the trained model
WORKSPACE_PATH = "/content/drive/MyDrive/Colab Notebooks/models/"
Mounted at /content/drive
Load the dataset¶
In [ ]:
# Read the dataset from github
url = 'https://raw.githubusercontent.com/quickheaven/scs-3546-deep-learning/master/datasets/'
X_train = pd.read_csv(url + 'X_train.csv', index_col=0)
X_test =pd.read_csv(url + 'X_test.csv', index_col=0)
y_train =pd.read_csv(url + 'y_train.csv', index_col=0)
y_test = pd.read_csv(url + 'y_test.csv', index_col=0)
In [ ]:
# Initialize input dim and output nodes
NUM_INPUT_DIM = X_train.shape[1]
NUM_OUTPUT_NODES = len(y_train['URL_Type_obf_Type'].unique())
#
y_train_dummy = to_categorical(y_train)
y_test_dummy = to_categorical(y_test)
Training and Tuning Model¶
The training and tuning of model is divided into three main experiments. Tuning of Deep Network, Tuning the Back Propagation and last is the Overfitting Management.
The first step before proceeding to other experiments is Tuning the Neural Network. These includes determining first the epoch and batch size to use, the number of hidden layers to add, the number of nodes in each layer, choosing the activation function and the weight initializer.
The second part is Tuning the Back Propagation. It involves whether to use Batch normalization or not, finding the right Optimizers and its Learning Rates.
Lastly is the Overfitting Management, I used Regularation and Dropout as part of the experiment.
In [ ]:
def get_base_model_config():
"""
This function returns the based model configuration for the experiments.
Parameters
----------
None
Returns
----------
dict - The dictionary containing the based model configuration.
"""
early_stopping_callback = EarlyStopping(monitor='val_accuracy', mode='max', patience=5, restore_best_weights=True)
callbacks = [early_stopping_callback]
model_config = {
# ##################################
"model_name": None,
"input_dim": NUM_INPUT_DIM,
"custom_layers": list(),
"callbacks": callbacks,
"validation_split": 0.20,
"loss": "categorical_crossentropy",
"output_nodes": NUM_OUTPUT_NODES,
"output_activation": "softmax",
"metrics": ['accuracy'],
# ##################################
"batch_size": 32,
"epochs": 200,
"hidden_activation": "relu",
"weights_initializer": "random_normal",
# ##################################
"normalization": None,
"optimizer": "adam",
"learning_rate": 0.001,
"regularizer": None,
"dropout_rate": None,
# ##################################
"is_save_model": False,
"workspace_path": WORKSPACE_PATH,
"verbose": 1
}
return model_config
In [ ]:
In [ ]:
def get_optimizer(optimizer_name, learning_rate):
"""
(str. float) -> keras.optimizers
This method returns the optimizer that will be use in the experiment.
Parameters
----------
optimizer_name - The name of the optimizer to use in the experiment. values are adagrad, rmsprop, adam and None.
learning_rate - The rate of learning to use in the optimizer.
Returns
----------
keras.optimizer - The keras optimizer object.
"""
optimizer=None
if optimizer_name == 'adagrad':
optimizer = keras.optimizers.Adagrad(learning_rate=learning_rate)
elif 'rmsprop':
optimizer = keras.optimizers.RMSprop(learning_rate=learning_rate)
elif'adam' :
optimizer = keras.optimizers.Adam(learning_rate=learning_rate)
else :
optimizer = keras.optimizers.SGD(learning_rate=learning_rate)
return optimizer
In [ ]:
def create_and_run_model(model_config, X, y):
# Build the new model
model = Sequential(layers=model_config['custom_layers'], name=model_config['model_name'])
model.summary()
optimizer = get_optimizer(model_config["optimizer"], model_config["learning_rate"])
# Compile the model
model.compile(optimizer=optimizer,
loss=model_config['loss'],
metrics=model_config['metrics'])
# Fit the model
history = model.fit(X, y,
batch_size=model_config['batch_size'],
callbacks=model_config['callbacks'],
epochs=model_config['epochs'],
validation_split=model_config['validation_split'],
verbose=model_config["verbose"])
if True == model_config['is_save_model']:
model_file = str(model_config['workspace_path']) + str(model_config['model_name']) + '.h5'
model.save(model_file)
return history
In [ ]:
def plot_accuracy_measures(accuracy_measures, title):
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 5))
for experiment in accuracy_measures.keys():
plt.plot(accuracy_measures[experiment],
label=experiment,
linewidth=3)
plt.title(title)
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend()
plt.show()
In [ ]:
def plot_learning_curves(history, title):
# plot curves for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Learning Curves ' + title)
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.plot(figsize=(8, 5))
plt.grid(True)
plt.show()
In [ ]:
model_config = get_base_model_config()
Tuning Deep Neural Network¶
Epoch and Batch size tuning¶
Figure out the right number of batches and epochs first, and then use that for further experimentation.
Batch Size: Experiment for the right size, batch size of 32 found most optimal for most use cases.
Epoch: Choose the earliest value when accuracy stabilizers.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
batch_sizes = [32, 64, 128]
for size in batch_sizes:
model_config['batch_size'] = size
custom_layers = [
layers.Dense(16, activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Epoch_and_batch_size_' + str(size)
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Epoch_and_batch_size_32"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_6 (Dense) (None, 16) 832
dense_7 (Dense) (None, 5) 85
=================================================================
Total params: 917 (3.58 KB)
Trainable params: 917 (3.58 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/200
636/636 [==============================] - 4s 5ms/step - loss: 1.3011 - accuracy: 0.5578 - val_loss: 1.0864 - val_accuracy: 0.6352
Epoch 2/200
636/636 [==============================] - 3s 5ms/step - loss: 1.0083 - accuracy: 0.6543 - val_loss: 0.9273 - val_accuracy: 0.6865
Epoch 3/200
636/636 [==============================] - 3s 5ms/step - loss: 0.8973 - accuracy: 0.6913 - val_loss: 0.8493 - val_accuracy: 0.6936
Epoch 4/200
636/636 [==============================] - 2s 4ms/step - loss: 0.8323 - accuracy: 0.7072 - val_loss: 0.8010 - val_accuracy: 0.7079
Epoch 5/200
636/636 [==============================] - 2s 4ms/step - loss: 0.7883 - accuracy: 0.7178 - val_loss: 0.7606 - val_accuracy: 0.7243
Epoch 6/200
636/636 [==============================] - 2s 4ms/step - loss: 0.7536 - accuracy: 0.7291 - val_loss: 0.7304 - val_accuracy: 0.7349
Epoch 7/200
636/636 [==============================] - 3s 4ms/step - loss: 0.7237 - accuracy: 0.7360 - val_loss: 0.7042 - val_accuracy: 0.7476
Epoch 8/200
636/636 [==============================] - 3s 5ms/step - loss: 0.6980 - accuracy: 0.7467 - val_loss: 0.6816 - val_accuracy: 0.7522
Epoch 9/200
636/636 [==============================] - 3s 5ms/step - loss: 0.6754 - accuracy: 0.7545 - val_loss: 0.6622 - val_accuracy: 0.7610
Epoch 10/200
636/636 [==============================] - 2s 4ms/step - loss: 0.6567 - accuracy: 0.7623 - val_loss: 0.6502 - val_accuracy: 0.7708
Epoch 11/200
636/636 [==============================] - 2s 4ms/step - loss: 0.6403 - accuracy: 0.7696 - val_loss: 0.6290 - val_accuracy: 0.7803
Epoch 12/200
636/636 [==============================] - 2s 4ms/step - loss: 0.6242 - accuracy: 0.7787 - val_loss: 0.6168 - val_accuracy: 0.7785
Epoch 13/200
636/636 [==============================] - 2s 4ms/step - loss: 0.6108 - accuracy: 0.7813 - val_loss: 0.6045 - val_accuracy: 0.7875
Epoch 14/200
636/636 [==============================] - 3s 5ms/step - loss: 0.5988 - accuracy: 0.7876 - val_loss: 0.5992 - val_accuracy: 0.7838
Epoch 15/200
636/636 [==============================] - 3s 5ms/step - loss: 0.5876 - accuracy: 0.7909 - val_loss: 0.5815 - val_accuracy: 0.7948
Epoch 16/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5777 - accuracy: 0.7959 - val_loss: 0.5745 - val_accuracy: 0.7962
Epoch 17/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5689 - accuracy: 0.7980 - val_loss: 0.5721 - val_accuracy: 0.7985
Epoch 18/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5613 - accuracy: 0.8001 - val_loss: 0.5612 - val_accuracy: 0.8011
Epoch 19/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5531 - accuracy: 0.8025 - val_loss: 0.5517 - val_accuracy: 0.8039
Epoch 20/200
636/636 [==============================] - 3s 5ms/step - loss: 0.5473 - accuracy: 0.8028 - val_loss: 0.5503 - val_accuracy: 0.8031
Epoch 21/200
636/636 [==============================] - 4s 6ms/step - loss: 0.5410 - accuracy: 0.8062 - val_loss: 0.5410 - val_accuracy: 0.8078
Epoch 22/200
636/636 [==============================] - 3s 4ms/step - loss: 0.5355 - accuracy: 0.8097 - val_loss: 0.5418 - val_accuracy: 0.8048
Epoch 23/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5297 - accuracy: 0.8109 - val_loss: 0.5372 - val_accuracy: 0.8109
Epoch 24/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5250 - accuracy: 0.8153 - val_loss: 0.5282 - val_accuracy: 0.8172
Epoch 25/200
636/636 [==============================] - 3s 4ms/step - loss: 0.5203 - accuracy: 0.8156 - val_loss: 0.5243 - val_accuracy: 0.8186
Epoch 26/200
636/636 [==============================] - 3s 5ms/step - loss: 0.5157 - accuracy: 0.8174 - val_loss: 0.5191 - val_accuracy: 0.8215
Epoch 27/200
636/636 [==============================] - 4s 6ms/step - loss: 0.5108 - accuracy: 0.8200 - val_loss: 0.5161 - val_accuracy: 0.8243
Epoch 28/200
636/636 [==============================] - 3s 4ms/step - loss: 0.5069 - accuracy: 0.8214 - val_loss: 0.5159 - val_accuracy: 0.8225
Epoch 29/200
636/636 [==============================] - 2s 4ms/step - loss: 0.5030 - accuracy: 0.8225 - val_loss: 0.5178 - val_accuracy: 0.8280
Epoch 30/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4993 - accuracy: 0.8234 - val_loss: 0.5106 - val_accuracy: 0.8186
Epoch 31/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4965 - accuracy: 0.8261 - val_loss: 0.5053 - val_accuracy: 0.8286
Epoch 32/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4920 - accuracy: 0.8269 - val_loss: 0.5009 - val_accuracy: 0.8255
Epoch 33/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4894 - accuracy: 0.8275 - val_loss: 0.5002 - val_accuracy: 0.8288
Epoch 34/200
636/636 [==============================] - 3s 4ms/step - loss: 0.4856 - accuracy: 0.8283 - val_loss: 0.4958 - val_accuracy: 0.8302
Epoch 35/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4830 - accuracy: 0.8300 - val_loss: 0.4990 - val_accuracy: 0.8290
Epoch 36/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4806 - accuracy: 0.8292 - val_loss: 0.5011 - val_accuracy: 0.8270
Epoch 37/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4774 - accuracy: 0.8292 - val_loss: 0.4926 - val_accuracy: 0.8280
Epoch 38/200
636/636 [==============================] - 3s 4ms/step - loss: 0.4757 - accuracy: 0.8302 - val_loss: 0.4861 - val_accuracy: 0.8300
Epoch 39/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4729 - accuracy: 0.8312 - val_loss: 0.4849 - val_accuracy: 0.8320
Epoch 40/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4696 - accuracy: 0.8331 - val_loss: 0.4939 - val_accuracy: 0.8298
Epoch 41/200
636/636 [==============================] - 3s 4ms/step - loss: 0.4673 - accuracy: 0.8334 - val_loss: 0.4845 - val_accuracy: 0.8357
Epoch 42/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4652 - accuracy: 0.8353 - val_loss: 0.4832 - val_accuracy: 0.8361
Epoch 43/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4631 - accuracy: 0.8358 - val_loss: 0.4829 - val_accuracy: 0.8347
Epoch 44/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4613 - accuracy: 0.8353 - val_loss: 0.4742 - val_accuracy: 0.8434
Epoch 45/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4583 - accuracy: 0.8368 - val_loss: 0.4727 - val_accuracy: 0.8361
Epoch 46/200
636/636 [==============================] - 3s 5ms/step - loss: 0.4571 - accuracy: 0.8372 - val_loss: 0.4715 - val_accuracy: 0.8363
Epoch 47/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4546 - accuracy: 0.8384 - val_loss: 0.4674 - val_accuracy: 0.8398
Epoch 48/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4526 - accuracy: 0.8395 - val_loss: 0.4717 - val_accuracy: 0.8335
Epoch 49/200
636/636 [==============================] - 2s 4ms/step - loss: 0.4513 - accuracy: 0.8387 - val_loss: 0.4694 - val_accuracy: 0.8410
Model: "Epoch_and_batch_size_64"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_8 (Dense) (None, 16) 832
dense_9 (Dense) (None, 5) 85
=================================================================
Total params: 917 (3.58 KB)
Trainable params: 917 (3.58 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/200
318/318 [==============================] - 2s 5ms/step - loss: 1.4376 - accuracy: 0.4816 - val_loss: 1.2671 - val_accuracy: 0.6103
Epoch 2/200
318/318 [==============================] - 2s 5ms/step - loss: 1.1368 - accuracy: 0.6318 - val_loss: 1.0315 - val_accuracy: 0.6657
Epoch 3/200
318/318 [==============================] - 2s 5ms/step - loss: 0.9771 - accuracy: 0.6662 - val_loss: 0.9236 - val_accuracy: 0.6745
Epoch 4/200
318/318 [==============================] - 2s 5ms/step - loss: 0.9007 - accuracy: 0.6826 - val_loss: 0.8686 - val_accuracy: 0.6842
Epoch 5/200
318/318 [==============================] - 2s 5ms/step - loss: 0.8551 - accuracy: 0.6953 - val_loss: 0.8291 - val_accuracy: 0.7018
Epoch 6/200
318/318 [==============================] - 1s 4ms/step - loss: 0.8221 - accuracy: 0.7056 - val_loss: 0.7998 - val_accuracy: 0.7034
Epoch 7/200
318/318 [==============================] - 1s 4ms/step - loss: 0.7949 - accuracy: 0.7133 - val_loss: 0.7768 - val_accuracy: 0.7178
Epoch 8/200
318/318 [==============================] - 1s 4ms/step - loss: 0.7726 - accuracy: 0.7222 - val_loss: 0.7555 - val_accuracy: 0.7250
Epoch 9/200
318/318 [==============================] - 1s 4ms/step - loss: 0.7525 - accuracy: 0.7264 - val_loss: 0.7380 - val_accuracy: 0.7347
Epoch 10/200
318/318 [==============================] - 1s 5ms/step - loss: 0.7352 - accuracy: 0.7357 - val_loss: 0.7245 - val_accuracy: 0.7384
Epoch 11/200
318/318 [==============================] - 1s 4ms/step - loss: 0.7190 - accuracy: 0.7409 - val_loss: 0.7055 - val_accuracy: 0.7453
Epoch 12/200
318/318 [==============================] - 1s 4ms/step - loss: 0.7037 - accuracy: 0.7465 - val_loss: 0.6946 - val_accuracy: 0.7541
Epoch 13/200
318/318 [==============================] - 2s 5ms/step - loss: 0.6904 - accuracy: 0.7508 - val_loss: 0.6803 - val_accuracy: 0.7518
Epoch 14/200
318/318 [==============================] - 2s 5ms/step - loss: 0.6771 - accuracy: 0.7561 - val_loss: 0.6740 - val_accuracy: 0.7606
Epoch 15/200
318/318 [==============================] - 2s 5ms/step - loss: 0.6651 - accuracy: 0.7595 - val_loss: 0.6573 - val_accuracy: 0.7628
Epoch 16/200
318/318 [==============================] - 2s 5ms/step - loss: 0.6543 - accuracy: 0.7650 - val_loss: 0.6471 - val_accuracy: 0.7691
Epoch 17/200
318/318 [==============================] - 1s 4ms/step - loss: 0.6445 - accuracy: 0.7673 - val_loss: 0.6397 - val_accuracy: 0.7712
Epoch 18/200
318/318 [==============================] - 1s 4ms/step - loss: 0.6350 - accuracy: 0.7709 - val_loss: 0.6334 - val_accuracy: 0.7803
Epoch 19/200
318/318 [==============================] - 1s 4ms/step - loss: 0.6254 - accuracy: 0.7742 - val_loss: 0.6208 - val_accuracy: 0.7744
Epoch 20/200
318/318 [==============================] - 1s 4ms/step - loss: 0.6172 - accuracy: 0.7787 - val_loss: 0.6170 - val_accuracy: 0.7824
Epoch 21/200
318/318 [==============================] - 1s 4ms/step - loss: 0.6095 - accuracy: 0.7821 - val_loss: 0.6061 - val_accuracy: 0.7824
Epoch 22/200
318/318 [==============================] - 1s 5ms/step - loss: 0.6021 - accuracy: 0.7839 - val_loss: 0.6021 - val_accuracy: 0.7877
Epoch 23/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5951 - accuracy: 0.7872 - val_loss: 0.5973 - val_accuracy: 0.7895
Epoch 24/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5877 - accuracy: 0.7903 - val_loss: 0.5876 - val_accuracy: 0.7909
Epoch 25/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5809 - accuracy: 0.7941 - val_loss: 0.5807 - val_accuracy: 0.7944
Epoch 26/200
318/318 [==============================] - 2s 6ms/step - loss: 0.5745 - accuracy: 0.7932 - val_loss: 0.5739 - val_accuracy: 0.7905
Epoch 27/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5672 - accuracy: 0.7982 - val_loss: 0.5690 - val_accuracy: 0.7946
Epoch 28/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5615 - accuracy: 0.7976 - val_loss: 0.5643 - val_accuracy: 0.7980
Epoch 29/200
318/318 [==============================] - 1s 5ms/step - loss: 0.5553 - accuracy: 0.8035 - val_loss: 0.5623 - val_accuracy: 0.7966
Epoch 30/200
318/318 [==============================] - 1s 5ms/step - loss: 0.5498 - accuracy: 0.8051 - val_loss: 0.5529 - val_accuracy: 0.8029
Epoch 31/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5441 - accuracy: 0.8073 - val_loss: 0.5462 - val_accuracy: 0.8086
Epoch 32/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5377 - accuracy: 0.8106 - val_loss: 0.5408 - val_accuracy: 0.8056
Epoch 33/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5328 - accuracy: 0.8123 - val_loss: 0.5379 - val_accuracy: 0.8131
Epoch 34/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5278 - accuracy: 0.8131 - val_loss: 0.5297 - val_accuracy: 0.8084
Epoch 35/200
318/318 [==============================] - 1s 5ms/step - loss: 0.5230 - accuracy: 0.8159 - val_loss: 0.5287 - val_accuracy: 0.8094
Epoch 36/200
318/318 [==============================] - 2s 6ms/step - loss: 0.5188 - accuracy: 0.8161 - val_loss: 0.5317 - val_accuracy: 0.8062
Epoch 37/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5141 - accuracy: 0.8179 - val_loss: 0.5228 - val_accuracy: 0.8117
Epoch 38/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5104 - accuracy: 0.8186 - val_loss: 0.5149 - val_accuracy: 0.8137
Epoch 39/200
318/318 [==============================] - 2s 5ms/step - loss: 0.5061 - accuracy: 0.8191 - val_loss: 0.5125 - val_accuracy: 0.8158
Epoch 40/200
318/318 [==============================] - 1s 4ms/step - loss: 0.5023 - accuracy: 0.8216 - val_loss: 0.5137 - val_accuracy: 0.8147
Epoch 41/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4983 - accuracy: 0.8218 - val_loss: 0.5069 - val_accuracy: 0.8156
Epoch 42/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4953 - accuracy: 0.8231 - val_loss: 0.5053 - val_accuracy: 0.8208
Epoch 43/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4921 - accuracy: 0.8242 - val_loss: 0.5045 - val_accuracy: 0.8190
Epoch 44/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4885 - accuracy: 0.8253 - val_loss: 0.4954 - val_accuracy: 0.8219
Epoch 45/200
318/318 [==============================] - 1s 5ms/step - loss: 0.4850 - accuracy: 0.8254 - val_loss: 0.4925 - val_accuracy: 0.8213
Epoch 46/200
318/318 [==============================] - 1s 5ms/step - loss: 0.4823 - accuracy: 0.8268 - val_loss: 0.4888 - val_accuracy: 0.8208
Epoch 47/200
318/318 [==============================] - 2s 6ms/step - loss: 0.4790 - accuracy: 0.8281 - val_loss: 0.4857 - val_accuracy: 0.8221
Epoch 48/200
318/318 [==============================] - 2s 5ms/step - loss: 0.4760 - accuracy: 0.8284 - val_loss: 0.4872 - val_accuracy: 0.8184
Epoch 49/200
318/318 [==============================] - 2s 6ms/step - loss: 0.4737 - accuracy: 0.8287 - val_loss: 0.4834 - val_accuracy: 0.8276
Epoch 50/200
318/318 [==============================] - 2s 5ms/step - loss: 0.4708 - accuracy: 0.8311 - val_loss: 0.4792 - val_accuracy: 0.8241
Epoch 51/200
318/318 [==============================] - 1s 5ms/step - loss: 0.4685 - accuracy: 0.8305 - val_loss: 0.4780 - val_accuracy: 0.8241
Epoch 52/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4661 - accuracy: 0.8314 - val_loss: 0.4746 - val_accuracy: 0.8257
Epoch 53/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4635 - accuracy: 0.8323 - val_loss: 0.4736 - val_accuracy: 0.8255
Epoch 54/200
318/318 [==============================] - 1s 4ms/step - loss: 0.4608 - accuracy: 0.8340 - val_loss: 0.4703 - val_accuracy: 0.8255
Model: "Epoch_and_batch_size_128"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_10 (Dense) (None, 16) 832
dense_11 (Dense) (None, 5) 85
=================================================================
Total params: 917 (3.58 KB)
Trainable params: 917 (3.58 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/200
159/159 [==============================] - 1s 5ms/step - loss: 1.5190 - accuracy: 0.4015 - val_loss: 1.3869 - val_accuracy: 0.5572
Epoch 2/200
159/159 [==============================] - 1s 4ms/step - loss: 1.2660 - accuracy: 0.5996 - val_loss: 1.1521 - val_accuracy: 0.6333
Epoch 3/200
159/159 [==============================] - 1s 4ms/step - loss: 1.0857 - accuracy: 0.6396 - val_loss: 1.0191 - val_accuracy: 0.6437
Epoch 4/200
159/159 [==============================] - 1s 5ms/step - loss: 0.9886 - accuracy: 0.6648 - val_loss: 0.9494 - val_accuracy: 0.6737
Epoch 5/200
159/159 [==============================] - 1s 5ms/step - loss: 0.9310 - accuracy: 0.6748 - val_loss: 0.9019 - val_accuracy: 0.6849
Epoch 6/200
159/159 [==============================] - 1s 6ms/step - loss: 0.8922 - accuracy: 0.6847 - val_loss: 0.8691 - val_accuracy: 0.6891
Epoch 7/200
159/159 [==============================] - 1s 5ms/step - loss: 0.8631 - accuracy: 0.6912 - val_loss: 0.8460 - val_accuracy: 0.7001
Epoch 8/200
159/159 [==============================] - 1s 6ms/step - loss: 0.8401 - accuracy: 0.6988 - val_loss: 0.8253 - val_accuracy: 0.7044
Epoch 9/200
159/159 [==============================] - 1s 6ms/step - loss: 0.8209 - accuracy: 0.7060 - val_loss: 0.8092 - val_accuracy: 0.7056
Epoch 10/200
159/159 [==============================] - 1s 7ms/step - loss: 0.8044 - accuracy: 0.7105 - val_loss: 0.7953 - val_accuracy: 0.7152
Epoch 11/200
159/159 [==============================] - 1s 6ms/step - loss: 0.7904 - accuracy: 0.7146 - val_loss: 0.7804 - val_accuracy: 0.7195
Epoch 12/200
159/159 [==============================] - 1s 6ms/step - loss: 0.7778 - accuracy: 0.7210 - val_loss: 0.7723 - val_accuracy: 0.7207
Epoch 13/200
159/159 [==============================] - 1s 5ms/step - loss: 0.7664 - accuracy: 0.7212 - val_loss: 0.7587 - val_accuracy: 0.7258
Epoch 14/200
159/159 [==============================] - 1s 4ms/step - loss: 0.7554 - accuracy: 0.7260 - val_loss: 0.7542 - val_accuracy: 0.7254
Epoch 15/200
159/159 [==============================] - 1s 4ms/step - loss: 0.7454 - accuracy: 0.7272 - val_loss: 0.7406 - val_accuracy: 0.7241
Epoch 16/200
159/159 [==============================] - 1s 4ms/step - loss: 0.7364 - accuracy: 0.7306 - val_loss: 0.7317 - val_accuracy: 0.7303
Epoch 17/200
159/159 [==============================] - 1s 5ms/step - loss: 0.7284 - accuracy: 0.7327 - val_loss: 0.7253 - val_accuracy: 0.7343
Epoch 18/200
159/159 [==============================] - 1s 4ms/step - loss: 0.7199 - accuracy: 0.7345 - val_loss: 0.7177 - val_accuracy: 0.7406
Epoch 19/200
159/159 [==============================] - 1s 4ms/step - loss: 0.7122 - accuracy: 0.7385 - val_loss: 0.7082 - val_accuracy: 0.7380
Epoch 20/200
159/159 [==============================] - 1s 5ms/step - loss: 0.7048 - accuracy: 0.7420 - val_loss: 0.7043 - val_accuracy: 0.7500
Epoch 21/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6981 - accuracy: 0.7444 - val_loss: 0.6954 - val_accuracy: 0.7410
Epoch 22/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6915 - accuracy: 0.7479 - val_loss: 0.6892 - val_accuracy: 0.7528
Epoch 23/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6850 - accuracy: 0.7512 - val_loss: 0.6860 - val_accuracy: 0.7590
Epoch 24/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6786 - accuracy: 0.7529 - val_loss: 0.6782 - val_accuracy: 0.7553
Epoch 25/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6726 - accuracy: 0.7558 - val_loss: 0.6716 - val_accuracy: 0.7586
Epoch 26/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6665 - accuracy: 0.7587 - val_loss: 0.6647 - val_accuracy: 0.7626
Epoch 27/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6598 - accuracy: 0.7615 - val_loss: 0.6592 - val_accuracy: 0.7669
Epoch 28/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6546 - accuracy: 0.7643 - val_loss: 0.6546 - val_accuracy: 0.7712
Epoch 29/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6487 - accuracy: 0.7680 - val_loss: 0.6504 - val_accuracy: 0.7702
Epoch 30/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6440 - accuracy: 0.7690 - val_loss: 0.6456 - val_accuracy: 0.7687
Epoch 31/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6387 - accuracy: 0.7698 - val_loss: 0.6396 - val_accuracy: 0.7771
Epoch 32/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6334 - accuracy: 0.7736 - val_loss: 0.6364 - val_accuracy: 0.7738
Epoch 33/200
159/159 [==============================] - 1s 6ms/step - loss: 0.6288 - accuracy: 0.7740 - val_loss: 0.6328 - val_accuracy: 0.7783
Epoch 34/200
159/159 [==============================] - 1s 5ms/step - loss: 0.6240 - accuracy: 0.7763 - val_loss: 0.6259 - val_accuracy: 0.7748
Epoch 35/200
159/159 [==============================] - 1s 5ms/step - loss: 0.6199 - accuracy: 0.7756 - val_loss: 0.6224 - val_accuracy: 0.7779
Epoch 36/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6158 - accuracy: 0.7778 - val_loss: 0.6254 - val_accuracy: 0.7773
Epoch 37/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6115 - accuracy: 0.7782 - val_loss: 0.6197 - val_accuracy: 0.7767
Epoch 38/200
159/159 [==============================] - 1s 4ms/step - loss: 0.6081 - accuracy: 0.7781 - val_loss: 0.6126 - val_accuracy: 0.7750
In [ ]:
accuracy_measures_batch_epoch = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Batch Size and Epoch")
The model with batch_size_32 offers higher accuracy.
In [ ]:
model_config['batch_size'] = 32
model_config['epochs'] = 50
Hidden layers tuning¶
One of the key model architecture hyperparameters is the number of hidden layers. As the number of layers increases, it increases the possibility of learning complex relationships between features and target variables, but it will also increase the cost and time needed for both training and inference. It is also has the risk of overfitting the training set.
A value of two has been sufficient for simple problems. It is recommended to increase the number of layers only based on experimentation if the set accuracy levels are not achieved. Otherwise, additional layers will take resources and time without providing any additional value.
In [ ]:
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
accuracy_measures = {}
# reused the previous model config from previous experiment.
# but on this case only increase the number of layers
# I will start small by checking if one or two nodes is enough to get a good result.
layer_list = []
for hidden_layer_count in range(1,11):
custom_layers = []
# Simply use 16 nodes per layer for now.
# Input
if hidden_layer_count == 1:
custom_layers.append(layers.Dense(16, activation=model_config['hidden_activation'], input_dim=model_config['input_dim']))
else:
# Additional Layers
custom_layers.append(layers.Dense(16, activation=model_config['hidden_activation'], input_dim=model_config['input_dim']))
for i in range(1, hidden_layer_count):
custom_layers.append(layers.Dense(16, activation=model_config['hidden_activation']))
# Output layers
custom_layers.append(layers.Dense(model_config['output_nodes'], activation=model_config['output_activation']))
model_name = 'Layers_' + str(hidden_layer_count)
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Layers_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_77 (Dense) (None, 16) 832
dense_78 (Dense) (None, 5) 85
=================================================================
Total params: 917 (3.58 KB)
Trainable params: 917 (3.58 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.3064 - accuracy: 0.5207 - val_loss: 1.0968 - val_accuracy: 0.6236
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0096 - accuracy: 0.6558 - val_loss: 0.9319 - val_accuracy: 0.6847
Epoch 3/50
636/636 [==============================] - 2s 4ms/step - loss: 0.9002 - accuracy: 0.6875 - val_loss: 0.8587 - val_accuracy: 0.6977
Epoch 4/50
636/636 [==============================] - 2s 4ms/step - loss: 0.8433 - accuracy: 0.7014 - val_loss: 0.8193 - val_accuracy: 0.6991
Epoch 5/50
636/636 [==============================] - 2s 4ms/step - loss: 0.8051 - accuracy: 0.7109 - val_loss: 0.7829 - val_accuracy: 0.7144
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7752 - accuracy: 0.7192 - val_loss: 0.7555 - val_accuracy: 0.7235
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7485 - accuracy: 0.7285 - val_loss: 0.7318 - val_accuracy: 0.7360
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7232 - accuracy: 0.7405 - val_loss: 0.7092 - val_accuracy: 0.7486
Epoch 9/50
636/636 [==============================] - 2s 4ms/step - loss: 0.7005 - accuracy: 0.7496 - val_loss: 0.6896 - val_accuracy: 0.7567
Epoch 10/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6814 - accuracy: 0.7593 - val_loss: 0.6760 - val_accuracy: 0.7614
Epoch 11/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6646 - accuracy: 0.7664 - val_loss: 0.6555 - val_accuracy: 0.7742
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6475 - accuracy: 0.7733 - val_loss: 0.6419 - val_accuracy: 0.7740
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6335 - accuracy: 0.7757 - val_loss: 0.6276 - val_accuracy: 0.7885
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6210 - accuracy: 0.7824 - val_loss: 0.6240 - val_accuracy: 0.7726
Epoch 15/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6092 - accuracy: 0.7857 - val_loss: 0.6065 - val_accuracy: 0.7846
Epoch 16/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5990 - accuracy: 0.7890 - val_loss: 0.5985 - val_accuracy: 0.7891
Epoch 17/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5900 - accuracy: 0.7919 - val_loss: 0.5918 - val_accuracy: 0.7921
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5814 - accuracy: 0.7936 - val_loss: 0.5871 - val_accuracy: 0.7925
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5728 - accuracy: 0.7994 - val_loss: 0.5745 - val_accuracy: 0.8080
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5664 - accuracy: 0.8026 - val_loss: 0.5720 - val_accuracy: 0.7995
Epoch 21/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5592 - accuracy: 0.8052 - val_loss: 0.5621 - val_accuracy: 0.8021
Epoch 22/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5526 - accuracy: 0.8073 - val_loss: 0.5628 - val_accuracy: 0.8056
Epoch 23/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5464 - accuracy: 0.8102 - val_loss: 0.5563 - val_accuracy: 0.8066
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5410 - accuracy: 0.8125 - val_loss: 0.5460 - val_accuracy: 0.8105
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5358 - accuracy: 0.8152 - val_loss: 0.5413 - val_accuracy: 0.8096
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5309 - accuracy: 0.8154 - val_loss: 0.5358 - val_accuracy: 0.8149
Epoch 27/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5257 - accuracy: 0.8183 - val_loss: 0.5323 - val_accuracy: 0.8184
Epoch 28/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5218 - accuracy: 0.8204 - val_loss: 0.5302 - val_accuracy: 0.8178
Epoch 29/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5175 - accuracy: 0.8215 - val_loss: 0.5285 - val_accuracy: 0.8211
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5136 - accuracy: 0.8219 - val_loss: 0.5275 - val_accuracy: 0.8210
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5101 - accuracy: 0.8229 - val_loss: 0.5204 - val_accuracy: 0.8221
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5056 - accuracy: 0.8226 - val_loss: 0.5192 - val_accuracy: 0.8229
Epoch 33/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5025 - accuracy: 0.8265 - val_loss: 0.5143 - val_accuracy: 0.8215
Epoch 34/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4988 - accuracy: 0.8259 - val_loss: 0.5083 - val_accuracy: 0.8296
Epoch 35/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4961 - accuracy: 0.8292 - val_loss: 0.5103 - val_accuracy: 0.8306
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4937 - accuracy: 0.8287 - val_loss: 0.5123 - val_accuracy: 0.8253
Epoch 37/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4908 - accuracy: 0.8318 - val_loss: 0.5046 - val_accuracy: 0.8270
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4886 - accuracy: 0.8305 - val_loss: 0.5005 - val_accuracy: 0.8333
Epoch 39/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4860 - accuracy: 0.8314 - val_loss: 0.4983 - val_accuracy: 0.8310
Epoch 40/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4833 - accuracy: 0.8324 - val_loss: 0.5015 - val_accuracy: 0.8325
Epoch 41/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4805 - accuracy: 0.8350 - val_loss: 0.5007 - val_accuracy: 0.8292
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4788 - accuracy: 0.8359 - val_loss: 0.4944 - val_accuracy: 0.8327
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4770 - accuracy: 0.8364 - val_loss: 0.4940 - val_accuracy: 0.8365
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4747 - accuracy: 0.8368 - val_loss: 0.4864 - val_accuracy: 0.8402
Epoch 45/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4724 - accuracy: 0.8368 - val_loss: 0.4854 - val_accuracy: 0.8379
Epoch 46/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4715 - accuracy: 0.8385 - val_loss: 0.4830 - val_accuracy: 0.8355
Epoch 47/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4687 - accuracy: 0.8396 - val_loss: 0.4812 - val_accuracy: 0.8381
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4668 - accuracy: 0.8401 - val_loss: 0.4884 - val_accuracy: 0.8333
Epoch 49/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4657 - accuracy: 0.8404 - val_loss: 0.4849 - val_accuracy: 0.8384
Model: "Layers_2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_79 (Dense) (None, 16) 832
dense_80 (Dense) (None, 16) 272
dense_81 (Dense) (None, 5) 85
=================================================================
Total params: 1189 (4.64 KB)
Trainable params: 1189 (4.64 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2987 - accuracy: 0.5184 - val_loss: 1.0414 - val_accuracy: 0.6608
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9330 - accuracy: 0.6813 - val_loss: 0.8518 - val_accuracy: 0.7024
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8168 - accuracy: 0.7121 - val_loss: 0.7742 - val_accuracy: 0.7099
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7515 - accuracy: 0.7362 - val_loss: 0.7250 - val_accuracy: 0.7471
Epoch 5/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7061 - accuracy: 0.7468 - val_loss: 0.6771 - val_accuracy: 0.7557
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6679 - accuracy: 0.7601 - val_loss: 0.6448 - val_accuracy: 0.7712
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6378 - accuracy: 0.7714 - val_loss: 0.6195 - val_accuracy: 0.7752
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6122 - accuracy: 0.7831 - val_loss: 0.6017 - val_accuracy: 0.7877
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5915 - accuracy: 0.7883 - val_loss: 0.5873 - val_accuracy: 0.7846
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5757 - accuracy: 0.7950 - val_loss: 0.5732 - val_accuracy: 0.7950
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5624 - accuracy: 0.7994 - val_loss: 0.5497 - val_accuracy: 0.8097
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5487 - accuracy: 0.8060 - val_loss: 0.5415 - val_accuracy: 0.8086
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5383 - accuracy: 0.8102 - val_loss: 0.5302 - val_accuracy: 0.8188
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5281 - accuracy: 0.8150 - val_loss: 0.5250 - val_accuracy: 0.8166
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5197 - accuracy: 0.8167 - val_loss: 0.5125 - val_accuracy: 0.8231
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5099 - accuracy: 0.8214 - val_loss: 0.5027 - val_accuracy: 0.8327
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5020 - accuracy: 0.8248 - val_loss: 0.5075 - val_accuracy: 0.8259
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4947 - accuracy: 0.8265 - val_loss: 0.4967 - val_accuracy: 0.8267
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4867 - accuracy: 0.8296 - val_loss: 0.4872 - val_accuracy: 0.8386
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4813 - accuracy: 0.8330 - val_loss: 0.4843 - val_accuracy: 0.8300
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4749 - accuracy: 0.8319 - val_loss: 0.4646 - val_accuracy: 0.8414
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4683 - accuracy: 0.8367 - val_loss: 0.4736 - val_accuracy: 0.8300
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4608 - accuracy: 0.8374 - val_loss: 0.4537 - val_accuracy: 0.8432
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4554 - accuracy: 0.8428 - val_loss: 0.4522 - val_accuracy: 0.8502
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4509 - accuracy: 0.8430 - val_loss: 0.4465 - val_accuracy: 0.8461
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4442 - accuracy: 0.8441 - val_loss: 0.4559 - val_accuracy: 0.8438
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4397 - accuracy: 0.8471 - val_loss: 0.4399 - val_accuracy: 0.8565
Epoch 28/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4359 - accuracy: 0.8484 - val_loss: 0.4326 - val_accuracy: 0.8477
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4328 - accuracy: 0.8495 - val_loss: 0.4450 - val_accuracy: 0.8479
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4288 - accuracy: 0.8512 - val_loss: 0.4251 - val_accuracy: 0.8569
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4271 - accuracy: 0.8520 - val_loss: 0.4310 - val_accuracy: 0.8603
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4221 - accuracy: 0.8542 - val_loss: 0.4187 - val_accuracy: 0.8579
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4200 - accuracy: 0.8543 - val_loss: 0.4158 - val_accuracy: 0.8548
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4165 - accuracy: 0.8562 - val_loss: 0.4105 - val_accuracy: 0.8569
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4136 - accuracy: 0.8570 - val_loss: 0.4142 - val_accuracy: 0.8536
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4118 - accuracy: 0.8582 - val_loss: 0.4159 - val_accuracy: 0.8563
Model: "Layers_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_82 (Dense) (None, 16) 832
dense_83 (Dense) (None, 16) 272
dense_84 (Dense) (None, 16) 272
dense_85 (Dense) (None, 5) 85
=================================================================
Total params: 1461 (5.71 KB)
Trainable params: 1461 (5.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2695 - accuracy: 0.5499 - val_loss: 0.9616 - val_accuracy: 0.6584
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8842 - accuracy: 0.6929 - val_loss: 0.8023 - val_accuracy: 0.7221
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7675 - accuracy: 0.7297 - val_loss: 0.7150 - val_accuracy: 0.7459
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6868 - accuracy: 0.7598 - val_loss: 0.6587 - val_accuracy: 0.7748
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6339 - accuracy: 0.7773 - val_loss: 0.6293 - val_accuracy: 0.7718
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5989 - accuracy: 0.7864 - val_loss: 0.5834 - val_accuracy: 0.7907
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5717 - accuracy: 0.7935 - val_loss: 0.5588 - val_accuracy: 0.7966
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5503 - accuracy: 0.7997 - val_loss: 0.5384 - val_accuracy: 0.8119
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5316 - accuracy: 0.8109 - val_loss: 0.5578 - val_accuracy: 0.8072
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5156 - accuracy: 0.8177 - val_loss: 0.5230 - val_accuracy: 0.8278
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5027 - accuracy: 0.8222 - val_loss: 0.4985 - val_accuracy: 0.8292
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4883 - accuracy: 0.8309 - val_loss: 0.5007 - val_accuracy: 0.8239
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4772 - accuracy: 0.8325 - val_loss: 0.4782 - val_accuracy: 0.8329
Epoch 14/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4660 - accuracy: 0.8377 - val_loss: 0.4800 - val_accuracy: 0.8335
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4552 - accuracy: 0.8428 - val_loss: 0.4771 - val_accuracy: 0.8373
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4469 - accuracy: 0.8465 - val_loss: 0.4805 - val_accuracy: 0.8432
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4390 - accuracy: 0.8494 - val_loss: 0.4804 - val_accuracy: 0.8337
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4317 - accuracy: 0.8509 - val_loss: 0.4401 - val_accuracy: 0.8508
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4243 - accuracy: 0.8550 - val_loss: 0.4459 - val_accuracy: 0.8506
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4179 - accuracy: 0.8568 - val_loss: 0.4525 - val_accuracy: 0.8506
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4119 - accuracy: 0.8618 - val_loss: 0.4333 - val_accuracy: 0.8575
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4062 - accuracy: 0.8623 - val_loss: 0.4434 - val_accuracy: 0.8426
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4007 - accuracy: 0.8628 - val_loss: 0.4185 - val_accuracy: 0.8626
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3950 - accuracy: 0.8648 - val_loss: 0.4156 - val_accuracy: 0.8612
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3917 - accuracy: 0.8681 - val_loss: 0.4122 - val_accuracy: 0.8603
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3853 - accuracy: 0.8677 - val_loss: 0.4014 - val_accuracy: 0.8711
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3822 - accuracy: 0.8703 - val_loss: 0.4081 - val_accuracy: 0.8626
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3764 - accuracy: 0.8722 - val_loss: 0.3896 - val_accuracy: 0.8695
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3736 - accuracy: 0.8714 - val_loss: 0.4111 - val_accuracy: 0.8624
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3689 - accuracy: 0.8733 - val_loss: 0.3997 - val_accuracy: 0.8677
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3659 - accuracy: 0.8756 - val_loss: 0.3907 - val_accuracy: 0.8689
Model: "Layers_4"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_86 (Dense) (None, 16) 832
dense_87 (Dense) (None, 16) 272
dense_88 (Dense) (None, 16) 272
dense_89 (Dense) (None, 16) 272
dense_90 (Dense) (None, 5) 85
=================================================================
Total params: 1733 (6.77 KB)
Trainable params: 1733 (6.77 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 1.2767 - accuracy: 0.5003 - val_loss: 1.0463 - val_accuracy: 0.6201
Epoch 2/50
636/636 [==============================] - 4s 7ms/step - loss: 0.9212 - accuracy: 0.6742 - val_loss: 0.8149 - val_accuracy: 0.7085
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7890 - accuracy: 0.7127 - val_loss: 0.7444 - val_accuracy: 0.7160
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7201 - accuracy: 0.7397 - val_loss: 0.6820 - val_accuracy: 0.7510
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6731 - accuracy: 0.7547 - val_loss: 0.6328 - val_accuracy: 0.7757
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6336 - accuracy: 0.7682 - val_loss: 0.6053 - val_accuracy: 0.7716
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5983 - accuracy: 0.7812 - val_loss: 0.5728 - val_accuracy: 0.7956
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5675 - accuracy: 0.7944 - val_loss: 0.5608 - val_accuracy: 0.7917
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5412 - accuracy: 0.8048 - val_loss: 0.5603 - val_accuracy: 0.7997
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5215 - accuracy: 0.8146 - val_loss: 0.5256 - val_accuracy: 0.8143
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5060 - accuracy: 0.8201 - val_loss: 0.4917 - val_accuracy: 0.8237
Epoch 12/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4912 - accuracy: 0.8267 - val_loss: 0.4790 - val_accuracy: 0.8296
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4820 - accuracy: 0.8296 - val_loss: 0.4723 - val_accuracy: 0.8377
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4722 - accuracy: 0.8341 - val_loss: 0.4802 - val_accuracy: 0.8282
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4648 - accuracy: 0.8366 - val_loss: 0.4669 - val_accuracy: 0.8286
Epoch 16/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4536 - accuracy: 0.8389 - val_loss: 0.5069 - val_accuracy: 0.8217
Epoch 17/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4472 - accuracy: 0.8424 - val_loss: 0.4515 - val_accuracy: 0.8459
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4406 - accuracy: 0.8438 - val_loss: 0.4411 - val_accuracy: 0.8430
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4335 - accuracy: 0.8466 - val_loss: 0.4445 - val_accuracy: 0.8439
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4276 - accuracy: 0.8494 - val_loss: 0.4785 - val_accuracy: 0.8339
Epoch 21/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4225 - accuracy: 0.8515 - val_loss: 0.4229 - val_accuracy: 0.8494
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4167 - accuracy: 0.8538 - val_loss: 0.4221 - val_accuracy: 0.8510
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4119 - accuracy: 0.8569 - val_loss: 0.4329 - val_accuracy: 0.8498
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4065 - accuracy: 0.8586 - val_loss: 0.4198 - val_accuracy: 0.8510
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4035 - accuracy: 0.8602 - val_loss: 0.4091 - val_accuracy: 0.8544
Epoch 26/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3999 - accuracy: 0.8593 - val_loss: 0.4136 - val_accuracy: 0.8610
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3946 - accuracy: 0.8642 - val_loss: 0.4116 - val_accuracy: 0.8624
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3917 - accuracy: 0.8643 - val_loss: 0.4030 - val_accuracy: 0.8548
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3882 - accuracy: 0.8649 - val_loss: 0.4378 - val_accuracy: 0.8502
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3846 - accuracy: 0.8670 - val_loss: 0.3890 - val_accuracy: 0.8675
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3829 - accuracy: 0.8686 - val_loss: 0.3958 - val_accuracy: 0.8640
Epoch 32/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3794 - accuracy: 0.8711 - val_loss: 0.3939 - val_accuracy: 0.8687
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3768 - accuracy: 0.8697 - val_loss: 0.4005 - val_accuracy: 0.8589
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3726 - accuracy: 0.8719 - val_loss: 0.4060 - val_accuracy: 0.8559
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3709 - accuracy: 0.8729 - val_loss: 0.3804 - val_accuracy: 0.8717
Epoch 36/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3683 - accuracy: 0.8740 - val_loss: 0.3906 - val_accuracy: 0.8650
Epoch 37/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3647 - accuracy: 0.8764 - val_loss: 0.4185 - val_accuracy: 0.8563
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3621 - accuracy: 0.8779 - val_loss: 0.3779 - val_accuracy: 0.8717
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3622 - accuracy: 0.8760 - val_loss: 0.3716 - val_accuracy: 0.8760
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3572 - accuracy: 0.8786 - val_loss: 0.4174 - val_accuracy: 0.8593
Epoch 41/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3558 - accuracy: 0.8789 - val_loss: 0.3803 - val_accuracy: 0.8738
Epoch 42/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3548 - accuracy: 0.8798 - val_loss: 0.3936 - val_accuracy: 0.8685
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3517 - accuracy: 0.8830 - val_loss: 0.3771 - val_accuracy: 0.8724
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3502 - accuracy: 0.8831 - val_loss: 0.4044 - val_accuracy: 0.8597
Model: "Layers_5"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_91 (Dense) (None, 16) 832
dense_92 (Dense) (None, 16) 272
dense_93 (Dense) (None, 16) 272
dense_94 (Dense) (None, 16) 272
dense_95 (Dense) (None, 16) 272
dense_96 (Dense) (None, 5) 85
=================================================================
Total params: 2005 (7.83 KB)
Trainable params: 2005 (7.83 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 1.1709 - accuracy: 0.5472 - val_loss: 0.9010 - val_accuracy: 0.6669
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8499 - accuracy: 0.6817 - val_loss: 0.7740 - val_accuracy: 0.7099
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7588 - accuracy: 0.7186 - val_loss: 0.7185 - val_accuracy: 0.7490
Epoch 4/50
636/636 [==============================] - 4s 7ms/step - loss: 0.6970 - accuracy: 0.7437 - val_loss: 0.6608 - val_accuracy: 0.7524
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6505 - accuracy: 0.7589 - val_loss: 0.6491 - val_accuracy: 0.7508
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6099 - accuracy: 0.7765 - val_loss: 0.5882 - val_accuracy: 0.7803
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5799 - accuracy: 0.7890 - val_loss: 0.5564 - val_accuracy: 0.7989
Epoch 8/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5516 - accuracy: 0.8040 - val_loss: 0.5449 - val_accuracy: 0.8054
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5308 - accuracy: 0.8137 - val_loss: 0.5541 - val_accuracy: 0.8017
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5110 - accuracy: 0.8184 - val_loss: 0.5145 - val_accuracy: 0.8241
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4982 - accuracy: 0.8260 - val_loss: 0.5099 - val_accuracy: 0.8147
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4845 - accuracy: 0.8304 - val_loss: 0.4761 - val_accuracy: 0.8400
Epoch 13/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4736 - accuracy: 0.8351 - val_loss: 0.5187 - val_accuracy: 0.8133
Epoch 14/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4649 - accuracy: 0.8362 - val_loss: 0.4719 - val_accuracy: 0.8361
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4574 - accuracy: 0.8386 - val_loss: 0.4834 - val_accuracy: 0.8178
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4496 - accuracy: 0.8418 - val_loss: 0.4477 - val_accuracy: 0.8410
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4403 - accuracy: 0.8470 - val_loss: 0.4389 - val_accuracy: 0.8510
Epoch 18/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4326 - accuracy: 0.8495 - val_loss: 0.4319 - val_accuracy: 0.8485
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4274 - accuracy: 0.8511 - val_loss: 0.4428 - val_accuracy: 0.8467
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4212 - accuracy: 0.8549 - val_loss: 0.4904 - val_accuracy: 0.8239
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4146 - accuracy: 0.8573 - val_loss: 0.4198 - val_accuracy: 0.8534
Epoch 22/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4088 - accuracy: 0.8602 - val_loss: 0.4188 - val_accuracy: 0.8550
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4021 - accuracy: 0.8614 - val_loss: 0.4057 - val_accuracy: 0.8581
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3956 - accuracy: 0.8630 - val_loss: 0.4097 - val_accuracy: 0.8571
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3914 - accuracy: 0.8642 - val_loss: 0.3944 - val_accuracy: 0.8634
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3860 - accuracy: 0.8653 - val_loss: 0.4025 - val_accuracy: 0.8640
Epoch 27/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3789 - accuracy: 0.8679 - val_loss: 0.3819 - val_accuracy: 0.8636
Epoch 28/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3743 - accuracy: 0.8712 - val_loss: 0.3721 - val_accuracy: 0.8665
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3707 - accuracy: 0.8711 - val_loss: 0.4164 - val_accuracy: 0.8510
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3670 - accuracy: 0.8704 - val_loss: 0.3666 - val_accuracy: 0.8713
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3625 - accuracy: 0.8724 - val_loss: 0.3987 - val_accuracy: 0.8589
Epoch 32/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3600 - accuracy: 0.8720 - val_loss: 0.3680 - val_accuracy: 0.8724
Epoch 33/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3537 - accuracy: 0.8776 - val_loss: 0.3573 - val_accuracy: 0.8717
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3525 - accuracy: 0.8749 - val_loss: 0.3823 - val_accuracy: 0.8660
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3487 - accuracy: 0.8768 - val_loss: 0.3589 - val_accuracy: 0.8732
Epoch 36/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3446 - accuracy: 0.8784 - val_loss: 0.3732 - val_accuracy: 0.8593
Epoch 37/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3402 - accuracy: 0.8787 - val_loss: 0.3806 - val_accuracy: 0.8630
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3400 - accuracy: 0.8804 - val_loss: 0.3475 - val_accuracy: 0.8783
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3382 - accuracy: 0.8821 - val_loss: 0.3570 - val_accuracy: 0.8713
Epoch 40/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3349 - accuracy: 0.8811 - val_loss: 0.3711 - val_accuracy: 0.8664
Epoch 41/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3346 - accuracy: 0.8829 - val_loss: 0.3776 - val_accuracy: 0.8650
Epoch 42/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3315 - accuracy: 0.8829 - val_loss: 0.3553 - val_accuracy: 0.8772
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3278 - accuracy: 0.8846 - val_loss: 0.3579 - val_accuracy: 0.8705
Model: "Layers_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_97 (Dense) (None, 16) 832
dense_98 (Dense) (None, 16) 272
dense_99 (Dense) (None, 16) 272
dense_100 (Dense) (None, 16) 272
dense_101 (Dense) (None, 16) 272
dense_102 (Dense) (None, 16) 272
dense_103 (Dense) (None, 5) 85
=================================================================
Total params: 2277 (8.89 KB)
Trainable params: 2277 (8.89 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 6s 8ms/step - loss: 1.2344 - accuracy: 0.5091 - val_loss: 1.0273 - val_accuracy: 0.6069
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9630 - accuracy: 0.6438 - val_loss: 0.8724 - val_accuracy: 0.6851
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8503 - accuracy: 0.6835 - val_loss: 0.7970 - val_accuracy: 0.7064
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7719 - accuracy: 0.7160 - val_loss: 0.7197 - val_accuracy: 0.7425
Epoch 5/50
636/636 [==============================] - 4s 7ms/step - loss: 0.7165 - accuracy: 0.7389 - val_loss: 0.6906 - val_accuracy: 0.7504
Epoch 6/50
636/636 [==============================] - 4s 7ms/step - loss: 0.6772 - accuracy: 0.7526 - val_loss: 0.6514 - val_accuracy: 0.7655
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6405 - accuracy: 0.7699 - val_loss: 0.6117 - val_accuracy: 0.7846
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6089 - accuracy: 0.7836 - val_loss: 0.5807 - val_accuracy: 0.7909
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5833 - accuracy: 0.7922 - val_loss: 0.5830 - val_accuracy: 0.8009
Epoch 10/50
636/636 [==============================] - 5s 7ms/step - loss: 0.5614 - accuracy: 0.8012 - val_loss: 0.5460 - val_accuracy: 0.8160
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5404 - accuracy: 0.8078 - val_loss: 0.5576 - val_accuracy: 0.7970
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5228 - accuracy: 0.8139 - val_loss: 0.5154 - val_accuracy: 0.8121
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5062 - accuracy: 0.8188 - val_loss: 0.5002 - val_accuracy: 0.8213
Epoch 14/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4923 - accuracy: 0.8260 - val_loss: 0.4736 - val_accuracy: 0.8363
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4818 - accuracy: 0.8288 - val_loss: 0.4924 - val_accuracy: 0.8192
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4673 - accuracy: 0.8333 - val_loss: 0.4969 - val_accuracy: 0.8263
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4585 - accuracy: 0.8382 - val_loss: 0.4809 - val_accuracy: 0.8320
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4494 - accuracy: 0.8403 - val_loss: 0.5144 - val_accuracy: 0.8111
Epoch 19/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4387 - accuracy: 0.8462 - val_loss: 0.4414 - val_accuracy: 0.8467
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4322 - accuracy: 0.8496 - val_loss: 0.4346 - val_accuracy: 0.8502
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4241 - accuracy: 0.8540 - val_loss: 0.4305 - val_accuracy: 0.8524
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4150 - accuracy: 0.8575 - val_loss: 0.4536 - val_accuracy: 0.8345
Epoch 23/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4103 - accuracy: 0.8599 - val_loss: 0.4159 - val_accuracy: 0.8599
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4012 - accuracy: 0.8634 - val_loss: 0.4128 - val_accuracy: 0.8622
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3992 - accuracy: 0.8625 - val_loss: 0.4248 - val_accuracy: 0.8546
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3926 - accuracy: 0.8665 - val_loss: 0.4158 - val_accuracy: 0.8648
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3877 - accuracy: 0.8678 - val_loss: 0.4139 - val_accuracy: 0.8587
Epoch 28/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3823 - accuracy: 0.8701 - val_loss: 0.3931 - val_accuracy: 0.8697
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3795 - accuracy: 0.8711 - val_loss: 0.4776 - val_accuracy: 0.8390
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3745 - accuracy: 0.8733 - val_loss: 0.4209 - val_accuracy: 0.8520
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3706 - accuracy: 0.8756 - val_loss: 0.3845 - val_accuracy: 0.8734
Epoch 32/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3661 - accuracy: 0.8754 - val_loss: 0.3805 - val_accuracy: 0.8738
Epoch 33/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3629 - accuracy: 0.8763 - val_loss: 0.3787 - val_accuracy: 0.8805
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3589 - accuracy: 0.8780 - val_loss: 0.3580 - val_accuracy: 0.8809
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3583 - accuracy: 0.8775 - val_loss: 0.3627 - val_accuracy: 0.8795
Epoch 36/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3519 - accuracy: 0.8799 - val_loss: 0.4032 - val_accuracy: 0.8654
Epoch 37/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3481 - accuracy: 0.8816 - val_loss: 0.4173 - val_accuracy: 0.8608
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3483 - accuracy: 0.8824 - val_loss: 0.3769 - val_accuracy: 0.8770
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3444 - accuracy: 0.8835 - val_loss: 0.3686 - val_accuracy: 0.8825
Epoch 40/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3430 - accuracy: 0.8826 - val_loss: 0.3695 - val_accuracy: 0.8778
Epoch 41/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3394 - accuracy: 0.8844 - val_loss: 0.3960 - val_accuracy: 0.8662
Epoch 42/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3396 - accuracy: 0.8852 - val_loss: 0.3609 - val_accuracy: 0.8809
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3348 - accuracy: 0.8859 - val_loss: 0.3731 - val_accuracy: 0.8744
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3337 - accuracy: 0.8869 - val_loss: 0.3759 - val_accuracy: 0.8748
Model: "Layers_7"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_104 (Dense) (None, 16) 832
dense_105 (Dense) (None, 16) 272
dense_106 (Dense) (None, 16) 272
dense_107 (Dense) (None, 16) 272
dense_108 (Dense) (None, 16) 272
dense_109 (Dense) (None, 16) 272
dense_110 (Dense) (None, 16) 272
dense_111 (Dense) (None, 5) 85
=================================================================
Total params: 2549 (9.96 KB)
Trainable params: 2549 (9.96 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 7s 8ms/step - loss: 1.3399 - accuracy: 0.4147 - val_loss: 1.1486 - val_accuracy: 0.5029
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 1.0867 - accuracy: 0.5592 - val_loss: 0.9868 - val_accuracy: 0.6183
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9242 - accuracy: 0.6548 - val_loss: 0.8404 - val_accuracy: 0.6922
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8093 - accuracy: 0.7016 - val_loss: 0.7470 - val_accuracy: 0.7303
Epoch 5/50
636/636 [==============================] - 5s 7ms/step - loss: 0.7508 - accuracy: 0.7255 - val_loss: 0.6885 - val_accuracy: 0.7581
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7011 - accuracy: 0.7479 - val_loss: 0.6495 - val_accuracy: 0.7630
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6596 - accuracy: 0.7637 - val_loss: 0.6170 - val_accuracy: 0.7809
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6224 - accuracy: 0.7756 - val_loss: 0.5773 - val_accuracy: 0.7938
Epoch 9/50
636/636 [==============================] - 5s 7ms/step - loss: 0.5787 - accuracy: 0.7959 - val_loss: 0.6041 - val_accuracy: 0.7903
Epoch 10/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5474 - accuracy: 0.8136 - val_loss: 0.5217 - val_accuracy: 0.8337
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5235 - accuracy: 0.8257 - val_loss: 0.4990 - val_accuracy: 0.8296
Epoch 12/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5040 - accuracy: 0.8340 - val_loss: 0.4835 - val_accuracy: 0.8329
Epoch 13/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4900 - accuracy: 0.8381 - val_loss: 0.4714 - val_accuracy: 0.8424
Epoch 14/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4771 - accuracy: 0.8425 - val_loss: 0.4665 - val_accuracy: 0.8365
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4679 - accuracy: 0.8459 - val_loss: 0.4911 - val_accuracy: 0.8270
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4545 - accuracy: 0.8491 - val_loss: 0.4653 - val_accuracy: 0.8461
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4450 - accuracy: 0.8504 - val_loss: 0.4510 - val_accuracy: 0.8424
Epoch 18/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4379 - accuracy: 0.8535 - val_loss: 0.4336 - val_accuracy: 0.8491
Epoch 19/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4295 - accuracy: 0.8567 - val_loss: 0.4280 - val_accuracy: 0.8530
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4238 - accuracy: 0.8577 - val_loss: 0.4239 - val_accuracy: 0.8593
Epoch 21/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4152 - accuracy: 0.8627 - val_loss: 0.4426 - val_accuracy: 0.8489
Epoch 22/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4117 - accuracy: 0.8642 - val_loss: 0.3941 - val_accuracy: 0.8664
Epoch 23/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4013 - accuracy: 0.8680 - val_loss: 0.4014 - val_accuracy: 0.8662
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3994 - accuracy: 0.8677 - val_loss: 0.3991 - val_accuracy: 0.8673
Epoch 25/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3903 - accuracy: 0.8704 - val_loss: 0.4292 - val_accuracy: 0.8432
Epoch 26/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3864 - accuracy: 0.8724 - val_loss: 0.3911 - val_accuracy: 0.8705
Epoch 27/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3817 - accuracy: 0.8736 - val_loss: 0.3948 - val_accuracy: 0.8664
Epoch 28/50
636/636 [==============================] - 4s 5ms/step - loss: 0.3753 - accuracy: 0.8768 - val_loss: 0.3738 - val_accuracy: 0.8730
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3742 - accuracy: 0.8760 - val_loss: 0.4051 - val_accuracy: 0.8603
Epoch 30/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3688 - accuracy: 0.8782 - val_loss: 0.3651 - val_accuracy: 0.8730
Epoch 31/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3647 - accuracy: 0.8792 - val_loss: 0.3734 - val_accuracy: 0.8772
Epoch 32/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3598 - accuracy: 0.8805 - val_loss: 0.3639 - val_accuracy: 0.8801
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3565 - accuracy: 0.8818 - val_loss: 0.3492 - val_accuracy: 0.8844
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3530 - accuracy: 0.8832 - val_loss: 0.3971 - val_accuracy: 0.8673
Epoch 35/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3520 - accuracy: 0.8837 - val_loss: 0.3482 - val_accuracy: 0.8813
Epoch 36/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3487 - accuracy: 0.8826 - val_loss: 0.3615 - val_accuracy: 0.8738
Epoch 37/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3461 - accuracy: 0.8853 - val_loss: 0.3570 - val_accuracy: 0.8789
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3419 - accuracy: 0.8871 - val_loss: 0.3590 - val_accuracy: 0.8787
Model: "Layers_8"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_112 (Dense) (None, 16) 832
dense_113 (Dense) (None, 16) 272
dense_114 (Dense) (None, 16) 272
dense_115 (Dense) (None, 16) 272
dense_116 (Dense) (None, 16) 272
dense_117 (Dense) (None, 16) 272
dense_118 (Dense) (None, 16) 272
dense_119 (Dense) (None, 16) 272
dense_120 (Dense) (None, 5) 85
=================================================================
Total params: 2821 (11.02 KB)
Trainable params: 2821 (11.02 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 7s 8ms/step - loss: 1.1880 - accuracy: 0.5346 - val_loss: 0.9894 - val_accuracy: 0.6059
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9586 - accuracy: 0.6191 - val_loss: 0.9218 - val_accuracy: 0.6279
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8689 - accuracy: 0.6659 - val_loss: 0.7943 - val_accuracy: 0.7036
Epoch 4/50
636/636 [==============================] - 4s 7ms/step - loss: 0.7931 - accuracy: 0.7049 - val_loss: 0.7394 - val_accuracy: 0.7341
Epoch 5/50
636/636 [==============================] - 5s 8ms/step - loss: 0.7417 - accuracy: 0.7280 - val_loss: 0.6908 - val_accuracy: 0.7500
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7074 - accuracy: 0.7445 - val_loss: 0.6703 - val_accuracy: 0.7551
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6765 - accuracy: 0.7575 - val_loss: 0.6384 - val_accuracy: 0.7814
Epoch 8/50
636/636 [==============================] - 4s 7ms/step - loss: 0.6528 - accuracy: 0.7672 - val_loss: 0.6656 - val_accuracy: 0.7634
Epoch 9/50
636/636 [==============================] - 5s 8ms/step - loss: 0.6278 - accuracy: 0.7765 - val_loss: 0.6537 - val_accuracy: 0.7761
Epoch 10/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6078 - accuracy: 0.7835 - val_loss: 0.5777 - val_accuracy: 0.7950
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5902 - accuracy: 0.7897 - val_loss: 0.5633 - val_accuracy: 0.7954
Epoch 12/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5725 - accuracy: 0.7963 - val_loss: 0.5511 - val_accuracy: 0.8046
Epoch 13/50
636/636 [==============================] - 5s 8ms/step - loss: 0.5578 - accuracy: 0.8006 - val_loss: 0.5256 - val_accuracy: 0.8117
Epoch 14/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5430 - accuracy: 0.8052 - val_loss: 0.5226 - val_accuracy: 0.8139
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5291 - accuracy: 0.8113 - val_loss: 0.4951 - val_accuracy: 0.8221
Epoch 16/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5155 - accuracy: 0.8138 - val_loss: 0.5413 - val_accuracy: 0.8029
Epoch 17/50
636/636 [==============================] - 5s 8ms/step - loss: 0.5027 - accuracy: 0.8190 - val_loss: 0.5156 - val_accuracy: 0.8190
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4921 - accuracy: 0.8237 - val_loss: 0.5024 - val_accuracy: 0.8221
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4808 - accuracy: 0.8257 - val_loss: 0.5032 - val_accuracy: 0.8274
Epoch 20/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4754 - accuracy: 0.8279 - val_loss: 0.5036 - val_accuracy: 0.8176
Epoch 21/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4668 - accuracy: 0.8329 - val_loss: 0.4508 - val_accuracy: 0.8428
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4549 - accuracy: 0.8364 - val_loss: 0.4386 - val_accuracy: 0.8432
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4468 - accuracy: 0.8423 - val_loss: 0.4356 - val_accuracy: 0.8459
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4409 - accuracy: 0.8462 - val_loss: 0.4466 - val_accuracy: 0.8381
Epoch 25/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4343 - accuracy: 0.8470 - val_loss: 0.4176 - val_accuracy: 0.8557
Epoch 26/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4258 - accuracy: 0.8529 - val_loss: 0.4332 - val_accuracy: 0.8538
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4186 - accuracy: 0.8551 - val_loss: 0.4106 - val_accuracy: 0.8616
Epoch 28/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4132 - accuracy: 0.8560 - val_loss: 0.4357 - val_accuracy: 0.8475
Epoch 29/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4080 - accuracy: 0.8580 - val_loss: 0.4473 - val_accuracy: 0.8487
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4025 - accuracy: 0.8601 - val_loss: 0.4073 - val_accuracy: 0.8551
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3970 - accuracy: 0.8645 - val_loss: 0.4189 - val_accuracy: 0.8599
Epoch 32/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3905 - accuracy: 0.8660 - val_loss: 0.4197 - val_accuracy: 0.8614
Model: "Layers_9"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_121 (Dense) (None, 16) 832
dense_122 (Dense) (None, 16) 272
dense_123 (Dense) (None, 16) 272
dense_124 (Dense) (None, 16) 272
dense_125 (Dense) (None, 16) 272
dense_126 (Dense) (None, 16) 272
dense_127 (Dense) (None, 16) 272
dense_128 (Dense) (None, 16) 272
dense_129 (Dense) (None, 16) 272
dense_130 (Dense) (None, 5) 85
=================================================================
Total params: 3093 (12.08 KB)
Trainable params: 3093 (12.08 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 8s 8ms/step - loss: 1.2945 - accuracy: 0.4298 - val_loss: 1.1867 - val_accuracy: 0.4780
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 1.1796 - accuracy: 0.4795 - val_loss: 1.1125 - val_accuracy: 0.5159
Epoch 3/50
636/636 [==============================] - 4s 7ms/step - loss: 1.0660 - accuracy: 0.5640 - val_loss: 0.9891 - val_accuracy: 0.6183
Epoch 4/50
636/636 [==============================] - 5s 8ms/step - loss: 0.9452 - accuracy: 0.6382 - val_loss: 0.9135 - val_accuracy: 0.6682
Epoch 5/50
636/636 [==============================] - 5s 7ms/step - loss: 0.8947 - accuracy: 0.6673 - val_loss: 0.8472 - val_accuracy: 0.6887
Epoch 6/50
636/636 [==============================] - 4s 7ms/step - loss: 0.8465 - accuracy: 0.6941 - val_loss: 0.8003 - val_accuracy: 0.7079
Epoch 7/50
636/636 [==============================] - 4s 7ms/step - loss: 0.7928 - accuracy: 0.7159 - val_loss: 0.7611 - val_accuracy: 0.7357
Epoch 8/50
636/636 [==============================] - 5s 8ms/step - loss: 0.7407 - accuracy: 0.7403 - val_loss: 0.7038 - val_accuracy: 0.7567
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6990 - accuracy: 0.7528 - val_loss: 0.6788 - val_accuracy: 0.7555
Epoch 10/50
636/636 [==============================] - 4s 7ms/step - loss: 0.6661 - accuracy: 0.7610 - val_loss: 0.6348 - val_accuracy: 0.7736
Epoch 11/50
636/636 [==============================] - 4s 7ms/step - loss: 0.6376 - accuracy: 0.7718 - val_loss: 0.6101 - val_accuracy: 0.7803
Epoch 12/50
636/636 [==============================] - 5s 8ms/step - loss: 0.6133 - accuracy: 0.7808 - val_loss: 0.5942 - val_accuracy: 0.7915
Epoch 13/50
636/636 [==============================] - 5s 7ms/step - loss: 0.5932 - accuracy: 0.7892 - val_loss: 0.5882 - val_accuracy: 0.7985
Epoch 14/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5699 - accuracy: 0.7981 - val_loss: 0.5436 - val_accuracy: 0.8196
Epoch 15/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5525 - accuracy: 0.8033 - val_loss: 0.5365 - val_accuracy: 0.8131
Epoch 16/50
636/636 [==============================] - 5s 8ms/step - loss: 0.5356 - accuracy: 0.8103 - val_loss: 0.5172 - val_accuracy: 0.8194
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5208 - accuracy: 0.8144 - val_loss: 0.5117 - val_accuracy: 0.8249
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5120 - accuracy: 0.8167 - val_loss: 0.4743 - val_accuracy: 0.8325
Epoch 19/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4993 - accuracy: 0.8201 - val_loss: 0.4955 - val_accuracy: 0.8322
Epoch 20/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4894 - accuracy: 0.8251 - val_loss: 0.4971 - val_accuracy: 0.8261
Epoch 21/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4769 - accuracy: 0.8305 - val_loss: 0.4495 - val_accuracy: 0.8449
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4680 - accuracy: 0.8341 - val_loss: 0.4427 - val_accuracy: 0.8422
Epoch 23/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4562 - accuracy: 0.8374 - val_loss: 0.4328 - val_accuracy: 0.8526
Epoch 24/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4491 - accuracy: 0.8453 - val_loss: 0.4505 - val_accuracy: 0.8500
Epoch 25/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4405 - accuracy: 0.8459 - val_loss: 0.4782 - val_accuracy: 0.8436
Epoch 26/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4354 - accuracy: 0.8497 - val_loss: 0.5480 - val_accuracy: 0.8131
Epoch 27/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4262 - accuracy: 0.8537 - val_loss: 0.4150 - val_accuracy: 0.8620
Epoch 28/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4187 - accuracy: 0.8586 - val_loss: 0.4332 - val_accuracy: 0.8532
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4142 - accuracy: 0.8589 - val_loss: 0.4265 - val_accuracy: 0.8589
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4103 - accuracy: 0.8616 - val_loss: 0.4038 - val_accuracy: 0.8620
Epoch 31/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4042 - accuracy: 0.8650 - val_loss: 0.4664 - val_accuracy: 0.8439
Epoch 32/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4024 - accuracy: 0.8651 - val_loss: 0.4126 - val_accuracy: 0.8632
Epoch 33/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3970 - accuracy: 0.8667 - val_loss: 0.3945 - val_accuracy: 0.8693
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3929 - accuracy: 0.8696 - val_loss: 0.4079 - val_accuracy: 0.8642
Epoch 35/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3875 - accuracy: 0.8701 - val_loss: 0.3906 - val_accuracy: 0.8734
Epoch 36/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3851 - accuracy: 0.8724 - val_loss: 0.3864 - val_accuracy: 0.8707
Epoch 37/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3785 - accuracy: 0.8722 - val_loss: 0.3997 - val_accuracy: 0.8717
Epoch 38/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3800 - accuracy: 0.8716 - val_loss: 0.3786 - val_accuracy: 0.8779
Epoch 39/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3737 - accuracy: 0.8746 - val_loss: 0.3863 - val_accuracy: 0.8728
Epoch 40/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3703 - accuracy: 0.8750 - val_loss: 0.3861 - val_accuracy: 0.8807
Epoch 41/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3675 - accuracy: 0.8767 - val_loss: 0.4811 - val_accuracy: 0.8432
Epoch 42/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3668 - accuracy: 0.8774 - val_loss: 0.4333 - val_accuracy: 0.8612
Epoch 43/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3605 - accuracy: 0.8799 - val_loss: 0.3847 - val_accuracy: 0.8742
Epoch 44/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3587 - accuracy: 0.8786 - val_loss: 0.4333 - val_accuracy: 0.8473
Epoch 45/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3554 - accuracy: 0.8793 - val_loss: 0.4042 - val_accuracy: 0.8616
Model: "Layers_10"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_131 (Dense) (None, 16) 832
dense_132 (Dense) (None, 16) 272
dense_133 (Dense) (None, 16) 272
dense_134 (Dense) (None, 16) 272
dense_135 (Dense) (None, 16) 272
dense_136 (Dense) (None, 16) 272
dense_137 (Dense) (None, 16) 272
dense_138 (Dense) (None, 16) 272
dense_139 (Dense) (None, 16) 272
dense_140 (Dense) (None, 16) 272
dense_141 (Dense) (None, 5) 85
=================================================================
Total params: 3365 (13.14 KB)
Trainable params: 3365 (13.14 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 6s 8ms/step - loss: 1.2983 - accuracy: 0.4487 - val_loss: 1.0936 - val_accuracy: 0.5936
Epoch 2/50
636/636 [==============================] - 5s 8ms/step - loss: 1.0024 - accuracy: 0.6199 - val_loss: 0.8863 - val_accuracy: 0.6641
Epoch 3/50
636/636 [==============================] - 5s 7ms/step - loss: 0.8863 - accuracy: 0.6713 - val_loss: 0.8145 - val_accuracy: 0.7009
Epoch 4/50
636/636 [==============================] - 4s 7ms/step - loss: 0.8233 - accuracy: 0.7037 - val_loss: 0.7639 - val_accuracy: 0.7374
Epoch 5/50
636/636 [==============================] - 5s 7ms/step - loss: 0.7738 - accuracy: 0.7233 - val_loss: 0.7962 - val_accuracy: 0.7162
Epoch 6/50
636/636 [==============================] - 6s 9ms/step - loss: 0.7222 - accuracy: 0.7484 - val_loss: 0.6966 - val_accuracy: 0.7626
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6722 - accuracy: 0.7647 - val_loss: 0.6418 - val_accuracy: 0.7763
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6386 - accuracy: 0.7792 - val_loss: 0.6182 - val_accuracy: 0.7828
Epoch 9/50
636/636 [==============================] - 5s 8ms/step - loss: 0.6060 - accuracy: 0.7903 - val_loss: 0.6195 - val_accuracy: 0.7901
Epoch 10/50
636/636 [==============================] - 5s 8ms/step - loss: 0.5758 - accuracy: 0.8037 - val_loss: 0.5379 - val_accuracy: 0.8202
Epoch 11/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5568 - accuracy: 0.8117 - val_loss: 0.5215 - val_accuracy: 0.8215
Epoch 12/50
636/636 [==============================] - 4s 7ms/step - loss: 0.5394 - accuracy: 0.8161 - val_loss: 0.5753 - val_accuracy: 0.8064
Epoch 13/50
636/636 [==============================] - 5s 8ms/step - loss: 0.5222 - accuracy: 0.8231 - val_loss: 0.4970 - val_accuracy: 0.8351
Epoch 14/50
636/636 [==============================] - 5s 7ms/step - loss: 0.5069 - accuracy: 0.8273 - val_loss: 0.4971 - val_accuracy: 0.8325
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4913 - accuracy: 0.8310 - val_loss: 0.5092 - val_accuracy: 0.8282
Epoch 16/50
636/636 [==============================] - 5s 7ms/step - loss: 0.4664 - accuracy: 0.8406 - val_loss: 0.4928 - val_accuracy: 0.8347
Epoch 17/50
636/636 [==============================] - 5s 9ms/step - loss: 0.4495 - accuracy: 0.8442 - val_loss: 0.4395 - val_accuracy: 0.8526
Epoch 18/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4386 - accuracy: 0.8496 - val_loss: 0.4919 - val_accuracy: 0.8331
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4256 - accuracy: 0.8537 - val_loss: 0.4902 - val_accuracy: 0.8402
Epoch 20/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4169 - accuracy: 0.8553 - val_loss: 0.4811 - val_accuracy: 0.8412
Epoch 21/50
636/636 [==============================] - 5s 9ms/step - loss: 0.4103 - accuracy: 0.8599 - val_loss: 0.4111 - val_accuracy: 0.8620
Epoch 22/50
636/636 [==============================] - 4s 7ms/step - loss: 0.4012 - accuracy: 0.8633 - val_loss: 0.3719 - val_accuracy: 0.8748
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3932 - accuracy: 0.8667 - val_loss: 0.3946 - val_accuracy: 0.8650
Epoch 24/50
636/636 [==============================] - 5s 8ms/step - loss: 0.3892 - accuracy: 0.8698 - val_loss: 0.4250 - val_accuracy: 0.8585
Epoch 25/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3833 - accuracy: 0.8698 - val_loss: 0.3872 - val_accuracy: 0.8742
Epoch 26/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3811 - accuracy: 0.8734 - val_loss: 0.4051 - val_accuracy: 0.8658
Epoch 27/50
636/636 [==============================] - 4s 7ms/step - loss: 0.3742 - accuracy: 0.8778 - val_loss: 0.3889 - val_accuracy: 0.8724
In [ ]:
accuracy_measures_hidden_layers = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Hidden Layers")
In [ ]:
accuracy_measures_tmp = {}
accuracy_measures_tmp['Layers_1'] = accuracy_measures['Layers_1']
accuracy_measures_tmp['Layers_2'] = accuracy_measures['Layers_2']
accuracy_measures_tmp['Layers_3'] = accuracy_measures['Layers_3']
accuracy_measures_tmp['Layers_7'] = accuracy_measures['Layers_7']
accuracy_measures_tmp['Layers_9'] = accuracy_measures['Layers_9']
accuracy_measures_tmp['Layers_10'] = accuracy_measures['Layers_10']
plot_accuracy_measures(accuracy_measures_tmp, "Compare Hidden Layers")
The Layers_9 Layers_10 completed faster and with higher accuracy but is not stable. Layer_1 was stable but lower accuracy. Layer_3 is a good candidate it shows signs of overfitting. Layer_2 is stable and have a decent accuracy. I will start small by choosing model with 2 layers.
Determining nodes in a layers¶
More nodes means more possibility for the model to learn complex relationships. However similar to number of layers, it will results to more training and resources.
On my experiment I started with a low number of 16 nodes.
In [ ]:
# reset the accuracy measures
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Exhaustive approach. Try all combinations of layers and the number of nodes;
node_list_layer_1 = [16, 24, 32]
node_list_layer_2 = [16, 24]
#node_list_layer_3 = [16]
for nbr_of_nodes_layer_1 in node_list_layer_1:
for nbr_of_nodes_layer_2 in node_list_layer_2:
#for nbr_of_nodes_layer_3 in node_list_layer_3:
custom_layers = [
layers.Dense(nbr_of_nodes_layer_1, activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(nbr_of_nodes_layer_2, activation=model_config['hidden_activation']),
#layers.Dense(nbr_of_nodes_layer_3, activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Nodes-' + str(nbr_of_nodes_layer_1) + '-' + str(nbr_of_nodes_layer_2)
#model_name = 'Nodes-' + str(nbr_of_nodes_layer_1) + '-' + str(nbr_of_nodes_layer_2) + '-' + str(nbr_of_nodes_layer_3)
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Nodes-16-16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_169 (Dense) (None, 16) 832
dense_170 (Dense) (None, 16) 272
dense_171 (Dense) (None, 5) 85
=================================================================
Total params: 1189 (4.64 KB)
Trainable params: 1189 (4.64 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 1.2631 - accuracy: 0.5350 - val_loss: 0.9916 - val_accuracy: 0.6496
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9089 - accuracy: 0.6751 - val_loss: 0.8480 - val_accuracy: 0.7036
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8114 - accuracy: 0.7106 - val_loss: 0.7743 - val_accuracy: 0.7223
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7476 - accuracy: 0.7350 - val_loss: 0.7279 - val_accuracy: 0.7376
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7038 - accuracy: 0.7474 - val_loss: 0.6873 - val_accuracy: 0.7522
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6702 - accuracy: 0.7583 - val_loss: 0.6617 - val_accuracy: 0.7533
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6440 - accuracy: 0.7659 - val_loss: 0.6371 - val_accuracy: 0.7740
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6230 - accuracy: 0.7759 - val_loss: 0.6239 - val_accuracy: 0.7803
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6052 - accuracy: 0.7797 - val_loss: 0.6112 - val_accuracy: 0.7828
Epoch 10/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5902 - accuracy: 0.7852 - val_loss: 0.6002 - val_accuracy: 0.7826
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5757 - accuracy: 0.7889 - val_loss: 0.5752 - val_accuracy: 0.7869
Epoch 12/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5610 - accuracy: 0.7934 - val_loss: 0.5742 - val_accuracy: 0.7834
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5502 - accuracy: 0.7979 - val_loss: 0.5535 - val_accuracy: 0.7950
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5388 - accuracy: 0.8040 - val_loss: 0.5501 - val_accuracy: 0.7952
Epoch 15/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5289 - accuracy: 0.8079 - val_loss: 0.5320 - val_accuracy: 0.8050
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5202 - accuracy: 0.8119 - val_loss: 0.5290 - val_accuracy: 0.8158
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5125 - accuracy: 0.8159 - val_loss: 0.5307 - val_accuracy: 0.8149
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5048 - accuracy: 0.8218 - val_loss: 0.5129 - val_accuracy: 0.8168
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4976 - accuracy: 0.8234 - val_loss: 0.5031 - val_accuracy: 0.8231
Epoch 20/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4925 - accuracy: 0.8282 - val_loss: 0.5000 - val_accuracy: 0.8280
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4870 - accuracy: 0.8287 - val_loss: 0.4927 - val_accuracy: 0.8304
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4805 - accuracy: 0.8331 - val_loss: 0.5017 - val_accuracy: 0.8190
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4756 - accuracy: 0.8328 - val_loss: 0.4845 - val_accuracy: 0.8316
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4712 - accuracy: 0.8358 - val_loss: 0.4794 - val_accuracy: 0.8392
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4672 - accuracy: 0.8378 - val_loss: 0.4768 - val_accuracy: 0.8400
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4617 - accuracy: 0.8400 - val_loss: 0.4723 - val_accuracy: 0.8426
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4587 - accuracy: 0.8415 - val_loss: 0.4754 - val_accuracy: 0.8375
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4538 - accuracy: 0.8435 - val_loss: 0.4673 - val_accuracy: 0.8363
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4504 - accuracy: 0.8441 - val_loss: 0.4803 - val_accuracy: 0.8410
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4470 - accuracy: 0.8451 - val_loss: 0.4605 - val_accuracy: 0.8396
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4446 - accuracy: 0.8460 - val_loss: 0.4549 - val_accuracy: 0.8449
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4407 - accuracy: 0.8474 - val_loss: 0.4482 - val_accuracy: 0.8544
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4380 - accuracy: 0.8483 - val_loss: 0.4558 - val_accuracy: 0.8439
Epoch 34/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4343 - accuracy: 0.8476 - val_loss: 0.4423 - val_accuracy: 0.8453
Epoch 35/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4319 - accuracy: 0.8512 - val_loss: 0.4465 - val_accuracy: 0.8457
Epoch 36/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4294 - accuracy: 0.8521 - val_loss: 0.4551 - val_accuracy: 0.8418
Epoch 37/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4260 - accuracy: 0.8511 - val_loss: 0.4479 - val_accuracy: 0.8422
Model: "Nodes-16-24"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_172 (Dense) (None, 16) 832
dense_173 (Dense) (None, 24) 408
dense_174 (Dense) (None, 5) 125
=================================================================
Total params: 1365 (5.33 KB)
Trainable params: 1365 (5.33 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2127 - accuracy: 0.5831 - val_loss: 0.9428 - val_accuracy: 0.6816
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8753 - accuracy: 0.6959 - val_loss: 0.8087 - val_accuracy: 0.7176
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7790 - accuracy: 0.7226 - val_loss: 0.7327 - val_accuracy: 0.7300
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7124 - accuracy: 0.7461 - val_loss: 0.6840 - val_accuracy: 0.7555
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6651 - accuracy: 0.7600 - val_loss: 0.6423 - val_accuracy: 0.7693
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6290 - accuracy: 0.7746 - val_loss: 0.6115 - val_accuracy: 0.7803
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6013 - accuracy: 0.7823 - val_loss: 0.5870 - val_accuracy: 0.7905
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5787 - accuracy: 0.7929 - val_loss: 0.5729 - val_accuracy: 0.7938
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5616 - accuracy: 0.7976 - val_loss: 0.5632 - val_accuracy: 0.8078
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5468 - accuracy: 0.8023 - val_loss: 0.5414 - val_accuracy: 0.8072
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5350 - accuracy: 0.8065 - val_loss: 0.5288 - val_accuracy: 0.8135
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5226 - accuracy: 0.8122 - val_loss: 0.5202 - val_accuracy: 0.8039
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5131 - accuracy: 0.8158 - val_loss: 0.5041 - val_accuracy: 0.8223
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5036 - accuracy: 0.8194 - val_loss: 0.5086 - val_accuracy: 0.8154
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4949 - accuracy: 0.8202 - val_loss: 0.4940 - val_accuracy: 0.8164
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4863 - accuracy: 0.8257 - val_loss: 0.5054 - val_accuracy: 0.8151
Epoch 17/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4795 - accuracy: 0.8281 - val_loss: 0.4846 - val_accuracy: 0.8324
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4729 - accuracy: 0.8305 - val_loss: 0.4721 - val_accuracy: 0.8353
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4660 - accuracy: 0.8332 - val_loss: 0.4730 - val_accuracy: 0.8304
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4613 - accuracy: 0.8330 - val_loss: 0.4748 - val_accuracy: 0.8276
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4551 - accuracy: 0.8364 - val_loss: 0.4636 - val_accuracy: 0.8408
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4499 - accuracy: 0.8393 - val_loss: 0.4747 - val_accuracy: 0.8251
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4445 - accuracy: 0.8418 - val_loss: 0.4478 - val_accuracy: 0.8424
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4389 - accuracy: 0.8418 - val_loss: 0.4465 - val_accuracy: 0.8382
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4349 - accuracy: 0.8424 - val_loss: 0.4394 - val_accuracy: 0.8443
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4299 - accuracy: 0.8450 - val_loss: 0.4367 - val_accuracy: 0.8402
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4259 - accuracy: 0.8468 - val_loss: 0.4338 - val_accuracy: 0.8473
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4218 - accuracy: 0.8470 - val_loss: 0.4265 - val_accuracy: 0.8461
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4176 - accuracy: 0.8485 - val_loss: 0.4554 - val_accuracy: 0.8426
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4127 - accuracy: 0.8510 - val_loss: 0.4242 - val_accuracy: 0.8534
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4104 - accuracy: 0.8499 - val_loss: 0.4143 - val_accuracy: 0.8565
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4061 - accuracy: 0.8524 - val_loss: 0.4149 - val_accuracy: 0.8550
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4032 - accuracy: 0.8530 - val_loss: 0.4209 - val_accuracy: 0.8538
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3980 - accuracy: 0.8552 - val_loss: 0.4152 - val_accuracy: 0.8392
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3956 - accuracy: 0.8582 - val_loss: 0.4078 - val_accuracy: 0.8573
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3930 - accuracy: 0.8583 - val_loss: 0.4056 - val_accuracy: 0.8561
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3883 - accuracy: 0.8619 - val_loss: 0.4139 - val_accuracy: 0.8457
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3870 - accuracy: 0.8626 - val_loss: 0.3938 - val_accuracy: 0.8610
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3841 - accuracy: 0.8637 - val_loss: 0.3973 - val_accuracy: 0.8591
Epoch 40/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3792 - accuracy: 0.8640 - val_loss: 0.4165 - val_accuracy: 0.8561
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3771 - accuracy: 0.8667 - val_loss: 0.4045 - val_accuracy: 0.8607
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3743 - accuracy: 0.8683 - val_loss: 0.3969 - val_accuracy: 0.8632
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3715 - accuracy: 0.8681 - val_loss: 0.3834 - val_accuracy: 0.8691
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3694 - accuracy: 0.8675 - val_loss: 0.3880 - val_accuracy: 0.8559
Epoch 45/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3648 - accuracy: 0.8691 - val_loss: 0.3969 - val_accuracy: 0.8601
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3638 - accuracy: 0.8706 - val_loss: 0.3729 - val_accuracy: 0.8685
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3620 - accuracy: 0.8725 - val_loss: 0.3733 - val_accuracy: 0.8730
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3584 - accuracy: 0.8738 - val_loss: 0.3985 - val_accuracy: 0.8550
Epoch 49/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3573 - accuracy: 0.8734 - val_loss: 0.3688 - val_accuracy: 0.8742
Epoch 50/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3546 - accuracy: 0.8763 - val_loss: 0.3797 - val_accuracy: 0.8671
Model: "Nodes-24-16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_175 (Dense) (None, 24) 1248
dense_176 (Dense) (None, 16) 400
dense_177 (Dense) (None, 5) 85
=================================================================
Total params: 1733 (6.77 KB)
Trainable params: 1733 (6.77 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2270 - accuracy: 0.5367 - val_loss: 0.9380 - val_accuracy: 0.6460
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8709 - accuracy: 0.6855 - val_loss: 0.7979 - val_accuracy: 0.7138
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7769 - accuracy: 0.7203 - val_loss: 0.7335 - val_accuracy: 0.7335
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7119 - accuracy: 0.7437 - val_loss: 0.6841 - val_accuracy: 0.7541
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6648 - accuracy: 0.7592 - val_loss: 0.6412 - val_accuracy: 0.7750
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6254 - accuracy: 0.7721 - val_loss: 0.6076 - val_accuracy: 0.7824
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5928 - accuracy: 0.7827 - val_loss: 0.5777 - val_accuracy: 0.7976
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5641 - accuracy: 0.7940 - val_loss: 0.5535 - val_accuracy: 0.7962
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5398 - accuracy: 0.8045 - val_loss: 0.5389 - val_accuracy: 0.8029
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5202 - accuracy: 0.8112 - val_loss: 0.5346 - val_accuracy: 0.8062
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5047 - accuracy: 0.8158 - val_loss: 0.5107 - val_accuracy: 0.8219
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4904 - accuracy: 0.8224 - val_loss: 0.4909 - val_accuracy: 0.8237
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4792 - accuracy: 0.8254 - val_loss: 0.4753 - val_accuracy: 0.8300
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4688 - accuracy: 0.8315 - val_loss: 0.4805 - val_accuracy: 0.8290
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4594 - accuracy: 0.8357 - val_loss: 0.4615 - val_accuracy: 0.8325
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4495 - accuracy: 0.8402 - val_loss: 0.4669 - val_accuracy: 0.8420
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4435 - accuracy: 0.8431 - val_loss: 0.4682 - val_accuracy: 0.8306
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4359 - accuracy: 0.8442 - val_loss: 0.4499 - val_accuracy: 0.8436
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4302 - accuracy: 0.8460 - val_loss: 0.4429 - val_accuracy: 0.8428
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4252 - accuracy: 0.8480 - val_loss: 0.4395 - val_accuracy: 0.8465
Epoch 21/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4204 - accuracy: 0.8499 - val_loss: 0.4316 - val_accuracy: 0.8445
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4145 - accuracy: 0.8531 - val_loss: 0.4417 - val_accuracy: 0.8414
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4105 - accuracy: 0.8530 - val_loss: 0.4312 - val_accuracy: 0.8530
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4056 - accuracy: 0.8570 - val_loss: 0.4188 - val_accuracy: 0.8518
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4023 - accuracy: 0.8579 - val_loss: 0.4147 - val_accuracy: 0.8506
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3972 - accuracy: 0.8603 - val_loss: 0.4104 - val_accuracy: 0.8563
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3935 - accuracy: 0.8599 - val_loss: 0.4089 - val_accuracy: 0.8540
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3898 - accuracy: 0.8630 - val_loss: 0.4077 - val_accuracy: 0.8534
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3876 - accuracy: 0.8622 - val_loss: 0.4247 - val_accuracy: 0.8518
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3829 - accuracy: 0.8654 - val_loss: 0.4102 - val_accuracy: 0.8557
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3805 - accuracy: 0.8665 - val_loss: 0.3881 - val_accuracy: 0.8620
Epoch 32/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3767 - accuracy: 0.8666 - val_loss: 0.3946 - val_accuracy: 0.8603
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3745 - accuracy: 0.8704 - val_loss: 0.3911 - val_accuracy: 0.8614
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3698 - accuracy: 0.8704 - val_loss: 0.3915 - val_accuracy: 0.8636
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3665 - accuracy: 0.8737 - val_loss: 0.3867 - val_accuracy: 0.8662
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3649 - accuracy: 0.8738 - val_loss: 0.3831 - val_accuracy: 0.8634
Epoch 37/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3611 - accuracy: 0.8736 - val_loss: 0.3923 - val_accuracy: 0.8612
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3588 - accuracy: 0.8759 - val_loss: 0.3691 - val_accuracy: 0.8734
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3561 - accuracy: 0.8765 - val_loss: 0.3710 - val_accuracy: 0.8758
Epoch 40/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3519 - accuracy: 0.8782 - val_loss: 0.3823 - val_accuracy: 0.8689
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3503 - accuracy: 0.8785 - val_loss: 0.3771 - val_accuracy: 0.8705
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3473 - accuracy: 0.8784 - val_loss: 0.3731 - val_accuracy: 0.8719
Epoch 43/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3440 - accuracy: 0.8807 - val_loss: 0.3760 - val_accuracy: 0.8699
Epoch 44/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3411 - accuracy: 0.8808 - val_loss: 0.3627 - val_accuracy: 0.8742
Model: "Nodes-24-24"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_178 (Dense) (None, 24) 1248
dense_179 (Dense) (None, 24) 600
dense_180 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1917 - accuracy: 0.5618 - val_loss: 0.9638 - val_accuracy: 0.6612
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8844 - accuracy: 0.6918 - val_loss: 0.8157 - val_accuracy: 0.7189
Epoch 3/50
636/636 [==============================] - 2s 4ms/step - loss: 0.7834 - accuracy: 0.7206 - val_loss: 0.7414 - val_accuracy: 0.7280
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7106 - accuracy: 0.7486 - val_loss: 0.6787 - val_accuracy: 0.7647
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6540 - accuracy: 0.7712 - val_loss: 0.6317 - val_accuracy: 0.7759
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6133 - accuracy: 0.7803 - val_loss: 0.5979 - val_accuracy: 0.7919
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5845 - accuracy: 0.7899 - val_loss: 0.5750 - val_accuracy: 0.7968
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5625 - accuracy: 0.7956 - val_loss: 0.5543 - val_accuracy: 0.7995
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5441 - accuracy: 0.8041 - val_loss: 0.5472 - val_accuracy: 0.7987
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5289 - accuracy: 0.8097 - val_loss: 0.5337 - val_accuracy: 0.8082
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5169 - accuracy: 0.8114 - val_loss: 0.5194 - val_accuracy: 0.8109
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5036 - accuracy: 0.8164 - val_loss: 0.5171 - val_accuracy: 0.8143
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4941 - accuracy: 0.8208 - val_loss: 0.4926 - val_accuracy: 0.8259
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4844 - accuracy: 0.8222 - val_loss: 0.4868 - val_accuracy: 0.8225
Epoch 15/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4757 - accuracy: 0.8274 - val_loss: 0.4738 - val_accuracy: 0.8257
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4663 - accuracy: 0.8310 - val_loss: 0.4796 - val_accuracy: 0.8300
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4600 - accuracy: 0.8352 - val_loss: 0.4748 - val_accuracy: 0.8282
Epoch 18/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4530 - accuracy: 0.8361 - val_loss: 0.4580 - val_accuracy: 0.8382
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4461 - accuracy: 0.8413 - val_loss: 0.4502 - val_accuracy: 0.8382
Epoch 20/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4407 - accuracy: 0.8442 - val_loss: 0.4495 - val_accuracy: 0.8434
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4358 - accuracy: 0.8465 - val_loss: 0.4391 - val_accuracy: 0.8502
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4293 - accuracy: 0.8473 - val_loss: 0.4489 - val_accuracy: 0.8412
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4243 - accuracy: 0.8508 - val_loss: 0.4287 - val_accuracy: 0.8524
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4200 - accuracy: 0.8545 - val_loss: 0.4222 - val_accuracy: 0.8526
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4162 - accuracy: 0.8540 - val_loss: 0.4158 - val_accuracy: 0.8585
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4107 - accuracy: 0.8552 - val_loss: 0.4180 - val_accuracy: 0.8601
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4069 - accuracy: 0.8584 - val_loss: 0.4157 - val_accuracy: 0.8561
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4030 - accuracy: 0.8611 - val_loss: 0.4074 - val_accuracy: 0.8608
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3986 - accuracy: 0.8635 - val_loss: 0.4279 - val_accuracy: 0.8496
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3954 - accuracy: 0.8652 - val_loss: 0.4216 - val_accuracy: 0.8481
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3931 - accuracy: 0.8649 - val_loss: 0.3875 - val_accuracy: 0.8730
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3886 - accuracy: 0.8675 - val_loss: 0.3876 - val_accuracy: 0.8675
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3849 - accuracy: 0.8701 - val_loss: 0.4013 - val_accuracy: 0.8610
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3816 - accuracy: 0.8719 - val_loss: 0.3903 - val_accuracy: 0.8642
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3777 - accuracy: 0.8732 - val_loss: 0.3889 - val_accuracy: 0.8660
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3759 - accuracy: 0.8737 - val_loss: 0.3824 - val_accuracy: 0.8699
Model: "Nodes-32-16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_181 (Dense) (None, 32) 1664
dense_182 (Dense) (None, 16) 528
dense_183 (Dense) (None, 5) 85
=================================================================
Total params: 2277 (8.89 KB)
Trainable params: 2277 (8.89 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1523 - accuracy: 0.5989 - val_loss: 0.9160 - val_accuracy: 0.6568
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8590 - accuracy: 0.6907 - val_loss: 0.8018 - val_accuracy: 0.7070
Epoch 3/50
636/636 [==============================] - 4s 5ms/step - loss: 0.7738 - accuracy: 0.7160 - val_loss: 0.7321 - val_accuracy: 0.7276
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7050 - accuracy: 0.7424 - val_loss: 0.6767 - val_accuracy: 0.7642
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6518 - accuracy: 0.7619 - val_loss: 0.6281 - val_accuracy: 0.7710
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6116 - accuracy: 0.7756 - val_loss: 0.5936 - val_accuracy: 0.7854
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5823 - accuracy: 0.7875 - val_loss: 0.5680 - val_accuracy: 0.7962
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5574 - accuracy: 0.7965 - val_loss: 0.5496 - val_accuracy: 0.8042
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5377 - accuracy: 0.8026 - val_loss: 0.5364 - val_accuracy: 0.8021
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5198 - accuracy: 0.8090 - val_loss: 0.5240 - val_accuracy: 0.8123
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5055 - accuracy: 0.8166 - val_loss: 0.5075 - val_accuracy: 0.8208
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4918 - accuracy: 0.8227 - val_loss: 0.4897 - val_accuracy: 0.8241
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4802 - accuracy: 0.8272 - val_loss: 0.4762 - val_accuracy: 0.8414
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4679 - accuracy: 0.8310 - val_loss: 0.4764 - val_accuracy: 0.8294
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4578 - accuracy: 0.8350 - val_loss: 0.4642 - val_accuracy: 0.8310
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4479 - accuracy: 0.8372 - val_loss: 0.4657 - val_accuracy: 0.8388
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4394 - accuracy: 0.8434 - val_loss: 0.4476 - val_accuracy: 0.8445
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4309 - accuracy: 0.8478 - val_loss: 0.4411 - val_accuracy: 0.8414
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4232 - accuracy: 0.8492 - val_loss: 0.4319 - val_accuracy: 0.8504
Epoch 20/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4162 - accuracy: 0.8523 - val_loss: 0.4336 - val_accuracy: 0.8518
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4103 - accuracy: 0.8556 - val_loss: 0.4119 - val_accuracy: 0.8599
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4034 - accuracy: 0.8588 - val_loss: 0.4230 - val_accuracy: 0.8506
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3960 - accuracy: 0.8623 - val_loss: 0.4040 - val_accuracy: 0.8620
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3913 - accuracy: 0.8656 - val_loss: 0.4039 - val_accuracy: 0.8612
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3876 - accuracy: 0.8654 - val_loss: 0.3910 - val_accuracy: 0.8681
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3820 - accuracy: 0.8655 - val_loss: 0.3855 - val_accuracy: 0.8699
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3769 - accuracy: 0.8722 - val_loss: 0.3939 - val_accuracy: 0.8614
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3731 - accuracy: 0.8724 - val_loss: 0.3786 - val_accuracy: 0.8734
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3698 - accuracy: 0.8730 - val_loss: 0.3933 - val_accuracy: 0.8673
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3650 - accuracy: 0.8745 - val_loss: 0.3901 - val_accuracy: 0.8618
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3620 - accuracy: 0.8738 - val_loss: 0.3842 - val_accuracy: 0.8679
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3575 - accuracy: 0.8774 - val_loss: 0.3826 - val_accuracy: 0.8646
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3552 - accuracy: 0.8769 - val_loss: 0.3690 - val_accuracy: 0.8760
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3504 - accuracy: 0.8783 - val_loss: 0.3651 - val_accuracy: 0.8774
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3465 - accuracy: 0.8810 - val_loss: 0.3791 - val_accuracy: 0.8699
Epoch 36/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3448 - accuracy: 0.8822 - val_loss: 0.3692 - val_accuracy: 0.8740
Epoch 37/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3411 - accuracy: 0.8821 - val_loss: 0.3790 - val_accuracy: 0.8669
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3389 - accuracy: 0.8820 - val_loss: 0.3601 - val_accuracy: 0.8779
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3357 - accuracy: 0.8836 - val_loss: 0.3472 - val_accuracy: 0.8846
Epoch 40/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3323 - accuracy: 0.8829 - val_loss: 0.3742 - val_accuracy: 0.8707
Epoch 41/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3307 - accuracy: 0.8835 - val_loss: 0.3630 - val_accuracy: 0.8740
Epoch 42/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3274 - accuracy: 0.8864 - val_loss: 0.3603 - val_accuracy: 0.8852
Epoch 43/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3254 - accuracy: 0.8858 - val_loss: 0.3461 - val_accuracy: 0.8805
Epoch 44/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3227 - accuracy: 0.8880 - val_loss: 0.3547 - val_accuracy: 0.8803
Epoch 45/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3197 - accuracy: 0.8883 - val_loss: 0.3623 - val_accuracy: 0.8748
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3187 - accuracy: 0.8900 - val_loss: 0.3382 - val_accuracy: 0.8829
Epoch 47/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3162 - accuracy: 0.8905 - val_loss: 0.3330 - val_accuracy: 0.8890
Epoch 48/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3132 - accuracy: 0.8919 - val_loss: 0.3687 - val_accuracy: 0.8675
Epoch 49/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3131 - accuracy: 0.8908 - val_loss: 0.3665 - val_accuracy: 0.8791
Epoch 50/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3099 - accuracy: 0.8925 - val_loss: 0.3423 - val_accuracy: 0.8840
Model: "Nodes-32-24"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_184 (Dense) (None, 32) 1664
dense_185 (Dense) (None, 24) 792
dense_186 (Dense) (None, 5) 125
=================================================================
Total params: 2581 (10.08 KB)
Trainable params: 2581 (10.08 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1537 - accuracy: 0.5736 - val_loss: 0.9125 - val_accuracy: 0.6712
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8366 - accuracy: 0.7012 - val_loss: 0.7697 - val_accuracy: 0.7319
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7373 - accuracy: 0.7354 - val_loss: 0.6961 - val_accuracy: 0.7382
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6694 - accuracy: 0.7593 - val_loss: 0.6381 - val_accuracy: 0.7651
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6210 - accuracy: 0.7734 - val_loss: 0.5970 - val_accuracy: 0.7785
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5837 - accuracy: 0.7871 - val_loss: 0.5652 - val_accuracy: 0.7960
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5541 - accuracy: 0.7978 - val_loss: 0.5349 - val_accuracy: 0.8090
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5298 - accuracy: 0.8075 - val_loss: 0.5221 - val_accuracy: 0.8092
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5104 - accuracy: 0.8154 - val_loss: 0.5058 - val_accuracy: 0.8247
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4931 - accuracy: 0.8234 - val_loss: 0.4856 - val_accuracy: 0.8331
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4782 - accuracy: 0.8290 - val_loss: 0.4703 - val_accuracy: 0.8357
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4636 - accuracy: 0.8365 - val_loss: 0.4605 - val_accuracy: 0.8438
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4524 - accuracy: 0.8411 - val_loss: 0.4454 - val_accuracy: 0.8439
Epoch 14/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4410 - accuracy: 0.8441 - val_loss: 0.4389 - val_accuracy: 0.8487
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4310 - accuracy: 0.8484 - val_loss: 0.4379 - val_accuracy: 0.8457
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4215 - accuracy: 0.8530 - val_loss: 0.4476 - val_accuracy: 0.8487
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4137 - accuracy: 0.8553 - val_loss: 0.4360 - val_accuracy: 0.8485
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4060 - accuracy: 0.8585 - val_loss: 0.4088 - val_accuracy: 0.8603
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3988 - accuracy: 0.8604 - val_loss: 0.4125 - val_accuracy: 0.8599
Epoch 20/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3924 - accuracy: 0.8626 - val_loss: 0.4058 - val_accuracy: 0.8636
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3863 - accuracy: 0.8661 - val_loss: 0.3923 - val_accuracy: 0.8634
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3793 - accuracy: 0.8691 - val_loss: 0.3995 - val_accuracy: 0.8561
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3726 - accuracy: 0.8715 - val_loss: 0.3926 - val_accuracy: 0.8660
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3687 - accuracy: 0.8749 - val_loss: 0.3801 - val_accuracy: 0.8717
Epoch 25/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3644 - accuracy: 0.8754 - val_loss: 0.3730 - val_accuracy: 0.8701
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3590 - accuracy: 0.8779 - val_loss: 0.3836 - val_accuracy: 0.8732
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3550 - accuracy: 0.8783 - val_loss: 0.3677 - val_accuracy: 0.8734
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3505 - accuracy: 0.8787 - val_loss: 0.3527 - val_accuracy: 0.8795
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3473 - accuracy: 0.8801 - val_loss: 0.4268 - val_accuracy: 0.8612
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3431 - accuracy: 0.8819 - val_loss: 0.3541 - val_accuracy: 0.8805
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3418 - accuracy: 0.8816 - val_loss: 0.3695 - val_accuracy: 0.8752
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3370 - accuracy: 0.8855 - val_loss: 0.3472 - val_accuracy: 0.8842
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3336 - accuracy: 0.8862 - val_loss: 0.3640 - val_accuracy: 0.8724
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3288 - accuracy: 0.8870 - val_loss: 0.3758 - val_accuracy: 0.8693
Epoch 35/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3239 - accuracy: 0.8902 - val_loss: 0.3518 - val_accuracy: 0.8783
Epoch 36/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3223 - accuracy: 0.8904 - val_loss: 0.3360 - val_accuracy: 0.8842
Epoch 37/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3175 - accuracy: 0.8910 - val_loss: 0.3603 - val_accuracy: 0.8803
In [ ]:
accuracy_measures_nodes = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Nodes in a Layer")
In [ ]:
In [ ]:
accuracy_measures_tmp = {}
accuracy_measures_tmp['Nodes-16-16'] = accuracy_measures['Nodes-16-16']
accuracy_measures_tmp['Nodes-24-24'] = accuracy_measures['Nodes-24-24']
accuracy_measures_tmp['Nodes-32-24'] = accuracy_measures['Nodes-32-24']
plot_accuracy_measures(accuracy_measures_tmp, "Compare Nodes in a Layer")
The model Nodes-32-24 result to higher accuracy but shows signs of overfitting. The model Nodes-16-16 was table but have a lower accuracy. The model Nodes-24-24 offer a good balance of accuracy and stability among the models.
Choosing Activation Function¶
There are number of algorithms available for activation functions. In general, rectified linear unit works best for regular artificial neural network (ANN) and convolutional network (CNN). Sigmoid works best for RNNs.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
activation_list = ['relu','sigmoid','tanh']
for activation in activation_list:
model_config['hidden_activation'] = activation
custom_layers = [
layers.Dense(24, activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Model-' + activation
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Model-relu"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_214 (Dense) (None, 24) 1248
dense_215 (Dense) (None, 24) 600
dense_216 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2074 - accuracy: 0.5374 - val_loss: 0.9435 - val_accuracy: 0.6537
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8686 - accuracy: 0.6835 - val_loss: 0.8099 - val_accuracy: 0.7103
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7719 - accuracy: 0.7191 - val_loss: 0.7341 - val_accuracy: 0.7303
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7040 - accuracy: 0.7431 - val_loss: 0.6764 - val_accuracy: 0.7500
Epoch 5/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6528 - accuracy: 0.7589 - val_loss: 0.6300 - val_accuracy: 0.7720
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6123 - accuracy: 0.7756 - val_loss: 0.5961 - val_accuracy: 0.7785
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5805 - accuracy: 0.7857 - val_loss: 0.5671 - val_accuracy: 0.7987
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5518 - accuracy: 0.7984 - val_loss: 0.5391 - val_accuracy: 0.8088
Epoch 9/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5259 - accuracy: 0.8107 - val_loss: 0.5189 - val_accuracy: 0.8182
Epoch 10/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5029 - accuracy: 0.8207 - val_loss: 0.5015 - val_accuracy: 0.8229
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4850 - accuracy: 0.8293 - val_loss: 0.4783 - val_accuracy: 0.8324
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4684 - accuracy: 0.8350 - val_loss: 0.4709 - val_accuracy: 0.8322
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4559 - accuracy: 0.8392 - val_loss: 0.4545 - val_accuracy: 0.8414
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4435 - accuracy: 0.8446 - val_loss: 0.4504 - val_accuracy: 0.8473
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4331 - accuracy: 0.8495 - val_loss: 0.4409 - val_accuracy: 0.8516
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4236 - accuracy: 0.8525 - val_loss: 0.4323 - val_accuracy: 0.8493
Epoch 17/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4160 - accuracy: 0.8546 - val_loss: 0.4388 - val_accuracy: 0.8459
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4092 - accuracy: 0.8583 - val_loss: 0.4233 - val_accuracy: 0.8489
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4037 - accuracy: 0.8598 - val_loss: 0.4212 - val_accuracy: 0.8563
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3973 - accuracy: 0.8619 - val_loss: 0.4166 - val_accuracy: 0.8567
Epoch 21/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3917 - accuracy: 0.8633 - val_loss: 0.4012 - val_accuracy: 0.8618
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3862 - accuracy: 0.8655 - val_loss: 0.4023 - val_accuracy: 0.8636
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3812 - accuracy: 0.8680 - val_loss: 0.3941 - val_accuracy: 0.8693
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3764 - accuracy: 0.8708 - val_loss: 0.3892 - val_accuracy: 0.8650
Epoch 25/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3729 - accuracy: 0.8720 - val_loss: 0.3979 - val_accuracy: 0.8638
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3685 - accuracy: 0.8712 - val_loss: 0.3857 - val_accuracy: 0.8754
Epoch 27/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3652 - accuracy: 0.8766 - val_loss: 0.3871 - val_accuracy: 0.8652
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3613 - accuracy: 0.8762 - val_loss: 0.3822 - val_accuracy: 0.8703
Epoch 29/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3586 - accuracy: 0.8761 - val_loss: 0.3885 - val_accuracy: 0.8664
Epoch 30/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3548 - accuracy: 0.8781 - val_loss: 0.3831 - val_accuracy: 0.8662
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3532 - accuracy: 0.8800 - val_loss: 0.3714 - val_accuracy: 0.8742
Model: "Model-sigmoid"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_217 (Dense) (None, 24) 1248
dense_218 (Dense) (None, 24) 600
dense_219 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5475 - accuracy: 0.3785 - val_loss: 1.4269 - val_accuracy: 0.4762
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2874 - accuracy: 0.5045 - val_loss: 1.1654 - val_accuracy: 0.5810
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1046 - accuracy: 0.5918 - val_loss: 1.0351 - val_accuracy: 0.6226
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 1.0010 - accuracy: 0.6373 - val_loss: 0.9575 - val_accuracy: 0.6433
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9339 - accuracy: 0.6579 - val_loss: 0.9003 - val_accuracy: 0.6673
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8890 - accuracy: 0.6736 - val_loss: 0.8634 - val_accuracy: 0.6794
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8565 - accuracy: 0.6876 - val_loss: 0.8361 - val_accuracy: 0.6967
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8316 - accuracy: 0.6981 - val_loss: 0.8138 - val_accuracy: 0.7085
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8105 - accuracy: 0.7071 - val_loss: 0.7943 - val_accuracy: 0.7136
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7936 - accuracy: 0.7136 - val_loss: 0.7837 - val_accuracy: 0.7197
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7785 - accuracy: 0.7181 - val_loss: 0.7620 - val_accuracy: 0.7225
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7632 - accuracy: 0.7209 - val_loss: 0.7515 - val_accuracy: 0.7227
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7502 - accuracy: 0.7236 - val_loss: 0.7369 - val_accuracy: 0.7286
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7375 - accuracy: 0.7268 - val_loss: 0.7321 - val_accuracy: 0.7333
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7254 - accuracy: 0.7318 - val_loss: 0.7146 - val_accuracy: 0.7339
Epoch 16/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7146 - accuracy: 0.7358 - val_loss: 0.7046 - val_accuracy: 0.7417
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7044 - accuracy: 0.7389 - val_loss: 0.6973 - val_accuracy: 0.7415
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6947 - accuracy: 0.7416 - val_loss: 0.6895 - val_accuracy: 0.7502
Epoch 19/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6847 - accuracy: 0.7466 - val_loss: 0.6768 - val_accuracy: 0.7429
Epoch 20/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6767 - accuracy: 0.7492 - val_loss: 0.6735 - val_accuracy: 0.7549
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6685 - accuracy: 0.7525 - val_loss: 0.6611 - val_accuracy: 0.7520
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6601 - accuracy: 0.7560 - val_loss: 0.6557 - val_accuracy: 0.7616
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6525 - accuracy: 0.7585 - val_loss: 0.6499 - val_accuracy: 0.7643
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6446 - accuracy: 0.7652 - val_loss: 0.6407 - val_accuracy: 0.7643
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6369 - accuracy: 0.7673 - val_loss: 0.6333 - val_accuracy: 0.7732
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6302 - accuracy: 0.7713 - val_loss: 0.6255 - val_accuracy: 0.7795
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6228 - accuracy: 0.7745 - val_loss: 0.6209 - val_accuracy: 0.7828
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6164 - accuracy: 0.7778 - val_loss: 0.6193 - val_accuracy: 0.7781
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6101 - accuracy: 0.7803 - val_loss: 0.6158 - val_accuracy: 0.7832
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6048 - accuracy: 0.7823 - val_loss: 0.6106 - val_accuracy: 0.7809
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5992 - accuracy: 0.7857 - val_loss: 0.6041 - val_accuracy: 0.7858
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5929 - accuracy: 0.7874 - val_loss: 0.5939 - val_accuracy: 0.7864
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5878 - accuracy: 0.7895 - val_loss: 0.5888 - val_accuracy: 0.7915
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5821 - accuracy: 0.7899 - val_loss: 0.5823 - val_accuracy: 0.7903
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5778 - accuracy: 0.7928 - val_loss: 0.5800 - val_accuracy: 0.7991
Epoch 36/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5729 - accuracy: 0.7948 - val_loss: 0.5863 - val_accuracy: 0.7889
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5680 - accuracy: 0.7964 - val_loss: 0.5743 - val_accuracy: 0.8042
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5644 - accuracy: 0.7965 - val_loss: 0.5660 - val_accuracy: 0.8011
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5596 - accuracy: 0.7997 - val_loss: 0.5623 - val_accuracy: 0.7991
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5547 - accuracy: 0.8020 - val_loss: 0.5650 - val_accuracy: 0.8021
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5509 - accuracy: 0.8048 - val_loss: 0.5582 - val_accuracy: 0.8084
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5469 - accuracy: 0.8059 - val_loss: 0.5551 - val_accuracy: 0.8105
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5432 - accuracy: 0.8071 - val_loss: 0.5490 - val_accuracy: 0.8194
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5397 - accuracy: 0.8083 - val_loss: 0.5408 - val_accuracy: 0.8109
Epoch 45/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5351 - accuracy: 0.8094 - val_loss: 0.5368 - val_accuracy: 0.8088
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5320 - accuracy: 0.8122 - val_loss: 0.5329 - val_accuracy: 0.8133
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5280 - accuracy: 0.8143 - val_loss: 0.5271 - val_accuracy: 0.8153
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5244 - accuracy: 0.8166 - val_loss: 0.5402 - val_accuracy: 0.8046
Model: "Model-tanh"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_220 (Dense) (None, 24) 1248
dense_221 (Dense) (None, 24) 600
dense_222 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 6ms/step - loss: 1.0765 - accuracy: 0.6218 - val_loss: 0.8518 - val_accuracy: 0.6873
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8098 - accuracy: 0.7056 - val_loss: 0.7666 - val_accuracy: 0.7184
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7463 - accuracy: 0.7251 - val_loss: 0.7141 - val_accuracy: 0.7225
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7004 - accuracy: 0.7420 - val_loss: 0.6824 - val_accuracy: 0.7502
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6698 - accuracy: 0.7523 - val_loss: 0.6556 - val_accuracy: 0.7555
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6449 - accuracy: 0.7660 - val_loss: 0.6271 - val_accuracy: 0.7750
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6240 - accuracy: 0.7732 - val_loss: 0.6067 - val_accuracy: 0.7858
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6037 - accuracy: 0.7818 - val_loss: 0.5974 - val_accuracy: 0.7915
Epoch 9/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5863 - accuracy: 0.7908 - val_loss: 0.5791 - val_accuracy: 0.7926
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5708 - accuracy: 0.7955 - val_loss: 0.5784 - val_accuracy: 0.7962
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5572 - accuracy: 0.8000 - val_loss: 0.5444 - val_accuracy: 0.8109
Epoch 12/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5433 - accuracy: 0.8054 - val_loss: 0.5405 - val_accuracy: 0.8027
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5311 - accuracy: 0.8098 - val_loss: 0.5257 - val_accuracy: 0.8182
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5192 - accuracy: 0.8151 - val_loss: 0.5223 - val_accuracy: 0.8086
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5097 - accuracy: 0.8167 - val_loss: 0.5209 - val_accuracy: 0.8170
Epoch 16/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4989 - accuracy: 0.8209 - val_loss: 0.5380 - val_accuracy: 0.8099
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4916 - accuracy: 0.8252 - val_loss: 0.5066 - val_accuracy: 0.8211
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4825 - accuracy: 0.8256 - val_loss: 0.4910 - val_accuracy: 0.8241
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4748 - accuracy: 0.8305 - val_loss: 0.4851 - val_accuracy: 0.8288
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4677 - accuracy: 0.8331 - val_loss: 0.4790 - val_accuracy: 0.8300
Epoch 21/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4617 - accuracy: 0.8343 - val_loss: 0.4631 - val_accuracy: 0.8355
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4537 - accuracy: 0.8380 - val_loss: 0.4773 - val_accuracy: 0.8318
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4466 - accuracy: 0.8404 - val_loss: 0.4576 - val_accuracy: 0.8373
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4397 - accuracy: 0.8436 - val_loss: 0.4462 - val_accuracy: 0.8408
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4345 - accuracy: 0.8464 - val_loss: 0.4384 - val_accuracy: 0.8424
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4279 - accuracy: 0.8461 - val_loss: 0.4397 - val_accuracy: 0.8516
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4229 - accuracy: 0.8511 - val_loss: 0.4332 - val_accuracy: 0.8453
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4173 - accuracy: 0.8505 - val_loss: 0.4178 - val_accuracy: 0.8514
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4122 - accuracy: 0.8562 - val_loss: 0.4624 - val_accuracy: 0.8430
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4063 - accuracy: 0.8542 - val_loss: 0.4183 - val_accuracy: 0.8487
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4032 - accuracy: 0.8561 - val_loss: 0.4058 - val_accuracy: 0.8567
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3973 - accuracy: 0.8585 - val_loss: 0.4049 - val_accuracy: 0.8587
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3929 - accuracy: 0.8611 - val_loss: 0.4020 - val_accuracy: 0.8557
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3872 - accuracy: 0.8618 - val_loss: 0.4065 - val_accuracy: 0.8585
Epoch 35/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3824 - accuracy: 0.8648 - val_loss: 0.4016 - val_accuracy: 0.8565
Epoch 36/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3795 - accuracy: 0.8652 - val_loss: 0.3928 - val_accuracy: 0.8595
Epoch 37/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3745 - accuracy: 0.8675 - val_loss: 0.4034 - val_accuracy: 0.8551
Epoch 38/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3713 - accuracy: 0.8700 - val_loss: 0.3704 - val_accuracy: 0.8707
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3671 - accuracy: 0.8676 - val_loss: 0.3717 - val_accuracy: 0.8779
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3617 - accuracy: 0.8731 - val_loss: 0.3977 - val_accuracy: 0.8593
Epoch 41/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3585 - accuracy: 0.8731 - val_loss: 0.3967 - val_accuracy: 0.8593
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3552 - accuracy: 0.8746 - val_loss: 0.3749 - val_accuracy: 0.8681
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3505 - accuracy: 0.8753 - val_loss: 0.3641 - val_accuracy: 0.8750
Epoch 44/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3480 - accuracy: 0.8763 - val_loss: 0.3692 - val_accuracy: 0.8650
In [ ]:
accuracy_activation = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Activation Functions")
The activation function relu have higher accuracy and completed faster.
In [ ]:
model_config['hidden_activation'] = 'relu'
Initialize Weights¶
The initial values of weights play a huge role in the speed of learning and finaly accuracy. Multiple initialization techniques exist.
Random normal works best for most cases but we will run an experiment if that is still the case on our dataset.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
initializer_list = ['random_normal', "random_uniform"]
for initializer in initializer_list:
model_config['weights_initializer'] = initializer
custom_layers = [
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Model-' + initializer
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Model-random_normal"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_223 (Dense) (None, 24) 1248
dense_224 (Dense) (None, 24) 600
dense_225 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 6ms/step - loss: 1.2255 - accuracy: 0.5372 - val_loss: 0.9506 - val_accuracy: 0.6586
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8902 - accuracy: 0.6822 - val_loss: 0.8291 - val_accuracy: 0.7109
Epoch 3/50
636/636 [==============================] - 4s 7ms/step - loss: 0.8137 - accuracy: 0.7029 - val_loss: 0.7737 - val_accuracy: 0.7201
Epoch 4/50
636/636 [==============================] - 7s 12ms/step - loss: 0.7602 - accuracy: 0.7225 - val_loss: 0.7305 - val_accuracy: 0.7303
Epoch 5/50
636/636 [==============================] - 5s 7ms/step - loss: 0.7157 - accuracy: 0.7394 - val_loss: 0.6847 - val_accuracy: 0.7520
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6746 - accuracy: 0.7537 - val_loss: 0.6483 - val_accuracy: 0.7598
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6403 - accuracy: 0.7644 - val_loss: 0.6169 - val_accuracy: 0.7809
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6099 - accuracy: 0.7762 - val_loss: 0.5960 - val_accuracy: 0.7854
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5849 - accuracy: 0.7842 - val_loss: 0.5820 - val_accuracy: 0.7932
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5632 - accuracy: 0.7926 - val_loss: 0.5640 - val_accuracy: 0.7991
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5455 - accuracy: 0.8016 - val_loss: 0.5309 - val_accuracy: 0.8113
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5282 - accuracy: 0.8082 - val_loss: 0.5283 - val_accuracy: 0.8001
Epoch 13/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5138 - accuracy: 0.8134 - val_loss: 0.5083 - val_accuracy: 0.8213
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5010 - accuracy: 0.8197 - val_loss: 0.5030 - val_accuracy: 0.8227
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4898 - accuracy: 0.8262 - val_loss: 0.4926 - val_accuracy: 0.8286
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4779 - accuracy: 0.8293 - val_loss: 0.4944 - val_accuracy: 0.8272
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4684 - accuracy: 0.8356 - val_loss: 0.4743 - val_accuracy: 0.8310
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4594 - accuracy: 0.8367 - val_loss: 0.4681 - val_accuracy: 0.8327
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4501 - accuracy: 0.8408 - val_loss: 0.4604 - val_accuracy: 0.8436
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4430 - accuracy: 0.8438 - val_loss: 0.4657 - val_accuracy: 0.8404
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4365 - accuracy: 0.8455 - val_loss: 0.4415 - val_accuracy: 0.8469
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4288 - accuracy: 0.8498 - val_loss: 0.4604 - val_accuracy: 0.8314
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4216 - accuracy: 0.8519 - val_loss: 0.4398 - val_accuracy: 0.8443
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4155 - accuracy: 0.8554 - val_loss: 0.4284 - val_accuracy: 0.8504
Epoch 25/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4106 - accuracy: 0.8553 - val_loss: 0.4296 - val_accuracy: 0.8485
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4034 - accuracy: 0.8587 - val_loss: 0.4153 - val_accuracy: 0.8650
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3993 - accuracy: 0.8630 - val_loss: 0.4084 - val_accuracy: 0.8573
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3935 - accuracy: 0.8646 - val_loss: 0.4082 - val_accuracy: 0.8563
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3895 - accuracy: 0.8630 - val_loss: 0.4269 - val_accuracy: 0.8544
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3850 - accuracy: 0.8649 - val_loss: 0.4254 - val_accuracy: 0.8471
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3827 - accuracy: 0.8666 - val_loss: 0.4320 - val_accuracy: 0.8544
Model: "Model-random_uniform"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_226 (Dense) (None, 24) 1248
dense_227 (Dense) (None, 24) 600
dense_228 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2744 - accuracy: 0.5081 - val_loss: 1.0436 - val_accuracy: 0.6079
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9706 - accuracy: 0.6395 - val_loss: 0.8900 - val_accuracy: 0.6753
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8587 - accuracy: 0.6831 - val_loss: 0.8172 - val_accuracy: 0.6958
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7985 - accuracy: 0.7058 - val_loss: 0.7717 - val_accuracy: 0.7138
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7556 - accuracy: 0.7205 - val_loss: 0.7256 - val_accuracy: 0.7254
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7162 - accuracy: 0.7336 - val_loss: 0.7001 - val_accuracy: 0.7256
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6833 - accuracy: 0.7421 - val_loss: 0.6636 - val_accuracy: 0.7484
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6553 - accuracy: 0.7550 - val_loss: 0.6437 - val_accuracy: 0.7598
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6334 - accuracy: 0.7620 - val_loss: 0.6336 - val_accuracy: 0.7608
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6151 - accuracy: 0.7679 - val_loss: 0.6201 - val_accuracy: 0.7614
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6011 - accuracy: 0.7736 - val_loss: 0.5936 - val_accuracy: 0.7795
Epoch 12/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5858 - accuracy: 0.7816 - val_loss: 0.5853 - val_accuracy: 0.7836
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5735 - accuracy: 0.7880 - val_loss: 0.5673 - val_accuracy: 0.7905
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5616 - accuracy: 0.7934 - val_loss: 0.5634 - val_accuracy: 0.7903
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5499 - accuracy: 0.7988 - val_loss: 0.5467 - val_accuracy: 0.8027
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5389 - accuracy: 0.8051 - val_loss: 0.5421 - val_accuracy: 0.8029
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5290 - accuracy: 0.8086 - val_loss: 0.5346 - val_accuracy: 0.8062
Epoch 18/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5198 - accuracy: 0.8135 - val_loss: 0.5161 - val_accuracy: 0.8139
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5094 - accuracy: 0.8188 - val_loss: 0.5064 - val_accuracy: 0.8243
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5006 - accuracy: 0.8209 - val_loss: 0.5043 - val_accuracy: 0.8253
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4932 - accuracy: 0.8246 - val_loss: 0.4976 - val_accuracy: 0.8247
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4850 - accuracy: 0.8278 - val_loss: 0.5053 - val_accuracy: 0.8241
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4769 - accuracy: 0.8302 - val_loss: 0.4829 - val_accuracy: 0.8335
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4679 - accuracy: 0.8365 - val_loss: 0.4762 - val_accuracy: 0.8320
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4615 - accuracy: 0.8391 - val_loss: 0.4785 - val_accuracy: 0.8282
Epoch 26/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4549 - accuracy: 0.8391 - val_loss: 0.4619 - val_accuracy: 0.8424
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4493 - accuracy: 0.8421 - val_loss: 0.4576 - val_accuracy: 0.8386
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4435 - accuracy: 0.8430 - val_loss: 0.4505 - val_accuracy: 0.8422
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4374 - accuracy: 0.8471 - val_loss: 0.4679 - val_accuracy: 0.8398
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4315 - accuracy: 0.8472 - val_loss: 0.4523 - val_accuracy: 0.8388
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4280 - accuracy: 0.8488 - val_loss: 0.4372 - val_accuracy: 0.8498
Epoch 32/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4229 - accuracy: 0.8504 - val_loss: 0.4273 - val_accuracy: 0.8518
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4190 - accuracy: 0.8508 - val_loss: 0.4315 - val_accuracy: 0.8463
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4139 - accuracy: 0.8516 - val_loss: 0.4216 - val_accuracy: 0.8548
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4105 - accuracy: 0.8548 - val_loss: 0.4301 - val_accuracy: 0.8455
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4070 - accuracy: 0.8572 - val_loss: 0.4290 - val_accuracy: 0.8485
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4034 - accuracy: 0.8573 - val_loss: 0.4343 - val_accuracy: 0.8506
Epoch 38/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4005 - accuracy: 0.8591 - val_loss: 0.4096 - val_accuracy: 0.8579
Epoch 39/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3976 - accuracy: 0.8575 - val_loss: 0.4083 - val_accuracy: 0.8577
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3937 - accuracy: 0.8616 - val_loss: 0.4459 - val_accuracy: 0.8447
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3919 - accuracy: 0.8609 - val_loss: 0.4191 - val_accuracy: 0.8512
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3890 - accuracy: 0.8632 - val_loss: 0.4156 - val_accuracy: 0.8553
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3846 - accuracy: 0.8630 - val_loss: 0.4041 - val_accuracy: 0.8542
In [ ]:
accuracy_measures_weight_init = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Weights Initializers")
Even though random_uniform completed faster, the random_normal result to higher accuracy thus choosing random_normal as weight initializer.
In [ ]:
model_config['weights_initializer'] = 'random_normal'
Tuning Back Propagation¶
Batch Normalization¶
Batch normalization is an important technique to vanishing and exploiting gradients during gradient descent. Batch normalization help achieve higher accuracies with lower epochs, hence is also an optimization technique.
This is no longer necessary because Scaler was already applied during data preparation. However, in order to show the difference, I am also running this experiment.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
normalization_list = ['batch','none']
for normalization in normalization_list:
custom_layers = []
if normalization == 'none':
custom_layers = [
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
else:
custom_layers = [
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.BatchNormalization(),
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.BatchNormalization(),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Normalization-' + normalization
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Normalization-batch"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_229 (Dense) (None, 24) 1248
batch_normalization_2 (Bat (None, 24) 96
chNormalization)
dense_230 (Dense) (None, 24) 600
batch_normalization_3 (Bat (None, 24) 96
chNormalization)
dense_231 (Dense) (None, 5) 125
=================================================================
Total params: 2165 (8.46 KB)
Trainable params: 2069 (8.08 KB)
Non-trainable params: 96 (384.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 0.7314 - accuracy: 0.7317 - val_loss: 0.6417 - val_accuracy: 0.7673
Epoch 2/50
636/636 [==============================] - 5s 7ms/step - loss: 0.5305 - accuracy: 0.8080 - val_loss: 0.5069 - val_accuracy: 0.8147
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4820 - accuracy: 0.8284 - val_loss: 0.5458 - val_accuracy: 0.8174
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4423 - accuracy: 0.8436 - val_loss: 0.4888 - val_accuracy: 0.8247
Epoch 5/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4177 - accuracy: 0.8534 - val_loss: 0.5098 - val_accuracy: 0.8186
Epoch 6/50
636/636 [==============================] - 5s 8ms/step - loss: 0.4033 - accuracy: 0.8589 - val_loss: 0.4688 - val_accuracy: 0.8302
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3966 - accuracy: 0.8599 - val_loss: 0.3701 - val_accuracy: 0.8799
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3771 - accuracy: 0.8673 - val_loss: 0.3977 - val_accuracy: 0.8638
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3763 - accuracy: 0.8661 - val_loss: 0.4601 - val_accuracy: 0.8329
Epoch 10/50
636/636 [==============================] - 5s 7ms/step - loss: 0.3627 - accuracy: 0.8716 - val_loss: 0.4455 - val_accuracy: 0.8526
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3551 - accuracy: 0.8739 - val_loss: 0.4270 - val_accuracy: 0.8504
Epoch 12/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3565 - accuracy: 0.8729 - val_loss: 0.3585 - val_accuracy: 0.8791
Model: "Normalization-none"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_232 (Dense) (None, 24) 1248
dense_233 (Dense) (None, 24) 600
dense_234 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2285 - accuracy: 0.5616 - val_loss: 0.9632 - val_accuracy: 0.6513
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8889 - accuracy: 0.6789 - val_loss: 0.8248 - val_accuracy: 0.7056
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8013 - accuracy: 0.7051 - val_loss: 0.7625 - val_accuracy: 0.7195
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7441 - accuracy: 0.7249 - val_loss: 0.7133 - val_accuracy: 0.7305
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6984 - accuracy: 0.7345 - val_loss: 0.6685 - val_accuracy: 0.7496
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6587 - accuracy: 0.7508 - val_loss: 0.6342 - val_accuracy: 0.7577
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6233 - accuracy: 0.7656 - val_loss: 0.6001 - val_accuracy: 0.7720
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5898 - accuracy: 0.7807 - val_loss: 0.5738 - val_accuracy: 0.7862
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5617 - accuracy: 0.7929 - val_loss: 0.5563 - val_accuracy: 0.8005
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5378 - accuracy: 0.8032 - val_loss: 0.5367 - val_accuracy: 0.8076
Epoch 11/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5188 - accuracy: 0.8103 - val_loss: 0.5069 - val_accuracy: 0.8149
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5003 - accuracy: 0.8196 - val_loss: 0.5006 - val_accuracy: 0.8117
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4853 - accuracy: 0.8227 - val_loss: 0.4785 - val_accuracy: 0.8329
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4713 - accuracy: 0.8293 - val_loss: 0.4709 - val_accuracy: 0.8255
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4595 - accuracy: 0.8339 - val_loss: 0.4598 - val_accuracy: 0.8377
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4481 - accuracy: 0.8375 - val_loss: 0.4598 - val_accuracy: 0.8367
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4381 - accuracy: 0.8443 - val_loss: 0.4594 - val_accuracy: 0.8363
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4304 - accuracy: 0.8458 - val_loss: 0.4376 - val_accuracy: 0.8400
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4217 - accuracy: 0.8511 - val_loss: 0.4348 - val_accuracy: 0.8432
Epoch 20/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4134 - accuracy: 0.8527 - val_loss: 0.4375 - val_accuracy: 0.8436
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4070 - accuracy: 0.8561 - val_loss: 0.4125 - val_accuracy: 0.8577
Epoch 22/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3989 - accuracy: 0.8593 - val_loss: 0.4226 - val_accuracy: 0.8432
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3921 - accuracy: 0.8596 - val_loss: 0.4026 - val_accuracy: 0.8591
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3840 - accuracy: 0.8646 - val_loss: 0.3934 - val_accuracy: 0.8624
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3792 - accuracy: 0.8657 - val_loss: 0.3824 - val_accuracy: 0.8662
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3713 - accuracy: 0.8682 - val_loss: 0.3801 - val_accuracy: 0.8695
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3667 - accuracy: 0.8722 - val_loss: 0.3805 - val_accuracy: 0.8650
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3613 - accuracy: 0.8719 - val_loss: 0.3695 - val_accuracy: 0.8664
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3570 - accuracy: 0.8756 - val_loss: 0.3944 - val_accuracy: 0.8587
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3517 - accuracy: 0.8756 - val_loss: 0.3728 - val_accuracy: 0.8695
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3497 - accuracy: 0.8758 - val_loss: 0.3724 - val_accuracy: 0.8681
In [ ]:
accuracy_measures_normalization = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Normalization Techniques")
Not applying a normalization layer results to a better result.
Optimizers¶
Optimizer are key tools, that help gradient descent, achieve faster results. Optimizers are algorithms, that helps speed up the training process.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
# Batch Normalization: not added
optimizer_list = ['sgd','rmsprop','adam','adagrad']
for optimizer in optimizer_list:
model_config['optimizer'] = optimizer
custom_layers = [
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Optimizer-' + optimizer
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Optimizer-sgd"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_235 (Dense) (None, 24) 1248
dense_236 (Dense) (None, 24) 600
dense_237 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2413 - accuracy: 0.5451 - val_loss: 0.9723 - val_accuracy: 0.6425
Epoch 2/50
636/636 [==============================] - 2s 4ms/step - loss: 0.9061 - accuracy: 0.6646 - val_loss: 0.8485 - val_accuracy: 0.6956
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8372 - accuracy: 0.6907 - val_loss: 0.8000 - val_accuracy: 0.7095
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7888 - accuracy: 0.7099 - val_loss: 0.7585 - val_accuracy: 0.7174
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7469 - accuracy: 0.7238 - val_loss: 0.7123 - val_accuracy: 0.7388
Epoch 6/50
636/636 [==============================] - 2s 4ms/step - loss: 0.7091 - accuracy: 0.7394 - val_loss: 0.6874 - val_accuracy: 0.7404
Epoch 7/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6789 - accuracy: 0.7489 - val_loss: 0.6566 - val_accuracy: 0.7579
Epoch 8/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6533 - accuracy: 0.7580 - val_loss: 0.6417 - val_accuracy: 0.7585
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6299 - accuracy: 0.7676 - val_loss: 0.6380 - val_accuracy: 0.7573
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6090 - accuracy: 0.7776 - val_loss: 0.6155 - val_accuracy: 0.7679
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5914 - accuracy: 0.7862 - val_loss: 0.5792 - val_accuracy: 0.7936
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5723 - accuracy: 0.7941 - val_loss: 0.5741 - val_accuracy: 0.7850
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5559 - accuracy: 0.8010 - val_loss: 0.5487 - val_accuracy: 0.8040
Epoch 14/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5396 - accuracy: 0.8053 - val_loss: 0.5402 - val_accuracy: 0.7964
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5246 - accuracy: 0.8103 - val_loss: 0.5212 - val_accuracy: 0.8147
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5101 - accuracy: 0.8187 - val_loss: 0.5109 - val_accuracy: 0.8278
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4973 - accuracy: 0.8257 - val_loss: 0.5090 - val_accuracy: 0.8247
Epoch 18/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4863 - accuracy: 0.8277 - val_loss: 0.4926 - val_accuracy: 0.8325
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4743 - accuracy: 0.8314 - val_loss: 0.4801 - val_accuracy: 0.8314
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4667 - accuracy: 0.8354 - val_loss: 0.4792 - val_accuracy: 0.8318
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4583 - accuracy: 0.8379 - val_loss: 0.4629 - val_accuracy: 0.8406
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4498 - accuracy: 0.8409 - val_loss: 0.4753 - val_accuracy: 0.8272
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4423 - accuracy: 0.8419 - val_loss: 0.4532 - val_accuracy: 0.8453
Epoch 24/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4345 - accuracy: 0.8471 - val_loss: 0.4493 - val_accuracy: 0.8381
Epoch 25/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4281 - accuracy: 0.8475 - val_loss: 0.4420 - val_accuracy: 0.8481
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4208 - accuracy: 0.8514 - val_loss: 0.4274 - val_accuracy: 0.8548
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4144 - accuracy: 0.8535 - val_loss: 0.4285 - val_accuracy: 0.8479
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4081 - accuracy: 0.8563 - val_loss: 0.4226 - val_accuracy: 0.8487
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4020 - accuracy: 0.8575 - val_loss: 0.4472 - val_accuracy: 0.8418
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3967 - accuracy: 0.8580 - val_loss: 0.4199 - val_accuracy: 0.8500
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3929 - accuracy: 0.8595 - val_loss: 0.4119 - val_accuracy: 0.8591
Epoch 32/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3872 - accuracy: 0.8627 - val_loss: 0.4069 - val_accuracy: 0.8628
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3827 - accuracy: 0.8630 - val_loss: 0.4057 - val_accuracy: 0.8550
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3769 - accuracy: 0.8665 - val_loss: 0.3950 - val_accuracy: 0.8632
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3723 - accuracy: 0.8667 - val_loss: 0.3969 - val_accuracy: 0.8585
Epoch 36/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3685 - accuracy: 0.8693 - val_loss: 0.4056 - val_accuracy: 0.8589
Epoch 37/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3640 - accuracy: 0.8703 - val_loss: 0.4065 - val_accuracy: 0.8573
Epoch 38/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3619 - accuracy: 0.8742 - val_loss: 0.3771 - val_accuracy: 0.8758
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3578 - accuracy: 0.8732 - val_loss: 0.3802 - val_accuracy: 0.8719
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3535 - accuracy: 0.8770 - val_loss: 0.4114 - val_accuracy: 0.8526
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3506 - accuracy: 0.8777 - val_loss: 0.3780 - val_accuracy: 0.8644
Epoch 42/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3466 - accuracy: 0.8804 - val_loss: 0.3880 - val_accuracy: 0.8699
Epoch 43/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3443 - accuracy: 0.8789 - val_loss: 0.3680 - val_accuracy: 0.8740
Model: "Optimizer-rmsprop"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_238 (Dense) (None, 24) 1248
dense_239 (Dense) (None, 24) 600
dense_240 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2415 - accuracy: 0.5203 - val_loss: 0.9733 - val_accuracy: 0.6511
Epoch 2/50
636/636 [==============================] - 2s 4ms/step - loss: 0.8988 - accuracy: 0.6809 - val_loss: 0.8336 - val_accuracy: 0.7056
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8137 - accuracy: 0.7036 - val_loss: 0.7698 - val_accuracy: 0.7274
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7545 - accuracy: 0.7225 - val_loss: 0.7282 - val_accuracy: 0.7284
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7046 - accuracy: 0.7379 - val_loss: 0.6735 - val_accuracy: 0.7514
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6634 - accuracy: 0.7540 - val_loss: 0.6401 - val_accuracy: 0.7612
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6316 - accuracy: 0.7652 - val_loss: 0.6124 - val_accuracy: 0.7752
Epoch 8/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6051 - accuracy: 0.7783 - val_loss: 0.5935 - val_accuracy: 0.7805
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5840 - accuracy: 0.7860 - val_loss: 0.5769 - val_accuracy: 0.7834
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5657 - accuracy: 0.7962 - val_loss: 0.5611 - val_accuracy: 0.7897
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5501 - accuracy: 0.7988 - val_loss: 0.5388 - val_accuracy: 0.8090
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5336 - accuracy: 0.8047 - val_loss: 0.5385 - val_accuracy: 0.7942
Epoch 13/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5202 - accuracy: 0.8100 - val_loss: 0.5058 - val_accuracy: 0.8251
Epoch 14/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5079 - accuracy: 0.8152 - val_loss: 0.5071 - val_accuracy: 0.8084
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4963 - accuracy: 0.8188 - val_loss: 0.4929 - val_accuracy: 0.8217
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4850 - accuracy: 0.8224 - val_loss: 0.4916 - val_accuracy: 0.8223
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4755 - accuracy: 0.8261 - val_loss: 0.4820 - val_accuracy: 0.8325
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4656 - accuracy: 0.8292 - val_loss: 0.4651 - val_accuracy: 0.8304
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4573 - accuracy: 0.8336 - val_loss: 0.4586 - val_accuracy: 0.8335
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4501 - accuracy: 0.8345 - val_loss: 0.4545 - val_accuracy: 0.8327
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4433 - accuracy: 0.8368 - val_loss: 0.4471 - val_accuracy: 0.8430
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4360 - accuracy: 0.8399 - val_loss: 0.4411 - val_accuracy: 0.8428
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4284 - accuracy: 0.8427 - val_loss: 0.4265 - val_accuracy: 0.8516
Epoch 24/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4225 - accuracy: 0.8458 - val_loss: 0.4193 - val_accuracy: 0.8487
Epoch 25/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4171 - accuracy: 0.8481 - val_loss: 0.4199 - val_accuracy: 0.8494
Epoch 26/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4111 - accuracy: 0.8500 - val_loss: 0.4185 - val_accuracy: 0.8500
Epoch 27/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4061 - accuracy: 0.8516 - val_loss: 0.4119 - val_accuracy: 0.8441
Epoch 28/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4018 - accuracy: 0.8541 - val_loss: 0.4024 - val_accuracy: 0.8544
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3967 - accuracy: 0.8558 - val_loss: 0.4232 - val_accuracy: 0.8467
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3915 - accuracy: 0.8559 - val_loss: 0.3990 - val_accuracy: 0.8546
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3885 - accuracy: 0.8591 - val_loss: 0.3939 - val_accuracy: 0.8612
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3835 - accuracy: 0.8605 - val_loss: 0.3942 - val_accuracy: 0.8544
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3805 - accuracy: 0.8614 - val_loss: 0.3972 - val_accuracy: 0.8587
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3759 - accuracy: 0.8634 - val_loss: 0.3932 - val_accuracy: 0.8579
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3702 - accuracy: 0.8658 - val_loss: 0.3913 - val_accuracy: 0.8567
Epoch 36/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3674 - accuracy: 0.8688 - val_loss: 0.3814 - val_accuracy: 0.8638
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3634 - accuracy: 0.8698 - val_loss: 0.3863 - val_accuracy: 0.8622
Epoch 38/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3604 - accuracy: 0.8728 - val_loss: 0.3683 - val_accuracy: 0.8721
Epoch 39/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3573 - accuracy: 0.8724 - val_loss: 0.3698 - val_accuracy: 0.8715
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3521 - accuracy: 0.8741 - val_loss: 0.3825 - val_accuracy: 0.8691
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3483 - accuracy: 0.8760 - val_loss: 0.3673 - val_accuracy: 0.8724
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3459 - accuracy: 0.8775 - val_loss: 0.3703 - val_accuracy: 0.8736
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3417 - accuracy: 0.8783 - val_loss: 0.3651 - val_accuracy: 0.8746
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3392 - accuracy: 0.8782 - val_loss: 0.3561 - val_accuracy: 0.8711
Epoch 45/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3358 - accuracy: 0.8805 - val_loss: 0.3666 - val_accuracy: 0.8715
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3343 - accuracy: 0.8818 - val_loss: 0.3461 - val_accuracy: 0.8838
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3309 - accuracy: 0.8836 - val_loss: 0.3419 - val_accuracy: 0.8852
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3282 - accuracy: 0.8856 - val_loss: 0.3642 - val_accuracy: 0.8750
Epoch 49/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3273 - accuracy: 0.8851 - val_loss: 0.3490 - val_accuracy: 0.8815
Epoch 50/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3229 - accuracy: 0.8883 - val_loss: 0.3401 - val_accuracy: 0.8864
Model: "Optimizer-adam"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_241 (Dense) (None, 24) 1248
dense_242 (Dense) (None, 24) 600
dense_243 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2883 - accuracy: 0.5144 - val_loss: 1.0342 - val_accuracy: 0.6046
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9284 - accuracy: 0.6533 - val_loss: 0.8543 - val_accuracy: 0.6893
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8350 - accuracy: 0.6923 - val_loss: 0.7947 - val_accuracy: 0.7072
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7781 - accuracy: 0.7141 - val_loss: 0.7480 - val_accuracy: 0.7250
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7278 - accuracy: 0.7326 - val_loss: 0.6928 - val_accuracy: 0.7563
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6791 - accuracy: 0.7542 - val_loss: 0.6568 - val_accuracy: 0.7563
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6396 - accuracy: 0.7686 - val_loss: 0.6216 - val_accuracy: 0.7756
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6075 - accuracy: 0.7816 - val_loss: 0.5969 - val_accuracy: 0.7830
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5818 - accuracy: 0.7926 - val_loss: 0.5806 - val_accuracy: 0.7873
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5601 - accuracy: 0.8004 - val_loss: 0.5626 - val_accuracy: 0.7952
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5428 - accuracy: 0.8044 - val_loss: 0.5386 - val_accuracy: 0.8096
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5257 - accuracy: 0.8121 - val_loss: 0.5279 - val_accuracy: 0.8103
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5131 - accuracy: 0.8181 - val_loss: 0.5090 - val_accuracy: 0.8249
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5004 - accuracy: 0.8217 - val_loss: 0.5122 - val_accuracy: 0.8147
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4892 - accuracy: 0.8240 - val_loss: 0.4933 - val_accuracy: 0.8231
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4776 - accuracy: 0.8275 - val_loss: 0.4911 - val_accuracy: 0.8316
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4686 - accuracy: 0.8321 - val_loss: 0.4797 - val_accuracy: 0.8353
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4598 - accuracy: 0.8344 - val_loss: 0.4649 - val_accuracy: 0.8288
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4505 - accuracy: 0.8366 - val_loss: 0.4599 - val_accuracy: 0.8398
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4444 - accuracy: 0.8394 - val_loss: 0.4575 - val_accuracy: 0.8412
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4377 - accuracy: 0.8425 - val_loss: 0.4499 - val_accuracy: 0.8455
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4309 - accuracy: 0.8453 - val_loss: 0.4590 - val_accuracy: 0.8298
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4247 - accuracy: 0.8462 - val_loss: 0.4321 - val_accuracy: 0.8459
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4183 - accuracy: 0.8511 - val_loss: 0.4267 - val_accuracy: 0.8485
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4147 - accuracy: 0.8514 - val_loss: 0.4293 - val_accuracy: 0.8538
Epoch 26/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4083 - accuracy: 0.8538 - val_loss: 0.4152 - val_accuracy: 0.8577
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4044 - accuracy: 0.8547 - val_loss: 0.4177 - val_accuracy: 0.8561
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4012 - accuracy: 0.8563 - val_loss: 0.4047 - val_accuracy: 0.8597
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3964 - accuracy: 0.8585 - val_loss: 0.4274 - val_accuracy: 0.8546
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3925 - accuracy: 0.8586 - val_loss: 0.4059 - val_accuracy: 0.8622
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3894 - accuracy: 0.8620 - val_loss: 0.4022 - val_accuracy: 0.8642
Epoch 32/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3845 - accuracy: 0.8630 - val_loss: 0.4025 - val_accuracy: 0.8585
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3810 - accuracy: 0.8673 - val_loss: 0.3971 - val_accuracy: 0.8630
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3769 - accuracy: 0.8646 - val_loss: 0.3997 - val_accuracy: 0.8607
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3730 - accuracy: 0.8693 - val_loss: 0.3962 - val_accuracy: 0.8644
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3693 - accuracy: 0.8710 - val_loss: 0.3949 - val_accuracy: 0.8646
Epoch 37/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3657 - accuracy: 0.8738 - val_loss: 0.3950 - val_accuracy: 0.8599
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3626 - accuracy: 0.8731 - val_loss: 0.3726 - val_accuracy: 0.8754
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3595 - accuracy: 0.8740 - val_loss: 0.3717 - val_accuracy: 0.8783
Epoch 40/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3551 - accuracy: 0.8767 - val_loss: 0.3926 - val_accuracy: 0.8620
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3534 - accuracy: 0.8771 - val_loss: 0.3711 - val_accuracy: 0.8732
Epoch 42/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3494 - accuracy: 0.8793 - val_loss: 0.3974 - val_accuracy: 0.8730
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3455 - accuracy: 0.8802 - val_loss: 0.3678 - val_accuracy: 0.8770
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3432 - accuracy: 0.8803 - val_loss: 0.3623 - val_accuracy: 0.8740
Model: "Optimizer-adagrad"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_244 (Dense) (None, 24) 1248
dense_245 (Dense) (None, 24) 600
dense_246 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 4ms/step - loss: 1.6071 - accuracy: 0.2388 - val_loss: 1.6049 - val_accuracy: 0.2472
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6037 - accuracy: 0.2870 - val_loss: 1.6011 - val_accuracy: 0.3255
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6002 - accuracy: 0.3393 - val_loss: 1.5974 - val_accuracy: 0.3618
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 1.5964 - accuracy: 0.3624 - val_loss: 1.5930 - val_accuracy: 0.3612
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 1.5915 - accuracy: 0.3636 - val_loss: 1.5874 - val_accuracy: 0.3638
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5859 - accuracy: 0.3705 - val_loss: 1.5812 - val_accuracy: 0.3785
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5796 - accuracy: 0.3862 - val_loss: 1.5742 - val_accuracy: 0.3968
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5725 - accuracy: 0.3975 - val_loss: 1.5663 - val_accuracy: 0.4100
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5645 - accuracy: 0.4070 - val_loss: 1.5575 - val_accuracy: 0.4208
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 1.5558 - accuracy: 0.4142 - val_loss: 1.5480 - val_accuracy: 0.4336
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 1.5464 - accuracy: 0.4229 - val_loss: 1.5378 - val_accuracy: 0.4357
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 1.5363 - accuracy: 0.4294 - val_loss: 1.5270 - val_accuracy: 0.4418
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5258 - accuracy: 0.4361 - val_loss: 1.5158 - val_accuracy: 0.4458
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 1.5150 - accuracy: 0.4401 - val_loss: 1.5044 - val_accuracy: 0.4503
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 1.5041 - accuracy: 0.4439 - val_loss: 1.4930 - val_accuracy: 0.4534
Epoch 16/50
636/636 [==============================] - 4s 6ms/step - loss: 1.4931 - accuracy: 0.4474 - val_loss: 1.4816 - val_accuracy: 0.4581
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 1.4824 - accuracy: 0.4487 - val_loss: 1.4705 - val_accuracy: 0.4605
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 1.4718 - accuracy: 0.4509 - val_loss: 1.4597 - val_accuracy: 0.4603
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 1.4616 - accuracy: 0.4511 - val_loss: 1.4493 - val_accuracy: 0.4607
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 1.4518 - accuracy: 0.4513 - val_loss: 1.4393 - val_accuracy: 0.4589
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 1.4425 - accuracy: 0.4515 - val_loss: 1.4298 - val_accuracy: 0.4597
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 1.4336 - accuracy: 0.4514 - val_loss: 1.4207 - val_accuracy: 0.4593
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 1.4250 - accuracy: 0.4519 - val_loss: 1.4120 - val_accuracy: 0.4603
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 1.4169 - accuracy: 0.4524 - val_loss: 1.4037 - val_accuracy: 0.4583
In [ ]:
accuracy_measures_optimizer = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Optimizers")
There are two possible good candidates for optimizer. The Optimizer-rmsprop and Optimizer-adam both offer similar accuracy but Optimizer-adam completed with fewer epoch thus using adam as optimizer.
In [ ]:
model_config['optimizer'] = 'adam'
Learning Rates¶
Learning rate is the rate at which the weights will change in response to the estimated error. It is the speed at which the model is expected to learn from the training data and adjust its weights. Learning rates work in conjunction with the optimizer.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
# Batch Normalization: not added
model_config['optimizer'] = 'adam'
learning_rate_list = [0.001, 0.005, 0.01, 0.1]
for learning_rate in learning_rate_list:
model_config['learning_rate'] = learning_rate
custom_layers = [
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Learning-Rate-' + str(learning_rate)
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
accuracy_measures[model_name] = history.history['accuracy']
Model: "Learning-Rate-0.001"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_247 (Dense) (None, 24) 1248
dense_248 (Dense) (None, 24) 600
dense_249 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2194 - accuracy: 0.5588 - val_loss: 0.9739 - val_accuracy: 0.6449
Epoch 2/50
636/636 [==============================] - 2s 4ms/step - loss: 0.8988 - accuracy: 0.6687 - val_loss: 0.8319 - val_accuracy: 0.6865
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7997 - accuracy: 0.7051 - val_loss: 0.7614 - val_accuracy: 0.7193
Epoch 4/50
636/636 [==============================] - 5s 8ms/step - loss: 0.7377 - accuracy: 0.7285 - val_loss: 0.7117 - val_accuracy: 0.7380
Epoch 5/50
636/636 [==============================] - 5s 8ms/step - loss: 0.6939 - accuracy: 0.7414 - val_loss: 0.6687 - val_accuracy: 0.7526
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6561 - accuracy: 0.7553 - val_loss: 0.6365 - val_accuracy: 0.7590
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6244 - accuracy: 0.7678 - val_loss: 0.6097 - val_accuracy: 0.7706
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5961 - accuracy: 0.7776 - val_loss: 0.5912 - val_accuracy: 0.7809
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5724 - accuracy: 0.7896 - val_loss: 0.5791 - val_accuracy: 0.7993
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5529 - accuracy: 0.7987 - val_loss: 0.5522 - val_accuracy: 0.8084
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5365 - accuracy: 0.8046 - val_loss: 0.5285 - val_accuracy: 0.8121
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5199 - accuracy: 0.8125 - val_loss: 0.5171 - val_accuracy: 0.8078
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5076 - accuracy: 0.8173 - val_loss: 0.5052 - val_accuracy: 0.8208
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4965 - accuracy: 0.8201 - val_loss: 0.4969 - val_accuracy: 0.8135
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4862 - accuracy: 0.8223 - val_loss: 0.4891 - val_accuracy: 0.8241
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4750 - accuracy: 0.8264 - val_loss: 0.4930 - val_accuracy: 0.8223
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4677 - accuracy: 0.8334 - val_loss: 0.4963 - val_accuracy: 0.8186
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4595 - accuracy: 0.8351 - val_loss: 0.4622 - val_accuracy: 0.8339
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4520 - accuracy: 0.8377 - val_loss: 0.4627 - val_accuracy: 0.8381
Epoch 20/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4456 - accuracy: 0.8422 - val_loss: 0.4557 - val_accuracy: 0.8381
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4403 - accuracy: 0.8422 - val_loss: 0.4434 - val_accuracy: 0.8426
Epoch 22/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4335 - accuracy: 0.8455 - val_loss: 0.4497 - val_accuracy: 0.8361
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4281 - accuracy: 0.8452 - val_loss: 0.4371 - val_accuracy: 0.8443
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4214 - accuracy: 0.8495 - val_loss: 0.4249 - val_accuracy: 0.8485
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4178 - accuracy: 0.8522 - val_loss: 0.4255 - val_accuracy: 0.8491
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4122 - accuracy: 0.8520 - val_loss: 0.4173 - val_accuracy: 0.8544
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4077 - accuracy: 0.8561 - val_loss: 0.4097 - val_accuracy: 0.8569
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4021 - accuracy: 0.8570 - val_loss: 0.4117 - val_accuracy: 0.8504
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3969 - accuracy: 0.8602 - val_loss: 0.4289 - val_accuracy: 0.8485
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3932 - accuracy: 0.8577 - val_loss: 0.4061 - val_accuracy: 0.8561
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3901 - accuracy: 0.8612 - val_loss: 0.3956 - val_accuracy: 0.8679
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3849 - accuracy: 0.8634 - val_loss: 0.3898 - val_accuracy: 0.8612
Epoch 33/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3813 - accuracy: 0.8629 - val_loss: 0.4038 - val_accuracy: 0.8526
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3767 - accuracy: 0.8652 - val_loss: 0.3934 - val_accuracy: 0.8548
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3722 - accuracy: 0.8682 - val_loss: 0.3958 - val_accuracy: 0.8567
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3697 - accuracy: 0.8698 - val_loss: 0.3903 - val_accuracy: 0.8603
Model: "Learning-Rate-0.005"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_250 (Dense) (None, 24) 1248
dense_251 (Dense) (None, 24) 600
dense_252 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9594 - accuracy: 0.6310 - val_loss: 0.7688 - val_accuracy: 0.7107
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7055 - accuracy: 0.7366 - val_loss: 0.6315 - val_accuracy: 0.7640
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6065 - accuracy: 0.7715 - val_loss: 0.5688 - val_accuracy: 0.8011
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5439 - accuracy: 0.7981 - val_loss: 0.5138 - val_accuracy: 0.8090
Epoch 5/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5104 - accuracy: 0.8099 - val_loss: 0.4861 - val_accuracy: 0.8172
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4761 - accuracy: 0.8257 - val_loss: 0.4623 - val_accuracy: 0.8428
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4482 - accuracy: 0.8390 - val_loss: 0.4463 - val_accuracy: 0.8408
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4280 - accuracy: 0.8443 - val_loss: 0.4127 - val_accuracy: 0.8553
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4133 - accuracy: 0.8511 - val_loss: 0.5588 - val_accuracy: 0.8121
Epoch 10/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3980 - accuracy: 0.8581 - val_loss: 0.4450 - val_accuracy: 0.8420
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3869 - accuracy: 0.8616 - val_loss: 0.3763 - val_accuracy: 0.8640
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3786 - accuracy: 0.8659 - val_loss: 0.3632 - val_accuracy: 0.8738
Epoch 13/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3691 - accuracy: 0.8683 - val_loss: 0.3816 - val_accuracy: 0.8691
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3608 - accuracy: 0.8704 - val_loss: 0.3737 - val_accuracy: 0.8726
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3520 - accuracy: 0.8750 - val_loss: 0.3815 - val_accuracy: 0.8703
Epoch 16/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3451 - accuracy: 0.8781 - val_loss: 0.3604 - val_accuracy: 0.8715
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3371 - accuracy: 0.8803 - val_loss: 0.3857 - val_accuracy: 0.8691
Model: "Learning-Rate-0.01"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_253 (Dense) (None, 24) 1248
dense_254 (Dense) (None, 24) 600
dense_255 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9885 - accuracy: 0.6218 - val_loss: 0.7768 - val_accuracy: 0.7166
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7121 - accuracy: 0.7363 - val_loss: 0.6421 - val_accuracy: 0.7551
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6189 - accuracy: 0.7656 - val_loss: 0.6004 - val_accuracy: 0.7704
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5644 - accuracy: 0.7923 - val_loss: 0.5139 - val_accuracy: 0.8113
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5380 - accuracy: 0.7994 - val_loss: 0.5028 - val_accuracy: 0.8237
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5146 - accuracy: 0.8119 - val_loss: 0.4859 - val_accuracy: 0.8188
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4975 - accuracy: 0.8204 - val_loss: 0.4540 - val_accuracy: 0.8436
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4786 - accuracy: 0.8272 - val_loss: 0.4704 - val_accuracy: 0.8318
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4675 - accuracy: 0.8336 - val_loss: 0.5144 - val_accuracy: 0.8235
Epoch 10/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4575 - accuracy: 0.8376 - val_loss: 0.5206 - val_accuracy: 0.8206
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4513 - accuracy: 0.8392 - val_loss: 0.4107 - val_accuracy: 0.8591
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4457 - accuracy: 0.8407 - val_loss: 0.4029 - val_accuracy: 0.8585
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4401 - accuracy: 0.8427 - val_loss: 0.4416 - val_accuracy: 0.8591
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4345 - accuracy: 0.8469 - val_loss: 0.4146 - val_accuracy: 0.8589
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4300 - accuracy: 0.8481 - val_loss: 0.4624 - val_accuracy: 0.8396
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4282 - accuracy: 0.8485 - val_loss: 0.4253 - val_accuracy: 0.8493
Model: "Learning-Rate-0.1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_256 (Dense) (None, 24) 1248
dense_257 (Dense) (None, 24) 600
dense_258 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.6557 - accuracy: 0.2059 - val_loss: 1.6092 - val_accuracy: 0.2140
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 1.6154 - accuracy: 0.2099 - val_loss: 1.6083 - val_accuracy: 0.2166
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6163 - accuracy: 0.2034 - val_loss: 1.6098 - val_accuracy: 0.2166
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6158 - accuracy: 0.2100 - val_loss: 1.6131 - val_accuracy: 0.2140
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6173 - accuracy: 0.2018 - val_loss: 1.6110 - val_accuracy: 0.2166
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6167 - accuracy: 0.2114 - val_loss: 1.6085 - val_accuracy: 0.2166
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 1.6162 - accuracy: 0.2096 - val_loss: 1.6059 - val_accuracy: 0.2134
In [ ]:
accuracy_measures_learning_rate = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Learning Rates")
The learning rate 0.01 and 0.005 offers a higher accuracy but it is unstable compared to 0.001. Thus, selecting the model with a learning rate of 0.001
In [ ]:
model_config['learning_rate'] = 0.001
Overfitting Management¶
Regularization¶
Regularization is an important technique for managing overfitting in neural networks. Regularization algorithms provide an adjustments to the model parameters after they are updated. The adjustment reduces the variance in the model by providing a penalty when overfitting.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
# Batch Normalization: not added
model_config['optimizer'] = 'adam'
model_config['learning_rate'] = 0.001
regularizer_list = ['l1','l2','l1_l2', 'None']
for regularizer in regularizer_list:
if regularizer == 'None':
regularizer = None
model_config['regularizer'] = regularizer
print('Testing: {}'.format(regularizer))
custom_layers = [
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
];
model_name = 'Regularizer-' + str(regularizer)
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
# Used the validation accuracy
accuracy_measures[model_name] = history.history['val_accuracy']
Testing: l1
Model: "Regularizer-l1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_259 (Dense) (None, 24) 1248
dense_260 (Dense) (None, 24) 600
dense_261 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.6437 - accuracy: 0.2150 - val_loss: 1.6144 - val_accuracy: 0.2140
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6160 - accuracy: 0.2159 - val_loss: 1.6149 - val_accuracy: 0.2134
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6160 - accuracy: 0.2137 - val_loss: 1.6149 - val_accuracy: 0.2140
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6160 - accuracy: 0.2140 - val_loss: 1.6148 - val_accuracy: 0.2140
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 1.6160 - accuracy: 0.2164 - val_loss: 1.6146 - val_accuracy: 0.2140
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 1.6160 - accuracy: 0.2156 - val_loss: 1.6148 - val_accuracy: 0.2140
Testing: l2
Model: "Regularizer-l2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_262 (Dense) (None, 24) 1248
dense_263 (Dense) (None, 24) 600
dense_264 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.4273 - accuracy: 0.4420 - val_loss: 1.2950 - val_accuracy: 0.5462
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2619 - accuracy: 0.5565 - val_loss: 1.2170 - val_accuracy: 0.6364
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 1.2001 - accuracy: 0.6094 - val_loss: 1.1524 - val_accuracy: 0.6291
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 1.1393 - accuracy: 0.6491 - val_loss: 1.1063 - val_accuracy: 0.6519
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1057 - accuracy: 0.6587 - val_loss: 1.0766 - val_accuracy: 0.6643
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0824 - accuracy: 0.6657 - val_loss: 1.0589 - val_accuracy: 0.6680
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0650 - accuracy: 0.6664 - val_loss: 1.0368 - val_accuracy: 0.6712
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0505 - accuracy: 0.6704 - val_loss: 1.0271 - val_accuracy: 0.6940
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 1.0358 - accuracy: 0.6766 - val_loss: 1.0326 - val_accuracy: 0.6920
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 1.0223 - accuracy: 0.6811 - val_loss: 1.0013 - val_accuracy: 0.6983
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0114 - accuracy: 0.6868 - val_loss: 0.9878 - val_accuracy: 0.6946
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9987 - accuracy: 0.6908 - val_loss: 0.9886 - val_accuracy: 0.6694
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9900 - accuracy: 0.6921 - val_loss: 0.9700 - val_accuracy: 0.7058
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9808 - accuracy: 0.6972 - val_loss: 0.9638 - val_accuracy: 0.7017
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9713 - accuracy: 0.6987 - val_loss: 0.9432 - val_accuracy: 0.7170
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9629 - accuracy: 0.7028 - val_loss: 0.9365 - val_accuracy: 0.7201
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9549 - accuracy: 0.7057 - val_loss: 0.9369 - val_accuracy: 0.7201
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9471 - accuracy: 0.7117 - val_loss: 0.9384 - val_accuracy: 0.7022
Epoch 19/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9376 - accuracy: 0.7144 - val_loss: 0.9252 - val_accuracy: 0.7075
Epoch 20/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9310 - accuracy: 0.7210 - val_loss: 0.9140 - val_accuracy: 0.7292
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9222 - accuracy: 0.7263 - val_loss: 0.9031 - val_accuracy: 0.7252
Epoch 22/50
636/636 [==============================] - 2s 4ms/step - loss: 0.9124 - accuracy: 0.7296 - val_loss: 0.8925 - val_accuracy: 0.7380
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9042 - accuracy: 0.7329 - val_loss: 0.8984 - val_accuracy: 0.7431
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8955 - accuracy: 0.7368 - val_loss: 0.8834 - val_accuracy: 0.7453
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8877 - accuracy: 0.7432 - val_loss: 0.8695 - val_accuracy: 0.7437
Epoch 26/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8816 - accuracy: 0.7435 - val_loss: 0.8660 - val_accuracy: 0.7508
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8750 - accuracy: 0.7470 - val_loss: 0.8562 - val_accuracy: 0.7465
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8690 - accuracy: 0.7465 - val_loss: 0.8812 - val_accuracy: 0.7386
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8632 - accuracy: 0.7513 - val_loss: 0.8669 - val_accuracy: 0.7508
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8567 - accuracy: 0.7497 - val_loss: 0.8503 - val_accuracy: 0.7524
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8522 - accuracy: 0.7548 - val_loss: 0.8296 - val_accuracy: 0.7636
Epoch 32/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8458 - accuracy: 0.7555 - val_loss: 0.8290 - val_accuracy: 0.7616
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8388 - accuracy: 0.7591 - val_loss: 0.8250 - val_accuracy: 0.7618
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8323 - accuracy: 0.7593 - val_loss: 0.8133 - val_accuracy: 0.7716
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8283 - accuracy: 0.7618 - val_loss: 0.8180 - val_accuracy: 0.7667
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8216 - accuracy: 0.7648 - val_loss: 0.8225 - val_accuracy: 0.7555
Epoch 37/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8167 - accuracy: 0.7676 - val_loss: 0.8452 - val_accuracy: 0.7484
Epoch 38/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8122 - accuracy: 0.7670 - val_loss: 0.7910 - val_accuracy: 0.7736
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8056 - accuracy: 0.7711 - val_loss: 0.7825 - val_accuracy: 0.7759
Epoch 40/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7975 - accuracy: 0.7744 - val_loss: 0.8334 - val_accuracy: 0.7506
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7917 - accuracy: 0.7768 - val_loss: 0.8095 - val_accuracy: 0.7622
Epoch 42/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7855 - accuracy: 0.7769 - val_loss: 0.7866 - val_accuracy: 0.7732
Epoch 43/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7810 - accuracy: 0.7800 - val_loss: 0.7636 - val_accuracy: 0.7842
Epoch 44/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7772 - accuracy: 0.7801 - val_loss: 0.7678 - val_accuracy: 0.7881
Epoch 45/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7709 - accuracy: 0.7822 - val_loss: 0.7749 - val_accuracy: 0.7801
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7677 - accuracy: 0.7836 - val_loss: 0.7535 - val_accuracy: 0.7868
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7627 - accuracy: 0.7853 - val_loss: 0.7496 - val_accuracy: 0.7966
Epoch 48/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7583 - accuracy: 0.7868 - val_loss: 0.7621 - val_accuracy: 0.7913
Epoch 49/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7552 - accuracy: 0.7870 - val_loss: 0.7618 - val_accuracy: 0.7985
Epoch 50/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7520 - accuracy: 0.7905 - val_loss: 0.7696 - val_accuracy: 0.7901
Testing: l1_l2
Model: "Regularizer-l1_l2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_265 (Dense) (None, 24) 1248
dense_266 (Dense) (None, 24) 600
dense_267 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 4ms/step - loss: 1.6468 - accuracy: 0.2143 - val_loss: 1.6146 - val_accuracy: 0.2140
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6162 - accuracy: 0.2153 - val_loss: 1.6148 - val_accuracy: 0.2134
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 1.6162 - accuracy: 0.2141 - val_loss: 1.6149 - val_accuracy: 0.2140
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 1.6162 - accuracy: 0.2149 - val_loss: 1.6148 - val_accuracy: 0.2140
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 1.6162 - accuracy: 0.2164 - val_loss: 1.6147 - val_accuracy: 0.2140
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 1.6162 - accuracy: 0.2153 - val_loss: 1.6148 - val_accuracy: 0.2140
Testing: None
Model: "Regularizer-None"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_268 (Dense) (None, 24) 1248
dense_269 (Dense) (None, 24) 600
dense_270 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.3225 - accuracy: 0.4952 - val_loss: 1.1006 - val_accuracy: 0.5951
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9706 - accuracy: 0.6601 - val_loss: 0.8569 - val_accuracy: 0.6991
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8194 - accuracy: 0.7036 - val_loss: 0.7788 - val_accuracy: 0.7144
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7575 - accuracy: 0.7245 - val_loss: 0.7384 - val_accuracy: 0.7305
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7165 - accuracy: 0.7327 - val_loss: 0.6933 - val_accuracy: 0.7465
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6789 - accuracy: 0.7504 - val_loss: 0.6583 - val_accuracy: 0.7640
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6438 - accuracy: 0.7671 - val_loss: 0.6243 - val_accuracy: 0.7809
Epoch 8/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6125 - accuracy: 0.7812 - val_loss: 0.6030 - val_accuracy: 0.7928
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5857 - accuracy: 0.7908 - val_loss: 0.5881 - val_accuracy: 0.7877
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5648 - accuracy: 0.7966 - val_loss: 0.5784 - val_accuracy: 0.7958
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5471 - accuracy: 0.8034 - val_loss: 0.5468 - val_accuracy: 0.8023
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5303 - accuracy: 0.8097 - val_loss: 0.5390 - val_accuracy: 0.8031
Epoch 13/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5166 - accuracy: 0.8146 - val_loss: 0.5170 - val_accuracy: 0.8206
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5038 - accuracy: 0.8199 - val_loss: 0.5102 - val_accuracy: 0.8176
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4919 - accuracy: 0.8247 - val_loss: 0.4920 - val_accuracy: 0.8255
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4806 - accuracy: 0.8275 - val_loss: 0.5007 - val_accuracy: 0.8325
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4714 - accuracy: 0.8335 - val_loss: 0.5006 - val_accuracy: 0.8247
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4609 - accuracy: 0.8385 - val_loss: 0.4641 - val_accuracy: 0.8353
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4502 - accuracy: 0.8412 - val_loss: 0.4575 - val_accuracy: 0.8447
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4419 - accuracy: 0.8446 - val_loss: 0.4581 - val_accuracy: 0.8449
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4342 - accuracy: 0.8477 - val_loss: 0.4365 - val_accuracy: 0.8493
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4241 - accuracy: 0.8486 - val_loss: 0.4380 - val_accuracy: 0.8449
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4157 - accuracy: 0.8542 - val_loss: 0.4257 - val_accuracy: 0.8528
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4088 - accuracy: 0.8572 - val_loss: 0.4193 - val_accuracy: 0.8579
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4031 - accuracy: 0.8595 - val_loss: 0.4119 - val_accuracy: 0.8585
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3962 - accuracy: 0.8599 - val_loss: 0.4091 - val_accuracy: 0.8618
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3901 - accuracy: 0.8651 - val_loss: 0.4010 - val_accuracy: 0.8642
Epoch 28/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3859 - accuracy: 0.8673 - val_loss: 0.4031 - val_accuracy: 0.8587
Epoch 29/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3801 - accuracy: 0.8687 - val_loss: 0.4438 - val_accuracy: 0.8457
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3753 - accuracy: 0.8705 - val_loss: 0.4045 - val_accuracy: 0.8599
Epoch 31/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3716 - accuracy: 0.8725 - val_loss: 0.3880 - val_accuracy: 0.8713
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3671 - accuracy: 0.8741 - val_loss: 0.3982 - val_accuracy: 0.8691
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3629 - accuracy: 0.8763 - val_loss: 0.3803 - val_accuracy: 0.8683
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3580 - accuracy: 0.8775 - val_loss: 0.3807 - val_accuracy: 0.8693
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3527 - accuracy: 0.8795 - val_loss: 0.3815 - val_accuracy: 0.8697
Epoch 36/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3507 - accuracy: 0.8798 - val_loss: 0.3733 - val_accuracy: 0.8677
In [ ]:
accuracy_measures_regularizer = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Regularizers")
Similar to Batch Normalization, adding regularization does not improve performance and instead produce unstability to the model. Hence, not applying any regularizer to the model is the better option.
In [ ]:
model_config['regularizer'] = None
Dropout¶
Dropout works during forward propagation. By default, during forward propagation, the output of each node in the layer is sent every node in the next layer. When using dropout, the outputs of some of the nodes in the layer are dropped randomly.
In [ ]:
accuracy_measures = {}
model_config = get_base_model_config()
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 2
# Number of nodes per each layer: 24-24
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
# Batch Normalization: not added
model_config['optimizer'] = 'adam'
model_config['learning_rate'] = 0.001
model_config['regularizer'] = None
model_config['dropout_rate'] = None
dropout_list = [0.0, 0.001, 0.1, 0.2, 0.5]
for dropout in dropout_list:
model_config['dropout_rate'] = dropout
custom_layers = []
if model_config['dropout_rate'] > 0.0:
model_name = 'Dropout' + str(dropout)
custom_layers = [
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dropout(model_config['dropout_rate']),
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dropout(model_config['dropout_rate']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
]
else:
model_name = 'Dropout-None'
custom_layers = [
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
]
model_config['model_name'] = model_name
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
history = create_and_run_model(model_config, X_train, y_train_dummy)
plot_learning_curves(history, model_name)
# Used the validation accuracy
accuracy_measures[model_name] = history.history['val_accuracy']
Model: "Dropout-None"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_271 (Dense) (None, 24) 1248
dense_272 (Dense) (None, 24) 600
dense_273 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 3s 4ms/step - loss: 1.2333 - accuracy: 0.5797 - val_loss: 0.9494 - val_accuracy: 0.6771
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8893 - accuracy: 0.6879 - val_loss: 0.8212 - val_accuracy: 0.7062
Epoch 3/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7964 - accuracy: 0.7118 - val_loss: 0.7532 - val_accuracy: 0.7296
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7320 - accuracy: 0.7352 - val_loss: 0.7106 - val_accuracy: 0.7494
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6877 - accuracy: 0.7479 - val_loss: 0.6624 - val_accuracy: 0.7618
Epoch 6/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6511 - accuracy: 0.7633 - val_loss: 0.6289 - val_accuracy: 0.7756
Epoch 7/50
636/636 [==============================] - 2s 4ms/step - loss: 0.6191 - accuracy: 0.7735 - val_loss: 0.5992 - val_accuracy: 0.7818
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5895 - accuracy: 0.7858 - val_loss: 0.5761 - val_accuracy: 0.7868
Epoch 9/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5633 - accuracy: 0.7948 - val_loss: 0.5598 - val_accuracy: 0.8019
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5409 - accuracy: 0.8035 - val_loss: 0.5446 - val_accuracy: 0.8084
Epoch 11/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5217 - accuracy: 0.8118 - val_loss: 0.5114 - val_accuracy: 0.8162
Epoch 12/50
636/636 [==============================] - 2s 4ms/step - loss: 0.5048 - accuracy: 0.8193 - val_loss: 0.5003 - val_accuracy: 0.8194
Epoch 13/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4920 - accuracy: 0.8249 - val_loss: 0.4839 - val_accuracy: 0.8237
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4792 - accuracy: 0.8312 - val_loss: 0.4813 - val_accuracy: 0.8243
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4680 - accuracy: 0.8341 - val_loss: 0.4610 - val_accuracy: 0.8343
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4568 - accuracy: 0.8376 - val_loss: 0.4671 - val_accuracy: 0.8377
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4481 - accuracy: 0.8421 - val_loss: 0.4605 - val_accuracy: 0.8400
Epoch 18/50
636/636 [==============================] - 2s 4ms/step - loss: 0.4386 - accuracy: 0.8452 - val_loss: 0.4405 - val_accuracy: 0.8382
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4296 - accuracy: 0.8471 - val_loss: 0.4324 - val_accuracy: 0.8506
Epoch 20/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4205 - accuracy: 0.8524 - val_loss: 0.4365 - val_accuracy: 0.8496
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4140 - accuracy: 0.8533 - val_loss: 0.4155 - val_accuracy: 0.8567
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4061 - accuracy: 0.8565 - val_loss: 0.4109 - val_accuracy: 0.8612
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3982 - accuracy: 0.8594 - val_loss: 0.4064 - val_accuracy: 0.8597
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3913 - accuracy: 0.8630 - val_loss: 0.3941 - val_accuracy: 0.8620
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3859 - accuracy: 0.8643 - val_loss: 0.3919 - val_accuracy: 0.8589
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3796 - accuracy: 0.8645 - val_loss: 0.3914 - val_accuracy: 0.8685
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3734 - accuracy: 0.8710 - val_loss: 0.3818 - val_accuracy: 0.8644
Epoch 28/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3689 - accuracy: 0.8703 - val_loss: 0.3719 - val_accuracy: 0.8715
Epoch 29/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3634 - accuracy: 0.8726 - val_loss: 0.4099 - val_accuracy: 0.8577
Epoch 30/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3582 - accuracy: 0.8745 - val_loss: 0.3632 - val_accuracy: 0.8734
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3558 - accuracy: 0.8747 - val_loss: 0.3633 - val_accuracy: 0.8738
Epoch 32/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3500 - accuracy: 0.8793 - val_loss: 0.3678 - val_accuracy: 0.8719
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3467 - accuracy: 0.8779 - val_loss: 0.3677 - val_accuracy: 0.8726
Epoch 34/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3406 - accuracy: 0.8826 - val_loss: 0.3586 - val_accuracy: 0.8768
Epoch 35/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3370 - accuracy: 0.8843 - val_loss: 0.3578 - val_accuracy: 0.8781
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3338 - accuracy: 0.8846 - val_loss: 0.3583 - val_accuracy: 0.8742
Epoch 37/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3288 - accuracy: 0.8848 - val_loss: 0.3775 - val_accuracy: 0.8650
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3270 - accuracy: 0.8883 - val_loss: 0.3373 - val_accuracy: 0.8862
Epoch 39/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3229 - accuracy: 0.8880 - val_loss: 0.3331 - val_accuracy: 0.8846
Epoch 40/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3191 - accuracy: 0.8891 - val_loss: 0.3708 - val_accuracy: 0.8734
Epoch 41/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3166 - accuracy: 0.8901 - val_loss: 0.3375 - val_accuracy: 0.8838
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3132 - accuracy: 0.8916 - val_loss: 0.3508 - val_accuracy: 0.8807
Epoch 43/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3084 - accuracy: 0.8948 - val_loss: 0.3328 - val_accuracy: 0.8882
Epoch 44/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3070 - accuracy: 0.8942 - val_loss: 0.3415 - val_accuracy: 0.8813
Epoch 45/50
636/636 [==============================] - 2s 4ms/step - loss: 0.3028 - accuracy: 0.8964 - val_loss: 0.3399 - val_accuracy: 0.8838
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3014 - accuracy: 0.8965 - val_loss: 0.3137 - val_accuracy: 0.8949
Epoch 47/50
636/636 [==============================] - 2s 4ms/step - loss: 0.2991 - accuracy: 0.8966 - val_loss: 0.3141 - val_accuracy: 0.8950
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.2956 - accuracy: 0.8980 - val_loss: 0.3380 - val_accuracy: 0.8817
Epoch 49/50
636/636 [==============================] - 4s 6ms/step - loss: 0.2949 - accuracy: 0.8993 - val_loss: 0.3255 - val_accuracy: 0.8901
Epoch 50/50
636/636 [==============================] - 3s 5ms/step - loss: 0.2910 - accuracy: 0.8998 - val_loss: 0.3401 - val_accuracy: 0.8809
Model: "Dropout0.001"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_274 (Dense) (None, 24) 1248
dropout (Dropout) (None, 24) 0
dense_275 (Dense) (None, 24) 600
dropout_1 (Dropout) (None, 24) 0
dense_276 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 4ms/step - loss: 1.2087 - accuracy: 0.5639 - val_loss: 0.9463 - val_accuracy: 0.6631
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8735 - accuracy: 0.6882 - val_loss: 0.8100 - val_accuracy: 0.7068
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7934 - accuracy: 0.7067 - val_loss: 0.7590 - val_accuracy: 0.7186
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7440 - accuracy: 0.7243 - val_loss: 0.7199 - val_accuracy: 0.7321
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7084 - accuracy: 0.7340 - val_loss: 0.6805 - val_accuracy: 0.7451
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6739 - accuracy: 0.7459 - val_loss: 0.6509 - val_accuracy: 0.7549
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6437 - accuracy: 0.7567 - val_loss: 0.6208 - val_accuracy: 0.7730
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6133 - accuracy: 0.7703 - val_loss: 0.6046 - val_accuracy: 0.7797
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5895 - accuracy: 0.7794 - val_loss: 0.5922 - val_accuracy: 0.7883
Epoch 10/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5697 - accuracy: 0.7880 - val_loss: 0.5633 - val_accuracy: 0.7962
Epoch 11/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5520 - accuracy: 0.7962 - val_loss: 0.5344 - val_accuracy: 0.8115
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5346 - accuracy: 0.8048 - val_loss: 0.5401 - val_accuracy: 0.7885
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5238 - accuracy: 0.8084 - val_loss: 0.5098 - val_accuracy: 0.8143
Epoch 14/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5115 - accuracy: 0.8141 - val_loss: 0.5110 - val_accuracy: 0.8056
Epoch 15/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4984 - accuracy: 0.8179 - val_loss: 0.5001 - val_accuracy: 0.8202
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4875 - accuracy: 0.8199 - val_loss: 0.4945 - val_accuracy: 0.8225
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4780 - accuracy: 0.8266 - val_loss: 0.4889 - val_accuracy: 0.8296
Epoch 18/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4688 - accuracy: 0.8281 - val_loss: 0.4692 - val_accuracy: 0.8294
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4602 - accuracy: 0.8317 - val_loss: 0.4642 - val_accuracy: 0.8351
Epoch 20/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4529 - accuracy: 0.8358 - val_loss: 0.4739 - val_accuracy: 0.8296
Epoch 21/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4473 - accuracy: 0.8391 - val_loss: 0.4504 - val_accuracy: 0.8345
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4399 - accuracy: 0.8398 - val_loss: 0.4574 - val_accuracy: 0.8296
Epoch 23/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4332 - accuracy: 0.8432 - val_loss: 0.4432 - val_accuracy: 0.8343
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4271 - accuracy: 0.8474 - val_loss: 0.4320 - val_accuracy: 0.8349
Model: "Dropout0.1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_277 (Dense) (None, 24) 1248
dropout_2 (Dropout) (None, 24) 0
dense_278 (Dense) (None, 24) 600
dropout_3 (Dropout) (None, 24) 0
dense_279 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 6ms/step - loss: 1.2816 - accuracy: 0.5011 - val_loss: 0.9922 - val_accuracy: 0.6688
Epoch 2/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9538 - accuracy: 0.6537 - val_loss: 0.8364 - val_accuracy: 0.7054
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8618 - accuracy: 0.6869 - val_loss: 0.7656 - val_accuracy: 0.7227
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8009 - accuracy: 0.7084 - val_loss: 0.7117 - val_accuracy: 0.7457
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7562 - accuracy: 0.7234 - val_loss: 0.6681 - val_accuracy: 0.7608
Epoch 6/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7274 - accuracy: 0.7371 - val_loss: 0.6445 - val_accuracy: 0.7642
Epoch 7/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6918 - accuracy: 0.7519 - val_loss: 0.6151 - val_accuracy: 0.7832
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6728 - accuracy: 0.7582 - val_loss: 0.5887 - val_accuracy: 0.7883
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6431 - accuracy: 0.7697 - val_loss: 0.5702 - val_accuracy: 0.8001
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6193 - accuracy: 0.7807 - val_loss: 0.5550 - val_accuracy: 0.8072
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6068 - accuracy: 0.7825 - val_loss: 0.5273 - val_accuracy: 0.8233
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5941 - accuracy: 0.7910 - val_loss: 0.5180 - val_accuracy: 0.8237
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5776 - accuracy: 0.7972 - val_loss: 0.5005 - val_accuracy: 0.8249
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5622 - accuracy: 0.8015 - val_loss: 0.4928 - val_accuracy: 0.8263
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5603 - accuracy: 0.8045 - val_loss: 0.4796 - val_accuracy: 0.8367
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5394 - accuracy: 0.8109 - val_loss: 0.4762 - val_accuracy: 0.8416
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5407 - accuracy: 0.8123 - val_loss: 0.4762 - val_accuracy: 0.8479
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5286 - accuracy: 0.8153 - val_loss: 0.4681 - val_accuracy: 0.8422
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5197 - accuracy: 0.8180 - val_loss: 0.4558 - val_accuracy: 0.8491
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5161 - accuracy: 0.8185 - val_loss: 0.4501 - val_accuracy: 0.8494
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5158 - accuracy: 0.8203 - val_loss: 0.4363 - val_accuracy: 0.8542
Epoch 22/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5022 - accuracy: 0.8215 - val_loss: 0.4383 - val_accuracy: 0.8520
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5012 - accuracy: 0.8241 - val_loss: 0.4267 - val_accuracy: 0.8520
Epoch 24/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4963 - accuracy: 0.8261 - val_loss: 0.4318 - val_accuracy: 0.8546
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4840 - accuracy: 0.8289 - val_loss: 0.4226 - val_accuracy: 0.8540
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4862 - accuracy: 0.8280 - val_loss: 0.4316 - val_accuracy: 0.8593
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4829 - accuracy: 0.8298 - val_loss: 0.4161 - val_accuracy: 0.8595
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4762 - accuracy: 0.8348 - val_loss: 0.4124 - val_accuracy: 0.8553
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4718 - accuracy: 0.8341 - val_loss: 0.4164 - val_accuracy: 0.8605
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4723 - accuracy: 0.8347 - val_loss: 0.4018 - val_accuracy: 0.8614
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4661 - accuracy: 0.8348 - val_loss: 0.4043 - val_accuracy: 0.8599
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4592 - accuracy: 0.8361 - val_loss: 0.3995 - val_accuracy: 0.8632
Epoch 33/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4635 - accuracy: 0.8378 - val_loss: 0.3954 - val_accuracy: 0.8595
Epoch 34/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4570 - accuracy: 0.8391 - val_loss: 0.3957 - val_accuracy: 0.8599
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4611 - accuracy: 0.8357 - val_loss: 0.4059 - val_accuracy: 0.8557
Epoch 36/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4576 - accuracy: 0.8397 - val_loss: 0.3890 - val_accuracy: 0.8648
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4510 - accuracy: 0.8435 - val_loss: 0.3958 - val_accuracy: 0.8573
Epoch 38/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4463 - accuracy: 0.8431 - val_loss: 0.3937 - val_accuracy: 0.8607
Epoch 39/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4466 - accuracy: 0.8442 - val_loss: 0.3887 - val_accuracy: 0.8671
Epoch 40/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4396 - accuracy: 0.8451 - val_loss: 0.4049 - val_accuracy: 0.8603
Epoch 41/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4418 - accuracy: 0.8468 - val_loss: 0.3873 - val_accuracy: 0.8601
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4354 - accuracy: 0.8477 - val_loss: 0.4049 - val_accuracy: 0.8597
Epoch 43/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4398 - accuracy: 0.8470 - val_loss: 0.3825 - val_accuracy: 0.8636
Epoch 44/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4337 - accuracy: 0.8503 - val_loss: 0.3716 - val_accuracy: 0.8715
Epoch 45/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4284 - accuracy: 0.8547 - val_loss: 0.3774 - val_accuracy: 0.8756
Epoch 46/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4331 - accuracy: 0.8514 - val_loss: 0.3761 - val_accuracy: 0.8650
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4285 - accuracy: 0.8498 - val_loss: 0.3724 - val_accuracy: 0.8776
Epoch 48/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4270 - accuracy: 0.8524 - val_loss: 0.3873 - val_accuracy: 0.8648
Epoch 49/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4249 - accuracy: 0.8529 - val_loss: 0.3739 - val_accuracy: 0.8728
Epoch 50/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4252 - accuracy: 0.8538 - val_loss: 0.3660 - val_accuracy: 0.8785
Model: "Dropout0.2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_280 (Dense) (None, 24) 1248
dropout_4 (Dropout) (None, 24) 0
dense_281 (Dense) (None, 24) 600
dropout_5 (Dropout) (None, 24) 0
dense_282 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 1.2917 - accuracy: 0.5118 - val_loss: 0.9977 - val_accuracy: 0.6509
Epoch 2/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9873 - accuracy: 0.6378 - val_loss: 0.8371 - val_accuracy: 0.7038
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8832 - accuracy: 0.6844 - val_loss: 0.7574 - val_accuracy: 0.7321
Epoch 4/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8184 - accuracy: 0.7109 - val_loss: 0.7059 - val_accuracy: 0.7535
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.7762 - accuracy: 0.7246 - val_loss: 0.6658 - val_accuracy: 0.7718
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7410 - accuracy: 0.7380 - val_loss: 0.6333 - val_accuracy: 0.7752
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.7101 - accuracy: 0.7513 - val_loss: 0.6073 - val_accuracy: 0.7875
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6795 - accuracy: 0.7599 - val_loss: 0.5769 - val_accuracy: 0.7934
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6544 - accuracy: 0.7704 - val_loss: 0.5497 - val_accuracy: 0.8084
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6357 - accuracy: 0.7752 - val_loss: 0.5459 - val_accuracy: 0.8076
Epoch 11/50
636/636 [==============================] - 4s 6ms/step - loss: 0.6154 - accuracy: 0.7816 - val_loss: 0.5145 - val_accuracy: 0.8249
Epoch 12/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6161 - accuracy: 0.7841 - val_loss: 0.5114 - val_accuracy: 0.8166
Epoch 13/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6020 - accuracy: 0.7881 - val_loss: 0.4952 - val_accuracy: 0.8324
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5866 - accuracy: 0.7955 - val_loss: 0.5046 - val_accuracy: 0.8162
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5827 - accuracy: 0.7963 - val_loss: 0.4810 - val_accuracy: 0.8290
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5733 - accuracy: 0.8003 - val_loss: 0.4779 - val_accuracy: 0.8363
Epoch 17/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5707 - accuracy: 0.8028 - val_loss: 0.4742 - val_accuracy: 0.8320
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5557 - accuracy: 0.8041 - val_loss: 0.4554 - val_accuracy: 0.8384
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5562 - accuracy: 0.8059 - val_loss: 0.4486 - val_accuracy: 0.8422
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5480 - accuracy: 0.8066 - val_loss: 0.4445 - val_accuracy: 0.8426
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5577 - accuracy: 0.8057 - val_loss: 0.4396 - val_accuracy: 0.8424
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5353 - accuracy: 0.8135 - val_loss: 0.4386 - val_accuracy: 0.8379
Epoch 23/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5341 - accuracy: 0.8100 - val_loss: 0.4227 - val_accuracy: 0.8510
Epoch 24/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5303 - accuracy: 0.8157 - val_loss: 0.4265 - val_accuracy: 0.8449
Epoch 25/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5293 - accuracy: 0.8172 - val_loss: 0.4187 - val_accuracy: 0.8561
Epoch 26/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5248 - accuracy: 0.8165 - val_loss: 0.4239 - val_accuracy: 0.8528
Epoch 27/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5208 - accuracy: 0.8222 - val_loss: 0.4100 - val_accuracy: 0.8618
Epoch 28/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5147 - accuracy: 0.8197 - val_loss: 0.4091 - val_accuracy: 0.8546
Epoch 29/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5168 - accuracy: 0.8209 - val_loss: 0.4079 - val_accuracy: 0.8589
Epoch 30/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5024 - accuracy: 0.8245 - val_loss: 0.4099 - val_accuracy: 0.8591
Epoch 31/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5130 - accuracy: 0.8207 - val_loss: 0.4052 - val_accuracy: 0.8583
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5059 - accuracy: 0.8252 - val_loss: 0.4097 - val_accuracy: 0.8577
Model: "Dropout0.5"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_283 (Dense) (None, 24) 1248
dropout_6 (Dropout) (None, 24) 0
dense_284 (Dense) (None, 24) 600
dropout_7 (Dropout) (None, 24) 0
dense_285 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 5s 6ms/step - loss: 1.4342 - accuracy: 0.4098 - val_loss: 1.1574 - val_accuracy: 0.6411
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 1.2126 - accuracy: 0.5290 - val_loss: 0.9914 - val_accuracy: 0.6785
Epoch 3/50
636/636 [==============================] - 3s 4ms/step - loss: 1.1308 - accuracy: 0.5720 - val_loss: 0.9120 - val_accuracy: 0.6881
Epoch 4/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0793 - accuracy: 0.5941 - val_loss: 0.8588 - val_accuracy: 0.7144
Epoch 5/50
636/636 [==============================] - 3s 4ms/step - loss: 1.0432 - accuracy: 0.6135 - val_loss: 0.8212 - val_accuracy: 0.7292
Epoch 6/50
636/636 [==============================] - 4s 6ms/step - loss: 1.0127 - accuracy: 0.6230 - val_loss: 0.7960 - val_accuracy: 0.7282
Epoch 7/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9970 - accuracy: 0.6343 - val_loss: 0.7700 - val_accuracy: 0.7441
Epoch 8/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9818 - accuracy: 0.6484 - val_loss: 0.7500 - val_accuracy: 0.7455
Epoch 9/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9628 - accuracy: 0.6520 - val_loss: 0.7375 - val_accuracy: 0.7490
Epoch 10/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9458 - accuracy: 0.6583 - val_loss: 0.7457 - val_accuracy: 0.7533
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9341 - accuracy: 0.6655 - val_loss: 0.7047 - val_accuracy: 0.7616
Epoch 12/50
636/636 [==============================] - 4s 6ms/step - loss: 0.9268 - accuracy: 0.6672 - val_loss: 0.6958 - val_accuracy: 0.7618
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9179 - accuracy: 0.6725 - val_loss: 0.6826 - val_accuracy: 0.7763
Epoch 14/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9080 - accuracy: 0.6758 - val_loss: 0.6775 - val_accuracy: 0.7677
Epoch 15/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9016 - accuracy: 0.6835 - val_loss: 0.6608 - val_accuracy: 0.7854
Epoch 16/50
636/636 [==============================] - 3s 4ms/step - loss: 0.9024 - accuracy: 0.6837 - val_loss: 0.6590 - val_accuracy: 0.7740
Epoch 17/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8910 - accuracy: 0.6853 - val_loss: 0.6673 - val_accuracy: 0.7712
Epoch 18/50
636/636 [==============================] - 4s 6ms/step - loss: 0.8821 - accuracy: 0.6919 - val_loss: 0.6438 - val_accuracy: 0.7797
Epoch 19/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8816 - accuracy: 0.6907 - val_loss: 0.6325 - val_accuracy: 0.7822
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.8758 - accuracy: 0.6981 - val_loss: 0.6454 - val_accuracy: 0.7761
In [ ]:
accuracy_measures_dropout = accuracy_measures.copy()
In [ ]:
plot_accuracy_measures(accuracy_measures, "Compare Dropout")
The result of dropout experiment shows a close result between Dropout-None and Dropout0.001. I will choose the model Dropout0.001 because of small reduction in overfitting but slightly higher accuracy.
In [ ]:
model_config['dropout_rate'] = 0.001
Model - Evaluation¶
In [ ]:
def get_best_param_model_config(model_name):
"""
(str) -> dict
This method returns the best hyper parameters from experiments.
Parameters
----------
model_name - The name of the model
Returns
----------
dict - The dictionary containing the best hyper parameters from experiments.
"""
# Default model config
model_config = get_base_model_config()
model_config['model_name'] = model_name
# Include F1-Score in the metrics so we can use that score during evaluation
model_config['metrics'] = ['accuracy', F1Score(average='macro')], # set the average so the F1-Score will not return an array per each individual classes.
# These are the final hyperparameters that I will use in training the Models:
# Hyperparameters:
model_config['batch_size'] = 32
model_config['epochs'] = 50
# Number of layers: 3
# Number of nodes per each layer: 32-16-16
model_config['hidden_activation'] = 'relu'
model_config['weights_initializer'] = 'random_normal'
# Batch Normalization: not added
model_config['optimizer'] = 'adam'
model_config['learning_rate'] = 0.001
model_config['regularizer'] = None
model_config['dropout_rate'] = 0.001
custom_layers = [
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation'], input_dim=model_config['input_dim']),
layers.Dropout(model_config['dropout_rate']),
layers.Dense(24, kernel_regularizer=model_config['regularizer'], kernel_initializer=model_config['weights_initializer'], activation=model_config['hidden_activation']),
layers.Dropout(model_config['dropout_rate']),
layers.Dense(model_config['output_nodes'], activation=model_config['output_activation'])
]
model_config['custom_layers'].clear()
model_config['custom_layers'].extend(custom_layers)
# we would like to save the model starting here for evaluation.
model_config['is_save_model'] = True
return model_config
In [ ]:
from keras.models import load_model
def evaluate_model(model_config, X, y):
"""
(dict, tf.data.Dataset)
This is a reusable function that load a previously trained model and run an evaluation.
Parameters
----------
model_config - The dictionary to use in model evaluation.
test_ds - The test dataset to use in the evaluation.
Returns
----------
Dataframe - The Panda Dataframe containing the scores from the experiment. The scores includes the Accuracy and Loss.
"""
model_file = str(model_config['workspace_path']) + str(model_config['model_name']) + '.h5'
model = load_model(model_file)
test_loss, test_accuracy, test_f1_score = model.evaluate(X, y, verbose=model_config['verbose'])
print(f"Test accuracy: {test_accuracy}, Test F1-Score: {test_f1_score}")
In [ ]:
# reset accuracy measures
accuracy_measures = {}
In [ ]:
model_name = 'Multiclass_Clarissification_of_Malicious_URL'
best_param_model_config = get_best_param_model_config(model_name)
In [ ]:
best_model_history = create_and_run_model(best_param_model_config, X_train, y_train_dummy)
Model: "Multiclass_Clarissification_of_Malicious_URL"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_292 (Dense) (None, 24) 1248
dropout_12 (Dropout) (None, 24) 0
dense_293 (Dense) (None, 24) 600
dropout_13 (Dropout) (None, 24) 0
dense_294 (Dense) (None, 5) 125
=================================================================
Total params: 1973 (7.71 KB)
Trainable params: 1973 (7.71 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/50
636/636 [==============================] - 4s 5ms/step - loss: 1.2485 - accuracy: 0.5534 - f1_score: 0.5265 - val_loss: 1.0466 - val_accuracy: 0.6240 - val_f1_score: 0.6094
Epoch 2/50
636/636 [==============================] - 3s 5ms/step - loss: 0.9458 - accuracy: 0.6700 - f1_score: 0.6592 - val_loss: 0.8416 - val_accuracy: 0.7129 - val_f1_score: 0.7014
Epoch 3/50
636/636 [==============================] - 3s 5ms/step - loss: 0.8074 - accuracy: 0.7079 - f1_score: 0.6970 - val_loss: 0.7599 - val_accuracy: 0.7164 - val_f1_score: 0.7056
Epoch 4/50
636/636 [==============================] - 4s 6ms/step - loss: 0.7410 - accuracy: 0.7308 - f1_score: 0.7217 - val_loss: 0.7103 - val_accuracy: 0.7406 - val_f1_score: 0.7321
Epoch 5/50
636/636 [==============================] - 3s 5ms/step - loss: 0.6960 - accuracy: 0.7426 - f1_score: 0.7347 - val_loss: 0.6703 - val_accuracy: 0.7565 - val_f1_score: 0.7523
Epoch 6/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6571 - accuracy: 0.7610 - f1_score: 0.7544 - val_loss: 0.6308 - val_accuracy: 0.7763 - val_f1_score: 0.7710
Epoch 7/50
636/636 [==============================] - 3s 4ms/step - loss: 0.6238 - accuracy: 0.7728 - f1_score: 0.7670 - val_loss: 0.6038 - val_accuracy: 0.7838 - val_f1_score: 0.7797
Epoch 8/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5939 - accuracy: 0.7861 - f1_score: 0.7811 - val_loss: 0.5829 - val_accuracy: 0.7860 - val_f1_score: 0.7793
Epoch 9/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5719 - accuracy: 0.7952 - f1_score: 0.7909 - val_loss: 0.5665 - val_accuracy: 0.7932 - val_f1_score: 0.7903
Epoch 10/50
636/636 [==============================] - 4s 6ms/step - loss: 0.5533 - accuracy: 0.8012 - f1_score: 0.7974 - val_loss: 0.5479 - val_accuracy: 0.8084 - val_f1_score: 0.8071
Epoch 11/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5374 - accuracy: 0.8057 - f1_score: 0.8024 - val_loss: 0.5256 - val_accuracy: 0.8117 - val_f1_score: 0.8091
Epoch 12/50
636/636 [==============================] - 3s 4ms/step - loss: 0.5209 - accuracy: 0.8138 - f1_score: 0.8113 - val_loss: 0.5177 - val_accuracy: 0.8042 - val_f1_score: 0.8016
Epoch 13/50
636/636 [==============================] - 3s 5ms/step - loss: 0.5082 - accuracy: 0.8156 - f1_score: 0.8132 - val_loss: 0.4992 - val_accuracy: 0.8229 - val_f1_score: 0.8206
Epoch 14/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4956 - accuracy: 0.8236 - f1_score: 0.8214 - val_loss: 0.4946 - val_accuracy: 0.8249 - val_f1_score: 0.8235
Epoch 15/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4851 - accuracy: 0.8273 - f1_score: 0.8255 - val_loss: 0.4826 - val_accuracy: 0.8261 - val_f1_score: 0.8257
Epoch 16/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4744 - accuracy: 0.8289 - f1_score: 0.8271 - val_loss: 0.4891 - val_accuracy: 0.8282 - val_f1_score: 0.8304
Epoch 17/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4655 - accuracy: 0.8336 - f1_score: 0.8323 - val_loss: 0.4771 - val_accuracy: 0.8298 - val_f1_score: 0.8290
Epoch 18/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4566 - accuracy: 0.8348 - f1_score: 0.8336 - val_loss: 0.4594 - val_accuracy: 0.8406 - val_f1_score: 0.8407
Epoch 19/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4507 - accuracy: 0.8387 - f1_score: 0.8377 - val_loss: 0.4517 - val_accuracy: 0.8351 - val_f1_score: 0.8369
Epoch 20/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4446 - accuracy: 0.8414 - f1_score: 0.8405 - val_loss: 0.4638 - val_accuracy: 0.8355 - val_f1_score: 0.8368
Epoch 21/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4378 - accuracy: 0.8428 - f1_score: 0.8421 - val_loss: 0.4459 - val_accuracy: 0.8398 - val_f1_score: 0.8406
Epoch 22/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4315 - accuracy: 0.8447 - f1_score: 0.8439 - val_loss: 0.4594 - val_accuracy: 0.8329 - val_f1_score: 0.8304
Epoch 23/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4240 - accuracy: 0.8471 - f1_score: 0.8464 - val_loss: 0.4230 - val_accuracy: 0.8534 - val_f1_score: 0.8537
Epoch 24/50
636/636 [==============================] - 4s 6ms/step - loss: 0.4194 - accuracy: 0.8502 - f1_score: 0.8497 - val_loss: 0.4233 - val_accuracy: 0.8494 - val_f1_score: 0.8508
Epoch 25/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4140 - accuracy: 0.8520 - f1_score: 0.8517 - val_loss: 0.4211 - val_accuracy: 0.8493 - val_f1_score: 0.8503
Epoch 26/50
636/636 [==============================] - 3s 5ms/step - loss: 0.4072 - accuracy: 0.8531 - f1_score: 0.8529 - val_loss: 0.4128 - val_accuracy: 0.8585 - val_f1_score: 0.8606
Epoch 27/50
636/636 [==============================] - 3s 4ms/step - loss: 0.4022 - accuracy: 0.8558 - f1_score: 0.8556 - val_loss: 0.4098 - val_accuracy: 0.8583 - val_f1_score: 0.8595
Epoch 28/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3985 - accuracy: 0.8582 - f1_score: 0.8580 - val_loss: 0.3959 - val_accuracy: 0.8601 - val_f1_score: 0.8611
Epoch 29/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3914 - accuracy: 0.8594 - f1_score: 0.8593 - val_loss: 0.4492 - val_accuracy: 0.8457 - val_f1_score: 0.8483
Epoch 30/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3872 - accuracy: 0.8606 - f1_score: 0.8606 - val_loss: 0.3975 - val_accuracy: 0.8601 - val_f1_score: 0.8619
Epoch 31/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3839 - accuracy: 0.8617 - f1_score: 0.8615 - val_loss: 0.3930 - val_accuracy: 0.8616 - val_f1_score: 0.8639
Epoch 32/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3795 - accuracy: 0.8670 - f1_score: 0.8670 - val_loss: 0.3875 - val_accuracy: 0.8669 - val_f1_score: 0.8687
Epoch 33/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3759 - accuracy: 0.8678 - f1_score: 0.8678 - val_loss: 0.3911 - val_accuracy: 0.8589 - val_f1_score: 0.8594
Epoch 34/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3703 - accuracy: 0.8677 - f1_score: 0.8675 - val_loss: 0.4159 - val_accuracy: 0.8487 - val_f1_score: 0.8474
Epoch 35/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3659 - accuracy: 0.8704 - f1_score: 0.8704 - val_loss: 0.3883 - val_accuracy: 0.8689 - val_f1_score: 0.8708
Epoch 36/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3626 - accuracy: 0.8718 - f1_score: 0.8718 - val_loss: 0.3762 - val_accuracy: 0.8652 - val_f1_score: 0.8668
Epoch 37/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3576 - accuracy: 0.8724 - f1_score: 0.8725 - val_loss: 0.3674 - val_accuracy: 0.8713 - val_f1_score: 0.8735
Epoch 38/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3545 - accuracy: 0.8755 - f1_score: 0.8756 - val_loss: 0.3562 - val_accuracy: 0.8746 - val_f1_score: 0.8763
Epoch 39/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3519 - accuracy: 0.8748 - f1_score: 0.8750 - val_loss: 0.3605 - val_accuracy: 0.8722 - val_f1_score: 0.8740
Epoch 40/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3438 - accuracy: 0.8792 - f1_score: 0.8792 - val_loss: 0.3666 - val_accuracy: 0.8730 - val_f1_score: 0.8752
Epoch 41/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3409 - accuracy: 0.8785 - f1_score: 0.8786 - val_loss: 0.3599 - val_accuracy: 0.8738 - val_f1_score: 0.8758
Epoch 42/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3387 - accuracy: 0.8810 - f1_score: 0.8810 - val_loss: 0.3540 - val_accuracy: 0.8783 - val_f1_score: 0.8800
Epoch 43/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3344 - accuracy: 0.8838 - f1_score: 0.8840 - val_loss: 0.3544 - val_accuracy: 0.8709 - val_f1_score: 0.8724
Epoch 44/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3321 - accuracy: 0.8827 - f1_score: 0.8828 - val_loss: 0.3657 - val_accuracy: 0.8646 - val_f1_score: 0.8637
Epoch 45/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3257 - accuracy: 0.8858 - f1_score: 0.8859 - val_loss: 0.3628 - val_accuracy: 0.8699 - val_f1_score: 0.8713
Epoch 46/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3241 - accuracy: 0.8860 - f1_score: 0.8862 - val_loss: 0.3331 - val_accuracy: 0.8809 - val_f1_score: 0.8821
Epoch 47/50
636/636 [==============================] - 3s 4ms/step - loss: 0.3200 - accuracy: 0.8902 - f1_score: 0.8905 - val_loss: 0.3249 - val_accuracy: 0.8872 - val_f1_score: 0.8894
Epoch 48/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3170 - accuracy: 0.8886 - f1_score: 0.8888 - val_loss: 0.3508 - val_accuracy: 0.8730 - val_f1_score: 0.8732
Epoch 49/50
636/636 [==============================] - 4s 6ms/step - loss: 0.3156 - accuracy: 0.8908 - f1_score: 0.8910 - val_loss: 0.3730 - val_accuracy: 0.8687 - val_f1_score: 0.8707
Epoch 50/50
636/636 [==============================] - 3s 5ms/step - loss: 0.3120 - accuracy: 0.8920 - f1_score: 0.8922 - val_loss: 0.3315 - val_accuracy: 0.8862 - val_f1_score: 0.8872
In [ ]:
plot_learning_curves(best_model_history, model_name)
In [ ]:
evaluate_model(best_param_model_config, X_test, y_test_dummy)
341/341 [==============================] - 1s 3ms/step - loss: 0.3266 - accuracy: 0.8869 - f1_score: 0.8860 Test accuracy: 0.8869014978408813, Test F1-Score: 0.8860093951225281
The study also conducted additional experiments by adding more layers and increasing the number of nodes and it results to higher accuracy but shows signs of overfitting and instability on the model. The experiment with 3 hidden layers with 128-128-64 nodes even results to 94% accuracy. Meanwhile, adding more layers on top of the three hidden layers shows no improvement and even deteriorates the accuracy.
Conclusion¶
This study was able to prove that we can use Deep Learning via Keras to perform multiclass classification to identify the malicious URLs. It was also able to produce a model that has a good balance between accuracy and stability.
However, using Random Forest (RF) proves a better choice for this particular use case.
In [ ]: