Machine Learning for Complete Intersection Calabi-Yau Manifolds¶
We then separately consider another ML approach to predict the Hodge numbers from the configuration matrix of CICY 3-folds: we implement neural networks (NN) in order to improve previous predictions using different kinds of architectures. In particular we will use convolutional NN (CNN) to process the configuration matrix using kernel convolutions (as if we were considering a "computer vision" task). We try different architectures such as a CNN with only convolutional hidden layers, as opposed to a CNN which uses a fully connected (FC) network to process the output.
We will use the Tensorflow backend to implement NNs on the GPU, using the Keras high level API.
Infrastructure¶
We print information about the current OS:
In [1]:
from mltools.libos import InfoOS
print('Current OS: {} (kernel release: {}, architecture: {})'.format(InfoOS().os, InfoOS().kernel, InfoOS().arch))
print('Number of available threads: {:d}'.format(InfoOS().threads))
print('Current CPU frequency: {:.0f} MHz (max: {:.0f} MHz)'.format(InfoOS().freq, InfoOS().freqm))
print('Available RAM memory: {:d} MB (tot: {:d} MB)'.format(InfoOS().vmav, InfoOS().vmtot))
# print info on GPU
!nvidia-smi
Current OS: Linux (kernel release: 5.6.11-arch1-1, architecture: x86_64)
Number of available threads: 8
Current CPU frequency: 2998 MHz (max: 3800 MHz)
Available RAM memory: 10792 MB (tot: 15758 MB)
Sat May 9 21:05:11 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce 940MX Off | 00000000:02:00.0 Off | N/A |
| N/A 68C P8 N/A / N/A | 5MiB / 2004MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 15009 G /usr/lib/Xorg 4MiB |
+-----------------------------------------------------------------------------+
Setup¶
We import the Python modules we use and print their versions to keep track of changes.
In [2]:
import json
import sys
import matplotlib as mpl
import random as rnd
import sklearn as skl
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend as K
import warnings
warnings.simplefilter(action='ignore', category=UserWarning) # ignore user warnings: nothing that I can really do anything about it...
%matplotlib inline
mpl.rc('axes', labelsize=12)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)
# print the version of the modules
print('Python version: {:d}.{:d}' .format(sys.version_info.major, sys.version_info.minor))
print('Matplot version: {}' .format(mpl.__version__))
print('Numpy version: {}' .format(np.__version__))
print('Pandas version: {}' .format(pd.__version__))
print('Scikit-learn version: {}' .format(skl.__version__))
print('Tensorflow version: {}' .format(tf.__version__))
print('Keras version: {} (backend: {})'.format(keras.__version__, K.backend()))
# fix random_seed
RAND = 42
rnd.seed(RAND)
np.random.seed(RAND)
tf.random.set_seed(RAND)
Python version: 3.7 Matplot version: 3.2.1 Numpy version: 1.18.4 Pandas version: 1.0.3 Scikit-learn version: 0.22.2.post1 Tensorflow version: 2.0.0 Keras version: 2.2.4-tf (backend: tensorflow)
Session Preparation¶
in order to save the results of the analysis, we define where to store images, log files and models:
In [3]:
from os import path, makedirs
ROOT_DIR = '.' #-------------------------------------------------- root directory
IMG_DIR = 'img' #------------------------------------------------ directory of images
MOD_DIR = 'models' #--------------------------------------------- directory of saved models
LOG_DIR = 'log' #------------------------------------------------ directory of logs
OUT_DIR = 'output' #--------------------------------------------- directory of outputs, data, csv files, etc.
DB_NAME = 'cicy3o' #---------------------------------------------- name of the dataset
DB_FILE = DB_NAME + '_analysis.h5' #------------------------------ full name with extension
DB_PATH = path.join(ROOT_DIR, DB_FILE) #-------------------------- full path of the dataset
DB_DIR = 'original' if DB_NAME == 'cicy3o' else 'favourable' #--- subdir where to store images, models, logs
# define full paths
IMG_PATH = path.join(ROOT_DIR, IMG_DIR, DB_DIR)
MOD_PATH = path.join(ROOT_DIR, MOD_DIR, DB_DIR)
LOG_PATH = path.join(ROOT_DIR, LOG_DIR, DB_DIR)
OUT_PATH = path.join(ROOT_DIR, OUT_DIR, DB_DIR)
# create directories if non existent
if not path.isdir(IMG_PATH):
makedirs(IMG_PATH, exist_ok=True)
if not path.isdir(MOD_PATH):
makedirs(MOD_PATH, exist_ok=True)
if not path.isdir(LOG_PATH):
makedirs(LOG_PATH, exist_ok=True)
if not path.isdir(OUT_PATH):
makedirs(OUT_PATH, exist_ok=True)
We also create a log file to store debug and related information:
In [4]:
import logging
from mltools.liblog import create_logfile
path_to_log = path.join(LOG_PATH,
DB_NAME + '_nn.log'
) #----------------------------------------------------- path to the log file
log = create_logfile(path_to_log,
name=DB_NAME + '_nn',
level=logging.DEBUG
) #-------------------------------------------------------- create a log file and a log session
# these lines provide the same setup also for the Jupyter logging
logger = logging.getLogger() #------------------------------------------------- get the current logging session
fmt = logging.Formatter('%(asctime)s: %(levelname)s ==> %(message)s') #-------- customise the formatting options
handler = logging.StreamHandler() #-------------------------------------------- handle the stream to the default (stderr)
handler.setLevel(logging.DEBUG) #---------------------------------------------- print everything
handler.setFormatter(fmt) #---------------------------------------------------- set the formatting options
logger.handlers = [handler] #-------------------------------------------------- override the default stream
# we are ready to go!
log.info('New logging session started. Log is at {}.'.format(path_to_log))
Rotating existing logs...
2020-05-09 21:05:17,798: INFO ==> New logging session started. Log is at ./log/original/cicy3o_nn.log.
We finally set the memory growth property of the GPU in order to avoid overflowing its RAM memory:
In [5]:
gpus = tf.config.experimental.list_physical_devices('GPU') #--------------------------------------- list of physical GPUs
if gpus: #----------------------------------------------------------------------------------------- set memory growth only if GPU is active
try:
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True) #---------------------------------- set memory growth
logical_gpus = tf.config.experimental.list_logical_devices('GPU') #------------------------ list of logical devices
print('GPU setup: {:d} physical GPUs, {:d} logical GPUs.'.format(len(gpus),
len(logical_gpus)
)
)
except RuntimeError as e:
print(e)
else:
print('No GPUs in the setup!')
GPU setup: 1 physical GPUs, 1 logical GPUs.
Loading the Dataset¶
We first load the dataset we built during the preanalysis.
In [6]:
import pandas as pd
# load the dataset
if path.isfile(DB_PATH):
df = pd.read_hdf(DB_PATH)
log.debug('Database loaded.')
log.info('Shape is {:d} rows x {:d} columns.'.format(df.shape[0], df.shape[1]))
else:
log.error('Cannot load database from {}!'.format(DB_PATH))
2020-05-09 21:05:18,886: DEBUG ==> Database loaded. 2020-05-09 21:05:18,887: INFO ==> Shape is 7851 rows x 7 columns.
We print the dtypes and the name of the keys inside the dataframe as a reference:
In [7]:
df.dtypes
Out[7]:
h11 int16 h21 int16 num_cp int8 dim_cp object dim_h0_amb object matrix object pca object dtype: object
Dense Format Extraction¶
We now extract the matrix from the available features and the labels. Notice that in this case we do not flatten the output as it will be processed by Conv2D layers in Keras: we reshape it as if it was a black and white image, that is we take each matrix of shape (width, height) and output a matrix of shape (width, height, 1):
In [8]:
from mltools.libtransformer import ExtractTensor
# extract the labels
h11 = df['h11'].values
h21 = df['h21'].values
# we also extract num_cp for the plots (not for training)
num_cp = np.array(ExtractTensor(flatten=True).fit_transform(df['num_cp']))
# extract the tensor features
matrix = np.array(ExtractTensor(flatten=False).fit_transform(df['matrix'])) #-------- old shape is (batch, width, height)
matrix = np.reshape(matrix, (-1, np.shape(matrix)[1], np.shape(matrix)[2], 1)) #----- new shape is (batch, width, height, 1)
log.debug('Matrix shape: {}'.format(np.shape(matrix)))
2020-05-09 21:05:21,027: DEBUG ==> Matrix shape: (7851, 12, 15, 1)
Training and Validation Strategy¶
We now define the validation strategy and split the dataset into training, validation and test sets.
We will take out 10% of the dataset to be our test set. For the remaining 90% of the dataset, we will use holdout validation with 19 of the training samples as validation (effectively we use 80% of the total set for training and 10% as validation).
In [9]:
from sklearn.model_selection import train_test_split
# divide into training and test sets
matrix_train, matrix_test, \
num_cp_train, num_cp_test, \
h11_train, h11_test, \
h21_train, h21_test = train_test_split(matrix, num_cp, h11, h21, test_size=0.1, shuffle=False)
# keep a holdout validation set
matrix_train, matrix_val, \
num_cp_train, num_cp_val, \
h11_train, h11_val, \
h21_train, h21_val = train_test_split(matrix_train, num_cp_train, h11_train, h21_train, test_size=1.0/9.0, shuffle=False)
log.debug('Train set size: {:d}'.format(np.shape(matrix_train)[0]))
log.debug('Validation set size: {:d}'.format(np.shape(matrix_val)[0]))
log.debug('Test set size: {:d}'.format(np.shape(matrix_test)[0]))
2020-05-09 21:05:21,115: DEBUG ==> Train set size: 6280 2020-05-09 21:05:21,119: DEBUG ==> Validation set size: 785 2020-05-09 21:05:21,123: DEBUG ==> Test set size: 786
We also provide a way to visualise the predictions by comparing the predictions as function of a (scalar) base feature (e.g.: num_cp) with respect to the ground truth:
In [10]:
def get_counts(base_feature, label):
'''
Returns unique values and counts of the label as a function of the base_feature.
Required arguments:
base_feature: the feature considered as base for the comparison,
label: the label we are interested in comparing.
Yields:
[ unique value of base_feature, unique value of label, count ].
'''
for n in np.sort(np.unique(base_feature)):
uniques, counts = np.unique(label[np.argwhere(base_feature == n)[:,0]], return_counts=True)
for u, c in np.c_[uniques, counts]:
yield [n, u, c]
Notice that the use of num_cp will only be necessary to show in a simple way how well the algorithm can predict h11 and h21. num_cp will be taken as a representative of the matrices in the test set and h11 and h21 will be shown in a 2D plot.
Feature Rescaling¶
Before proceeding we rescale the input matrices to have entries in [0,1]: we divide each entry of each matrix by the range of variation of its entries (since they are all positive, we divide by the maximum) in the training set. We then apply the same scaling to validation and test sets.
In [11]:
# compute the scale factor
scale_factor = np.max(matrix_train) - np.min(matrix_train)
# rescale the sets
matrix_train = matrix_train / scale_factor
matrix_val = matrix_val / scale_factor
matrix_test = matrix_test / scale_factor
# define the shape of the tensors (batch axis excluded) since we will use it later one
input_shape = np.shape(matrix_train)[1:]
Input Visualisation¶
Using matplotlib we can try to visualise a few samples of the training set to better understand what kind of input we are dealing with and come up with a good strategy for the architecture of the neural network.
In [12]:
from mltools.libplot import Plot
# choose 4 random indices
rnd = np.random.randint(low=0, high=np.shape(matrix_train)[0], size=(9,))
# decide the colour map
cmap = 'plasma'
# plot the matrices
plot = Plot(rows=3, columns=3)
plot.matrix(data=np.reshape(matrix_train[rnd[0]], (np.shape(matrix_train[rnd[0]])[0], np.shape(matrix_train[rnd[0]])[1])),
axis=(0,0),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[0]], '{21}', h21_train[rnd[0]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[1]], (np.shape(matrix_train[rnd[1]])[0], np.shape(matrix_train[rnd[1]])[1])),
axis=(0,1),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[1]], '{21}', h21_train[rnd[1]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[2]], (np.shape(matrix_train[rnd[2]])[0], np.shape(matrix_train[rnd[2]])[1])),
axis=(0,2),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[2]], '{21}', h21_train[rnd[2]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[3]], (np.shape(matrix_train[rnd[3]])[0], np.shape(matrix_train[rnd[3]])[1])),
axis=(1,0),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[3]], '{21}', h21_train[rnd[3]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[4]], (np.shape(matrix_train[rnd[4]])[0], np.shape(matrix_train[rnd[4]])[1])),
axis=(1,1),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[4]], '{21}', h21_train[rnd[4]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[5]], (np.shape(matrix_train[rnd[5]])[0], np.shape(matrix_train[rnd[5]])[1])),
axis=(1,2),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[5]], '{21}', h21_train[rnd[5]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[6]], (np.shape(matrix_train[rnd[6]])[0], np.shape(matrix_train[rnd[6]])[1])),
axis=(2,0),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[6]], '{21}', h21_train[rnd[6]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[7]], (np.shape(matrix_train[rnd[7]])[0], np.shape(matrix_train[rnd[7]])[1])),
axis=(2,1),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[7]], '{21}', h21_train[rnd[7]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.matrix(data=np.reshape(matrix_train[rnd[8]], (np.shape(matrix_train[rnd[8]])[0], np.shape(matrix_train[rnd[8]])[1])),
axis=(2,2),
title='$h_{} = {:d}$, $h_{} = {:d}$'.format('{11}', h11_train[rnd[8]], '{21}', h21_train[rnd[8]]),
cmap=cmap,
vmin=0.0,
vmax=1.0
)
plot.save_and_close(path.join(IMG_PATH, 'nn_input_visualisation'))
log.debug('Plot saved in {}.'.format(path.join(IMG_PATH, 'nn_input_visualisation.pdf')))
2020-05-09 21:05:30,882: DEBUG ==> Plot saved in ./img/original/nn_input_visualisation.pdf.
Architecture Building¶
We then move to the problem of building the correct architecture. We will first implement a function which will allow us to build different models using similar structure blocks: the idea is to first process the images through convolutional layers, then add a FC network at the end (we will also add dropout and normalization layers).
In [13]:
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv2D, Dense, LeakyReLU, Dropout, ZeroPadding2D, BatchNormalization, Flatten, concatenate
from tensorflow.keras.regularizers import l1_l2
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import model_to_dot
def cnn_model(input_shape,
model_name='cnn_model',
learning_rate=0.01,
conv_layers=[32],
conv_paddings=['same'],
conv_kernels=[(2,2)],
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
out_name='output',
l1_reg=0.0,
l2_reg=0.0
):
'''
Build a CNN model.
Required arguments:
input_size: the size of the input tensor.
Optional arguments:
model_name: the name of the model,
learning_rate: the learning rate of the gradient descent,
conv_layers: a list-like object with the no. of filters for each hiddend convolution layer,
conv_paddings: a list-like object with the corresponding padding for each layer,
conv_kernels: a list-like object with the kernel size of each layer,
conv_alpha: the slope of the LeakyReLU activation (ReLU if 0.0) of the convolution layers,
fc_layers: a list-like object with the no. of units for each hidden dense layer,
fc_alpha: the slope of the LeakyReLU activation (ReLU if 0.0) of the FC network,
dropout: the dropout rate (do not use dropout if <= 0.0),
full_dropout: use this dropout rate after every layer (disabled if <= 0.0),
normalization: the momentum of the batch normalization (do not use normalization if <= 0.0),
last_relu: whether to use ReLU activation in the output layer (force positive output),
out_name: the name of the output layer,
l1_reg: the L1 kernel regularization factor,
l2_reg: the L2 kernel regularization factor.
Returns:
the compiled model.
'''
# define the regularizer
regularizer = l1_l2(l1=l1_reg, l2=l2_reg) #--------------------------------------------- regularizer
# build the model
I = Input(shape=input_shape, name=model_name + '_input') #------------------------------ input layer
x = I
# build convolutional layers
for n in range(np.shape(conv_layers)[0]): #--------------------------------------------- loop through the conv. layers
x = Conv2D(filters=conv_layers[n],
kernel_size=conv_kernels[n],
padding=conv_paddings[n],
kernel_regularizer=regularizer,
name=model_name + '_conv2d_' + str(n)
)(x) #-------------------------------------------------------------------- add conv. layer
x = LeakyReLU(alpha=conv_alpha,
name=model_name + '_conv2d_' + str(n) + '_activation'
)(x) #----------------------------------------------------------------- add activation
if normalization > 0.0:
x = BatchNormalization(momentum=normalization,
name=model_name + '_conv2d_' + str(n) + '_normalization'
)(x) #---------------------------------------------------- add batch normalization (if requested)
if full_dropout > 0.0:
x = Dropout(rate=full_dropout,
name=model_name + '_conv2d_' + str(n) + '_full_dropout'
)(x) #--------------------------------------------------------------- add dropout (if requested)
# add dropout
if dropout > 0.0 and full_dropout <= 0.0:
x = Dropout(rate=dropout,
name=model_name + '_dropout'
)(x) #------------------------------------------------------------------- add dropout (if requested)
# flatten the output
x = Flatten(name=model_name + '_flatten')(x) #------------------------------------------ flatten the output
# build FC network
for n in range(np.shape(fc_layers)[0]):
x = Dense(units=fc_layers[n],
kernel_regularizer=regularizer,
name=model_name + '_fc_' + str(n)
)(x) #--------------------------------------------------------------------- add dense layers
x = LeakyReLU(alpha=fc_alpha,
name=model_name + '_fc_' + str(n) + '_activation'
)(x) #----------------------------------------------------------------- add activation
if normalization > 0.0:
x = BatchNormalization(momentum=normalization,
name=model_name + '_fc_' + str(n) + '_normalization'
)(x) #---------------------------------------------------- add batch normalization (if requested)
if full_dropout > 0.0:
x = Dropout(rate=full_dropout,
name=model_name + '_fc_' + str(n) + '_full_dropout'
)(x) #--------------------------------------------------------------- add dropout (if requested)
if last_relu: #------------------------------------------------------------------------- output layer
F = Dense(1, activation='relu', name=model_name + '_' + out_name)(x)
else:
F = Dense(1, name=model_name + '_' + out_name)(x)
# define the model
model = Model(inputs=I, outputs=F, name=model_name)
# compile the model
model.compile(optimizer=Adam(learning_rate=learning_rate),
loss='mean_squared_error',
metrics=['mean_squared_error']
)
# return the compiled model
return model
def scan_inception_model(input_shape,
model_name='inception_model',
learning_rate=0.01,
conv_layers=[32],
conv_padding='same',
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
out_name='output',
l1_reg=0.0,
l2_reg=0.0
):
'''
Build a CNN 'scan-inception' model: scan over rows and columns and merge.
Required arguments:
input_size: the size of the input tensor.
Optional arguments:
model_name: the name of the model,
learning_rate: the learning rate of the gradient descent,
conv_layers: a list-like object with the no. of filters for each 'inception' modules,
conv_padding: the padding to use for the convolutional scans,
conv_alpha: the slope of the LeakyReLU activation (ReLU if 0.0) of the convolution layers,
fc_layers: a list-like object with the no. of units for each hidden dense layer,
fc_alpha: the slope of the LeakyReLU activation (ReLU if 0.0) of the FC network,
dropout: the dropout rate (do not use dropout if <= 0.0),
full_dropout: use this dropout rate after every layer (disabled if <= 0.0),
normalization: the momentum of the batch normalization (do not use normalization if <= 0.0),
last_relu: whether to use ReLU activation in the output layer (force positive output),
out_name: the name of the output layer,
l1_reg: the L1 kernel regularization factor,
l2_reg: the L2 kernel regularization factor.
Returns:
the compiled model.
'''
# define the regularizer
regularizer = l1_l2(l1=l1_reg, l2=l2_reg) #--------------------------------------------- regularizer
# build the model
I = Input(shape=input_shape, name=model_name + '_input') #------------------------------ input layer
if conv_padding == 'same':
x = I
else:
x = ZeroPadding2D(padding=((0,3),(0,0)), data_format='channels_last')(I)
# build convolutional layers
for n in range(np.shape(conv_layers)[0]): #--------------------------------------------- loop through the conv. layers
a = Conv2D(filters=conv_layers[n],
kernel_size=(x.shape[1],1),
padding=conv_padding,
kernel_regularizer=regularizer,
name=model_name + '_conv2d_rows_' + str(n)
)(x) #-------------------------------------------------------------------- add conv. layer over rows
a = LeakyReLU(alpha=conv_alpha,
name=model_name + '_conv2d_rows_' + str(n) + '_activation'
)(a) #----------------------------------------------------------------- add activation
b = Conv2D(filters=conv_layers[n],
kernel_size=(1,x.shape[2]),
padding=conv_padding,
kernel_regularizer=regularizer,
name=model_name + '_conv2d_columns_' + str(n)
)(x) #-------------------------------------------------------------------- add conv. layer over columns
b = LeakyReLU(alpha=conv_alpha,
name=model_name + '_conv2d_columns_' + str(n) + '_activation'
)(b) #----------------------------------------------------------------- add activation
x = concatenate([a, b],
name=model_name + '_concatenation_' + str(n)
) if conv_padding == 'same' \
else concatenate([a, tf.einsum('bij...->bji...',b)], #------------- swap columns and rows
name=model_name + '_concatenation_' + str(n)
) #------------------------------------------------ concatenate layers
if normalization > 0.0:
x = BatchNormalization(momentum=normalization,
name=model_name + '_conv2d_' + str(n) + '_normalization'
)(x) #---------------------------------------------------- add batch normalization (if requested)
if full_dropout > 0.0:
x = Dropout(rate=full_dropout,
name=model_name + '_conv2d_' + str(n) + '_full_dropout'
)(x) #--------------------------------------------------------------- add dropout (if requested)
# add dropout
if dropout > 0.0 and full_dropout <= 0.0:
x = Dropout(rate=dropout,
name=model_name + '_dropout'
)(x) #------------------------------------------------------------------- add dropout (if requested)
# flatten the output
x = Flatten(name=model_name + '_flatten')(x) #------------------------------------------ flatten the output
# build FC network
for n in range(np.shape(fc_layers)[0]):
x = Dense(units=fc_layers[n],
kernel_regularizer=regularizer,
name=model_name + '_fc_' + str(n)
)(x) #--------------------------------------------------------------------- add dense layers
x = LeakyReLU(alpha=fc_alpha,
name=model_name + '_fc_' + str(n) + '_activation'
)(x) #----------------------------------------------------------------- add activation
if normalization > 0.0:
x = BatchNormalization(momentum=normalization,
name=model_name + '_fc_' + str(n) + '_normalization'
)(x) #---------------------------------------------------- add batch normalization (if requested)
if full_dropout > 0.0:
x = Dropout(rate=full_dropout,
name=model_name + '_fc_' + str(n) + '_full_dropout'
)(x) #--------------------------------------------------------------- add dropout (if requested)
if last_relu: #------------------------------------------------------------------------- output layer
F = Dense(1, activation='relu', name=model_name + '_' + out_name)(x)
else:
F = Dense(1, name=model_name + '_' + out_name)(x)
# define the model
model = Model(inputs=I, outputs=F, name=model_name)
# compile the model
model.compile(optimizer=Adam(learning_rate=learning_rate),
loss='mean_squared_error',
metrics=['mean_squared_error']
)
# return the compiled model
return model
We will consider different models and train them all and then compare and discuss the results. The models we implement will compare these peculiarities:
- small vs. large kernels,
- FC network vs. no FC network,
- padding vs. no padding.
The idea behind the small kernels is to have a small no. of operations and to be fairly quick to train (smaller kernels will probably need more layers, though), while larger kernels usually improve the results in the field of semantic segmentation and may be worth the try (we will also need less hidden layers). The fully connected layers can then help in processing the output, however the no. of parameters will increase substantially since they are dense layers (one of the goals will be to build the smallest NN). The padding operation will instead guide the convolution operation by reducing the output (and input) size of each layer (when using padding we could also add a FC network to a very small convolutional layer and keep the no. of parameters small).
We will then compare the results with Bull et al.
In [14]:
from tensorflow.keras.backend import clear_session
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau
from mltools.libscore import Score
def cnn_fit(model,
X,
y,
X_val,
y_val,
batch_size=32,
epochs=100,
early_stopping=10,
reduce_lr=5,
verbose=0
):
'''
Fit the model.
Required arguments:
model: the model to fit,
X: the training features,
y: the training labels,
X_val: the validation features,
y_val: the validation labels.
Optional arguments:
batch_size: the size of the batch used in forward pass,
epochs: the number of epochs,
early_stopping: patience of early stopping,
reduce_lr: patience to reduce learning rate,
verbose: verbosity level (debug)
Returns:
the fitted model, the history of the model.
'''
# clear the TF graph
clear_session()
# define callbacks
callbacks = [EarlyStopping(monitor='val_mean_squared_error',
patience=early_stopping,
verbose=verbose,
restore_best_weights=True
),
ReduceLROnPlateau(monitor='val_mean_squared_error',
factor=0.3,
patience=reduce_lr,
verbose=verbose
),
ModelCheckpoint(filepath=path.join(MOD_PATH, model.name + '.h5'),
monitor='val_mean_squared_error',
verbose=verbose,
save_best_only=True,
save_format='h5'
)
]
# fit the model
history = model.fit(x=X,
y=y,
batch_size=batch_size,
epochs=epochs,
verbose=verbose,
callbacks=callbacks,
validation_data=(X_val, y_val)
)
# return the fitted model and its history
return model, history.history
The test results will then be saved to file along with other quantities which can be reused later on.
In [15]:
test_results = {'h11_true': h11_test.tolist(),
'h21_true': h21_test.tolist(),
'num_cp': num_cp_test.tolist(),
'matrix': matrix_test.tolist()
} #--------- dictionary with the predictions
Analysis for h11¶
We now have the instruments to proceed with the analysis, starting with the prediction of h11.
Base Analysis and Comparison¶
First of all we build the architecture of the network in Bull et al. and train it as a reference.
In [16]:
h11_base_model = cnn_model(input_shape=input_shape,
model_name='h11_bull_et_al',
learning_rate=1.0e-3,
conv_layers=[],
conv_paddings=[],
conv_kernels=[],
conv_alpha=0.0,
fc_layers=[876, 461, 437, 929, 404],
fc_alpha=0.0,
dropout=0.0,
full_dropout=0.2072,
normalization=0.99,
last_relu=True,
l1_reg=0.0,
l2_reg=0.0
)
# print characteristics
h11_base_model.summary()
# print architecture
h11_base_model_dot = model_to_dot(model=h11_base_model, show_shapes=True, show_layer_names=False)
h11_base_model_dot.write_pdf(path.join(IMG_PATH, h11_base_model.name + '.pdf'))
# train the model
log.debug('Training base model...')
h11_base_model, h11_base_model_history = cnn_fit(model=h11_base_model,
X=matrix_train, y=h11_train,
X_val=matrix_val, y_val=h11_val,
batch_size=np.shape(matrix_train)[0],
epochs=5000,
early_stopping=200,
reduce_lr=75,
verbose=0
)
log.debug('Base model has been trained.')
# compute the predictions
h11_base_model_val_predictions = np.reshape(h11_base_model.predict(matrix_val), (-1,)) #---- reshape to match the labels
h11_base_model_test_predictions = np.reshape(h11_base_model.predict(matrix_test), (-1,))
test_results[str(h11_base_model.name)] = h11_base_model_test_predictions.tolist()
# save the history of the model
with open(path.join(OUT_PATH, h11_base_model.name + '.json'), 'w') as f:
json.dump(str(h11_base_model_history), f)
log.debug('History of {} has been saved to {}.'.format(h11_base_model.name, path.join(OUT_PATH, h11_base_model.name + '.json')))
# save results to file
with open(path.join(OUT_PATH, 'convnet_test_results_h11_h21.json'), 'w') as f:
json.dump(test_results, f)
log.debug('History of {} has been saved to {}.'.format(h11_base_model.name, path.join(OUT_PATH, 'convnet_test_results_h11_h21.json')))
# compute the accuracy and score of the predictions
rounding = np.rint
h11_base_model_score_val = Score(y_true=h11_val, y_pred=h11_base_model_val_predictions, rounding=rounding)
h11_base_model_score_test = Score(y_true=h11_test, y_pred=h11_base_model_test_predictions, rounding=rounding)
print('Accuracy of {} on the validation set: {:.3f}%'.format(h11_base_model.name, h11_base_model_score_val.accuracy()*100))
print('Accuracy of {} on the test set: {:.3f}%'.format(h11_base_model.name, h11_base_model_score_test.accuracy()*100))
# plot the distribution of the predictions and the error difference
plot = Plot(rows=1, columns=5)
plot.scatter2D(np.array(list(get_counts(num_cp_val, h11_val))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='real values',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_val, rounding(h11_base_model_val_predictions)))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, h11_test))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='real values',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, rounding(h11_base_model_test_predictions)))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.hist2D(h11_base_model_score_val.error(),
axis=2,
title='Distance from the Real Value',
legend='validation set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.hist2D(h11_base_model_score_test.error(),
axis=2,
title='Distance from the Real Value',
legend='test set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.series2D(data=h11_base_model_history['loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='training loss',
ylog=True,
binstep=int(np.shape(h11_base_model_history['loss'])[0] / 10)
)
plot.series2D(data=h11_base_model_history['val_loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='validation loss',
ylog=True,
binstep=int(np.shape(h11_base_model_history['val_loss'])[0] / 10)
)
plot.series2D(data=h11_base_model_history['mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='training MSE',
ylog=True,
binstep=int(np.shape(h11_base_model_history['mean_squared_error'])[0] / 10)
)
plot.series2D(data=h11_base_model_history['val_mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='validation MSE',
ylog=True,
binstep=int(np.shape(h11_base_model_history['val_mean_squared_error'])[0] / 10)
)
plot.save_and_close(path.join(IMG_PATH, h11_base_model.name))
log.debug('Plot saved to {}.'.format(path.join(IMG_PATH, h11_base_model.name)))
2020-05-09 21:05:35,229: DEBUG ==> Training base model...
Model: "h11_bull_et_al" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_bull_et_al_input (InputL [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_bull_et_al_flatten (Flat (None, 180) 0 _________________________________________________________________ h11_bull_et_al_fc_0 (Dense) (None, 876) 158556 _________________________________________________________________ h11_bull_et_al_fc_0_activati (None, 876) 0 _________________________________________________________________ h11_bull_et_al_fc_0_normaliz (None, 876) 3504 _________________________________________________________________ h11_bull_et_al_fc_0_full_dro (None, 876) 0 _________________________________________________________________ h11_bull_et_al_fc_1 (Dense) (None, 461) 404297 _________________________________________________________________ h11_bull_et_al_fc_1_activati (None, 461) 0 _________________________________________________________________ h11_bull_et_al_fc_1_normaliz (None, 461) 1844 _________________________________________________________________ h11_bull_et_al_fc_1_full_dro (None, 461) 0 _________________________________________________________________ h11_bull_et_al_fc_2 (Dense) (None, 437) 201894 _________________________________________________________________ h11_bull_et_al_fc_2_activati (None, 437) 0 _________________________________________________________________ h11_bull_et_al_fc_2_normaliz (None, 437) 1748 _________________________________________________________________ h11_bull_et_al_fc_2_full_dro (None, 437) 0 _________________________________________________________________ h11_bull_et_al_fc_3 (Dense) (None, 929) 406902 _________________________________________________________________ h11_bull_et_al_fc_3_activati (None, 929) 0 _________________________________________________________________ h11_bull_et_al_fc_3_normaliz (None, 929) 3716 _________________________________________________________________ h11_bull_et_al_fc_3_full_dro (None, 929) 0 _________________________________________________________________ h11_bull_et_al_fc_4 (Dense) (None, 404) 375720 _________________________________________________________________ h11_bull_et_al_fc_4_activati (None, 404) 0 _________________________________________________________________ h11_bull_et_al_fc_4_normaliz (None, 404) 1616 _________________________________________________________________ h11_bull_et_al_fc_4_full_dro (None, 404) 0 _________________________________________________________________ h11_bull_et_al_output (Dense (None, 1) 405 ================================================================= Total params: 1,560,202 Trainable params: 1,553,988 Non-trainable params: 6,214 _________________________________________________________________
2020-05-09 21:18:58,955: DEBUG ==> Base model has been trained. 2020-05-09 21:19:00,370: DEBUG ==> History of h11_bull_et_al has been saved to ./output/original/h11_bull_et_al.json. 2020-05-09 21:19:01,004: DEBUG ==> History of h11_bull_et_al has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_bull_et_al on the validation set: 77.325% Accuracy of h11_bull_et_al on the test set: 77.481%
2020-05-09 21:19:03,547: DEBUG ==> Plot saved to ./img/original/h11_bull_et_al.
Improved Analysis and Differential Models¶
We then implement our own models and compare their performance to the base model. We are interested in smaller networks which can give same or better performance or network comparable in size but which can give better overall performance.
In [17]:
h11_deep_model = cnn_model(input_shape=input_shape,
model_name='h11_large_kernel_same_conv_fc',
learning_rate=1.0e-3,
conv_layers=[80, 40, 20],
conv_paddings=['same']*3,
conv_kernels=[(5,5)]*3,
conv_alpha=0.0,
fc_layers=[1000, 1000, 100],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_deep_model_no_fc = cnn_model(input_shape=input_shape,
model_name='h11_large_kernel_same_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[180, 100, 40, 20],
conv_paddings=['same']*4,
conv_kernels=[(5,5)]*4,
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_model_padding = cnn_model(input_shape=input_shape,
model_name='h11_valid_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[180, 100, 75, 40, 20],
conv_paddings=['valid']*5,
conv_kernels=[(4,5), (4,5), (3,4), (3,3), (2,2)],
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_small_kernel_deep_model = cnn_model(input_shape=input_shape,
model_name='h11_small_kernel_same_conv_fc',
learning_rate=1.0e-3,
conv_layers=[80, 40, 20],
conv_paddings=['same']*3,
conv_kernels=[(3,3), (3,3), (2,2)],
conv_alpha=0.0,
fc_layers=[1000, 1000, 100],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_small_kernel_deep_model_padding = cnn_model(input_shape=input_shape,
model_name='h11_small_kernel_valid_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[350, 300, 250, 200, 150, 100, 50],
conv_paddings=['valid']*7,
conv_kernels=[(3,4), (3,4), (3,4), (3,3), (2,2), (2,2), (2,2)],
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_inception_model_padding = scan_inception_model(input_shape=input_shape,
model_name='h11_inception_valid_conv_fc',
learning_rate=1.0e-2,
conv_layers=[60],
conv_padding='valid',
conv_alpha=0.0,
fc_layers=[500, 100],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h11_deep_inception_model = scan_inception_model(input_shape=input_shape,
model_name='h11_inception_same_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[32, 64, 32],
conv_padding='same',
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.3,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-4,
l2_reg=1.0e-4
)
# create a list of models to train
models = [h11_deep_inception_model,
h11_inception_model_padding,
h11_small_kernel_deep_model_padding,
h11_small_kernel_deep_model,
h11_deep_model,
h11_deep_model_no_fc,
h11_model_padding
]
# train and display properties for each of them
rounding = np.rint
for model in models:
# print characteristics
model.summary()
# print architecture
model_dot = model_to_dot(model=model, show_shapes=True, show_layer_names=False)
model_dot.write_pdf(path.join(IMG_PATH, model.name + '_model.pdf'))
# train the model
log.debug('Training model {}...'.format(model.name))
model, model_history = cnn_fit(model=model,
X=matrix_train, y=h11_train,
X_val=matrix_val, y_val=h11_val,
batch_size=32,
epochs=5000,
early_stopping=400,
reduce_lr=150,
verbose=0
)
log.debug('Model {} has been trained.'.format(model.name))
# compute the predictions
model_val_predictions = np.reshape(model.predict(matrix_val), (-1,)) #---- reshape to match the labels
model_test_predictions = np.reshape(model.predict(matrix_test), (-1,))
test_results[str(model.name)] = model_test_predictions.tolist()
# save the history of the model
with open(path.join(OUT_PATH, model.name + '.json'), 'w') as f:
json.dump(str(model_history), f)
log.debug('History of {} has been saved to {}.'.format(model.name, path.join(OUT_PATH, model.name + '.json')))
# save results to file
with open(path.join(OUT_PATH, 'convnet_test_results_h11_h21.json'), 'w') as f:
json.dump(test_results, f)
log.debug('History of {} has been saved to {}.'.format(model.name, path.join(OUT_PATH, 'convnet_test_results_h11_h21.json')))
# compute the accuracy and score of the predictions
model_score_val = Score(y_true=h11_val, y_pred=model_val_predictions, rounding=rounding)
model_score_test = Score(y_true=h11_test, y_pred=model_test_predictions, rounding=rounding)
print('Accuracy of {} on the validation set: {:.3f}%'.format(model.name, model_score_val.accuracy()*100))
print('Accuracy of {} on the test set: {:.3f}%'.format(model.name, model_score_test.accuracy()*100))
# plot the distribution of the predictions and the error difference
plot = Plot(rows=1, columns=5)
plot.scatter2D(np.array(list(get_counts(num_cp_val, h11_val))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='real values',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_val, rounding(model_val_predictions)))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, h11_test))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='real values',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, rounding(model_test_predictions)))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.hist2D(model_score_val.error(),
axis=2,
title='Distance from the Real Value',
legend='validation set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.hist2D(model_score_test.error(),
axis=2,
title='Distance from the Real Value',
legend='test set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.series2D(data=model_history['loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='training loss',
ylog=True,
binstep=int(np.shape(model_history['loss'])[0] / 10)
)
plot.series2D(data=model_history['val_loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='validation loss',
ylog=True,
binstep=int(np.shape(model_history['val_loss'])[0] / 10)
)
plot.series2D(data=model_history['mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='training MSE',
ylog=True,
binstep=int(np.shape(model_history['mean_squared_error'])[0] / 10)
)
plot.series2D(data=model_history['val_mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='validation MSE',
ylog=True,
binstep=int(np.shape(model_history['val_mean_squared_error'])[0] / 10)
)
plot.save_and_close(path.join(IMG_PATH, model.name))
log.debug('Plot saved to {}.'.format(path.join(IMG_PATH, model.name)))
Model: "h11_inception_same_conv_no_fc"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
h11_inception_same_conv_no_fc_i [(None, 12, 15, 1)] 0
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 416 h11_inception_same_conv_no_fc_inp
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 512 h11_inception_same_conv_no_fc_inp
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 0 h11_inception_same_conv_no_fc_con
h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 256 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 49216 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 61504 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 128) 0 h11_inception_same_conv_no_fc_con
h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 128) 512 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 49184 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 61472 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 0 h11_inception_same_conv_no_fc_con
h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_c (None, 12, 15, 64) 256 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_d (None, 12, 15, 64) 0 h11_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_f (None, 11520) 0 h11_inception_same_conv_no_fc_dro
__________________________________________________________________________________________________
h11_inception_same_conv_no_fc_o (None, 1) 11521 h11_inception_same_conv_no_fc_fla
==================================================================================================
Total params: 234,849
Trainable params: 234,337
Non-trainable params: 512
__________________________________________________________________________________________________
2020-05-09 21:19:12,954: DEBUG ==> Training model h11_inception_same_conv_no_fc... 2020-05-10 00:12:20,488: DEBUG ==> Model h11_inception_same_conv_no_fc has been trained. 2020-05-10 00:12:22,902: DEBUG ==> History of h11_inception_same_conv_no_fc has been saved to ./output/original/h11_inception_same_conv_no_fc.json. 2020-05-10 00:12:26,063: DEBUG ==> History of h11_inception_same_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_inception_same_conv_no_fc on the validation set: 99.236% Accuracy of h11_inception_same_conv_no_fc on the test set: 99.364%
2020-05-10 00:12:34,179: DEBUG ==> Plot saved to ./img/original/h11_inception_same_conv_no_fc.
Model: "h11_inception_valid_conv_fc"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
h11_inception_valid_conv_fc_inp [(None, 12, 15, 1)] 0
__________________________________________________________________________________________________
zero_padding2d (ZeroPadding2D) (None, 15, 15, 1) 0 h11_inception_valid_conv_fc_input
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 15, 1, 60) 960 zero_padding2d[0][0]
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 1, 15, 60) 960 zero_padding2d[0][0]
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 15, 1, 60) 0 h11_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 1, 15, 60) 0 h11_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
tf_op_layer_transpose (TensorFl [(None, 1, 15, 60)] 0 h11_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 1, 15, 120) 0 h11_inception_valid_conv_fc_conv2
tf_op_layer_transpose[0][0]
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_con (None, 1, 15, 120) 480 h11_inception_valid_conv_fc_conca
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_dro (None, 1, 15, 120) 0 h11_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fla (None, 1800) 0 h11_inception_valid_conv_fc_dropo
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 500) 900500 h11_inception_valid_conv_fc_flatt
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 500) 0 h11_inception_valid_conv_fc_fc_0[
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 500) 2000 h11_inception_valid_conv_fc_fc_0_
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 100) 50100 h11_inception_valid_conv_fc_fc_0_
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 100) 0 h11_inception_valid_conv_fc_fc_1[
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_fc_ (None, 100) 400 h11_inception_valid_conv_fc_fc_1_
__________________________________________________________________________________________________
h11_inception_valid_conv_fc_out (None, 1) 101 h11_inception_valid_conv_fc_fc_1_
==================================================================================================
Total params: 955,501
Trainable params: 954,061
Non-trainable params: 1,440
__________________________________________________________________________________________________
2020-05-10 00:12:34,704: DEBUG ==> Training model h11_inception_valid_conv_fc... 2020-05-10 00:56:04,809: DEBUG ==> Model h11_inception_valid_conv_fc has been trained. 2020-05-10 00:56:05,708: DEBUG ==> History of h11_inception_valid_conv_fc has been saved to ./output/original/h11_inception_valid_conv_fc.json. 2020-05-10 00:56:07,666: DEBUG ==> History of h11_inception_valid_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_inception_valid_conv_fc on the validation set: 81.274% Accuracy of h11_inception_valid_conv_fc on the test set: 81.298%
2020-05-10 00:56:13,878: DEBUG ==> Plot saved to ./img/original/h11_inception_valid_conv_fc.
Model: "h11_small_kernel_valid_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_small_kernel_valid_conv_ [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 10, 12, 350) 4550 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 10, 12, 350) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 10, 12, 350) 1400 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 8, 9, 300) 1260300 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 8, 9, 300) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 8, 9, 300) 1200 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 6, 6, 250) 900250 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 6, 6, 250) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 6, 6, 250) 1000 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 4, 4, 200) 450200 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 4, 4, 200) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 4, 4, 200) 800 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 3, 3, 150) 120150 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 3, 3, 150) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 3, 3, 150) 600 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 2, 2, 100) 60100 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 2, 2, 100) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 2, 2, 100) 400 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 1, 1, 50) 20050 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 1, 1, 50) 200 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 50) 0 _________________________________________________________________ h11_small_kernel_valid_conv_ (None, 1) 51 ================================================================= Total params: 2,821,251 Trainable params: 2,818,451 Non-trainable params: 2,800 _________________________________________________________________
2020-05-10 00:56:14,767: DEBUG ==> Training model h11_small_kernel_valid_conv_no_fc... 2020-05-10 04:12:32,987: DEBUG ==> Model h11_small_kernel_valid_conv_no_fc has been trained. 2020-05-10 04:12:35,605: DEBUG ==> History of h11_small_kernel_valid_conv_no_fc has been saved to ./output/original/h11_small_kernel_valid_conv_no_fc.json. 2020-05-10 04:12:37,465: DEBUG ==> History of h11_small_kernel_valid_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_small_kernel_valid_conv_no_fc on the validation set: 89.172% Accuracy of h11_small_kernel_valid_conv_no_fc on the test set: 90.331%
2020-05-10 04:12:43,559: DEBUG ==> Plot saved to ./img/original/h11_small_kernel_valid_conv_no_fc.
Model: "h11_small_kernel_same_conv_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_small_kernel_same_conv_f [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 80) 800 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 80) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 80) 320 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 40) 28840 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 40) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 40) 160 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 20) 3220 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 20) 80 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 3600) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 3601000 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 4000 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 1001000 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1000) 4000 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 100) 100100 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 100) 0 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 100) 400 _________________________________________________________________ h11_small_kernel_same_conv_f (None, 1) 101 ================================================================= Total params: 4,744,021 Trainable params: 4,739,541 Non-trainable params: 4,480 _________________________________________________________________
2020-05-10 04:12:44,182: DEBUG ==> Training model h11_small_kernel_same_conv_fc... 2020-05-10 05:32:12,056: DEBUG ==> Model h11_small_kernel_same_conv_fc has been trained. 2020-05-10 05:32:12,861: DEBUG ==> History of h11_small_kernel_same_conv_fc has been saved to ./output/original/h11_small_kernel_same_conv_fc.json. 2020-05-10 05:32:13,968: DEBUG ==> History of h11_small_kernel_same_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_small_kernel_same_conv_fc on the validation set: 90.573% Accuracy of h11_small_kernel_same_conv_fc on the test set: 90.458%
2020-05-10 05:32:17,000: DEBUG ==> Plot saved to ./img/original/h11_small_kernel_same_conv_fc.
Model: "h11_large_kernel_same_conv_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_large_kernel_same_conv_f [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 80) 2080 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 80) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 80) 320 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 40) 80040 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 40) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 40) 160 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 20) 20020 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 20) 80 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 3600) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 3601000 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 4000 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 1001000 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1000) 4000 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 100) 100100 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 100) 0 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 100) 400 _________________________________________________________________ h11_large_kernel_same_conv_f (None, 1) 101 ================================================================= Total params: 4,813,301 Trainable params: 4,808,821 Non-trainable params: 4,480 _________________________________________________________________
2020-05-10 05:32:17,255: DEBUG ==> Training model h11_large_kernel_same_conv_fc... 2020-05-10 07:32:48,180: DEBUG ==> Model h11_large_kernel_same_conv_fc has been trained. 2020-05-10 07:32:49,043: DEBUG ==> History of h11_large_kernel_same_conv_fc has been saved to ./output/original/h11_large_kernel_same_conv_fc.json. 2020-05-10 07:32:50,100: DEBUG ==> History of h11_large_kernel_same_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_large_kernel_same_conv_fc on the validation set: 93.376% Accuracy of h11_large_kernel_same_conv_fc on the test set: 94.656%
2020-05-10 07:32:53,556: DEBUG ==> Plot saved to ./img/original/h11_large_kernel_same_conv_fc.
Model: "h11_large_kernel_same_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_large_kernel_same_conv_n [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 180) 4680 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 180) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 180) 720 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 100) 450100 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 100) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 100) 400 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 40) 100040 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 40) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 40) 160 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 20) 20020 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 20) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 20) 80 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 12, 15, 20) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 3600) 0 _________________________________________________________________ h11_large_kernel_same_conv_n (None, 1) 3601 ================================================================= Total params: 579,801 Trainable params: 579,121 Non-trainable params: 680 _________________________________________________________________
2020-05-10 07:32:53,862: DEBUG ==> Training model h11_large_kernel_same_conv_no_fc... 2020-05-10 09:04:57,375: DEBUG ==> Model h11_large_kernel_same_conv_no_fc has been trained. 2020-05-10 09:04:58,397: DEBUG ==> History of h11_large_kernel_same_conv_no_fc has been saved to ./output/original/h11_large_kernel_same_conv_no_fc.json. 2020-05-10 09:04:59,431: DEBUG ==> History of h11_large_kernel_same_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_large_kernel_same_conv_no_fc on the validation set: 93.503% Accuracy of h11_large_kernel_same_conv_no_fc on the test set: 94.402%
2020-05-10 09:05:02,260: DEBUG ==> Plot saved to ./img/original/h11_large_kernel_same_conv_no_fc. 2020-05-10 09:05:02,516: DEBUG ==> Training model h11_valid_conv_no_fc...
Model: "h11_valid_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h11_valid_conv_no_fc_input ( [(None, 12, 15, 1)] 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 9, 11, 180) 3780 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 9, 11, 180) 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 9, 11, 180) 720 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 6, 7, 100) 360100 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 6, 7, 100) 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 6, 7, 100) 400 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 4, 4, 75) 90075 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 4, 4, 75) 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 4, 4, 75) 300 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 2, 2, 40) 27040 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 2, 2, 40) 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 2, 2, 40) 160 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 1, 1, 20) 3220 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 1, 1, 20) 0 _________________________________________________________________ h11_valid_conv_no_fc_conv2d_ (None, 1, 1, 20) 80 _________________________________________________________________ h11_valid_conv_no_fc_dropout (None, 1, 1, 20) 0 _________________________________________________________________ h11_valid_conv_no_fc_flatten (None, 20) 0 _________________________________________________________________ h11_valid_conv_no_fc_output (None, 1) 21 ================================================================= Total params: 485,896 Trainable params: 485,066 Non-trainable params: 830 _________________________________________________________________
2020-05-10 10:16:11,728: DEBUG ==> Model h11_valid_conv_no_fc has been trained. 2020-05-10 10:16:12,362: DEBUG ==> History of h11_valid_conv_no_fc has been saved to ./output/original/h11_valid_conv_no_fc.json. 2020-05-10 10:16:13,329: DEBUG ==> History of h11_valid_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h11_valid_conv_no_fc on the validation set: 86.369% Accuracy of h11_valid_conv_no_fc on the test set: 87.277%
2020-05-10 10:16:16,380: DEBUG ==> Plot saved to ./img/original/h11_valid_conv_no_fc.
As announced, this is the only case already considered in the reference paper. We see that the accuracy of the fully connected model can be reached and outperformed by simpler (smaller) convolutional networks. Surprisingly scanning over rows and columns of the matrix and then combining the results leads to a large improvement of the accuracy, reaching almost 100% on the test set.
Analysis for h21¶
We can then follow the same line of ideas and implement models of neural networks to predict h21. In general they will have to be deeper with respect to the previous models since the dependences on the features are more complicated in this case.
In [18]:
h21_deep_model = cnn_model(input_shape=input_shape,
model_name='h21_large_kernel_same_conv_fc',
learning_rate=1.0e-3,
conv_layers=[250, 150, 100, 20],
conv_paddings=['same']*4,
conv_kernels=[(6,6)]*4,
conv_alpha=0.0,
fc_layers=[1500, 1500, 100],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_deep_model_no_fc = cnn_model(input_shape=input_shape,
model_name='h21_large_kernel_same_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[250, 150, 100, 50],
conv_paddings=['same']*4,
conv_kernels=[(6,6)]*4,
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_model_padding = cnn_model(input_shape=input_shape,
model_name='h21_large_kernel_valid_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[360, 180, 100, 50],
conv_paddings=['valid']*4,
conv_kernels=[(5,6), (5,6), (3,4), (2,2)],
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_small_kernel_deep_model = cnn_model(input_shape=input_shape,
model_name='h21_small_kernel_same_conv_fc',
learning_rate=1.0e-3,
conv_layers=[250, 200, 150, 100, 50, 10],
conv_paddings=['same']*6,
conv_kernels=[(3,3)]*6,
conv_alpha=0.0,
fc_layers=[800, 800, 100],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_small_kernel_deep_model_padding = cnn_model(input_shape=input_shape,
model_name='h21_small_kernel_valid_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[350, 300, 250, 200, 150, 100, 50],
conv_paddings=['valid']*7,
conv_kernels=[(3,4), (3,4), (3,4), (3,3), (2,2), (2,2), (2,2)],
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.4,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_inception_model_padding = scan_inception_model(input_shape=input_shape,
model_name='h21_inception_valid_conv_fc',
learning_rate=1.0e-2,
conv_layers=[120],
conv_padding='valid',
conv_alpha=0.0,
fc_layers=[500, 100],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-5,
l2_reg=1.0e-5
)
h21_deep_inception_model = scan_inception_model(input_shape=input_shape,
model_name='h21_inception_same_conv_no_fc',
learning_rate=1.0e-3,
conv_layers=[128, 128, 32],
conv_padding='same',
conv_alpha=0.0,
fc_layers=[],
fc_alpha=0.0,
dropout=0.2,
full_dropout=0.0,
normalization=0.99,
last_relu=True,
l1_reg=1.0e-4,
l2_reg=1.0e-4
)
# create a list of models to train
models = [h21_deep_inception_model,
h21_inception_model_padding,
h21_deep_model,
h21_deep_model_no_fc,
h21_model_padding,
h21_small_kernel_deep_model,
h21_small_kernel_deep_model_padding
]
# train and display properties for each of them
rounding = np.rint
for model in models:
# print characteristics
model.summary()
# print architecture
model_dot = model_to_dot(model=model, show_shapes=True, show_layer_names=False)
model_dot.write_pdf(path.join(IMG_PATH, model.name + '_model.pdf'))
# train the model
log.debug('Training model {}...'.format(model.name))
model, model_history = cnn_fit(model=model,
X=matrix_train, y=h21_train,
X_val=matrix_val, y_val=h21_val,
batch_size=32,
epochs=5000,
early_stopping=400,
reduce_lr=125,
verbose=0
)
log.debug('Model {} has been trained.'.format(model.name))
# compute the predictions
model_val_predictions = np.reshape(model.predict(matrix_val), (-1,)) #---- reshape to match the labels
model_test_predictions = np.reshape(model.predict(matrix_test), (-1,))
test_results[str(model.name)] = model_test_predictions.tolist()
# save the history of the model
with open(path.join(OUT_PATH, model.name + '.json'), 'w') as f:
json.dump(str(model_history), f)
log.debug('History of {} has been saved to {}.'.format(model.name, path.join(OUT_PATH, model.name + '.json')))
# save results to file
with open(path.join(OUT_PATH, 'convnet_test_results_h11_h21.json'), 'w') as f:
json.dump(test_results, f)
log.debug('History of {} has been saved to {}.'.format(model.name, path.join(OUT_PATH, 'convnet_test_results_h11_h21.json')))
# compute the accuracy and score of the predictions
model_score_val = Score(y_true=h21_val, y_pred=model_val_predictions, rounding=rounding)
model_score_test = Score(y_true=h21_test, y_pred=model_test_predictions, rounding=rounding)
print('Accuracy of {} on the validation set: {:.3f}%'.format(model.name, model_score_val.accuracy()*100))
print('Accuracy of {} on the test set: {:.3f}%'.format(model.name, model_score_test.accuracy()*100))
# plot the distribution of the predictions and the error difference
plot = Plot(rows=1, columns=5)
plot.scatter2D(np.array(list(get_counts(num_cp_val, h21_val))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='real values',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_val, rounding(model_val_predictions)))).T,
axis=0,
title='Predictions for the Validation Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (validation set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, h21_test))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='real values',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.scatter2D(np.array(list(get_counts(num_cp_test, rounding(model_test_predictions)))).T,
axis=1,
title='Predictions for the Test Set for $h_{11}$',
legend='predictions',
xlabel='num_cp (test set)',
ylabel='$h_{11}$',
colour=False,
alpha=0.65
)
plot.hist2D(model_score_val.error(),
axis=2,
title='Distance from the Real Value',
legend='validation set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.hist2D(model_score_test.error(),
axis=2,
title='Distance from the Real Value',
legend='test set',
xlabel='error difference',
ylabel='#',
binstep=1
)
plot.series2D(data=model_history['loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='training loss',
ylog=True,
binstep=int(np.shape(model_history['loss'])[0] / 10)
)
plot.series2D(data=model_history['val_loss'],
axis=3,
title='Loss Function',
xlabel='epoch',
legend='validation loss',
ylog=True,
binstep=int(np.shape(model_history['val_loss'])[0] / 10)
)
plot.series2D(data=model_history['mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='training MSE',
ylog=True,
binstep=int(np.shape(model_history['mean_squared_error'])[0] / 10)
)
plot.series2D(data=model_history['val_mean_squared_error'],
axis=4,
title='Metric Function',
xlabel='epoch',
legend='validation MSE',
ylog=True,
binstep=int(np.shape(model_history['val_mean_squared_error'])[0] / 10)
)
plot.save_and_close(path.join(IMG_PATH, model.name))
log.debug('Plot saved to {}.'.format(path.join(IMG_PATH, model.name)))
Model: "h21_inception_same_conv_no_fc"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
h21_inception_same_conv_no_fc_i [(None, 12, 15, 1)] 0
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 1664 h21_inception_same_conv_no_fc_inp
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 2048 h21_inception_same_conv_no_fc_inp
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 256) 0 h21_inception_same_conv_no_fc_con
h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 256) 1024 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 393344 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 491648 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 128) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 256) 0 h21_inception_same_conv_no_fc_con
h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 256) 1024 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 32) 98336 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 32) 122912 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 32) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 64) 0 h21_inception_same_conv_no_fc_con
h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_c (None, 12, 15, 64) 256 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_d (None, 12, 15, 64) 0 h21_inception_same_conv_no_fc_con
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_f (None, 11520) 0 h21_inception_same_conv_no_fc_dro
__________________________________________________________________________________________________
h21_inception_same_conv_no_fc_o (None, 1) 11521 h21_inception_same_conv_no_fc_fla
==================================================================================================
Total params: 1,123,777
Trainable params: 1,122,625
Non-trainable params: 1,152
__________________________________________________________________________________________________
2020-05-10 10:16:22,569: DEBUG ==> Training model h21_inception_same_conv_no_fc... 2020-05-10 14:08:07,556: DEBUG ==> Model h21_inception_same_conv_no_fc has been trained. 2020-05-10 14:08:09,280: DEBUG ==> History of h21_inception_same_conv_no_fc has been saved to ./output/original/h21_inception_same_conv_no_fc.json. 2020-05-10 14:08:10,318: DEBUG ==> History of h21_inception_same_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_inception_same_conv_no_fc on the validation set: 45.350% Accuracy of h21_inception_same_conv_no_fc on the test set: 47.583%
2020-05-10 14:08:13,870: DEBUG ==> Plot saved to ./img/original/h21_inception_same_conv_no_fc.
Model: "h21_inception_valid_conv_fc"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
h21_inception_valid_conv_fc_inp [(None, 12, 15, 1)] 0
__________________________________________________________________________________________________
zero_padding2d (ZeroPadding2D) (None, 15, 15, 1) 0 h21_inception_valid_conv_fc_input
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 15, 1, 120) 1920 zero_padding2d[0][0]
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 1, 15, 120) 1920 zero_padding2d[0][0]
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 15, 1, 120) 0 h21_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 1, 15, 120) 0 h21_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
tf_op_layer_transpose (TensorFl [(None, 1, 15, 120)] 0 h21_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 1, 15, 240) 0 h21_inception_valid_conv_fc_conv2
tf_op_layer_transpose[0][0]
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_con (None, 1, 15, 240) 960 h21_inception_valid_conv_fc_conca
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_dro (None, 1, 15, 240) 0 h21_inception_valid_conv_fc_conv2
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fla (None, 3600) 0 h21_inception_valid_conv_fc_dropo
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 500) 1800500 h21_inception_valid_conv_fc_flatt
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 500) 0 h21_inception_valid_conv_fc_fc_0[
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 500) 2000 h21_inception_valid_conv_fc_fc_0_
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 100) 50100 h21_inception_valid_conv_fc_fc_0_
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 100) 0 h21_inception_valid_conv_fc_fc_1[
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_fc_ (None, 100) 400 h21_inception_valid_conv_fc_fc_1_
__________________________________________________________________________________________________
h21_inception_valid_conv_fc_out (None, 1) 101 h21_inception_valid_conv_fc_fc_1_
==================================================================================================
Total params: 1,857,901
Trainable params: 1,856,221
Non-trainable params: 1,680
__________________________________________________________________________________________________
2020-05-10 14:08:14,155: DEBUG ==> Training model h21_inception_valid_conv_fc... 2020-05-10 15:19:56,855: DEBUG ==> Model h21_inception_valid_conv_fc has been trained. 2020-05-10 15:19:58,573: DEBUG ==> History of h21_inception_valid_conv_fc has been saved to ./output/original/h21_inception_valid_conv_fc.json. 2020-05-10 15:19:59,317: DEBUG ==> History of h21_inception_valid_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_inception_valid_conv_fc on the validation set: 26.879% Accuracy of h21_inception_valid_conv_fc on the test set: 29.008%
2020-05-10 15:20:02,922: DEBUG ==> Plot saved to ./img/original/h21_inception_valid_conv_fc.
Model: "h21_large_kernel_same_conv_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h21_large_kernel_same_conv_f [(None, 12, 15, 1)] 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 250) 9250 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 250) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 250) 1000 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 150) 1350150 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 150) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 150) 600 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 100) 540100 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 100) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 100) 400 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 20) 72020 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 20) 80 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 12, 15, 20) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 3600) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 5401500 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 6000 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 2251500 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1500) 6000 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 100) 150100 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 100) 0 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 100) 400 _________________________________________________________________ h21_large_kernel_same_conv_f (None, 1) 101 ================================================================= Total params: 9,789,201 Trainable params: 9,781,961 Non-trainable params: 7,240 _________________________________________________________________
2020-05-10 15:20:03,260: DEBUG ==> Training model h21_large_kernel_same_conv_fc... 2020-05-10 21:10:19,813: DEBUG ==> Model h21_large_kernel_same_conv_fc has been trained. 2020-05-10 21:10:22,414: DEBUG ==> History of h21_large_kernel_same_conv_fc has been saved to ./output/original/h21_large_kernel_same_conv_fc.json. 2020-05-10 21:10:22,971: DEBUG ==> History of h21_large_kernel_same_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_large_kernel_same_conv_fc on the validation set: 32.739% Accuracy of h21_large_kernel_same_conv_fc on the test set: 33.461%
2020-05-10 21:10:25,482: DEBUG ==> Plot saved to ./img/original/h21_large_kernel_same_conv_fc.
Model: "h21_large_kernel_same_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h21_large_kernel_same_conv_n [(None, 12, 15, 1)] 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 250) 9250 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 250) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 250) 1000 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 150) 1350150 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 150) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 150) 600 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 100) 540100 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 100) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 100) 400 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 50) 180050 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 50) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 50) 200 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 12, 15, 50) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 9000) 0 _________________________________________________________________ h21_large_kernel_same_conv_n (None, 1) 9001 ================================================================= Total params: 2,090,751 Trainable params: 2,089,651 Non-trainable params: 1,100 _________________________________________________________________
2020-05-10 21:10:25,704: DEBUG ==> Training model h21_large_kernel_same_conv_no_fc... 2020-05-11 03:18:10,393: DEBUG ==> Model h21_large_kernel_same_conv_no_fc has been trained. 2020-05-11 03:18:12,824: DEBUG ==> History of h21_large_kernel_same_conv_no_fc has been saved to ./output/original/h21_large_kernel_same_conv_no_fc.json. 2020-05-11 03:18:13,862: DEBUG ==> History of h21_large_kernel_same_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_large_kernel_same_conv_no_fc on the validation set: 35.924% Accuracy of h21_large_kernel_same_conv_no_fc on the test set: 40.458%
2020-05-11 03:18:17,357: DEBUG ==> Plot saved to ./img/original/h21_large_kernel_same_conv_no_fc.
Model: "h21_large_kernel_valid_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h21_large_kernel_valid_conv_ [(None, 12, 15, 1)] 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 8, 10, 360) 11160 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 8, 10, 360) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 8, 10, 360) 1440 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 4, 5, 180) 1944180 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 4, 5, 180) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 4, 5, 180) 720 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 2, 2, 100) 216100 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 2, 2, 100) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 2, 2, 100) 400 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 1, 1, 50) 20050 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 1, 1, 50) 200 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 50) 0 _________________________________________________________________ h21_large_kernel_valid_conv_ (None, 1) 51 ================================================================= Total params: 2,194,301 Trainable params: 2,192,921 Non-trainable params: 1,380 _________________________________________________________________
2020-05-11 03:18:17,662: DEBUG ==> Training model h21_large_kernel_valid_conv_no_fc... 2020-05-11 05:39:48,703: DEBUG ==> Model h21_large_kernel_valid_conv_no_fc has been trained. 2020-05-11 05:39:49,761: DEBUG ==> History of h21_large_kernel_valid_conv_no_fc has been saved to ./output/original/h21_large_kernel_valid_conv_no_fc.json. 2020-05-11 05:39:50,780: DEBUG ==> History of h21_large_kernel_valid_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_large_kernel_valid_conv_no_fc on the validation set: 29.554% Accuracy of h21_large_kernel_valid_conv_no_fc on the test set: 27.354%
2020-05-11 05:39:53,998: DEBUG ==> Plot saved to ./img/original/h21_large_kernel_valid_conv_no_fc.
Model: "h21_small_kernel_same_conv_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h21_small_kernel_same_conv_f [(None, 12, 15, 1)] 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 250) 2500 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 250) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 250) 1000 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 200) 450200 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 200) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 200) 800 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 150) 270150 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 150) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 150) 600 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 100) 135100 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 100) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 100) 400 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 50) 45050 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 50) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 50) 200 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 10) 4510 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 10) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 10) 40 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 12, 15, 10) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 1800) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 1440800 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 3200 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 640800 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 800) 3200 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 100) 80100 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 100) 0 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 100) 400 _________________________________________________________________ h21_small_kernel_same_conv_f (None, 1) 101 ================================================================= Total params: 3,079,151 Trainable params: 3,074,231 Non-trainable params: 4,920 _________________________________________________________________
2020-05-11 05:39:54,297: DEBUG ==> Training model h21_small_kernel_same_conv_fc... 2020-05-11 07:46:32,883: DEBUG ==> Model h21_small_kernel_same_conv_fc has been trained. 2020-05-11 07:46:34,588: DEBUG ==> History of h21_small_kernel_same_conv_fc has been saved to ./output/original/h21_small_kernel_same_conv_fc.json. 2020-05-11 07:46:35,703: DEBUG ==> History of h21_small_kernel_same_conv_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_small_kernel_same_conv_fc on the validation set: 27.898% Accuracy of h21_small_kernel_same_conv_fc on the test set: 29.517%
2020-05-11 07:46:39,101: DEBUG ==> Plot saved to ./img/original/h21_small_kernel_same_conv_fc.
Model: "h21_small_kernel_valid_conv_no_fc" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= h21_small_kernel_valid_conv_ [(None, 12, 15, 1)] 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 10, 12, 350) 4550 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 10, 12, 350) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 10, 12, 350) 1400 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 8, 9, 300) 1260300 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 8, 9, 300) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 8, 9, 300) 1200 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 6, 6, 250) 900250 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 6, 6, 250) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 6, 6, 250) 1000 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 4, 4, 200) 450200 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 4, 4, 200) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 4, 4, 200) 800 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 3, 3, 150) 120150 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 3, 3, 150) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 3, 3, 150) 600 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 2, 2, 100) 60100 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 2, 2, 100) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 2, 2, 100) 400 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 1, 1, 50) 20050 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 1, 1, 50) 200 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 1, 1, 50) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 50) 0 _________________________________________________________________ h21_small_kernel_valid_conv_ (None, 1) 51 ================================================================= Total params: 2,821,251 Trainable params: 2,818,451 Non-trainable params: 2,800 _________________________________________________________________
2020-05-11 07:46:39,462: DEBUG ==> Training model h21_small_kernel_valid_conv_no_fc... 2020-05-11 12:54:53,903: DEBUG ==> Model h21_small_kernel_valid_conv_no_fc has been trained. 2020-05-11 12:54:55,463: DEBUG ==> History of h21_small_kernel_valid_conv_no_fc has been saved to ./output/original/h21_small_kernel_valid_conv_no_fc.json. 2020-05-11 12:54:56,636: DEBUG ==> History of h21_small_kernel_valid_conv_no_fc has been saved to ./output/original/convnet_test_results_h11_h21.json.
Accuracy of h21_small_kernel_valid_conv_no_fc on the validation set: 28.662% Accuracy of h21_small_kernel_valid_conv_no_fc on the test set: 27.481%
2020-05-11 12:55:00,180: DEBUG ==> Plot saved to ./img/original/h21_small_kernel_valid_conv_no_fc.