Exercise 5.3: Neural Networks in Keras - Solution

In [1]:
import numpy as np
import matplotlib.pyplot as plt

# See https://keras.io/
# for extennsive documentation
import tensorflow as tf
from tensorflow import keras

from keras.models import Sequential
from keras.layers import Dense

Let us visit the problem of wine quality prediction previously encountered in Exercises 3.2 and 4.1 one final time. After linear regression and a self-made network, we can now explore the comfort provided by the Keras library.

In [2]:
# The code snippet below is responsible for downloading the dataset to
# Google. You can directly download the file using the link
# if you work with a local anaconda setup
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv
/bin/sh: wget: command not found
In [3]:
# load all examples from the file
data = np.genfromtxt('winequality-white.csv',delimiter=";",skip_header=1)

print("data:", data.shape)

# Prepare for proper training
np.random.shuffle(data) # randomly sort examples

# take the first 3000 examples for training
X_train = data[:3000,:11] # all features except last column
y_train = data[:3000,11]  # quality column

# and the remaining examples for testing
X_test = data[3000:,:11] # all features except last column
y_test = data[3000:,11] # quality column

print("First example:")
print("Features:", X_train[0])
print("Quality:", y_train[0])
data: (4898, 12)
First example:
Features: [7.0000e+00 3.1000e-01 3.5000e-01 1.6000e+00 6.3000e-02 1.3000e+01
 1.1900e+02 9.9184e-01 3.2200e+00 5.0000e-01 1.0700e+01]
Quality: 5.0

Below is the simple network from exercise 4.1 implemented using Keras. In addition to the network we define the loss function and optimiser.

In [24]:
# See: https://keras.io/api/models/sequential/ and 
# https://keras.io/api/layers/core_layers/dense/
# We can use the Sequential class to very easiliy
# build a simple architecture
model = Sequential()
# 11 inputs, 20 outputs, relu
model.add(Dense(20, input_dim=11, activation='relu')) 
model.add(Dense(20, activation='relu')) 
model.add(Dense(20, activation='relu')) 
# 20 inputs (automatically detected by Keras), 1 output, linear activation
model.add(Dense(1, activation='linear'))


# Set loss function and optimiser algorithm
# Remove comments from of these versions:

# Initial:
#model.compile(loss='mse',  # mean squared error
#              optimizer='sgd'# stochastic gradient descent
#             ) 
#
# sgd with mometum
# instead of passing a string, we can explicitely construct the optimizer object
# this gives us more control over its properties
#opt = keras.optimizers.SGD(momentum=0.8) 
#model.compile(loss='mse',  # mean squared error
#              optimizer='sgd'
#             ) 
#
# Adam:
opt = keras.optimizers.Adam(learning_rate=0.0005) 
model.compile(loss='mse',  # mean squared error
              optimizer=opt
             ) 

Training and evaluation below

The code below trains the network for 5 epochs using the loss function and optimiser defined above. Each example is individually passed to the network

In [25]:
history = model.fit(X_train, y_train, 
                    validation_data=(X_test, y_test),
                    epochs=70, batch_size=10) # changed batch size to 10 
Epoch 1/70
300/300 [==============================] - 1s 2ms/step - loss: 46.4118 - val_loss: 3.1073
Epoch 2/70
300/300 [==============================] - 0s 1ms/step - loss: 2.7097 - val_loss: 2.1891
Epoch 3/70
300/300 [==============================] - 0s 1ms/step - loss: 1.9155 - val_loss: 0.9903
Epoch 4/70
300/300 [==============================] - 0s 1ms/step - loss: 0.8393 - val_loss: 0.7024
Epoch 5/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6665 - val_loss: 0.6581
Epoch 6/70
300/300 [==============================] - 0s 1ms/step - loss: 0.7041 - val_loss: 0.6317
Epoch 7/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6754 - val_loss: 0.6689
Epoch 8/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6605 - val_loss: 0.6133
Epoch 9/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6512 - val_loss: 0.6183
Epoch 10/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6678 - val_loss: 0.6162
Epoch 11/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6542 - val_loss: 0.5944
Epoch 12/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6498 - val_loss: 0.6002
Epoch 13/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6381 - val_loss: 0.5874
Epoch 14/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6452 - val_loss: 0.6921
Epoch 15/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6827 - val_loss: 0.6189
Epoch 16/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5984 - val_loss: 0.7090
Epoch 17/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6073 - val_loss: 0.5954
Epoch 18/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6351 - val_loss: 0.6501
Epoch 19/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6004 - val_loss: 0.5765
Epoch 20/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6296 - val_loss: 0.5682
Epoch 21/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6307 - val_loss: 0.5854
Epoch 22/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6311 - val_loss: 0.6989
Epoch 23/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5797 - val_loss: 0.6385
Epoch 24/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6446 - val_loss: 0.5486
Epoch 25/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6366 - val_loss: 0.5558
Epoch 26/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6019 - val_loss: 0.5446
Epoch 27/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6299 - val_loss: 0.5682
Epoch 28/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5982 - val_loss: 0.5423
Epoch 29/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5921 - val_loss: 0.5433
Epoch 30/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5791 - val_loss: 0.5554
Epoch 31/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5792 - val_loss: 0.6534
Epoch 32/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5904 - val_loss: 0.5771
Epoch 33/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5871 - val_loss: 0.5595
Epoch 34/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5635 - val_loss: 0.5373
Epoch 35/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6022 - val_loss: 0.7051
Epoch 36/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5985 - val_loss: 0.5677
Epoch 37/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5580 - val_loss: 0.5455
Epoch 38/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6386 - val_loss: 0.5383
Epoch 39/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6253 - val_loss: 0.6117
Epoch 40/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5911 - val_loss: 0.5612
Epoch 41/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5735 - val_loss: 0.5583
Epoch 42/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5878 - val_loss: 0.5711
Epoch 43/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5955 - val_loss: 0.5366
Epoch 44/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5770 - val_loss: 0.5332
Epoch 45/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5794 - val_loss: 0.5375
Epoch 46/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6078 - val_loss: 0.5349
Epoch 47/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5853 - val_loss: 0.5300
Epoch 48/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6130 - val_loss: 0.5279
Epoch 49/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5680 - val_loss: 0.7001
Epoch 50/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5703 - val_loss: 0.6029
Epoch 51/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6161 - val_loss: 0.5407
Epoch 52/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5793 - val_loss: 0.5336
Epoch 53/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5859 - val_loss: 0.5460
Epoch 54/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5511 - val_loss: 0.5384
Epoch 55/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5948 - val_loss: 0.5303
Epoch 56/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5912 - val_loss: 0.5345
Epoch 57/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5816 - val_loss: 0.5278
Epoch 58/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5890 - val_loss: 0.5268
Epoch 59/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5852 - val_loss: 0.5261
Epoch 60/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5851 - val_loss: 0.5566
Epoch 61/70
300/300 [==============================] - 0s 1ms/step - loss: 0.6098 - val_loss: 0.5239
Epoch 62/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5671 - val_loss: 0.5541
Epoch 63/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5868 - val_loss: 0.5280
Epoch 64/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5735 - val_loss: 0.5238
Epoch 65/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5701 - val_loss: 0.5417
Epoch 66/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5502 - val_loss: 0.5333
Epoch 67/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5938 - val_loss: 0.5467
Epoch 68/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5668 - val_loss: 0.5323
Epoch 69/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5883 - val_loss: 0.5417
Epoch 70/70
300/300 [==============================] - 0s 1ms/step - loss: 0.5458 - val_loss: 0.5390
In [26]:
# The history object returned by the model training above 
# contains the values of the loss function (the mean-squared-error)
# at different epochs
# We discard the first epoch as the loss value is very high,
# obscuring the rest of the distribution
train_loss = history.history["loss"][1:]
test_loss = history.history["val_loss"][1:]
In [27]:
# Prepare and plot loss over time
plt.plot(train_loss,label="train")
plt.plot(test_loss,label="test")
plt.legend()
plt.xlabel("Epoch-1")
plt.ylabel("Loss")
plt.show()
In [28]:
# After the training:

# Prepare scatter plot
y_pred = model.predict(X_test)[:,0]

print("Correlation coefficient:", np.corrcoef(y_pred,y_test)[0,1])
plt.scatter(y_pred,y_test)
plt.xlabel("Predicted")
plt.ylabel("True")
plt.show()
Correlation coefficient: 0.5500396765061618