Exercise 5.3: Neural Networks in Keras - Solution¶
In [1]:
import numpy as np
import matplotlib.pyplot as plt
# See https://keras.io/
# for extennsive documentation
import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense
Let us visit the problem of wine quality prediction previously encountered in Exercises 3.2 and 4.1 one final time. After linear regression and a self-made network, we can now explore the comfort provided by the Keras library.
In [2]:
# The code snippet below is responsible for downloading the dataset to
# Google. You can directly download the file using the link
# if you work with a local anaconda setup
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv
/bin/sh: wget: command not found
In [3]:
# load all examples from the file
data = np.genfromtxt('winequality-white.csv',delimiter=";",skip_header=1)
print("data:", data.shape)
# Prepare for proper training
np.random.shuffle(data) # randomly sort examples
# take the first 3000 examples for training
X_train = data[:3000,:11] # all features except last column
y_train = data[:3000,11] # quality column
# and the remaining examples for testing
X_test = data[3000:,:11] # all features except last column
y_test = data[3000:,11] # quality column
print("First example:")
print("Features:", X_train[0])
print("Quality:", y_train[0])
data: (4898, 12) First example: Features: [7.0000e+00 3.1000e-01 3.5000e-01 1.6000e+00 6.3000e-02 1.3000e+01 1.1900e+02 9.9184e-01 3.2200e+00 5.0000e-01 1.0700e+01] Quality: 5.0
Below is the simple network from exercise 4.1 implemented using Keras. In addition to the network we define the loss function and optimiser.
In [24]:
# See: https://keras.io/api/models/sequential/ and
# https://keras.io/api/layers/core_layers/dense/
# We can use the Sequential class to very easiliy
# build a simple architecture
model = Sequential()
# 11 inputs, 20 outputs, relu
model.add(Dense(20, input_dim=11, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(20, activation='relu'))
# 20 inputs (automatically detected by Keras), 1 output, linear activation
model.add(Dense(1, activation='linear'))
# Set loss function and optimiser algorithm
# Remove comments from of these versions:
# Initial:
#model.compile(loss='mse', # mean squared error
# optimizer='sgd'# stochastic gradient descent
# )
#
# sgd with mometum
# instead of passing a string, we can explicitely construct the optimizer object
# this gives us more control over its properties
#opt = keras.optimizers.SGD(momentum=0.8)
#model.compile(loss='mse', # mean squared error
# optimizer='sgd'
# )
#
# Adam:
opt = keras.optimizers.Adam(learning_rate=0.0005)
model.compile(loss='mse', # mean squared error
optimizer=opt
)
Training and evaluation below¶
The code below trains the network for 5 epochs using the loss function and optimiser defined above. Each example is individually passed to the network
In [25]:
history = model.fit(X_train, y_train,
validation_data=(X_test, y_test),
epochs=70, batch_size=10) # changed batch size to 10
Epoch 1/70 300/300 [==============================] - 1s 2ms/step - loss: 46.4118 - val_loss: 3.1073 Epoch 2/70 300/300 [==============================] - 0s 1ms/step - loss: 2.7097 - val_loss: 2.1891 Epoch 3/70 300/300 [==============================] - 0s 1ms/step - loss: 1.9155 - val_loss: 0.9903 Epoch 4/70 300/300 [==============================] - 0s 1ms/step - loss: 0.8393 - val_loss: 0.7024 Epoch 5/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6665 - val_loss: 0.6581 Epoch 6/70 300/300 [==============================] - 0s 1ms/step - loss: 0.7041 - val_loss: 0.6317 Epoch 7/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6754 - val_loss: 0.6689 Epoch 8/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6605 - val_loss: 0.6133 Epoch 9/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6512 - val_loss: 0.6183 Epoch 10/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6678 - val_loss: 0.6162 Epoch 11/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6542 - val_loss: 0.5944 Epoch 12/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6498 - val_loss: 0.6002 Epoch 13/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6381 - val_loss: 0.5874 Epoch 14/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6452 - val_loss: 0.6921 Epoch 15/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6827 - val_loss: 0.6189 Epoch 16/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5984 - val_loss: 0.7090 Epoch 17/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6073 - val_loss: 0.5954 Epoch 18/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6351 - val_loss: 0.6501 Epoch 19/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6004 - val_loss: 0.5765 Epoch 20/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6296 - val_loss: 0.5682 Epoch 21/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6307 - val_loss: 0.5854 Epoch 22/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6311 - val_loss: 0.6989 Epoch 23/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5797 - val_loss: 0.6385 Epoch 24/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6446 - val_loss: 0.5486 Epoch 25/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6366 - val_loss: 0.5558 Epoch 26/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6019 - val_loss: 0.5446 Epoch 27/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6299 - val_loss: 0.5682 Epoch 28/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5982 - val_loss: 0.5423 Epoch 29/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5921 - val_loss: 0.5433 Epoch 30/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5791 - val_loss: 0.5554 Epoch 31/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5792 - val_loss: 0.6534 Epoch 32/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5904 - val_loss: 0.5771 Epoch 33/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5871 - val_loss: 0.5595 Epoch 34/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5635 - val_loss: 0.5373 Epoch 35/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6022 - val_loss: 0.7051 Epoch 36/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5985 - val_loss: 0.5677 Epoch 37/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5580 - val_loss: 0.5455 Epoch 38/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6386 - val_loss: 0.5383 Epoch 39/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6253 - val_loss: 0.6117 Epoch 40/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5911 - val_loss: 0.5612 Epoch 41/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5735 - val_loss: 0.5583 Epoch 42/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5878 - val_loss: 0.5711 Epoch 43/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5955 - val_loss: 0.5366 Epoch 44/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5770 - val_loss: 0.5332 Epoch 45/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5794 - val_loss: 0.5375 Epoch 46/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6078 - val_loss: 0.5349 Epoch 47/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5853 - val_loss: 0.5300 Epoch 48/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6130 - val_loss: 0.5279 Epoch 49/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5680 - val_loss: 0.7001 Epoch 50/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5703 - val_loss: 0.6029 Epoch 51/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6161 - val_loss: 0.5407 Epoch 52/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5793 - val_loss: 0.5336 Epoch 53/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5859 - val_loss: 0.5460 Epoch 54/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5511 - val_loss: 0.5384 Epoch 55/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5948 - val_loss: 0.5303 Epoch 56/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5912 - val_loss: 0.5345 Epoch 57/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5816 - val_loss: 0.5278 Epoch 58/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5890 - val_loss: 0.5268 Epoch 59/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5852 - val_loss: 0.5261 Epoch 60/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5851 - val_loss: 0.5566 Epoch 61/70 300/300 [==============================] - 0s 1ms/step - loss: 0.6098 - val_loss: 0.5239 Epoch 62/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5671 - val_loss: 0.5541 Epoch 63/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5868 - val_loss: 0.5280 Epoch 64/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5735 - val_loss: 0.5238 Epoch 65/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5701 - val_loss: 0.5417 Epoch 66/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5502 - val_loss: 0.5333 Epoch 67/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5938 - val_loss: 0.5467 Epoch 68/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5668 - val_loss: 0.5323 Epoch 69/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5883 - val_loss: 0.5417 Epoch 70/70 300/300 [==============================] - 0s 1ms/step - loss: 0.5458 - val_loss: 0.5390
In [26]:
# The history object returned by the model training above
# contains the values of the loss function (the mean-squared-error)
# at different epochs
# We discard the first epoch as the loss value is very high,
# obscuring the rest of the distribution
train_loss = history.history["loss"][1:]
test_loss = history.history["val_loss"][1:]
In [27]:
# Prepare and plot loss over time
plt.plot(train_loss,label="train")
plt.plot(test_loss,label="test")
plt.legend()
plt.xlabel("Epoch-1")
plt.ylabel("Loss")
plt.show()
In [28]:
# After the training:
# Prepare scatter plot
y_pred = model.predict(X_test)[:,0]
print("Correlation coefficient:", np.corrcoef(y_pred,y_test)[0,1])
plt.scatter(y_pred,y_test)
plt.xlabel("Predicted")
plt.ylabel("True")
plt.show()
Correlation coefficient: 0.5500396765061618
