Sketch Recognition using Simple Neural Network (NN)¶
- 🤖 See full list of Machine Learning Experiments on GitHub
- ▶️ Interactive Demo: try this model and other machine learning experiments in action
Experiment overview¶
In this experiment we will build a Multilayer Perceptron (MLP) model using Tensorflow to recognize handwritten sketches by using a quick-draw dataset.
A multilayer perceptron (MLP) is a class of feedforward artificial neural network. An MLP consists of, at least, three layers of nodes: an input layer, a hidden layer and an output layer. Except for the input nodes, each node is a neuron that uses a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training. Its multiple layers and non-linear activation distinguish MLP from a linear perceptron. It can distinguish data that is not linearly separable.
Import dependencies¶
In [325]:
# Selecting Tensorflow version v2 (the command is relevant for Colab only).
# %tensorflow_version 2.x
In [673]:
import tensorflow as tf
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt
import numpy as np
import math
import datetime
import platform
import pathlib
import random
print('Python version:', platform.python_version())
print('Tensorflow version:', tf.__version__)
print('Keras version:', tf.keras.__version__)
In [674]:
cache_dir = 'tmp';
In [675]:
# Create cache folder.
!mkdir tmp
Load dataset¶
In [676]:
# List all available datasets to see how the wikipedia dataset is called.
tfds.list_builders()
Out[676]:
In [677]:
DATASET_NAME = 'quickdraw_bitmap'
dataset, dataset_info = tfds.load(
name=DATASET_NAME,
data_dir=cache_dir,
with_info=True,
split=tfds.Split.TRAIN,
)
Explore dataset¶
In [678]:
print(dataset_info)
In [679]:
image_shape = dataset_info.features['image'].shape
num_classes = dataset_info.features['label'].num_classes
num_examples = dataset_info.splits['train'].num_examples
print('num_examples: ', num_examples)
print('image_shape: ', image_shape)
print('num_classes: ', num_classes)
In [680]:
label_index_to_string = dataset_info.features['label'].int2str
classes = []
for class_index in range(num_classes):
classes.append(label_index_to_string(class_index))
print('classes:\n\n', classes)
In [681]:
print(dataset)
In [682]:
fig = tfds.show_examples(dataset_info, dataset)
In [700]:
def dataset_preview(dataset, image_shape, preview_images_num=100):
num_cells = math.ceil(math.sqrt(preview_images_num))
plt.figure(figsize=(17, 17))
image_size = image_shape[0]
for image_index, example in enumerate(dataset.take(preview_images_num)):
image = example['image']
label = example['label']
class_index = label.numpy()
class_name = label_index_to_string(class_index)
plt.subplot(num_cells, num_cells, image_index + 1)
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(
np.reshape(image, (image_size, image_size)),
cmap=plt.cm.binary
)
plt.xlabel('{} ({})'.format(class_name, class_index))
plt.show()
In [701]:
def dataset_normalized_preview(dataset, image_shape, preview_images_num=100):
num_cells = math.ceil(math.sqrt(preview_images_num))
plt.figure(figsize=(17, 17))
image_size = image_shape[0]
for image_index, example in enumerate(dataset.take(preview_images_num)):
image = example[0]
label = example[1]
class_index = label.numpy()
class_name = label_index_to_string(class_index)
plt.subplot(num_cells, num_cells, image_index + 1)
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(
np.reshape(image, (image_size, image_size)),
cmap=plt.cm.binary
)
plt.xlabel('{} ({})'.format(class_name, class_index))
plt.show()
In [684]:
def dataset_head(ds):
for example in ds.take(1):
image = example['image']
label = example['label']
class_index = label.numpy()
class_name = label_index_to_string(class_index)
print('{} ({})'.format(class_name, class_index), '\n')
print('Image shape: ', image.shape, '\n')
print(np.reshape(image.numpy(), (28, 28)), '\n')
In [685]:
dataset_head(dataset)
In [702]:
dataset_preview(dataset, image_shape)
Normalize dataset¶
In [691]:
def normalize_example(example):
image = example['image']
label = example['label']
image = tf.math.divide(image, 255)
return (image, label)
In [692]:
def augment_example(image, label):
image = tf.image.random_flip_left_right(image)
return (image, label)
In [693]:
dataset_normalized = dataset.map(normalize_example).map(augment_example)
In [694]:
for (image, label) in dataset_normalized.take(1):
class_index = label.numpy()
class_name = label_index_to_string(class_index)
print('{} ({})'.format(class_name, class_index), '\n')
print('Image shape: ', image.shape, '\n')
print(np.reshape(image.numpy(), (28, 28)), '\n')
In [705]:
dataset_normalized_preview(dataset_normalized, image_shape)
Prepare Train/Validation/Test dataset splits¶
In [706]:
# A quick example of how we're going to split the dataset for train/test/validation subsets.
tmp_ds = tf.data.Dataset.range(10)
print('tmp_ds:', list(tmp_ds.as_numpy_iterator()))
tmp_ds_test = tmp_ds.take(2)
print('tmp_ds_test:', list(tmp_ds_test.as_numpy_iterator()))
tmp_ds_val = tmp_ds.skip(2).take(3)
print('tmp_ds_val:', list(tmp_ds_val.as_numpy_iterator()))
tmp_ds_train = tmp_ds.skip(2 + 3)
print('tmp_ds_train:', list(tmp_ds_train.as_numpy_iterator()))
In [707]:
# Dataset split
test_dataset_batches = 1
val_dataset_batches = 1
# Dataset batching and shuffling
shuffle_buffer_size = 20000
batch_size = 20000
prefetch_buffer_batches = 10
# Training
epochs = 40
steps_per_epoch = 200
In [708]:
dataset_batched = dataset_normalized \
.shuffle(
buffer_size=shuffle_buffer_size,
reshuffle_each_iteration=True
) \
.batch(batch_size=batch_size)
In [709]:
# TEST dataset.
dataset_test = dataset_batched \
.take(test_dataset_batches)
# VALIDATION dataset.
dataset_val = dataset_batched \
.skip(test_dataset_batches) \
.take(val_dataset_batches)
# TRAIN dataset.
dataset_train = dataset_batched \
.skip(test_dataset_batches + val_dataset_batches) \
.prefetch(buffer_size=prefetch_buffer_batches) \
.repeat()
In [710]:
for (image_test, label_test) in dataset_test.take(1):
print('label_test.shape: ', label_test.shape)
print('image_test.shape: ', image_test.shape)
print()
for (image_val, label_val) in dataset_val.take(1):
print('label_val.shape: ', label_val.shape)
print('image_val.shape: ', image_val.shape)
print()
for (image_train, label_train) in dataset_train.take(1):
print('label_train.shape: ', label_train.shape)
print('image_train.shape: ', image_train.shape)
Create model¶
In [711]:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Flatten(
input_shape=image_shape
))
model.add(tf.keras.layers.Dense(
units=512,
activation=tf.keras.activations.relu
))
model.add(tf.keras.layers.Dense(
units=512,
activation=tf.keras.activations.relu
))
model.add(tf.keras.layers.Dense(
units=512,
activation=tf.keras.activations.relu
))
model.add(tf.keras.layers.Dense(
units=num_classes,
activation=tf.keras.activations.softmax
))
In [712]:
model.summary()
In [713]:
tf.keras.utils.plot_model(
model,
show_shapes=True,
show_layer_names=True,
)
Out[713]:
In [714]:
adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(
optimizer=adam_optimizer,
loss=tf.keras.losses.sparse_categorical_crossentropy,
metrics=['accuracy']
)
Train model¶
In [715]:
early_stopping_callback = tf.keras.callbacks.EarlyStopping(
patience=5,
monitor='val_accuracy',
restore_best_weights=True,
verbose=1
)
In [716]:
training_history = model.fit(
x=dataset_train,
epochs=epochs,
steps_per_epoch=steps_per_epoch,
validation_data=dataset_val,
callbacks=[
early_stopping_callback
]
)
In [717]:
# Renders the charts for training accuracy and loss.
def render_training_history(training_history):
loss = training_history.history['loss']
val_loss = training_history.history['val_loss']
accuracy = training_history.history['accuracy']
val_accuracy = training_history.history['val_accuracy']
plt.figure(figsize=(14, 4))
plt.subplot(1, 2, 1)
plt.title('Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.plot(loss, label='Training set')
plt.plot(val_loss, label='Test set', linestyle='--')
plt.legend()
plt.grid(linestyle='--', linewidth=1, alpha=0.5)
plt.subplot(1, 2, 2)
plt.title('Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.plot(accuracy, label='Training set')
plt.plot(val_accuracy, label='Test set', linestyle='--')
plt.legend()
plt.grid(linestyle='--', linewidth=1, alpha=0.5)
plt.show()
In [718]:
render_training_history(training_history)
Evaluate model accuracy¶
Training set accuracy¶
In [719]:
%%capture
train_loss, train_accuracy = model.evaluate(dataset_train.take(1))
In [720]:
print('Train loss: ', '{:.2f}'.format(train_loss))
print('Train accuracy: ', '{:.2f}'.format(train_accuracy))
Validation set accuracy¶
In [721]:
%%capture
val_loss, val_accuracy = model.evaluate(dataset_val)
In [722]:
print('Validation loss: ', '{:.2f}'.format(val_loss))
print('Validation accuracy: ', '{:.2f}'.format(val_accuracy))
Test set accuracy¶
In [723]:
%%capture
test_loss, test_accuracy = model.evaluate(dataset_test)
In [724]:
print('Test loss: ', '{:.2f}'.format(test_loss))
print('Test accuracy: ', '{:.2f}'.format(test_accuracy))
Save the model¶
We will save the entire model to a HDF5 file. The .h5 extension of the file indicates that the model should be saved in Keras format as HDF5 file. To use this model on the front-end we will convert it (later in this notebook) to Javascript understandable format (tfjs_layers_model with .json and .bin files) using tensorflowjs_converter as it is specified in the main README.
In [725]:
model_name = 'sketch_recognition_mlp.h5'
model.save(model_name, save_format='h5')
Converting the model to web-format¶
To use this model on the web we need to convert it into the format that will be understandable by tensorflowjs. To do so we may use tfjs-converter as following:
tensorflowjs_converter --input_format keras \
./experiments/sketch_recognition_mlp/sketch_recognition_mlp.h5 \
./demos/public/models/sketch_recognition_mlpYou find this experiment in the Demo app and play around with it right in you browser to see how the model performs in real life.