0. Quick overview¶

The goal of this notebook is to introduce convolutional neural networks; A neural network architecture that is (loosely) inspired by the biological vision system and used to process visual image (or video) data (e.g., to identify the objects depicted in an image).

You will further get a basic introduction to Tensorflow and Keras; two of the most widely used deep learning libraries.

If you stick around, you will also learn a bit about Google's deep dream algorithm at the end of the notebook.

1. Classifying images of clothing pieces¶

In this notebook, we will be using the Fashion-MNIST dataset.

This dataset contains 70,000 28x28 pixel grey-scale images of clothing pieces, encompassing a total of 10 different categories of clothing.

We can load the dataset with the fashion_mnist.load_data() function from tensorflow.keras.datasets.

(More details on Tensorflow and Keras will follow later).

This dataset is already divided into a distinct training and test dataset, each containing 60000 and 10000 images:

To make these images easily digestible for an artificial neural network, we will normalize the range of the pixel values to a range between 0 and 1:

Ok, the normalization worked; The high frequency of 0 values is based on the consistent black background of each image.

Let's take a look at a few of the training examples:

2. Convolutional Neural Networks¶

2.1 Biological inspiration¶

A very simple way to classify this dataset would be to flatten each sample to a vector of 784 values (28x28) and to then feed them to a fully-connected (or dense) artificial neural network (as we did with the handwritten digits in the previous notebook (1-Neural-Networks-Backpropagation.ipynb).

However, there is another common type of artificial neural network architecture that was specifically designed for this type of computer vision problem:

These networks are called convolutional neural networks (or CNNs) and are (very) loosely inspired by our knowledge of the neurobiological processes underlying the vision system.

On a superficial level, the brain processes visual information in the following steps (see the figure below):

First, a projection of the image is registered on the retina. The retina is composed of various types of neurons, each sensitive to specific characteristics of the perceived image. Each neuron further has a receptive field, restricting its sensitivity to specific area of the input image.

These retina neurons are connected to the optical nerve, which sends their output signals to the part of the visual system in the brain.

The visual system encompasses multiple brain regions (see for example the primary visual cortex, which each seem to respond to different patterns in (or aspects of) the visual input.

Importantly, lower-level brain regions of the visual system (which are closer to the visual input) are generally more sensitive to basic properties of the input (such as orientations, colors, or contrasts), while higher-level brain regions generally respond to more complex patterns (famous examples are the fusiform face area and parahippocampal place area which specifically respond to faces and scenes).

Adapted from: Kubilius, Jonas (2017): Ventral visual stream. figshare. Figure. https://doi.org/10.6084/m9.figshare.106794.v3

2.2. Convolution Kernels (or filters)¶

Convolution kernels (or filters) represent the core building block of a convolutional neural networks. Conceptually, each kernel mimics a neuron of the visual system and responds to a specific pattern in the visual input.

Computationally, each kernel $k$ is composed of a small square matrix (typically with a size 3x3 or 5x5 values).

To obtain an activation map with a kernel (indicating whether the pattern to which the kernel responds is present at each location of the input), we can perform a spatial convolution between the kernel and image.

To do this, we simply move the kernel over the image and compute a weighted sum between the values of the kernel and the values of the underlying image at each location. Similarly to a Perceptron, a bias term $b$ is added to the weighted sum and it is scaled through a non-linear activation function ($\phi$). Typically, convolution kernels are activated through the rectified linear unit function (ReLu.

Importantly, the distance between the applications of the kernel to the input image is called the stride. A stride of 1 pixel creates an activation map of the same size as the input image, while larger strides reduce the size of the activation map:

Let's try this ourselves with some data:

Here, we will be using a few exemplary edge detection kernels (based on the Sobel Operator):

What kind of activation maps do we get when we convolve these three kernels with a few examples of our dataset?

As expected, the activation maps highlight the characteristics of the image that each kernel is sensitive to: vertical edges, horizontal edges, all edges!

2.3 Stacking many kernels to form a convolutional neural network¶

A CNN is nothing more than a sequence of convolution layers, which are each composed of a stack of convolution kernels!

Importantly, the kernels of each layer are applied to the activation maps resulting from the previous layer. This trick allows higher-level convolution kernels (which are deeper into the network) to learn very abstracted features, based on the activation maps of the preceding lower-level convolution kernels (for more details on this, see this amazing paper: https://distill.pub/2018/building-blocks/).

2.4 Pooling kernels¶

Classical CNN architectures contain one more type of kernel, which is used to down-sample the activation maps.

These kernels are called pooling kernels.

Two classical pooling kernels are average and max pooling.

They do nothing else then return the average or maximum value of the input in their receptive field.

Importantly, they are moved over the input in non-overlapping steps: A pooling kernel of size 2x2 would therefore be moved over the input in 2 pixel steps; thereby down-sampling the input by a factor of 2:

2.5 Building a convolutional neural network with Keras and Tensorflow¶

Enough with the theory, let's build our first CNN!

To do this, we will be using one of the most widely used deep learning libraries: Tensorflow.

Tensorflow is a high-level Python library that makes it extremely easy to build and train state-of-the-art artificial neural network architectures.

Specifically, we will be using the Keras library that is part of Tensorflow.

In Keras, a deep learning model can be specified by the use of the Sequential model class.

Calling models.Sequential initiates an empty model.

We can then sequentially add layers to this model from Keras' layers module. Here, we will be focusing on the Conv2Dlayer type (representing a 2D-convolution layer) as well as AveragePooling2Dand Dense layers. Dense layers represent fully connected artificial neural network layers, such as the ones that we used in the previous notebook (1-Neural-Networks-Backpropagation.ipynb), each containing a set of individual Perceptrons.

For an overview of all available layer types, see this.

Here, we will re-create the classical LeNet-5 architecture.

Note that this architecture, in spite of its historical importance, is outdated! Nowadays, one would probably use smaller kernels, ReLu activations, and more convolution layers (instead of the three added fully connected layers; see below).

LeNet-5 includes three dense layers at the end of the network, with 120, 84, and 10 neurons each (see the code below). To allow for the addition of these layer, we need to flatten the activation maps resulting from the last convolution layer, such that they are each represented as a single row vector and can be processed by the dense layer:

Ok, lets take a look at our full model, by calling model.summary():

This is looking good.

We can see that the size of the activation maps is decreasing as the data is passed through the network.

We can also see that the activation maps of each convolution layer are stacked along the last dimension (look at the "Output Shape" column).

The flat activation maps resulting from our last convolution layer have a size of 288 values and the output layer contains 10 neurons (one for each class).

Overall, this model has almost 45,000 trainable parameters - mostly resulting from the first dense layer!

2.6 Training the model¶

Now that we have build our model, let's train it:

In Keras, this requires two steps:

First, we compile the model, by calling model.compile. Here, Keras builds a computational graph for the model in the background, automatically specifying all gradient computations (which we previously needed to specify by hand) and initializing all the weights.

In this step, we also specify the loss function that we want to minimize during training. Similar to our previous examples, we are using the cross entropy loss for multiple classes.

We further define the optimizer that we want to use to minimize our loss function: In our previous examples, we used a vanilla version of stochastic gradient descent. There exist, however, more sophisticated optimizers (for an overview, see here and here); We will use RMSprop (the default in Keras).

Lastly, we tell Keras to also track the predictive accuracy of our model during training (next to the loss; this solely means that Keras repeatedly computes the accuracy in the training and validation data during training and saves it; we do not optimize the predictive accuracy!)

Second, we specify everything else that is missing to actually perform stochastic gradient descent: e.g., the training and validation datasets, the batch size, and number of training epochs that we want to perform.

In Keras, an epoch is defined as an entire iteration over the training dataset:

Let's take a look at the training statistics:

Great! Our model is able to classify more than 80% of the images in the test dataset correctly (this value might vary on different machines due to difficulties with random initializations).

Note however that it is also slightly overfitting the training dataset (as indicated by the slightly higher predictive accuracy in the training data).

3. Your exercise:¶

Below you will find a copy of the code that we just ran to build and train our CNN.

• Try adapting the specifications of the individual layers (e.g., reduce / increase the number of kernels or change the size of the kernels (e.g., to 3x3 or 7x7) or their activation funcion) and see how it changes the performance of the model.

• You can also try out different optimizers (e.g., 'adam' or 'SGD'), batch sizes, and number of training epochs.

• Lastly, you can also try to remove the pooling layers from the model and replace them with an increasing stride size of the convolution layers that are subsequent to them. An increase in stride size should have the same down-sampling effect as a pooling layer!

Be reasonable: try to avoid very large numbers of kernels per layer, more than 10 layers in the network, or a very large batch sizes. Otherwise you might crash the notebook that you are currently running; Chances are that you are running this with very limited resources on the Jupyter Binder Servers.

4. One last bonus: activation maximization with deep dream¶

One last bonus for those of you who want to dig a bit deeper into the mechanics of convolutional neural networks:

Remember that I previously said that the convolution kernels respond to increasingly complex patterns in the data as you move further down into the network?

We can actually generate images that give us an insight into these patterns: To do this, we will use the deep dream algorithm. The code for this is buried in the deep_dream.py script, which you can find in the same directory as this notebook.

The basic idea of deep dream is very simple: We can turn the gradient descent procedure around, by optimizing an input image instead of the network parameters. Simply put, we are trying to find the image that maximally activates a specific kernel of the network (by performing gradient ascent).

For this analysis, we will use the VGG16 architecture, which was pre-trained on the ImageNet dataset.

We will then run the deep dream algorithm for three kernels of the layers 'block1_conv1', 'block2_conv2', 'block3_conv3', and 'block4_conv3'.

Importantly, I have pre-computed the output of this analysis. This analysis is computationally expensive, so I would not recommend running it on the Jupyter Binder Servers.

Instead, you can run it at home on your personal machine, if you like.

To do this, simply set the switch run_analysis to True..

We can see that the resulting patterns, which maximally excite the individual kernels, get increasingly more abstract and complex the deeper we travel into the network (from left to right in the figure). Each row represents a the pattern that maximally excites the respective kernel of each layer.

Super cool, no?

Instead of maximizing the output of a single kernel, we could also maximize the output of entire layers:

As expected, entire layers respond to even more complicated patterns; So far as that the last layer that we are plotting (on the far right) seems to be sensitive to very specific features (such as eyes?).