Welcome to your first programming assignment for this week!
This notebook was produced together with NVIDIA's Deep Learning Institute.
from tensorflow.keras.layers import Bidirectional, Concatenate, Permute, Dot, Input, LSTM, Multiply
from tensorflow.keras.layers import RepeatVector, Dense, Activation, Lambda
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import load_model, Model
import tensorflow.keras.backend as K
import tensorflow as tf
import numpy as np
from faker import Faker
import random
from tqdm import tqdm
from babel.dates import format_date
from nmt_utils import *
import matplotlib.pyplot as plt
%matplotlib inline
We will train the model on a dataset of 10,000 human readable dates and their equivalent, standardized, machine readable dates. Let's run the following cells to load the dataset and print some examples.
m = 10000
dataset, human_vocab, machine_vocab, inv_machine_vocab = load_dataset(m)
dataset[:10]
You've loaded:
dataset
: a list of tuples of (human readable date, machine readable date).human_vocab
: a python dictionary mapping all characters used in the human readable dates to an integervalued index.machine_vocab
: a python dictionary mapping all characters used in machine readable dates to an integervalued index. human_vocab
. inv_machine_vocab
: the inverse dictionary of machine_vocab
, mapping from indices back to characters. Let's preprocess the data and map the raw text data into the index values.
human_vocab
tf.config.list_physical_devices()
Tx = 30
Ty = 10
X, Y, Xoh, Yoh = preprocess_data(dataset, human_vocab, machine_vocab, Tx, Ty)
print("X.shape:", X.shape)
print("Y.shape:", Y.shape)
print("Xoh.shape:", Xoh.shape)
print("Yoh.shape:", Yoh.shape)
You now have:
X
: a processed version of the human readable dates in the training set.human_vocab
. X.shape = (m, Tx)
where m is the number of training examples in a batch.Y
: a processed version of the machine readable dates in the training set.machine_vocab
. Y.shape = (m, Ty)
. Xoh
: onehot version of X
X
is converted to the onehot representation (if the index is 2, the onehot version has the index position 2 set to 1, and the remaining positions are 0.Xoh.shape = (m, Tx, len(human_vocab))
Yoh
: onehot version of Y
Y
is converted to the onehot representation. Yoh.shape = (m, Ty, len(machine_vocab))
. len(machine_vocab) = 11
since there are 10 numeric digits (0 to 9) and the 
symbol.index
in the cell below to navigate the dataset and see how source/target dates are preprocessed. index = 0
print("Source date:", dataset[index][0])
print("Target date:", dataset[index][1])
print()
print("Source after preprocessing (indices):", X[index])
print("Target after preprocessing (indices):", Y[index])
print()
print("Source after preprocessing (onehot):", Xoh[index])
print("Target after preprocessing (onehot):", Yoh[index])
In this part, you will implement the attention mechanism presented in the lecture videos.
Here are some properties of the model that you may notice:
Postattention LSTM: at the top of the diagram comes after the attention mechanism.
The postattention LSTM passes the hidden state $s^{\langle t \rangle}$ and cell state $c^{\langle t \rangle}$ from one time step to the next.
RepeatVector
node to copy $s^{\langle t1 \rangle}$'s value $T_x$ times.Concatenation
to concatenate $s^{\langle t1 \rangle}$ and $a^{\langle t \rangle}$.RepeatVector
and Concatenation
in Keras below. Let's implement this neural translator. You will start by implementing two functions: one_step_attention()
and model()
.
Implement one_step_attention()
.
model()
will call the layers in one_step_attention()
$T_y$ times using a forloop.one_step_attention
function. For example, defining the objects as global variables would work.model
would technically work, since model
will then call the one_step_attention
function. For the purposes of making grading and troubleshooting easier, we are defining these as global variables. Note that the automatic grader will expect these to be global variables as well.var_repeated = repeat_layer(var1)
concatenated_vars = concatenate_layer([var1,var2,var3])
var_out = dense_layer(var_in)
activation = activation_layer(var_in)
dot_product = dot_layer([var1,var2])
# Defined shared layers as global variables
repeator = RepeatVector(Tx)
concatenator = Concatenate(axis=1)
densor1 = Dense(10, activation = "tanh")
densor2 = Dense(1, activation = "relu")
activator = Activation(softmax, name='attention_weights') # We are using a custom softmax(axis = 1) loaded in this notebook
dotor = Dot(axes = 1)
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: one_step_attention
def one_step_attention(a, s_prev):
"""
Performs one step of attention: Outputs a context vector computed as a dot product of the attention weights
"alphas" and the hidden states "a" of the BiLSTM.
Arguments:
a  hidden state output of the BiLSTM, numpyarray of shape (m, Tx, 2*n_a)
s_prev  previous hidden state of the (postattention) LSTM, numpyarray of shape (m, n_s)
Returns:
context  context vector, input of the next (postattention) LSTM cell
"""
### START CODE HERE ###
# Use repeator to repeat s_prev to be of shape (m, Tx, n_s) so that you can concatenate it with all hidden states "a" (≈ 1 line)
s_prev = repeator(s_prev)
# Use concatenator to concatenate a and s_prev on the last axis (≈ 1 line)
# For grading purposes, please list 'a' first and 's_prev' second, in this order.
concat = concatenator([a,s_prev])
# Use densor1 to propagate concat through a small fullyconnected neural network to compute the "intermediate energies" variable e. (≈1 lines)
e = densor1(concat)
# Use densor2 to propagate e through a small fullyconnected neural network to compute the "energies" variable energies. (≈1 lines)
energies = densor2(e)
# Use "activator" on "energies" to compute the attention weights "alphas" (≈ 1 line)
alphas = activator(energies)
# Use dotor together with "alphas" and "a", in this order, to compute the context vector to be given to the next (postattention) LSTMcell (≈ 1 line)
context = dotor([alphas,a])
### END CODE HERE ###
return context
# UNIT TEST
def one_step_attention_test(target):
m = 10
Tx = 30
n_a = 32
n_s = 64
#np.random.seed(10)
a = np.random.uniform(1, 0, (m, Tx, 2 * n_a)).astype(np.float32)
s_prev =np.random.uniform(1, 0, (m, n_s)).astype(np.float32) * 1
context = target(a, s_prev)
assert type(context) == tf.python.framework.ops.EagerTensor, "Unexpected type. It should be a Tensor"
assert tuple(context.shape) == (m, 1, n_s), "Unexpected output shape"
assert np.all(context.numpy() > 0), "All output values must be > 0 in this example"
assert np.all(context.numpy() < 1), "All output values must be < 1 in this example"
#assert np.allclose(context[0][0][0:5].numpy(), [0.50877404, 0.57160693, 0.45448175, 0.50074816, 0.53651875]), "Unexpected values in the result"
print("\033[92mAll tests passed!")
one_step_attention_test(one_step_attention)
Implement modelf()
as explained in figure 1 and the instructions:
modelf
first runs the input through a BiLSTM to get $[a^{<1>},a^{<2>}, ..., a^{<T_x>}]$. modelf
calls one_step_attention()
$T_y$ times using a for
loop. At each iteration of this loop:Again, we have defined global layers that will share weights to be used in modelf()
.
n_a = 32 # number of units for the preattention, bidirectional LSTM's hidden state 'a'
n_s = 64 # number of units for the postattention LSTM's hidden state "s"
# Please note, this is the post attention LSTM cell.
post_activation_LSTM_cell = LSTM(n_s, return_state = True) # Please do not modify this global variable.
output_layer = Dense(len(machine_vocab), activation=softmax)
Now you can use these layers $T_y$ times in a for
loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps:
X
into a bidirectional LSTM.Sample code:
sequence_of_hidden_states = Bidirectional(LSTM(units=..., return_sequences=...))(the_input_X)
Iterate for $t = 0, \cdots, T_y1$:
one_step_attention()
, passing in the sequence of hidden states $[a^{\langle 1 \rangle},a^{\langle 2 \rangle}, ..., a^{ \langle T_x \rangle}]$ from the preattention bidirectional LSTM, and the previous hidden state $s^{<t1>}$ from the postattention LSTM to calculate the context vector $context^{<t>}$.Give $context^{<t>}$ to the postattention LSTM cell.
This outputs the new hidden state $s^{<t>}$ and the new cell state $c^{<t>}$.
Sample code:
next_hidden_state, _ , next_cell_state =
post_activation_LSTM_cell(inputs=..., initial_state=[prev_hidden_state, prev_cell_state])
Please note that the layer is actually the "post attention LSTM cell". For the purposes of passing the automatic grader, please do not modify the naming of this global variable. This will be fixed when we deploy updates to the automatic grader.
output = output_layer(inputs=...)
Create your Keras model instance.
X
, the onehot encoded inputs to the model, of shape ($T_{x}, humanVocabSize)$model = Model(inputs=[...,...,...], outputs=...)
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: model
def modelf(Tx, Ty, n_a, n_s, human_vocab_size, machine_vocab_size):
"""
Arguments:
Tx  length of the input sequence
Ty  length of the output sequence
n_a  hidden state size of the BiLSTM
n_s  hidden state size of the postattention LSTM
human_vocab_size  size of the python dictionary "human_vocab"
machine_vocab_size  size of the python dictionary "machine_vocab"
Returns:
model  Keras model instance
"""
# Define the inputs of your model with a shape (Tx,)
# Define s0 (initial hidden state) and c0 (initial cell state)
# for the decoder LSTM with shape (n_s,)
X = Input(shape=(Tx, human_vocab_size))
s0 = Input(shape=(n_s,), name='s0')
c0 = Input(shape=(n_s,), name='c0')
s = s0
c = c0
# Initialize empty list of outputs
outputs = []
### START CODE HERE ###
# Step 1: Define your preattention BiLSTM. (≈ 1 line)
a = Bidirectional(LSTM(n_a, return_sequences=True))(X)
# Step 2: Iterate for Ty steps
for t in range(Ty):
# Step 2.A: Perform one step of the attention mechanism to get back the context vector at step t (≈ 1 line)
context = one_step_attention(a, s)
# Step 2.B: Apply the postattention LSTM cell to the "context" vector.
# Don't forget to pass: initial_state = [hidden state, cell state] (≈ 1 line)
s, _, c = post_activation_LSTM_cell(context,initial_state=[s, c])
# Step 2.C: Apply Dense layer to the hidden state output of the postattention LSTM (≈ 1 line)
out = output_layer(s)
# Step 2.D: Append "out" to the "outputs" list (≈ 1 line)
outputs.append(out)
# Step 3: Create model instance taking three inputs and returning the list of outputs. (≈ 1 line)
model = Model(inputs=[X, s0, c0],outputs=outputs)
### END CODE HERE ###
return model
# UNIT TEST
from test_utils import *
def modelf_test(target):
m = 10
Tx = 30
n_a = 32
n_s = 64
len_human_vocab = 37
len_machine_vocab = 11
model = target(Tx, Ty, n_a, n_s, len_human_vocab, len_machine_vocab)
print(summary(model))
expected_summary = [['InputLayer', [(None, 30, 37)], 0],
['InputLayer', [(None, 64)], 0],
['Bidirectional', (None, 30, 64), 17920],
['RepeatVector', (None, 30, 64), 0, 30],
['Concatenate', (None, 30, 128), 0],
['Dense', (None, 30, 10), 1290, 'tanh'],
['Dense', (None, 30, 1), 11, 'relu'],
['Activation', (None, 30, 1), 0],
['Dot', (None, 1, 64), 0],
['InputLayer', [(None, 64)], 0],
['LSTM',[(None, 64), (None, 64), (None, 64)], 33024,[(None, 1, 64), (None, 64), (None, 64)],'tanh'],
['Dense', (None, 11), 715, 'softmax']]
comparator(summary(model), expected_summary)
modelf_test(modelf)
Run the following cell to create your model.
model = modelf(Tx, Ty, n_a, n_s, len(human_vocab), len(machine_vocab))
Let's get a summary of the model to check if it matches the expected output.
model.summary()
Expected Output:
Here is the summary you should see


**Total params:**  52,960 
**Trainable params:**  52,960 
**Nontrainable params:**  0 
**bidirectional_1's output shape **  (None, 30, 64) 
**repeat_vector_1's output shape **  (None, 30, 64) 
**concatenate_1's output shape **  (None, 30, 128) 
**attention_weights's output shape **  (None, 30, 1) 
**dot_1's output shape **  (None, 1, 64) 
**dense_3's output shape **  (None, 11) 
Sample code
optimizer = Adam(lr=..., beta_1=..., beta_2=..., decay=...)
model.compile(optimizer=..., loss=..., metrics=[...])
### START CODE HERE ### (≈2 lines)
opt = Adam(lr=0.005, beta_1=0.9, beta_2=0.999, decay=0.01)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
### END CODE HERE ###
# UNIT TESTS
assert opt.lr == 0.005, "Set the lr parameter to 0.005"
assert opt.beta_1 == 0.9, "Set the beta_1 parameter to 0.9"
assert opt.beta_2 == 0.999, "Set the beta_2 parameter to 0.999"
assert opt.decay == 0.01, "Set the decay parameter to 0.01"
assert model.loss == "categorical_crossentropy", "Wrong loss. Use 'categorical_crossentropy'"
assert model.optimizer == opt, "Use the optimizer that you have instantiated"
assert model.compiled_metrics._user_metrics[0] == 'accuracy', "set metrics to ['accuracy']"
print("\033[92mAll tests passed!")
The last step is to define all your inputs and outputs to fit the model:
s0
and c0
to initialize your post_attention_LSTM_cell
with zeros.model()
you coded, you need the "outputs" to be a list of 10 elements of shape (m, T_y). outputs[i][0], ..., outputs[i][Ty]
represents the true labels (characters) corresponding to the $i^{th}$ training example (X[i]
). outputs[i][j]
is the true label of the $j^{th}$ character in the $i^{th}$ training example.s0 = np.zeros((m, n_s))
c0 = np.zeros((m, n_s))
outputs = list(Yoh.swapaxes(0,1))
Let's now fit the model and run it for one epoch.
model.fit([Xoh, s0, c0], outputs, epochs=1, batch_size=100)
While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:
We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)
model.load_weights('models/model.h5')
You can now see the results on new examples.
EXAMPLES = ['3 May 1979', '5 April 09', '21th of August 2016', 'Tue 10 Jul 2007', 'Saturday May 9 2018', 'March 3 2001', 'March 3rd 2001', '1 March 2001']
s00 = np.zeros((1, n_s))
c00 = np.zeros((1, n_s))
for example in EXAMPLES:
source = string_to_int(example, Tx, human_vocab)
#print(source)
source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1)
source = np.swapaxes(source, 0, 1)
source = np.expand_dims(source, axis=0)
prediction = model.predict([source, s00, c00])
prediction = np.argmax(prediction, axis = 1)
output = [inv_machine_vocab[int(i)] for i in prediction]
print("source:", example)
print("output:", ''.join(output),"\n")
def translate_date(sentence):
source = string_to_int(sentence, Tx, human_vocab)
source = np.array(list(map(lambda x: to_categorical(x, num_classes=len(human_vocab)), source))).swapaxes(0,1)
source = np.swapaxes(source, 0, 1)
source = np.expand_dims(source, axis=0)
prediction = model.predict([source, s00, c00])
prediction = np.argmax(prediction, axis = 1)
output = [inv_machine_vocab[int(i)] for i in prediction]
print("source:", sentence)
print("output:", ''.join(output),"\n")
example = "4th of july 2001"
translate_date(example)
You can also change these examples to test with your own examples. The next part will give you a better sense of what the attention mechanism is doingi.e., what part of the input the network is paying attention to when generating a particular output character.
Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (such as the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can visualize what each part of the output is looking at which part of the input.
Consider the task of translating "Saturday 9 May 2018" to "20180509". If we visualize the computed $\alpha^{\langle t, t' \rangle}$ we get this:
Notice how the output ignores the "Saturday" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We also see that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's "18" in order to generate "2018."
Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of $\alpha^{\langle t, t' \rangle}$.
To figure out where the attention values are located, let's start by printing a summary of the model .
model.summary()
Navigate through the output of model.summary()
above. You can see that the layer named attention_weights
outputs the alphas
of shape (m, 30, 1) before dot_2
computes the context vector for every time step $t = 0, \ldots, T_y1$. Let's get the attention weights from this layer.
The function attention_map()
pulls out the attention values from your model and plots them.
Note: We are aware that you might run into an error running the cell below despite a valid implementation for Exercise 2  modelf
above. If you get the error kindly report it on this Topic on Discourse as it'll help us improve our content.
If you haven’t joined our Discourse community you can do so by clicking on the link: http://bit.ly/dlsdiscourse
And don’t worry about the error, it will not affect the grading for this assignment.
attention_map = plot_attention_map(model, human_vocab, inv_machine_vocab, "Tuesday 09 Oct 1993", num = 7, n_s = 64);
On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that the places where the network is paying attention makes sense to you.
In the date translation application, you will observe that most of the time attention helps predict the year, and doesn't have much impact on predicting the day or month.
You have come to the end of this assignment
Congratulations on finishing this assignment! You are now able to implement an attention model and use it to learn complex mappings from one sequence to another.