This session will cover a few more advanced concepts around Deep Learning in Python with Keras. We will build upon the ideas from the last session and learn about ways to customize the workflow further in detail. We will also learn how to solve some problems that we faced during the last session.
import tensorflow as tf
import numpy as np
This time, we will simply use a wrapper provided by Keras to load up the IMDB dataset that we explored in the last session. For reference, see the API docs.
INDEX_FROM=3
NUM_WORDS=1000
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.imdb.load_data(num_words=NUM_WORDS,index_from=INDEX_FROM)
Note that this already provides us with a train-test split.
This dataset is already built using word indices instead of word strings. For transforming text from and to indices using the word index, see this example.
Exercise: Explore the first 3 samples of X_train by converting them back to strings. What is going wrong? Why?
word_index = tf.keras.datasets.imdb.get_word_index()
inverted_word_index = {v:k for k,v in word_index.items()}
for i in range(3):
decoded_sequence = " ".join(inverted_word_index[ind] for ind in X_train[i])
print("label:",["negative","positive"][y_train[i]])
print(decoded_sequence)
print()
label: positive the as you with out themselves powerful and and their becomes and had and of lot from anyone to have after out atmosphere never more room and it so heart shows to years of every never going and help moments or of every and and movie except her was several of enough more with is now and film as you of and and unfortunately of you than him that with out themselves her get for was and of you movie sometimes movie that with scary but and to story wonderful that in seeing in character to of and and with heart had and they of here that with her serious to have does when from why what have and they is you that isn't one will very to as itself with other and in of seen over and for anyone of and br and to whether from than out themselves history he name half some br of and and was two most of mean for 1 any an and she he should is thought and but of script you not while history he heart to real at and but when from one bit then have two of script their with her and most that with wasn't to with and acting watch an for with and film want an label: negative the thought and thought and do making to is and and and while he of jack in where and as getting on was did hands fact characters to always life and not as me can't in at are br of sure your way of little it and and to view of love it so and of guy it used and of where it of here and film of and to don't all unique some like of direction it if out her and and keep of and he and to makes this and and of and it thought begins br and and budget and though ok and and for ever better were and and for budget look and any to of making it out and and for effects show to show cast this family us scenes more it and making and to and finds tv and to of and these thing wants but and an and and as it is video do you david see and it in few those are of and for with of and to one is very work dark they don't do dvd with those them label: negative the as there in at by br of sure many br of and no only women was than doesn't as you never of and night that with and they bad out and plays of how star so stories film comes and and of and they don't do that had with of hollywood br of my seeing fan this of and out body shots in having because and it's and and first were and for from look and sense from me and die in character as and and but is you that isn't one song just is him less are and not are you that different just even by this of you there is and when it part are film's love film's and was big also light don't and as it in character looked cinematography so stories is far br man acting
word_index = tf.keras.datasets.imdb.get_word_index()
inverted_word_index = {v+INDEX_FROM:k for k,v in word_index.items()}
inverted_word_index |= {0:"<PAD>",
1: "<START>",
2:"<UNK>",
3: "<UNUSED>"}
for i in range(3):
decoded_sequence = " ".join(inverted_word_index[ind] for ind in X_train[i])
print("label:",["negative","positive"][y_train[i]])
print(decoded_sequence)
print()
label: positive <START> this film was just brilliant casting <UNK> <UNK> story direction <UNK> really <UNK> the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same <UNK> <UNK> as myself so i loved the fact there was a real <UNK> with this film the <UNK> <UNK> throughout the film were great it was just brilliant so much that i <UNK> the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the <UNK> <UNK> was amazing really <UNK> at the end it was so sad and you know what they say if you <UNK> at a film it must have been good and this definitely was also <UNK> to the two little <UNK> that played the <UNK> of <UNK> and paul they were just brilliant children are often left out of the <UNK> <UNK> i think because the stars that play them all <UNK> up are such a big <UNK> for the whole film but these children are amazing and should be <UNK> for what they have done don't you think the whole story was so <UNK> because it was true and was <UNK> life after all that was <UNK> with us all label: negative <START> big <UNK> big <UNK> bad music and a <UNK> <UNK> <UNK> these are the words to best <UNK> this terrible movie i love cheesy horror movies and i've seen <UNK> but this had got to be on of the worst ever made the plot is <UNK> <UNK> and ridiculous the acting is an <UNK> the script is completely <UNK> the best is the end <UNK> with the <UNK> and how he worked out who the killer is it's just so <UNK> <UNK> written the <UNK> are <UNK> and funny in <UNK> <UNK> the <UNK> is big lots of <UNK> <UNK> men <UNK> those cut <UNK> <UNK> that show off their <UNK> <UNK> that men actually <UNK> them and the music is just <UNK> <UNK> that plays over and over again in almost every scene there is <UNK> music <UNK> and <UNK> taking away <UNK> and the <UNK> still doesn't close for <UNK> all <UNK> <UNK> this is a truly bad film whose only <UNK> is to look back on the <UNK> that was the <UNK> and have a good old laugh at how bad everything was back then label: negative <START> this has to be one of the worst films of the <UNK> when my friends i were watching this film being the <UNK> audience it was <UNK> at we just <UNK> watched the first half an hour with our <UNK> <UNK> the <UNK> at how bad it really was the rest of the time everyone else in the <UNK> just started talking to each other <UNK> or <UNK> <UNK> into their <UNK> that they actually <UNK> money they had <UNK> working to watch this <UNK> <UNK> for a film it must have looked like a great idea on <UNK> but on film it looks like no one in the film has a <UNK> what is going on crap acting crap <UNK> i can't get across how <UNK> this is to watch save yourself an hour a bit of your life
tf.data.Dataset
¶Using tf.data.Dataset
, we can represent very large datasets (will become very important later in the semester). Tensorflow will handle many features necessary for that internally.
Exercise: Use tf.data.Dataset.from_generator
(docs) to convert our ndarray-based dataset to tf.data.Dataset
. Provide an output_signature=(X,y)
(you will also have to have the generator return this format).
Using tf.data.Dataset.from_tensor_slices
is probably difficult because the data is not padded yet.
def gen(Xs, ys):
for X,y in zip(Xs,ys):
yield (X,y)
output_signature=(tf.TensorSpec(shape=[None], dtype=tf.int32),tf.TensorSpec(shape=[],dtype=tf.int32))
train_ds = tf.data.Dataset.from_generator(lambda: gen(X_train,y_train), output_signature=output_signature)
test_ds = tf.data.Dataset.from_generator(lambda: gen(X_test,y_test), output_signature=output_signature)
Converting the data back to numpy is easy:
next(train_ds.as_numpy_iterator())
(array([ 1, 14, 22, 16, 43, 530, 973, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 2, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16, 480, 66, 2, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 2, 88, 12, 16, 283, 5, 16, 2, 113, 103, 32, 15, 16, 2, 19, 178, 32]), 1)
Tensorflow makes it quite easy to save and load tf.data.Dataset
.
tf.data.experimental.save
and load
¶tf.data.experimental.save
(docs) and load
(docs) can be used to persist a Dataset to storage. This will create multiple files (shards).
Exercise: Save and load our dataset to storage.
tf.data.experimental.save(train_ds,path="imdb_train")
tf.data.experimental.save(test_ds,path="imdb_test")
train_ds=tf.data.experimental.load(path="imdb_train")
test_ds=tf.data.experimental.load(path="imdb_test")
Let's test it again:
next(train_ds.as_numpy_iterator())
(array([ 1, 14, 22, 16, 43, 530, 973, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 2, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16, 480, 66, 2, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 2, 88, 12, 16, 283, 5, 16, 2, 113, 103, 32, 15, 16, 2, 19, 178, 32]), 1)
Note: The TFRecord format is the traditional method to save serialized data, which might save memory.
map
and filter
¶Using map
(docs) and filter
(docs) on tf.data.Dataset
is very convenient, as the used functions are applied on the fly, controlled by demand.
It is recommended to use the tf.function
decorator (docs) to improve performance if possible.
Exercise: From the train_ds
, filter out all reviews shorter than 100 tokens.
@tf.function
def filter_func(X,y):
return tf.shape(X)[0]>=100
train_ds = train_ds.filter(filter_func)
@tf.function
def map_func(X,y):
size=tf.shape(X)[0]
r=tf.range(0,(size//100)*100,1)
r=tf.reshape(r,[size//100,100])
Xs=tf.gather(X,r)
ys=tf.repeat(y,size//100)
return tf.data.Dataset.from_tensor_slices((Xs,ys))
train_ds = train_ds.flat_map(map_func)
it=train_ds.as_numpy_iterator()
for i in range(3):
print(next(it))
(array([ 1, 14, 22, 16, 43, 530, 973, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 2, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16]), 1) (array([480, 66, 2, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16]), 1) (array([ 1, 194, 2, 194, 2, 78, 228, 5, 6, 2, 2, 2, 134, 26, 4, 715, 8, 118, 2, 14, 394, 20, 13, 119, 954, 189, 102, 5, 207, 110, 2, 21, 14, 69, 188, 8, 30, 23, 7, 4, 249, 126, 93, 4, 114, 9, 2, 2, 5, 647, 4, 116, 9, 35, 2, 4, 229, 9, 340, 2, 4, 118, 9, 4, 130, 2, 19, 4, 2, 5, 89, 29, 952, 46, 37, 4, 455, 9, 45, 43, 38, 2, 2, 398, 4, 2, 26, 2, 5, 163, 11, 2, 2, 4, 2, 9, 194, 775, 7, 2]), 0)
In order to make our dataset usable for training, we will need to batch it (split it up into batches), repeat it (so you can train on multiple epochs) and shuffle it (to avoid using the same order every time).
In this task, you will learn that the order of these operations indeed matters!
Let's create a dummy dataset:
DUMMY_DS_SIZE=30
dummy_ds=tf.data.Dataset.range(DUMMY_DS_SIZE)
DUMMY_BATCHSIZE=10
DUMMY_BUFFERSIZE=2*10
Exercise: Roll the dice to determine the order in which you will implement shuffle, batch and repeat. Try to spot flaws in the results by inspecting 5 epochs.
print(np.random.choice(["Shuffle, repeat, batch",
"Repeat, shuffle, batch",
"Batch, shuffle, repeat"]))
Repeat, shuffle, batch
new_dummy_ds=dummy_ds.shuffle(DUMMY_BUFFERSIZE).repeat().batch(DUMMY_BATCHSIZE)
for epoch in range(5):
for batch in new_dummy_ds.take(DUMMY_DS_SIZE//DUMMY_BATCHSIZE).as_numpy_iterator():
print(batch)
print()
# Observation: Despite being a bit unintuitive (repeating after shuffling?!), this is indeed the correct solution.
[ 3 4 0 21 7 22 5 15 27 28] [ 9 14 24 6 10 13 29 1 26 17] [19 18 20 8 25 23 16 11 2 12] [10 16 6 21 7 12 3 20 11 26] [29 24 1 13 0 4 5 8 9 25] [27 14 17 15 28 22 2 23 19 18] [16 10 9 12 17 11 19 6 8 5] [13 7 27 29 18 14 22 28 1 2] [23 21 25 26 15 20 0 3 24 4] [19 11 8 16 10 7 22 17 12 23] [15 2 25 4 9 3 13 24 14 0] [ 5 26 1 18 21 20 28 29 6 27] [ 8 0 19 7 16 22 5 24 12 13] [25 4 6 17 29 23 26 18 21 3] [ 1 2 20 11 10 27 14 15 9 28]
new_dummy_ds=dummy_ds.repeat().shuffle(DUMMY_BUFFERSIZE).batch(DUMMY_BATCHSIZE)
for epoch in range(5):
for batch in new_dummy_ds.take(DUMMY_DS_SIZE//DUMMY_BATCHSIZE).as_numpy_iterator():
print(batch)
print()
# Observation: Some samples occur multiple times inside the epoch (and sometimes even inside a batch)
[11 2 20 3 15 16 19 13 10 12] [ 5 9 17 29 0 8 0 4 7 28] [ 6 6 5 1 7 26 24 27 22 12] [ 6 9 7 19 4 1 21 16 14 11] [17 13 20 5 2 0 29 3 1 8] [ 9 3 11 7 10 22 14 15 23 0] [17 7 11 9 21 2 25 6 4 20] [13 27 14 18 19 15 5 4 24 6] [22 29 26 0 10 5 7 2 12 14] [16 14 0 4 5 18 7 1 25 2] [ 8 13 9 12 17 0 23 3 11 22] [ 6 10 8 24 11 19 29 3 27 5] [ 0 13 7 4 2 19 12 18 1 17] [ 6 3 25 24 29 14 5 9 8 7] [28 3 16 26 10 4 20 11 0 5]
new_dummy_ds=dummy_ds.batch(DUMMY_BATCHSIZE).shuffle(DUMMY_BUFFERSIZE).repeat()
for epoch in range(5):
for batch in new_dummy_ds.take(DUMMY_DS_SIZE//DUMMY_BATCHSIZE).as_numpy_iterator():
print(batch)
print()
# Observation: Only shuffles batches -> batch should be called last
[20 21 22 23 24 25 26 27 28 29] [0 1 2 3 4 5 6 7 8 9] [10 11 12 13 14 15 16 17 18 19] [10 11 12 13 14 15 16 17 18 19] [20 21 22 23 24 25 26 27 28 29] [0 1 2 3 4 5 6 7 8 9] [20 21 22 23 24 25 26 27 28 29] [10 11 12 13 14 15 16 17 18 19] [0 1 2 3 4 5 6 7 8 9] [10 11 12 13 14 15 16 17 18 19] [0 1 2 3 4 5 6 7 8 9] [20 21 22 23 24 25 26 27 28 29] [20 21 22 23 24 25 26 27 28 29] [10 11 12 13 14 15 16 17 18 19] [0 1 2 3 4 5 6 7 8 9]
Exercise: Shuffle, repeat and batch (using padded_batch
) our train_ds
.
BATCHSIZE=64
BUFFERSIZE=2*64
train_ds=train_ds.shuffle(BUFFERSIZE).repeat().padded_batch(BATCHSIZE)
Keras allows you to define custom layers. This is useful for:
Exercise: Re-implement a dense layer using a subclass of the tf.keras.layers.Layer
class (docs).
class CustomDenseLayer(tf.keras.layers.Layer):
def __init__(self, units=32):
super().__init__()
self.units = units
def build(self, input_shape):
w_init = tf.random_normal_initializer()
self.w = tf.Variable(
initial_value=w_init(shape=(input_shape[-1], self.units),
dtype='float32'),
trainable=True)
b_init = tf.zeros_initializer()
self.b = tf.Variable(
initial_value=b_init(shape=(self.units,), dtype='float32'),
trainable=True)
def call(self, inputs):
return tf.matmul(inputs, self.w) + self.b
class CustomDropoutLayer(tf.keras.layers.Layer):
def __init__(self, rate):
super().__init__()
self.rate = rate
def call(self, inputs, training=True):
#if training is None:
# training = tf.keras.backend.learning_phase()
random=tf.where(tf.random.uniform(inputs.shape,minval=0,maxval=1,dtype=tf.float32)<self.rate,0.0,1.0)
dropped_out=inputs*random
dropped_out=dropped_out/(1.0 - self.rate) # sum over all inputs should be unchanged
return tf.where(training,dropped_out,inputs)
If you want to dive deeper into defining custom layers, see this guide.
def custom_loss(y_true, y_pred):
return tf.reduce_sum(tf.where(y_true==tf.ones_like(y_true),
-tf.math.log(y_pred)*(tf.cast(tf.size(y_true)/tf.reduce_sum(y_true),tf.float32)),
-tf.math.log(1.0-y_pred)*(tf.cast(tf.size(y_true)/tf.reduce_sum(1-y_true),tf.float32))))
By defining a subclass to tf.keras.Model
, we can customize what is happening during fit()
on a more fine-grained level than using callbacks.
class CustomModel(tf.keras.Model):
def train_step(self, data):
# Unpack the data. Its structure depends on your model and
# on what you pass to `fit()`.
x, y = data
with tf.GradientTape() as tape:
y_pred = self(x, training=True) # Forward pass
# Compute the loss value
# (the loss function is configured in `compile()`)
loss = self.compiled_loss(y, y_pred, regularization_losses=self.losses)
# Compute gradients
trainable_vars = self.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
# Update weights
self.optimizer.apply_gradients(zip(gradients, trainable_vars))
# Update metrics (includes the metric that tracks the loss)
self.compiled_metrics.update_state(y, y_pred)
# Return a dict mapping metric names to current value
return {m.name: m.result() for m in self.metrics}
Further details on customizing the behavior of fit()
can be found in this guide.
TensorBoard is a browser application that allows you to supervise the training progress. To access the generated logs, use the following command:
%load_ext tensorboard
%tensorboard --logdir ./logs
# alternatively: !tensorboard --logdir ./logs
Reusing TensorBoard on port 6006 (pid 12512), started 0:05:55 ago. (Use '!kill 12512' to kill it.)
We use a special callback to generate the data that TensorBoard will be visualizing:
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="./logs",update_freq=1)
inputs = tf.keras.Input(shape=(None,),batch_size=BATCHSIZE)
x = tf.keras.layers.Embedding(len(inverted_word_index), 16)(inputs)
x = tf.keras.layers.GlobalAveragePooling1D()(x)
x = CustomDenseLayer(16)(x)
x = tf.keras.layers.Activation("relu")(x)
x = CustomDropoutLayer(0.1)(x)
outputs = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid)(x)
model = CustomModel(inputs,outputs)
print(model.summary())
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
loss=custom_loss)
model.fit(train_ds,validation_data=test_ds.batch(1),callbacks=[tensorboard_callback],epochs=10,steps_per_epoch=1000)