Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.
A simple character-level RNN to generate new bits of text based on text from a novel.
%load_ext watermark
%watermark -a 'Sebastian Raschka' -v -p torch
import torch
import torch.nn.functional as F
from torchtext import data
from torchtext import datasets
import time
import random
import unidecode
import string
import random
import re
torch.backends.cudnn.deterministic = True
Sebastian Raschka CPython 3.7.1 IPython 7.4.0 torch 1.0.1.post2
RANDOM_SEED = 123
torch.manual_seed(RANDOM_SEED)
DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
TEXT_PORTION_SIZE = 200
NUM_ITER = 20000
LEARNING_RATE = 0.005
EMBEDDING_DIM = 100
HIDDEN_DIM = 100
NUM_HIDDEN = 1
Download A Tale of Two Cities by Charles Dickens from the Gutenberg Project:
!wget http://www.gutenberg.org/files/98/98-0.txt
--2019-04-26 04:03:36-- http://www.gutenberg.org/files/98/98-0.txt Resolving www.gutenberg.org (www.gutenberg.org)... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47 Connecting to www.gutenberg.org (www.gutenberg.org)|152.19.134.47|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 804335 (785K) [text/plain] Saving to: ‘98-0.txt.11’ 98-0.txt.11 100%[===================>] 785.48K 1.68MB/s in 0.5s 2019-04-26 04:03:36 (1.68 MB/s) - ‘98-0.txt.11’ saved [804335/804335]
Convert all characters into ASCII characters provided by string.printable
:
string.printable
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'
with open('./98-0.txt', 'r') as f:
textfile = f.read()
# convert special characters
textfile = unidecode.unidecode(textfile)
# strip extra whitespaces
textfile = re.sub(' +',' ', textfile)
TEXT_LENGTH = len(textfile)
print(f'Number of characters in text: {TEXT_LENGTH}')
Number of characters in text: 776058
Divide the text into smaller portions:
random.seed(RANDOM_SEED)
def random_portion(textfile):
start_index = random.randint(0, TEXT_LENGTH - TEXT_PORTION_SIZE)
end_index = start_index + TEXT_PORTION_SIZE + 1
return textfile[start_index:end_index]
print(random_portion(textfile))
left his saw sticking in the firewood he was cutting, set it in motion again; the women who had left on a door-step the little pot of hot ashes, at which she had been trying to soften the pain in her
Define a function to convert characters into tensors of integers (type long):
def char_to_tensor(text):
lst = [string.printable.index(c) for c in text]
tensor = torch.tensor(lst).long()
return tensor
print(char_to_tensor('abcDEF'))
tensor([10, 11, 12, 39, 40, 41])
Putting it together to make a function that draws random batches for training:
def draw_random_sample(textfile):
text_long = char_to_tensor(random_portion(textfile))
inputs = text_long[:-1]
targets = text_long[1:]
return inputs, targets
draw_random_sample(textfile)
(tensor([94, 17, 18, 28, 94, 32, 18, 23, 13, 24, 32, 94, 10, 28, 94, 18, 15, 94, 29, 17, 10, 29, 94, 32, 14, 27, 14, 94, 27, 30, 21, 14, 13, 94, 15, 24, 27, 94, 15, 18, 16, 30, 27, 14, 28, 94, 29, 24, 24, 73, 94, 10, 23, 13, 94, 14, 31, 14, 27, 34, 29, 17, 18, 23, 16, 96, 30, 23, 13, 14, 27, 94, 29, 17, 14, 94, 12, 21, 24, 30, 13, 28, 94, 32, 14, 27, 14, 94, 10, 94, 28, 30, 22, 75, 96, 96, 63, 43, 10, 21, 21, 24, 10, 62, 63, 94, 28, 10, 18, 13, 94, 48, 27, 75, 94, 54, 29, 27, 34, 31, 14, 27, 75, 94, 63, 43, 24, 32, 94, 13, 24, 94, 34, 24, 30, 94, 13, 24, 82, 94, 44, 94, 17, 24, 25, 14, 94, 34, 24, 30, 94, 10, 27, 14, 94, 32, 14, 21, 21, 62, 63, 96, 96, 44, 29, 94, 32, 10, 28, 94, 54, 29, 27, 34, 31, 14, 27, 68, 28, 94, 16, 27, 10, 23, 13, 94, 25, 14, 12, 30, 21, 18, 10, 27, 18, 29, 34, 94, 29, 17]), tensor([17, 18, 28, 94, 32, 18, 23, 13, 24, 32, 94, 10, 28, 94, 18, 15, 94, 29, 17, 10, 29, 94, 32, 14, 27, 14, 94, 27, 30, 21, 14, 13, 94, 15, 24, 27, 94, 15, 18, 16, 30, 27, 14, 28, 94, 29, 24, 24, 73, 94, 10, 23, 13, 94, 14, 31, 14, 27, 34, 29, 17, 18, 23, 16, 96, 30, 23, 13, 14, 27, 94, 29, 17, 14, 94, 12, 21, 24, 30, 13, 28, 94, 32, 14, 27, 14, 94, 10, 94, 28, 30, 22, 75, 96, 96, 63, 43, 10, 21, 21, 24, 10, 62, 63, 94, 28, 10, 18, 13, 94, 48, 27, 75, 94, 54, 29, 27, 34, 31, 14, 27, 75, 94, 63, 43, 24, 32, 94, 13, 24, 94, 34, 24, 30, 94, 13, 24, 82, 94, 44, 94, 17, 24, 25, 14, 94, 34, 24, 30, 94, 10, 27, 14, 94, 32, 14, 21, 21, 62, 63, 96, 96, 44, 29, 94, 32, 10, 28, 94, 54, 29, 27, 34, 31, 14, 27, 68, 28, 94, 16, 27, 10, 23, 13, 94, 25, 14, 12, 30, 21, 18, 10, 27, 18, 29, 34, 94, 29, 17, 10]))
class RNN(torch.nn.Module):
def __init__(self, input_size, embed_size,
hidden_size, output_size, num_layers):
super(RNN, self).__init__()
self.num_layers = num_layers
self.hidden_size = hidden_size
self.embed = torch.nn.Embedding(input_size, hidden_size)
self.gru = torch.nn.GRU(input_size=embed_size,
hidden_size=hidden_size,
num_layers=num_layers)
self.fc = torch.nn.Linear(hidden_size, output_size)
self.init_hidden = torch.nn.Parameter(torch.zeros(
num_layers, 1, hidden_size))
def forward(self, features, hidden):
embedded = self.embed(features.view(1, -1))
output, hidden = self.gru(embedded.view(1, 1, -1), hidden)
output = self.fc(output.view(1, -1))
return output, hidden
def init_zero_state(self):
init_hidden = torch.zeros(self.num_layers, 1, self.hidden_size).to(DEVICE)
return init_hidden
torch.manual_seed(RANDOM_SEED)
model = RNN(len(string.printable), EMBEDDING_DIM, HIDDEN_DIM, len(string.printable), NUM_HIDDEN)
model = model.to(DEVICE)
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
def evaluate(model, prime_str='A', predict_len=100, temperature=0.8):
## based on https://github.com/spro/practical-pytorch/
## blob/master/char-rnn-generation/char-rnn-generation.ipynb
hidden = model.init_zero_state()
prime_input = char_to_tensor(prime_str)
predicted = prime_str
# Use priming string to "build up" hidden state
for p in range(len(prime_str) - 1):
_, hidden = model(prime_input[p].to(DEVICE), hidden.to(DEVICE))
inp = prime_input[-1]
for p in range(predict_len):
output, hidden = model(inp.to(DEVICE), hidden.to(DEVICE))
# Sample from the network as a multinomial distribution
output_dist = output.data.view(-1).div(temperature).exp()
top_i = torch.multinomial(output_dist, 1)[0]
# Add predicted character to string and use as next input
predicted_char = string.printable[top_i]
predicted += predicted_char
inp = char_to_tensor(predicted_char)
return predicted
start_time = time.time()
for iteration in range(NUM_ITER):
### FORWARD AND BACK PROP
hidden = model.init_zero_state()
optimizer.zero_grad()
loss = 0.
inputs, targets = draw_random_sample(textfile)
inputs, targets = inputs.to(DEVICE), targets.to(DEVICE)
for c in range(TEXT_PORTION_SIZE):
outputs, hidden = model(inputs[c], hidden)
loss += F.cross_entropy(outputs, targets[c].view(1))
loss /= TEXT_PORTION_SIZE
loss.backward()
### UPDATE MODEL PARAMETERS
optimizer.step()
### LOGGING
with torch.set_grad_enabled(False):
if iteration % 1000 == 0:
print(f'Time elapsed: {(time.time() - start_time)/60:.2f} min')
print(f'Iteration {iteration} | Loss {loss.item():.2f}\n\n')
print(evaluate(model, 'Th', 200), '\n')
print(50*'=')
Time elapsed: 0.00 min Iteration 0 | Loss 4.58 Th4izvh?=lw2ZaCV_}xEt5y.gA^+rPgO2z@<$.1KRQe/c\ {a5A55Dun}_*czf.o6Hmy$l"W@fi{7rjKsvnEMJ mr`PaKygiE+VSbR#RF|SC^g^CZK,aenDc)t.O_ D^(M]1w'^Wd_HDws\>_2)iavp?*c-npOvoQE>i L ================================================== Time elapsed: 2.63 min Iteration 1000 | Loss 1.81 Th Prost into he forn a wock, abrould with his lother the star a caide with the Jue turnd face. Breaknay when and and of or, street were work have the long is on the proseing bove wabres. Throk a mean h ================================================== Time elapsed: 5.29 min Iteration 2000 | Loss 1.72 Ther face. And civery ire head shook the lange's was note my booked she cray. The grance for that the with Lerary swere were, and for young to-the wank the tanger brother whas at a for the requestone-st ================================================== Time elapsed: 7.91 min Iteration 3000 | Loss 1.73 Thou my menal known a purntatieful a might Frent fargefuch by sour that reforned after as as a mists and the countice of the Founk "I among him your for the you glason in?" "I constrance yhabuing a ================================================== Time elapsed: 10.55 min Iteration 4000 | Loss 1.77 The seeantelition pricomer; I have had the passess bestious had be patriender one up thow, such the even the line and that ins show was somen of his openey, but fine had a raghter? "I! And at a sifulra ================================================== Time elapsed: 13.17 min Iteration 5000 | Loss 1.46 The Bask tree. "The intame!" "Neothing and fam and if you brow lisert, to the mouther desk to an to the Gells that immered of the indence an aftionation bud, undering to went remark down off work; doe! ================================================== Time elapsed: 15.83 min Iteration 6000 | Loss 1.64 The Pross. What of moon, and worth her knitting, and is he see myself the was seeper on prisoner her been on him our, and yet in the poors; is stooness of a morned this things more, were benthell name, ================================================== Time elapsed: 18.47 min Iteration 7000 | Loss 1.64 The here an the ferty care it was of the streach. As Miss Pross Borring of her surfounds of comprelegual saken which his returnes, shall in Heaved the arrows of the retore, then for Defarge. Jark, he wa ================================================== Time elapsed: 21.12 min Iteration 8000 | Loss 1.66 Thur and the decients than any. Monsiever such put her cite out over the cermarded and in herce then the repariey who grance the stalled be of the own and conversicted way of his anterom cold the cirse ================================================== Time elapsed: 23.77 min Iteration 9000 | Loss 1.53 Thrat to his man that extenss of the said her and had and world at it, she had was as breat--how had asseet triatile of the pationed, and that worked he works of one and nobainly, and out of that at the ================================================== Time elapsed: 26.43 min Iteration 10000 | Loss 1.30 Ther his moth wooten a new blood, a sile, the lactriden nother were noter, who had from his father to prettorers his fation. Then. He are is him a sloke it soits in him woired to the paper women, maning ================================================== Time elapsed: 29.07 min Iteration 11000 | Loss 1.73 The eighs while Miss Pross was saying a could they last the done by, and pressed to the been hackeful hight in mending, and to the done into-raid to have little faming shall now, with the said to go of ================================================== Time elapsed: 31.72 min Iteration 12000 | Loss 1.68 The here. It were would done. "It alread, I was say?" seen in not in the culles the sunded Miss, sure to be there were the would close he dark see radfe taken it is instend me had done-all I spy so str ================================================== Time elapsed: 34.35 min Iteration 13000 | Loss 1.42 The part, but, at had were tosen in it are of a proined serverently passing the fars, and the friended that a fiffer the knouttial backle and day, list, and from to could my deting; and very dark of the ================================================== Time elapsed: 37.00 min Iteration 14000 | Loss 1.78 Thre it was a days. "You and deperianned of the moved there way, and a socions of the proppiouse and this must a dively? "Yest!" (And in the care befon this there," asked nother, to the two in this ex ================================================== Time elapsed: 39.63 min Iteration 15000 | Loss 1.54 Tho a man in all looking the mannen were trangs; he more at man and in the had believe the sick of their an than the man the prioned in a golderate scattered no stup, and look, all thoused shall law sca ================================================== Time elapsed: 42.28 min Iteration 16000 | Loss 1.52 Thrishe forth, his have like him and words of it is a peeched in the eyes farge what it went exciect the deing and the mittions. The mounged the repalling's citines of mineurmt you not thinks, Charlee ================================================== Time elapsed: 44.93 min Iteration 17000 | Loss 1.58 Thrithest the prisonened I staid be, short, and not morright door with with the mitting to my worthud no paid it." "He do I as a more through a passed and go more. No and me, the far bold to fears and ================================================== Time elapsed: 47.56 min Iteration 18000 | Loss 1.49 Tho you would fro in his intides rather sation and chocal went in the things, asked the have hand of the distened did of the cately roar chifulures. What the His of a not his have pourty the took this l ================================================== Time elapsed: 50.19 min Iteration 19000 | Loss 1.78 Thragges," said some of a puncher in the Gabody old, was a Fants tall to know of the complight--seat more inten asse interancame my any went med Courable hands in that he behing make no will never see t ==================================================