%load_ext autoreload
%autoreload 2
%matplotlib inline
#export
from exp.nb_11a import *
We will use the IMDB dataset that consists of 50,000 labeled reviews of movies (positive or negative) and 50,000 unlabelled ones.
path = datasets.untar_data(datasets.URLs.IMDB)
path.ls()
[PosixPath('/home/jupyter/.fastai/data/imdb/unsup'),
PosixPath('/home/jupyter/.fastai/data/imdb/tmp_clas'),
PosixPath('/home/jupyter/.fastai/data/imdb/ld.pkl'),
PosixPath('/home/jupyter/.fastai/data/imdb/test'),
PosixPath('/home/jupyter/.fastai/data/imdb/train'),
PosixPath('/home/jupyter/.fastai/data/imdb/README'),
PosixPath('/home/jupyter/.fastai/data/imdb/models'),
PosixPath('/home/jupyter/.fastai/data/imdb/tmp_lm'),
PosixPath('/home/jupyter/.fastai/data/imdb/imdb.vocab')]
We define a subclass of ItemList that will read the texts in the corresponding filenames.
#export
def read_file(fn):
with open(fn, 'r', encoding = 'utf8') as f: return f.read()
class TextList(ItemList):
@classmethod
def from_files(cls, path, extensions='.txt', recurse=True, include=None, **kwargs):
return cls(get_files(path, extensions, recurse=recurse, include=include), path, **kwargs)
def get(self, i):
if isinstance(i, Path): return read_file(i)
return i
Just in case there are some text log files, we restrict the ones we take to the training, test, and unsupervised folders.
il = TextList.from_files(path, include=['train', 'test', 'unsup'])
We should expect a total of 100,000 texts.
len(il.items)
100000
Here is the first one as an example.
txt = il[0]
txt
'Comedian Adam Sandler\'s last theatrical release "I Now Pronounce You Chuck and Larry" served as a loud and proud plea for tolerance of the gay community. The former "Saturday Night Live" funnyman\'s new movie "You Don\'t Mess with the Zohan" (*** out of ****) constitutes his plea for tolerance toward Israeli and Palestinian immigrants in America. These unfortunate people are often punished in America for the crimes of their counterparts in the war-ravaged Middle East. Although "Zohan" advocates a lofty cause, Sandler doesn\'t let his political agenda overshadow his usual crude, below-the-belt, juvenile shenanigans that rely on obscene bodily functions, promiscuous sex, and far-fetched harebrained idiocy. Indeed, the hysterical horseplay that Sandler and company revel in may distract you from the plight of these uprooted, misplaced misfits that had fled to Uncle Sam\'s shores because they believe America is a Utopia. Interestingly, Sandler plays a Jewish counterterrorist agent of the Mossad, Israel\'s secret police, with a hopelessly corny accent. Zohan\'s exploits appear to foreshadow Will Smith\'s upcoming "Hancock." Zohan is the best Jewish secret agent in the whole wide world. He is literally indestructible. He catches bullets in his nose. He can swim faster than a dolphin, and a razor-toothed piranha fish in his bikini swim trunks amuses him.<br /><br />Zohan (Adam Sandler) is cooking fish at the beach when his superiors interrupt his vacation and inform him that the dreaded Arab terrorist, the Phantom (John Turturro of "Transformers"), is up to his old tricks again. Naturally, Zohan is furious! Actually, Zohan captured the Phantom three months ago, but the politicians have exchanged the Phantom for political prisoners. Now, Zohan must nab his nemesis again! The Phantom and Zohan tangle in a spectacular fight in the sea and Zohan doesn\'t survive. In reality, Zohan deliberately fakes his death so that he can immigrate to New York City and realize his life-long dream of cutting hair for Paul Mitchell. Zohan gives himself an obsolete Frankie Avalon haircut, trims his beard, and smuggles himself onto a plane bound for America. If what happens before his flight seems outlandish, once he is on the jet, he spends his time in the cargo hold with two fluffy dogs named "Scrappy" and "Coco." Zohan styles their hair from photos in his Paul Mitchell haircut book.<br /><br />At first, Zohan has no luck getting a job with Paul Mitchell, much less cutting hair. Zohan defends Michael (Nick Swardson of "Reno 911, The Movie") in a street brawl after a motorist blames Michael for his accident with a delivery truck. A grateful Michael invites Zohan to stay with his mother, Gail (Lainie Kazan of "Dayton\'s Devils"), and him. Zohan practices cutting Gail\'s hair when he isn\'t having in lusty sex with her. Eventually, Zohan gets a job sweeping up hair at a salon owned by Dalia (Emmanuelle Chriqui of "Wrong Turn") who as it turns out is a Palestinian. Indeed, Zohan knows about her heritage but doesn\'t let it bother him. One day when one of Dalia\'s hair stylists doesn\'t show up, Zohan takes advantage of her absence to cut hair. Much to Dalia\'s surprise, Zohan wins the allegiance of the over sixty crowd. Older woman line up around the block to have him fashion their hair. After each session, Zohan takes each older lady in the back and assuages their sexual appetites.<br /><br />Meanwhile, a millionaire real estate developer Walbridge (Michael Buffer of "Rocky Balboa") hikes the rent to force Dalia and others like her out of her store to make way for his mall with a roller-coaster. Zohan surprises both Dalia and Walbridge\'s people and forks over the money for her to pay the rent. An angry Walbridge contacts a white supremacy group to ignite a neighborhood war between the Israelis and Palestinians. This happens about the same time that Zohan falls in love with Dalia. Perennial Sandler cohort Rob Schneider of "Deuce Bigalow" appears as a cretinous Palestinian named Salim who doesn\'t know the difference between nitroglycerin and Neosporin. He tries to blow up Zohan for an old grudge. It seems Zohan beat Salim up and stole his goat.<br /><br />"You Don\'t Mess with the Zohan" qualifies as a surreal comedy. Scenarists Robert Smigel of "Saturday Night Live," Judd Apatow of "The 40-Year Old Virgin," and Sandler himself vigorously ignore the laws of logic in this zany comedy. The movie that most closely resembles "Zohan" is "Little Nicky," because both characters boast supernatural abilities. "You Don\'t Mess with the Zohan" will keep Adam Sandler fans in stitches.'
For text classification, we will split by the grand parent folder as before, but for language modeling, we take all the texts and just put 10% aside.
sd = SplitData.split_by_func(il, partial(random_splitter, p_valid=0.1))
sd
SplitData
Train: TextList (89885 items)
[PosixPath('/home/jupyter/.fastai/data/imdb/unsup/30860_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/36250_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/24690_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/21770_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/9740_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/40778_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/44512_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/22672_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/25946_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/40866_0.txt')...]
Path: /home/jupyter/.fastai/data/imdb
Valid: TextList (10115 items)
[PosixPath('/home/jupyter/.fastai/data/imdb/unsup/1041_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/38186_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/16367_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/47167_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/58_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/49861_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/306_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/18238_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/34952_0.txt'), PosixPath('/home/jupyter/.fastai/data/imdb/unsup/24288_0.txt')...]
Path: /home/jupyter/.fastai/data/imdb
We need to tokenize the dataset first, which is splitting a sentence in individual tokens. Those tokens are the basic words or punctuation signs with a few tweaks: don't for instance is split between do and n't. We will use a processor for this, in conjunction with the spacy library.
#export
import spacy,html
Before even tokenizeing, we will apply a bit of preprocessing on the texts to clean them up (we saw the one up there had some HTML code). These rules are applied before we split the sentences in tokens.
#export
#special tokens
UNK, PAD, BOS, EOS, TK_REP, TK_WREP, TK_UP, TK_MAJ = "xxunk xxpad xxbos xxeos xxrep xxwrep xxup xxmaj".split()
def sub_br(t):
"Replaces the <br /> by \n"
re_br = re.compile(r'<\s*br\s*/?>', re.IGNORECASE)
return re_br.sub("\n", t)
def spec_add_spaces(t):
"Add spaces around / and #"
return re.sub(r'([/#])', r' \1 ', t)
def rm_useless_spaces(t):
"Remove multiple spaces"
return re.sub(' {2,}', ' ', t)
def replace_rep(t):
"Replace repetitions at the character level: cccc -> TK_REP 4 c"
def _replace_rep(m:Collection[str]) -> str:
c,cc = m.groups()
return f' {TK_REP} {len(cc)+1} {c} '
re_rep = re.compile(r'(\S)(\1{3,})')
return re_rep.sub(_replace_rep, t)
def replace_wrep(t):
"Replace word repetitions: word word word -> TK_WREP 3 word"
def _replace_wrep(m:Collection[str]) -> str:
c,cc = m.groups()
return f' {TK_WREP} {len(cc.split())+1} {c} '
re_wrep = re.compile(r'(\b\w+\W+)(\1{3,})')
return re_wrep.sub(_replace_wrep, t)
def fixup_text(x):
"Various messy things we've seen in documents"
re1 = re.compile(r' +')
x = x.replace('#39;', "'").replace('amp;', '&').replace('#146;', "'").replace(
'nbsp;', ' ').replace('#36;', '$').replace('\\n', "\n").replace('quot;', "'").replace(
'<br />', "\n").replace('\\"', '"').replace('<unk>',UNK).replace(' @.@ ','.').replace(
' @-@ ','-').replace('\\', ' \\ ')
return re1.sub(' ', html.unescape(x))
default_pre_rules = [fixup_text, replace_rep, replace_wrep, spec_add_spaces, rm_useless_spaces, sub_br]
default_spec_tok = [UNK, PAD, BOS, EOS, TK_REP, TK_WREP, TK_UP, TK_MAJ]
replace_rep('cccc')
' xxrep 4 c '
replace_wrep('word word word word word ')
' xxwrep 5 word '
These rules are applies after the tokenization on the list of tokens.
#export
def replace_all_caps(x):
"Replace tokens in ALL CAPS by their lower version and add `TK_UP` before."
res = []
for t in x:
if t.isupper() and len(t) > 1: res.append(TK_UP); res.append(t.lower())
else: res.append(t)
return res
def deal_caps(x):
"Replace all Capitalized tokens in by their lower version and add `TK_MAJ` before."
res = []
for t in x:
if t == '': continue
if t[0].isupper() and len(t) > 1 and t[1:].islower(): res.append(TK_MAJ)
res.append(t.lower())
return res
def add_eos_bos(x): return [BOS] + x + [EOS]
default_post_rules = [deal_caps, replace_all_caps, add_eos_bos]
replace_all_caps(['I', 'AM', 'SHOUTING'])
['I', 'xxup', 'am', 'xxup', 'shouting']
deal_caps(['My', 'name', 'is', 'Jeremy'])
['xxmaj', 'my', 'name', 'is', 'xxmaj', 'jeremy']
Since tokenizing and applying those rules takes a bit of time, we'll parallelize it using ProcessPoolExecutor to go faster.
#export
from spacy.symbols import ORTH
from concurrent.futures import ProcessPoolExecutor
def parallel(func, arr, max_workers=4):
if max_workers<2: results = list(progress_bar(map(func, enumerate(arr)), total=len(arr)))
else:
with ProcessPoolExecutor(max_workers=max_workers) as ex:
return list(progress_bar(ex.map(func, enumerate(arr)), total=len(arr)))
if any([o is not None for o in results]): return results
#export
class TokenizeProcessor(Processor):
def __init__(self, lang="en", chunksize=2000, pre_rules=None, post_rules=None, max_workers=4):
self.chunksize,self.max_workers = chunksize,max_workers
self.tokenizer = spacy.blank(lang).tokenizer
for w in default_spec_tok:
self.tokenizer.add_special_case(w, [{ORTH: w}])
self.pre_rules = default_pre_rules if pre_rules is None else pre_rules
self.post_rules = default_post_rules if post_rules is None else post_rules
def proc_chunk(self, args):
i,chunk = args
chunk = [compose(t, self.pre_rules) for t in chunk]
docs = [[d.text for d in doc] for doc in self.tokenizer.pipe(chunk)]
docs = [compose(t, self.post_rules) for t in docs]
return docs
def __call__(self, items):
toks = []
if isinstance(items[0], Path): items = [read_file(i) for i in items]
chunks = [items[i: i+self.chunksize] for i in (range(0, len(items), self.chunksize))]
toks = parallel(self.proc_chunk, chunks, max_workers=self.max_workers)
return sum(toks, [])
def proc1(self, item): return self.proc_chunk([item])[0]
def deprocess(self, toks): return [self.deproc1(tok) for tok in toks]
def deproc1(self, tok): return " ".join(tok)
tp = TokenizeProcessor()
txt[:250]
'Comedian Adam Sandler\'s last theatrical release "I Now Pronounce You Chuck and Larry" served as a loud and proud plea for tolerance of the gay community. The former "Saturday Night Live" funnyman\'s new movie "You Don\'t Mess with the Zohan" (*** out o'
' • '.join(tp(il[:100])[0])[:400]
'xxbos • xxmaj • comedian • xxmaj • adam • xxmaj • sandler • \'s • last • theatrical • release • " • i • xxmaj • now • xxmaj • pronounce • xxmaj • you • xxmaj • chuck • and • xxmaj • larry • " • served • as • a • loud • and • proud • plea • for • tolerance • of • the • gay • community • . • xxmaj • the • former • " • xxmaj • saturday • xxmaj • night • xxmaj • live • " • funnyman • \'s • new • movie •'
Once we have tokenized our texts, we replace each token by an individual number, this is called numericalizing. Again, we do this with a processor (not so different from the CategoryProcessor).
#export
import collections
class NumericalizeProcessor(Processor):
def __init__(self, vocab=None, max_vocab=60000, min_freq=2):
self.vocab,self.max_vocab,self.min_freq = vocab,max_vocab,min_freq
def __call__(self, items):
#The vocab is defined on the first use.
if self.vocab is None:
freq = Counter(p for o in items for p in o)
self.vocab = [o for o,c in freq.most_common(self.max_vocab) if c >= self.min_freq]
for o in reversed(default_spec_tok):
if o in self.vocab: self.vocab.remove(o)
self.vocab.insert(0, o)
if getattr(self, 'otoi', None) is None:
self.otoi = collections.defaultdict(int,{v:k for k,v in enumerate(self.vocab)})
return [self.proc1(o) for o in items]
def proc1(self, item): return [self.otoi[o] for o in item]
def deprocess(self, idxs):
assert self.vocab is not None
return [self.deproc1(idx) for idx in idxs]
def deproc1(self, idx): return [self.vocab[i] for i in idx]
When we do language modeling, we will infer the labels from the text during training, so there's no need to label. The training loop expects labels however, so we need to add dummy ones.
proc_tok,proc_num = TokenizeProcessor(max_workers=8),NumericalizeProcessor()
%time ll = label_by_func(sd, lambda x: 0, proc_x = [proc_tok,proc_num])
CPU times: user 23.9 s, sys: 5.06 s, total: 29 s Wall time: 3min 13s
Once the items have been processed they will become list of numbers, we can still access the underlying raw data in x_obj (or y_obj for the targets, but we don't have any here).
ll.train.x_obj(0)
'xxbos xxmaj comedian xxmaj adam xxmaj sandler \'s last theatrical release " i xxmaj now xxmaj pronounce xxmaj you xxmaj chuck and xxmaj larry " served as a loud and proud plea for tolerance of the gay community . xxmaj the former " xxmaj saturday xxmaj night xxmaj live " funnyman \'s new movie " xxmaj you xxmaj do n\'t xxmaj mess with the xxmaj zohan " ( * * * out of xxrep 4 * ) constitutes his plea for tolerance toward xxmaj israeli and xxmaj palestinian immigrants in xxmaj america . xxmaj these unfortunate people are often punished in xxmaj america for the crimes of their counterparts in the war - ravaged xxmaj middle xxmaj east . xxmaj although " xxmaj zohan " advocates a lofty cause , xxmaj sandler does n\'t let his political agenda overshadow his usual crude , below - the - belt , juvenile shenanigans that rely on obscene bodily functions , promiscuous sex , and far - fetched harebrained idiocy . xxmaj indeed , the hysterical horseplay that xxmaj sandler and company revel in may distract you from the plight of these uprooted , misplaced misfits that had fled to xxmaj uncle xxmaj sam \'s shores because they believe xxmaj america is a xxmaj utopia . xxmaj interestingly , xxmaj sandler plays a xxmaj jewish xxunk agent of the xxmaj mossad , xxmaj israel \'s secret police , with a hopelessly corny accent . xxmaj zohan \'s exploits appear to foreshadow xxmaj will xxmaj smith \'s upcoming " xxmaj hancock . " xxmaj zohan is the best xxmaj jewish secret agent in the whole wide world . xxmaj he is literally indestructible . xxmaj he catches bullets in his nose . xxmaj he can swim faster than a dolphin , and a razor - toothed piranha fish in his bikini swim trunks amuses him . \n\n xxmaj zohan ( xxmaj adam xxmaj sandler ) is cooking fish at the beach when his superiors interrupt his vacation and inform him that the dreaded xxmaj arab terrorist , the xxmaj phantom ( xxmaj john xxmaj turturro of " xxmaj transformers " ) , is up to his old tricks again . xxmaj naturally , xxmaj zohan is furious ! xxmaj actually , xxmaj zohan captured the xxmaj phantom three months ago , but the politicians have exchanged the xxmaj phantom for political prisoners . xxmaj now , xxmaj zohan must nab his nemesis again ! xxmaj the xxmaj phantom and xxmaj zohan tangle in a spectacular fight in the sea and xxmaj zohan does n\'t survive . xxmaj in reality , xxmaj zohan deliberately fakes his death so that he can immigrate to xxmaj new xxmaj york xxmaj city and realize his life - long dream of cutting hair for xxmaj paul xxmaj mitchell . xxmaj zohan gives himself an obsolete xxmaj frankie xxmaj avalon haircut , trims his beard , and smuggles himself onto a plane bound for xxmaj america . xxmaj if what happens before his flight seems outlandish , once he is on the jet , he spends his time in the cargo hold with two fluffy dogs named " xxmaj scrappy " and " xxmaj coco . " xxmaj zohan styles their hair from photos in his xxmaj paul xxmaj mitchell haircut book . \n\n xxmaj at first , xxmaj zohan has no luck getting a job with xxmaj paul xxmaj mitchell , much less cutting hair . xxmaj zohan defends xxmaj michael ( xxmaj nick xxmaj swardson of " xxmaj reno 911 , xxmaj the xxmaj movie " ) in a street brawl after a motorist blames xxmaj michael for his accident with a delivery truck . a grateful xxmaj michael invites xxmaj zohan to stay with his mother , xxmaj gail ( xxmaj lainie xxmaj kazan of " xxmaj dayton \'s xxmaj devils " ) , and him . xxmaj zohan practices cutting xxmaj gail \'s hair when he is n\'t having in lusty sex with her . xxmaj eventually , xxmaj zohan gets a job sweeping up hair at a salon owned by xxmaj dalia ( xxmaj emmanuelle xxmaj chriqui of " xxmaj wrong xxmaj turn " ) who as it turns out is a xxmaj palestinian . xxmaj indeed , xxmaj zohan knows about her heritage but does n\'t let it bother him . xxmaj one day when one of xxmaj dalia \'s hair stylists does n\'t show up , xxmaj zohan takes advantage of her absence to cut hair . xxmaj much to xxmaj dalia \'s surprise , xxmaj zohan wins the allegiance of the over sixty crowd . xxmaj older woman line up around the block to have him fashion their hair . xxmaj after each session , xxmaj zohan takes each older lady in the back and xxunk their sexual appetites . \n\n xxmaj meanwhile , a millionaire real estate developer xxmaj walbridge ( xxmaj michael xxmaj buffer of " xxmaj rocky xxmaj balboa " ) hikes the rent to force xxmaj dalia and others like her out of her store to make way for his mall with a roller - coaster . xxmaj zohan surprises both xxmaj dalia and xxmaj walbridge \'s people and forks over the money for her to pay the rent . xxmaj an angry xxmaj walbridge contacts a white supremacy group to ignite a neighborhood war between the xxmaj israelis and xxmaj palestinians . xxmaj this happens about the same time that xxmaj zohan falls in love with xxmaj dalia . xxmaj perennial xxmaj sandler cohort xxmaj rob xxmaj schneider of " xxmaj deuce xxmaj bigalow " appears as a cretinous xxmaj palestinian named xxmaj salim who does n\'t know the difference between nitroglycerin and xxmaj xxunk . xxmaj he tries to blow up xxmaj zohan for an old grudge . xxmaj it seems xxmaj zohan beat xxmaj salim up and stole his goat . \n\n " xxmaj you xxmaj do n\'t xxmaj mess with the xxmaj zohan " qualifies as a surreal comedy . xxmaj scenarists xxmaj robert xxmaj smigel of " xxmaj saturday xxmaj night xxmaj live , " xxmaj judd xxmaj apatow of " xxmaj the 40-year xxmaj old xxmaj virgin , " and xxmaj sandler himself vigorously ignore the laws of logic in this zany comedy . xxmaj the movie that most closely resembles " xxmaj zohan " is " xxmaj little xxmaj nicky , " because both characters boast supernatural abilities . " xxmaj you xxmaj do n\'t xxmaj mess with the xxmaj zohan " will keep xxmaj adam xxmaj sandler fans in stitches . xxeos'
Since the preprocessing takes time, we save the intermediate result using pickle. Don't use any lambda functions in your processors or they won't be able to pickle.
pickle.dump(ll, open(path/'ld.pkl', 'wb'))
ll = pickle.load(open(path/'ld.pkl', 'rb'))
We have a bit of work to convert our LabelList in a DataBunch as we don't just want batches of IMDB reviews. We want to stream through all the texts concatenated. We also have to prepare the targets that are the newt words in the text. All of this is done with the next object called LM_PreLoader. At the beginning of each epoch, it'll shuffle the articles (if shuffle=True) and create a big stream by concatenating all of them. We divide this big stream in bs smaller streams. That we will read in chunks of bptt length.
# Just using those for illustration purposes, they're not used otherwise.
from IPython.display import display,HTML
import pandas as pd
Let's say our stream is:
stream = """
In this notebook, we will go back over the example of classifying movie reviews we studied in part 1 and dig deeper under the surface.
First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we'll have another example of the Processor used in the data block API.
Then we will study how we build a language model and train it.\n
"""
tokens = np.array(tp([stream])[0])
Then if we split it in 6 batches it would give something like this:
bs,seq_len = 6,15
d_tokens = np.array([tokens[i*seq_len:(i+1)*seq_len] for i in range(bs)])
df = pd.DataFrame(d_tokens)
display(HTML(df.to_html(index=False,header=None)))
| xxbos | \n | xxmaj | in | this | notebook | , | we | will | go | back | over | the | example | of |
| classifying | movie | reviews | we | studied | in | part | 1 | and | dig | deeper | under | the | surface | . |
| \n | xxmaj | first | we | will | look | at | the | processing | steps | necessary | to | convert | text | into |
| numbers | and | how | to | customize | it | . | xxmaj | by | doing | this | , | we | 'll | have |
| another | example | of | the | xxmaj | processor | used | in | the | data | block | api | . | \n | xxmaj |
| then | we | will | study | how | we | build | a | language | model | and | train | it | . | \n\n |
Then if we have a bptt of 5, we would go over those three batches.
bs,bptt = 6,5
for k in range(3):
d_tokens = np.array([tokens[i*seq_len + k*bptt:i*seq_len + (k+1)*bptt] for i in range(bs)])
df = pd.DataFrame(d_tokens)
display(HTML(df.to_html(index=False,header=None)))
| xxbos | \n | xxmaj | in | this |
| classifying | movie | reviews | we | studied |
| \n | xxmaj | first | we | will |
| numbers | and | how | to | customize |
| another | example | of | the | xxmaj |
| then | we | will | study | how |
| notebook | , | we | will | go |
| in | part | 1 | and | dig |
| look | at | the | processing | steps |
| it | . | xxmaj | by | doing |
| processor | used | in | the | data |
| we | build | a | language | model |
| back | over | the | example | of |
| deeper | under | the | surface | . |
| necessary | to | convert | text | into |
| this | , | we | 'll | have |
| block | api | . | \n | xxmaj |
| and | train | it | . | \n\n |
#export
class LM_PreLoader():
def __init__(self, data, bs=64, bptt=70, shuffle=False):
self.data,self.bs,self.bptt,self.shuffle = data,bs,bptt,shuffle
total_len = sum([len(t) for t in data.x])
self.n_batch = total_len // bs
self.batchify()
def __len__(self): return ((self.n_batch-1) // self.bptt) * self.bs
def __getitem__(self, idx):
source = self.batched_data[idx % self.bs]
seq_idx = (idx // self.bs) * self.bptt
return source[seq_idx:seq_idx+self.bptt],source[seq_idx+1:seq_idx+self.bptt+1]
def batchify(self):
texts = self.data.x
if self.shuffle: texts = texts[torch.randperm(len(texts))]
stream = torch.cat([tensor(t) for t in texts])
self.batched_data = stream[:self.n_batch * self.bs].view(self.bs, self.n_batch)
dl = DataLoader(LM_PreLoader(ll.valid, shuffle=True), batch_size=64)
Let's check it all works ok: x1, y1, x2 and y2 should all be of size bs by bptt. The texts in each row of x1 should continue in x2. y1 and y2 should have the same texts as their x counterpart, shifted of one position to the right.
iter_dl = iter(dl)
x1,y1 = next(iter_dl)
x2,y2 = next(iter_dl)
x1.size(),y1.size()
vocab = proc_num.vocab
" ".join(vocab[o] for o in x1[0])
" ".join(vocab[o] for o in y1[0])
" ".join(vocab[o] for o in x2[0])
And let's prepare some convenience function to do this quickly.
#export
def get_lm_dls(train_ds, valid_ds, bs, bptt, **kwargs):
return (DataLoader(LM_PreLoader(train_ds, bs, bptt, shuffle=True), batch_size=bs, **kwargs),
DataLoader(LM_PreLoader(valid_ds, bs, bptt, shuffle=False), batch_size=2*bs, **kwargs))
def lm_databunchify(sd, bs, bptt, **kwargs):
return DataBunch(*get_lm_dls(sd.train, sd.valid, bs, bptt, **kwargs))
bs,bptt = 64,70
data = lm_databunchify(ll, bs, bptt)
When we will want to tackle classification, gathering the data will be a bit different: first we will label our texts with the folder they come from, and then we will need to apply padding to batch them together. To avoid mixing very long texts with very short ones, we will also use Sampler to sort (with a bit of randomness for the training set) our samples by length.
First the data block API calls shold look familiar.
proc_cat = CategoryProcessor()
il = TextList.from_files(path, include=['train', 'test'])
sd = SplitData.split_by_func(il, partial(grandparent_splitter, valid_name='test'))
ll = label_by_func(sd, parent_labeler, proc_x = [proc_tok, proc_num], proc_y=proc_cat)
pickle.dump(ll, open(path/'ll_clas.pkl', 'wb'))
ll = pickle.load(open(path/'ll_clas.pkl', 'rb'))
Let's check the labels seem consistent with the texts.
[(ll.train.x_obj(i), ll.train.y_obj(i)) for i in [1,12552]]
We saw samplers in notebook 03. For the validation set, we will simply sort the samples by length, and we begin with the longest ones for memory reasons (it's better to always have the biggest tensors first).
#export
from torch.utils.data import Sampler
class SortSampler(Sampler):
def __init__(self, data_source, key): self.data_source,self.key = data_source,key
def __len__(self): return len(self.data_source)
def __iter__(self):
return iter(sorted(list(range(len(self.data_source))), key=self.key, reverse=True))
For the training set, we want some kind of randomness on top of this. So first, we shuffle the texts and build megabatches of size 50 * bs. We sort those megabatches by length before splitting them in 50 minibatches. That way we will have randomized batches of roughly the same length.
Then we make sure to have the biggest batch first and shuffle the order of the other batches. We also make sure the last batch stays at the end because its size is probably lower than batch size.
#export
class SortishSampler(Sampler):
def __init__(self, data_source, key, bs):
self.data_source,self.key,self.bs = data_source,key,bs
def __len__(self) -> int: return len(self.data_source)
def __iter__(self):
idxs = torch.randperm(len(self.data_source))
megabatches = [idxs[i:i+self.bs*50] for i in range(0, len(idxs), self.bs*50)]
sorted_idx = torch.cat([tensor(sorted(s, key=self.key, reverse=True)) for s in megabatches])
batches = [sorted_idx[i:i+self.bs] for i in range(0, len(sorted_idx), self.bs)]
max_idx = torch.argmax(tensor([self.key(ck[0]) for ck in batches])) # find the chunk with the largest key,
batches[0],batches[max_idx] = batches[max_idx],batches[0] # then make sure it goes first.
batch_idxs = torch.randperm(len(batches)-2)
sorted_idx = torch.cat([batches[i+1] for i in batch_idxs]) if len(batches) > 1 else LongTensor([])
sorted_idx = torch.cat([batches[0], sorted_idx, batches[-1]])
return iter(sorted_idx)
Padding: we had the padding token (that as an id of 1) at the end of each sequence to make them all the same size when batching them. Note that we need padding at the end to be able to use PyTorch convenience functions that will let us ignore that padding (see 12c).
#export
def pad_collate(samples, pad_idx=1, pad_first=False):
max_len = max([len(s[0]) for s in samples])
res = torch.zeros(len(samples), max_len).long() + pad_idx
for i,s in enumerate(samples):
if pad_first: res[i, -len(s[0]):] = LongTensor(s[0])
else: res[i, :len(s[0]) ] = LongTensor(s[0])
return res, tensor([s[1] for s in samples])
bs = 64
train_sampler = SortishSampler(ll.train.x, key=lambda t: len(ll.train[int(t)][0]), bs=bs)
train_dl = DataLoader(ll.train, batch_size=bs, sampler=train_sampler, collate_fn=pad_collate)
iter_dl = iter(train_dl)
x,y = next(iter_dl)
lengths = []
for i in range(x.size(0)): lengths.append(x.size(1) - (x[i]==1).sum().item())
lengths[:5], lengths[-1]
The last one is the minimal length. This is the first batch so it has the longest sequence, but if look at the next one that is more random, we see lengths are roughly the sames.
x,y = next(iter_dl)
lengths = []
for i in range(x.size(0)): lengths.append(x.size(1) - (x[i]==1).sum().item())
lengths[:5], lengths[-1]
We can see the padding at the end:
x
And we add a convenience function:
#export
def get_clas_dls(train_ds, valid_ds, bs, **kwargs):
train_sampler = SortishSampler(train_ds.x, key=lambda t: len(train_ds.x[t]), bs=bs)
valid_sampler = SortSampler(valid_ds.x, key=lambda t: len(valid_ds.x[t]))
return (DataLoader(train_ds, batch_size=bs, sampler=train_sampler, collate_fn=pad_collate, **kwargs),
DataLoader(valid_ds, batch_size=bs*2, sampler=valid_sampler, collate_fn=pad_collate, **kwargs))
def clas_databunchify(sd, bs, **kwargs):
return DataBunch(*get_clas_dls(sd.train, sd.valid, bs, **kwargs))
bs,bptt = 64,70
data = clas_databunchify(ll, bs)
!python notebook2script.py 12_text.ipynb