01 - Introduction to NLP in fastai2¶

Things work a little differently in fastai2 for text compared to the other two modules (vision and tab)

• We pre-tokenize our text
• The training outline is different
• ULM-FiT get's fine-tuned differently too

In today's lesson we'll explore the high-level API for Text, and understand what makes it different

Starting with the data¶

We're going to use a subset of the IMDB dataset, a sentiment-analysis dataset where you try to see if a review was positive or negative:

What's in our path?

Alright! So what's the general plan for training?

1. Language Model (LM) DataLoaders
2. LM Training
3. Classification DataLoaders
4. Fine-Tune with LM encoder

We're just going to touch on how to use the API, in the next lesson we'll touch on the nitty-gritty of what is going on. For now, let's look at how to build this LM DataLoader using the DataBlock API:

TextBlock¶

As it's a text problem, we'll probably want something like a TextBlock right? Well we can't simply do this:

It won't throw any errors yet, so why?

TextBlock needs to know how we plan to tokenize our words (our tok_tfm), if we want to use a vocab already, if it's a language model, our sequence length, and a few other parameters. So there's a lot going on there!

Along with this there's a few class methods for this too:

So we can see if we don't necissarily want to define everything ourself we can use quick and easy from_df and from_folder methods. We'll use from_df here.

But what? What is this res_col_name? res_col_name is the column where our tokenized text will be added to. This becomes very important as our get_x is going to want to pull from this column rather than where our untokenized input is. So let's build a TextBlock for our problem. So we can see a different output, we'll change our res_col_name to tok_text:

For the rest of our DataBlock, we want our get_x to read that res_col_name column, and our splitter to split our data 90%, 10%:

The more data you can train your LM on, the better:

And now we build the DataLoaders. We need to declare how long our sequence length is going to be here as well:

We'll also set num_workers to 4, the rule of thumb is 4 workers / 1 GPU, source

Let's look at a batch of data, both in a raw form and as a show_batch:

So this looks a bit odd, what happened?

• Tokenized and Numericalized our text (we'll see the latter in a moment)
• LM's want to predict the next word in a sequence

This is where that Numericalization comes into play. Each token/vocabulary gets converted into a number for us to pass to our model, also:

We can see that since we passed in a seq_len of 72, each individual text input is 72 words!

Cool, can we train already?

Training the LM:¶

We have a special Learner for language models, language_model_learner we'll use. The only thing you need to pass in is your DataLoaders, specify the arch, and pass in some metrics:

Potentially, if you have your own pre-trained base model you want to use you can pass in a pretrained_fname, otherwise we'll use a pretrained WikiText103 model.

For our metrics we'll be using both accuracy and Perplexity

What is Perplexity? Teaching our model how to deal with uncertainty in language.

Perplexity metric in NLP is a way to capture the degree of 'uncertainty' a model has in predicting (assigning probabilities to) some text. Lower the entropy (uncertainty), lower the perplexity. If a model, which is trained on good blogs and is being evaluated on similarly looking good blogs, assigns higher probability, we say the model has lower perplexity than a model which assigns lower probability. source

So now let's build our Learner. For an arch we'll use the AWD_LSTM (we'll explore more later on) pretrained on WikiText:

For training our model, we'll simply use fine_tune for a few epochs (mostly due to the small sample size):

That's about what we want in accuracy, ~30-40%, don't expect higher unless you know the model can. Remember: we're predicting the next word given the previous few, a hard task! Now that we have our LM, we want to save those embeddings away.

Embeddings?

These embeddings! This is essentially our ImageNet weights. Along with this, for the downstream task we'll also want our vocab, so we'll actually go ahead and save our DataLoaders too.

How do we do this? Using torch.save and save_encoder:

Our model has been saved to the models directory now. Let's work on our downstream task:

Classification¶

Now classification is what we actually want to do: is this review pos or neg

Let's look at our DataFrame one more time to figure out how to frame our DataBlock:

So we'll want another ColReader to grab our label and to split our data by the is_valid column. First let's get everything ready by loading in those DataLoaders from earlier:

We need these because we already have a vocab to use, and a sequence length:

Then we'll build our DataBlock:

And finally the DataLoaders:

Those with a keen eye will notice how we called .vocab, but before that meant how we grab our classes! So how do we do this here?

Since Categorize (what CategoryBlock adds) is a transform that's stored as an attribute, we can grab our classes that way:

text_classifier_learner¶

Next up is the text_classifier_learner. This is what we'll use to make our classification AWD_LSTM:

Now we have our pretrained embeddings right? Let's look at it in our learn:

It's right there! So we have a load_encoder function that will copy over our weights to there:

Note: It will automatically assume that your model is saved in learn.path in the models folder and has the extention .pth

So now we have our frozen model and our pretrained weights. Let's train!

We're going to use a training methodology that Jeremy and Sebastian Ruder came up with for fine-tuning:

1. Find a learning rate
2. Lower that learning rate each time, slowly unfreezing fitting for 1 epoch at a time
3. Finally unfreeze the model and fit for two epochs:

That's about 4e-2, so I'll agree with it, let's train on this schema:

First one epoch completely frozen:

Then we freeze to -2 and adjust our lr:

This adjuster schema is how we will divide our lr during fitting (we'll also adjust our learning rate outside of it):

Then -3:

73.5% is as high as we got given a subset of only 1,000 texts! Not to shabby, with the full version we can expect around 94.5% accuracy (this was their results in the paper).

Now let's show an example predict, and for fun, compare fastai2 to fastinference

Warning: the ULMFiT model will not export to ONNX

What about fastinference?

We can still shave off some time!

While also still getting our classes back each time too