If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it.
#! pip install datasets transformers
If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.
To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow.
First you have to store your authentication token from the Hugging Face website (sign up here if you haven't already!) then execute the following cell and input your username and password:
from huggingface_hub import notebook_login
notebook_login()
Then you need to install Git-LFS. Uncomment the following instructions:
# !apt install git-lfs
Make sure your version of Transformers is at least 4.11.0 since the functionality was introduced in that version:
import transformers
print(transformers.__version__)
We also quickly upload some telemetry - this tells us which examples and software versions are getting used so we know where to prioritize our maintenance efforts. We don't collect (or care about) any personally identifiable information, but if you'd prefer not to be counted, feel free to skip this step or delete this cell entirely.
from transformers.utils import send_example_telemetry
send_example_telemetry("multiple_choice_notebook", framework="pytorch")
In this notebook, we will see how to fine-tune one of the 🤗 Transformers model to a multiple choice task, which is the task of selecting the most plausible inputs in a given selection. The dataset used here is SWAG but you can adapt the pre-processing to any other multiple choice dataset you like, or your own data. SWAG is a dataset about commonsense reasoning, where each example describes a situation then proposes four options that could go after it.
This notebook is built to run with any model checkpoint from the Model Hub as long as that model has a version with a mutiple choice head. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those two parameters, then the rest of the notebook should run smoothly:
model_checkpoint = "bert-base-uncased"
batch_size = 16
We will use the 🤗 Datasets library to download the data. This can be easily done with the functions load_dataset
.
from datasets import load_dataset, load_metric
load_dataset
will cache the dataset to avoid downloading it again the next time you run this cell.
datasets = load_dataset("swag", "regular")
Reusing dataset swag (/home/sgugger/.cache/huggingface/datasets/swag/regular/0.0.0/f9784740e0964a3c799d68cec0d992cc267d3fe94f3e048175eca69d739b980d)
The dataset
object itself is DatasetDict
, which contains one key for the training, validation and test set (with more keys for the mismatched validation and test set in the special case of mnli
).
datasets
DatasetDict({ train: Dataset({ features: ['video-id', 'fold-ind', 'startphrase', 'sent1', 'sent2', 'gold-source', 'ending0', 'ending1', 'ending2', 'ending3', 'label'], num_rows: 73546 }) validation: Dataset({ features: ['video-id', 'fold-ind', 'startphrase', 'sent1', 'sent2', 'gold-source', 'ending0', 'ending1', 'ending2', 'ending3', 'label'], num_rows: 20006 }) test: Dataset({ features: ['video-id', 'fold-ind', 'startphrase', 'sent1', 'sent2', 'gold-source', 'ending0', 'ending1', 'ending2', 'ending3', 'label'], num_rows: 20005 }) })
To access an actual element, you need to select a split first, then give an index:
datasets["train"][0]
{'ending0': 'passes by walking down the street playing their instruments.', 'ending1': 'has heard approaching them.', 'ending2': "arrives and they're outside dancing and asleep.", 'ending3': 'turns the lead singer watches the performance.', 'fold-ind': '3416', 'gold-source': 'gold', 'label': 0, 'sent1': 'Members of the procession walk down the street holding small horn brass instruments.', 'sent2': 'A drum line', 'startphrase': 'Members of the procession walk down the street holding small horn brass instruments. A drum line', 'video-id': 'anetv_jkn6uvmqwh4'}
To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset.
from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML
def show_random_elements(dataset, num_examples=10):
assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
picks = []
for _ in range(num_examples):
pick = random.randint(0, len(dataset)-1)
while pick in picks:
pick = random.randint(0, len(dataset)-1)
picks.append(pick)
df = pd.DataFrame(dataset[picks])
for column, typ in dataset.features.items():
if isinstance(typ, ClassLabel):
df[column] = df[column].transform(lambda i: typ.names[i])
display(HTML(df.to_html()))
show_random_elements(datasets["train"])
ending0 | ending1 | ending2 | ending3 | fold-ind | gold-source | label | sent1 | sent2 | startphrase | video-id | |
---|---|---|---|---|---|---|---|---|---|---|---|
0 | are seated on a field. | are skiing down the slope. | are in a lift. | are pouring out in a man. | 16668 | gold | 1 | A man is wiping the skiboard. | Group of people | A man is wiping the skiboard. Group of people | anetv_JmL6BiuXr_g |
1 | performs stunts inside a gym. | shows several shopping in the water. | continues his skateboard while talking. | is putting a black bike close. | 11424 | gold | 0 | The credits of the video are shown. | A lady | The credits of the video are shown. A lady | anetv_dWyE0o2NetQ |
2 | is emerging into the hospital. | are strewn under water at some wreckage. | tosses the wand together and saunters into the marketplace. | swats him upside down. | 15023 | gen | 1 | Through his binoculars, someone watches a handful of surfers being rolled up into the wave. | Someone | Through his binoculars, someone watches a handful of surfers being rolled up into the wave. Someone | lsmdc3016_CHASING_MAVERICKS-6791 |
3 | spies someone sitting below. | opens the fridge and checks out the photo. | puts a little sheepishly. | staggers up to him. | 5475 | gold | 3 | He tips it upside down, and its little umbrella falls to the floor. | Back inside, someone | He tips it upside down, and its little umbrella falls to the floor. Back inside, someone | lsmdc1008_Spider-Man2-75503 |
4 | carries her to the grave. | laughs as someone styles her hair. | sets down his glass. | stares after her then trudges back up into the street. | 6904 | gen | 1 | Someone kisses her smiling daughter on the cheek and beams back at the camera. | Someone | Someone kisses her smiling daughter on the cheek and beams back at the camera. Someone | lsmdc1028_No_Reservations-83242 |
5 | stops someone and sweeps all the way back from the lower deck to join them. | is being dragged towards the monstrous animation. | beats out many events at the touch of the sword, crawling it. | reaches into a pocket and yanks open the door. | 14089 | gen | 1 | But before he can use his wand, he accidentally rams it up the troll's nostril. | The angry troll | But before he can use his wand, he accidentally rams it up the troll's nostril. The angry troll | lsmdc1053_Harry_Potter_and_the_philosophers_stone-95867 |
6 | sees someone's name in the photo. | gives a surprised look. | kneels down and touches his ripped specs. | spies on someone's clock. | 8407 | gen | 1 | Someone keeps his tired eyes on the road. | Glancing over, he | Someone keeps his tired eyes on the road. Glancing over, he | lsmdc1024_Identity_Thief-82693 |
7 | stops as someone speaks into the camera. | notices how blue his eyes are. | is flung out of the door and knocks the boy over. | flies through the air, its a fireball. | 4523 | gold | 1 | Both people are knocked back a few steps from the force of the collision. | She | Both people are knocked back a few steps from the force of the collision. She | lsmdc0043_Thelma_and_Luise-68271 |
8 | sits close to the river. | have pet's supplies and pets. | pops parked outside the dirt facility, sending up a car highway to catch control. | displays all kinds of power tools and website. | 8112 | gold | 1 | A guy waits in the waiting room with his pet. | A pet store and its van | A guy waits in the waiting room with his pet. A pet store and its van | anetv_9VWoQpg9wqE |
9 | the slender someone, someone turns on the light. | , someone gives them to her boss then dumps some alcohol into dough. | liquids from a bowl, she slams them drunk. | wags his tail as someone returns to the hotel room. | 10867 | gold | 3 | Inside a convenience store, she opens a freezer case. | Dolce | Inside a convenience store, she opens a freezer case. Dolce | lsmdc3090_YOUNG_ADULT-43871 |
Each example in the dataset has a context composed of a first sentence (in the field sent1
) and an introduction to the second sentence (in the field sent2
). Then four possible endings are given (in the fields ending0
, ending1
, ending2
and ending3
) and the model must pick the right one (indicated in the field label
). The following function lets us visualize a give example a bit better:
def show_one(example):
print(f"Context: {example['sent1']}")
print(f" A - {example['sent2']} {example['ending0']}")
print(f" B - {example['sent2']} {example['ending1']}")
print(f" C - {example['sent2']} {example['ending2']}")
print(f" D - {example['sent2']} {example['ending3']}")
print(f"\nGround truth: option {['A', 'B', 'C', 'D'][example['label']]}")
show_one(datasets["train"][0])
Context: Members of the procession walk down the street holding small horn brass instruments. A - A drum line passes by walking down the street playing their instruments. B - A drum line has heard approaching them. C - A drum line arrives and they're outside dancing and asleep. D - A drum line turns the lead singer watches the performance. Ground truth: option A
show_one(datasets["train"][15])
Context: Now it's someone's turn to rain blades on his opponent. A - Someone pats his shoulder and spins wildly. B - Someone lunges forward through the window. C - Someone falls to the ground. D - Someone rolls up his fast run from the water and tosses in the sky. Ground truth: option C
Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers Tokenizer
which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.
To do all of this, we instantiate our tokenizer with the AutoTokenizer.from_pretrained
method, which will ensure:
That vocabulary will be cached, so it's not downloaded again the next time we run the cell.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
We pass along use_fast=True
to the call above to use one of the fast tokenizers (backed by Rust) from the 🤗 Tokenizers library. Those fast tokenizers are available for almost all models, but if you got an error with the previous call, remove that argument.
You can directly call this tokenizer on one sentence or a pair of sentences:
tokenizer("Hello, this one sentence!", "And this sentence goes with it.")
{'input_ids': [101, 7592, 1010, 2023, 2028, 6251, 999, 102, 1998, 2023, 6251, 3632, 2007, 2009, 1012, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in this tutorial if you're interested.
To preprocess our dataset, we will thus need the names of the columns containing the sentence(s). The following dictionary keeps track of the correspondence task to column names:
We can them write the function that will preprocess our samples. The tricky part is to put all the possible pairs of sentences in two big lists before passing them to the tokenizer, then un-flatten the result so that each example has four input ids, attentions masks, etc.
When calling the tokenizer
, we use the argument truncation=True
. This will ensure that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model.
ending_names = ["ending0", "ending1", "ending2", "ending3"]
def preprocess_function(examples):
# Repeat each first sentence four times to go with the four possibilities of second sentences.
first_sentences = [[context] * 4 for context in examples["sent1"]]
# Grab all second sentences possible for each context.
question_headers = examples["sent2"]
second_sentences = [[f"{header} {examples[end][i]}" for end in ending_names] for i, header in enumerate(question_headers)]
# Flatten everything
first_sentences = sum(first_sentences, [])
second_sentences = sum(second_sentences, [])
# Tokenize
tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)
# Un-flatten
return {k: [v[i:i+4] for i in range(0, len(v), 4)] for k, v in tokenized_examples.items()}
This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists of lists for each key: a list of all examples (here 5), then a list of all choices (4) and a list of input IDs (length varying here since we did not apply any padding):
examples = datasets["train"][:5]
features = preprocess_function(examples)
print(len(features["input_ids"]), len(features["input_ids"][0]), [len(x) for x in features["input_ids"][0]])
5 4 [30, 25, 30, 28]
To check we didn't do anything group when grouping all possibilites then unflattening, let's have a look at the decoded inputs for a given example:
idx = 3
[tokenizer.decode(features["input_ids"][idx][i]) for i in range(4)]
['[CLS] a drum line passes by walking down the street playing their instruments. [SEP] members of the procession are playing ping pong and celebrating one left each in quick. [SEP]', '[CLS] a drum line passes by walking down the street playing their instruments. [SEP] members of the procession wait slowly towards the cadets. [SEP]', '[CLS] a drum line passes by walking down the street playing their instruments. [SEP] members of the procession makes a square call and ends by jumping down into snowy streets where fans begin to take their positions. [SEP]', '[CLS] a drum line passes by walking down the street playing their instruments. [SEP] members of the procession play and go back and forth hitting the drums while the audience claps for them. [SEP]']
We can compare it to the ground truth:
show_one(datasets["train"][3])
Context: A drum line passes by walking down the street playing their instruments. A - Members of the procession are playing ping pong and celebrating one left each in quick. B - Members of the procession wait slowly towards the cadets. C - Members of the procession makes a square call and ends by jumping down into snowy streets where fans begin to take their positions. D - Members of the procession play and go back and forth hitting the drums while the audience claps for them. Ground truth: option D
This seems alright, so we can apply this function on all the examples in our dataset, we just use the map
method of our dataset
object we created earlier. This will apply the function on all the elements of all the splits in dataset
, so our training, validation and testing data will be preprocessed in one single command.
encoded_datasets = datasets.map(preprocess_function, batched=True)
Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/swag/regular/0.0.0/f9784740e0964a3c799d68cec0d992cc267d3fe94f3e048175eca69d739b980d/cache-975c81cf12e5b7ac.arrow Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/swag/regular/0.0.0/f9784740e0964a3c799d68cec0d992cc267d3fe94f3e048175eca69d739b980d/cache-d4806d63f1eaf5cd.arrow Loading cached processed dataset at /home/sgugger/.cache/huggingface/datasets/swag/regular/0.0.0/f9784740e0964a3c799d68cec0d992cc267d3fe94f3e048175eca69d739b980d/cache-258c9cd71b0182db.arrow
Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass load_from_cache_file=False
in the call to map
to not use the cached files and force the preprocessing to be applied again.
Note that we passed batched=True
to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently.
Now that our data is ready, we can download the pretrained model and fine-tune it. Since all our task is about mutliple choice, we use the AutoModelForMultipleChoice
class. Like with the tokenizer, the from_pretrained
method will download and cache the model for us.
from transformers import AutoModelForMultipleChoice, TrainingArguments, Trainer
model = AutoModelForMultipleChoice.from_pretrained(model_checkpoint)
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMultipleChoice: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'] - This IS expected if you are initializing BertForMultipleChoice from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing BertForMultipleChoice from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of BertForMultipleChoice were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The warning is telling us we are throwing away some weights (the vocab_transform
and vocab_layer_norm
layers) and randomly initializing some other (the pre_classifier
and classifier
layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.
To instantiate a Trainer
, we will need to define three more things. The most important is the TrainingArguments
, which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:
model_name = model_checkpoint.split("/")[-1]
args = TrainingArguments(
f"{model_name}-finetuned-swag",
evaluation_strategy = "epoch",
learning_rate=5e-5,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
num_train_epochs=3,
weight_decay=0.01,
push_to_hub=True,
)
Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the batch_size
defined at the top of the notebook and customize the number of epochs for training, as well as the weight decay.
The last argument to setup everything so we can push the model to the Hub regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the hub_model_id
argument to set the repo name (it needs to be the full name, including your namespace: for instance "sgugger/bert-finetuned-swag"
or "huggingface/bert-finetuned-swag"
).
Then we need to tell our Trainer
how to form batches from the pre-processed inputs. We haven't done any padding yet because we will pad each batch to the maximum length inside the batch (instead of doing so with the maximum length of the whole dataset). This will be the job of the data collator. A data collator takes a list of examples and converts them to a batch (by, in our case, applying padding). Since there is no data collator in the library that works on our specific problem, we will write one, adapted from the DataCollatorWithPadding
:
from dataclasses import dataclass
from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
from typing import Optional, Union
import torch
@dataclass
class DataCollatorForMultipleChoice:
"""
Data collator that will dynamically pad the inputs for multiple choice received.
"""
tokenizer: PreTrainedTokenizerBase
padding: Union[bool, str, PaddingStrategy] = True
max_length: Optional[int] = None
pad_to_multiple_of: Optional[int] = None
def __call__(self, features):
label_name = "label" if "label" in features[0].keys() else "labels"
labels = [feature.pop(label_name) for feature in features]
batch_size = len(features)
num_choices = len(features[0]["input_ids"])
flattened_features = [[{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features]
flattened_features = sum(flattened_features, [])
batch = self.tokenizer.pad(
flattened_features,
padding=self.padding,
max_length=self.max_length,
pad_to_multiple_of=self.pad_to_multiple_of,
return_tensors="pt",
)
# Un-flatten
batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
# Add back labels
batch["labels"] = torch.tensor(labels, dtype=torch.int64)
return batch
When called on a list of examples, it will flatten all the inputs/attentions masks etc. in big lists that it will pass to the tokenizer.pad
method. This will return a dictionary with big tensors (of shape (batch_size * 4) x seq_length
) that we then unflatten.
We can check this data collator works on a list of features, we just have to make sure to remove all features that are not inputs accepted by our model (something the Trainer
will do automatically for us after):
accepted_keys = ["input_ids", "attention_mask", "label"]
features = [{k: v for k, v in encoded_datasets["train"][i].items() if k in accepted_keys} for i in range(10)]
batch = DataCollatorForMultipleChoice(tokenizer)(features)
Again, all those flatten/un-flatten are sources of potential errors so let's make another sanity check on our inputs:
[tokenizer.decode(batch["input_ids"][8][i].tolist()) for i in range(4)]
['[CLS] someone walks over to the radio. [SEP] someone hands her another phone. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]', '[CLS] someone walks over to the radio. [SEP] someone takes the drink, then holds it. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]', '[CLS] someone walks over to the radio. [SEP] someone looks off then looks at someone. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]', '[CLS] someone walks over to the radio. [SEP] someone stares blearily down at the floor. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]']
show_one(datasets["train"][8])
Context: Someone walks over to the radio. A - Someone hands her another phone. B - Someone takes the drink, then holds it. C - Someone looks off then looks at someone. D - Someone stares blearily down at the floor. Ground truth: option D
All good!
The last thing to define for our Trainer
is how to compute the metrics from the predictions. We need to define a function for this, which will just use the metric
we loaded earlier, the only preprocessing we have to do is to take the argmax of our predicted logits:
import numpy as np
def compute_metrics(eval_predictions):
predictions, label_ids = eval_predictions
preds = np.argmax(predictions, axis=1)
return {"accuracy": (preds == label_ids).astype(np.float32).mean().item()}
Then we just need to pass all of this along with our datasets to the Trainer
:
trainer = Trainer(
model,
args,
train_dataset=encoded_datasets["train"],
eval_dataset=encoded_datasets["validation"],
tokenizer=tokenizer,
data_collator=DataCollatorForMultipleChoice(tokenizer),
compute_metrics=compute_metrics,
)
We can now finetune our model by just calling the train
method:
trainer.train()
Epoch | Training Loss | Validation Loss | Accuracy |
---|---|---|---|
1 | 0.154598 | 0.828017 | 0.766520 |
2 | 0.296633 | 0.667454 | 0.786814 |
3 | 0.111786 | 0.994927 | 0.789363 |
TrainOutput(global_step=6897, training_loss=0.19714653808275168)
You can now upload the result of the training to the Hub, just execute this instruction:
trainer.push_to_hub()
You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier "your-username/the-name-you-picked"
so for instance:
from transformers import AutoModelForMultipleChoice
model = AutoModelForMultipleChoice.from_pretrained("sgugger/my-awesome-model")