In this notebook, we will see how to fine-tune one of the 🤗 Transformers model to a token classification task, which is the task of predicting a label for each token.
The most common token classification tasks are:
We will see how to easily load a dataset for these kinds of tasks and use the Trainer
API to fine-tune a model on it.
This notebook is built to run on any token classification task, with any model checkpoint from the Model Hub as long as that model has a version with a token classification head and a fast tokenizer (check on this table if this is the case). It might just need some small adjustments if you decide to use a different dataset than the one used here. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those three parameters, then the rest of the notebook should run smoothly:
task = "ner" # Should be one of "ner", "pos" or "chunk"
# model_checkpoint = "neuralmind/bert-base-portuguese-cased"
# model_checkpoint = "neuralmind/bert-large-portuguese-cased"
model_checkpoint = "pierreguillou/bert-base-cased-pt-lenerbr"
from google.colab import drive
drive.mount('/content/drive', force_remount=True)
Mounted at /content/drive
If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it.
%%capture
!pip install datasets transformers seqeval
If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.
To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow.
First you have to store your authentication token from the Hugging Face website (sign up here if you haven't already!) then execute the following cell and input your username and password:
from huggingface_hub import notebook_login
notebook_login()
Login successful
Your token has been saved to /root/.huggingface/token
Authenticated through git-credential store but this isn't the helper defined on your machine.
You might have to re-authenticate when pushing to the Hugging Face Hub. Run the following command in your terminal in case you want to set this credential helper as the default
git config --global credential.helper store
Then you need to install Git-LFS. Uncomment the following instructions:
%%capture
!apt install git-lfs
Make sure your version of Transformers is at least 4.11.0 since the functionality was introduced in that version:
import transformers
print(transformers.__version__)
# 4.13.0
4.13.0
import datasets
print(datasets.__version__)
# 1.17.0
1.17.0
import pathlib
from pathlib import Path
import pandas as pd
from datasets import Dataset, DatasetDict
You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs here.
We will use the 🤗 Datasets library to download the data and get the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the functions load_dataset
and load_metric
.
from datasets import load_dataset, load_metric
For our example here, we'll use the LeNER-Br dataset. The notebook should work with any token classification dataset provided by the 🤗 Datasets library. If you're using your own dataset defined from a JSON or csv file (see the Datasets documentation on how to load them), it might need some adjustments in the names of the columns used.
datasets = load_dataset("lener_br")
Downloading: 0%| | 0.00/2.20k [00:00<?, ?B/s]
Downloading: 0%| | 0.00/1.51k [00:00<?, ?B/s]
Downloading and preparing dataset lener_br/lener_br (download: 2.84 MiB, generated: 5.27 MiB, post-processed: Unknown size, total: 8.12 MiB) to /root/.cache/huggingface/datasets/lener_br/lener_br/1.0.0/4a8c97e6813b5c2d85a50faf0a3e6c24ea82f4a9044e6e9e8b24997d27399382...
0%| | 0/3 [00:00<?, ?it/s]
Downloading: 0%| | 0.00/436k [00:00<?, ?B/s]
Downloading: 0%| | 0.00/70.8k [00:00<?, ?B/s]
Downloading: 0%| | 0.00/94.5k [00:00<?, ?B/s]
0%| | 0/3 [00:00<?, ?it/s]
0 examples [00:00, ? examples/s]
0 examples [00:00, ? examples/s]
0 examples [00:00, ? examples/s]
Dataset lener_br downloaded and prepared to /root/.cache/huggingface/datasets/lener_br/lener_br/1.0.0/4a8c97e6813b5c2d85a50faf0a3e6c24ea82f4a9044e6e9e8b24997d27399382. Subsequent calls will reuse this data.
0%| | 0/3 [00:00<?, ?it/s]
The datasets
object itself is DatasetDict
, which contains one key for the training, validation and test set.
datasets
DatasetDict({ train: Dataset({ features: ['id', 'tokens', 'ner_tags'], num_rows: 7828 }) validation: Dataset({ features: ['id', 'tokens', 'ner_tags'], num_rows: 1177 }) test: Dataset({ features: ['id', 'tokens', 'ner_tags'], num_rows: 1390 }) })
We can see the training, validation and test sets all have a column for the tokens (the input texts split into words) and one column of labels for each kind of task we introduced before.
To access an actual element, you need to select a split first, then give an index:
datasets["train"][0]
{'id': '0', 'ner_tags': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'tokens': ['EMENTA', ':', 'APELAÇÃO', 'CÍVEL', '-', 'AÇÃO', 'DE', 'INDENIZAÇÃO', 'POR', 'DANOS', 'MORAIS', '-', 'PRELIMINAR', '-', 'ARGUIDA', 'PELO', 'MINISTÉRIO', 'PÚBLICO', 'EM', 'GRAU', 'RECURSAL', '-', 'NULIDADE', '-', 'AUSÊNCIA', 'DE', 'INTERVENÇÃO', 'DO', 'PARQUET', 'NA', 'INSTÂNCIA', 'A', 'QUO', '-', 'PRESENÇA', 'DE', 'INCAPAZ', '-', 'PREJUÍZO', 'EXISTENTE', '-', 'PRELIMINAR', 'ACOLHIDA', '-', 'NULIDADE', 'RECONHECIDA', '.']}
The labels are already coded as integer ids to be easily usable by our model, but the correspondence with the actual categories is stored in the features
of the dataset:
datasets["train"].features[f"ner_tags"]
Sequence(feature=ClassLabel(num_classes=13, names=['O', 'B-ORGANIZACAO', 'I-ORGANIZACAO', 'B-PESSOA', 'I-PESSOA', 'B-TEMPO', 'I-TEMPO', 'B-LOCAL', 'I-LOCAL', 'B-LEGISLACAO', 'I-LEGISLACAO', 'B-JURISPRUDENCIA', 'I-JURISPRUDENCIA'], names_file=None, id=None), length=-1, id=None)
So for the NER tags, 0 corresponds to 'O', 1 to 'B-PER' etc... On top of the 'O' (which means no special entity), there are four labels for NER here, each prefixed with 'B-' (for beginning) or 'I-' (for intermediate), that indicate if the token is the first one for the current group with the label or not:
Since the labels are lists of ClassLabel
, the actual names of the labels are nested in the feature
attribute of the object above:
label_list = datasets["train"].features[f"{task}_tags"].feature.names
label_list
['O', 'B-ORGANIZACAO', 'I-ORGANIZACAO', 'B-PESSOA', 'I-PESSOA', 'B-TEMPO', 'I-TEMPO', 'B-LOCAL', 'I-LOCAL', 'B-LEGISLACAO', 'I-LEGISLACAO', 'B-JURISPRUDENCIA', 'I-JURISPRUDENCIA']
To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset (automatically decoding the labels in passing).
from datasets import ClassLabel, Sequence
import random
import pandas as pd
from IPython.display import display, HTML
def show_random_elements(dataset, num_examples=10):
assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
picks = []
for _ in range(num_examples):
pick = random.randint(0, len(dataset)-1)
while pick in picks:
pick = random.randint(0, len(dataset)-1)
picks.append(pick)
df = pd.DataFrame(dataset[picks])
for column, typ in dataset.features.items():
if isinstance(typ, ClassLabel):
df[column] = df[column].transform(lambda i: typ.names[i])
elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):
df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])
display(HTML(df.to_html()))
show_random_elements(datasets["train"])
id | tokens | ner_tags | |
---|---|---|---|
0 | 1567 | [POSSIBILIDADE, .] | [O, O] |
1 | 4517 | [Relator, :, Ministro, Napoleão, Nunes, Maia, Filho, .] | [O, O, O, B-PESSOA, I-PESSOA, I-PESSOA, I-PESSOA, O] |
2 | 4072 | [Publique-se, e, encaminhe-se, ao, Instituto, de, Defesa, do, Consumidor, -, PROCON, ,, para, as, providências, pertinentes, .] | [O, O, O, O, B-ORGANIZACAO, I-ORGANIZACAO, I-ORGANIZACAO, I-ORGANIZACAO, I-ORGANIZACAO, O, B-ORGANIZACAO, O, O, O, O, O, O] |
3 | 52 | [AÇÃO, DECLARATÓRIA, .] | [O, O, O] |
4 | 4307 | [AGRAVO, INTERNO, NO, RECURSO, ESPECIAL, .] | [O, O, O, O, O, O] |
5 | 5238 | [Aceita, a, denúncia, à, fls, ., 39, e, interrogado, o, réu, a, fls, ., 174, ,, sua, Advogada, apresentou, defesa, prévia, a, fls, ., 93/98, .] | [O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O] |
6 | 958 | [PROJETO, DE, AUTORIA, PARLAMENTAR, .] | [O, O, O, O, O] |
7 | 3353 | [O, presente, feito, carece, de, condição, de, procedibilidade, ,, devendo, ser, reconhecida, ,, preliminarmente, e, de, ofício, ,, a, ausência, de, formalidade, essencial, à, sua, continuidade, .] | [O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O] |
8 | 2085 | [Também, incompleta, a, documentação, relativa, a, despesas, com, táxi, ,, consultores, ,, alimentação, ,, advogados, e, prestadores, de, serviços, diversos, .] | [O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O] |
9 | 7633 | [Inicialmente, ,, não, há, que, se, falar, em, contradição, ,, pois, o, caso, era, mesmo, de, extinção, da, execução, ,, ainda, que, por, fundamento, diverso, ,, devendo, ser, mantida, a, decisão, de, origem, e, ,, consequentemente, ,, negado, provimento, ao, Agravo, da, União, Federal, .] | [O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-ORGANIZACAO, O, O] |
Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers Tokenizer
which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.
To do all of this, we instantiate our tokenizer with the AutoTokenizer.from_pretrained
method, which will ensure:
That vocabulary will be cached, so it's not downloaded again the next time we run the cell.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
Downloading: 0%| | 0.00/530 [00:00<?, ?B/s]
Downloading: 0%| | 0.00/205k [00:00<?, ?B/s]
Downloading: 0%| | 0.00/428k [00:00<?, ?B/s]
Downloading: 0%| | 0.00/112 [00:00<?, ?B/s]
The following assertion ensures that our tokenizer is a fast tokenizers (backed by Rust) from the 🤗 Tokenizers library. Those fast tokenizers are available for almost all models, and we will need some of the special features they have for our preprocessing.
import transformers
assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)
You can check which type of models have a fast tokenizer available and which don't on the big table of models.
You can directly call this tokenizer on one sentence:
tokenizer("Hello, this is one sentence!")
{'input_ids': [101, 15044, 22280, 117, 12230, 145, 847, 3185, 22279, 5440, 1710, 106, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in this tutorial if you're interested.
If, as is the case here, your inputs have already been split into words, you should pass the list of words to your tokenzier with the argument is_split_into_words=True
:
tokenizer(["Hello", ",", "this", "is", "one", "sentence", "split", "into", "words", "."], is_split_into_words=True)
{'input_ids': [101, 15044, 22280, 117, 12230, 145, 847, 3185, 22279, 5440, 1710, 139, 863, 284, 19124, 2702, 824, 22281, 119, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
Note that transformers are often pretrained with subword tokenizers, meaning that even if your inputs have been split into words already, each of those words could be split again by the tokenizer. Let's look at an example of that:
example = datasets["train"][5]
print(example["tokens"])
['V.v', 'APELAÇÃO', 'CÍVEL', '-', 'NULIDADE', 'PROCESSUAL', '-', 'INTIMAÇÃO', 'DO', 'MINISTÉRIO', 'PÚBLICO', '-', 'INCAPAZ', 'ACOMPANHADA', 'DE', 'REPRESENTANTE', 'LEGAL', 'E', 'DE', 'ADVOGADO', '-', 'EXERCÍCIO', 'DO', 'CONTRADITÓRIO', 'E', 'DA', 'AMPLA', 'DEFESA', '-', 'AUSÊNCIA', 'DE', 'PREJUÍZOS', '-', 'VÍCIO', 'AFASTADO', '-', 'IMPROCEDÊNCIA', 'DO', 'PEDIDO', '-', 'INEXISTÊNCIA', 'DE', 'PROVA', 'QUANTO', 'AO', 'FATO', 'CONSTITUTIVO', 'DO', 'DIREITO', '.']
tokenized_input = tokenizer(example["tokens"], is_split_into_words=True)
tokens = tokenizer.convert_ids_to_tokens(tokenized_input["input_ids"])
print(tokens)
['[CLS]', 'V', '.', 'v', 'AP', '##EL', '##A', '##Ç', '##ÃO', 'C', '##Í', '##V', '##EL', '-', 'N', '##UL', '##ID', '##AD', '##E', 'PR', '##OC', '##ES', '##S', '##UA', '##L', '-', 'IN', '##TI', '##MA', '##Ç', '##ÃO', 'DO', 'M', '##IN', '##IS', '##T', '##É', '##RI', '##O', 'P', '##Ú', '##B', '##L', '##IC', '##O', '-', 'IN', '##CA', '##PA', '##Z', 'AC', '##OM', '##PA', '##N', '##HA', '##DA', 'DE', 'R', '##EP', '##RE', '##SE', '##NT', '##AN', '##TE', 'L', '##E', '##GA', '##L', 'E', 'DE', 'A', '##D', '##V', '##O', '##GA', '##DO', '-', 'E', '##X', '##ER', '##C', '##Í', '##CI', '##O', 'DO', 'CON', '##T', '##RA', '##DI', '##T', '##Ó', '##RI', '##O', 'E', 'D', '##A', 'AM', '##P', '##LA', 'DE', '##F', '##ES', '##A', '-', 'A', '##US', '##Ê', '##N', '##CI', '##A', 'DE', 'PR', '##E', '##J', '##U', '##Í', '##Z', '##OS', '-', 'V', '##Í', '##CI', '##O', 'A', '##FA', '##ST', '##AD', '##O', '-', 'I', '##MP', '##RO', '##CE', '##D', '##Ê', '##N', '##CI', '##A', 'DO', 'P', '##ED', '##ID', '##O', '-', 'IN', '##E', '##X', '##IS', '##T', '##Ê', '##N', '##CI', '##A', 'DE', 'PR', '##O', '##VA', 'Q', '##UA', '##NT', '##O', 'A', '##O', 'FA', '##TO', 'CON', '##ST', '##IT', '##UT', '##IV', '##O', 'DO', 'D', '##IR', '##EI', '##TO', '.', '[SEP]']
Here the words "Zwingmann" and "sheepmeat" have been split in three subtokens.
This means that we need to do some processing on our labels as the input ids returned by the tokenizer are longer than the lists of labels our dataset contain, first because some special tokens might be added (we can a [CLS]
and a [SEP]
above) and then because of those possible splits of words in multiple tokens:
len(example[f"{task}_tags"]), len(tokenized_input["input_ids"])
(50, 178)
Thankfully, the tokenizer returns outputs that have a word_ids
method which can help us.
print(tokenized_input.word_ids())
[None, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 6, 7, 7, 7, 7, 7, 8, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 11, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 14, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 17, 18, 19, 19, 19, 19, 19, 19, 20, 21, 21, 21, 21, 21, 21, 21, 22, 23, 23, 23, 23, 23, 23, 23, 23, 24, 25, 25, 26, 26, 26, 27, 27, 27, 27, 28, 29, 29, 29, 29, 29, 29, 30, 31, 31, 31, 31, 31, 31, 31, 32, 33, 33, 33, 33, 34, 34, 34, 34, 34, 35, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 38, 38, 38, 38, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 42, 42, 42, 43, 43, 43, 43, 44, 44, 45, 45, 46, 46, 46, 46, 46, 46, 47, 48, 48, 48, 48, 49, None]
As we can see, it returns a list with the same number of elements as our processed input ids, mapping special tokens to None
and all other tokens to their respective word. This way, we can align the labels with the processed input ids.
word_ids = tokenized_input.word_ids()
aligned_labels = [-100 if i is None else example[f"{task}_tags"][i] for i in word_ids]
print(len(aligned_labels), len(tokenized_input["input_ids"]))
178 178
Here we set the labels of all special tokens to -100 (the index that is ignored by PyTorch) and the labels of all other tokens to the label of the word they come from. Another strategy is to set the label only on the first token obtained from a given word, and give a label of -100 to the other subtokens from the same word. We propose the two strategies here, just change the value of the following flag:
label_all_tokens = True
We're now ready to write the function that will preprocess our samples. We feed them to the tokenizer
with the argument truncation=True
(to truncate texts that are bigger than the maximum size allowed by the model) and is_split_into_words=True
(as seen above). Then we align the labels with the token ids using the strategy we picked:
def tokenize_and_align_labels(examples):
tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True, max_length=512)
labels = []
for i, label in enumerate(examples[f"{task}_tags"]):
word_ids = tokenized_inputs.word_ids(batch_index=i)
previous_word_idx = None
label_ids = []
for word_idx in word_ids:
# Special tokens have a word id that is None. We set the label to -100 so they are automatically
# ignored in the loss function.
if word_idx is None:
label_ids.append(-100)
# We set the label for the first token of each word.
elif word_idx != previous_word_idx:
label_ids.append(label[word_idx])
# For the other tokens in a word, we set the label to either the current label or -100, depending on
# the label_all_tokens flag.
else:
label_ids.append(label[word_idx] if label_all_tokens else -100)
previous_word_idx = word_idx
labels.append(label_ids)
tokenized_inputs["labels"] = labels
return tokenized_inputs
This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:
tokenize_and_align_labels(datasets['train'][:5])
{'input_ids': [[101, 192, 7463, 8427, 22301, 131, 12127, 9008, 22301, 22402, 16484, 187, 22360, 22339, 9008, 118, 177, 22402, 16484, 10836, 13760, 7545, 22320, 22323, 22351, 22301, 22402, 16484, 212, 8718, 250, 7665, 6072, 213, 8718, 22301, 6538, 118, 11635, 9008, 13270, 7073, 6765, 118, 11741, 22328, 22341, 6392, 22301, 212, 9008, 22317, 213, 7073, 6538, 22321, 22352, 21748, 22317, 212, 22371, 22318, 22327, 6162, 22317, 192, 22311, 278, 5650, 22341, 257, 5476, 15289, 5903, 22327, 118, 248, 18199, 6392, 11836, 22309, 118, 177, 10409, 22420, 22320, 14298, 22301, 10836, 13760, 16017, 22322, 22339, 12547, 22402, 16484, 15040, 18868, 22322, 22349, 22341, 9208, 248, 22301, 13760, 11846, 22379, 22320, 14298, 22301, 177, 5226, 22341, 22317, 118, 11635, 3341, 12547, 22402, 22301, 10836, 13760, 4529, 5869, 22351, 118, 11635, 22309, 22333, 22341, 22360, 22351, 22317, 192, 22348, 6538, 16017, 8427, 22309, 118, 11635, 9008, 13270, 7073, 6765, 11247, 7918, 22340, 6392, 22301, 118, 248, 18199, 6392, 11836, 22309, 257, 5476, 12234, 22340, 5476, 6392, 22301, 119, 102], [101, 118, 231, 1328, 119, 7048, 117, 1349, 117, 171, 8250, 22304, 543, 1128, 301, 179, 7740, 22279, 320, 3890, 7638, 558, 3718, 529, 8309, 173, 179, 1307, 6399, 125, 22250, 117, 2586, 243, 146, 1328, 119, 2680, 22315, 171, 653, 8178, 179, 146, 1673, 2810, 1444, 326, 625, 146, 3890, 7638, 346, 344, 14353, 201, 221, 12079, 146, 2160, 173, 179, 1981, 558, 3718, 119, 102], [101, 118, 9633, 214, 118, 176, 125, 2973, 18233, 2446, 4977, 1442, 2530, 240, 2760, 12787, 117, 253, 18604, 123, 7418, 171, 3890, 7638, 229, 5061, 125, 13816, 180, 2601, 10629, 119, 102], [101, 118, 8457, 146, 1666, 523, 908, 3828, 286, 423, 5124, 22286, 3660, 17231, 2687, 20477, 162, 117, 1004, 271, 4271, 180, 12385, 125, 8407, 2483, 399, 171, 4794, 170, 1037, 229, 7036, 125, 9176, 1555, 298, 20165, 442, 6960, 17106, 117, 525, 4122, 118, 176, 146, 5035, 180, 1444, 3093, 1076, 171, 2160, 117, 125, 547, 123, 6675, 320, 3890, 7638, 558, 3718, 11925, 202, 2160, 173, 652, 5032, 119, 102], [101, 118, 4407, 4453, 2623, 19903, 328, 221, 11413, 123, 1444, 3093, 1076, 180, 12385, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'labels': [[-100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -100], [-100, 0, 0, 9, 10, 10, 10, 10, 10, 10, 10, 10, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -100], [-100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, -100], [-100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, -100], [-100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -100]]}
To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the map
method of our dataset
object we created earlier. This will apply the function on all the elements of all the splits in dataset
, so our training, validation and testing data will be preprocessed in one single command.
tokenized_datasets = datasets.map(tokenize_and_align_labels, batched=True)
0%| | 0/8 [00:00<?, ?ba/s]
0%| | 0/2 [00:00<?, ?ba/s]
0%| | 0/2 [00:00<?, ?ba/s]
Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass load_from_cache_file=False
in the call to map
to not use the cached files and force the preprocessing to be applied again.
Note that we passed batched=True
to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently.
Now that our data is ready, we can download the pretrained model and fine-tune it. Since all our tasks are about token classification, we use the AutoModelForTokenClassification
class. Like with the tokenizer, the from_pretrained
method will download and cache the model for us. The only thing we have to specify is the number of labels for our problem (which we can get from the features, as seen before):
from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer
model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=len(label_list))
Downloading: 0%| | 0.00/893 [00:00<?, ?B/s]
Downloading: 0%| | 0.00/416M [00:00<?, ?B/s]
Some weights of the model checkpoint at pierreguillou/bert-base-cased-pt-lenerbr were not used when initializing BertForTokenClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias'] - This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of BertForTokenClassification were not initialized from the model checkpoint at pierreguillou/bert-base-cased-pt-lenerbr and are newly initialized: ['classifier.weight', 'classifier.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The warning is telling us we are throwing away some weights (the vocab_transform
and vocab_layer_norm
layers) and randomly initializing some other (the pre_classifier
and classifier
layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.
To instantiate a Trainer
, we will need to define three more things. The most important is the TrainingArguments
, which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:
model_name = model_checkpoint.split("/")[-1]
# hyperparameters, which are passed into the training job
per_device_batch_size = 4
gradient_accumulation_steps = 2
learning_rate = 2e-5 # (AdamW) we started with 3e-4, then 1e-4, then 5e-5 but the model overfits fastly
num_train_epochs = 3 # we started with 10 epochs but the model overfits fastly
weight_decay = 0.01
save_total_limit = 3
logging_steps = 290 # melhor evaluate frequently (5000 seems too high)
eval_steps = logging_steps
evaluation_strategy = 'steps'
logging_strategy = 'steps'
save_strategy = 'steps'
save_steps = logging_steps
load_best_model_at_end = True
fp16 = True
# folders
model_name = model_checkpoint.split("/")[-1]
folder_model = 'e' + str(num_train_epochs) + '_lr' + str(learning_rate)
output_dir = '/content/drive/MyDrive/' + 'ner-lenerbr-' + str(model_name) + '/checkpoints/' + folder_model
logging_dir = '/content/drive/MyDrive/' + 'ner-lenerbr-' + str(model_name) + '/logs/' + folder_model
# get best model through a metric
metric_for_best_model = 'eval_f1'
if metric_for_best_model == 'eval_f1':
greater_is_better = True
elif metric_for_best_model == 'eval_loss':
greater_is_better = False
args = TrainingArguments(
output_dir=output_dir,
learning_rate=learning_rate,
per_device_train_batch_size=per_device_batch_size,
per_device_eval_batch_size=per_device_batch_size*2,
gradient_accumulation_steps=gradient_accumulation_steps,
num_train_epochs=num_train_epochs,
weight_decay=weight_decay,
save_total_limit=save_total_limit,
logging_steps = logging_steps,
eval_steps = logging_steps,
load_best_model_at_end = load_best_model_at_end,
metric_for_best_model = metric_for_best_model,
greater_is_better = greater_is_better,
gradient_checkpointing = False,
do_train = True,
do_eval = True,
do_predict = True,
evaluation_strategy = evaluation_strategy,
logging_dir=logging_dir,
logging_strategy = logging_strategy,
save_strategy = save_strategy,
save_steps = save_steps,
fp16 = fp16,
push_to_hub=False,
)
Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the batch_size
defined at the top of the notebook and customize the number of epochs for training, as well as the weight decay.
The last argument to setup everything so we can push the model to the Hub regularly during training. Remove it if you didn't follow the installation steps at the top of the notebook. If you want to save your model locally in a name that is different than the name of the repository it will be pushed, or if you want to push your model under an organization and not your name space, use the hub_model_id
argument to set the repo name (it needs to be the full name, including your namespace: for instance "sgugger/bert-finetuned-ner"
or "huggingface/bert-finetuned-ner"
).
Then we will need a data collator that will batch our processed examples together while applying padding to make them all the same size (each pad will be padded to the length of its longest example). There is a data collator for this task in the Transformers library, that not only pads the inputs, but also the labels:
from transformers import DataCollatorForTokenClassification
data_collator = DataCollatorForTokenClassification(tokenizer)
The last thing to define for our Trainer
is how to compute the metrics from the predictions. Here we will load the seqeval
metric (which is commonly used to evaluate results on the CONLL dataset) via the Datasets library.
metric = load_metric("seqeval")
Downloading: 0%| | 0.00/2.48k [00:00<?, ?B/s]
This metric takes list of labels for the predictions and references:
labels = [label_list[i] for i in example[f"{task}_tags"]]
metric.compute(predictions=[labels], references=[labels])
{'ORGANIZACAO': {'f1': 1.0, 'number': 1, 'precision': 1.0, 'recall': 1.0}, 'overall_accuracy': 1.0, 'overall_f1': 1.0, 'overall_precision': 1.0, 'overall_recall': 1.0}
So we will need to do a bit of post-processing on our predictions:
The following function does all this post-processing on the result of Trainer.evaluate
(which is a namedtuple containing predictions and labels) before applying the metric:
import numpy as np
def compute_metrics(p):
predictions, labels = p
predictions = np.argmax(predictions, axis=2)
# Remove ignored index (special tokens)
true_predictions = [
[label_list[p] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
true_labels = [
[label_list[l] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
results = metric.compute(predictions=true_predictions, references=true_labels)
return {
"precision": results["overall_precision"],
"recall": results["overall_recall"],
"f1": results["overall_f1"],
"accuracy": results["overall_accuracy"],
}
Note that we drop the precision/recall/f1 computed for each category and only focus on the overall precision/recall/f1/accuracy.
Then we just need to pass all of this along with our datasets to the Trainer
:
from transformers.trainer_callback import EarlyStoppingCallback
# wait early_stopping_patience x eval_steps before to stp the training in order to get a better model
early_stopping_patience = save_total_limit
trainer = Trainer(
model,
args,
train_dataset=tokenized_datasets["train"],
eval_dataset=tokenized_datasets["validation"],
data_collator=data_collator,
tokenizer=tokenizer,
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=early_stopping_patience)],
)
Using amp half precision backend
We can now finetune our model by just calling the train
method:
trainer.train()
The following columns in the training set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running training ***** Num examples = 7828 Num Epochs = 3 Instantaneous batch size per device = 4 Total train batch size (w. parallel, distributed & accumulation) = 8 Gradient Accumulation steps = 2 Total optimization steps = 2934
Step | Training Loss | Validation Loss | Precision | Recall | F1 | Accuracy |
---|---|---|---|---|---|---|
290 | 0.314600 | 0.163042 | 0.735828 | 0.697849 | 0.716336 | 0.949198 |
580 | 0.086900 | 0.123495 | 0.779540 | 0.824301 | 0.801296 | 0.965807 |
870 | 0.072800 | 0.106785 | 0.798481 | 0.858925 | 0.827600 | 0.968626 |
1160 | 0.046300 | 0.109921 | 0.824576 | 0.877419 | 0.850177 | 0.973243 |
1450 | 0.036600 | 0.102495 | 0.848792 | 0.899355 | 0.873342 | 0.975940 |
1740 | 0.033400 | 0.121514 | 0.821681 | 0.899785 | 0.858961 | 0.967071 |
2030 | 0.034700 | 0.115568 | 0.846849 | 0.887097 | 0.866506 | 0.970607 |
2320 | 0.018000 | 0.108600 | 0.840258 | 0.895914 | 0.867194 | 0.973730 |
The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290/special_tokens_map.json The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580/special_tokens_map.json The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870/special_tokens_map.json The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160/special_tokens_map.json Deleting older checkpoint [/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-290] due to args.save_total_limit The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450/special_tokens_map.json Deleting older checkpoint [/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-580] due to args.save_total_limit The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740/special_tokens_map.json Deleting older checkpoint [/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-870] due to args.save_total_limit The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2030 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2030/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2030/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2030/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2030/special_tokens_map.json Deleting older checkpoint [/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1160] due to args.save_total_limit The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8 Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2320 Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2320/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2320/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2320/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-2320/special_tokens_map.json Deleting older checkpoint [/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1740] due to args.save_total_limit Training completed. Do not forget to share your model on huggingface.co/models =) Loading best model from /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/checkpoints/e3_lr2e-05/checkpoint-1450 (score: 0.8733423827921062).
TrainOutput(global_step=2320, training_loss=0.0804201787915723, metrics={'train_runtime': 2231.4621, 'train_samples_per_second': 10.524, 'train_steps_per_second': 1.315, 'total_flos': 901649209129776.0, 'train_loss': 0.0804201787915723, 'epoch': 2.37})
The evaluate
method allows you to evaluate again on the evaluation dataset or on another dataset:
trainer.evaluate()
The following columns in the evaluation set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Evaluation ***** Num examples = 1177 Batch size = 8
{'epoch': 2.37, 'eval_accuracy': 0.9759397808828684, 'eval_f1': 0.8733423827921062, 'eval_loss': 0.10249536484479904, 'eval_precision': 0.8487923685812868, 'eval_recall': 0.8993548387096775, 'eval_runtime': 50.5934, 'eval_samples_per_second': 23.264, 'eval_steps_per_second': 2.925}
To get the precision/recall/f1 computed for each category now that we have finished training, we can apply the same function as before on the result of the predict
method:
predictions, labels, _ = trainer.predict(tokenized_datasets["validation"])
predictions = np.argmax(predictions, axis=2)
# Remove ignored index (special tokens)
true_predictions = [
[label_list[p] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
true_labels = [
[label_list[l] for (p, l) in zip(prediction, label) if l != -100]
for prediction, label in zip(predictions, labels)
]
results = metric.compute(predictions=true_predictions, references=true_labels)
results
The following columns in the test set don't have a corresponding argument in `BertForTokenClassification.forward` and have been ignored: tokens, id, ner_tags. ***** Running Prediction ***** Num examples = 1177 Batch size = 8
{'JURISPRUDENCIA': {'f1': 0.7069834413246942, 'number': 657, 'precision': 0.6707650273224044, 'recall': 0.7473363774733638}, 'LEGISLACAO': {'f1': 0.8256227758007118, 'number': 571, 'precision': 0.8390596745027125, 'recall': 0.8126094570928196}, 'LOCAL': {'f1': 0.7688564476885645, 'number': 194, 'precision': 0.728110599078341, 'recall': 0.8144329896907216}, 'ORGANIZACAO': {'f1': 0.8548387096774193, 'number': 1340, 'precision': 0.8062169312169312, 'recall': 0.9097014925373135}, 'PESSOA': {'f1': 0.9826697892271662, 'number': 1072, 'precision': 0.9868297271872061, 'recall': 0.9785447761194029}, 'TEMPO': {'f1': 0.9615846338535414, 'number': 816, 'precision': 0.9423529411764706, 'recall': 0.9816176470588235}, 'overall_accuracy': 0.9759397808828684, 'overall_f1': 0.8733423827921062, 'overall_precision': 0.8487923685812868, 'overall_recall': 0.8993548387096775}
model_dir = '/content/drive/MyDrive/' + 'ner-lenerbr-' + str(model_name) + '/model/'
trainer.save_model(model_dir)
Saving model checkpoint to /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/ Configuration saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/config.json Model weights saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/pytorch_model.bin tokenizer config file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/tokenizer_config.json Special tokens file saved in /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/special_tokens_map.json
You can now upload the result of the training to the Hub, just execute this instruction:
ner_tags = datasets["train"].features[f"ner_tags"].feature.names
from transformers import AutoModelForTokenClassification, AutoTokenizer, AutoConfig
id2label = {str(i): label for i, label in enumerate(ner_tags)}
label2id = {v: k for k, v in id2label.items()}
model = AutoModelForTokenClassification.from_pretrained(
str(model_dir),
id2label=id2label,
label2id=label2id,
)
tokenizer = AutoTokenizer.from_pretrained(str(model_dir))
loading configuration file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/config.json Model config BertConfig { "_name_or_path": "/content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/", "architectures": [ "BertForTokenClassification" ], "attention_probs_dropout_prob": 0.1, "classifier_dropout": null, "directionality": "bidi", "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "id2label": { "0": "O", "1": "B-ORGANIZACAO", "10": "I-LEGISLACAO", "11": "B-JURISPRUDENCIA", "12": "I-JURISPRUDENCIA", "2": "I-ORGANIZACAO", "3": "B-PESSOA", "4": "I-PESSOA", "5": "B-TEMPO", "6": "I-TEMPO", "7": "B-LOCAL", "8": "I-LOCAL", "9": "B-LEGISLACAO" }, "initializer_range": 0.02, "intermediate_size": 3072, "label2id": { "B-JURISPRUDENCIA": "11", "B-LEGISLACAO": "9", "B-LOCAL": "7", "B-ORGANIZACAO": "1", "B-PESSOA": "3", "B-TEMPO": "5", "I-JURISPRUDENCIA": "12", "I-LEGISLACAO": "10", "I-LOCAL": "8", "I-ORGANIZACAO": "2", "I-PESSOA": "4", "I-TEMPO": "6", "O": "0" }, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "bert", "num_attention_heads": 12, "num_hidden_layers": 12, "output_past": true, "pad_token_id": 0, "pooler_fc_size": 768, "pooler_num_attention_heads": 12, "pooler_num_fc_layers": 3, "pooler_size_per_head": 128, "pooler_type": "first_token_transform", "position_embedding_type": "absolute", "torch_dtype": "float32", "transformers_version": "4.15.0", "type_vocab_size": 2, "use_cache": true, "vocab_size": 29794 } loading weights file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/pytorch_model.bin All model checkpoint weights were used when initializing BertForTokenClassification. All the weights of BertForTokenClassification were initialized from the model checkpoint at /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/. If your task is similar to the task the model of the checkpoint was trained on, you can already use BertForTokenClassification for predictions without further training. Didn't find file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/added_tokens.json. We won't load it. loading file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/vocab.txt loading file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/tokenizer.json loading file None loading file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/special_tokens_map.json loading file /content/drive/MyDrive/ner-lenerbr-bert-base-cased-pt-lenerbr/model/tokenizer_config.json
if model_checkpoint == "pierreguillou/bert-base-cased-pt-lenerbr":
model.push_to_hub('pierreguillou/ner-bert-base-cased-pt-lenerbr')
tokenizer.push_to_hub('pierreguillou/ner-bert-base-cased-pt-lenerbr')
else:
model.push_to_hub('pierreguillou/ner-bert-large-cased-pt-lenerbr')
tokenizer.push_to_hub('pierreguillou/ner-bert-large-cased-pt-lenerbr')
/usr/local/lib/python3.7/dist-packages/huggingface_hub/hf_api.py:726: FutureWarning: `create_repo` now takes `token` as an optional positional argument. Be sure to adapt your code! FutureWarning, Cloning https://huggingface.co/pierreguillou/ner-bert-base-cased-pt-lenerbr into local empty directory. Configuration saved in pierreguillou/ner-bert-base-cased-pt-lenerbr/config.json Model weights saved in pierreguillou/ner-bert-base-cased-pt-lenerbr/pytorch_model.bin
Upload file pytorch_model.bin: 0%| | 3.37k/413M [00:00<?, ?B/s]
To https://huggingface.co/pierreguillou/ner-bert-base-cased-pt-lenerbr 17c0eab..181dfa1 main -> main tokenizer config file saved in pierreguillou/ner-bert-base-cased-pt-lenerbr/tokenizer_config.json Special tokens file saved in pierreguillou/ner-bert-base-cased-pt-lenerbr/special_tokens_map.json To https://huggingface.co/pierreguillou/ner-bert-base-cased-pt-lenerbr 181dfa1..aca0f80 main -> main
You can now share this model with all your friends, family, favorite pets: they can all load it with the identifier "your-username/the-name-you-picked"
so for instance:
from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained("sgugger/my-awesome-model")