This notebook regroups the code sample of the video below, which is a part of the Hugging Face course.

In [ ]:

#@title
from IPython.display import HTML

HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/grlLV8AIXug?rel=0&amp;controls=0&amp;showinfo=0" frameborder="0" allowfullscreen></iframe>')

Out[ ]:

Install the Transformers and Datasets libraries to run this notebook.

In [ ]:

! pip install datasets transformers[sentencepiece]

In [ ]:

from transformers import AutoTokenizerFast

tokenizer = AutoTokenizerFast.from_pretrained('albert-base-v1’)

text = "3.2.1: let's get started!"

print(tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(text))