Transformers documentation

Use tokenizers from 🤗 Tokenizers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v4.38.2).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Use tokenizers from 🤗 Tokenizers

The PreTrainedTokenizerFast depends on the 🤗 Tokenizers library. The tokenizers obtained from the 🤗 Tokenizers library can be loaded very simply into 🤗 Transformers.

Before getting in the specifics, let’s first start by creating a dummy tokenizer in a few lines:

>>> from tokenizers import Tokenizer
>>> from tokenizers.models import BPE
>>> from tokenizers.trainers import BpeTrainer
>>> from tokenizers.pre_tokenizers import Whitespace

>>> tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
>>> trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

>>> tokenizer.pre_tokenizer = Whitespace()
>>> files = [...]
>>> tokenizer.train(files, trainer)

We now have a tokenizer trained on the files we defined. We can either continue using it in that runtime, or save it to a JSON file for future re-use.

Loading directly from the tokenizer object

Let’s see how to leverage this tokenizer object in the 🤗 Transformers library. The PreTrainedTokenizerFast class allows for easy instantiation, by accepting the instantiated tokenizer object as an argument:

>>> from transformers import PreTrainedTokenizerFast

>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)

This object can now be used with all the methods shared by the 🤗 Transformers tokenizers! Head to the tokenizer page for more information.

Loading from a JSON file

In order to load a tokenizer from a JSON file, let’s first start by saving our tokenizer:

>>> tokenizer.save("tokenizer.json")

The path to which we saved this file can be passed to the PreTrainedTokenizerFast initialization method using the tokenizer_file parameter:

>>> from transformers import PreTrainedTokenizerFast

>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")

This object can now be used with all the methods shared by the 🤗 Transformers tokenizers! Head to the tokenizer page for more information.