Tokenizer¶

A tokenizer is in charge of preparing the inputs for a model. The library comprise tokenizers for all the models. Most of the tokenizers are available in two flavors: a full python implementation and a “Fast” implementation based on the Rust library tokenizers. The “Fast” implementations allows (1) a significant speed-up in particular when doing batched tokenization and (2) additional methods to map between the original string (character and words) and the token space (e.g. getting the index of the token comprising a given character or the span of characters corresponding to a given token). Currently no “Fast” implementation is available for the SentencePiece-based tokenizers (for T5, ALBERT, CamemBERT, XLMRoBERTa and XLNet models).

The base classes PreTrainedTokenizer and PreTrainedTokenizerFast implements the common methods for encoding string inputs in model inputs (see below) and instantiating/saving python and “Fast” tokenizers either from a local file or directory or from a pretrained tokenizer provided by the library (downloaded from HuggingFace’s AWS S3 repository).

PreTrainedTokenizer and PreTrainedTokenizerFast thus implements the main methods for using all the tokenizers:

tokenizing (spliting strings in sub-word token strings), converting tokens strings to ids and back, and encoding/decoding (i.e. tokenizing + convert to integers),
adding new tokens to the vocabulary in a way that is independant of the underlying structure (BPE, SentencePiece…),
managing special tokens like mask, beginning-of-sentence, etc tokens (adding them, assigning them to attributes in the tokenizer for easy access and making sure they are not split during tokenization)

BatchEncoding holds the output of the tokenizer’s encoding methods (__call__, encode_plus and batch_encode_plus) and is derived from a Python dictionary. When the tokenizer is a pure python tokenizer, this class behave just like a standard python dictionary and hold the various model inputs computed by these methodes (input_ids, attention_mask…). When the tokenizer is a “Fast” tokenizer (i.e. backed by HuggingFace tokenizers library), this class provides in addition several advanced alignement methods which can be used to map between the original string (character and words) and the token space (e.g. getting the index of the token comprising a given character or the span of characters corresponding to a given token).

`PreTrainedTokenizer`¶

class transformers.PreTrainedTokenizer(**kwargs)[source]¶

Base class for all slow tokenizers.

Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.

This class also contain the added tokens in a unified way on top of all tokenizers so we don’t have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece…).

Class attributes (overridden by derived classes):

vocab_files_names: a python dict with, as keys, the __init__ keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string).
pretrained_vocab_files_map: a python dict of dict the high-level keys being the __init__ keyword name of each vocabulary file required by the model, the low-level being the short-cut-names (string) of the pretrained models with, as associated values, the url (string) to the associated pretrained vocabulary file.
max_model_input_sizes: a python dict with, as keys, the short-cut-names (string) of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or None if the model has no maximum input size.
pretrained_init_configuration: a python dict with, as keys, the short-cut-names (string) of the pretrained models, and as associated values, a dictionnary of specific arguments to pass to the __init__``method of the tokenizer class for this pretrained model when loading the tokenizer with the ``from_pretrained() method.

Parameters

model_max_length (-) – (Optional) int: the maximum length in number of tokens for the inputs to the transformer model. When the tokenizer is loaded with from_pretrained, this will be set to the value stored for the associated model in max_model_input_sizes (see above). If no value is provided, will default to VERY_LARGE_INTEGER (int(1e30)). no associated max_length can be found in max_model_input_sizes.
padding_side (-) – (Optional) string: the side on which the model should have padding applied. Should be selected between [‘right’, ‘left’]
model_input_names (-) – (Optional) List[string]: the list of the forward pass inputs accepted by the model (“token_type_ids”, “attention_mask”…).
bos_token (-) – (Optional) string: a beginning of sentence token. Will be associated to self.bos_token and self.bos_token_id
eos_token (-) – (Optional) string: an end of sentence token. Will be associated to self.eos_token and self.eos_token_id
unk_token (-) – (Optional) string: an unknown token. Will be associated to self.unk_token and self.unk_token_id
sep_token (-) – (Optional) string: a separation token (e.g. to separate context and query in an input sequence). Will be associated to self.sep_token and self.sep_token_id
pad_token (-) – (Optional) string: a padding token. Will be associated to self.pad_token and self.pad_token_id
cls_token (-) – (Optional) string: a classification token (e.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model). Will be associated to self.cls_token and self.cls_token_id
mask_token (-) – (Optional) string: a masking token (e.g. when training a model with masked-language modeling). Will be associated to self.mask_token and self.mask_token_id
additional_special_tokens (-) – (Optional) list: a list of additional special tokens. Adding all special tokens here ensure they won’t be split by the tokenization process. Will be associated to self.additional_special_tokens and self.additional_special_tokens_ids

__call__(text: Union[str, List[str], List[List[str]]], text_pair: Optional[Union[str, List[str], List[List[str]]]] = None, add_special_tokens: bool = True, padding: Union[bool, str] = False, truncation: Union[bool, str] = False, max_length: Optional[int] = None, stride: int = 0, is_pretokenized: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, transformers.tokenization_utils_base.TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs) → transformers.tokenization_utils_base.BatchEncoding¶

Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classification and the overflowing elements if a max_length is specified.

Parameters

text (str, List[str], List[List[str]]`) – The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pre-tokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_pretokenized=True (to lift the ambiguity with a batch of sequences)
text_pair (str, List[str], List[List[str]]`) – The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pre-tokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_pretokenized=True (to lift the ambiguity with a batch of sequences)
add_special_tokens (bool, optional, defaults to True) – If set to True, the sequences will be encoded with the special tokens relative to their model.
padding (Union[bool, str], optional, defaults to False) –
Activate and control padding. Accepts the following values:
- True or ‘longest’: pad to the longest sequence in the batch (or no padding if only a single sequence if provided),
- ’max_length’: pad to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None)
- False or ‘do_not_pad’ (default): No padding (i.e. can output batch with sequences of uneven lengths)
truncation (Union[bool, str], optional, defaults to False) –
Activate and control truncation. Accepts the following values:
- True or ‘longest_first’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided,
- ’only_first’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
- ’only_second’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
- False or ‘do_not_truncate’ (default): No truncation (i.e. can output batch with sequences length greater than the model max admissible input size)
max_length (Union[int, None], optional, defaults to None) –
Control the length for padding/truncation. Accepts the following values
- None (default): This will use the predefined model max length if required by one of the truncation/padding parameters. If the model has no specific max input length (e.g. XLNet) truncation/padding to max length is deactivated.
- any integer value (e.g. 42): Use this specific maximum length value if required by one of the truncation/padding parameters.
stride (int, optional, defaults to 0) – If set to a number along with max_length, the overflowing tokens returned when return_overflowing_tokens=True will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflow ing sequences. The value of this argument defines the number of overlapping tokens.
is_pretokenized (bool, defaults to False) – Set to True to indicate the input is already tokenized
pad_to_multiple_of – (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability >= 7.5 (Volta).
return_tensors (str, optional, defaults to None) – Can be set to ‘tf’, ‘pt’ or ‘np’ to return respectively TensorFlow tf.constant, PyTorch torch.Tensor or Numpy :oj: np.ndarray instead of a list of python integers.
return_token_type_ids (bool, optional, defaults to None) –
Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer’s default, defined by the return_outputs attribute.

What are token type IDs?
return_attention_mask (bool, optional, defaults to none) –
Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer’s default, defined by the return_outputs attribute.

What are attention masks?
return_overflowing_tokens (bool, optional, defaults to False) – Set to True to return overflowing token sequences (default False).
return_special_tokens_mask (bool, optional, defaults to False) – Set to True to return special tokens mask information (default False).
return_offsets_mapping (bool, optional, defaults to False) – Set to True to return (char_start, char_end) for each token (default False). If using Python’s tokenizer, this method will raise NotImplementedError. This one is only available on fast tokenizers inheriting from PreTrainedTokenizerFast.
**kwargs – passed to the self.tokenize() method

Returns

A Dictionary of shape:

{
    input_ids: list[int],
    token_type_ids: list[int] if return_token_type_ids is True (default)
    attention_mask: list[int] if return_attention_mask is True (default)
    overflowing_tokens: list[int] if the tokenizer is a slow tokenize, else a List[List[int]] if a ``max_length`` is specified and ``return_overflowing_tokens=True``
    special_tokens_mask: list[int] if ``add_special_tokens`` if set to ``True``
    and return_special_tokens_mask is True
}

With the fields:

input_ids: list of token ids to be fed to a model
token_type_ids: list of token type ids to be fed to a model
attention_mask: list of indices specifying which tokens should be attended to by the model
overflowing_tokens: list of overflowing tokens sequences if a max length is specified and return_overflowing_tokens=True.
special_tokens_mask: if adding special tokens, this is a list of [0, 1], with 0 specifying special added tokens and 1 specifying sequence tokens.

convert_ids_to_tokens(ids: Union[int, List[int]], skip_special_tokens: bool = False) → Union[str, List[str]][source]¶

Converts a single index or a sequence of indices (integers) in a token ” (resp.) a sequence of tokens (str), using the vocabulary and added tokens.

Parameters: skip_special_tokens – Don’t decode special tokens (self.all_special_tokens). Default: False

convert_tokens_to_ids(tokens)[source]¶: Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary.

convert_tokens_to_string(tokens: List[str]) → str[source]¶: Converts a sequence of tokens (string) in a single string. The most simple way to do it is ‘ ‘.join(self.convert_ids_to_tokens(token_ids)) but we often want to remove sub-word tokenization artifacts at the same time.

decode(token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True) → str[source]¶

Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids)).

Parameters

token_ids – list of tokenized input ids. Can be obtained using the encode or encode_plus methods.
skip_special_tokens – if set to True, will replace special tokens.
clean_up_tokenization_spaces – if set to True, will clean up the tokenization spaces.

get_special_tokens_mask(token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False) → List[int][source]¶

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer prepare_for_model method.

Parameters

token_ids_0 – list of ids (must not contain special tokens)
token_ids_1 – Optional list of ids (must not contain special tokens), necessary when fetching sequence ids for sequence pairs
already_has_special_tokens – (default False) Set to True if the token list is already formated with special tokens for the model

Returns

1 for a special token, 0 for a sequence token.

Return type

A list of integers in the range [0, 1]

get_vocab()[source]¶: Returns the vocabulary as a dict of {token: index} pairs. tokenizer.get_vocab()[token] is equivalent to tokenizer.convert_tokens_to_ids(token) when token is in the vocab.

num_special_tokens_to_add(pair=False)[source]¶

Returns the number of added tokens when encoding a sequence with special tokens.

Note

This encodes inputs and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop.

Parameters: pair – Returns the number of added tokens in the case of a sequence pair if set to True, returns the number of added tokens in the case of a single sequence if set to False.
Returns: Number of tokens added to sequences

prepare_for_tokenization(text: str, is_pretokenized=False, **kwargs) -> (<class 'str'>, <class 'dict'>)[source]¶

Performs any necessary transformations before tokenization.

This method should pop the arguments from kwargs and return kwargs as well. We test kwargs at the end of the encoding process to be sure all the arguments have been used.

save_vocabulary(save_directory) → Tuple[str][source]¶

Save the tokenizer vocabulary to a directory. This method does NOT save added tokens and special token mappings.

Please use save_pretrained() () to save the full Tokenizer state if you want to reload it using the from_pretrained() class method.

tokenize(text: str, **kwargs)[source]¶

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).

Take care of added tokens.

Parameters

text (string) – The sequence to be encoded.
( (**kwargs) – obj: dict): Arguments passed to the model-specific prepare_for_tokenization preprocessing method.

property vocab_size¶: Size of the base vocabulary (without the added tokens)

`PreTrainedTokenizerFast`¶

class transformers.PreTrainedTokenizerFast(tokenizer: tokenizers.implementations.base_tokenizer.BaseTokenizer, **kwargs)[source]¶

Base class for all fast tokenizers (wrapping HuggingFace tokenizers library).

Inherit from PreTrainedTokenizer.

Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary.

This class also contain the added tokens in a unified way on top of all tokenizers so we don’t have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece…).

Class attributes (overridden by derived classes):

vocab_files_names: a python dict with, as keys, the __init__ keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string).
pretrained_vocab_files_map: a python dict of dict the high-level keys being the __init__ keyword name of each vocabulary file required by the model, the low-level being the short-cut-names (string) of the pretrained models with, as associated values, the url (string) to the associated pretrained vocabulary file.
max_model_input_sizes: a python dict with, as keys, the short-cut-names (string) of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or None if the model has no maximum input size.
pretrained_init_configuration: a python dict with, as keys, the short-cut-names (string) of the pretrained models, and as associated values, a dictionnary of specific arguments to pass to the __init__``method of the tokenizer class for this pretrained model when loading the tokenizer with the ``from_pretrained() method.

Parameters

tokenizer (-) – A Fast tokenizer from the HuggingFace tokenizer library (in low level Rust language)
model_max_length (-) – (Optional) int: the maximum length in number of tokens for the inputs to the transformer model. When the tokenizer is loaded with from_pretrained, this will be set to the value stored for the associated model in max_model_input_sizes (see above). If no value is provided, will default to VERY_LARGE_INTEGER (int(1e30)). no associated max_length can be found in max_model_input_sizes.
padding_side (-) – (Optional) string: the side on which the model should have padding applied. Should be selected between [‘right’, ‘left’]
model_input_names (-) – (Optional) List[string]: the list of the forward pass inputs accepted by the model (“token_type_ids”, “attention_mask”…).
bos_token (-) – (Optional) string: a beginning of sentence token. Will be associated to self.bos_token and self.bos_token_id
eos_token (-) – (Optional) string: an end of sentence token. Will be associated to self.eos_token and self.eos_token_id
unk_token (-) – (Optional) string: an unknown token. Will be associated to self.unk_token and self.unk_token_id
sep_token (-) – (Optional) string: a separation token (e.g. to separate context and query in an input sequence). Will be associated to self.sep_token and self.sep_token_id
pad_token (-) – (Optional) string: a padding token. Will be associated to self.pad_token and self.pad_token_id
cls_token (-) – (Optional) string: a classification token (e.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model). Will be associated to self.cls_token and self.cls_token_id
mask_token (-) – (Optional) string: a masking token (e.g. when training a model with masked-language modeling). Will be associated to self.mask_token and self.mask_token_id
additional_special_tokens (-) – (Optional) list: a list of additional special tokens. Adding all special tokens here ensure they won’t be split by the tokenization process. Will be associated to self.additional_special_tokens and self.additional_special_tokens_ids

__call__(text: Union[str, List[str], List[List[str]]], text_pair: Optional[Union[str, List[str], List[List[str]]]] = None, add_special_tokens: bool = True, padding: Union[bool, str] = False, truncation: Union[bool, str] = False, max_length: Optional[int] = None, stride: int = 0, is_pretokenized: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, transformers.tokenization_utils_base.TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs) → transformers.tokenization_utils_base.BatchEncoding¶

Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classification and the overflowing elements if a max_length is specified.

Parameters

text (str, List[str], List[List[str]]`) – The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pre-tokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_pretokenized=True (to lift the ambiguity with a batch of sequences)
text_pair (str, List[str], List[List[str]]`) – The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pre-tokenized string). If the sequences are provided as list of strings (pretokenized), you must set is_pretokenized=True (to lift the ambiguity with a batch of sequences)
add_special_tokens (bool, optional, defaults to True) – If set to True, the sequences will be encoded with the special tokens relative to their model.
padding (Union[bool, str], optional, defaults to False) –
Activate and control padding. Accepts the following values:
- True or ‘longest’: pad to the longest sequence in the batch (or no padding if only a single sequence if provided),
- ’max_length’: pad to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None)
- False or ‘do_not_pad’ (default): No padding (i.e. can output batch with sequences of uneven lengths)
truncation (Union[bool, str], optional, defaults to False) –
Activate and control truncation. Accepts the following values:
- True or ‘longest_first’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided,
- ’only_first’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
- ’only_second’: truncate to a max length specified in max_length or to the max acceptable input length for the model if no length is provided (max_length=None). This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
- False or ‘do_not_truncate’ (default): No truncation (i.e. can output batch with sequences length greater than the model max admissible input size)
max_length (Union[int, None], optional, defaults to None) –
Control the length for padding/truncation. Accepts the following values
- None (default): This will use the predefined model max length if required by one of the truncation/padding parameters. If the model has no specific max input length (e.g. XLNet) truncation/padding to max length is deactivated.
- any integer value (e.g. 42): Use this specific maximum length value if required by one of the truncation/padding parameters.
stride (int, optional, defaults to 0) – If set to a number along with max_length, the overflowing tokens returned when return_overflowing_tokens=True will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflow ing sequences. The value of this argument defines the number of overlapping tokens.
is_pretokenized (bool, defaults to False) – Set to True to indicate the input is already tokenized
pad_to_multiple_of – (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability >= 7.5 (Volta).
return_tensors (str, optional, defaults to None) – Can be set to ‘tf’, ‘pt’ or ‘np’ to return respectively TensorFlow tf.constant, PyTorch torch.Tensor or Numpy :oj: np.ndarray instead of a list of python integers.
return_token_type_ids (bool, optional, defaults to None) –
Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer’s default, defined by the return_outputs attribute.

What are token type IDs?
return_attention_mask (bool, optional, defaults to none) –
Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer’s default, defined by the return_outputs attribute.

What are attention masks?
return_overflowing_tokens (bool, optional, defaults to False) – Set to True to return overflowing token sequences (default False).
return_special_tokens_mask (bool, optional, defaults to False) – Set to True to return special tokens mask information (default False).
return_offsets_mapping (bool, optional, defaults to False) – Set to True to return (char_start, char_end) for each token (default False). If using Python’s tokenizer, this method will raise NotImplementedError. This one is only available on fast tokenizers inheriting from PreTrainedTokenizerFast.
**kwargs – passed to the self.tokenize() method

Returns

A Dictionary of shape:

{
    input_ids: list[int],
    token_type_ids: list[int] if return_token_type_ids is True (default)
    attention_mask: list[int] if return_attention_mask is True (default)
    overflowing_tokens: list[int] if the tokenizer is a slow tokenize, else a List[List[int]] if a ``max_length`` is specified and ``return_overflowing_tokens=True``
    special_tokens_mask: list[int] if ``add_special_tokens`` if set to ``True``
    and return_special_tokens_mask is True
}

With the fields:

input_ids: list of token ids to be fed to a model
token_type_ids: list of token type ids to be fed to a model
attention_mask: list of indices specifying which tokens should be attended to by the model
overflowing_tokens: list of overflowing tokens sequences if a max length is specified and return_overflowing_tokens=True.
special_tokens_mask: if adding special tokens, this is a list of [0, 1], with 0 specifying special added tokens and 1 specifying sequence tokens.

convert_ids_to_tokens(ids: Union[int, List[int]], skip_special_tokens: bool = False) → Union[str, List[str]][source]¶

Converts a single index or a sequence of indices (integers) in a token ” (resp.) a sequence of tokens (str), using the vocabulary and added tokens.

Parameters: skip_special_tokens – Don’t decode special tokens (self.all_special_tokens). Default: False

convert_tokens_to_ids(tokens: Union[str, List[str]]) → Union[int, List[int]][source]¶: Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary.

decode(token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True) → str[source]¶

Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids)).

Parameters

token_ids – list of tokenized input ids. Can be obtained using the encode or encode_plus methods.
skip_special_tokens – if set to True, will replace special tokens.
clean_up_tokenization_spaces – if set to True, will clean up the tokenization spaces.

set_truncation_and_padding(padding_strategy: transformers.tokenization_utils_base.PaddingStrategy, truncation_strategy: transformers.tokenization_utils_base.TruncationStrategy, max_length: int, stride: int, pad_to_multiple_of: Optional[int])[source]¶

Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards.

The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding/truncation when exiting the managed section.

Parameters

padding_strategy (PaddingStrategy) – The kind of padding that will be applied to the input
truncation_strategy (TruncationStrategy) – The kind of truncation that will be applied to the input
max_length (int) – The maximum size of the sequence
stride (int) – The stride to use when handling overflow
pad_to_multiple_of (int, optional, defaults to None) –

`BatchEncoding`¶

class transformers.BatchEncoding(data: Optional[Dict[str, Any]] = None, encoding: Optional[Union[tokenizers.Encoding, Sequence[tokenizers.Encoding]]] = None, tensor_type: Union[None, str, transformers.tokenization_utils_base.TensorType] = None, prepend_batch_axis: bool = False)[source]¶

BatchEncoding hold the output of the encode and batch_encode methods (tokens, attention_masks, etc). This class is derived from a python Dictionary and can be used as a dictionnary. In addition, this class expose utility methods to map from word/char space to token space.

Parameters

data (dict) – Dictionary of lists/arrays returned by the encode/batch_encode methods (‘input_ids’, ‘attention_mask’…)
encoding (EncodingFast, list(EncodingFast), optional, defaults to None) – If the tokenizer is a fast tokenizer which outputs additional informations like mapping from word/char space to token space the EncodingFast instance or list of instance (for batches) hold these informations.
tensor_type (Union[None, str, TensorType], optional, defaults to None) – You can give a tensor_type here to convert the lists of integers in PyTorch/TF/Numpy Tensors at initialization
prepend_batch_axis (bool, optional, defaults to False) – Set to True to add a batch axis when converting in Tensors (see tensor_type above)

char_to_token(batch_or_char_index: int, char_index: Optional[int] = None) → int[source]¶

Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch.

Can be called as:

self.char_to_token(char_index) if batch size is 1
self.char_to_token(batch_index, char_index) if batch size is greater or equal to 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words.

Parameters

batch_or_char_index (int) – Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence
char_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the word in the sequence.

Returns

Index of the token.

Return type

int

char_to_word(batch_or_char_index: int, char_index: Optional[int] = None) → int[source]¶

Get the word in the original string corresponding to a character in the original string of a sequence of the batch.

Can be called as:

self.char_to_word(char_index) if batch size is 1
self.char_to_word(batch_index, char_index) if batch size is greater than 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words.

Parameters

batch_or_char_index (int) – Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the orginal string.
char_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the character in the orginal string.

Returns

Index or indices of the associated encoded token(s).

Return type

int or List[int]

property encodings¶

Return the list all encoding from the tokenization process

Returns: List[EncodingFast] or None if input was tokenized through Python (i.e. not fast) tokenizer

property is_fast¶: Indicate if this BatchEncoding was generated from the result of a PreTrainedTokenizerFast Returns: True if generated from subclasses of PreTrainedTokenizerFast, else otherwise

items() → a set-like object providing a view on D’s items[source]¶

keys() → a set-like object providing a view on D’s keys[source]¶

to(device: str)[source]¶: Send all values to device by calling v.to(device)

token_to_chars(batch_or_token_index: int, token_index: Optional[int] = None) → transformers.tokenization_utils_base.CharSpan[source]¶

Get the character span corresponding to an encoded token in a sequence of the batch.

Character spans are returned as a CharSpan NamedTuple with:

start: index of the first character in the original string associated to the token
end: index of the character following the last character in the original string associated to the token

Can be called as:

self.token_to_chars(token_index) if batch size is 1
self.token_to_chars(batch_index, token_index) if batch size is greater or equal to 1

Parameters

batch_or_token_index (int) – Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence
token_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the token or tokens in the sequence.

Returns

Span of characters in the original string.

CharSpan are NamedTuple with:

start: index of the first character in the original string
end: index of the character following the last character in the original string

Return type

CharSpan

token_to_word(batch_or_token_index: int, token_index: Optional[int] = None) → int[source]¶

Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch.

Can be called as:

self.token_to_word(token_index) if batch size is 1
self.token_to_word(batch_index, token_index) if batch size is greater than 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words.

Parameters

batch_or_token_index (int) – Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence
token_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the token in the sequence.

Returns

index of the word in the input sequence.

Return type

int

values() → an object providing a view on D’s values[source]¶

word_to_chars(batch_or_word_index: int, word_index: Optional[int] = None) → transformers.tokenization_utils_base.CharSpan[source]¶

Get the character span in the original string corresponding to given word in a sequence of the batch.

Character spans are returned as a CharSpan NamedTuple with:

start: index of the first character in the original string
end: index of the character following the last character in the original string

Can be called as:

self.word_to_chars(word_index) if batch size is 1
self.word_to_chars(batch_index, word_index) if batch size is greater or equal to 1

Parameters

batch_or_word_index (int) – Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence
word_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the word in the sequence.

Returns

Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with:

start: index of the first character associated to the token in the original string
end: index of the character following the last character associated to the token in the original string

Return type

CharSpan or List[CharSpan]

word_to_tokens(batch_or_word_index: int, word_index: Optional[int] = None) → transformers.tokenization_utils_base.TokenSpan[source]¶

Get the encoded token span corresponding to a word in the sequence of the batch.

Token spans are returned as a TokenSpan NamedTuple with:

start: index of the first token
end: index of the token following the last token

Can be called as:

self.word_to_tokens(word_index) if batch size is 1
self.word_to_tokens(batch_index, word_index) if batch size is greater or equal to 1

This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words.

Parameters

batch_or_word_index (int) – Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence
word_index (int, optional) – If a batch index is provided in batch_or_token_index, this can be the index of the word in the sequence.

Returns

Span of tokens in the encoded sequence.

TokenSpan are NamedTuple with:

start: index of the first token
end: index of the token following the last token

Return type

TokenSpan

`SpecialTokensMixin`¶

class transformers.SpecialTokensMixin(verbose=True, **kwargs)[source]¶

SpecialTokensMixin is derived by PreTrainedTokenizer and PreTrainedTokenizerFast and handles specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access to these special tokens in a model-independant manner and allow to set and update the special tokens.

add_special_tokens(special_tokens_dict: Dict[str, Union[str, tokenizers.AddedToken]]) → int[source]¶

Add a dictionary of special tokens (eos, pad, cls…) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary).

Using add_special_tokens will ensure your special tokens can be used in several ways:

special tokens are carefully handled by the tokenizer (they are never split)
you can easily refer to special tokens using tokenizer class attributes like tokenizer.cls_token. This makes it easy to develop model-agnostic training and fine-tuning scripts.

When possible, special tokens are already registered for provided pretrained models (ex: BertTokenizer cls_token is already registered to be ‘[CLS]’ and XLM’s one is also registered to be ‘</s>’)

Parameters

special_tokens_dict –

dict of string. Keys should be in the list of predefined special attributes: [bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, additional_special_tokens].

Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the unk_token to them).

Returns

Number of tokens added to the vocabulary.

Examples:

# Let's see how to add a new classification token to GPT-2
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')

special_tokens_dict = {'cls_token': '<CLS>'}

num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.

assert tokenizer.cls_token == '<CLS>'

add_tokens(new_tokens: Union[str, tokenizers.AddedToken, List[str], List[tokenizers.AddedToken]], special_tokens=False) → int[source]¶

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary.

Parameters

new_tokens – string or list of string or AddedToken. Each string is a token to add. Tokens are only added if they are not already in the vocabulary. AddedToken wrap a string token to let you personnalize it’s behavior (Whether this token should only match against single word, whether this token should strip all potential whitespaces on the left side, Whether this token should strip all potential whitespaces on the right side…).
special_token –
can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance)

See details for AddedToken in HuggingFace tokenizers library.

Returns

Number of tokens added to the vocabulary.

Examples:

# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.

property additional_special_tokens¶: All the additional special tokens you may want to use (list of strings). Log an error if used while not having been set.

property additional_special_tokens_ids¶: Ids of all the additional special tokens in the vocabulary (list of integers). Log an error if used while not having been set.

property all_special_ids¶: List the vocabulary indices of the special tokens (‘<unk>’, ‘<cls>’…) mapped to class attributes (cls_token, unk_token…).

property all_special_tokens¶: List all the special tokens (‘<unk>’, ‘<cls>’…) mapped to class attributes Convert tokens of AddedToken type in string. All returned tokens are strings (cls_token, unk_token…).

property all_special_tokens_extended¶

List all the special tokens (‘<unk>’, ‘<cls>’…) mapped to class attributes Keep the tokens as AddedToken if they are of this type.

AddedToken can be used to control more finely how special tokens are tokenized.

property bos_token¶: Beginning of sentence token (string). Log an error if used while not having been set.

property bos_token_id¶: Id of the beginning of sentence token in the vocabulary. Log an error if used while not having been set.

property cls_token¶: Classification token (string). E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set.

property cls_token_id¶: Id of the classification token in the vocabulary. E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set.

property eos_token¶: End of sentence token (string). Log an error if used while not having been set.

property eos_token_id¶: Id of the end of sentence token in the vocabulary. Log an error if used while not having been set.

property mask_token¶: Mask token (string). E.g. when training a model with masked-language modeling. Log an error if used while not having been set.

property mask_token_id¶: Id of the mask token in the vocabulary. E.g. when training a model with masked-language modeling. Log an error if used while not having been set.

property pad_token¶: Padding token (string). Log an error if used while not having been set.

property pad_token_id¶: Id of the padding token in the vocabulary. Log an error if used while not having been set.

property pad_token_type_id¶: Id of the padding token type in the vocabulary.

sanitize_special_tokens() → int[source]¶

Make sure that all the special tokens attributes of the tokenizer (tokenizer.mask_token, tokenizer.cls_token, …) are in the vocabulary. Add the missing ones to the vocabulary if needed.

Returns: Number of tokens added in the vocaulary during the operation.

property sep_token¶: Separation token (string). E.g. separate context and query in an input sequence. Log an error if used while not having been set.

property sep_token_id¶: Id of the separation token in the vocabulary. E.g. separate context and query in an input sequence. Log an error if used while not having been set.

property special_tokens_map¶: A dictionary mapping special token class attribute (cls_token, unk_token…) to their values (‘<unk>’, ‘<cls>’…) Convert tokens of AddedToken type in string. All returned tokens are strings

property special_tokens_map_extended¶

A dictionary mapping special token class attribute (cls_token, unk_token…) to their values (‘<unk>’, ‘<cls>’…) Keep the tokens as AddedToken if they are of this type.

AddedToken can be used to control more finely how special tokens are tokenized.

property unk_token¶: Unknown token (string). Log an error if used while not having been set.

property unk_token_id¶: Id of the unknown token in the vocabulary. Log an error if used while not having been set.

Tokenizer¶

PreTrainedTokenizer¶

PreTrainedTokenizerFast¶

BatchEncoding¶

SpecialTokensMixin¶

`PreTrainedTokenizer`¶

`PreTrainedTokenizerFast`¶

`BatchEncoding`¶

`SpecialTokensMixin`¶