Encoding
Encoding
The Encoding represents the output of a Tokenizer.
The attention mask
This indicates to the LM which tokens should be attended to, and which should not. This is especially important when batching sequences, where we need to applying padding.
The generated IDs
The IDs are the main input to a Language Model. They are the token indices, the numerical representations that a LM understands.
The offsets associated to each token
These offsets letβs you slice the input string, and thus retrieve the original part that led to producing the corresponding token.
A List
of overflowing Encoding
When using truncation, the Tokenizer takes care of splitting the output into as many pieces as required to match the specified maximum length. This field lets you retrieve all the subsequent pieces.
When you use pairs of sequences, the overflowing pieces will contain enough variations to cover all the possible combinations, while respecting the provided maximum length.
The generated sequence indices.
They represent the index of the input sequence associated to each token. The sequence id can be None if the token is not related to any input sequence, like for example with special tokens.
The special token mask
This indicates which tokens are special tokens, and which are not.
The generated tokens
They are the string representation of the IDs.
The generated type IDs
Generally used for tasks like sequence classification or question answering, these tokens let the LM know which input sequence corresponds to each tokens.
The generated word indices.
They represent the index of the word associated to each token. When the input is pre-tokenized, they correspond to the ID of the given input label, otherwise they correspond to the words indices as defined by the PreTokenizer that was used.
For special tokens and such (any token that was generated from something that was
not part of the input), the output is None
The generated word indices.
This is deprecated and will be removed in a future version.
Please use ~tokenizers.Encoding.word_ids
instead.
They represent the index of the word associated to each token. When the input is pre-tokenized, they correspond to the ID of the given input label, otherwise they correspond to the words indices as defined by the PreTokenizer that was used.
For special tokens and such (any token that was generated from something that was
not part of the input), the output is None
Get the token that contains the char at the given position in the input sequence.
Get the word that contains the char at the given position in the input sequence.
merge
( encodings growing_offsets = True ) β Encoding
Merge the list of encodings into one final Encoding
pad
( length direction = 'right' pad_id = 0 pad_type_id = 0 pad_token = '[PAD]' )
Parameters
-
length (
int
) — The desired lengthdirection — (
str
, defaults toright
): The expected padding direction. Can be eitherright
orleft
-
pad_id (
int
, defaults to0
) — The ID corresponding to the padding token -
pad_type_id (
int
, defaults to0
) — The type ID corresponding to the padding token -
pad_token (
str
, defaults to [PAD]) — The pad token to use
Pad the Encoding at the given length
Set the given sequence index
Set the given sequence index for the whole range of tokens contained in this Encoding.
Get the offsets of the token at the given index.
The returned offsets are related to the input sequence that contains the
token. In order to determine in which input sequence it belongs, you
must call ~tokenizers.Encoding.token_to_sequence()
.
Get the index of the sequence represented by the given token.
In the general use case, this method returns 0
for a single sequence or
the first sequence of a pair, and 1
for the second sequence of a pair
Get the index of the word that contains the token in one of the input sequences.
The returned word index is related to the input sequence that contains
the token. In order to determine in which input sequence it belongs, you
must call ~tokenizers.Encoding.token_to_sequence()
.
Truncate the Encoding at the given length
If this Encoding represents multiple sequences, when truncating this information is lost. It will be considered as representing a single sequence.
Get the offsets of the word at the given index in one of the input sequences.