Utilities for Tokenizers¶

This page lists all the utility functions used by the tokenizers, mainly the class PreTrainedTokenizerBase that implements the common methods between PreTrainedTokenizer and PreTrainedTokenizerFast and the mixin SpecialTokensMixin.

Most of those are only useful if you are studying the code of the tokenizers in the library.