Tokenizers

Fast State-of-the-art tokenizers, optimized for both research and production

🤗 Tokenizers provides an implementation of today’s most used tokenizers, with a focus on performance and versatility. These tokenizers are also used in 🤗 Transformers.

Main features:

• Train new vocabularies and tokenize, using today’s most used tokenizers.

• Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server’s CPU.

• Easy to use, but also extremely versatile.

• Designed for both research and production.

• Full alignment tracking. Even with destructive normalization, it’s always possible to get the part of the original sentence that corresponds to any token.

• Does all the pre-processing: Truncation, Padding, add the special tokens your model needs.

Getting Started

• Quicktour
  • Build a tokenizer from scratch
  • Using a pretrained tokenizer
  • Importing a pretrained tokenizer from legacy vocabulary files
• Installation
  • Installation with pip
  • Installation from sources
• The tokenization pipeline
  • Normalization
  • Pre-Tokenization
  • The Model
  • Post-Processing
  • All together: a BERT tokenizer from scratch
  • Decoding
• Components
  • Normalizers
  • Pre tokenizers
  • Models
  • PostProcessor
  • Decoders

Using 🤗 Tokenizers

• Training from memory
  • The most basic way
  • Using the 🤗 Datasets library
  • Using gzip files

API Reference

• Input sequences
• Encode inputs
• Tokenizer
• Encoding
• Added Tokens
• Models
• Normalizers
• Pre-tokenizers
• Post-processor
• Trainers
• Decoders
• Visualizer