tokenizers

Getting Started

  • Quicktour
  • Installation
  • The tokenization pipeline
  • Components

API Reference

  • Input sequences
  • Encode inputs
  • Tokenizer
  • Encoding
  • Added Tokens
tokenizers

ⓘ You are viewing legacy docs. Go to latest documentation instead.

  • »
  • Search

© Copyright 2020, huggingface

Built with Sphinx using a theme provided by Read the Docs.