B y T 5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022)

Abstract: Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1

## introduction an important consideration when designing nlp models is the way that text is represented. one common choice is to assign a unique token id to each word in a fixed finite vocabulary. a given piece of text is thus converted into a sequence of tokens by a tokenizer before being fed into a model for processing. an issue with using a fixed vocabulary of words is that there is no obvious way to process a piece of text that contains an out-of-vocabulary word. a standard approach is to map all unknown words to the @xmath0, but this prevents the model from distinguishing between different out-of-vocabulary words. subword tokenizers @xcite @xcite present an elegant solution to the out-of-vocabulary problem. instead of mapping each word to a single token, subword tokenizers decompose words into smaller subword units with a goal of minimizing the total length of the token sequences for a fixed vocabulary size. as an example, a subword tokenizer might tokenize the word doghouse as the pair of tokens dog and house even if doghouse is not in the subword vocabulary. this flexibility has caused subword tokenizers to become the de facto way to tokenize text over the past few years. however, subword tokenizers still exhibit various undesirable behaviors. typos, variants in spelling and capitalization, and morphological changes can all cause the token representation of a word or phrase to change completely, which can result in mispredictions. furthermore, unknown characters (e.g., from a language that was not used when the subword vocabulary was built) are typically out-of-vocabulary for a subword model. a more natural solution that avoids the aforementioned pitfalls would be to create token-free nlp models that do not rely on a learned vocabulary to map words or subword units to tokens. such models operate on raw text directly. we are not the first to make the case for token-free models, and a more comprehensive treatment of their various benefits can be found in recent work by @xcite . in this work, we make use of the fact that text data is generally stored as a sequence of bytes. thus, feeding byte sequences directly into the model enables the processing of arbitrary text sequences. this approach is well-aligned with the figure 1: pre-training example creation and network architecture of mt5 @xcite vs. byt5 (this work). mt5: text is split into sentencepiece tokens, spans of ∼3 tokens are masked (red), and the encoder/decoder transformer stacks have equal depth. byt5: text is processed as utf-8 bytes, spans of ∼20 bytes are masked, and the encoder is 3× deeper than the decoder. x , y , and z represent sentinel tokens. the main drawback of byte-level models is that byte sequences tend to be significantly longer than token sequences. because computational costs of machine learning models tend to scale with sequence length, much previous work on character-and byte-level models has explored ways to process long sequences efficiently using convolutions with pooling @xcite or adaptive computation time @xcite . in this work, we take a simpler approach and show that the transformer architecture can be straightforwardly adapted to process byte sequences without a dramatically unfavorable increase in computational cost. we focus on the t5 framework @xcite , where all text-based nlp problems are cast to a text-to-text format. this approach makes it simple to tackle an nlp task by generating a sequence of bytes conditioned on some input bytes. our proposed byt5 architecture is described in section 3. the design stays fairly close to mt5 (the multilingual variant of t5 introduced by @xcite ), with the differences illustrated in figure 1 . through extensive experiments on a diverse set of english and multilingual tasks (presented in section 4), we show that byt5 is competitive with a subwordlevel baseline, despite being pre-trained on 4× less text. we also confirm in section 5 that byte-level models are more robust to corruptions of the input text. throughout, we characterize the trade-offs of our design decisions in terms of computational cost and parameter count, discussed in more detail in sections 6 and 7. the end result is a set of pre-trained byt5 models that we release alongside this paper. ## related work the early neural language models of @xcite and @xcite operated directly on character sequences. this precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures @xcite @xcite @xcite . @xcite showed byte language models can match the perplexity of word-level models when given the same parameter budget. however, standard practice in real-world scenarios has remained to use word-or subword-level models. a number of character-aware architectures have been proposed that make use of characterlevel features but still rely on a tokenizer to identify word boundaries. these approaches include elmo @xcite , @xcite , and many others @xcite @xcite @xcite . separately, some work has endeavored to ameliorate issues with tokenization, for example, by adapting vocabularies to new languages @xcite or randomly choosing different subword segmentations to improve robustness in low-resource and out-of-domain settings @xcite . these methods do not meet our goal of simplifying the nlp pipeline by removing text preprocessing. there have been a few recent efforts to develop general-purpose token-free pre-trained language models for transfer learning. 2 @xcite show strong results on sequence labeling with character-level pre-training and release models covering four languages. more recently, @xcite develop canine, which shows gains over multilingual bert by working with characters instead of word-piece tokens, though the ''canine-s'' model still uses a tokenizer during pre-training to define targets for the masked language modeling task. our work differs from these in that (i) we train encoder-decoder models that extend to generative tasks, (ii) our models work directly with utf-8 bytes, and (iii) we explore the effect of model scale, training models beyond 10 billion parameters. ## byt5 design our goal in designing byt5 is to take an existing token-based model and perform the minimal set of modifications to make it token-free, thereby limiting experimental confounds. we base byt5 on the recent mt5 model @xcite , which was trained on mc4 (a large corpus of unlabeled multilingual text data) and achieved state-of-theart on many community benchmarks. we release byt5 in five sizes analogous to t5 and mt5 (small, base, large, xl, xxl). we aim for byt5 to cover the same use cases as mt5: it is a general-purpose pre-trained text-to-text model covering 100+ languages. we expect byt5 will be particular useful for tasks operating on shortto-medium length text sequences (a few sentences or less), as these will incur less slowdown in finetuning and inference. ## core results in this section, we compare byt5 against mt5 on a wide range of tasks. we show that byt5 is competitive with mt5 on standard english and multilingual nlp benchmarks and outperforms mt5 at small model sizes. additionally, byt5 excels on free-form generation tasks and wordlevel tasks. for each downstream task, we fine-tune mt5 and byt5 models for 262,144 steps, using a constant learning rate of 0.001 and a dropout rate of 0.1. 4 we use a batch size of 2 17 tokens by default, but increased this to 2 20 for several tasks with larger training sets (glue, superglue, xnli, tweetqa), and decreased to 2 16 for the dakshina task. in all cases, we select the best model checkpoint based on validation set performance. ## experiments on synthetic noise text on modern digital platforms is noisy and exhibits complex character-level phenomena such as typos, character repetitions, and non-standard case changes @xcite . beyond these, errors can be introduced by nlp systems such as predictive input methods and automatic speech recognition. we have already seen strong byt5 performance on the ''messy'' text in tweetqa. in this section, we move to even noisier text and explore model performance on inputs that have been corrupted with artificial noise of various kinds. across a range of noising schemes, we find that byt5 outperforms mt5, demonstrating higher robustness to noise across tasks and languages. we experiment with five noising schemes: (1) drop: each character (i.e., unicode codepoint) has a 10% chance of being dropped. (2) repetitions: each character has a 20% chance of being selected for repetition. if selected, 1-3 repetitions (with equal likelihood) are appended after the original character. (3) antspeak: each character is capitalized and padded with spaces, so ''an owl'' becomes '' a n o w l ''. (4) uppercase: each character is converted to uppercase. (5) random case: each character is set to a random case (upper or lower). for the last two noise types, we restrict to languages whose scripts distinguish case. we first consider the easier setting of learnable noise, where noise is applied during both fine-tuning and evaluation. we evaluate on xnli zero-shot and tydiqa-goldp. for xnli, both the premise and hypothesis are noised, and the model predicts an entailment label as usual. for tydiqa, we add noise to the question and the context, but leave the answer unchanged. thus, in many cases, the model needs to first locate the noisy answer, and then ''undo'' the noise to produce the target. we fine-tune all models for 30,000 steps following the procedure in section 4. we also test robustness to noise that is unseen during training but injected during evaluation. this is relevant in making models more future-proof as well as more resilient to accidental or adversarial spelling mistakes @xcite . we evaluate only xnli and skip tydiqa-goldp in this setting, as it is unreasonable to expect a generative model that was fine-tuned to always copy spans from the context to spontaneously ''undo'' corruptions and predict novel spans. the rightmost column of our findings echo the results of @xcite , who find that character-level models are more robust to real and synthetic noise than bpe or word-based models, across a range of morphological, syntactic, and semantic tagging tasks. the more general conclusion that emerges is that token-free models are more robust to noise across many tasks. ## ablation study to better understand the importance of various design choices, we train ablation models and compare these against our baselines on three tasks: xnli zero-shot, tydiqa-goldp, and gem-xsum. our baselines and ablations are listed in ## speed comparisons the time required for fine-tuning is also variable across tasks. when holding batch size constant at a fixed number of tokens, we find that byt5 typically takes more fine-tuning steps than mt5 to reach optimal performance on a holdout set. for example, byt5-large took 1.2× as many steps as mt5-large to reach peak validation performance on xnli zero-shot, 2.6× as many steps for tydiqa-goldp, and 4.5× as many for gem-xsum. this overall trend is expected, in that fewer labeled examples fit into each byt5 fine-tuning batch. however, on tasks that strongly favor byte-level representations, byt5 reaches peak performance in fewer fine-tuning steps, suggesting that the model can generalize better from a small number of training examples. for example, byt5-large took 2.5× fewer steps than mt5-large to reach peak performance on dakshina. overall, we believe that the additional pretraining cost (roughly +33% wall time) and the additional fine-tuning cost (for some tasks) is justified in non-latency-sensitive applications by the benefits of reduced system complexity, better robustness to noise, and improved task performance on many benchmarks. ## conclusion in this work, we presented byt5, a token-free variant of multilingual t5 @xcite that simplifies the nlp pipeline by doing away with vocabulary building, text preprocessing and tokenization. on downstream task quality, byt5 is competitive with parameter-matched mt5 models that rely on sentencepiece vocabulary. specifically, byt5 outperforms mt5 in any of these five scenarios: (1) at model sizes under 1 billion parameters, (2) on generative tasks, (3) on multilingual tasks with in-language labels, (4) on word-level tasks sensitive to spelling and/or pronunciation, and (5) in the presence of various types of noise. while beating mt5 in many cases, byt5 slightly underperformed in certain conditionsmost notably, on english classification tasks for model sizes over 1 billion parameters. in future work, it will also be important to evaluate tokenfree approaches on a more diverse set of tasks, especially those where character-based models have traditionally struggled. these include word similarity tasks @xcite , syntactic and semantic tagging tasks @xcite , and machine translation from a non-english source into english @xcite . through ablations, we showed that byte-level encoder-decoder models benefit from a ''heavier'' encoder (decoupling encoder and decoder depth), and that the pre-training task benefits from masking longer id sequences. we also showed that for fixed parameter count, character-level models give similar but somewhat worse results. interestingly, the gains we observe with byt5 are achieved despite the model being pre-trained on 4× less text than mt5. this suggests that bytelevel models may be more data efficient learners. these gains in design simplicity, task quality and data efficiency come at the cost of additional computation. our ''hands-off'' approach of feeding raw utf-8 bytes directly into the transformer costs +33% pre-training time, as well as longer inference time (up to 10× slower in the worst case). as such, there is significant room for improvement. we believe techniques such as hash embeddings, local attention and down-sampling @xcite , as well as sparse computation @xcite can help address latency issues, removing the remaining barriers to a tokenfree future.