MANT a: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022)

Abstract: Static subword tokenization algorithms have been an essential component of recent works on language modeling. However, their static nature results in important flaws that degrade the models’ downstream performance and robustness. In this work, we propose MANTa, a Module for Adaptive Neural TokenizAtion. MANTa is a differentiable tokenizer trained end-to-end with the language model. The resulting system offers a trade-off between the expressiveness of byte-level models and the speed of models trained using subword tokenization. In addition, our tokenizer is highly explainable since it produces an explicit segmentation of sequences into blocks. We evaluate our pre-trained model on several English datasets from different domains as well as on synthetic noise. We find that MANTa improves robustness to character perturbations and out-of-domain data. We then show that MANTa performs comparably to other models on the general-domain GLUE benchmark. Finally, we show that it is considerably faster than strictly byte-level models.

## introduction in order to improve language models (lms), the natural language processing field has removed most of the system-induced biases in the last few years. for instance, practices that were once standard such as lemmatization, stemming and feature engineering have progressively disappeared in favor of general architectures trained on huge amounts of data, learning end-to-end which features may be leveraged to attain better performances. however, one essential part of lms has seen little evolution: tokenization. tokenizers convert sequences of characters into sequences of tokens (substrings of smaller length) which can then be embedded by the model. subword tokenization algorithms @xcite ., * equal contribution. 2016; @xcite are a specific class of tokenizers designed in such a way that almost every string can be encoded and decoded with very few outof-vocabulary tokens. they are used in the vast majority of recent lms, but have been an essential part of nlp systems since much longer @xcite . the success of these algorithms can be attributed to several reasons. firstly, they produce token sequences whose length is greatly reduced compared to the original character sequence. this characteristic is helpful because limitations in compute power and architectural constraints, such as the quadratic complexity with respect to sequence length of transformers @xcite , prevent models from processing arbitrary long sequences. sec-ondly, they compress the corpus using occurrence statistics that may help lms. for instance, if a word appears frequently in the training corpus, it will be encoded as a single token in the vocabulary and the model will be able to build a representation for that particular token more easily. however, the induced biases of tokenizers are also harmful for modelization. one such limitation lies in their brittleness to character deformations which are commonly found in real world, noisy data. for instance, bert's tokenizer @xcite encodes "performance" as ["performance"] but "perfonmance" as ['per', '##fo', '##n', '##man', '##ce'], which makes it hard for the model to behave similarly in both cases. moreover, the tokenizer is fixed after its training and is therefore impossible to update, for instance to reflect new domains @xcite where tokenization might over-segment specific or technical terms. @xcite list other issues emerging when using static subword tokenizers, especially when modeling languages with a more complex morphology than english. to overcome these issues, tokenization-free models @xcite @xcite produce character-based or byte-based embeddings for lms instead of subword embeddings. these methods improve the robustness of lms to naturally occurring noise as well as their expressiveness when dealing with out-of-domain or multilingual data. in order to cope with increased input lengths, some of these methods compress sequences with constant reduction rates obtained using specialized modules @xcite , subsequently removing any notion of subwords. we argue that learning a subword tokenization together with input representations in an end-toend fashion is beneficial for language modeling. in this work, we introduce manta, a gradient-based tokenizer and embedding module. it can easily be plugged-in to replace the classical combination of fixed tokenizers and trainable subword embedding matrices existing in most encoder-decoder models, without any increase in the total number of trainable parameters. we also introduce manta-lm, a transformer encoder-decoder that incorporates manta and that is trained end-to-end. by learning a soft, adaptive segmentation of input sequences jointly with the lm pre-training objective, manta-lm produces byte-based representations with sequence lengths similar to those produced by static subword tokenizers. additionally, by propagating gradients through our soft segmentation module during fine-tuning as well, we are able to adapt the segmentation to new domains, removing the limitations imposed by static subword tokenization. we show that manta-lm is robust to noisy text data and able to adapt to new domains while being significantly faster than byte-level models. interestingly, manta learns a simple but explainable segmentation using only the lm objective while effectively reducing the length of byte sequences. in summary, the contributions of this paper are the following: ## related work non-neural subword-level tokenization methods have dominated in the last few years as the default way to encode textual data, the most used being bpe @xcite , wordpiece @xcite and unigram @xcite . however, they have inherent flaws that limit their multilingual performance @xcite , their adaptability to new languages and new domains after pre-training @xcite and the downstream performance of language models in general @xcite . to alleviate these issues, tokenization-free (or character-level) models leverage characters instead of subwords to build text representations. some of the first neural networks for sequence generation used characters directly as inputs @xcite , and following works modified the approach to create input word representations based on characters @xcite @xcite . similar architectures were recently adapted to work with transformers @xcite . nevertheless, they still rely on fixed tokenization heuristics (for instance segmenting using whitespaces) which may not be suited to some languages or certain types of language variations. recent works have tried to remove these induced biases by working purely with characters or bytes as input @xcite @xcite . however, they either have to use various tricks to reduce the sequence lengths based on other induced biases like downsampling rates @xcite or have extremely low training and inference speeds @xcite . @xcite create tokens in a differentiable manner by predicting frontiers and using the representations of each character inside a "token", but it remains unclear how their model could be adapted to be used with newer architectures such as transformers. @xcite propose to segment tokens using a trained "frontier predictor." nevertheless, this differentiable tokenizer is not trained with the main language model objective but instead mimics a bpe subword tokenizer, carrying some of its flaws. ## training speedups in terms of speed, we compare our model to manta-lm small to t5 small counterparts: one that is trained at the classical subword-level, and one trained at byte-level, hence using sequences that are roughly 4 times longer. we also report the speed of the larger byt5 small architecture as described in @xcite . manta-lm is approximately 4 times faster than byte-level t5 small , and 5 times faster than byt5 small , which can be explained by the reduced sequence length we use in the encoder-decoder model. manta-lm is only 2.3 times slower than t5 small which furthermore benefits from already tokenized sequences at training time. ## conclusion in this work, we present manta, a fully differentiable module that learns to segment input byte sequences into blocks of arbitrary lengths, and constructs a robust representation for these blocks. we train this module jointly with an encoderdecoder lm on a span denoising objective to obtain manta-lm. we then show that manta-lm is more robust when applied to noisy or out-ofdomain data than models using static subword tokenizers. at the same time, it performs on par with fully byte-level models on these setups while operating with a much reduced computational cost. beyond the noisy and out-of-domain settings, we believe that our approach could lead to interesting results for a number of languages, especially those whose writing system do not use whitespace separators, such as chinese. finally, tokenizers are hypothesized to be an important limiting factor when segmenting multilingual data @xcite . we believe manta could be used in the multilingual setting to ensure a more balanced segmentation between languages.