An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers (2022)

Abstract: We introduce FLOTA (Few Longest Token Approximation), a simple yet effective method to improve the tokenization of pretrained language models (PLMs). FLOTA uses the vocabulary of a standard tokenizer but tries to preserve the morphological structure of words during tokenization. We evaluate FLOTA on morphological gold segmentations as well as a text classification task, using BERT, GPT-2, and XLNet as example PLMs. FLOTA leads to performance gains, makes inference more efficient, and enhances the robustness of PLMs with respect to whitespace noise.

## introduction the first step in nlp architectures using pretrained language models (plms) is to map text to a sequence of tokens corresponding to input embeddings. the tokenizers used to accomplish this have been shown to exhibit various undesirable properties such as generating segmentations that blur word meaning @xcite @xcite and generalizing suboptimally to new domains @xcite @xcite . in this paper, we propose flota (few longest token approximation), a simple yet effective method to mitigate some shortcomings of plm tokenizers. flota is motivated by the following hypothesis: rather than finding a segmentation that covers all characters of a word but destroys its morphological structure, it can be more beneficial to find a segmentation that does not cover all characters but preserves key aspects of the morphology. we confirm this hypothesis in this paper. our study investigates three plms and corresponding tokenizers: bert (base, uncased; @xcite , which uses wordpiece @xcite , gpt-2 (base, cased; @xcite , which uses byte-pair encoding (bpe; @xcite , and xlnet (base, cased; @xcite , which uses unigram @xcite . we find that flota increases the morphological quality of all tokenizers as evaluated on human-annotated gold segmentations as well as the performance of all plms on a text classification challenge set. contributions. we introduce flota, a simple yet effective method to improve the tokenization of plms during finetuning. flota uses the vocabulary of a standard tokenizer but tries to preserve the morphological structure of words during tokenization. we show that flota has three advantages compared to standard tokenization: (i) it can increase the performance of plms on certain tasks, sometimes substantially; (ii) it makes inference more efficient by shortening the processed token sequences; (iii) it enhances the robustness of plms with respect to certain types of noise in the data. all this is achieved without requiring any additional parameters or resources compared to vanilla plm finetuning. we also release a text classification challenge set that can serve as a benchmark for future studies on plm tokenizers. foot_0 ## few longest token approximation let v be a set of tokens that constitute the vocabulary of a tokenizer. for the tokenizers discussed in this paper, v contains words, subwords, and characters. let φ be a model used by the tokenizer to map text to a sequence of tokens from v . flota (few longest token approximation) discards φ and uses v in a modified way. given a word w not in v , flota tokenizes it by determining the longest substring @xmath0, returning s, and recursing on w \ s, the string(s) remaining when s is removed from w. we stop after k recursive calls or when the residue is null. figure 1 provides pseudocode. for the example word undesirable and @xmath1, flota first searches on maxsubwordsplit(w, v ) flota is based on a recursive function flotatokenize that uses a hash table f to store the longest substring s and its index i on each recursive call. s and i are found by means of a second function maxsubwordsplit, which also returns a residue r. in practice, to ensure correct indexing throughout different recursive calls as well as prevent using discontinuous substrings for tokenization, we compute r using an operation ⊕ j that concatenates two strings by putting j (length of s) hyphens between them. the recursion stops after k recursive calls or when r only consists of hyphens (determined by a boolean function hyphen). the hash table returned by flotatokenize is converted to a tokenization using a simple wrapper function that sorts the found substrings by their indices (not shown). if maxsubwordsplit does not find a substring @xmath2, flotatokenize returns an empty hash table (not shown). flota is guided by the following observations: many words not in v are made up of smaller and typically more frequent elements that determine their meaning (e.g., they are derivatives such as undesirable); many of these elements are in v . 2 by recursively searching for the longest substrings, we hope to recover the most important meaningful 2 existing tokenizers have been shown to be able to recover these elements only to a very limited extent @xcite @xcite . elements. this is also why it makes sense to stop after k recursions: if flota returns the most important meaningful elements as the first few tokens, we expect to not lose much by stopping. ## evaluation on gold segmentations english inflection is simple, but the language has highly complex word formation, i.e., derivation and compounding @xcite . to evaluate the morphological quality of flota against the standard tokenizers, we thus focus on derivatives and compounds. data. our evaluation uses celex @xcite and ladec @xcite , two large datasets of human-annotated gold segmentations of morphologically complex words. we merge both datasets and extract all words consisting of a prefix and a stem (prefixed derivatives), a stem and a suffix (suffixed derivatives), or two stems (compounds). we create for each plm a subset of words where both morphological elements (i.e., stems and affixes) are in the tokenizer vocabulary, but the word itself is not in the tokenizer vocabulary. in such cases, a word needs to be segmented, and it is guaranteed that the gold segmentation is possible given the tokenizer vocabulary. this procedure results in 11,272, 11,253, 10,848 words for bert, gpt-2, xlnet, respectively. experimental setup. we define three metrics to analyze how closely flota matches the gold segmentations. we compare against two alternative tokenization strategies: representing words as the k first tokens returned by the standard tokenizer (first) and representing words as the k longest tokens returned by the standard tokenizer (longest). recall that the wordpiece tokenization of the running example undesirable is und, es, ira, ble. with @xmath3, first is und, es, ira (i.e., it simply returns the first k tokens) and longest is und, ira, ble (i.e., it returns the k longest tokens in the order in which they occur in the standard tokenization). ## evaluation on downstream task we investigate whether the enhanced quality of flota tokenizations translates to performance on downstream tasks. we focus on text classification as one of the most common tasks in nlp. data. we create two text classification challenge sets based on arxiv, foot_2 each consisting of three datasets. specifically, for the subject areas of computer science, maths, and physics, we extract titles for the 20 most frequent subareas (e.g., computation and language). we then sample 100/1,000 titles per subarea, resulting in three text classification datasets of 2,000/20,000 titles each, which we bundle together as arxiv-s/l. ensures that arxiv-s/l require challenging generalization from a small number of short training examples with highly complex language. see appendix a.1 for more details. experimental setup. we split the six datasets of arxiv-s and arxiv-l into 60% train, 20% dev, and 20% test. we then train the three plms with classification heads on the six train splits, once with the standard tokenizers and once with flota. see appendix a.2 for hyperparameters. for flota, we treat k as an additional tunable hyperparameter. we use f1 as the evaluation metric. performance. the flota models perform better than the models with standard tokenization, albeit to varying degrees for the three plms ( the advantage of flota mirrors the differences observed in the morphological analysis, indicating that flota helps close the morphological quality gap between standard tokenizations and gold segmentations. where the gap is large, gains due to flota are large (gpt-2/bpe); where it is small, gains due to flota are small (xlnet/unigram). bert/wordpiece again lies in between. ## limitations while we find flota to work well on text classification, there are tasks for which flota might prove a less suitable tokenization method: e.g., for small values of k, flota often discards suffixes, which can be important for tasks with a syntactic component such as pos tagging. similar considerations hold for transfer to languages other than english: e.g., in the case of languages with a non-linear morphology such as arabic, flota is expected to inherit the insufficiencies of the underlying tokenizer @xcite . ## related work the question how plms are affected by their tokenizer has attracted growing interest recently. bostrom and durrett (2020), church (2020), @xcite @xcite focus on the linguistic properties of tokenizers. we contribute to this line of work by conducting the first comparative analysis of all three common plm tokenizers and releasing a challenge set as a benchmark for future studies. another strand of research has sought to improve plm tokenizers by training models from scratch @xcite @xcite or modifying the tokenizer during finetuning, mostly by adding tokens and corresponding embeddings @xcite @xcite . flota crucially differs in that it can be used during finetuning but does not add any parameters to the plm. furthermore, there has been work improving tokenization by variously exploiting the probabilistic nature of tokenizers @xcite @xcite . by contrast, our method does not need access to the underlying model. our study also relates to computational work on derivational morphology @xcite @xcite @xcite and word segmentation @xcite @xcite @xcite @xcite . we are the first to systematically evaluate the segmentations of plm tokenizers on human-annotated gold data. conceptually, the findings of our study are in line with evidence from the cognitive sciences that knowledge of a longer (i.e., more detailed and informative) sequence takes priority over any knowledge about smaller sequences @xcite @xcite . ## conclusion we introduce flota (few longest token approximation), a simple yet effective method to improve the tokenization of pretrained language models (plms). flota uses the vocabulary of a standard tokenizer but tries to preserve the morphological structure of words during tokenization. flota leads to performance gains, makes inference more efficient, and substantially enhances the robustness of plms with respect to whitespace noise.