Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance (2024)

Abstract: Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compression, that can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. We also demonstrate the empirical importance of compression for downstream success of pre-trained language models. We control the compression ability of several BPE tokenizers by varying the amount of documents available during their training: from 1 million documents to a character-based tokenizer equivalent to no training data at all. We then pre-train English language models based on those tokenizers and fine-tune them over several tasks. We show that there is a correlation between tokenizers’ compression and models’ downstream performance, suggesting that compression is a reliable intrinsic indicator of tokenization quality. These correlations are more pronounced for generation tasks (over classification) or for smaller models (over large ones). We replicated a representative part of our experiments on Turkish and found similar results, confirming that our results hold for languages with typological characteristics dissimilar to English. We conclude that building better compressing tokenizers is a fruitful avenue for further research and for improving overall model performance.

## introduction while language modeling pipelines employ a multitude of sophisticated techniques to achieve success in many nlp tasks, their presupposed tokenization, i.e., the step of discretizing text into processable units, is often done with less scrutiny or deviation from the common practices. this tokenization stage, which segments space-delimited words into subwords, forms the foundation of most large language models @xcite @xcite and influences their modus operandi in subsequent usage. among other open questions regarding tokenization, it is unclear whether tokenization is even needed @xcite @xcite and how much poor tokenization influences model performance, especially for non-english languages @xcite @xcite . as the tokenizers serve language models, it is straightforward that the primary method to assess their quality is by measuring their contribution to the model performance over the nlp tasks it is meant to solve, i.e., evaluating the tokenizers on tasks extrinsic to tokenization itself. however, this method requires pretraing expensive llms whenever an evaluation of a tokenizer is needed. for this reason an intrisic indicator of tokenization quality is warrented. and indeed the literature is teeming with intrinsic evaluations of tokenization. for example, @xcite used text compression as the main indicator of the tokenizer's intrinsic quality, whereas @xcite suggested assessing tokenizers based on segmentation overlap with a morphologically segmented reference. in this paper we carefully distinguish between intrinsic and extrinsic evaluation of tokenizers, and examine to what extent they are correlated. as a specific intrinsic evaluation we focus on compression, the metric underpinning bpe @xcite , the most prevalent tokenization algorithm that requires character co-occurrence statistics over a large corpus of raw text to achieve minimal length in tokens. we control the tokenizer's ability to compress by limiting its support, i.e., the amount of data available in the tokenizer's training corpus. by doing so we skew the statistics available to the tokenizer. we compared tokenizers trained with a million supporting documents to ones trained on less and less data, down to a single document, and to a character-level tokenizer, equivalent to zero support. we then pre-trained from scratch copies of a decoder-only transformer-based model @xcite , with the different tokenizers, and finetuned them on several downstream tasks. in this work we hypothesize that the downstream success should be correlated with the compression ability of the underlying tokenizers. we experimented with three model sizes, tokenizers of six different volumes of supporting data, and two languages, english and turkish. our results show that in terms of intrinsic performance, the tokenizers' compression ability is highly influenced by the amount of supporting data, with tokenizers trained on a minimal amount of data having tokenized texts more than 60% longer compared to the best compressing tokenizer. however, the discrepancy in compression is significantly more marked for less frequent words. extrinsically, we also found that downstream success monotonically increases with the increase in the tokenizer's support. the correlation between the intrinsic and extrinsic measures of tokenization quality points to the conclusion that better compressing tokenizers is a desired goal on the road to better language models. a conclusion that may be true even for models dealing with other modalities @xcite . while we evaluated the downstream performance on both classification and generative tasks, we observed that the correlation to compression is stronger for the latter type of tasks. this discrepancy could be attributed to the fact that generative tasks require the use of the tokenizer more exten-sively than in classification tasks, aligning with the number of generation steps involved. we therefore conclude that tokenization's effect is better assessed through generation tasks, rather than classification tasks. our results also show that smaller models are especially vulnerable to poor tokenizations, with the smallest 10m parameter model suffering from more significant drops in performance compared to our largest 1b model. finally, experimentation with turkish revealed the same trends, ruling out the option of an english-specific phenomenon. in the remainder of the paper we will describe the common practices in assessing tokenizers (section 2) and argue for the theoretical sensibility of compression as an intrinsic tokenization evaluation (section 3). we will then describe our experiments (section 4) and their results (section 5). ## measuring tokenization quality from the very early days of nlp, models have always assumed text discretized to tokens as input @xcite . for the most part, these tokens were whitespace separated words, but in the recent decade non-trivial tokenization algorithms have surfaced @xcite @xcite , primarily to deal with unseen tokens without smoothing techniques or other convoluted methods @xcite . the underlying reasoning behind all modern tokenization methods is that some subwords, e.g., morphemes, may carry independent information that is of value to the model even if the word as a whole is rare or unseen. therefore, better tokenization is assumed to improve models' performance tokenizer sentence char _t h i s _i s _a b o u t _c o m p r e s s i n g _t o k e n i z e r s 1-doc _th is _is _a b ou t _comp re ss ing _to k en i z ers 10-doc _this _is _about _comp res sing _to k en iz ers 100-doc _this _is _about _comp ress ing _tok en izers 1k-doc _this _is _about _comp ressing _to ken izers 1m-doc _this _is _about _compress ing _token izers over rare words, while also carrying computational benefits, like smaller models and the elimination of unknown tokens. it is not surprising then, that whenever tokenizers are presented or tested, they are usually accompanied with an array of evaluations that assess the tokenization's influence on the model's downstream success, mostly on translation tasks @xcite @xcite , although monolingual tasks are also used @xcite . other works circle back and assess tokenization with respect to the desiderata it is supposed to serve as a stand-alone algorithm, independently from the model trained on top of it. this is done usually in addition to evaluation over downstream performance. however, most works disagree on the desiderata themselves. many emphasize alignment to linguistically meaningful units @xcite @xcite or to human cognitive preferences in general @xcite . 1 others include analyses of token length and frequency @xcite , mostly in addition to the above, assuming that ideal tokenizers use longer and more frequent tokens. the two types of tokenization evaluations, extrinsic over downstream success and intrinsic over a plethora of metrics, are usually not compared directly. they are only used to demonstrate the superiority of a specific tokenizer, and the relations between the evaluation approaches is glossed over. in this work we explicitly focus on compression as a potential intrinsic indicator of tokenization quality, as has been suggested in past works in other settings @xcite , and check to what extent it is correlated with extrinsic downstream success. we conclude that compression is a desideratum for tokenization not only due to its theoretical virtue, expanded on in the next section, but first and foremost because it correlates with downstream performance. ## the role of compression in tokenization in the realm of intrinsic measures for evaluating tokenization quality, compression particularly stands out in prominence. it has garnered considerable attention, notably due to its pivotal role as the cornerstone of the byte pair encoding tokenization algorithm (bpe; @xcite , an algorithm that initially conceived for general data compression purposes @xcite . given data composed of sequences of atomic symbols, the algorithm minimizes the overall data length, for a given dictionary budget, by iteratively substituting a new symbol in place of the symbol pair most frequently occurring in a large corpus of supporting data. in the domain of language modeling, the symbols are usually characters and the supporting corpus is a subset of the text designated to be used as a training set for the language model which the tokenizer is meant to serve. but in a sense, compression-driven tokenization may be viewed as language modeling in and of itself. consider that language models are aimed at assessing and possibly maximizing the likelihood of produced texts expressed as a product of token probabilities, in terms of n-gram language modeling, where the probability of each token is approximated as dependent only on a context of length n -1, and the probability of x k given its context is further approximated by the number of appearances of the relevant n-gram in a training corpus, a compressor may be considered a 0-gram language model, where the relevant n-gram is of length 0 and the probability of each token is not even a function of its own frequency in the training data, setting uniformly, although simplistic when thinking about language modeling with predefined whitespaceseparated words, this type of objective is sensible when considering that it is used to determine the symbols themselves. from this point of view, prioritizing compression as an indicator for tokenization quality is very reasonable. since bpe optimizes an approximation, albeit crude, of the downstream objective, doing better under this approximated objective should translate into better downstream performance which will justify the focus on compression as a metric. moreover, from an information theoretic perspective, shannon's source coding theorem (shannon, 1948) links the limit on the compression to the entropy of the source of the data to be compressed. as language models aim to increase the log-likelihood of texts, hence decrease the entropy of the distribution, they inadvertently also increase the possible compression of the texts. our claim is that this relationship is symmetric, and bpe tokenizers, as they compress texts, may also inadvertently increase their log-likelihood. we set to empirically examine our hypothesis by assessing the correlation between the tokenizer's compression ability and the performance of language models of various sizes over a set of downstream tasks. to explicitly control the compression ability, while fixing any other intervening factors as much as possible, we deal only with bpe tokenizers. this is in contrast with other works that compared compression across different tokenization algorithms @xcite . to create bpe tokenizers with varied compression rate we recall that bpe's maximal compression is guaranteed only over its supporting corpus. normally, for large enough corpora, a minimal discrepancy is assumed between the character distribution in the corpus and the "true" distribution in the language. in this work however, we explicitly emphasize and expand this discrepancy by limiting the size of the support to a great extent. we will show that this intervention severely hinders the compression capacity of the tokenizer and that it also leads to deteriorating downstream performance. ## analysis from mc4). we averaged the token-per-word ratio over all words whose occurrences are of the same order of magnitude and provided the number of words in each bin. a similar analysis was done for turkish and it is shown in figure 4 . the figures show that the token-to-word ratio is extremely similar across tokenizers for words that are the most frequent. on the other hand, the different tokenizers diverge in token-to-word ratio when presented with rarer words, with less supported tokenizers being more sensitive to word frequency, compared to better supported tokenizers. it is worth noting that the same trend applies to the char tokenizer, for which the number of tokens per word is simply its length in characters. this should not be surprising due to the tendency of frequently used words to be shorter in accordance to zipf's law of abbreviation @xcite . performance over frequent words to complement the analysis above we also broke down the results of the generation tasks by average frequency of the words in the targeted output of each example. the results, plotted in figure 5 , include the difference in rouge-l per example from the best 1m-doc model per task and model size. it shows that the differences between differently tokenizing models are more pronounced over examples with rarer words. together, this and the previous analysis shed some light on the reasons for the correlation found in our main result. we demonstrate that the differences in performance between the various models are indeed more pronounced in the presence of rarer words, which are exactly the words that the tokenizers compress differently. it is thus highly probable that word frequency is a major confounding factor the connects compression with downstream performance. in addition, this analysis may point to the benefit of challenge sets, comprised of examples with rarer words, in the evaluation of tokenization. ## conclusions in this paper we demonstrated the importance of compression to tokenization as an intrinsic evaluation of tokenization quality that indicates the performance on extrinsic downstream tasks. we argued in favor of compression-driven tokenization from a theoretical perspective, since it may make the tokenizer act as a simple standalone language model, and we showed its correlation with downstream model success. our experiments point to generation tasks as better downstream evaluators of tokenization since their results are both more sensitive to the tokenizer and better correlate with tokenization quality as expressed in compression ability. in terms of linguistic diversity, the similarity in the results and analyses across two very different languages, english and turkish, points to our conclusions being independent of specific typological characteristics. yet, ample room is left for studying the effects of tokenization on more languages that are even more typologically diverse. moreover, other intrinsic evaluators are still to be assessed even for the languages we did work with. we conclude that tokenization matters, as poorly compressing tokenizers hinder the results of language models, and that investment in better compressing tokenizer has a potential of improving model performance while being relatively cheap in terms of compute. we therefore call for research to better understand the factors controlling the quality of the tokenization and its relation to overall success of the llms.