Tokenization Is More Than Compression (2024)

Abstract: Tokenization is a foundational step in natural language processing (NLP) tasks, bridging raw text and language models. Existing tokenization approaches like Byte-Pair Encoding (BPE) originate from the field of data compression, and it has been suggested that the effectiveness of BPE stems from its ability to condense text into a relatively small number of tokens. We test the hypothesis that fewer tokens lead to better downstream performance by introducing PathPiece, a new tokenizer that segments a document’s text into the minimum number of tokens for a given vocabulary. Through extensive experimentation we find this hypothesis not to be the case, casting doubt on the understanding of the reasons for effective tokenization. To examine which other factors play a role, we evaluate design decisions across all three phases of tokenization: pre-tokenization, vocabulary construction, and segmentation, offering new insights into the design of effective tokenizers. Specifically, we illustrate the importance of pre-tokenization and the benefits of using BPE to initialize vocabulary construction. We train 64 language models with varying tokenization, ranging in size from 350M to 2.4B parameters, all of which are made publicly available.

## introduction tokenization is an essential step in nlp that translates human-readable text into a sequence of distinct tokens that can be subsequently used by statistical models @xcite . recently, a growing number of studies have researched the effects of tokenization, both in an intrinsic manner and as it affects downstream model performance @xcite @xcite @xcite . to rigorously inspect the impact of tokenization, we consider tokenization as three distinct, sequential stages: as an example, segmentation might decide to split the text intractable into "int ract able", "in trac table", "in tractable", or "int r act able". we will refer to this step as segmentation, although in other works it is also called "inference" or even "tokenization". the widely used byte-pair encoding (bpe) tokenizer @xcite originated in the field of data compression @xcite . @xcite argues that it is effective because it compresses text to a short sequence of tokens. @xcite varied the number of documents in the tokenizer training data for bpe, and found a correlation between ctc and downstream performance. to investigate the hypothesis that having fewer tokens necessarily leads to better downstream performance, we design a novel tokenizer, pathpiece, that, for a given document d and vocabulary v, finds a segmentation with the minimum possible k d . the pathpiece vocabulary construction routine is a top-down procedure that heuristically minimizes ctc on a training corpus. pathpiece is ideal for studying the effect of ctc on downstream performance, as we can vary decisions at each tokenization stage. we extend these experiments to the most commonly used tokenizers, focusing on how downstream task performance is impacted by the major stages of tokenization and vocabulary sizes. toward this aim, we conducted experiments by training 64 language models (lms): 54 lms with 350m parameters; 6 lms with 1.3b parameters; and 4 lms with 2.4b parameters. we provide open-source, public access to pathpiece 1 , and our trained vocabularies and lms 2 . ## pathpiece several efforts over the last few years @xcite have suggested that the empirical advantage of bpe as a tokenizer in many nlp applications, despite its unawareness of language structure, can be traced to its superior compression abilities, providing models with overall shorter sequences during learning and inference. inspired by this claim we introduce pathpiece, a lossless subword tokenizer that, given a vocabulary v and document d, produces a segmentation minimizing the total number of tokens needed to split d. we additionally provide a vocabulary construction procedure that, using this segmentation, attempts to find a v minimizing the corpus token count (ctc). 5 pathpiece provides an ideal testing laboratory for the compression hypothesis by virtue of its maximally efficient segmentation. ## experiments we used the pile corpus @xcite for language model pre-training, which contains 825gb of english text data from 22 high-quality datasets. we constructed the tokenizer vocabularies over the minipile dataset @xcite , a 6gb subset of the pile. we use the mo-saicml pretrained transformers (mpt) decoderonly language model architecture. 8 appendix b gives the full set of model parameters, and appendix d discusses model convergence. ## results ## analysis we now analyze the results across the various experiments in a more controlled manner. our experiments allow us to examine changes in each stage of tokenization, holding the rest constant, revealing design decisions making a significant difference. 13 ## conclusion we investigate the hypothesis that reducing the corpus token count (ctc) would improve downstream performance, as suggested by @xcite and @xcite when they varied aspects of bpe. when comparing ctc and downstream accuracy across all our experimental settings in figure 3 , we do not find a clear relationship between the two. we expand on the findings of @xcite who did not find a strong relation when comparing 3 tokenizers, as we run 18 experiments varying the tokenizer, initial vocabulary, pre-tokenizer, and inference method. our results suggest compression is not a straightforward explanation of what makes a tokenizer effective. finally, this work makes several practical contributions: (1) vocabulary size has little impact on downstream performance over the range of sizes we examined ( §5.1); (2) five different tokenizers all perform comparably, with none outperforming at statistical significance ( §5.2); (3) bpe initial vocabularies work best for top-down vocabulary construction ( §6.3). to further encourage research in this direction, we make all of our trained vocabularies publicly available, along with the model weights from our 64 language models.