Tokenizer Choice For LLM Training: Negligible or Crucial? (2024)

Abstract: The recent success of large language models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot.Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations. Our studies highlight that the tokenizer choice can significantly impact the model’s downstream performance and training costs. In particular, we find that the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model’s downstream performance. Furthermore, we show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English. While English-centric tokenizers have been applied to the training of multi-lingual LLMs in the past, we find that this approach results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary.

## introduction llms have shown impressive capabilities in many downstream tasks in a zero/few-shot setting such as summarization, reading comprehension, translation, and commonsense reasoning @xcite . to train a llm, the currently established approach is to employ a tokenizer that splits the training documents into tokens where a token represents a word @xcite , a sub-word @xcite @xcite , or a single character @xcite , and each token is represented in the model by an embedding vector that can be further processed. the quality of a tokenizer can be assessed intrinsically and extrinsically. an intrinsic evaluation solely addresses the characteristics of tokenizers and their generated output in isolation, whereas the extrinsic evaluation measures the impact of the tokenizer on a downstream component, e.g., the large language @xcite . while many different tokenization approaches have been proposed, ranging from character-based to word-based methods, the potential impact of different tokenizers is underexplored w.r.t. llms, especially in the context of multilingual llms. recent works proposed by @xcite and @xcite demonstrate that carelessly designed tokenizers applied to the training of multilingual llms result in severe inequalities and limitations across languages. text passages translated into different languages resulted in tokenized sequences that differ in length up to a factor of 15, affecting inference costs and latency during inference. furthermore, it is known that the learning of long-range dependencies @xcite , is an essential property for effectively learning transformer-based llms. given a fixed sequence length, learning to relate words far apart in the input text is impossible for languages whose text is excessively fragmented by the tokenizer. despite the importance of tokenizers and the potentially severe impact of poorly performing tokenizers, there exists no extensive study so far that holistically investigates the intrinsic and extrinsic tokenizer performance in a monolingual and multilingual setting with a focus on decoder-only models, which represent the backbone of current llms. in this work, we address this gap and conduct an extensive study in which we measure the impact of the tokenizer on the model performance. in particular, we make the following contributions: this section provides an overview of tokenization algorithms and their usage in encoder-and decoderonly transformer models. ## approach to investigate the tokenizer impact on the model performance, we conducted an extensive ablation study. in detail, we created dedicated datasets for the training of the tokenizers and the models, trained bpe and unigram tokenizers, and for each tokenizer we trained decoder-only models with a size of 2.6b parameters while keeping the remaining configuration (i.e., dataset and model hyper-parameters) fixed. this allowed us to measure the tokenizer's impact on the model's downstream performance in isolation. ## intrinsic tokenizer evaluation in our intrinsic evaluation, we first compare the fertility and parity of the trained tokenizers (section 4.1) and subsequently the overlap of their vocabularies (section 4.2). ## extrinsic tokenizer evaluation in the following, we describe the results of our extrinsic evaluation of tokenizers. section 5.1 describes the experimental setup, section 5.2 presents the downstream performance of the trained models based on the investigated tokenizers, and section 5.3 analyzes the computational costs associated with each tokenizer when employed in a specific model. ## correlation between intrinsic and as highlighted in the correlation heatmaps in fig. 4 , we find that there is no distinct correlation across all tasks and languages, demanding a more granular analysis. while for non-english tasks, we mainly observe a correlation between low fertility and higher downstream performance, the non-english tasks yield seemingly random positive and negative correlations. however, it should be noted that the number of multilingual tasks per language is much lower than for english and that for several multilingual tasks such as xsqa and lambada, a similar correlation behaviour between the english tasks and their translated version can be observed. taking the fertility trends with varying vocabulary sizes (see fig. 1 ) into consideration, we hypothesize that fertility only correlates with downstream performance in certain language-specific vocabulary size limits. for the english language, the tokenizers already provide low, close-to-convergence fertility scores for vocabulary sizes of 33k tokens. while additional tokens yield only minute fertility improvements, we presume that they do not capture morphological segmentations and, thus, can harm downstream performance and significantly increase the computation costs (see section 5.3) in the end. in contrast, for multilingual tokenizers, we observe significant fertility improvements with increasing vocabulary sizes. due to the larger thesaurus induced by the additional languages, the tokenizer requires a larger vocabulary to allow a model to perform convincingly on all languages. therefore, only within the non-convergence vocabulary range, we achieve a strong, negative correlation between fertility and downstream performance with varying vocabulary sizes. in conclusion, intrinsic tokenizer metrics such as fertility and parity need to be taken with a grain of salt and supposedly are only predictive of downstream model performance in certain bounds. low fertility scores might be regarded as a necessary criterion but not as a sufficient one. ## conclusion & future work this work represents a fundamental step to a better understanding of the impact of the tokenizer on the models' downstream performance. we have shown that training tokenizers with a balanced share across languages achieve comparable low fertility and parity scores across all languages, which has important implications. higher fertility results in up to 68% more computational costs during training and prevents the model from learning longrange dependencies in limited context windows. furthermore, we highlight that the tokenizer choice can significantly impact the model's downstream performance. we could show that the bpe algorithm applies well to mono-and multilingual settings. for english, we show that a vocabulary size of 33k is sufficient, whereas multilingual models based on our five considered languages require a up to three times larger vocabulary size. moreover, we could show that the sentencepiece library outperforms the huggingface tokenizer library. finally, we could demonstrate that there is no clear correlation between intrinsic and extrinsic tokenizer performance, but the correlation is rather task-specific. a small fertility value might be a necessary condition for good downstream performance but not a sufficient one. in the future, we aim to investigate tokenizers for a larger set of languages, including very diverse languages, and investigate the impact of alternative tokenization approaches such as sage @xcite ) that focus on context information during tokenizer training.