Tokenization is Sensitive to Language Variation (2025)

Abstract: Variation in language is ubiquitous and often systematically linked to regional, social, and contextual factors. Tokenizers split texts into smaller units and might behave differently for less common linguistic forms. This might affect downstream LLM performance differently on two types of tasks: Tasks where the model should be robust to language variation (e.g., for semantic tasks like NLI, labels do not depend on whether a text uses British or American spelling) and tasks where the model should be sensitive to language variation (e.g., for form-based tasks like authorship verification, labels depend on whether a text uses British or American spelling). We pre-train BERT base models with the popular Byte-Pair Encoding algorithm to investigate how key tokenization design choices impact the performance of downstream models: the corpus used to train the tokenizer, the pre-tokenizer and the vocabulary size. We find that the best tokenizer varies on the two task types and that the pre-tokenizer has the biggest overall impact on performance. Further, we introduce a new approach to estimate tokenizer impact on downstream LLM performance, showing substantial improvement over metrics like Rényi efficiency. We encourage more work on language variation and its relation to tokenizers and thus LLM performance.

## introduction variation in language is ubiquitous, manifesting in forms such as lexical variation (e.g., vocabulary choice like "lift" vs. "elevator"), spelling variation (e.g., spelling "u" instead of "you"), and syntactic variation (e.g., word order like "i gave her the ball." and "i gave the ball to her."). such variation is often systematic rather than random, and is linked to but regional, social, and contextual factors @xcite @xcite . tokenizers break up input strings and determine the tokens that are fed into language models. to-tasks robust to language variation e.g., nli guy learnt guitar a man learned an instrument. tasks sensitive to language variation e.g., authorship verification guy learnt guitar a man learned an instrument. ## related work next to byte-pair encoding @xcite , there are several other subword (and other) tokenizers @xcite . for example, character and byte-based tokenizers have been argued to be more robust to spelling variations @xcite @xcite . in this study we focus on bpe as it has been the most common tokenization algorithm in recent llms (e.g., llama 3, mixtral, deepseek v3 and gpt-4). there is no universally agreed-upon standard to evaluate tokenizers. tokenizers have been evaluated intrinsically (i.e., without training larger lms) and extrinsically (i.e., considering the performance of larger lms pre-trained with the considered tokenizer). common intrinsic methods include: the average number of subwords produced per word and correlated measures like corpus token count @xcite @xcite @xcite , information theoretic measures like shannon entropy or rényi efficiency @xcite and morphological alignment @xcite . extrinsic measures include lm perplexity @xcite @xcite , cross-entropy loss (rajaraman et al., 2024), computational training cost @xcite and downstream task performances @xcite . it is computationally infeasible to train sota llms endto-end for each version of the tokenizer one wants to evaluate. recent work tackled this by training "smaller" generative language models with 350m-2.5b parameters for each tokenizer @xcite . however, even such smaller models still can take several days to train. we measure downstream task performance on models with 110m parameters taking less than 15 gpu hours to train per model. tokenizer algorithms and settings can affect an llm's performance, for example, on tasks including numbers like arithmetic @xcite , on tasks including domain specific vocabulary or jargon like coding or medicine @xcite @xcite , on different scripts and languages @xcite @xcite @xcite and when translating between languages @xcite @xcite . to the best of our knowledge monolingual tokenizers have been underinvestigated in relation to language variation. monolingual tokenizers and tokenizer settings have recently been investigated on broader selections of nlu tasks @xcite , presumably with the underlying assumption that for a given language like english, there exists a best tokenizer for most, if not all, tasks. we investigate this assumption for two systematically different types of tasks: tasks requiring robustness and tasks requiring sensitivity to language variation. ## tokenizer settings tokenizers break up texts into smaller units. these units are fed into the language model as input. we investigate different variations of the most popular tokenization algorithm: byte-pair encoding or bpe @xcite . bpe is a subword tokenization algorithm (i.e., it breaks rare words down into subword units, bottoming out in bytes in the worst case). the vocabulary is built iteratively: it starts out with a base vocabulary of distinct bytes and merges them based on token frequency in the fitting corpus until the desired vocabulary size is reached. we vary the algorithm on three parameters: (1) the vocabulary size, (2) the pre-tokenizer and (3) the fitting corpus. ## evaluation tasks we compare tokenizers on classification tasks that require robustness to language variation ( §4.1) and tasks that require sensitivity to language variation ( §4.2). models solving these two types of tasks should need to make use of more semantic and form-based signals respectively and might have different requirements for a tokenizer (cf. figure 1 ). we select tasks that strike a balance between being sufficiently challenging and staying within the capabilities of our pre-trained bert models. see an overview of selected tasks in ## modeling for each investigated tokenizer (cf. §3), we pretrain three bert models with different seeds. we use encoder instead of decoder models, as encoder models tend to reach higher performance for classification tasks for low parameter settings. this allows us to train models using fewer gpu hours. a bigger model size p and fewer tokens t improves the training loss. chinchilla's scaling law might also hold for smaller encoder models, specifically optimal parameter count could scale with the token size for a fixed compute p op t ≈ t 23/27 (hoffmann et al., 2024). for the remainder of this work, we use the base bert model architecture with 110m parameters, initialize all weights randomly and pre-train on 750m tokens sampled in sequences of 512 from the miscellaneous corpus ( §3.3) and use a batch size of 32 and 45k steps. for further details and hyperparameters, see appendix c. fine-tuning bert. unless otherwise specified, we use 3 epochs, a max sequence length of 128, a batch size of 32 and a learning rate of 2e-5 to perform the classification tasks. we evaluate on the dev set for glue. for comparability, we use the same setup for tasks requiring sensitivity to language variation. only for the authorship verification task we use a contrastive training setup, then use the dev set to find an optimal cosine similarity threshold and calculate accuracy on the test set. ## results we show the performance of fine-tuned bert models on tasks that require robustness to language variation in note that the performance differences tend to be relatively small, which is consistent with previous work comparing different tokenizer algorithms on downstream tasks @xcite . to ensure significance, we compute the pairwise mcnemar, 1947's test for the pre-tokenizer, fitting corpus and vocabulary size settings, see figure 2. for the significance testing, we consider classifications by models with the same settings but different seeds to be stemming from the same rater. we use the bonferroni correction @xcite for our total of 26 tests. rq1: tokenizer settings perform differently on tasks robust and sensitive to language variation. overall, tasks sensitive to language variation profit from tokenizers that encode more variation through a larger vocabulary size than 4k ( §6.3). note that the best-performing tokenizer settings are not always consistent across the individual variation-robust and variation-sensitive tasks (e.g., vocabulary size for glue+typo and glue in pre-tokenizer performance differs more between individual tasks than between task types. for both task types, _ws is among the best performing pre-tokenizers. combining leading whitespaces with letters (_ws) generally improved performance over isolating whitespaces (ws). notably, _ws cuts the input tokens to a model in half as single whitespaces between words are not separate tokens anymore. for dialect and spelling-transformed glue, core, nucle and dialect, the whitespace-based pre-tokenizers perform the best. for av, pan and the original glue tasks, gpt2 and llama perform better. tasks like grammatical error detection (nu-cle) could be seen as identifying deviations from a standard, and might benefit from pre-tokenizers that include tokens with typical combinations of unicode character categories. one of the main differences of gpt2 and llama to other pre-tokenizers is that the combination of different unicode character categories (e.g., punctuation and letters) within the same token is less often allowed. by preventing combinations of categories like punctuation and letters, the vocabulary of gpt2 can include a broader range of tokens that only consist of letters, e.g., "_queer" in ## pre-evaluating downstream tokenizer impact a popular alternative to training several larger lms to evaluate tokenizers is using fast intrinsic measures on the token distribution on the downstream we display examples of sequences tokenized by tokenizers with varying fitting corpora, pre-tokenizers and vocabulary size. we represent whitespaces within tokens as _. the domain of the fitting corpus affects the set of unique words in the vocabulary (e.g., "queer" is part of twitter but not pubmed). the size determines the number of "rarer" words in the fitting corpus that are added to the vocabulary (e.g., the british variant "jumper" of "sweater" is only added with size 64k). the pre-tokenizer determines what types of words can be part of the vocabulary (e.g., "b4" can not be part of gpt2 and llama3). note that allowing fewer unicode character category combinations (e.g., numbers and letters) increases the broadness of the vocabulary (e.g., in our fitting corpora, gpt2 allows for fewer combinations than other pre-tokenizers and is the only one to include "queer" in its vocabulary). ## conclusion in this work, we investigated tokenizer settings for tasks that require robustness and tasks that re-