On the Effect of (Near) Duplicate Subwords in Language Modelling (2024)

Abstract: Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned random indices before being served to the LM. However, this process—while typically lossless—may lead to less efficient LM training, because it removes character-level information, thereby making it more difficult to generalise across similar subwords, such as *now* and *Now*. We refer to such subwords as **near duplicates**. In this paper, we study the impact of near duplicate subwords on LM training efficiency. First, we design an experiment that gives us an upper bound to how much we should expect a model to improve if we could perfectly generalise across near duplicates. We do this, by duplicating each token in our LM’s vocabulary, creating perfectly equivalent classes of subwords. Experimentally, we find that LMs need roughly 17% more data when trained in a fully duplicated setting. Second, we investigate the impact of naturally occurring near duplicates on LMs. Here, we see that deduplicating them considerably hurts LM performance; but that this loss in performance can be easily mitigated.

## introduction most modern language models (lms) do not have direct access to the bits or characters which make up the text that they must model. rather, they operate on higher-level units, so-called tokens, which are elements of a finite set of previously defined subwords. this set of subwords is typically obtained as the output of a tokenisation method and forms an lm's vocabulary (gage, * shared supervision. ## language modelling let σ be a vocabulary of subwords. a language model p is formally defined as a probability distribution over the set of all finite sequences of sub- in order for p to approximate the true distributions over natural strings p(w), we train this model to minimise its cross-entropy with p: ## subword duplication now, let's assume there exist in our alphabet pairs or groups of subwords which are nearly identicalin both their orthography and semantics. such near duplicates can arise from various sources, including but not limited to capitalization differences (e.g., now vs. now), typographical errors (language vs. langauge), the presence or absence of whitespace (e.g., the vs. _the), foot_2 and variations in spelling (e.g., modeling vs. modelling). the question we are concerned with is: how might such subword duplication affect the performance of language models? to define this question formally, let s be a set of disjoint sets of near duplicate subwords: we index these as s i to represent the i'th set of near duplicates. further, let @xmath0. to find out how duplication affects lms, we can create a map s : σ → σ which deduplicates subwords, mapping duplicates to the corresponding canonical symbols. this map is defined as: with σ @xmath1. we are now in a position to define a distribution over deduplicated subword sequences: in words, the conditional probability p s (c t | @xmath2. ## experimental setup we implement all of our experiments in the codebase of the languini kitchen @xcite . below, we provide an overview of the models and datasets used here. we refer the reader to @xcite work for more details regarding implementation choices, training setup, and dataset collection. model. we use the gpt model from languini, which is a gpt-2 style transformer decoder @xcite . unless otherwise noted, we use the "small" configuration with 12 layers, hidden size 768, and 85m non-embedding parameters. we train models with sequence length 512, batch size 128, the adam optimiser @xcite , and a cosine learning rate schedule from 6e-4 to 6e-6 with 500 warmup steps. data. we train on the languini training data, a filtered version of the books3 subset from the pile @xcite . for evaluation, we use the heldout languini test set, which contains 11m tokens. this data is pre-tokenised into a vocabulary of size 16k using a bpe tokeniser @xcite trained using sentencepiece @xcite . unless otherwise noted, we train our models for 18,265 steps-i.e., the first 1.2b tokens in our dataset-which corresponds to training the small gpt model for 6h on an rtx 3090 gpu; this is languini's gpt small 6h setting. evaluation. we generally report our model's perplexity on the test set as our evaluation metric. to ensure sufficient context for all predictions, we use a sliding window with steps of 128: we fill in a 512 tokens context, ignore the model's outputs on the initial 384, and evaluate it only using the last 128 tokens. for models p(c) trained on the deduplicated setting, we simply report their perplexities, defined as the exponentiated cross-entropy evaluated on a held-out test set d eval : ppl(c) = exp (l( p(c); d eval )). when evaluating models p(w), trained in the duplicated setting, we report their projected perplexity, defined as: or a synthetically duplicated vocabulary; this gives us an upper bound for the impact of real near duplicates, as synthetically duplicated subwords are perfectly comparable in terms of their semantics. in the natural duplication setting, we deduplicate the default vocabulary by merging real near duplicates. by comparing performance on the default and the deduplicated vocabulary, we verify whether the effect of natural near duplicates in lms is comparable to the effect of perfect duplicates. ## related work our experiments are partly inspired by a number of works on cross-lingual lm generalisation. these works also use duplicated vocabularies, terming the duplicates "fake-english". unlike our work, however, they sample entire sequences of either english or "fake-english" tokens (we perform token-wise i.i.d. sampling; @xcite @xcite . relatedly, huang et al.'s (2023) lex-invariant lms can be interpreted as an extreme case of token duplication; essentially creating infinite "fake languages", they show that lms can learn, to a certain extent, even in the absence of a fixed set of embeddings. more similar to our duplication method is the work of @xcite , who use duplicates sampled at the token level to isolate the effect of vocabulary size when investigating the role of bpe tokenisation in a model's ability to memorise training data. another set of related work studies subword regularisation techniques, such as, e.g., bpe dropout @xcite . certain character sequences can be represented by multiple equivalent sequences of subwords. these techniques then non-deterministically choose between these equivalent choices at training time, which can lead to improved lms. our perfect duplication experiments, where we train lms with either of two duplicated tokens, can be seen as similar to these methods. unlike them, however, our approach duplicates a model's vocabulary to introduce this ambiguity, which may explain the fact that our results diverge. relatedly, our projected entropy measure (which sums over subword duplicates) is analogous to the marginalisation proposed by @xcite , which sums over spurious tokenisations. ## conclusion in this paper, we investigate to what extent (near) duplicate subwords impact language modelling performance, conducting controlled experiments on synthetic perfect duplicates and natural near duplicates. we find that lms can generalise across duplicated subwords, although this incurs extra cost. when operating on a fully duplicated vocabulary, the lm is about 17% less data efficient. this number depends roughly linearly on the fraction of the vocabulary that is duplicated. assuming that roughly 40% of subwords are near duplicates in common lm vocabularies, our findings imply that lms with improved generalisation across duplicates, e.g., by modelling language at the characterlevel, could achieve data efficiency gains of up to 10%. this bound is reached for perfectly equivalent duplicates. however, we find that natural near duplicates are not perfectly equivalent: in practice, the potential for such performance improvements is likely limited.