Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models (2024)

Abstract: The disconnect between tokenizer creation and model training in language models allows for specific inputs, such as the infamous SolidGoldMagikarp token, to induce unwanted model behaviour. Although such ‘glitch tokens’, tokens present in the tokenizer vocabulary but that are nearly or entirely absent during model training, have been observed across various models, a reliable method to identify and address them has been missing. We present a comprehensive analysis of Large Language Model tokenizers, specifically targeting this issue of detecting under-trained tokens. Through a combination of tokenizer analysis, model weight-based indicators, and prompting techniques, we develop novel and effective methods for automatically detecting these problematic tokens. Our findings demonstrate the prevalence of such tokens across a diverse set of models and provide insights into improving the efficiency and safety of language models.

## introduction large language models (llms) have undergone remarkable advancements, becoming increasingly capable of understanding and generating humanlike text. while most components of these models are trained in an unsupervised fashion on vast amounts of data, the tokenizer typically remains a separately trained component based on custom algorithms and smaller datasets. gpt-2 laid the foundation for much of currentday transformer-based language modelling @xcite , including a framework for tokenization building on previous work in byte-pair encoding (bpe) @xcite , that has since been widely adopted. tokenization using bpe converts input text to a sequence of subword tokens by iteratively merging two neighbouring tokens using a fixed set of merge rules. these rules are learned using a greedy training algorithm on a smaller dataset, which is ideally representative of the llm's training data. recent work in this area has primarily focused on techniques to remove the need for tokenization altogether by moving to raw byte input @xcite . this choice typically comes at a significant inference speed cost, which can be compensated for by specialized architectures at the initial and final layers @xcite , or variable compute at intermediate layers (slagle, 2024). however, these techniques have not been widely adopted, and the vast majority of contemporary models still rely on subword tokenization. the main alternative to bpe for subword tokenization is the unigram method @xcite , which despite work suggesting it outperforms bpe @xcite is not in common use. for an in-depth overview of tokenization methods and their history, see @xcite . despite its widespread use, the tokenization step has generally been found to be unsatisfactory, responsible for many unwanted llm behaviours @xcite . the disconnect between tokenizer and model training creates the potential for some tokens to rarely or never be seen in training. the presence of such tokens in model inputs can lead to unexpected model behaviour including hallucination or the generation of garbled outputs, leading to such tokens commonly being referred to as 'glitch tokens' @xcite . we refer to these as 'under-trained' or 'untrained' tokens, reserving the latter term only for cases in which we have clear indication that the specific token had no model training data occurrences. the presence of such under-trained tokens has several drawbacks. firstly, they occupy capacity in a fixed-size tokenizer that could be better utilized for more frequently occurring tokens, reducing average input/output length and inference costs. secondly, their deliberate or accidental presence in input data has the potential to cause unwanted model outputs and break downstream applications. robustness to such unexpected or malicious input data is increasingly important with the proliferation of tool use and agents in llms that retrieve and process external data. lastly, these tokens can potentially be exploited to more easily circumvent guardrails by pushing the model beyond its trained distribution @xcite . although previous work exists on identifying such tokens through model and tokenizer analysis @xcite @xcite , there is a lack of reliable, automated methods that are well-tested and perform consistently across a wide range of models. automated tools for detecting tokenizer issues provide ways to test and iteratively improve the development of tokenizers, and can also provide methods for protecting deployed models from unwanted inputs, for example through sanitization. in this work, we present effective and efficient techniques for identifying such problematic tokens based on the model embedding weights and tokenizer configuration. we apply these methods to a wide range of popular and recent open-weight models. finally, we include a brief exploration of extensions of these techniques to closed-source models. to the best of our knowledge, this is the first work to present a set of automated, efficient, and theoretically sound methods that systematically and demonstrably identify 'glitch' tokens across various models and tokenizers. we also publish a general analysis tool compatible with hugging face models @xcite , along with detailed results for each analyzed model. ## methods our approach consists of three main steps: i) first, we perform an in-depth tokenizer analysis by inspecting its vocabulary and observing its encoding and decoding behaviour, ii) second, we calculate several indicators to identify candidate undertrained tokens, and iii) third, we verify whether identified candidate tokens are indeed out of distribution by prompting the target model. ## results in this section, we present a summary of our key findings on under-trained token detection. given the model-specific nature and the extensive volume of results, we elaborate on some common findings and showcase representative examples for particular models. comprehensive reports covering an increasing number of tested models and token types are available in our repository. ## application to closed-source models as our techniques involve directly using model weights, they are not directly applicable to closedsource models whose weights are not publicly available. however, the experience gained in inspecting a large variety of open-weight models provides insight which we adapt and transfer to closed models. for these tests, we use a custom prompt designed to exactly repeat strings and see if models appear incapable of doing so. for details of prompts and results, see appendix d. mistral's flagship api models do not consistently include information about tokenizers, but tokenizers are available for their openly-released models. due to a confirmed leak of an early version of their 'medium' model as 'miqu', we have some indication of the 'medium' model being potentially derived from llama2 70b. by prompting both the 'medium' and 'large' models, we can confirm that the 'medium' model is unable to repeat strings that are typically under-trained in llama2 models, and the 'large' model fails on typical tokens from the 'small' and 'mixtral' model series. in addition, in experimenting with such prompts we find that the 'large' model occasionally responds with special tokens including [tool_calls] and [control_331], which were recently confirmed to be part of the tokenizer for the 8x22b model, further highlighting the effectiveness of this approach. anthropic's models have limited documentation on their tokenizers. the anthropic sdk contains some tokenizer utilities for claude 2, with remarks that they are not accurate for claude 3. using the tokenizer provided for claude 2, we can identify some candidates for intermediate fragments that are likely under-trained by looking for long tokens which are included as part of even longer tokens. this results in candidates such as candidatefacinum (icandidatefacinum), trileptonpattuple (trileptonpattuplemc), bfrontend (dvbfrontend) and others. some of these tokens can be confirmed as problematic in claude 2.1, although none appear effective in the claude 3 family of models, consistent with the change in tokenizer implied by the sdk code. openai's models have well-documented tokenizers from the tiktoken package. in addition, by using models that share a tokenizer (refer to section 3.2), we already have access to a list of potential under-trained token candidates for gpt-3.5 and gpt-4, including _forcanbeconverted, $postalcodesnl, useralative, _typingsjapgolly, and others. we find that all openai models older than gpt-4o fail to handle many of them correctly, resulting in hallucinations followed by an inability to tell the difference between the inputs and incorrect outputs, or model output degrading into repetition. the gpt-4o model family uses a different tokenizer with a larger vocabulary, but the same techniques for tokenizer analysis are effective in finding under-trained tokens, including ម្បី , which appears to induce an 'end of text' token, as well as various tokens apparently derived from chinese advertisements, such as _天天中彩票app. ## discussion and conclusion the presence of under-trained tokens has several negative consequences for language models, including inefficient inference and the potential to bypass guardrails. our investigations show that a wide variety of untrained and under-trained tokens are present in model tokenizers, across a wide variety of model classes. even with our relatively conservative threshold for verification, we detect the presence of such tokens across all tested models, with typically around 0.1-1% of the vocabulary consisting of severely under-trained tokens, although their prevalence varies significantly. the most salient factors contributing to a model having many under-trained tokens, aside from simply having a large vocabulary, appears to be whether the tokenizer was trained on similar data as the model. models which re-use a large external tokenizer, and then train on distinct data from scratch, are among those with the highest number of detected under-trained tokens. analyzing tokenizer characteristics directly can provide immediate sigal helping to identify several of these issues without the need for any additional training. this includes detecting unreachable tokens that fail to encode back to their original text representation, and unused byte fallback tokens. this approach can also be particularly useful to quickly detect and mitigate errors with tokenizer configuration, which appears to be a particularly common issue when a custom vocabulary is manually added to that of the original tokenizer. additionally, using the model embedding weights directly is a reliable way to detect under-trained tokens, although care should be taken to account for model architecture. based on our findings, we summarize a number of recommendations within the scope of current llm development tooling. firstly, we recommend ensuring that input data pre-processing is identical across tokenizer training data, model training data, and model inference. in particular, by carefully considering how to handle carriage returns, and special tokens present as plain text in training data and user input. secondly, careful consideration of tokenizer training data is re-quired, ensuring that it is representative of model training data. next, after training a tokenizer, we recommend checking for unreachable tokens by encoding and decoding the vocabulary to ensure that manually added tokens are handled correctly. finally, when training base models, checking for under-trained tokens after smaller test runs, or testing on a different corpus to reveal pre-processing bugs that may cause unrepresentative inputs in the main training data, provides a valuable sanity check. in addition to providing a set of useful tools for improving models and tokenizers, our work indicates several directions for future research. firstly, the results from starcoder2 (see section 3.2) highlight a potential limitation in bpe training where occurrences in a single document or repository can define a token by themselves. strategies to prevent this, such as limiting the count for pairs to be merged by document, can be explored to prevent this. secondly, we note that byte-based bpe tokenization produces more intermediate fragments which additionally have the ability to cause outputs to be undecodable. the trade-off between more efficient encoding methods and these downsides is particularly under-explored. although allowing such tokens may lead to lower average token counts, this also leads to the presence of more untrained 'fragments' and tokens which are less semantically meaningful. techniques such as bpedropout @xcite have been proposed to compensate for under-trained intermediate fragments, but direct comparisons on state-ofthe-art models are lacking. finally, we observe differences across models in terms of how weight decay is applied to tokens that are not present in the input, including not applying weight decay to embeddings, applying it only to tokens seen in a batch, or applying it across all model weights. this choice may affect the ability of models to learn richer semantic representations for rare tokens, and likely mitigate the severity and impact of under-trained tokens. although this choice has been studied in older models @xcite , we are not aware of systematic ablations in recent llms. our findings highlight a range of tokenizer issues, the severity of which varies across models. by analyzing tokenizers and model embeddings, we can identify under-trained tokens and improve the efficiency and security of llms.