Tokenization Falling Short: On Subword Robustness in Large Language Models (2024)

Abstract: Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length variations, and largely oblivious to the internal structure of tokens—issues we term *the curse of tokenization*. In this study, we delve into these drawbacks and demonstrate that large language models (LLMs) remain susceptible to these problems. This study systematically investigates these challenges and their impact on LLMs through three critical research questions: (1) complex problem solving, (2) token structure probing, and (3) resilience to typographical variation. Our findings reveal that scaling model parameters can mitigate the issue of tokenization; however, LLMs still suffer from biases induced by typos and other text format variations. Our experiments show that subword regularization such as BPE-dropout can mitigate this issue. We release our evaluation code and data at https://github.com/FloatAI/TKEval.

## introduction tokenization is a fundamental step in the preprocessing pipeline of large language models (llms) @xcite @xcite @xcite , converting raw text into a sequence of subword units derived from a predefined vocabulary @xcite . this process, while effective in many scenarios, presents significant challenges that can hinder the performance and robustness of llms. these challenges include sensitivity to typographical errors @xcite , length variations @xcite , and a lack of awareness of the internal structure of tokens @xcite )-collectively termed the curse of tokenization. figure 1 : compositional challenges in token embeddings. (a) "assignment" decomposed into "assign" and "ment" shows a cosine similarity of 0.21 and an angle of 78.16°. (b) "import" decomposed into "im" and "port" shows a cosine similarity of 0.13 and an angle of 82.47°. these results indicate that existing llms do not accurately capture surface form composition. typographical errors, such as minor misspellings or misplaced characters, can drastically affect the tokenization process. unlike humans, who can easily overlook these errors and understand the intended meaning, llms can misinterpret or fail to recognize these variations, leading to degraded performance @xcite . this typo-sensitivity reveals a crucial gap in current tokenization methods, which do not sufficiently mimic human reading capabilities. another critical issue is the length unawareness of current tokenization approaches. llms often struggle to accurately represent the organizational structure of text, being insensitive to the number of characters or words @xcite . this insensitivity affects their ability to understand and process text effectively, particularly in tasks requiring a nuanced understanding of text length and compositional structure. furthermore, existing tokenization methods are often blind to the internal structure of tokens. the decoupled embedding space and lookup table approach fail to account for the hierarchical composition of language, spanning characters, subwords, and words, as depicted in figure 1 . this lack of integration across different levels of token composition limits the model's ability to fully grasp semantic relationships and differences. case insensitivity in certain languages adds another layer of complexity, where variations in capitalization can lead to different token representations, further complicating the model's understanding and processing of text. to address these challenges, we conducted a comprehensive study examining the limitations of current tokenization methods and their impact on llm performance. our study is guided by three critical research questions: a. complex problem solving ( §3). as a pilot experiment, we firstly investigate the performance of llms on complex problems that are sensitive to tokenization, involving anagram task and complex mathematical language understanding. b. token structure probing ( §4). we study the token structural tasks such as case manipulation, length counting, and length-sensitive tasks to probe the token structural understanding of llms. c. typographical variation ( §5). we designed a robust set of evaluation benchmarks on top of various datasets such as mmlu, truthfulqa, gsm8k, and humaneval, covering diverse tasks and linguistic phenomena. these benchmarks allow us to systematically test and analyze the llms' resilience to tokenization. our findings highlight that while scaling model parameters can enhance the robustness to tokenization, llms still suffer from biases introduced by typographical errors and text format variations. we demonstrate the persistent nature of these tokenization challenges. contribution to conclude, our main contributions are as follows: ## complex problem solving complex problem-solving tasks are critical benchmarks for evaluating the complex reasoning and comprehension capabilities of llms. we explore the llm's ability to perform intricate operations on tokenized inputs, as the tokenization process is fundamental to determine how the raw text is segmented and processed, directly impacting the model's interpretation and prediction. anagram solving and mathematical language comprehension were selected to elucidate the relationship between tokenization quality and the model's performance on complex problem-solving. anagram tasks require models to decode and rearrange jumbled letters into coherent words, emphasizing the importance of precise token boundaries and recognition accuracy. on the other hand, mathematical language comprehension, particularly expressed with l a t e x-formatted expressions, demands an exact interpretation of specialized symbols and structured notation, challenging the tokenization process's robustness. ## token structure probe tokenization is a key preprocessing step in llms, yet it introduces several significant challenges, which we defined as the curse of tokenization. these challenges include length unawareness, case insensitivity, and a lack of awareness of the internal structure of tokens. tokenization transforms text into sequences of token identifiers, often obscuring the surface form and internal structure of the original text. this conversion can lead to deficiencies in the model's ability to understand and process textual data accurately. the curse of tokenization manifests in several ways, which refers to the inherent challenges: a) length unawareness: models struggle to recognize the organizational structure of text, such as the number of characters or words. b) case insensitivity: variations in capitalization can lead to different token identifiers and representations, complicating the model's processing of text. c) blindness to internal structure: the decoupled embedding space and lookup table approach used in llms fail to preserve the hierarchical and relational structure within tokens, obscuring the surface form and internal relationships between characters and subwords. to address these challenges, we construct a set of probing tasks to evaluate the model's understanding of token structure. these tasks are divided into intra-token ( §4.1) and inter-token probes ( §4.2). ## typographical variation to evaluate the robustness of llms to typographical variations, we constructed tasks that introduce character-level and token-level typographical errors into the input text. these tasks are designed to test the models' ability to maintain semantic understanding despite the presence of such errors. the datasets include mmlu @xcite , truthfulqa @xcite , gsm8k @xcite , and humaneval @xcite , ensuring diverse coverage. ## conclusion in this study, we investigated the challenge of the curse of tokenization, comprehensively evaluating mainstream llms across thirteen tasks sensitive to conventional subword tokenization. our findings reveal that while larger models and increased shot counts can partially mitigate these issues, llms still struggle with understanding internal structures and token compositions. moderate bpe-dropout can alleviate some of these challenges, whereas larger drop rates lead to performance degradation. we encourage the research community to develop more flexible approaches to further address these limitations and enhance model robustness.