CUTE : Measuring LLM s’ Understanding of Their Tokens (2024)

Abstract: Large Language Models (LLMs) show remarkable performance on a wide variety of tasks. Most LLMs split text into multi-character tokens and process them as atomic units without direct access to individual characters. This raises the question: To what extent can LLMs learn orthographic information? To answer this, we propose a new benchmark, CUTE, which features a collection of tasks designed to test the orthographic knowledge of LLMs. We evaluate popular LLMs on CUTE, finding that most of them seem to know the spelling of their tokens, yet fail to use this information effectively to manipulate text, calling into question how much of this knowledge is generalizable.

## introduction large language models (llms) attract a lot of interest due to their strong performance on many nlp tasks. they have demonstrated a level of fluency rivaling humans. however, it is often overlooked that llms lack direct access to the characters composing their tokens. they can only infer knowledge about the characters from the context during pretraining or instruction tuning. while there are models that use characters as input units, none of them have been instruction-tuned (to our knowledge). our work examines how well llms understand the composition of their tokens. this knowledge enables llms to better generalize to new languages and to perform well on a variety of tasks involving character-level understanding. tasks such as word puzzles, poetry generation (e.g. alliterations), or parsing ciphers all require very explicit use of characters to achieve. more popular tasks such as code completion, morphological inflection, or spelling correction also require character-level information to a lesser extent, though these tasks also require semantic knowledge, which we wish to ablate. we introduce character-level understanding of tokens evaluation (cute) @xcite , a benchmark consisting of several tasks designed to be easy for humans to complete, given our ability to process characters individually. we evaluate several llms ranging from 7b to 132b parameters in size on cute to answer the following questions: we address these questions with a set of tasks primarily on the character level, and additionally on the word level to see the difference in performance on the two granularities, thereby separating the understanding of the task from the understanding of what makes up a token. other benchmarks testing orthographic knowledge often focus on morphology, such as the sig-morphon inflection tasks (e.g. goldman et al. ( 2023 )). inflection is fairly regular for most languages, and could be memorized by a language model. given that many developers of llms do not disclose the sources of their pretraining data, it is possible that llms memorize these inflection patterns. therefore, from this we cannot conclude that llms can inflect a new word given a new set of rules, and furthermore we cannot conclude that llms can apply an arbitrary manipulation to a sequence. most llms are trained with performance on english in mind, and english being less morphological in nature makes inflection a nonideal measure for a model's understanding of the composition of tokens. the most similar benchmark to the one we propose is lmentry (efrat et al., 2023). the purpose of lmentry is similar in that the goal is to test models on tasks that are trivial to humans. some of the tasks included test orthography and are similar to our tasks, for example, they ask the model to write a word containing a letter, or ask to write the first or last letter of a given word. our benchmark is distinguished from lmentry as most of our tasks explicitly require knowledge of every character in a word. testing whether a model knows the first letter of a word is not sufficient for concluding that it understands orthography, as there may be more pressure to learn about the first letter of a word from pretraining and/or instruction tuning (e.g. via alliterations or acronyms). as such, the task to write a word containing a letter could be more trivially solved by writing a word that starts with said letter. there is an extensive body of research on character-level models, where each character forms a token (lee et al., 2017; xue et al., 2022; tay et al., 2022, inter alia). several works compared these models to subword models (libovický et al., 2022; edman et al., 2022, 2024, inter alia), but they evaluated on tasks requiring additional training. we assume that character-based models would perform spell out the word: there write the word that is spelled out (no spaces): t h e r e input is there a 'c' in 'there'? is there a 'the' in 'the sky is blue'? closer in levenshtein distance to 'happy': glad or apply? more semantically related to 'happy': glad or apply? add 'b' after every 'e' in 'there' add 'is' after every 'the' in 'the sky is blue' delete every 'e' in 'there' delete every 'the' in 'the sky is blue' replace every 'e' with 'a' in 'there replace every 'the' with 'is' in 'the sky is blue' swap 't' and 'r' in 'there' swap 'the' and 'is' in 'the sky is blue' contains char inverse spelling spelling task contains word orthographic semantic char insertion word insertion char deletion word deletion char substitution word substitution char swapping word swapping sim. composition manipulation no there t h e r e output thebreb the is sky is blue yes apply glad thr sky is blue thara is sky is blue rhete is sky the blue well on our benchmark, but we cannot test it since none of these models have been instruction-tuned. ## benchmark we split our tasks into 3 categories: understanding composition, understanding orthographic similarity, and ability to manipulate sequences. figure 1 shows an example for each task. foot_1 data gathering and processing details can be found in appendix c. our tasks are synthetically generated from existing corpora. there are non-synthetically generated datasets which partially test our research questions, but these datasets have external factors (e.g. domain and/or language in a translation dataset) that would likely obscure our findings. some of these datasets also might have been leaked into the llm's pretraining data and been memorized, resulting in an unrealistically good performance. ## experimental setup models we use the models shown in prompts we use a template inspired by bsharat et al. (2023)'s few-shot template to prompt our models with 4 in-context examples. further details and a sample prompt can be found in appendix b. russian experiments we additionally create cute-rus, a russian version of our benchmark. we discuss this fully in appendix d. the trends largely follow what we see for english. ## results figure 2 shows the results for each model. the random baseline is 50% for the contains and similarity tasks, and 0% for all other tasks. ## conclusion while current llms with bpe vocabularies lack direct access to a token's characters, they perform well on some tasks requiring this information, but perform poorly on others. the models seem to understand the composition of their tokens in direct probing, but mostly fail to understand the concept of orthographic similarity. their performance on text manipulation tasks at the character level lags far behind their performance at the word level. llm developers currently apply no methods which specifically address these issues (to our knowledge), and so we recommend more research to better master orthography. character-level models are a promising direction. with instruction tuning, they might provide a solution to many of the shortcomings exposed by our cute benchmark.