How Much Semantic Information is Available in Large Language Model Tokens? (2025)

Abstract: Large language models segment many words into multiple tokens, and companies that make those models claim that meaningful subword tokens are essential. To investigate whether subword tokens bear meaning, we segmented tens of thousands of words from each of 41 languages according to three generations of GPT tokenizers. We found that words sharing tokens are more semantically similar than expected by chance or expected from length alone, that tokens capture morphological information even when they don’t look like morphemes, and that tokens capture more information than is explained by morphology. In languages that use a script other than the Latin alphabet, GPT-4 tokens are uninformative, but GPT-4o has improved this situation. These results suggest that comparing tokens to morphemes overlooks the wider variety of semantic information available in word form and that standard tokenization methods successfully capture much of that information.

## introduction subword tokens in large language models (llms) are often compared to morphemes (i.e., productive constituents such as pre and school). in a popular tokenization resource (github.com/openai/ tiktoken), openai claims that llms ''will often split encoding into tokens like encod and ing (instead of e.g. enc and oding)'', and in another resource (huggingface.co/transformers), hugging-face is explicit that ''rare words should be decomposed into meaningful subwords''. however, tokenization is typically determined by the frequency of strings of characters, not by semantics (for a review, see @xcite , so tokens don't reliably correspond to morphemes @xcite . con-sider how three generations of gpt tokenizers segment enc+odings, enc+od+ification, and enc+od+ified, contra openai's own example. perhaps llms treat tokens more like the letters of a very large alphabet, memorizing how to spell words token by token. memorization can't account for unfamiliar words, though, and an important function of subword tokenization is to help llms represent rare words @xcite . this problem is compounded in low-resource languages, which, on average, make up less of an llm's training data and contain more multi-token words (e.g., @xcite . accurately representing rare words requires extracting information from word form, so in this study, we investigate how reliably tokens imply word meanings and whether this varies across languages. we build on previous research that compares tokens to morphemes (e.g., @xcite we take a broader view of the semantic information available in word form, which includes etymological relationships and other patterns (e.g., purse and bursary or glow and glisten; for reviews, see @xcite @xcite . ## gpt-4 benchmark performance do llms use the semantic information available in tokens to represent word meanings? openai reports gpt-4's performance on the massive multi-task language understanding benchmark for 27 languages (mmlu; @xcite , so we investigated whether the semantic information available in those languages' gpt-4 tokens predicts mmlu scores. nine of the 27 languages have word-frequency ranks available from fast-text but not wordfreq (i.e., they're not among our original 41 languages), so we gathered the 26,201 most-frequent words in each of those nine languages (according to fasttext word frequency ranks), which is the median number of words among the 50,000 most-frequent words in both databases for the other 41 languages. we followed the same procedure of measuring the cosine similarity of fasttext and word2vec embeddings for 100,000 pairs of words in each of three conditions for each language. to quantify the semantic information available in tokens for each language, we divided the mean cosine similarity of word embeddings for pairs that share tokens by the mean similarity of randomly paired words, according to fasttext and word2vec, as above. we ignored the length-matched condition because when it comes to llm performance, we care only about the amount of information available in tokens, not whether that information is stumbled across by virtue of length. that is, penalizing languages for having long tokens will obscure any impact of token informativity. multi-simlex provides ratings fasttext word2vec β p β p intercept 77.7 <.0001 79.5 <.0001 crawl % 3.7 <.0001 2.8 .0020 distance -1.8 .1423 -2.0 .0390 info. 1.0 .4110 2.3 .0530 info.*cc% -2.5 .0070 -2.2 .0227 info.*dist. -1.5 .3610 0.5 .6328 adj. r 2 .699 .700 for only eight of the 27 languages with mmlu scores. as we report in the online supplement, the effects are consistent with fasttext and word2vec, but the adjusted r 2 is over .99 because the number of parameters in the models described below is almost equal to the number of multi-simlex observations, so the model is clearly overfit. in two linear regression models, for fasttext and word2vec embeddings, we regressed mmlu score on the interaction of semantic information available in gpt-4 tokens with the percentage of the common crawl that a language makes up plus its interaction with orthographic distance from english. we again excluded english because it is an outlier. in both cases, as reported in ## general discussion and conclusion we compared the semantic relatedness of words that share tokens, such as bat+on and bat+he, to words that share length-matched segments, such as bat+on and gl+utton, and to randomly paired words. words that share tokens are more closely related than words in the control conditions; words that share tokens more often share morphemes than words in the control conditions do, even when the tokens are not themselves morphemes; and when words that share tokens don't share any morphemes, they are still more closely related than words in the control conditions. these findings suggest that standard frequency-based tokenization (i.e., byte-pair encoding) successfully picks out meaningful constituents, that those tokens don't need to look like morphemes in order to identify morphological patterns, and that frequency-based tokenization recovers other sorts of patterns, such as etymological relationships (e.g., purse and bursary). of course, doing better than chance in no way entails optimal segmentation, and the shortcomings of gpt tokens are most obvious in languages that are orthographically dissimilar to english. gpt-4o mitigates this problem, segmenting words that use non-latin scripts into more meaningful tokens than gpt-4 does, yet gpt-4o also amplifies the disadvantage for lower resource languages (i.e., those which make up less of the common crawl). we've provided some evidence that llms use the information available in tokens to compensate for the challenge of processing low-resource languages: gpt-4 performs better on lower resource languages that have more meaningful tokens, but the corollary is that performance drops on lower resource languages with uninformative tokens. it remains to be seen whether this pattern of effects holds in gpt-4o and other llms and when controlling for potential confounds. reliable representations of word meanings matter because llms are black boxes, so content moderation depends on natural language. for example, when the text-to-image model dall-e generates images, openai appends user-generated prompts with system prompts such as ''do not create any imagery that would be offensive'' and ''do not discuss copyright policies'' (github.com/spdustin). engineers exert some control over llm output through the curation of training data and through reinforcement learning from human feedback (e.g., @xcite , but the most direct method of content moderation involves talking to llms and assuming they interpret words like people do. @xcite argue that llms can learn human-like meanings because ''words and concepts must be used jointly together in a coherent way that mimics how humans would.'' patterns in words do convey valuable semantic information, but recall our first, simplest finding: that most words in every language other than english comprise multiple tokens. even english, in a larger sample of the 57,531 words which occur in multiple films and tv shows in the subtlex-us database @xcite , follows this trend, with two thirds of those words being segmented into multiple tokens by gpt-2, gpt-4, and gpt-4o. the outsize proportion of multi-token words belies common descriptions of llms as learning patterns in words by predicting the next word in a sequence (e.g., @xcite . llms clearly do represent the meanings of many multi-token words (e.g., @xcite , but again, most words are rare, and interpreting rare words reliably and in a way that aligns with human interpretations requires inference based on word form. we've found that in many languages, subword tokens occur in semantically coherent sets of words, creating opportunities for human-like inference, yet token utility decreases when it's needed most, in lower resource languages. if llms continue to carve those words far from their joints, disparities in performance will persist.