Token Erasure as a Footprint of Implicit Vocabulary Items in LLM s (2024)

Abstract: LLMs process text as sequences of tokens that roughly correspond to words, where less common words are represented by multiple tokens. However, individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise. For example, Llama-2-7b’s tokenizer splits the word “patrolling” into two tokens, “pat” and “rolling”, neither of which correspond to semantically meaningful units like “patrol” or "-ing.” Similarly, the overall meanings of named entities like “Neil Young” and multi-word expressions like “break a leg” cannot be directly inferred from their constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups of tokens into useful higher-level representations? In this work, we find that last token representations of named entities and multi-token words exhibit a pronounced “erasure” effect, where information about previous and current tokens is rapidly forgotten in early layers. Using this observation, we propose a method to “read out” the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers, and present results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is the first attempt to probe the implicit vocabulary of an LLM.

## introduction despite their widespread use, the specific mechanisms by which llms are able to "understand" and generate coherent text are not well understood. one mystery is the process by which groups of subword tokens are converted into meaningful representations, a process described by @xcite as detokenization. current language models process text as a series of tokens drawn from a set token vocabulary: one token can correspond to a single word (_fish), figure 1: we observe "erasure" of token-level information in later layers of llms for multi-token words and entities @xcite . we hypothesize that this is a result of a process that converts token embeddings into useful lexical representations, and introduce a new method for enumerating these lexical items (bottom). or to a piece of a larger word (mon in "salmon"). the vocabulary of tokens available to a model is typically determined before training with byte-pair encoding @xcite , which is based on a specific dataset and can lead to unintuitive results. for example, llama-2-7b's @xcite tokenizer breaks the word "northeastern" into the tokens @xcite , none of which correspond to semantically meaningful units like "north" or "east." capitalization also creates unexpected issues: for example, the word "hawaii" is split into two tokens if the first letter is capitalized @xcite , but four if the first letter is lowercase @xcite . in spite of these challenges, large models are apparently able to "understand" such idiosyncratic tokenizations of multi-token words with few observable effects on downstream performance @xcite , unless these weaknesses are directly targeted @xcite . how is this possible? we hypothesize that during pretraining, llms develop an implicit vocabulary that maps from groups of arbitrary tokens to semantically meaningful units. these lexical units may be multi-token words ("northeastern"), named entities ("neil young"), or idiomatic multi-word expressions ("break a leg") and can be understood as "item[s] that function as single unit[s] of meaning" in a model's vocabulary @xcite . lexical items are also non-compositional: just as the meaning of "break a leg" cannot be predicted from the individual meanings of "break" and "leg," the meaning of "patrolling" cannot be predicted from its constituent tokens pat and rolling. this arbitrariness necessitates some kind of storage system, implicit or otherwise @xcite . how exactly do llms deal with these cases mechanistically? in this paper, we begin to answer this question by investigating token-level information stored in llm representations. we interpret this erasure effect as a "footprint" of a mechanism in early layers that orchestrates the formation of meaningful lexical items. ## background previous work has shown that knowledge about a multi-token entity is often stored in the last token of that entity. for example, @xcite found that factual information about a subject like "the space needle" would be concentrated in the representation for le. @xcite find evidence for a subject enrichment stage during factual recall, where information about an entity is collected at its last token in early layers, which is also seen in other work on factual recall using the same dataset @xcite , and corroborated by research on @xmath0. this phenomenon may be due to the autoregressive nature of decoder transformer models: models cannot enrich "space" with information about seattle until after "needle" is seen, as "space" could refer to a number of unrelated concepts ("space jam," "space station"). 2 other work in interpretability has also started to uncover evidence of models encoding lexical items. @xcite observe neurons in early layers that fire on the last tokens of multitoken words, names of famous people, generic nouns, compound words, and latex commands. they also find late-layer neurons that seem to be relevant to retokenization, i.e., conversion from internal representations back into tokens. for example, a retokenization neuron might fire on _st and promote rag in order to facilitate the output of the word "straggler." gurnee et al. (2023) also find examples of polysemantic neurons in pythia models @xcite that activate for a number of multi-token constructions like "apple developer," "bloom.ington," and "research.gate." ## building a vocabulary after examination of probe behavior for multitoken words and entities, we hypothesize that this "erasure" effect is a result of the implicit formation of lexical representations in early layers. to characterize this phenomenon, we propose an erasure score ψ to identify token sequences that follow the pattern observed in section 3. we then introduce an approach to "reading out" a list of implicit vocabulary entries for a given model using this score. accuracy on all other tokens shown on the left. we see an erasing effect for multi-token words, similar to the effect seen for counterfact subjects in figure 2 . ## conclusion in this work, we present preliminary evidence for the existence of an implicit vocabulary that allows models to convert from byte-pair encoded tokens to useful lexical items. we posit that the "erasure" effect we observe for llama-2-7b and llama-3-8b is a result of model processes that deal with multi-token expressions, and use this insight to propose a new method for "reading out" an llm's implicit vocabulary. this is a first step towards understanding the formation of lexical representations in llms, and may serve as a useful tool for elucidation of words that a given model "knows."