Racing Thoughts: Explaining Contextualization Errors in Large Language Models (2025)

Abstract: The profound success of transformer-based language models can largely be attributed to their ability to integrate relevant contextual information from an input sequence in order to generate a response or complete a task. However, we know very little about the algorithms that a model employs to implement this capability, nor do we understand their failure modes. For example, given the prompt “John is going fishing, so he walks over to the bank. Can he make an ATM transaction?”, a model may incorrectly respond “Yes” if it has not properly contextualized “bank” as a geographical feature, rather than a financial institution. We propose the LLM Race Conditions Hypothesis as an explanation of contextualization errors of this form. This hypothesis identifies dependencies between tokens (e.g., “bank” must be properly contextualized before the final token, "?", integrates information from “bank”), and claims that contextualization errors are a result of violating these dependencies. Using a variety of techniques from mechanistic interpretability, we provide correlational and causal evidence in support of the hypothesis and suggest inference-time interventions to address it.

## introduction large language models (llms) have demonstrated a remarkable capacity for accomplishing a wide variety of language generation and classification tasks @xcite @xcite . many of these capabilities crucially rely on contextualization-integrating relevant contextual information from the input sequence at inference time. indeed, llms' profound success at contextualization has given rise to a novel method of adapting models to new tasks: in-context learning @xcite . although contextualization is fundamental to the success of large language models, we understand fairly little about the mechanisms underlying successful contextualization. at an implementation level, we understand that attention heads allow information to be read from and written to token representations across different positions @xcite . however, we are currently in the dark about the algorithm-level factors that result in the success or failure of contextualization. consider the contextualization error presented in fig. 1 @xcite . this error was generated by a state of the art llm, gemini foot_0 @xcite . crucially, it betrays that the model has not properly contextualized the representation of bank. the model first interprets the word to refer to a geographical feature, and then interprets that same instance of bank in the same context as a financial institution. in this work, we propose an explanation for contextualization errors in llms, denoted the llm race conditions hypothesis. a race condition is a circumstance where a system deploys two or more subroutines in parallel, but relies on these operations executing in a particular order @xcite . we identify race conditions occurring throughout the layers of llms (right side of fig. 1 ). these race conditions occur whenever one token (e.g., the final question-mark token of can he make an atm transaction at this bank?) must read from a contextualized representation of another (e.g., the earlier instance of the token bank). this establishes a dependency where the contextualization of bank must complete at an earlier layer than the contextualization of the end-of-sentence delimiter. outof-order contextualization can result in incorrect or inconsistent llm generations. we verify the llm race conditions hypothesis using state-of-the-art techniques in mechanistic interpretability, offering qualitative and quantitative evidence as well as several inference-time interventions to ameliorate the problem. we conclude by arguing that race conditions are endemic to feedforward language models and point toward potential solutions. 2 in summary, our key contributions are: ## methods in this section, we describe the setting in which we study contextualization errors, datasets we construct for this purpose, the tools used for our analyses, and the models under investigation. task. to investigate contextualization errors, we define an llm question-answering task that requires integrating a particular cue that is found in context. specifically, instances of our task consist of the following components: for example: the game's success led sega to develop an extensive media franchise. i am holding a fishing rod. i see a bank . is it a geographical feature? for each subject entity and cue, we specify two questions -the correct answer to one of which is 2 code and data are available here. for each subject entity and cue, we systematically vary the number of distractors included in the prompt, as well as the position of the cue amongst these distractors. to ensure that our analyses are robust to the particular distractors that we randomly select, we include three independent random samples of distractors for each subject entity and cue. importantly, these datasets are not meant to be reflective of real-world contexts, but are instead constructed to stress-test models. by exacerbating problems that are present in normal llm computation, we seek to gain a deeper understanding of contextualization within llms. ## behavioral failure modes in this section we document a simple and robust failure mode of contextualization in llms. inspired by recent work investigating context integration in multi-document question answering @xcite and long-context retrieval (kam-0 1 2 3 4 5 num distractors 0.0 0.2 0.4 0.6 0.8 1.0 accuracy behavioral results: gemma-2 dataset polysemous factual gender figure 2: model accuracy vs. number of distractors. we find that injecting distractor text into the prompt causes sharp performance degradations across all datasets. ## the llm race conditions hypothesis to explain the behavioral results presented in § 3, we propose the llm race conditions hypothesis, illustrated pictorially in fig. 1 . ## related work this work takes inspiration from the burgeoning field of mechanistic interpretability, which attempts to understand the circuits and algorithms that a neural network implements in order to accomplish a particular task @xcite . our work contributes to a trend of using patching to study algorithms implemented in production-level llms @xcite @xcite @xcite @xcite . several recent works have attempted to understand in-context learning (icl), which might be viewed as an extreme case of context-dependent generation (e.g., @xcite @xcite @xcite . our work complements this line of research by studying contextualization in more natural settings. additionally, recent work has analyzed knowledge conflictscontexts in which information encoded within the weights of an llm conflict with information provided in context @xcite . the factual dataset is comprised of knowledge conflicts regarding countries and their capitals. thus, we might hypothesize that race conditions may be present in other contexts that contain knowledge conflicts. finally, this work contributes to a body of work that identifies distinct processing stages throughout the layers of language models @xcite . recent mechanistic work has localized refusal mechanisms to the early layers of llms @xcite and suggested the existence of universal stages of processing @xcite . earlier work in interpretability has studied the impact of training objectives on processing stages @xcite . the present study demonstrates distinct stages of contextualization over layers and suggests interventions that are informed by these stages. ## conclusion in this work, we introduced a hypothesis to explain contextualization errors in llms: the llm race conditions hypothesis. after establishing a setting for studying contextualization errors systematically, we tested two predictions of the llm race conditions hypothesis: 1) that there exists a critical window when the question tokens are contextualized, and 2) that model failures are caused by improper contextualization of the subject entity during this window. this is a natural consequence of purely feedforward architectures, as early layers simply do not have access to later layer representations. this work provides an example of using techniques from mechanistic interpretability to understand a problem found in state-of-the-art llms. by exposing and understanding algorithmic shortcomings of these models, one can suggest plausible solutions for addressing them. our work-in-progress aims to explore two potential solutions. first, new models might include recurrent connections, allowing later layer representations of earlier tokens to influence earlier layer representations of later tokens. this would enable deep, well-contextualized representations to influence the entire residual stream of all ensuing tokens. second, this problem may be ameliorated in existing pretrained models by adopting more advanced versions of the inference-time interventions presented in § 4.2.