Internal states before wait modulate reasoning patterns (2025)

Abstract: Prior work has shown that a significant driver of performance in reasoning models is their ability to reason and self-correct. A distinctive marker in these reasoning traces is the token \textit{wait} , which often signals reasoning behavior such as backtracking. Despite being such a complex behavior, little is understood of exactly why models do or do not decide to reason in this particular manner, which limits our understanding of what makes a reasoning model so effective. In this work, we address the question whether model’s latents preceding \textit{wait} tokens contain relevant information for modulating the subsequent reasoning process. We train crosscoders at multiple layers of \texttt{DeepSeek-R1-Distill-Llama-8B} and its base version, and introduce a latent attribution technique in the crosscoder setting. We locate a small set of features relevant for promoting/suppressing wait tokens’ probabilities. Finally, through a targeted series of experiments analyzing max-activating examples and causal interventions, we show that many of our identified features indeed are relevant for the reasoning process and give rise to different types of reasoning patterns such as restarting from the beginning, recalling prior knowledge, expressing uncertainty, and double-checking.

## introduction a growing class of language models known as reasoning models, for instance, deepseek-r1 @xcite , openai o1 series @xcite , and others @xcite @xcite , produce detailed internal reasoning chains before generating responses. while they showcase sophisticated reasoning capabilities, our understanding of their internal reasoning mechanisms remains limited. a recent work by @xcite categorized deepseek r1's reasoning process into behavioral patterns such as example testing, uncertainty estimation, and backtracking to show how models solve reasoning tasks. this leads us to ask the following question: when and how do these reasoning patterns form? this is a broad question that encompasses both the training dynamics, such as when and how reasoning circuits form, and the emergence of reasoning patterns and behaviors during inference. in this work, we focus on the latter, specifically within the distilled r1 model deepseek-r1-distill-llama-8b. 1 from observation, in this model the token wait is strongly associated with the self-reflection of the model @xcite , which could relate to reasoning patterns such as backtracking, deduction, and uncertainty estimation @xcite . hence, we hypothesize that the features that modulate this token prediction can be useful in understanding the shifts in the reasoning patterns of the reasoning model. contributions. motivated by this observation, we train sparse crosscoders @xcite a recent mechanistic interpretability technique that allows to learn features within a paired base and finetuned model in an unsupervised way and can be used to perform model diffing. we leverage our crosscoders to discover latent features that modulate reasoning behaviors. to reduce the search space, our case study focuses on features promoting and suppressing wait tokens that we hypothesize to play a key role in r1's reasoning. to this end, we extend attribution patching @xcite to the crosscoder setting. our crosscoder latent attribution method allows us to efficiently score all of our 32,768 features regarding their contribution to the logits of wait tokens. as a result, we obtain two short lists of relevant features, the 50 top features with the largest contributions to wait logits and the 50 bottom ones, that we can effectively investigate. by utilizing the crosscoders classification of the features in to base, shared, and reasoning-finetuned we found that surprisingly many reasoning related features fall inside of the bottom bucket, suggesting that both amplifying as well as suppressing wait are of key importance to reasoning. by studying max-activating examples we observe a similar pattern. while top features mostly occur in the frequently reported backtracking and self-verification behaviors, we again notice that bottom features stand out in terms of the diversity of the reasoning behaviors within which they occur (e.g., starting again from an earlier step, expressing uncertainty, wrapping up, etc.). finally, we perform a causal analysis via activation steering, in which we observe effects matching our interpretations based on the max-activating examples for many features, in particular the ones mentioned in parenthesis before. our work showcases how to reduce the combinatorial complexities of crosscoder feature-based interpretability with the help of our novel crosscoder latent feature attribution technique. we thereby open the door for future researchers to explore such findings in other chat and reasoning models. ## methods to discover features, we create reasoning data instances ( section 2.2) and train sparse crosscoders ( section 2.1). we then apply latent attribution patching ( section 2.3) to filter features that strongly modulate wait tokens. finally, we steer these features ( section 2.4) to evaluate their impact on the model's reasoning behavior, helping us determine how actionable they are. ## related work to understand reasoning models, @xcite looks into the chain-of-thought texts and categorizes the trace into various reasoning behavior patterns such as deduction, adding knowledge, and example testing. they further find steering vectors to increase or decrease the appearance of a particular reasoning behavior. @xcite investigates reasoning features, learned by crosscoders, via qualitative analysis of max activating examples. an additional perspective on token-level and feature-level analysis of reasoning features is a concurrent work by @xcite that performs a case study on the self-verification process of a task-specific reasoning model. ## conclusion our study sheds light on how the model's latents preceding wait tokens signal behaviors like backtracking. we introduce latent attribution patching for identifying and testing which internal features influence these tokens in crosscoder models. we show that these features not only predict wait tokens but also shape how the model reasons.