Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals (2024)

Abstract: Interpretability research aims to bridge the gap between the empirical success and our scientific understanding of the inner workings of large language models (LLMs). However, most existing research in this area focused on analyzing a single mechanism, such as how models copy or recall factual knowledge. In this work, we propose the formulation of competition of mechanisms, which instead of individual mechanisms focuses on the interplay of multiple mechanisms, and traces how one of them becomes dominant in the final prediction. We uncover how and where the competition of mechanisms happens within LLMs using two interpretability methods, logit inspection and attention modification. Our findings show traces of the mechanisms and their competition across various model components, and reveal attention positions that effectively control the strength of certain mechanisms.

## introduction recent advancements in large language models (llms) have brought unprecedented performance improvements to nlp @xcite @xcite . however, the black-box nature of these models obfuscates our scientific understanding of how these models achieve certain capabilities, and how can we trace the problem when they fail at other tasks. this has brought an increasing focus on interpretability research to help us understand the inner workings of llms. existing interpretability research has been largely focused on discovering the existence of single mechanisms, such as the copy mechanism in induction heads of llms @xcite , and factual knowledge recall in the mlp layers @xcite @xcite . however, different from discovering what mechanisms exist in llms, we propose a more fundamental question: how do different mechanisms interact in the decision-making of llms? we show a motivating example in figure 1 , where the model fails to recognize the correct mechanism when it needs to judge between two possible mechanisms: whether to recall the factual knowledge on who developed the iphone (i.e., mechanism 1) or to follow its counterfactual redefinition in the new given context (i.e., mechanism 2). we propose a novel formulation of competition of mechanisms, which focuses on tracing each mechanism in the model and understanding how one of them becomes dominant in the final prediction by winning the "competition". specifically, we build our work on two single mechanisms that are well-studied separately in literature: (1) the factual knowledge recall mechanism, which can be located in the mlp layers @xcite @xcite ; and (2) the in-context adaptation to a counterfactual statement, which is enabled by the copy mechanism conducted by induction heads of attention layers @xcite . based on the latest tools to inspect each of these two mechanisms @xcite @xcite , we then unfold how and where the competition of the two mechanisms happen, and how it leads to the overall success or failure of llms. technically, we deploy two main methods: logit inspection @xcite by projecting the outputs of each model component by an unembedding matrix, and attention modification @xcite . using these methods, we assess the contributions of various model components, both from a macroscopic view (e.g., each layer) and a microscopic view (e.g., attention heads), and identify critical positions and attention heads involved in the competition of the two mechanisms. moreover, we locate a few localized positions of some attention head matrices that can significantly control the strength of the factual mechanism. we summarize our main findings as follows: ## related work on interpretability as deep learning approaches show increasingly impressive performance in nlp, their black-box nature has hindered the scientific understanding of these models and their effective future improvements. to this end, interpretability research has been a rising research direction to understand the internal workings of these models. ## problem setup following the setup of many existing interpretability studies @xcite @xcite , inter alia), we look into the next token prediction behavior of autoregressive llms in their inference mode, namely next, we design the task to incorporate the competition of mechanisms as in figure 1 . specifically, for each factual statement f := (t f 1 , . . . , t f k ) consisting of k tokens (e.g., "iphone was developed by apple."), we compose a corresponding counterfactual statement c := (t c 1 , . . . , t c k ′ ) (e.g., "iphone was developed by google."). then, we compose a prompt connecting the two statements as "redefine: c. f 1:k-1 .", such as "redefine: iphone was developed by google. iphone was developed by ___". the two mechanisms can be traced by inspecting the rise and fall of the factual token t f k and the counterfactual token t c k ′ . for the simplicity of notation, we take the tokens out of the context of their exact position and denote them as t fact and t cofa , respectively, in the rest of the paper. an llm produces the initial residual stream x 0 i by applying an embedding matrix w @xmath0, where |v | is the size of the vocabulary. then, it modifies the residual stream by a sequence of l layers, each consisting of an attention block a l and mlp m l . finally, after the last layer, it projects the internal state of the residual stream back to the vocabulary space with an unembedding matrix w @xmath1. formally, the update of the residual stream at the l th layer is: to understand which token the residual stream x l favors, we follow the common practice in previous work @xcite @xcite to project it to the vocabulary space using the aforementioned unembedding matrix w u which maps the latent embeddings to actual tokens in the vocabulary, enabling us to obtain the logits of the factual t fact and counterfactual token t cofa . known as the logit lens @xcite , this method is broadly adopted due to its consistent success in yielding interpretable results, demonstrating its effectiveness through broad empirical usage. however, it is important to note that it can occasionally fail to reflect the actual importance of vocabulary items, especially in the early layers of the network @xcite . ## results and findings in this section, we trace the competition of the mechanisms within the llm via the two methods introduced in section 4, i.e., inspecting the residual stream and intervening on the attention. we provide mechanistic analyses on five research questions in the following subsections: ## conclusion in this work, we have proposed the formulation of the competition of mechanisms as a powerful interpretation when llms need to handle multiple mechanisms, only one of which leads to the correct answer. we deployed two mechanistic interpretability tools, logit inspection and attention modification, and identified critical positions and model components involved in competing for the mechanisms. finally, we discovered a few localized positions in the attention map, which largely control the strength of the factual mechanism. our study sheds light on future work on interpretability research for llms.