Paper: Circuit Component Reuse Across Tasks in Transformer Language Models (ICLR-2024)

Reviewer comment (Writing):
My main concern with the paper is that it is not easy to read and follow, especially for people outside the MI field. There is a heavy use of jargon and a liberal use of terms. The authors should give a thorough pass of the work and try to "cleanse it" to make it more accessible to a wider audience.

Author response:
We would like to thank the reviewer for their comments. We understand that some parts are hard to follow and we would like this work to be accessible to as wide of an audience as possible. It’s true that this type of work makes use of a lot of jargon. We have focused so far primarily on providing additional controls to strengthen the core claims of the paper. We would like to add a glossary in the appendix and properly explain some of the language in the main paper. We definitely appreciate the reviewer’s points on clarity, we will need to take some time to address them all.

Reviewer comment (Writing):
The term `circuit`, although present in large part of the MI literature, should be refreshed and defined for clarity.

Author response:
We would like to thank the reviewer for their comments. We understand that some parts are hard to follow and we would like this work to be accessible to as wide of an audience as possible. It’s true that this type of work makes use of a lot of jargon. We have focused so far primarily on providing additional controls to strengthen the core claims of the paper. We would like to add a glossary in the appendix and properly explain some of the language in the main paper. We definitely appreciate the reviewer’s points on clarity, we will need to take some time to address them all.

Reviewer comment (Writing):
The term `residual stream` / `write in the residual stream` should be explicitly defined.

Author response:
We would like to thank the reviewer for their comments. We understand that some parts are hard to follow and we would like this work to be accessible to as wide of an audience as possible. It’s true that this type of work makes use of a lot of jargon. We have focused so far primarily on providing additional controls to strengthen the core claims of the paper. We would like to add a glossary in the appendix and properly explain some of the language in the main paper. We definitely appreciate the reviewer’s points on clarity, we will need to take some time to address them all.

Reviewer comment (Writing):
The notation `[end]` is not defined; it is presumed to refer to the last token in the sentence, but this is unclear.

Author response:
We would like to thank the reviewer for their comments. We understand that some parts are hard to follow and we would like this work to be accessible to as wide of an audience as possible. It’s true that this type of work makes use of a lot of jargon. We have focused so far primarily on providing additional controls to strengthen the core claims of the paper. We would like to add a glossary in the appendix and properly explain some of the language in the main paper. We definitely appreciate the reviewer’s points on clarity, we will need to take some time to address them all.

Reviewer comment (Writing):
The term `inhibition` is used but not specifically defined.

Author response:
We would like to thank the reviewer for their comments. We understand that some parts are hard to follow and we would like this work to be accessible to as wide of an audience as possible. It’s true that this type of work makes use of a lot of jargon. We have focused so far primarily on providing additional controls to strengthen the core claims of the paper. We would like to add a glossary in the appendix and properly explain some of the language in the main paper. We definitely appreciate the reviewer’s points on clarity, we will need to take some time to address them all.

Reviewer comment (Evaluation):
Do the MLPs play no role in the circuits explained? Are the MLPs even considered in the analysis?

Author response:
>>Do the MLPs play no role in the circuits explained? Are the MLPs even considered in the analysis?

The short answer is no, they pay a negligible role. We can include the actual numbers in the revision, but we wanted to focus on following the setup from Wang, et al. which excludes them.

Reviewer comment (Experiments):
The paper only investigates reuse of components between two tasks, which are pointed out as similar to each other in the paper. The claims in the paper would be strengthened by investigating component reuse across additional tasks that are less similar to the original tasks studied (IOI & Colored Objects). Perhaps studying the behavior of these components on a larger distribution of text (i.e. open web text or some equivalent) that is not task-specific would provide insight into how generally we can understand the inhibition-mover components.

Author response:
To address this, we performed path patching on two additional different datasets (see the new Appendix H). These are the Greater Than task (Hanna, et al. 2023) and a simple factual recall task in which the model must predict the capital of a country (“The capital of France is”). The first represents a task that we expect will have minimal overlap (based on the Hanna, et al. circuit) and the latter tests a specific hypothesis about the reuse of mover heads, as well as provides one additional differing datapoint (see our general comment). In the first task, the overlap is only 16.7% and 22.7% for the IOI and Colored Objects tasks, respectively. The World Capitals task uses 7 mover heads to perform the final copying step, 4 of which appear in the IOI and Colored Objects circuit. The remainder of the circuit uses very different components giving us 25% overlap. We hope this puts the overlap displayed in the main paper into perspective.

Reviewer comment (Experiments):
While the intervention in section 5 correctly predicts the interaction between the inhibition heads and mover heads will improve performance, this ignores the other half of the circuit - induction heads, duplicate token heads, and content gatherer heads. A more thorough analysis of the interactions between these elements and the inhibition heads/mover heads would strengthen the claim that these components are part of the circuit. For example, is there an intervention on induction heads/previous token heads (or something else) that would reliably produce the attention pattern you artificially induced into the inhibition heads/negative mover heads? This would show better understanding of the entire circuit, and not just the interaction of the last two sub-components.

Author response:
We agree with the reviewer that this would be satisfying, but this does not appear to work. As reported in Wang, et al. the inhibition heads activate in response to detected duplication of a token (passed by induction/duplicate heads). We show that they work even when no duplication is present, the inhibition heads can still be coerced to work. The action of inhibition heads is fascinating and we are directly following up on them in followup work, but they are complicated and it’s outside the scope of this paper. We believe that our current experiments satisfy our original hypothesis about circuit level overlap, even if it raises some new questions about inhibition heads.

Reviewer comment (Experiments):
Although it is briefly touched upon at the end, there is is an important question to be answered more concretely: why don’t inhibition heads appear to work properly in the Colored Objects setting when they work just fine for the IOI task? One hypothesis might be that the induction heads, which were important for influencing attention of inhibition heads in IOI, aren’t passing forward the proper information but it might be improved with more shots. For example: if you provide some more ICL examples do the inhibition heads start to work as the IOI task would expect them to?

Author response:
To your final questions: We are glad you also find this interesting, we are hoping to encourage discussion around these topics with this paper. We did try adding more shots, but accuracy does not improve. Our current hypothesis is that since the content gatherer and inhibition heads are close together in the model, the input signal to the inhibition heads is perturbed by the output of the gatherer heads. This gets a bit outside what we aimed to accomplish in this work, but we plan on addressing this.

Reviewer comment (Evaluation):
**Overlap Measurement**: The paper mentions an overlap in in-circuit attention heads but lacks a detailed explanation of how this overlap is computed. Although "putting an exact number on how similar two circuits are is a very complex task that [the authors] do not attempt to fully answer in this paper," the results in the paper heavily rely on this metric. Thus, authors should (a) provide a more detailed description of the overlap metric; (b) quantify the uncertainty of this metric and the results associated to it; and (c) provide clues to better understand its range. As an illustration, authors wrote that 78% of the IOI circuit is used in the Colored Object task. However, without more information, it is difficult to say how significant this overlap is. Providing control baselines, such as comparing the IOI circuit overlap with a set of random tasks, or computing the overlap regarding another circuit (which could be a random circuit), could help to determine if 78% is high or not. Without such information (uncertainty quantification, control baselines, etc.), the significance of the results is hard to interpret. Note that authors briefly mention such an experiment in Appendix I. This should be expanded and included in the main paper.

Author response:
> As an illustration, authors wrote that 78% of the IOI circuit is used in the Colored Object task. However, without more information, it is difficult to say how significant this overlap is… would it be possible to provide reference baselines, such as comparing the IOI circuit overlap with a set of random tasks?

In the new Appendix H, we run path patching experiments on two additional tasks that are significantly different from each other and the original IOI and Colored Objects tasks. These are the Greater Than task (Hanna, et al. 2023) and a simple factual recall task in which the model must predict the capital of a country (“The capital of France is”). The first represents a task that we expect will have minimal overlap (based on the Hanna, et al. circuit) and the latter tests a specific hypothesis about the reuse of mover heads, as well as provides one additional differing datapoint (see our general comment). In the first task, the overlap is only 16.7% and 22.7% for the IOI and Colored Objects tasks, respectively. The World Capitals task uses 7 mover heads to perform the final copying step, 4 of which appear in the IOI and Colored Objects circuit. The remainder of the circuit uses very different components giving us 25% overlap. We hope this puts the overlap displayed in the main paper into perspective.

Reviewer comment (Experiments):
**Interpretation of Intervention Experiment**: In order to prove that some heads are performing the role of gathered heads, the authors conducted an intervention experiment in the model. They did so by blocking attentions to some words in these heads and then measuring the impact on accuracy. They observed that the intervention causes accuracy to decrease and concluded that it successfully demonstrates the role of these heads. However, once again, this experiment lacks a control baseline. Authors should, for instance, reproduce the same intervention on similar heads (but that are not expected to perform the role of CG) and see if the result is different. If the interventions on random heads do not produce the decrease in accuracy observed with candidates’ heads, then it is reasonable to say that these candidates are CGs. Without these control baselines, it is not possible to draw any conclusions from the intervention.

Author response:
> In order to prove that some heads are performing the role of gathered heads, the authors conducted an intervention experiment in the model. They did so by blocking attentions to some words in these heads and then measuring the impact on accuracy… this experiment lacks a control baseline.

To address this, we run three new control experiments. There are 3 content gatherer heads, so we repeat the intervention using three randomly sampled heads that aren’t in the circuit from the layers that the 3 real content gatherers come from (without replacement, so 9 different heads were tried). Across these three control baselines the accuracy barely changes: 49.9 +/- 1%. Recall that the original accuracy is 49.7%. This shows quite decisively that the content gatherers have a particularly strong impact in the way we predict. Thank you to the reviewer for pointing this out.

Reviewer comment (Novelty):
**Related Work**: The section on related work is too concise. To provide a broader context, the authors should expand on related work in the field of language model interpretability, specifically those that address the task-specificity of interpretability findings. This will help readers better understand where this work fits in the existing literature and how it contributes to addressing the challenge of task-specificity in interpretability studies.

Author response:
> The section on related work is too concise.

Agreed, unfortunately we cut back to save the much needed space. After revisions we will see if we can reclaim this space, or at least provide a further related work section in the appendix.

Reviewer comment (Evaluation):
**Overlap Measurement (Major Question)**: The paper relies heavily on the concept of overlap in in-circuit attention heads, particularly in demonstrating the generalization of mechanisms. While it is acknowledged that putting an exact number on this similarity is complex, can the authors provide a more detailed description of the metric used for measuring overlap? Additionally, could they quantify the uncertainty associated with this metric and provide insights into the expected range of overlap values? To help readers assess the significance of the 78% overlap, would it be possible to provide reference baselines, such as comparing the IOI circuit overlap with a set of random tasks or comparing it with the overlap regarding another circuit, even if it's a randomly generated one?

Author response:
> As an illustration, authors wrote that 78% of the IOI circuit is used in the Colored Object task. However, without more information, it is difficult to say how significant this overlap is… would it be possible to provide reference baselines, such as comparing the IOI circuit overlap with a set of random tasks?

In the new Appendix H, we run path patching experiments on two additional tasks that are significantly different from each other and the original IOI and Colored Objects tasks. These are the Greater Than task (Hanna, et al. 2023) and a simple factual recall task in which the model must predict the capital of a country (“The capital of France is”). The first represents a task that we expect will have minimal overlap (based on the Hanna, et al. circuit) and the latter tests a specific hypothesis about the reuse of mover heads, as well as provides one additional differing datapoint (see our general comment). In the first task, the overlap is only 16.7% and 22.7% for the IOI and Colored Objects tasks, respectively. The World Capitals task uses 7 mover heads to perform the final copying step, 4 of which appear in the IOI and Colored Objects circuit. The remainder of the circuit uses very different components giving us 25% overlap. We hope this puts the overlap displayed in the main paper into perspective.

Reviewer comment (Novelty):
**Related Work (Major Question)**: The related work section, while present, is relatively concise. To provide a more comprehensive context for the readers, can the authors expand on related work in the field of language model interpretability, particularly focusing on those works that address the task-specificity of interpretability findings? This would help clarify where this research fits in the existing literature and how it contributes to addressing the challenge of task-specificity in interpretability studies. Are there any specific prior works that have addressed or attempted to address the same problem or similar questions, and how does this paper compare to them? In particular, how does this work relate to the induction and duplicate heads found in Olsson et al. 2020?

Author response:
> The section on related work is too concise.

Agreed, unfortunately we cut back to save the much needed space. After revisions we will see if we can reclaim this space, or at least provide a further related work section in the appendix.

Reviewer comment (Presentation):
**Clarity (Major Question)**: Can the authors address the issues related to clarity, particularly in Section 4.3 and with the legends in the figures? Could they provide clearer explanations for terms and visual elements like "attn prob on token" and "dot w/ token embed" (Fig 4)? Additionally, what is the significance of the color information in the figures (e.g. Fig 4), and could the authors provide more context or commentary regarding its use in the visualizations? Lastly, are there plans to improve the clarity of explanations in the appendix, given its complexity, to enhance the accessibility of the research for a broader audience? Clarifying these elements would greatly benefit the overall presentation of the work.

Author response:
> Clarity: Can the authors address the issues related to clarity, particularly in Section 4.3 and with the legends in the figures? Could they provide clearer explanations for terms and visual elements like "attn prob on token" and "dot w/ token embed" (Fig 4) ?

Yes, this kind of work makes use of a lot of jargon. We would like to add a glossary to the appendix on top of providing better explanations in the main paper. Please allow us to address this in our next revision

Reviewer comment (Experiments):
**Failure to Utilize IOI Circuit (Minor Question)**: Why does the language model fail to correctly utilize the IOI circuit on the Colored Objects task? Is this failure potentially influenced by the number of distractors in the task? If so, it would be informative to explore varying the number of distractors in the task and analyzing the resulting overlap and accuracy to gain deeper insights into the model's behavior in different contexts.

Author response:
>Why does the language model fail to correctly utilize the IOI circuit on the Colored Objects task? Is this failure potentially influenced by the number of distractors in the task? If so, it would be informative to explore varying the number of distractors in the task and analyzing the resulting overlap and accuracy to gain deeper insights into the model's behavior in different contexts

Ultimately it seems to come down to the inhibition heads. In the original IOI paper the inhibition heads ‘activated’ in response to duplication detection. Our intervention shows that the inhibition heads work even without duplication (the color words are not duplicated in the prompt). It appears that the inhibition heads are more complicated than was initially believed and getting to the bottom of it is out of the scope of this work. Also, we tried additional distractors but the baseline model performance was too low to be useful.

Reviewer comment (Writing):
**Clarification of [end] Position (Minor Question)**: What does the term [end] position refer to in the text? It is mentioned but not clearly explained.

Author response:
>What does the term [end] position refer to in the text? It is mentioned but not clearly explained.  
This is just the last token in the prompt. So the “:” token in Colored Objects. Good catch.

Reviewer comment (Reproducibility):
**Definition of Important Heads (Minor Question)**: In Section 4.5, the paper discusses the top 2% of important heads. Could the authors provide a more detailed explanation of how these important heads are defined, particularly in relation to the path patching technique?

Author response:
>In Section 4.5, the paper discusses the top 2% of important heads. Could the authors provide a more detailed explanation of how these important heads are defined, particularly in relation to the path patching technique?

This is based on the heads that cause the highest logit difference when patched. We use the top 2% per path patching step (e.g., the top 2% most important heads that affect the logits, or the 2% most important heads that affect the queries of the mover heads). We attached a number to this to be more objective and concise, but we would prefer the analysis experiments demonstrating the similarity to be the main point of reference for similarity. Typically what counts as in-circuit vs. out-of circuit is up to the discretion of the researchers, we offer this number as at least some way of leaving our judgment out of it, but it isn’t meant to be perfect.

Reviewer comment (Novelty):
The finding that there are circuits that are reused within transformer networks is not new, since this is known e.g., for induction heads. (This is also pointed out by the authors in their related work section.) Therefore, it is unclear what the new conceptual contribution of this paper is.

Author response:
>So I am having trouble wrapping my mind around what the new conceptual contribution of this paper is

The question we set out to answer with this paper is “Does circuit analysis actually help us understand language models?” The primary criticism of this type of work is that so far it hasn’t been clear whether reverse-engineering LMs at this level of granularity grants us any higher level understanding of what they do. Circuit analyses all look very different, if it turned out that there weren’t patterns across different datasets/behaviors, this type of interpretability work would end up offering very little. Our paper is the first to perform the comparisons needed to answer positively, and show that we can accomplish a surprising amount of control over the LM as a result. The reviewers point out a few areas where we can strengthen this core claim; in our revised draft we added path patching experiments on two new tasks to provide more reference points and provide a more decisive “yes” to our motivating question.

## Rebuttal 2 (ID: StbM5udJS5)
Thank you for the reply

>I am not sure I buy this. Aren't induction heads already known to appear in multiple LM circuits? And circuit reuse has also been studied outside of LM models (e.g., specialized neurons such as corner detectors in convnets).

There are two points that we’d like to make here.
1.) We do not just show that heads are doing something consistently across contexts, but instead that they interact in predictable ways, *and* that lessons from lower level analysis on smaller models and toy tasks can directly guide interpretability efforts on larger models and more complex tasks. For example, building on the insights from Wang et al. allow us to design the inhibition head intervention. All of this shows that circuit analysis builds towards a higher level of understanding of LMs, not just one-off lessons about how a single model solves a single task. 
2.) More specifically with respect to induction heads:. Induction heads will always fire to the same patterns in text: “A B… A__(B)” (perhaps analogous to an edge detector) so we know exactly when and how they will fire, but not how the information they write out is used. In Wang, et al. they are careful not to overclaim what the _Name_ mover and _Subject_ Inhibition heads do because they are dependent on the context in a way that induction heads are not. So it’s not trivial to show that these perform similar operations in new contexts. We appreciate that the reviewer is probably very familiar with the related work and has maybe even observed reuse of these more complex components themselves, but it’s important to establish rigorously in controlled settings if we are to make progress and convince others that this type of analysis work is worthwhile. Within the community, we have fallen into these kinds of traps before. Take for example, the word embedding analogies findings of [Mikolov, et al. 2013](https://aclanthology.org/N13-1090/) which showed that static word embeddings encode semantic relationships like (Russia - Moscow)+France = Paris. This led to people overgeneralizing their belief in what kinds of analogical reasoning was possible with word embeddings. It wasn’t until years later that it was shown that this breaks down for many types of relations ([Gladkova, et. al 2016](https://vecto.space/projects/BATS/)). This is why this type of work is important.

Reviewer comment (Novelty):
In terms of techniques, the identification of the IOI subnetwork in GPT-2 medium reproduces an analysis of Wang et al. (2022) using their path patching method. The Colored Objects task network is identified using the same previously-known path patching method. The novelty in identifying this network seems to be mainly in determining what the different heads in this subnetwork do and expressing this as a human-interpretable algorithm, but this style of analysis already appears in the IOI paper.

Author response:
> But this style of analysis already appears in the IOI paper.

We see this is a strength of the paper, not a weakness. The point of our paper is to perfectly replicate the method used in the IOI paper and change one variable at a time: first the model (switching to GPT2-Medium), then the task (switching to the Colored Objects dataset) and analyze how the results change.

Reviewer comment (Theory):
The analysis is mentioned in the appendix to not apply to GPT-2 Large & GPT-2 XL, where there is no significant overlap between the IOI and Colored Objects subnetworks. While the authors’ honesty in reporting this negative finding is appreciated, this seems like a major strike against the phenomenon of circuit reuse advocated by the paper. An in-depth explanation of why we could expect GPT-2 Large and GPT-2 XL to not have circuit reuse would be valuable.

Author response:
To the point about GPT2-Large and XL, as well as the reviewer’s question about Figure 23: We may have miscommunicated the results here a bit, apologies for that. For both tasks and models, many of the heads affecting the logits are mover heads for both tasks. In, e.g., GPT2-Large, 5 of the top 10 heads are shared between tasks and they are mover heads for both. Most of the other heads also appear to be mover heads (or negative movers in the case of IOI), even if they aren’t shared as the most important between the two tasks. To answer the second question, this is what we mean when we say “large number of mover heads”. In these larger models with many more heads, there are more ‘paths’ through the network, so it seems like literal overlap in the exact heads being used is less likely, but the mechanism used to solve the tasks is the same. Should this be considered circuit-level overlap? To us, this is a consideration when working with larger models, but not a counterpoint to our claims. There is no consensus on this, so we respect the reviewer’s opinion if they still feel the same way, but we think this additional information might be helpful to consider. We have expanded on this appendix section to make this point clearer.

## Rebuttal 2 (ID: StbM5udJS5)
>Thank you for clarifying. From your response it sounds like there is not as much circuit reuse, literally speaking, for these larger models. Is that correct?

In a literal sense, yes, path patching tells us that some heads are contributing more to one task than another, but they are all contributing and are generally being used for the same functionality (moving). If a model has 100 mover heads, is every task that uses them going to assign importance the same way across them all? This was the sort of question that motivated running the experiment. The results show potential pitfalls of using a simple overlap metric and the importance of properly understanding what the heads are actually doing to establish whether functionally similar things are happening. We hope this point is now clear, because we don’t want to mislead the readers into an oversimplified picture of what the models are doing.