Paper: Copy Suppression: Comprehensively Understanding an Attention Head (ICLR-2024)

Reviewer comment (Experiments):
The paper is limited to a very specific model architecture and probably checkpoint, so whether the conclusion is generalizable and the method is scalable is questionable.

Author response:
Our initial findings already demonstrated the scaling of copy suppression across different models but in an Appendix that we’ve now to the main text (Section 4.1). To further bolster this, we've added a new appendix focusing on Llama models, showcasing the presence and implications of copy suppression in these larger and more complex architectures (Appendix A).

Reviewer comment (Experiments):
While reading the paper, I was wondering how did you find and choose to target L10H7?

Author response:
The project initially sprung out of an analysis of the roles of heads L10H7 and L11H10 in the indirect object identification circuit. We started out with the theory that these heads were responding to predictions made by the earlier name mover heads, rather than to task-specific features. After accruing evidence for this hypothesis, we moved to studying these heads on the entire OWT distribution, and found that the hypothesis still held up. We focused more on L10H7 because it was a cleaner example of copy suppression, and scored higher on our metrics, however we also believe L11H10 is implementing a weaker version of the same algorithm. As mentioned previously, our analysis of attention heads over a large class of models in Section 4.1 provides evidence for copy suppression being a universal phenomenon.

Reviewer comment (Experiments):
The conclusions given in the paper about transferability across different model classes, sizes, and data are not clear. In my opinion, this is the biggest issue with this paper. Although the author's experiments involve other models such as GPT-2 medium and Pythia besides GPT-2 small, it still does not eliminate concerns about this issue. The unclear applicability of the conclusions makes it difficult to assess the paper's contribution.

Author response:
> The conclusions given in the paper about transferability across different model classes, sizes, and data are not clear

As mentioned in our general comments, we acknowledge that our initial presentation did not communicate the universality of the copy suppression phenomenon across different models, and we have revised Section 4 to emphasize this point. We believe this highlights that copy suppression is a phenomena that exists across different model classes, sizes, and data, as our experiments include Pythia, SoLU, an GPT models (which are models of different classes and sizes, trained on different datasets). Notably, Pythia (unlike GPT-2 Small) uses rotary positional encodings which alters the functioning of the attention mechanism, yet copy suppression heads still form. Pythia was also trained without dropout, creating less of a need for self-repair.

Reviewer comment (Presentation):
The presentation of this paper is not clear enough. For example, Figure 1 shows a illustration of L10H7's copy suppression mechanism, which is not easy for readers to understand. In Section 3.1, the WU matrix appears for the first time but without any clear explanation, which may cause difficulties for readers to understand.

Author response:
> The presentation of this paper is not clear enough. 

We appreciate your comments on improving the clarity of the paper. From your feedback, we have taken steps to make Figure 1 clearer and more legible, removing the unnecessary details and adding more to the figure caption (highlighted in red), and have additional inserted various definitions in Section 2-4 (including definitions for the W_U matrix, the effective embedding, logit lens, and Q-composition). We hope this adds additional clarity to these sections of our paper. Has this addressed your concerns? If not, we would appreciate pointers to how else our presentation may be improved.

Reviewer comment (Writing):
The connection between copy suppression and self-repair may not be intuitive. Can authors elaborate on the relationship between copy suppression and self-repair? Also, what motivated authors to explore the role of copy suppression in self-repair?

Author response:
> The connection between copy suppression and self-repair may not be intuitive. Can authors elaborate on the relationship between copy suppression and self-repair? Also, what motivated authors to explore the role of copy suppression in self-repair?

We agree that the connection between self-repair and copy suppression is not entirely intuitive. 

Here is one example we think is intuitive - if the model is currently predicting the next token after the phrase “All is fair in love and”, there will be naive copying heuristics in the model which copy the token ‘ love’ from earlier in the context. The role of copy suppression is to ablate this naive prediction.

Now imagine we were to ablate the original copying mechanism, preventing the ‘ love’ token from being copied. This change doesn’t happen in isolation - it also subsequently deactivates copy suppression.

As a result, if one is measuring the effect of an ablation solely by measuring the presence of the ‘ love’ token in the unebedding, they will not see a significant change, despite the fact that they ablate the original copiers. This is self-repair in action, caused by copy suppression (and the lack of it upon the ablation of copying).

The authors had originally discovered copy suppression while studying the roles of heads L10H7 and L11H10 in the indirect object identification circuit, in which these heads clearly self-repair. Once we discovered copy suppression, it was natural to next attempt to connect both phenomena.

Reviewer comment (Novelty):
"negative heads" and "anti-induction heads" have already been discussed in prior work. It was not clear to me what this work contributes on top of these works.

Author response:
We think this misunderstands prior work as well as what our work contributes. Anti-Induction (Olsson et al., 2022: https://arxiv.org/abs/2209.11895) and Negative Name Moving (Wang et al., 2022: https://arxiv.org/abs/2211.00593) heads were found due to their behavior, not their mechanism. Specifically, researchers were studying specific tasks and found that there were model components that seemed unhelpful, for reasons that the researchers did not understand. We unify both of these components as common instances of copy suppression heads through their mechanism of detecting high confidence prediction, attention and suppression (1-3 in our introduction). Put another way, our contribution on top of these prior works is that we point out that their assumptions about task-specific negative components were both wrong – copy suppression is a general motif and is not specifically implementing a form of induction or name moving.

We are thankful that you studied this result, and we have moved it to the maintext (Section 4.1) as it is important for addressing the limitations you raised.

Reviewer comment (Experiments):
It is not clear why we should care about a single head in GPT-2 Small (a model that few people use in practical settings anymore). If similar heads could be identified across a wider variety of models that would increase the generality and perhaps interest.

Author response:
As addressed in the prior question, our results on negative heads and anti-induction heads are examples of copy suppression heads across GPT, Pythia and SoLU architectures, which is surprising since the copy suppression motif still arises in both of these very different tasks. We have moved this into the main text and we hope this is clearer. Further to your concern that the model studied is not practically deployed, we found evidence of a divide between positive heads and suppression heads in Llama-2-7B and Llama-2-13B (models used in practice) in our new Appendix A, with some limitations due to the nature of the tasks studied. We found suppression heads (that also performed anti-induction) in these larger models, showing generalization of our claims.

Reviewer comment (Evaluation):
Even if "1." and "2." is achieved, it is not clear to me how these findings would be actionable. I don't necessarily think that all interpretability research needs to be actionable, but it makes the relevance of the interpretations much more convincing if there is a path to better system building, more fair/generalizable models, etc.

Author response:
Mirroring two paragraphs from our introduction, we think that i) our work has actionable consequences for any work involving ablations in language model forward passes, including beyond interpretability and ii) our work has relevance to fundamental research into understanding deep learning models.

i): consider self-repair, which is partially explained by copy suppression (Section 4.2). Whenever techniques involve ablations in forward passes, self-repair can arise automatically through copy suppression, an observation that our work introduces, and which limits the utility of these techniques. Examples of work that intervenes on the forward pass for practical uses include Inference Time Interventions (Li et al., 2023: https://arxiv.org/abs/2306.03341) and Activation Addition (Turner et al., 2023: https://arxiv.org/abs/2308.10248). We made these arguments in our paper but have added citations thanks to this feedback. If copy suppression heads can be identified and frozen, this may reduce self-repair and make these interventions more effective.

Further, as touched upon in our introduction, “negative components may be ignored by attribution methods that only find the positive components”. Our work finds that some negative components – copy suppression heads – have developed an algorithm that sometimes suppresses the correct answer but improves performance across the training distribution. Though prior work had observed these negative algorithms, they were not studied in detail, and it was plausible that it was merely a spurious bug in the model. Therefore our work provides evidence that it is crucial that attribution methods in interpretability assume some internal model components may be negative.

ii): finally, independent of the arguments in i) we believe our work is a significant contribution to the scientific understanding of complex deep learning models. As discussed in our work, the achievement of reverse-engineering language model components across the whole pretraining distribution is a significant improvement upon the depth of understanding achieved in prior work. We have essentially no ground-up understanding of how large language models are computing their outputs, and a rigorous understanding of the smaller and more tractable models could provide a better groundwork for scaling to a true understanding of models. This is because existing methods of scaling explanations have had limited success (Huang et al., 2023: https://arxiv.org/abs/2309.10312). We believe that a deep understanding of model internals may in the long-term form a basis for more robust and transparent improvements and debiasing of models. A vision for the importance of mechanistic interpretability via basic science is written about in ‘Interpretability Dreams’ (Chris Olah; https://transformer-circuits.pub/2023/interpretability-dreams/index.html).