Paper: Towards Best Practices of Activation Patching in Language Models: Metrics and Methods (ICLR-2024)

Reviewer comment (Novelty):
The work overall is very interesting but it feels a bit light on the novelty. Perhaps proposing additional novel methods for input corruption and improvements for the activation patching techniques can help to increase the novelty in this paper.

Author response:
We believe that our work is significant, since it gives the first rigorous and comprehensive evaluation of a foundational technique.  As we surveyed in the paper and [a rebuttal comment above](https://openreview.net/forum?id=Hf17y6u9BC&noteId=swlVw4U9Z4), a large number of recent papers in interpretability have all used activation patching (AP).
Yet, many works use somewhat different variants in terms of metric or implementation details. Prior to our work, there was no comprehensive study on the consistency of AP. Our paper fills this gap by (i) rigorously evaluating a wide range of techniques within AP and (ii) providing recommendations for its best practices. Hence, while the main contribution of our work may be unglamorous, we believe it is crucial as it provides the rigor for the field moving forward. 

In terms of novelty, our investigation into GN corruption suggests that it may introduce OOD inputs to the model's internal components. We further argue that it may lead to unreliable interpretability claims. To the best of our knowledge, this insight is novel in the literature. It provides an importance evidence for the limitation of the technique.

Reviewer comment (Writing):
The way the paper is written it might be a good fit for a workshop.

Author response:
Our hope was to write a paper that would be of value to practitioners in LM interpretability who use activation patching in their research, as well as of broad interest to the community. We would appreciate any advice on improving the presentation of our paper.

Reviewer comment (Writing):
Some terminology could be explained in the paper. E.g. Name Mover. It seems that the paper requires prior knowledge of another paper Wang et.al. Some concepts such as: 0.10 negative detection is not very clear.

Author response:
Thanks for pointing this out! In the end of the first paragraph of Section 3.2, we added the definition that “we say a detection is negative if the patching effect of the component is negative (under a given metric).” Intuitively, this means that patching suggests the component hurts model performance. We also added Appendix B, an overview of the attention heads in the IOI circuit of GPT-2 small, as found by Wang et al. We hope these clarify the setting of Table 1.

Reviewer comment (Novelty):
What are some of the novel contributions of the paper ?

Author response:
We believe that our work is significant, since it gives the first rigorous and comprehensive evaluation of a foundational technique.  As we surveyed in the paper and [a rebuttal comment above](https://openreview.net/forum?id=Hf17y6u9BC&noteId=swlVw4U9Z4), a large number of recent papers in interpretability have all used activation patching (AP).
Yet, many works use somewhat different variants in terms of metric or implementation details. Prior to our work, there was no comprehensive study on the consistency of AP. Our paper fills this gap by (i) rigorously evaluating a wide range of techniques within AP and (ii) providing recommendations for its best practices. Hence, while the main contribution of our work may be unglamorous, we believe it is crucial as it provides the rigor for the field moving forward. 

In terms of novelty, our investigation into GN corruption suggests that it may introduce OOD inputs to the model's internal components. We further argue that it may lead to unreliable interpretability claims. To the best of our knowledge, this insight is novel in the literature. It provides an importance evidence for the limitation of the technique.

Reviewer comment (Reproducibility):
How was the content of Table 1 computed ?

Author response:
As detailed in Section 3.1, we say that a head is detected if its patching effect is 2 standard deviations (SD) away from the mean effect. We have also clarified this in the main text: Table 1 is computed by patching each individual head, calculating the patching effects, listing the heads that are detected (according to the definition above) and contrasting with the results from Wang et al.

Reviewer comment (Experiments):
I am not very familiar with the details of the existing activation patching methods. Therefore, I am not sure whether the methods included in the paper are diverse and representative enough for the mechanistic interpretability community.

Author response:
Thank you for evaluating and appreciating our work! 

> I am not very familiar with the details of the existing activation patching methods. Therefore, I am not sure whether the methods included in the paper are diverse and representative enough for the mechanistic interpretability community.

First, we note that activation patching (AP), as a generic method, is foundational and widely used in mechanistic interpretability. In particular, the third paragraph of our introduction lists 5 prior works that apply the method. We also refer the reviewer to consider the related work section that gives more examples. In total, one can  identify over 20 papers using AP or its variants in the literature (which [we list in a separate comment below](https://openreview.net/forum?id=Hf17y6u9BC&noteId=swlVw4U9Z4)).

Second, the metrics and particular variants considered by our work cover a large fraction of existing techniques in AP. Specifically, this includes techniques used in [Meng et al., 2022](https://rome.baulab.info/), [Wang et al., 2023](https://arxiv.org/abs/2211.00593), [Hanna et al., 2023](https://arxiv.org/abs/2305.00586), [Heimersheim & Janiak, 2023](https://www.alignmentforum.org/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only) and [Stolfo et al., 2023](https://arxiv.org/abs/2305.15054). All of them are studied in detail in our paper. As we surveyed in the related work section and the comment below, these same techniques are used in other works as well. In addition, we also investigate KL divergence as a metric, used in [Conmy et al., 2023](https://arxiv.org/abs/2304.14997). 

Finally, in section 6, we consider the effect of varying the corrupted token position, a rather neglected aspect in the literature. This again adds to the comprehensiveness of our study.