Paper: Successor Heads: Recurring, Interpretable Attention Heads In The Wild (ICLR-2024)

Reviewer comment (Writing):
Effective OV circuit: what is meant by effective here?

Author response:
> Effective OV circuit : what is meant by effective here ?

We use “effective” to emphasise that i) this includes MLP0, which is not standard in prior work and ii) this more closely matched how the model uses attention heads to transform information from embeddings to unembeddings

Reviewer comment (Experiments):
What was the motivation for using MLP$_0$ rather than the original word embeddings? Given that MLP$_0$ occurs after a first self-attention layer, I would have not expected it to map embeddings to a representation amenable for the analysis presented in the paper.

Author response:
> What was the motivation for using MLP rather than the original word embeddings ?

Lots of literature on Language Models has found that MLP0 is crucial for model representations, e.g Wang et al., 2022 (https://arxiv.org/abs/2211.00593), Hanna et al., 2023 (https://arxiv.org/abs/2305.00586). Further, Pythia models use parallel attention (introduced in https://arxiv.org/pdf/2204.06745.pdf), ie the input to MLP0 is just the embedding and does not include the attention layer so it is even more natural for this architecture to not just use the word embeddings.

Reviewer comment (Experiments):
It is mentioned that 128 samples are sampled from the Pile dataset. Did you happen to find a token where the successor head was important for each of the 128 samples? Or did you bias your sampling towards extracts where the successor head is important?

Author response:
> Did you happen to find a token where the successor head was important for each of the 128 samples ? Or did you bias your sampling towards extracts where the successor head is important ?’:

We apologize for not being clearer here; the 128 samples from the Pile were randomly sampled (with no bias), and we have now specified this.

Reviewer comment (Reproducibility):
In Section 4, direct effect mean ablation is used to show "that when the successor head is contributing usefully, the prompts often required some kind of incrementation". Then in Appendix J, direct effect mean ablation is used again to show that the successor head is the most important head across 64 prompts. Though this is stated with some evidence, not enough quantifiable evidence is shown here to justify the reach of these claims, such as the statement of "mostly solved". More analysis can also be shown about the "direct effect" to separate it from "indirect effects". The paper also did not clarify the details of the ablation, such as if it used resampling ablation (there may be issues if it used mean ablation from the same dataset, as there are known issues with mean ablation [1]), and/or path patching (to obtain direct effects). [1] https://www.lesswrong.com/posts/kcZZAsEjwrbczxN2i/causal-scrubbing-appendix#3_Further_discussion_of_zero_and_mean_ablation

Author response:
> not enough quantifiable evidence is shown here to justify the reach of these claims, such as the statement of "mostly solved”

This is a good point, and we agree that the indirect effects should have been studied. We have now analysed the indirect and the total effect for the loss reducing experiments, and evidence that indirect effects do not play a significant role in successorship. These results are described in Appendix L.

> The paper also did not clarify the details of the ablation

We apologize for not specifying this. Our experiments in Section 4 used direct effect mean ablation, taking the mean on the same batch, and we have now included this detail. Additionally, we have repeated our loss reducing experiments using resampling ablation instead of mean ablation and see similar results, with results described in Appendix L.

Reviewer comment (Experiments):
The paper mentions that vector addition was performed successfully for 89% of the cases for digits 20-29, and mentions how it was performed on token '35'. It does not mention how this performed for other digits. This is likewise the case for number words only showing ten to twenty. Presumably, the performance is similar, but the paper should explicitly mention this to avoid criticism of cherry-picking. In Figure 7, it's also unclear what "target residues modulo 10" means when referring to the column headers. Presumably, this is stating something similar to how the vector arithmetic on MLP0(E('fifth') makes it "behave more like MLP0(E('seventh')". The wording can be made clearer to avoid confusion that it means the number "7" rather than the word "seventh". Additionally, the checkmarks are given when "the max logits are on the successor token". This is an interesting result, but how big is the logit difference between logits for the successor token and other tokens? Appendix D states that scaling was used on the additive feature terms. A quick explanation of why a particular scaling factor was used would be helpful.

Author response:
> Presumably, the performance is similar, but the paper should explicitly mention this to avoid criticism of cherry-picking

We have now mentioned that performance is similar across other tokens and have included some additional arithmetic tables, displayed in Appendix O.

> it's also unclear what "target residues modulo 10" means when referring to the column headers

We have included an example of what we mean by ‘target residue’ in the caption of Figure 6 (this figure was Figure 7). We hope this makes things clearer.

> how big is the logit difference between logits for the successor token and other tokens?

We have included the logit distributions for randomly sampled checkmarked cells in Appendix P to give a picture as to how the logits differ across tokens.

> A quick explanation of why a particular scaling factor was used would be helpful

We agree that this aspect should be described further and have added some more explanation behind choosing the scaling factor in Appendix D.

Reviewer comment (Novelty):
The paper states: "to the best of our knowledge the presence of both successorship and acronym behavior in head L12H0 is the cleanest example of polysemantic behavior identified so far in an LLM." Why is this the cleanest example of polysemantic behavior, compared to other studies on the topic such as in [2]? Similarly for this statement, "which to the best of our knowledge are the most closely studied components in LLMs that occur in both small and large models", what other components are you comparing to that are not as closely studied? [2] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023

Author response:
> what other components are you comparing to that are not as closely studied?

We are comparing to induction heads (https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html), the only other interpretable, recurring language model component in small and large models that we claim do not have as crisp an explanation as successor heads (analyzing attention patterns is not an end-to-end explanation) - we have now reviewed this in the related work section.

> Why is this the cleanest example of polysemantic behavior?

Thank you for the reference to the Gurnee et al. work. We agree that this provides helpful evidence about polysemanticity in neurons. We think that our work is cleaner, since we measure the loss explained by the succession, acronym and other behaviors (Figure 10). This shows that this attention head’s direct effect has over half of its behavior explained by these behaviors – Gurnee et al show that some neurons implement several different tasks, but don’t make claims about the role of these neurons across the whole training distribution. It is possible that their bigram examples do not explain the majority of the role of this neuron in the model.

Reviewer comment (Novelty):
This paper discovers novel and interesting observations, but it does not elaborate much on why this observation is impactful enough.

Author response:
> This paper discovers novel and interesting observations, but it does not elaborate much on why this observation is impactful enough

We agree that the primary contribution of our work is about interesting phenomena without a directly impactful outcome. However, as elaborated in the introduction, there are several second-order effects of our work. Firstly, we provide useful evidence about the universality of representations in models: successor heads generalize to larger models, but cross-task generalization is more complicated, such as how steering the successor head on days of the week did not produce as strong results. Additionally, we find evidence that models have compositional representations, such as combining a mod 10 feature with task-based information to produce certain completions, suggesting that LLMs have sophisticated internal structure, a similar conclusion to https://arxiv.org/abs/2310.02207, but from a different angle (numeracy). Finally, our work shows another bit of evidence for the strength of Sparse Autoencoders, which in parallel to our work were found to have a lot of success at almost completely explaining a small language model (https://transformer-circuits.pub/2023/monosemantic-features), which we hope the interpretability field will find useful.

Reviewer comment (Experiments):
While using MLP0 with successor heads, in isolation, was shown to be sufficient to perform incrementation, it is not shown how they interact with the rest of the models they belong to that the paper studied. How is this information about numerical features processed in later layers, from MLP0 to the successor head L12H0, then to the rest of the model? How does this end-to-end path interact with alternative paths, and inhibitory ones? The mod-10 features are obtained from MLP0's outputs, to study how the end-to-end path of MLP0 to the successor head processes these features. However, the paper does not show how this information is processed through other layers and MLPs in the models. Can such mod-10 features be found in other layers?

Author response:
> How is this information about numerical features processed in later layers, from MLP0 to the successor head L12H0, then to the rest of the model? How does this end-to-end path interact with alternative paths, and inhibitory ones? (...) the paper does not show how this information is processed through other layers and MLPs in the models.

We believe that a full exploration of these details is out-of-scope for our work, however we have added analysis of the indirect effect to a new Appendix to our work (Appendix L). This suggests that processes occurring after the successor head’s output may not play a significant role to succession, hence an analysis beyond the direct effect is less important.

Reviewer comment (Experiments):
There are also the cases of adding non-mod-10 features to numeric tokens, and adding the features to non-numeric tokens. There may be cases of "confounding" factors where U(OV(MLP0(E('twelve')) - f_2 + f_4)) has high logits for 'fourteen', but is it possible that adding non-numeric features will also shift twelve to thirteen, fourteen, etc.? If so, then perhaps these mod-10 features are just obtaining the correct change because of how the numeric features are scaled with one another? In other words, would subtracting non-numeric features R then adding RR obtain the same result? This is not a weak point of the paper as it may be beyond its scope, and this situation seems unlikely to be the case, but it can be further investigated for thoroughness.

Author response:
> would subtracting non-numeric features R then adding RR obtain the same result?

We are not sure we understand this concern. We have included additional experiments exploring the separability of the numeric and task subspaces in Appendix M, which we hope you appreciate. These demonstrate that we can isolate a common numeric subspace within embedding space, that for any given token (e.g. ‘February’), encodes the index of that token within its ordinal sequence (e.g. months). If this does not address your concern, let us know.

Reviewer comment (Novelty):
This paper tackles a similar topic as a previous project from earlier this year, which also found a successor attention head by performing OV circuit analysis on inputs of numbers, number words, days, months, and letters. It also was stated to have inspected the effects of vector addition on number tokens [3]. Could the authors elaborate on the similarities and differences? [3] https://alignmentjam.com/project/one-is-1-analyzing-activations-of-numerical-words-vs-digits

Author response:
> Could the authors elaborate on the similarities and differences?

Though the referenced project similarly studies model behaviour on tasks involving incrementation and finds a successor head, it is limited to a single successor head in a small model (GPT-2 Small), whereas our work finds that this is a more general and recurring phenomena in larger models of varying architecture, and we describe an end-to-end circuit that can perform incrementation using these heads. Our work also performs a weight-level analysis of successor heads, allowing us to discover mod-10 features, however the referenced project does not perform such an analysis. They also do not investigate how their found successor head performs on natural language datasets and the tasks for which it is important for, which we analyze in Section 4. We were not aware of this work (and it is outside the time period of necessary consideration and non-archival) but have added it as a reference and are glad that some of our results were found independently.