Paper: Summing Up the Facts: Additive Mechanisms behind Factual Recall in LLMs (ICLR-2024)

Reviewer comment (Evaluation):
This finding seems to be not profound enough. It only demonstrates that LLMs perform better under the additive motif, but it appears insufficient to prove that the additive motif is the underlying factual recall behind LLMs.

Author response:
Thanks for raising this – we have provided additional discussion about our main findings, and why we believe them to be novel, in a top level comment to all reviewers. See the sections titled “**additivity**” and “**novelty**”.

Reviewer comment (Theory):
This finding may explain the fact that models trained on “A is B” fail to generalize to “B is A”. Is there any possible to explain the CoT prompting such as “let’s think step by step” by using your findings? Does it bring any other insights or explanations for other phenomena that are difficult to explain in LLMs? For example, is there any possible to explain the CoT prompting such as “let’s think step by step” by using your findings?

Author response:
Understanding why chain of thought prompting works so well via interpretability is an interesting problem. We however think our set up would not be a good place to study this, due to the simple one step nature of the problem of factual recall, which is fundamentally different to the multi-hop nature of chain of thought prompting. 

That said, an interesting area of further investigation may be studying “multi-hop factual recall”, which our results may help explain. Consider prompts of form “The largest church in the world is located in the city of”. Additivity may help models solve this task in one step, even though a human may reason about this problem in a sequential manner. We have added this idea to future work.

Reviewer comment (Experiments):
The main claim, additivity of the multiple mechanisms, is not very clearly demonstrated in the paper. The separation of the subject/relation heads (as displayed in Fig. 2) is impressive. However, neither the roles of the "mixed head" mechanism, the MLP, and additivity of all these mechanisms are not clearly demonstrated.

Author response:
We address your concerns regarding the main claim of additivity in a top level comment to all reviewers above titled **additivity**. The clearer distinction for additivity is into two clusters of updates - relating either to the relation or subject. Each of the four model mechanisms contributes to one or both (in the case of mixed heads) of these clusters.

Reviewer comment (Evaluation):
Visualizations are quite useful for observing some of the results. However, the discussion of findings based on a more quantitative measure (e.g., DLA difference between factual and counterfactual attributes) would be much more convincing, precise, repeatable, and general.

Author response:
Thanks for this comment. We found qualitative figures generally a more useful form to present our results. Our main aim was to demonstrate a range of mechanisms with different qualitative properties exist. This is what is necessary to demonstrate additivity, which is our main goal in the paper.  

That said, we do agree that a more raw form of data can also be valuable. In Appendix E we present several instances of such data, on a per-prompt basis. We include for instance tables of raw ordered output tokens, logit ranks, and DLA scores.

Reviewer comment (Presentation):
Overall, the paper is somewhat difficult to follow, relying on data in the appendix for some of the main claims and discussion points. Appendices should not really be used for circumventing page-limits. Ideally, most readers should not even need to look at them.

Author response:
Thank you for the feedback, sorry for this. We have since restructured to emphasise the additivity claim more prominently. This is the main claim in the paper. In order to show additivity, it suffices to have two mechanisms (namely, subject and relation heads). These are distinct, and we explain these in detail. We in fact find four mechanisms, which we include for completeness.  We struggled to fit figures for all four mechanisms in the main body. We therefore made the decision to prioritise “summary” figures in the first results section to give a high level picture of the various mechanisms, and a detailed explanation of two mechanisms.

Reviewer comment (Reproducibility):
The head type (subject/relation) definition uses an arbitrary threshold. Although it sounds like a rather conservative choice, it would still be good to know how it was determined.

Author response:
Thanks for this comment. We agree this choice is somewhat arbitrary, but we believe it’s useful to draw this distinction. See the top level comment, under “**classifying attention heads**”.

Reviewer comment (Presentation):
The "categories" defined at the beginning of the results section come as a surprise, and seem to be an important part of the analysis throughout. This should be defined/explained earlier.

Author response:
Thanks for raising this point of confusion. The categories are important to the analysis on the individual component level, but are not conceptually important to the main claim of additivity of the paper. There are several levels of granularity discussed in the paper. From high to low: two additive clusters, four separate groups of mechanisms, and many individual components implementing these. Categories operate on the lowest component level, and merely state individual components (like individual attention heads), have a narrow purpose. We have removed references to categories from the “summary of results” section, to make the main claims regarding additivity flow better and be more clear.

Reviewer comment (Novelty):
There have been many works [1, 2] interpreting the model behavior of Factual Recall. It seems that the novelty is insufficient with only a deeper zooming into attention heads using similar interpretability methods. Additionally, the discovery of the additive motif is not surprising enough, as already explained in work [3] that "Attention heads can be understood as independent operations, each outputting a result which is added into the residual stream."

Author response:
> (1) There have been many works [1, 2] interpreting the model behaviour of Factual Recall. It seems that the novelty is insufficient with only a deeper zooming into attention heads using similar interpretability methods. 

Our approach is fundamentally different to [1] and [2], and our findings subsequently different. As you noted, we chose to zoom in deeply into individual model components, which [1] did not do at all, and [2] only briefly studied. Through using a “circuits style” approach, we were able to find insights that both [1] and [2] missed. [1] studies where factual knowledge is stored in models, but not how such knowledge is used to predict the next token. [2] attempts to study this problem, but only does so through coarse grained ablations. They suggest a mechanism based on this, which we search for in individual model components. In doing so, we instead find several independent mechanisms that explain how information regarding factual knowledge is moved. We additionally find an extra place where factual information is stored – namely in the relationship token “sport” itself - prior work missed this important aspect, which we found through careful study of a range of counterfactual attributes S and R. This motivated the focus on “additivity” (see the top level comment titled "**additivity**" for more discussion on this). We also discuss the "**novelty**" of this work in another top level comment.

> Additionally, the discovery of the additive motif is not surprising enough, as already explained in work [3] that "Attention heads can be understood as independent operations, each outputting a result which is added into the residual stream." 

Thanks for raising this confusion. We address what we mean by additivity in a top level comment to all reviewers, and have amended the paper to clarify this. Our use of additivity is distinct from [3].

Reviewer comment (Reproducibility):
Is direct logit attribution (DLA) the same as the interpretability method of Path Patching [4] or Causal Mediation Analysis [5]? If so, it is necessary to explain how the counterfactual data is applied for causal intervention. If it is not, it is necessary to provide a detailed description of the algorithm flow of DLA.

Author response:
> (2) Is direct logit attribution (DLA) the same as the interpretability method of Path Patching [4] or Causal Mediation Analysis [5]? If so, it is necessary to explain how the counterfactual data is applied for causal intervention. If it is not, it is necessary to provide a detailed description of the algorithm flow of DLA.

Thanks for raising this confusion. Direct logit attribution (DLA) is not the same technique as path patching or causal mediation analysis. DLA is a simple technique that we describe in the paper in section 2. We have now added further discussion on this, including a detailed description of the algorithm, in Appendix D. It is not based on a causal intervention. It builds on a technique named the “logit lens”. Both are based on the insight by [3] that the residual stream is an accumulated sum of model components, and that the map to logits from the final residual stream vector is approximately linear. While running a single forward pass, we may take the output from individual model components, such as attention heads, and directly map these to logit space, by immediately applying the model unembedding (aka language model decoder head/final linear layer). Given a set of logits, we can read off the DLA for any possible output token, including for counterfactual output tokens.

Reviewer comment (Reproducibility):
This work extends “DLA by source token group” with a weighted sum of outputs corresponding to distinct attention source position. But how to obtain the “weights”? How to attribute multiple tokens simultaneously? These missing implementation details make it difficult to understand the method and reproduce the results.

Author response:
> (3) This work extends “DLA by source token group” with a weighted sum of outputs corresponding to distinct attention source position. But how to obtain the “weights”? How to attribute multiple tokens simultaneously? These missing implementation details make it difficult to understand the method and reproduce the results.

Our extension is to consider the attention head output as a weighted sum over individual source positions, where the weighting is given by the attention probability. This is entirely faithful to transformer computation. By unravelling this sum, we can consider DLA for attention heads to be attributed to all attention source tokens individually. We have provided a detailed discussion with formulae of this technique in Appendix D.

Reviewer comment (Theory):
We wonder if it is possible to explain the behavior of MLPs explicitly, similar to explaining Attention Heads via attention patterns?

Author response:
> (2) We wonder if it is possible to explain the behaviour of MLPs explicitly, similar to explaining Attention Heads via attention patterns?

Explaining the behaviour of MLPs is well documented to be significantly harder than interpreting attention heads in the interpretability literature. The function of attention heads can be thought of as primarily of moving information around different token positions. MLPs instead perform meaningful computation, and have many more parameters. They have also been noted to operate on features under a large amount of superposition [1] – meaning individual neurons encode many different concepts and are in general uninterpretable in networks in isolation. In this work, we only attempt to interpret the entire output of MLP layers, but agree that further work could investigate this computation more deeply. As this is not central to the key claims of the paper, we note this as a possible area of future investigation in the future work section.

[1] FINDING NEURONS IN A HAYSTACK: CASE STUDIES WITH SPARSE PROBING

Reviewer comment (Evaluation):
The paper introduction and further discussions claim that the results reported here provide a mechanistic explanation for the limitations of LLMs to learn "B is A" from training on "A is B" [1]. However, I do not see sufficient evidence to support this claim.

Author response:
We thank the reviewer for their feedback. We believe that our work does provide a mechanistic explanation for the reversal curse, but thank the reviewer for pointing out the need to communicate this more clearly. The reversal curse claims that models trained on “A is B” fail to generalise to “B is A”. Importantly, this is difficult to verify with purely pre-trained models on common facts (e.g. Michael Jordan plays basketball) where the model likely saw the fact in both forms (basketball is played by Michael Jordan, Michael Jordan plays basketball). The reversal curse paper demonstrates this with fine-tuning on novel facts, ensuring the reverse-direction was not seen. 

In our work, we only study a pre-trained model, and so the evidence we provide is indirect and suggestive, as we do not fine-tune ourselves. We find a circuit by which models may learn to output “A is B”, involving subject enrichment on the A tokens, and some attention head attending to A and extracting B. Importantly, this is a unidirectional circuit with two unidirectional components - it extracts the fact “B” from “A”.  This suggests that the reason fine-tuning on “A is B” does not boost “B is A” in general is because training on “A is B” only boosts the unidirectional A -> B mechanisms, and has no effect on potential B -> A mechanisms.