Paper: Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching (ICLR-2024)

Reviewer comment (Theory):
The definitions seem rather brittle. Requiring strict equality in the definitions of causally disconnected and dormant seems to be very limiting for practice.

Author response:
**We acknowledge that requiring strict equality in the definition of a dormant
subspace is limiting. In our revision of the paper, we have changed the
definition to allow for approximate equality instead**, which reflects the reality
better. In our experiments (Figure 20 and Figure 12) we find that the orthogonal
complements of the causally disconnected components of the subspaces we find are
relatively dormant compared to the causally disconnected components.

As for causally disconnected subspaces, **it is not too limiting to require strict
equality, and all our empirical examples of causally disconnected subspaces obey
strict equality**. Recall that a subspace of the activations of some model
component is causally disconnected if changing the activation only along this
subspace (leaving the component orthogonal to the subspace unchanged) does not
change the model’s output. This assumption is provably satisfied when, for
example, we consider the post-GELU activations of an MLP layer, and
$v_{disconnected}$ is in the kernel of the layer’s down-projection. This is
because the only causal pathway by which the MLP activations affect the rest of
the model is via the layer’s down-projection. A down-projection is a linear
operation, and adding a vector in its kernel does not change its output.

Reviewer comment (Theory):
Are there a set of empirical predictions (e.g. about the directions of activations or gradients) that could either falsify or support the theoretical model?

Author response:
We believe this experiment is an important signal for the presence illusion, but
we would also stress that it is somewhat specific to subspaces of hidden MLP
activations.

Specifically, for an MLP layer, there is a clear notion of a rowspace-nullspace
decomposition of a given activation vector. Namely, since we are considering the
post-GELU activations, and the only way such activations influence model output
is through multiplication with the down-projection matrix $W_{out}$ of the
layer, it is clear that, given an activation v, we can write $v = v_{nullspace} + v_{rowspace}$ where $v_{nullspace}\in \ker W_{out}$ and $v_{rowspace} \perp
v_{nullspace}$, and we know that intervening by modifying this activation to
change its projection along $v_{nullspace}$ will not change the model's outputs.
This is because $W_{out}(\alpha v_{nullspace} + v_{rowspace}) =
W_{out}v_{rowspace} = W_{out}(v_{nullspace}+v_{rowspace})$ for any $\alpha\in\mathbb{R}$. 

On the other hand, in the residual stream of a transformer, an analogous
decomposition would be vacuous. This is because the residual stream is
relatively low-dimensional (e.g., 768 in GPT2-Small), but at the same time it is
read by many attention heads in subsequent layers. This means that it is quite
likely the intersection of the nullspaces of all these attention heads'
key/query matrices is an empty subspace. We have also verified this empirically. 

This is the reason why we consider only the query matrices of name mover heads
in Section 5 of the paper. This gives us a non-vacuous notion of a
nullspace-rowspace decomposition. The downside is that it relies on assumptions
about the "important" parts of the model. We base these assumptions on the detailed previous work on the IOI circuit.

To sum up, this experiment is best applied to subspaces of hidden activations of MLP layers.

Reviewer comment (Theory):
The paper does not pose an explicit hypothesis that can be falsified, limiting its scientific validity. Further, the "illusion" seems to make problematic assumptions and has unclear implications.

Author response:
We thank the reviewer for their thoughtful and thorough review. We are sorry
that reading the paper was difficult, and we thank the reviewer for pointing out
several confusing parts of the paper. **However, there are some fundamental
aspects of our work that were miscommunicated, and we would like to clarify them
below**. We have uploaded a revision that we hope will alleviate these concerns,
with changes marked in red.

> The paper does not pose an explicit hypothesis that can be falsified, limiting
its scientific validity. [...] The hypothesis seems to be that every patched
subspace “v” discovered by DAS can be decomposed into two directions
“v_disconnected” and “v_dormant” (with specific properties)

The key message of our work is not that the illusion described will always
happen given some set of assumptions. Instead, **our central focus is showing
the *existence* of the illusion in practical cases of interest** for ML
interpretability. In the cases that we empirically discovered, the subspaces
found indeed decompose as a sum of two orthogonal vectors with the properties
given in Section 3 of the paper, and we found this decomposition a helpful way
to think about the mechanistic underpinnings of the illusion. **We do not claim
to provide specific conditions that will *guarantee* that the illusion will
occur**, or that such a decomposition will exist. This means that our work takes
the form of demonstrating the existence of the illusion, rather than proving a
falsifiable hypothesis that it should always happen, which we expect is too
strong a claim.

As other reviewers have noted, this is a valuable contribution to the
interpretability literature, as **the existence of such cases demonstrates that
one should not apply subspace patching techniques blindly**. In particular, a
key takeaway is that one should not rely solely on end-to-end metrics when
evaluating optimization-based subspace patching methods for the purpose of
localizing features in language models. 

We remark that we also provide several theoretical and heuristic reasons to
expect the illusion to be prevalent in practice in Section 7, but they are not
load-bearing to our main message.

Reviewer comment (Theory):
The hypothesis seems to be that every patched subspace “v” discovered by DAS can be decomposed into two directions “v_disconnected” and “v_dormant” (with specific properties), and that "patching along the sum of these directions, the variation in the disconnected part activates the dormant part, which then achieves the causal effect". The latter statement assumes that the model behaves somewhat linearly along these subspaces (i.e., the output of the model along direction "v" is given by a sum of its outputs along "v_disconnected" and "v_dormant"), which is a strong hypothesis given that these models are fundamentally non-linear. While the paper provides some evidence for the existence of some "disconnected" and "dormant" directions, there is no evidence justifying the apparent linear behaviour of the underlying model.

Author response:
> ..."patching along the sum of these directions, the variation in the
disconnected part activates the dormant part, which then achieves the causal
effect". The latter statement assumes that the model behaves somewhat linearly
along these subspaces (i.e., the output of the model along direction "v" is
given by a sum of its outputs along "v_disconnected" and "v_dormant"), which is
a strong hypothesis given that these models are fundamentally non-linear.

**We make no assumptions of linearity, only about the behavior of the model when
changing the activation only along v_disconnected, or only along v_dormant**. In
particular, to address the reviewer’s objection, we only need a weaker set of
assumptions: that **changing the activation only along v_disconnected does not
change model outputs**. This assumption is provably satisfied when, for example,
we consider the post-GELU activations of an MLP layer, and v_disconnected is in
the kernel of the layer’s down-projection. This is because the only causal
pathway by which the MLP activations affect the rest of the model is via the
layer’s down-projection. A down-projection is a linear operation (though
subsequent model layers are not linear), and adding a vector in its kernel does
not change its output, thus adding v_dormant, or adding v_dormant and
v_disconnected must have exactly the same downstream effect.

To unpack this in more detail, we believe that in the review, the words “...the
output of the model along direction "v" is given by a sum of its outputs along
"v_disconnected" and "v_dormant"...” are intended to mean “when changing the
activation along v, this results in the same change in model output as the sum
of the changes when changing only along v_disconnected and when changing only
along v_dormant, adjusting the magnitudes of the changes by the projection of v
on the two vectors”. 

**This is a true mathematical fact, but it is independent of any linearity
assumptions on the neural network**. Recall that by construction v =
(v_disconnected + v_dormant) / sqrt(2). Changing some activation A along v alone
is the same as changing it along v_disconnected and v_dormant simultaneously
(with coefficient 1/sqrt(2)), and leaving it unchanged along all directions
orthogonal to the plane spanned by v_disconnected and v_dormant. Let’s call this
new activation A’. By assumption, changing the activation A’ along
v_disconnected alone results in no change in model output; so this activation A’
will produce the same model output as the activation A’’ where only v_dormant
was changed. This shows that the change in outputs introduced by changing along
v is the same as the change in outputs introduced by changing along v_dormant
alone, plus the change in outputs after changing along v_disconnected alone
(which is zero).

Reviewer comment (Experiments):
The definition of the "dormant" subspaces is confusing in the context of this work. If the model output remains unchanged for in-distribution patching and changes only for out-of-distribution samples $x,y \sim \mathcal{D}$, what procedures presented in this work result in out-of-distribution samples/activations? As far as can be determined, all procedures described involve patching with in-distribution data.

Author response:
> The definition of the "dormant" subspaces is confusing in the context of this work. If the model output remains unchanged for in-distribution patching and changes only for out-of-distribution samples, what procedures presented in this work result in out-of-distribution samples/activations? As far as I can tell, all procedures described involve patching with in-distribution data.

While the reviewer’s concern would be true in the case of full component activation patching, **it is possible for subspace activation patching between in-distribution data to result in out-of-distribution activations** when the subspace is a proper subspace of activation space.  

To explain in more detail, imagine a 2-dimensional activation space, where
in-distribution activations are always on the x-axis, ie the y-coordinate is
always zero. Consider subspace activation patching along the vector
$v=(1/\sqrt(2), 1/\sqrt(2))$ which makes equal angles with the x and y axes. If
we patch along $v$ from an example with activation $(x_1, 0)$ into an example
with activation $(x_2, 0)$ with $x_1 \neq x_2$, the result is the activation

$$(x_2, 0) + 1/\sqrt(2) * (x_1-x_2) * v =  ((x_1+x_2)/2, (x_1-x_2)/2)$$

In particular, this activation is now out-of-distribution, since it has a
nonzero component along the y axis.

Reviewer comment (Writing):
It is unclear whether the hypothesis is that such a decomposition does not exist for “true” subspace directions.

Author response:
> Is the hypothesis that such a decomposition does not exist for “true” subspace directions?

**A definition of “true” subspace directions is outside the scope of our work**.
Rather, our central message is that the existence of such a decomposition, and
an empirical observation showing that removing the causally disconnected
component of a subspace greatly reduces the effect of the subspace activation
patch, should be taken as cause for concern when attributing features to
particular subspaces.

Reviewer comment (Presentation):
The main paper on its own does not seem to be self-contained and contains a significant number of references to the appendix.

Author response:
> The main paper on its own does not seem to be self-contained and seems to contain a significant number of references to the appendix.

We acknowledge that there is quite a bit of material in the appendix, and are
sorry that this made our work harder to follow. We have tried to condense the
central points of our work in the main body of the paper by making the following
changes, which have significantly reduced the dependence of the main body in
sections 4 and 6 on the supplementary material by including the following
details in the main body:
- IOI dataset we used (Section 4.1)
- Computing the direction $v_{grad}$ (Section 4.2)
- Computing directions using DAS (Section 4.2)
- Details for fact patching experiments (Section 6.1)
- We moved what was previously Figure 2 to the supplementary material, as we felt that it makes an insufficient contribution relative to the space it occupies.
- Similarly, we moved the statement of what was previously Lemma 6.1. To the
  appendix, and explained its result informally

We hope this alleviates your concern. If there are any other parts of the
appendices which felt important to understanding the main text, we would
appreciate you bringing them to our attention.

Reviewer comment (Evaluation):
The main hypothesis involves the claims that (1) subspace directions can be decomposed into two directions (disconnected, dormant) with specific properties, and (2) the effect of such directions adds somewhat linearly to the model outputs. However, the paper fails to connect this terminology ("disconnected, dormant directions") in sections 4-7, making it difficult to verify whether the experiments confirm or deny the hypothesis.

Author response:
> [...] However, the paper fails to connect this terminology ("disconnected, dormant directions") in sections 4-7, making it difficult to verify whether the experiments confirm or deny the hypothesis.

We do connect the terminology of disconnected and dormant directions to our experiments in several ways:
- For the IOI task, we orthogonally decompose the $v_{MLP}$ subspace as a sum $v_{MLP} = v_{MLP}^{nullspace} + v_{MLP}^{rowspace}$, where $v_{MLP}^{nullspace}$ is the orthogonal projection of $v_{MLP}$ on the kernel of $W_{out}$, and $v_{MLP}^{rowspace}$ is the orthogonal projection on the rowspace of $W_{out}$. 
  - By definition, $v_{MLP}^{nullspace}$ is a causally disconnected direction. 
  - We show that the direction $v_{MLP}^{rowspace}$ is significantly less activated by the feature we are patching than the direction $v_{MLP}^{nullspace}$ in Figure 20 in the Appendix. This shows that $v_{MLP}^{rowspace}$ is relatively dormant, compared to $v_{MLP}^{nullspace}$. 
  - In our revision of the paper, we have rewritten large parts of Subsection 4.3. to clarify the relationship of the experiments to our model of the illusion.
- For factual recall, we again decompose the directions found by DAS (which are in the post-gelu activations of MLP layers) in the same way. 
  - Again by definition, the nullspace components are causally disconnected.
  - To check how dormant the rowspace components are, we ran the experiment
  shown in Figure 12 in the paper, where we project the difference of
  activations of the two examples we are patching between on the causally
  disconnected component (the orthogonal projection on $\ker W_{out}$), and
  observe cosine similarity between this projection and the original direction
  of ~0.9; this implies that the cosine similarity with the rowspace component
  is ~0.4; thus, the rowspace component is relatively dormant compared to the
  nullspace one.
  - In our revision of the paper, we have included a note on this in the main body of the text.

Reviewer comment (Presentation):
The writing and presentation are sloppy. For example, in Section 4, “ker W_out” is never defined, and it is unclear how the results in Table 1 and Figure 3 relate to the description in section 4.3. It is unclear what is represented by ABB / BAB in Figure 3, and what “connected” in Table 1 refers to. Overall, Table 1, Figure 3, and Figure 4 do not clearly illustrate support for the presented hypothesis.

Author response:
> The writing and presentation are sloppy. For example, in Section 4, “ker W_out” is never defined, and it is unclear how the results in Table 1 and Figure 3 relate to the description in section 4.3. What is ABB / BAB in Figure 3? What does “connected” in Table 1 refer to? Overall, how do Table 1, Figure 3, and Figure 4 illustrate support for the presented hypothesis?

We have addressed these issues as well as others pointed out by the other reviewers in our updated draft. In particular, we rewrote Subsection 4.3 to fix various problems and better connect the writing to the table and figure. To answer the reviewer’s questions specifically:
- $\ker W_{out}$ is the kernel of the down-projection of the MLP layer where the
  illusory direction v_MLP is contained.
- The main text now contains detailed descriptions of all interventions involved, with short names matching those appearing in Table 1. 
- Figure 3 relates to the text in Section 4.3 through the paragraph now titled
“Patching $v_{\text{MLP}}$ activates a dormant pathway through the
MLP.” Specifically, Figure 3 illustrates how the subspace activation patch along
$v_{MLP}$ takes the output of the MLP layer along the gradient direction of the
name mover query matrices off-distribution.
- Here and elsewhere in the paper, “ABB” means prompts where the IO name comes
first, and “BAB” means prompts where the S name comes first. We thank the
reviewer for pointing out this omission. The notation is now explained in the
caption of Figure 3.

Overall, Table 1 supports our claim for the existence illusion by showing that
the subspace patch along $v_{MLP}$ has a significant effect on model outputs
(much stronger than the effect of patching the full MLP layer’s activations),
but this effect disappears when we restrict to the causally relevant component
of $v_{MLP}$, denoted $v_{MLP}^{rowspace}$ in our revision and introduced in the
“Methodology” part of Subsection 4.3.  Figure 3 further supports our model of
the illusion by showing how the MLP layer’s output on the causally relevant
direction $v_{grad}$ (given by the gradient of name-mover attention scores) is
taken significantly out of distribution by the patch along $v_{MLP}$. Finally,
Figure 4 (which is about factual recall) has similar message as Table 1: it
shows that subspace patching along the directions found by DAS is much more
effective at changing a model’s completion of a fact than patching the entire
MLP layer, or patching along only the causally relevant component.

Reviewer comment (Reproducibility):
The only “major” weakness is that code is not provided.

Author response:
We are working on releasing an anonymized version of our code in the next few
days. We will provide a link to it in a subsequent response once it is
available.

Reviewer comment (Experiments):
Why do the authors solely focus on subspace activation patching? The identified interpretability illusion should also hold for (automatic) component activation patching (i.e., consider the entire activation space as the (sub)space).

Author response:
This is a great question. A key property of subspace activation patching not
shared with full-component activation patching is that it **may create
out-of-distribution activations for the given component, even when patching only
between in-distribution examples**. This makes it particularly vulnerable to an
MLP-in-the-middle type of illusion, where we take the output of the MLP layer
off-distribution in order to write some causally important information in the
residual stream. Intuitively, this would not be possible to such an extent when
patching the full hidden activation of the MLP layer, because its activations
for in-distribution examples do not generally write important information to the
residual stream (as can be confirmed by doing the full patch). 

However, full-component activation patching can still take the activations of
the model as a whole off-distribution. It is an interesting question for future
work whether a variant of the illusion can be exhibited for full-component
patching. For example, if two heads always cancel each other out (one outputs +v
and the other -v), we could patch just one of them such that they no longer
cancel out. Thus, it may be possible to patch only one of these components and
throw the model off-distribution. However, we expect such scenarios of “perfect
cancellation” to be rarer in practice.

Reviewer comment (Writing):
There seems to be an error in the indices in the toy example in Appendix A.3.

Author response:
This is correct; the sentence “... the linear subspace of H_2 and H_3 defined by
the unit vector…” should be “...H_1 and H_2…” instead. We thank the reviewer for
pointing this out, and we have corrected it in our revision of the paper.

Reviewer comment (Presentation):
While Fig. 1 clearly demonstrates the interpretability illusion, it is hard to parse. It may be good to make it more accessible/easier to parse, as it demonstrates the main insight of the paper.

Author response:
We acknowledge that Figure 1 is somewhat involved and thank the reviewer for
pointing this out. To make it more readable, we have included a “decomposition”
of this figure into four figures to be read sequentially. Due to space
limitations, we include this new figure as Figure 26 in the Appendix, and we
point to it from the main text. We also re-did the original figure in TikZ to
improve readability. We hope that this makes the phenomenon clearer.

Reviewer comment (Writing):
It’d be good to add the relation of the vector $v$ to the subspace $U$ in Sec. 3.

Author response:
We thank the reviewer for pointing this out; the relationship is that, in the
special case of 1-dimensional subspace patching we consider, $U$ is the subspace
spanned by the (unit) vector $v$. We hope that clarifies this part of the paper;
we have added it to our revision.