Paper: The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (ICLR-2024)

Reviewer comment (Evaluation):
The paper has so many experiments and results that it could benefit from a richer discussion of how to interpret all of the results.

Author response:
We agree that this paper is packed with experiments. We have added some exposition to the experiment sections to more fully explain how we interpret the results.

Reviewer comment (Experiments):
Only tests one model, so it's unclear how generalizable the findings are (the authors, of course, acknowledge this).

Author response:
We have now added confirmatory results on LLaMA-2-13B! Please see the global comment.

Reviewer comment (Presentation):
I would have also appreciated a bit more detail about the datasets and how they were generated to be in the body of the paper instead of appendices.

Author response:
Thank you for this feedback! We’d be happy to add more detail on the datasets. Is there anything in particular you would find it useful to know about the datasets which you would like for us to discuss in the body of the paper?

Reviewer comment (Experiments):
Page 8, $\ell = 10$ for the intervention experiments: why that layer (especially since layer 12 was used earlier)?

Author response:
Good question! Intuitively, one should want to perform an intervention on a representation of truth as soon as it begins to form; in contrast, one wants to probe a concept of truth later in the model, after the representation has finished forming. This is why we performed our intervention in an earlier layer than we did our probing experiments, with the earlier layer originally chosen in an ad-hoc way. 
But we’re happy to say that we now have a more principled way to select the layers at which to probe and intervene! For the full details, please see the global comment about the newly added patching experiment. In short, we swap certain hidden states from a true input into the forward pass from a false input and measure which hidden states have a large effect on the model outputting “TRUE” vs. “FALSE.” This reveals that the relevant information seems to be encoded above two tokens – the final token of the statement and the period token – in a range of layers. For probing, we choose the latest such layer, since that’s the layer at which most truth-relevant information is present. In our modified and simplified causal intervention experiments, we intervene at all hidden states in this range.

Reviewer comment (Theory):
Is there a natural way of generalizing mass-mean probing to multi-class tasks beyond binary ones?

Author response:
Great question, and indeed there is! Suppose X is a random vector and f(X) is some labeling function feature (not necessarily binary). Consider the direction theta such that the orthogonal projection proj_theta away from theta makes it as hard as possible for a linear probe to predict f(X) from proj_theta(X). Then, in the case that f(X) is binary label, theta coincides with the mass-mean direction! (The proof is theorem 3.1 here https://arxiv.org/pdf/2306.03819.pdf.) But even if f(X) is continuous, the definition makes sense, and can be computed as in the linked paper. This would give a reasonable generalization of mass-mean probing. We are saving this idea for future work.

Reviewer comment (Experiments):
Table 2: the $\alpha$ values seem extremely large here, to the point where the resulting vector is almost going to look just like the truth vector. How do you think about these large values? (Does it make sense to do a baseline where the model just sees $\alpha\theta$?)

Author response:
In the new revision, we have considerably improved and simplified this experiment, so that it’s now easier to interpret the results. 

But if you’re curious, we’ll also address the question about the original experiment. We included this alpha for two conceptually distinct reasons: (1) because the truth value of the statement might be redundantly encoded in layers other than the one we intervened in, requiring an outsized intervention; and (2) because the truth directions identified by probes might be only partially aligned with the genuine truth direction, which would be reflected in requiring a larger alpha to be causally effective. So when alpha values were large, we were interpreting this as meaning that the probe direction was partially aligned with the truth direction but mostly pointing into an unmeaningful part of the model’s activation space.

Reviewer comment (Novelty):
The majority of the conclusions and claims are not unique. For example, from Li et al. and Burns et al. (which the paper cites), we already knew that such truth representation is linearly separable and one can apply casual intervention to such representation.

Author response:
> The majority of the conclusions and claims are not unique. For example, from Li et al. and Burns et al. (which the paper cites), we already knew that such truth representation is linearly separable and one can apply casual intervention to such representation.

Our paper certainly has significant overlap with the work of Burns et al. and Li et al.! To address this point in more detail, we’ve posted a self-contained comment discussing the state of the prior work and the novelty of our contribution.

## Rebuttal 2 (ID: ren5D0uDdw)
This comment discusses in more detail the state of the knowledge on LLM truth representations prior to this paper and the novelty of our contribution.

As background, we’ll say that this paper was motivated by wide-spread skepticism about the results of Burns et al. and Li et al. among LM interpretability researchers we’ve spoken with. We didn’t feel it was appropriate to go into detail about the controversy in the paper’s main body, but let us summarize some key points here:

* **The experimental results in Burns et al. are not actually very strong.** While the Burns et al. paper is commendable for introducing a novel technique and perspective, many observers have pointed out that the evidential basis for its claims is weak (see for instance the comments from its review process: https://openreview.net/forum?id=ETKGuby0hcs). Indeed, the results in the main body appear to be cherry-picked, with many LM interpretability researchers citing the full results table (figure 9 in the appendix of Burns et al.) as strong evidence against the paper’s central claims! See also the critique paper by Levenstein & Hermann [1].

* **The setting of Li et al. is not clearly about truth.** Li et al. use the TruthfulQA dataset, consisting of questions which are tricky or misleading in some way (e.g. “During the Middle Ages, what did scholars think the shape of the Earth was?”). For around 15% of the data, the “truthful” response is to not answer at all: “I have no comment.” Given the small model size (Li et al. work with LLaMA-7B) and the ambiguity of the questions, it is unclear whether the directions identified really correspond to truth. Indeed, the interventions of Li et al. also increase the probability of the model outputting “I have no comment.”

* **Burns et al. studies OOD classification accuracy and Li et al. studies IID causal efficacy, but these are both insufficient.** The core concern here is that probes may be picking up on non-truth features which correlate with truth. A probe which attends to all truth-correlated features on a distribution may well be causally implicated in model outputs on that same distribution. But showing that a probe trained to predict truth on one distribution can be used to causally intervene on another distribution is significantly stronger evidence, and has not yet been demonstrated in the literature.

Given the importance of this topic and the controversy surrounding prior work, we felt it would be a significant contribution to treat the question of LLM truth representations with the rigor it deserved. To this end we:

* **Curated datasets which precisely captured a well-scoped notion of truth**, unlike the long and complicated inputs used in Burns et al. and the intentionally misleading Q/A pairs used in Li et al.

* **Worked with models capable enough, and statements simple enough, that we can expect the LLMs we study to actually understand their inputs.**

* **Verified that the directions we identify are causally implicated OOD.** Note that even though the different probing techniques are comparable for OOD classification, they vary wildly in OOD causal efficacy, indicating that this previously-unstudied question is important for evaluating the validity of truth directions!

Finally, let us reiterate some additional contributions this paper makes which are not present in previous literature:

* **Localizing representations of truth to certain model components**. Especially with the improved and simplified experiments in section 5 (see the global comment), we localize LLMs’ representation of truth to the residual stream in certain layers above certain token positions. (In contrast, Li et al. add their probe directions into dozens of attention heads over every token during generation.) In particular, this reveals an interesting “summarization” behavior, where high-level information about a clause (e.g. its truth value) is stored above the token which marks the end of that clause; this phenomenon was also noted in the concurrent work of Tigges et al. [2]. 

* **Studying in detail cases of generalization failure.** Levenstein & Hermann [1] noted that probes trained on some datasets failed to generalize to negations, but incorrectly concluded that this shows LLMs do not represent truth. We dig into this phenomenon in significantly more detail.

* **Studying mass-mean probing and its properties in detail**, including providing a theoretical argument motivating its use. It was this theoretical argument which led us to discover the IID modification of mass-mean probing which significantly improves IID accuracy.

We have made various updates to relevant sections of the text to clarify these contributions.

[1] B. A. Levinstein and Daniel A. Herrmann. Still no lie detector for language models: Probing empirical and conceptual roadblocks, 2023.

[2] Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023.

Reviewer comment (Experiments):
Lack of model variances. The experiments are only conducted on LLaMA-13B. As a result, we don't really know the effects of scale or the effects of the pretraining paradigm on such representations.

Author response:
> Lack of model variances. The experiments are only conducted on LLaMA-13B. As a result, we don't really know the effects of scale or the effects of the pretraining paradigm on such representations.

We have now added results for LLaMA-2-13B! Please see the global comment.

Reviewer comment (Evaluation):
Mass-Mean probe doesn't really improve generalization accuracy over some of the other methods that much.

Author response:
> Mass-Mean probe doesn't really improve generalization accuracy over some of the other methods that much.

We agree with this, though we’ll make some remarks here which put these probing results into context. As revealed by our PCA visualizations, the true and false point clusters are rather far apart in their high-dimensional space. In such a setting, it is actually very easy for a linear probe to have seemingly impressive accuracy: it only needs to have a non-negligible cosine similarity with the genuine feature direction and be oriented in the correct direction. And since there is a diminishing density of points as one moves away from the cluster centers towards the decision boundary, there are also diminishing accuracy gains to better aligning with the genuine feature direction. Thus, it is extremely easy to go from 50% to 75% accuracy (in fact, Roger [1] shows that random directions with only a choice of orientation can often do it!) but harder to go from 80% to 85% accuracy.

In contrast, we aren’t aware of analogous subtleties for interpreting causal intervention experiments, so we think they give a much clearer picture of which probing methods are best identifying LLMs’ truth directions.

[1] Fabien Roger. What Discovering Latent Knowledge Did and Did Not Find, 2023.

Reviewer comment (Presentation):
The figure on page 17 is disproportionate.

Author response:
> The figure on page 17 is disproportionate

Thank you, we have fixed this.