Paper: Language Models Linearly Represent Sentiment (ICLR-2024)

Reviewer comment (Presentation):
There is no caption provided for Table 1 in the paper. Furthermore, Figure 2 is predominantly featured in Section 3, but there exists a substantial gap between its initial reference and the section in which it is discussed.

Author response:
Thank you very much for pointing out these slips in presentation. In the revised version we have ensured that all tables and figures are appropriately captioned and, where possible, placed in proximity to the relevant text for ease of reference and to facilitate a coherent narrative flow.

Reviewer comment (Experiments):
The proposed ToyMovieReview format, "I thought this movie was ADJECTIVE, I VERBed it. Conclusion: This movie is," appears to be more suitable for sentence-level sentiment analysis. It may not be as effective for other sentiment analysis tasks, such as document-level sentiment analysis or fine-grained sentiment analysis. Furthermore, the rationale for emphasizing adjectives and verbs in this specific prompt design remains unclear, as it has not been adequately explained in previous sections of the paper.

Author response:
Your point about the ToyMovieReview dataset is well-taken. To address this, we provide the additional experiment of replicating the SST all-token ablation experiment on the IMDB reviews dataset and have added the results to Section 3.3. These prompts are significantly longer and thus the results better demonstrate our findings at the document level.

We realize that our choice to focus on adjectives and verbs in the ToyMovieReview dataset was not fully elucidated. This choice was based on their simplicity and their efficacy as a sentiment-laden source of data. To further clarify, we have included a reference to a cosine similarity analysis with a toy adverb dataset in the appendix, which supports our choice.

Reviewer comment (Presentation):
The writing seems extremely sloppy. For an experimental paper, many of the plots/discussions are not clear, especially for a broad ICLR audience. For example, what are the x- and y-axes of figure 2b?

Author response:
Regarding the writing clarity, we regret any confusion and have extensively improved our paper’s clarity and readability in the latest revision. We have conducted a thorough review and made many small improvements, shown in red font for your convenience. We have brought key definitions forwards from the glossary into the main body and improved figure explanations, particularly for Figure 2b. We would also like to draw your attention to the helpful glossary in Appendix A.7.

Reviewer comment (Theory):
I feel like the authors are using the term "causal" loosely, do they mean that the sentiment is sensitive to activation addition, and hence they deem it causal? Usually the term causal representations are reserved for representations that generate the data, not the other way around.

Author response:
By “causally significant,” we do not mean that the sentiment direction is sensitive to activation addition, but rather that causal mediation (i.e. activation patching) experiments demonstrate the causal effect that these directions exert on the models’ outputs in real-world contexts. We would like to draw your attention to methods section 2.3 Causal Interventions, where we outline exactly two causal methods: “Activation Patching” and “Ablations”. Then in results section 3.3 Causal Evaluation, the first sentence reads “We evaluate the sentiment direction using directional patching”. In the figure captions for our causal results, we preface 2d with “Directional patching results” and 2e with “Patching results”. From the opening to Section 4: “Through an iterative process of path patching (see Section 2.3)”. In the reproducibility statement: “Our causal analysis techniques of activation patching, ablation, and directional patching are presented in Section 2.3”. 

The activation addition results were not central to our claims and thus were relegated to the Appendix in Figure A.3. We regret that these may have seemed like our basis for using the term “causal”. To avoid any further confusion, in our latest revision we have removed the single reference to activation addition from the main body of the paper. We appreciate your attention to this detail and are open to any further suggestions or questions you might have.

Reviewer comment (Evaluation):
As the authors note, attention heads are used for the analysis and a more thorough analysis should include activations at MLP layers too.

Author response:
Thank you for raising this point. We note that our core results focus on studying directions in the model’s residual stream (i.e. in the skip connections around layers), which are the sum of the outputs of *all* prior layers, including both attention and MLP layers, and thus includes the contributions of the MLP layers.. 

We also provide more detailed circuit analysis that we agree would benefit from a broader examination that includes MLP layers. However, this is not central to our core contribution in this paper, and as such, though we have provided initial findings, we acknowledge this as a limitation and an avenue for future work. In addition, in our path patching experiments we observed that the MLPs were of relatively low importance to the models’ performance on the tasks we analyzed, a finding that parallels that of past circuit analyses such as that in Wang et al. ([https://arxiv.org/abs/2211.00593](https://arxiv.org/abs/2211.00593)) Since attention heads had a more significant effect on model performance, we focused on them in our circuit-level analyses.

Reviewer comment (Experiments):
The authors claimed that the results were consistent across a range of models. However, I found mostly GPT2-small or Pythia in the experiment results -- one model for each, not repeated. While this is already an interesting finding, it has weak support for the claim. The author should either clarify this or lower the claim.

Author response:
> Models. The authors claimed that the results were consistent across a range of models… author should either clarify this or lower the claim.

Where possible, we have tried to run the same experiments across different models. In particular, the core experiments of patching the sentiment direction in different layers when evaluating on our real-world dataset (SST) were performed for 4 different sizes of GPT-2 and 4 different sizes of Pythia. We included a representative selection of these in Figure 2e. Some tasks could not be performed by smaller models (such as the mood stories) and other experiments were not performed on larger models due to compute time constraints (such as evaluating Pythia-2.8b on SST with patching at each layer). 

To add support to our claim of universality, we have repeated the experiments in Figures 2a and 2b for Pythia-1.4b, in order to provide a comparison with GPT2-small. These show that in both models, the different methods converge on a single direction which can classify sentiment on OpenWebText.  In addition, to provide additional support for our “summarization motif” results, we have added activation patching experiments for Pythia 6.9b to Appendix A.4.2, and are currently working on adding SST direction ablation experiments for other models to a future revision.

Reviewer comment (Evaluation):
The authors claimed that the sentiment was linearly represented. Still, the evidence was rather weak on the real dataset (SST), especially the vectors discovered by simple linear methods like PCA or logistic regression. The effect was more prominent than the random directions, but it did not support the linearity claim. While the authors explained that the direction was discovered from a small dataset, we could also explain that there was more than one direction or simply nonlinear sentiment space.

Author response:
> Linearity. The authors claimed that the sentiment was linearly represented… we could also explain that there was more than one direction or simply nonlinear sentiment space.

Thank you for pointing this out; we agree that this claim could benefit from further clarification and we have added a section to the Appendix (A.2) in the revised version, where we include some text expressing similar limitations to the above for our linearity claim.

Our core contribution lies in identifying a key sentiment direction that is causally influential and consistent across various tasks, as evidenced by our directional patching experiment's ability to leverage a sentiment direction identified from a toy dataset to effectively classify sentiment in more complex datasets like OpenWebText and Stanford Sentiment Treebank. While we acknowledge the potential for additional sentiment representations tailored to specific tasks or non-linear dimensions, our findings suggest the dominance of a primary, shared sentiment direction.

We propose that this primary direction acts as a "first principal component of sentiment," and that there could be further more nuanced, task-specific directions. This concept echoes the "feature splitting" phenomenon described by Bricken et al. in “Towards Monosemanticity.” Although exploring these finer-grained directions is outside our current scope, we believe pinpointing this overarching sentiment direction is nevertheless a meaningful contribution to the literature.

We would also like to draw your attention to Appendix Figure A.6, where we show that allowing DAS to learn a higher dimensional subspace does not improve the loss out of distribution. This supports the linear perspective, showing that it is hard to beat even by increasing the dimension of the DAS subspace when seeking a representation of sentiment across a sufficiently wide range of tasks.

Reviewer comment (Experiments):
While the author experimented with a toy dataset and injected irrelevant content, I found the relationship between the "summarization" and the context length interesting. However, this could be made stronger -- the authors could find sentences in SST with varied lengths and commas to support the relationship.

Author response:
> Distance. While the author experimented with a toy dataset and injected irrelevant content, I found the relationship between the "summarization" and the context length interesting. However, this could be made stronger…

Thank you for your interest in our result showing increasing summarization with distance. We would also love to see further study of this result, but we have struggled with a number of confounders in real-world datasets. As this is a very interesting phenomenon but not a central one to our core claims in this paper, we decided to merely flag the finding here and leave these complexities for future researchers.

Reviewer comment (Evaluation):
In the abstract, the authors wrote 76% and 36%, but these were not consistent with the text in Section 4.3.

Author response:
> Inconsistency. In the abstract, the authors wrote 76% and 36%, but these were not consistent with the text in Section 4.3.

Thank you for raising this. In hindsight, we agree that reporting the same result with two different denominators was confusing. We reported the same result in terms of the 76% reduction in accuracy above “chance” (i.e. 50% for a balanced binary dataset) and in the body we reported this in terms of a drop of the 38% drop in raw accuracy (i.e. above 0%), resulting in a factor of two discrepancy. The same was true for the 36% and 18% accuracy drops reported with comma-only ablation. We have since revised our numbers reported in the abstract to only use the more conservative framing

Reviewer comment (Evaluation):
Can you clarify why the results were consistent across different models?

Author response:
> Models. The authors claimed that the results were consistent across a range of models… author should either clarify this or lower the claim.

Where possible, we have tried to run the same experiments across different models. In particular, the core experiments of patching the sentiment direction in different layers when evaluating on our real-world dataset (SST) were performed for 4 different sizes of GPT-2 and 4 different sizes of Pythia. We included a representative selection of these in Figure 2e. Some tasks could not be performed by smaller models (such as the mood stories) and other experiments were not performed on larger models due to compute time constraints (such as evaluating Pythia-2.8b on SST with patching at each layer). 

To add support to our claim of universality, we have repeated the experiments in Figures 2a and 2b for Pythia-1.4b, in order to provide a comparison with GPT2-small. These show that in both models, the different methods converge on a single direction which can classify sentiment on OpenWebText.  In addition, to provide additional support for our “summarization motif” results, we have added activation patching experiments for Pythia 6.9b to Appendix A.4.2, and are currently working on adding SST direction ablation experiments for other models to a future revision.

Reviewer comment (Writing):
Can you explain why the logit flip in the abstract is doubled from the text?

Author response:
> Inconsistency. In the abstract, the authors wrote 76% and 36%, but these were not consistent with the text in Section 4.3.

Thank you for raising this. In hindsight, we agree that reporting the same result with two different denominators was confusing. We reported the same result in terms of the 76% reduction in accuracy above “chance” (i.e. 50% for a balanced binary dataset) and in the body we reported this in terms of a drop of the 38% drop in raw accuracy (i.e. above 0%), resulting in a factor of two discrepancy. The same was true for the 36% and 18% accuracy drops reported with comma-only ablation. We have since revised our numbers reported in the abstract to only use the more conservative framing