Language Models Linearly Represent Sentiment (2024)

Abstract: Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs). In this study, we reveal that across a range of models, sentiment is represented linearly: a single direction in activation space mostly captures the feature across a range of tasks with one extreme for positive and the other for negative. In a causal analysis, we isolate this direction using interventions and show it is causal in both toy tasks and real world datasets such as Stanford Sentiment Treebank. We analyze the mechanisms that involve this direction and discover a phenomenon which we term the summarization motif: sentiment is not just represented on valenced words, but is also summarized at intermediate positions without inherent sentiment, such as punctuation and names. We show that in SST classification, ablating the sentiment direction across all tokens results in a drop in accuracy from 100% to 62% (vs. 50% random baseline), while ablating the summarized sentiment direction at comma positions alone produces close to half this result (reducing accuracy to 82%).

## introduction large language models (llms) have displayed increasingly impressive capabilities @xcite @xcite , but their internal workings remain poorly understood. nevertheless, recent evidence @xcite has suggested that llms are capable of forming models of the world, i.e., inferring hidden variables of the data generation process rather than simply modeling surface word co-occurrence statistics. there is significant interest @xcite , @xcite ) in deciphering the latent structure of such representations. in this work, we investigate how llms represent sentiment, a variable in the data generation process that is relevant and interesting across a wide variety of language tasks @xcite . approaching our investigations through the frame of causal mediation analysis @xcite , we show that these sentiment features are represented linearly by the models, are causally significant, and are utilized by human-interpretable circuits @xcite . we find the existence of a single direction scientifically interesting as further evidence for the linear representation hypothesis @xcite @xcite , that models tend to extract properties of the input and internally represent them as directions in activation space. understanding the structure of internal representations is crucial to begin to decode them. linear representations are particularly amenable to detailed reverse-engineering @xcite and have seen recent interest in the context of sparse autoencoders @xcite . we believe that interpreting internal representations in llms shows promise for mitigating problematic behaviours. we show evidence of a phenomenon which we have labeled the "summarization motif" foot_0 , where rather than sentiment being directly moved from valenced tokens to the final token, it is first aggregated on intermediate summarization tokens without inherent valence such as commas, periods and particular nouns. this can be seen as a naturally emerging analogue to the explicit classification token in bert-like models @xcite , and in that context the phenomenon was observed by @xcite . we show that the sentiment stored on summarization tokens is causally relevant for the final prediction. we find this an intriguing example of an "information bottleneck", where the data generation process is funnelled through a small subset of tokens used as information stores. understanding the existence and location of information bottlenecks is a key first step to deciphering world models. this finding additionally suggests the models' ability to create summaries at various levels of abstraction, in this case at a sentence or clause rather than a token. our contributions are as follows. in section 3, we demonstrate that standard methods can find a linear representation of sentiment using a toy dataset, and show that this direction correlates with sentiment information in the wild. we use causal analysis to show that this linear representation matters causally in both toy and crowdsourced datasets. in section 4, we show through activation patching @xcite and ablations (techniques defined in section 2.2) that the learned sentiment direction is used in summarization behavior that is causally important to circuits performing sentiment tasks. we replicate these findings across gpt2, pythia, gemma, qwen and stablelm models (section 2.1). in sum, we provide a novel, detailed case study of how to analyse a feature's representation in activation space. ## finding a "sentiment direction" the first question we investigate is whether there exists a direction in the residual stream in a transformer model that represents the sentiment of the input text, as a special case of the linear representation hypothesis @xcite @xcite , that features are represented linearly as directions in activation space. we show that the methods discussed above (e.g. k-means, lr and das, see section 2.4) all arrive at a similar sentiment direction. we can visualize the feature being represented by this direction by projecting the residual stream at a given token/layer onto it, using some text from the training distribution. we will call this the "sentiment activation". finding and comparing the directions to find initial directions corresponding with sentiment, we first fit directions from the residual stream over the adjective token in the toymoviereview dataset (section 2.1), using methods from section 2.4. we find extremely high cosine similarity @xcite between the directions yielded by each of these methods in pythia-2.8b (cf. a.7 for other models). this suggests that these are all noisy approximations of the same direction, and indeed our results appear robust to choice of fitting method. ## the summarization motif for sentiment though we do not focus on circuit foot_2 analysis here, we note that initial patching experiments in the style of @xcite revealed patterns which motivated our definition of the "summarization motif": when there is information (e.g. sentiment) stored at certain 'placeholder' or 'summary' tokens (e.g. commas, periods and certain nouns) despite these tokens not inherently having the information. moreover, this information is causally significant for the model to complete a certain task (e.g. sentiment classification). we provide a detailed circuitbased analysis of this phenomenon in appendix a.8. in this section, we focus on verifying this behaviour in pythia-2.8b, and we replicate for other models in the appendix ( ## conclusion the two central novel findings of this research are the existence of a linear representation of sentiment and the use of summarization to store sentiment information. we have seen that the sentiment direction is causal and central to the circuitry of sentiment processing. remarkably, this direction is so stark in the residual stream space that it can be found even with the most basic methods and on a tiny toy dataset, yet generalize to diverse real-world datasets. summarization is a motif present in larger models with longer context lengths and greater proficiency in zero-shot classification. these summaries present a tantalising glimpse into the worldmodelling behavior of transformers.