Mechanistic Interpretability of Emotion Inference in Large Language Models (2025)

Abstract: Large language models (LLMs) show promising capabilities in predicting human emotions from text. However, the mechanisms through which these models process emotional stimuli remain largely unexplored. Our study addresses this gap by investigating how autoregressive LLMs infer emotions, showing that emotion representations are functionally localized to specific regions in the model. Our evaluation includes diverse model families and sizes, and is supported by robustness checks. We then show that the identified representations are psychologically plausible by drawing on cognitive appraisal theory—a well-established psychological framework positing that emotions emerge from evaluations (appraisals) of environmental stimuli. By causally intervening on construed appraisal concepts, we steer the generation and show that the outputs align with theoretical and intuitive expectations. This work highlights a novel way to causally intervene and control emotion inference, potentially benefiting safety and alignment in sensitive affective domains.

## introduction large language models (llms) demonstrate remarkable capabilities in emotion recognition and reasoning tasks, occasionally surpassing human performance @xcite . prior research primarily engages with llms as black boxes, utilizing zero-shot inference or in-context learning to gauge their performance on tasks such as emotion classification @xcite , emotional decision-making and situational appraisal @xcite , emotional intelligence @xcite , emotional dialogue understanding @xcite , and generation of emotional text @xcite . however, there remains a limited understanding of how ## experimental setup dataset and prompt design. we employ the crowd-envent dataset developed by @xcite , which comprises 6,800 emotional vignettes annotated with self-reported emotions among a list of 13 options and 23 self-rated appraisal variables, reflecting nuanced stimuli evaluations, including: pleasantness/unpleasantness, selfagency/other-agency, predictability/suddenness. appendix a.1 presents more details on the dataset, including a detailed list of appraisal variables, along with the scales used for measurement. to evaluate the model's ability to infer emotions from textual contexts, we design prompts that guide the model to predict the appropriate emotion as the next immediate output token, framing the task as a causal language modeling problem. subsequently, we consider a classification problem and evaluate the model by inspecting the logits confined to the set of targeted emotion labels. the primary prompt template used in this study is shown in figure 4 . emotion attribution is inherently subjective, making it challenging to define a single ground truth label for each input, particularly given our fine-grained list of emotions. thus, we focus on the correctly classified examples when inspecting each language model. in other words, we only analyze the samples for which the llm and the human annotator agreed on the same label, totaling at least 2,700 samples among different language models (see appendix a.3 for more details). this ensures that we assess tasks where the model performs reliably to understand the underlying mechanisms. ## notations and preliminaries prior research suggests that both mhsa and ffn units drive the generation in specific downstream tasks such as indirect object identification or concept promotion @xcite . by examining activations immediately after these units, we aim to evaluate their respective contributions to emotion processing within each transformer layer. more formally, let h (l) @xmath0, • • • , t }, where d is the dimensionality of the model's hidden representations and t is the input sequence length. then, ## probing for emotion signals building on the linear representation hypothesis @xcite @xcite , we perform probing experiments to assess the presence and strength of emotion-related signals at different activations within the model. specifically, we train linear classifiers (hewitt and liang, 2019) to predict the corresponding emotions. we formalize the linear classifiers as follows: we perform probing separately over different activation locations and layers across the model for the last token index. the probing results in figure 2 , measured as the accuracy on a held-out test set, indicate that the models begin consolidating emotional information in the hidden states h (l) neither too early nor too late, but predominantly around the mid-layers across all models. for example, in the first row corresponding to llama 3.2 1b in figure 2 , the emotional signal peaks by layer @xmath1. beyond layer 10, there is no significant increase in probe accuracy, suggesting that the model effectively captures emotional content by this stage. there is no clear distinction in probing performance between m (l) and h (l) . measurements from ffn closely track the hidden state dynamics, showing a steady increase in emotional conceptualization that peaks around the mid-layers. however, the heatmap corresponding to a (l) shows a more dispersed pattern while following the same consistent increasing trend observed in other locations. our experiments reveal that emotion-processing mechanisms in llms are most pronounced in the middle layers across model families and sizes. these findings suggest that the model has largely determined the output emotion by the mid-layers, with subsequent layers adding little additional processing. our observation aligns with the understanding that higher transformer layers capture more abstract and task-specific features. lastly, to evaluate the hypothesis regarding the importance of the last token in causal modeling, we repeat the analysis on the last five tokens for llama 3.2 1b in appendix c.3. we observe a consistent increase in signal strength from earlier to later tokens, reinforcing the focus on the last token as the primary contributor to output generation. ## emotion transfer by activation patching given evidence suggesting that the model's internal representation of emotional content stabilizes around the mid-layers, we explore causal intervention in these regions to test their functional importance. specifically, we assess whether the output emotion of a source example can be transferred to a target example, with a different emotion, by selectively patching activations from the source com- the goal is to determine whether substituting specific activations with those from another example can manipulate the model's final prediction to reflect the intended label. we conduct experiments by substituting activations at the last token and within a window spanning five layers, consistently across all model families and sizes foot_0 . the results shown in figure 3 demonstrate consistent behavior across model sizes and families. when intervening on hidden state activations, we observe a clear increase in intervention success (see figure 3 right). take llama 3.2 1b as an example. patching hidden states in the early layers is entirely ineffective. there is a critical point, e.g. layer 10 in this model, after which copying the hidden state transfers the emotion label with a high success rate. a high success rate at the final layers is naturally an expected behavior, as the residual stream of hidden states aggregates the information as it gets closer to the final layers. however, remarkably, the chance of success peaks and stabilizes around the mid-layers in all models, an observation that aligns with our findings in the previous section. to dig deeper, we look at the patching effect of mhsa and ffn units (see figure 3 left and middle, respectively), which provides clear evi-dence for functional localization of emotion processing. more precisely, interventions targeting a (l) and m (l) show clear evidence of success localized to specific mid-layers, e.g., mhsa units of layers @xmath2.2 1b. in other words, successful patching of both mhsa and ffn units occurs only in a subset of layers and predominantly in the middle rather than the final layers, with the ffn's most successful patchings happening only slightly later than the mhsa's patching. we hypothesize that there are a few consecutive layers in each language model whose mhsa units are responsible for gathering emotional information from the rest of the tokens and integrating it into the hidden state of the last token. this mechanism is immediately followed by a processing in the subsequent ffn units. to complement these findings, we perform an additional experiment, where a set of activations is knocked out in the forward pass to assess their impact. the results in appendix b consistently align with those of our activation patching and probing, reinforcing the evidence of functional localization in emotion processing. these observations hold across different models and experimental methods. our results in all previous sections are not prompt-dependent. specifically, changes in the format, wording, structure, or the number of demonstrations in the prompt do not affect the findings (see appendix c.5). additionally, we provide a control experiment in appendix c.4 to show that the identified units are not critical when performing a different yet syntactically similar task. in fact, the final layers are most critical for this isomorphic syntactic task, which differs significantly from the units we found for emotion processing. to further explain our findings, we analyze the attention patterns in llama 3.2 1b. specifically, we record the top 3 tokens attended to by all attention heads in the last token of each layer. we conduct this analysis for all samples in the dataset, provid- question template sample shot #1 sample shot #2 query what_are_the_inferred_emotions_in_the_following_contexts?_context:_my_first_child_was_born._answer:_joy _context:_my_dog_died_last_week._answer:_sadness_context:_i_got_an_a_in_my_exam_answer: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 layers ing insight into which tokens are most frequently attended to. figure 4 presents the aggregate attention patterns. the results indicate that early layers primarily focus on syntax. around the mid-layers, the model shifts its attention to emotionally relevant tokens in the query and maintains this focus until the final layers. in the last few layers, attention predominantly focuses on the last token in the sequence, suggesting that it primarily carries forward the last token's hidden state from earlier layers. these patterns provide evidence that the identified middle units are meaningfully related to emotion processing. ## investigating appraisal concepts we draw inspiration from cognitive appraisal theory to show an existing emotion structure in llms' latent representations. appraisals are known to have significant associations with emotions and, in some accounts, are considered causal factors in their emergence @xcite ). figure 5 illustrates three primary appraisal dimensions, including pleasantness, other-agency, and predictability, in addition to their associations with a set of basic emotions. for instance, guilt and pride are both materialized in situations with high self-agency, with the former happening in pleasant situations while the latter comes with an unpleasant experience. these mappings, which align closely with prior findings in appraisal theory @xcite , are extracted from our dataset by averaging appraisal scores for each emotion label. we begin by training linear probes for appraisal signals within the model representations. in contrast to the linear classification probes used in section 5, here we solve multiple independent regression tasks for each appraisal. more formally, consider a set of n appraisals and let v @xmath3, • • • , n}. let x be an activation vector, as introduced in section 4. we train the regression weight v a and the bias b @xmath4, figure 5 : appraisal emotion associations extracted from the dataset. for example, when participants report anger, they typically perceive a high degree of otheragency and a low level of pleasantness in the situation. ## emotion-appraisal mappings in this section, we analyze the representations of emotions and appraisals to reveal a structure within the latent llm representations. remember the weight matrix @xmath5. let w @xmath6, • • • , c}. we define the cosine similarity of appraisal a with emotion e as sim(a,e) = v ⊤ a we ∥va∥ 2 ∥we∥ 2 . figure 6 shows the similarity score of emotion vectors with two appraisal vectors, i.e. the pleasantness and other-agency, throughout llama 3.2 1b layers. notably, we observe psychologically plausible appraisal-emotion mappings across all layers. however, the projection strength peaks in the early layers and fades to near zero in the final layer, suggesting orthogonality in the final layers. we hypothesize that in the earlier layers, there exists a meaningful structure capturing appraisal and emotion concepts, which aligns with our expectations from the appraisal theory. however, these concepts gradually decouple as the processing progresses through the network, and by the final layers, they become orthogonal and fully decoupled, reflecting the specialization of the network toward higher-level tasks. we finish this section by drawing the connection to our findings in previous sections. notice that the decoupling starts around the critical layers, e.g. layers 9 and 10 in llama 3.2 1b, which we identified in previous sections. therefore, our observations suggest that the appraisals build a foundation to understand emotion representations in llm hidden states, but the structure vanishes as we progress through the network. ## intervention on appraisal concepts after finding the appraisal vectors, we investigate the possibility of indirectly modifying the emotion of an input example by modulating its appraisals within the model representations. for this purpose, we need to isolate the role of each appraisal a, by considering its associated latent vector v a and distinguishing it from other appraisal vectors. more precisely, we define v -a := [v 1 , . . . , v a-1 , v a+1 , . . . , v n ] by concatenating all appraisal vectors except v a . next, we introduce the unique effect vector of appraisal a as z a := (i -p -a )v a , where figure 7 illustrates concept modulation results with different magnitudes of β on layer 9 of llama 3.2 1b. notably, we observe a remarkable alignment with theoretical and intuitive expectations. for instance, increasing the pleasantness appraisal promotes both joy and pride, aligning with the fact that both of these emotions have high associations with the pleasantness appraisal. in contrast to these results, the appraisal modulation when applied to earlier layers, does not generate psychologically valid results and is totally ineffective when applied to later layers. this observation matches the intuitions on the mechanism we provided earlier; intervening on the early layers is not valid since modifications to latent representations are overwritten by emotion-specialized mid-layers. on the other hand, intervention on final layers is not effective because of the orthogonality of concepts as demonstrated in section 8. appendix f presents the full results, including intervention across all layers of llama 3.2 1b. appraisal theory is also predictive of the situations in which two appraisals are promoted simultaneously. to test this capability in llms, we perform an intervention on the superposition of two appraisal dimensions: other-agency and pleasantness, with mathematical details provided in appendix f. the results, depicted in figure 7 , show a successful promotion of emotion pride with no further occurrences of joy. these findings provide strong evidence that layer 9 in llama 3.2 1b directly contributes to cognitive processes related to emotions (see appendix f for experiments with other appraisal concepts). additionally, we provide complementary experiments such as direct emotion promotion using emotion vectors in appendix d. ## discussion we employed mechanistic interpretability techniques to investigate the inner workings of emotion inference in llms. our results reveal that mid-layer mhsa units within these models are responsible for processing emotional content. by applying linear algebraic manipulations to modulate the antecedents of emotions, i.e. the appraisal concepts, we steered the model outputs in controlled and predictable ways. this is particularly important for ensuring the reliability and steerability of llms in high-stakes affective domains such as legal decision-making and clinical therapy. a key distinction of our work is that we grounded our experiments on psychological theory and applied mi analysis on in-the-wild examples, rather than relying on synthetically generated simplistic structures, as seen in prior studies @xcite . for example, @xcite study the indirect object identification task, by considering a fixed input structure, such as "per-son1 and person2 had fun at school. person2 gave a ring to" where the model is expected to predict "person1". hanna et al. (2023) study the "greaterthan" task using a dataset of examples like "the war lasted from 1517 to 15", where the model is expected to predict any two-digit number larger than 17. while such notable efforts have demonstrated the feasibility of identifying end-to-end specialized circuits for these tasks, effectively interpreting them still requires structured inputs, making it challenging to generalize findings to more naturalistic settings. our work also highlights new opportunities for future research. despite significant advancements in understanding human emotions, debates persist regarding the definition of emotion, the role of cognition in emotion, and the mechanisms underlying emotion inference @xcite @xcite . in parallel, cognitive neuroscience has explored the neural basis of emotion in support of differing theoretical perspectives @xcite . the study of llms, combined with insights from emotion theory and neuroscience, opens a unique intersection for advancing our understanding of emotions @xcite . furthermore, our steering approach opens promising possibilities for conditioning llms to exhibit specific personality traits or moods, which could benefit applications requiring tailored affective responses @xcite @xcite @xcite . however, to ensure these interventions do not introduce unintended disruptions to other critical language-processing functions, it is essential to rigorously evaluate models on standard nlp benchmarks after inducing traits or moods. given llms' increasing societal impact -spanning areas such as mental health, legal decisionmaking, and human-ai interaction-it is imperative to deepen our understanding of their internal mechanisms. our study breaks new ground in the interpretability of emotion inference in llms, offering a novel way to causally intervene in emotional text generation. these findings hold promise for improving safety and alignment in sensitive affective domains. moving beyond black-box approaches to rigorously test and refine llm emotion processing will not only advance the field of llm interpretability but also unlock new pathways for more responsible ai systems. ## limitations in this study, we build upon the linear representation hypothesis @xcite @xcite -the idea that high-level concepts are encoded linearly within model representations @xcite . this hypothesis is particularly appealing because, if true, it could enable simple and effective methods for interpreting and controlling llms-an approach we leveraged to localize and manipulate latent emotion representations. however, despite notable efforts to formalize the notions of linearity @xcite and orthogonality @xcite in model representations, recent research suggests that not all features are encoded linearly @xcite . further investigation is needed to improve clarity and robustness in this area. furthermore, we demonstrated the ability to manipulate affective outputs by modifying appraisal concepts. nevertheless, the precise nature of this relationship remains unclear-it is possible that appraisals are merely correlated with emotions rather than exerting a direct causal influence or that the relationship follows an inverse causal pattern. establishing causality requires further investigation in future studies to disentangle directional depen-dencies. a deeper understanding of the interplay between llm emotional inference, emotion theory, and neuroscience will be crucial for both theoretical insights and practical applications. addressing these challenges will refine our understanding of llms and enhance their reliability in affective computing. ## ethical impact statement this study re-analyzes previously collected, deidentified data that had already undergone ethical review. the dataset is used for investigating the inner mechanisms by which auto-regressive llms process emotion. however, caution must be exercised when generalizing these results to models not examined in this work, to superficially similar tasks, or to different languages. our analysis highlights potential concerns for those deploying llms in high-stakes affective domains or for generating emotionally charged content. given the risks associated with emotional manipulation by llms, it is crucial to develop a deeper understanding of how these models process emotions. to this end, we advocate for further research in this domain to ensure that llms align with ethical standards and human-centered ai principles. phoebe c ellsworth and klaus r scherer. 2003. appraisal processes in emotion. oxford university press. zohar elyoseph, dorit hadar-shoval, kfir asraf, and maya lvovsky. 2023. chatgpt outperforms humans in emotional awareness evaluations. frontiers in psychology, 14:1199058. joshua engels, eric j michaud, isaac liao, wes gurnee, and max tegmark. 2025. not all language model features are linear. in the thirteenth international conference on learning representations. allyson ettinger, ahmed elgohary, and philip resnik. 2016. probing for semantic evidence of composition by means of simple classification tasks. in proceedings of the 1st workshop on evaluating vector-space representations for nlp, pages 134-139. jiahai feng, stuart russell, and jacob steinhardt. 2025. monitoring latent world states in language models with propositional probes. in the thirteenth international conference on learning representations. nico h frijda, peter kuipers, and elisabeth ter schure. 1989. relations among emotion, appraisal, and emotional action readiness. journal of personality and social psychology, 57(2):212. chris gagne and peter dayan. 2023. the inner sentiments of a thought. arxiv preprint arxiv:2307.01784. mor geva, avi caciularu, kevin wang, and yoav goldberg. 2022. transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 30-45, abu dhabi, united arab emirates. association for computational linguistics. asma ghandeharioun, avi caciularu, adam pearce, lucas dixon, and mor geva. 2024. patchscopes: a unifying framework for inspecting hidden representations of language models. in forty-first international conference on machine learning. mario giulianelli, jack harding, florian mohnert, dieuwke hupkes, and willem zuidema. 2018. under the hood: using diagnostic classifiers to investigate and improve how language models track agreement information. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 240-248, brussels, belgium. association for computational linguistics. aaron grattafiori, abhimanyu dubey, abhinav jauhri, abhinav pandey, et al. 2024. the llama 3 herd of models. abhijeet gupta, gemma boleda, marco baroni, and sebastian padó. 2015. distributional vectors encode referential attributes. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 12-21, lisbon, portugal. association for computational linguistics. michael hanna, ollie liu, and alexandre variengien. 2023. how does gpt-2 compute greater-than?: interpreting mathematical abilities in a pre-trained language model. in thirty-seventh conference on neural information processing systems. stefan heimersheim and neel nanda. 2024. how to use and interpret activation patching. arxiv preprint arxiv:2404.15255. john hewitt and percy liang. 2019. designing and interpreting probes with control tasks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 2733-2743, hong kong, china. association for computational linguistics. dieuwke hupkes and willem zuidema. 2018. visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. in proceedings of the twenty-seventh international joint conference on artificial intelligence, ijcai-18, pages 5617-5621. international joint conferences on artificial intelligence organization. guangyuan jiang, manjie xu, song-chun zhu, wenjuan han, chi zhang, and yixin zhu. 2023. evaluating and inducing personality in pre-trained language models. in neurips. hang jiang, xiajie zhang, xubo cao, cynthia breazeal, deb roy, and jad kabbara. 2024a. personallm: investigating the ability of large language models to express personality traits. in findings of the association for computational linguistics: naacl 2024, pages 3605-3627, mexico city, mexico. association for computational linguistics. yibo jiang, bryon aragam, and victor veitch. 2024b. uncovering meanings of embeddings via partial orthogonality. advances in neural information processing systems, 36. akos kádár, grzegorz chrupała, and afra alishahi. 2017. representation of linguistic form and function in recurrent neural networks. computational linguistics, 43(4):761-780. arne köhn. 2015. what's in an embedding? analyzing word embeddings through multilingual evaluation. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 2067-2073, lisbon, portugal. association for computational linguistics. philip a kragel, david sander, and kevin s labar. 2024. can brain data be used to arbitrate among emotion theories? in emotion theory: the routledge comprehensive guide, pages 511-542. routledge. sreejan kumar, theodore r sumers, takateru yamakoshi, ariel goldstein, uri hasson, kenneth a norman, thomas l griffiths, robert d hawkins, and samuel a nastase. 2024. shared functional specialization in transformer-based language models and the human brain. nature communications, 15(1):5523. nutchanon yongsatianchot, parisa ghanad torshizi, and stacy marsella. 2023. investigating large language models' perception of emotion using appraisal theory. in 2023 11th international conference on affective computing and intelligent interaction workshops and demos (aciiw), pages 1-8. ieee. weixiang zhao, yanyan zhao, xin lu, shilong wang, yanpeng tong, and bing qin. 2023. is chatgpt equipped with emotional dialogue capabilities? arxiv preprint arxiv:2304.09582.