Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion (2024)

Abstract: Situations and events evoke emotions in humans, but to what extent do they inform the prediction of emotion detection models? This work investigates how well human-annotated emotion triggers correlate with features that models deemed salient in their prediction of emotions. First, we introduce a novel dataset EmoTrigger, consisting of 900 social media posts sourced from three different datasets; these were annotated by experts for emotion triggers with high agreement. Using EmoTrigger, we evaluate the ability of large language models (LLMs) to identify emotion triggers, and conduct a comparative analysis of the features considered important for these tasks between LLMs and fine-tuned models. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection.

## introduction understanding perceived emotions and how they are expressed can be immensely useful for providing emotional support, sharing joyful situations, or in a therapy session, thus emotion detection has become a well-studied task @xcite @xcite @xcite @xcite @xcite @xcite . however, though existing work has sought to identify what triggers or causes a particular emotion @xcite @xcite @xcite , the relationship between those triggers and the prediction of emotion detection models is little understood. this relationship is crucial to investigate, without which the interpretation of perceived emotions-or claims for a model to be able to do so-is hollow @xcite . while humans can intuitively construe emotional reactions with events that trigger them @xcite , it is unclear to what extent, if any, current nlp models are doing so. prior work trained models to learn to recognize and summarize emotion triggers or causes. in this work, we instead ask the question: what roles do emotion triggers play in emotion prediction? in addition to fine-tuned transformer models shown to be performant on this task, we additionally put an emphasis on large language models (llms) including both api-based and open-sourced ones, since their capability for trigger prediction has not been explored. to ground our analysis, we present emotrig-ger, a linguist-annotated dataset of emotion triggers (as extractive text spans), over three social media corpora with labeled emotions across different themes: canceremo @xcite , hurricaneemo @xcite , and goemotions @xcite . this is, to the best of our knowledge, the first dataset annotated with high-quality triggers focusing on short social media texts. engaging with tools that attribute model prediction @xcite and prompts that elicit natural language explanations from llms, we aim to answer the following research questions: we release emotrigger at https://github. com/smritisingh26/emotrigger . ## the emotrigger dataset we first present emotrigger, a dataset annotated with ground-truth emotion triggers. emotrigger consists of subsets from three well-known datasets commonly used in emotion prediction, on top of which we engage expert annotators to highlight triggers for the annotated emotion. although prior emotion cause datasets exist, they largely focus on explicit emotions @xcite . @xcite 's dataset for reddit posts involves multi-sentence triggers for summarization, while this work focuses on word or phrase level attributions and explanations. source datasets (1) hurricaneemo @xcite consists of tweets related to 3 hurricanes. each tweet is annotated for 24 fine-grained emotions in plutchik's wheel of emotions @xcite . we map the emotions onto plutchik's 8 basic emotions (appendix a). to balance the emotions in emotrigger, we sample 20 examples per emotion from each constituent dataset. in total, we have 900 samples in our dataset, 160 each from hurricaneemo and canceremo, and 580 samples from goemotions. we make sure our samples do not contain links or images, i.e., the text stands alone from other media. ## study design we study to what extent words most informative to the final model predictions constitute triggers (as annotated in emotrigger) of the emotion they are predicting. this section describes our strategy to do so for each model type. llms we experimented with gpt-4, llama2-chat-13b @xcite , and alpaca-13b @xcite . our main llm prompt asks the model to predict emotions present in a given piece of text, and elicit explanations as words most important for detecting those emotions; annotated triggers were used as "salient features". due to the poor zero-shot performance (appendix example 1. unless he has actually threatened to or used them against you i think you're overreacting @xcite . ask him to please delete them @xcite . ## results for binary token-level comparison of annotated triggers vs. words the models identified as important for emotion detection (henceforth "salient features"), we look at: (1) word-level macro-averaged f1 comparing salient features against the union of annotator-highlighted triggers. are llms capable of detecting emotions and identifying their triggers? we first show each models' performance on the emotion prediction task across all datasets in the first portion of as seen in while there is no emotion for which any model can consistently identify the triggers most accurately, it is worth noting that for gpt-4 and llama2chat, the lowest scores are corresponding to the identification of triggers for the emotion 'anticipation'. this is reflected in tables 8 , 9 , and 10 in the appendix. to what extent do emotion prediction models rely on features that reflect emotion triggers? the bottom portion of how often do the triggers overlap with keyphrases or emotion words? we further hypothesize that keyphrases in the dataset, as well as explicit emotion words, might align with salient features that models pick up. surprisingly, the correlations with keyphrases are much higher than with emolex for all models and across datasets. this indicates that models rely on explicit emotion words to a lesser extent than expected, and keyphrases help characterize this discrepancy. this is especially true for fine-tuned model (emoberta) on themed datasets (hurricaneemo, canceremo): the shap values are much more correlated with emolex especially in goemotions. we find that emotions like anger, joy and sadness are expressed more through emolex words, whereas emotions like anticipation, fear and disgust are expressed more through keyphrases. this is reflected in figure 5 . we observe that gpt4, llama2 and alpaca rely on keyphrases an average of 27.3%, 35.9% and 36.3% more than emolex words respectively, whereas emoberta relies on keyphrases an average of 40.8% more than emolex words. this is also reflected in ## qualitative analysis in this section, we provide details about what we observe in our analysis, in terms of what large language models get right, and what they get wrong. we also delve deeper into the comparison of the results of the experiments with llms and fine-tuned models. ## discussion and conclusion in this paper, we present emotrigger, a linguistannotated dataset of emotion triggers (as extractive text spans), over three social media corpora with labeled emotions across different themes. we use this dataset to analyze what role emotion triggers play in emotion detection. overall, we believe this work provides evidence that with the exception of very large models like gpt-4 (few-shot), open-sourced ones like llama2chat and alpaca do not have a good understanding of what triggers an emotion. the finding that salient features correlate substantially with keyphrases, rather than emotion triggers, means that models are better at picking up corpus-level topical cues rather than possessing a deep understanding of emotions per se as humans do. in psychology, emotion is viewed as triggered by subjective evaluations (or appraisals) of particular events @xcite ; thus future work on more sophisticated emotional support open-source language models should address this flaw.