Unsupervised Extractive Summarization of Emotion Triggers (2023)

Abstract: Understanding what leads to emotions during large-scale crises is important as it can provide groundings for expressed emotions and subsequently improve the understanding of ongoing disasters. Recent approaches trained supervised models to both detect emotions and explain emotion triggers (events and appraisals) via abstractive summarization. However, obtaining timely and qualitative abstractive summaries is expensive and extremely time-consuming, requiring highly-trained expert annotators. In time-sensitive, high-stake contexts, this can block necessary responses. We instead pursue unsupervised systems that extract triggers from text. First, we introduce CovidET-EXT, augmenting (Zhan et al., 2022)’s abstractive dataset (in the context of the COVID-19 crisis) with extractive triggers. Second, we develop new unsupervised learning models that can jointly detect emotions and summarize their triggers. Our best approach, entitled Emotion-Aware Pagerank, incorporates emotion information from external sources combined with a language understanding module, and outperforms strong baselines. We release our data and code at https://github.com/tsosea2/CovidET-EXT .

## introduction language plays a central role in social, clinical, and cognitive psychology @xcite , and social media presents a gold mine for such analysis: people turn to social media to share experiences around challenges in their personal lives and seek diagnosis, treatment, and emotional support for their conditions @xcite . during crises, such as natural disasters or global pandemics, large-scale analysis of language on social media -both how people feel and what's going on in their lives to lead to these feelings -can have a profound impact on improving mental health solutions as well emotion triggers in a reddit post 1. it finally happened. 2. took an older relative for her first pfizer dose. 3. not that many people showed up so all accompanying family members were offered the shot and give papers for a second dose. 4. i wasnt due to get my shot for the next couple of months and have had some scares, many for which ive gotten support from you lovely awesome people. 5. i now have a whole different perspective on my governments organization. 6. they have a good oiled vaccination machine. 7. they just need more doses. 8. best part is i was told i could come back for my second dose whenever my relative was scheduled to get hers. 9. i have a lil arm pain. 10. but its the slightest arm pain when moving it past certain angles. 11. ive noticed that a lil blood drop shows on my vaccination spot @xcite shower half an hour later). 12. is this normal? trust fear joy figure 1 : an example post from covidet-ext annotated with emotion triggers. the highlighted sentences represent triggers of the tagged emotions. recent work @xcite taps into this broad challenge by jointly detecting emotions and generating a natural language description about what triggers them (triggers include both objective events and subjective appraisals of those events @xcite ). trigger explanation is formulated as a supervised, abstractive summarization task that is emotion-specific. unlike generic summarization however, due to the high cognitive load to provide judgments for each emotion, obtaining humanwritten summaries for this task is time-consuming and requires significant annotator training. this results in small, domain-specific datasets that are difficult to scale -especially in the face of new crisis events where the timing of such analysis is often pivotal. this work instead takes a fully unsupervised approach such that we do not rely on any labeled data, thus becoming agnostic to distributional shifts in domain or types of crisis, and robust for time-critical events. we posit that emotion triggers can be summarized effectively in an extractive manner where unsupervised methods are well-suited; we thus tackle the challenge of simultaneous emotion prediction and trigger extraction. for this new task, we first introduce covidet-ext, augmenting @xcite 's covidet with manually annotated extractive summaries corresponding to each of their abstractive summaries. the result is a dataset of 1, 883 reddit posts about the covid-19 pandemic, manually annotated with 7 fine-grained emotions (from covidet) and their corresponding extractive triggers (figure 1 ). for every emotion present in a post, our annotators highlight sentences that summarize the emotion triggers, resulting in 6, 741 extractive summaries in total. qualitative analyses of the dataset indicate good agreement among the annotators, and followup human validations of the annotations also reveal high correctness. covidet-ext provides an ideal test bed to facilitate the development of extractive (supervised or unsupervised) techniques for the tasks of emotion detection and trigger summarization in crisis contexts. we propose emotion-aware pagerank (eap), a novel, fully unsupervised, graph-based approach for extractive emotion trigger summarization from text. the core of our method is to decompose the traditional pagerank @xcite ranking algorithm into multiple biased pageranks @xcite , one for each emotion. to bias our model towards various emotions, our approach harnesses lexical information from emotion lexicons @xcite . critically, unlike previous graph-based unsupervised approaches @xcite @xcite @xcite , which represent the text as a bag-of-words or word embeddings, eap incorporates a language understanding module leveraging large language models to ensure that the summaries for an emotion are coherent in the context of that emotion. results on our covidet-ext indicate the effectiveness of our eap, which significantly pushes the rouge-l score of our summaries by an average of 2.7% over strong baselines. our contributions are as follows: 1) we introduce covidet-ext, a manually annotated benchmark dataset for the task of emotion detection and trigger summarization. 2) we propose emotion-aware pagerank, a variation of pagerank that combines a language understanding module and external emotion knowledge to generate emotion-specific extractive summaries. 3) we carry out a comprehensive set of experiments using numerous baselines to evaluate the performance on covidet-ext and show that our proposed eap significantly outperforms strong baselines. ## dataset construction since there is no annotated data for extractive emotion triggers summarization in crisis contexts, we first bridge this gap by extending @xcite 's abstractive-only dataset with extractive trigger summaries. doing so (a) creates benchmark data for extractive systems; (b) allows in-depth analyses to understand how and when emotion triggers are expressed on social media. this will also create a parallel abstractiveextractive dataset for future research. we name our new dataset covidet-ext (covidet {extractive, extension}). ## unsupervised extractive summarization in this section we introduce emotion-aware pagerank (eap), our fully unsupervised, graph-based, emotion trigger extractive summarization method that incorporates information from emotion lexicons to calculate a biased pagerank score of each sentence in a post. eap then fuses this score with an additional similarity-based sentence-level score that ensures the summary for a specific emotion e does not diverge in meaning from other summaries of the same emotion e. we show an overview of our model architecture in figure 3 . ## experiments and results in this section, we first introduce our emotionagnostic and emotion-specific baselines. next, we present our experimental setup and discuss the results obtained by eap against the baselines. emotion-agnostic baselines. we explore two standard heuristic baselines, namely 1) extracting the first sentence in the post (1 sent) and 2) extracting the first three sentences in the post (3 sent). next, we design three graph centrality measurebased methods: 3) pacsum (zheng and lapata, 2019), 4) presum (liu and lapata, 2019) and wordlevel 5) textrank @xcite . note that these methods are emotion-oblivious and the generated summary will be identical for different emotions. emotion-specific baselines. we first employ two lexical-based methods: 6) emolex -we use the emolex @xcite lexicon to identify lexical cues that indicate the expression of emotions. if a sentence contains a word that is associated with an emotion e, we consider the sentence to express e. the final summary for e contains all sentences expressing e. 7) emointensity -we leverage the nrc affect intensity lexicon @xcite to build a more fine-grained approach of identifying if a sentence expresses an emotion or not. for each sentence and emotion, we calculate the average emotion word intensity and compare it to a pre-defined threshold t. if the average intensity for e is higher than t we label the sentence with e. t is a tunable parameter that we select based on our validation set performance. finally, we leverage models trained on emotion detection datasets to build our emotion-specific summaries. for a post p , we use our model to make predictions on each sentence in p and build summaries by concatenating sentences that express the same emotions. we mainly experiment with a model trained on the 8) goemotions @xcite dataset. experimental setup. we carry out our experiments on an nvidia a5000 gpu. we use the hug-gingface transformers @xcite library for our sentence-bert implementation and we will make the code for our methods and data available for reasearch purposes. we report the performance in terms of rouge-2 and rouge-l @xcite to evaluate the summarization performance. additionally, we also calculate the performance in terms of f1 and show the results in appendix i. we provide extensive details about the hyperparameters used in eap and the baselines, such as our various thresholds and constants in appendix §g. results. we show the results obtained in eap consistently yields the highest results both in terms of rouge-2 and rouge-l compared to the other approaches. concretely, we obtain an average improvement of 2.7% in rouge-l and 2.5% in rouge-2 score over our strongest emointensity baseline. for example, on anger and joy we see improvements in rouge-2 of 1.7% and 6.3% respectively. moreover, our emotion-aware pagerank considerably outperforms textrank @xcite ) by as much as 5.5% rouge-l and 4.5% rouge-2 on average. ## conclusion we introduce covidet-ext, a new benchmark dataset composed of 1, 883 reddit posts annotated for the task emotion detection and extractive trigger summarization in the context of the covid-19 pandemic. our proposed emotion-aware pagerank approach yields strong results on our datasets, consistently outperforming prior work in an unsupervised learning context. in the future, we plan to study abstractive trigger summarization from an unsupervised point of view to bridge the gap between the extractive and abstractive summarization performance.