SRL4E – S emantic R ole L abeling for E motions: A Unified Evaluation Framework (2022)

Abstract: In the field of sentiment analysis, several studies have highlighted that a single sentence may express multiple, sometimes contrasting, sentiments and emotions, each with its own experiencer, target and/or cause. To this end, over the past few years researchers have started to collect and annotate data manually, in order to investigate the capabilities of automatic systems not only to distinguish between emotions, but also to capture their semantic constituents. However, currently available gold datasets are heterogeneous in size, domain, format, splits, emotion categories and role labels, making comparisons across different works difficult and hampering progress in the area. In this paper, we tackle this issue and present a unified evaluation framework focused on Semantic Role Labeling for Emotions (SRL4E), in which we unify several datasets tagged with emotions and semantic roles by using a common labeling scheme. We use SRL4E as a benchmark to evaluate how modern pretrained language models perform and analyze where we currently stand in this task, hoping to provide the tools to facilitate studies in this complex area.

## introduction emotion detection -a long-standing open problem in natural language processing (nlp) -is the task of automatically associating one or more emotions with a text. even though emotional states are highly subjective and often depend on several factors, such as one's past experiences, culture and education, the automatic identification, categorization and analysis of emotions in texts has been found to be beneficial in a wide array of downstream tasks, such as hate speech detection @xcite , sarcasm detection @xcite , and modeling political discourse @xcite , inter alia. in the past decade, deep learning techniques have become ubiquitous in the development of au-tomatic systems for an increasing number of nlp tasks, including emotion detection @xcite . however, most of the effective neural-based approaches still require significant amounts of training data in order to learn to perform at their best. for this reason, with a view to bootstrapping the development of neural systems for emotion detection, there have been several efforts to annotate corpora with emotions manually @xcite . nevertheless, over the past few years, numerous studies have indicated that a short text, even a single sentence, may contain multiple -at times concurring, at other times contrasting -sentiments and emotions. and not only this, two emotions in the same sentence may be experienced, targeted, and/or caused by different semantic constituents which, similarly to predicate-argument structures in semantic role labeling (srl), can be linked to form abstract semantic structures. the potential applications in social media analysis, abuse detection, and other actively studied areas in nlp @xcite of such automatically-extracted emotion-focused semantic structures have prompted researchers to create datasets aimed at investigating the capabilities of modern systems to parse emotional events @xcite . unfortunately, despite the increasing interest in this area, currently available gold datasets feature heterogeneous structures and characteristics, ranging from varying sizes to different domains, file format, splits and, most importantly, non-overlapping emotion categories. we argue that this heterogeneity obstructs, or at least hinders, further progress in this relatively new area of sentiment analysis. in this paper, we take a step towards addressing the above-mentioned issues and introduce a unified framework for semantic role labeling for emotions (srl4e). in srl4e, we unify several gold but heterogeneous datasets that contain anno-tations both for emotions and for their semantic constituents, so as to obtain a new homogeneous dataset that covers diverse domains and that can be used to train, validate and evaluate current and future work in this task. our contributions can be summarized as follows: we release srl4e at https://github.com/ sapienzanlp/srl4e in the hope that our unified framework will become a stepping stone for the development and evaluation of current and future approaches to semantic role labeling for emotions. ## related work emotion classification datasets. currently, there are a wide variety of datasets annotated with emotion classes, ranging across different domains and using different annotation schemes. among others, we can find datasets on emotional experiences @xcite ), children's fairy tales @xcite , news headlines @xcite , blog posts @xcite , news @xcite , social media posts and reviews @xcite , dialogs @xcite , facebook posts @xcite , with many focusing on tweets @xcite @xcite @xcite due to their tendency to have dense emotional content. to meet such a diversity of contents and formats, bostan and klinger (2018) created a unified resource for emotion classification comprising many of the aforementioned datasets, while @xcite , instead, added an additional clause-level annotation layer to some existing resources. more recent efforts, such as goemotion @xcite , xed @xcite and canceremo @xcite , provide, respectively, emotion annotations for reddit comments, multilingual subtitles and blog posts about health problems. although the above-mentioned corpora have enabled systems to perform emotion detection across different domains, their annotations are sentencelevel and, therefore, introduce an oversimplification: they indicate only the overall sentiment and/or emotion that appears in a given text, neglecting the cases in which a short text, even a single sentence, may express multiple emotions. furthermore, the aforementioned datasets do not indicate which part of the text elicits an emotion and who experiences, is the target of, or causes that emotion. as a consequence, a system trained on these datasets may produce predictions that are hard to interpret and more difficult to use in real-world applications. to overcome these problems, we rely on resources that not only indicate emotions, but also identify their semantic constituents, namely, emotional cues, ex-periencers, targets and stimuli. emotion taxonomy. among the studies that aim to identify the fundamental emotions, @xcite proposed a set of six categories: anger, disgust, fear, joy, sadness and surprise; @xcite shared the same set with two additions: anticipation and trust. instead of relying on discrete categories, @xcite proposed the circumplex model where every emotion can be described by three continuous values: arousal, dominance and valence. more recent studies in psychology use more fine-grained sets of emotions, ranging from 12 @xcite to 28 categories @xcite , devised depending on the context of the study, e.g., speech prosody and facial expressions. however, the analysis of demszky et al. (2020) over a fine-grained set of 28 emotions suggests that a large number of categories results in more frequent disagreements on similar classes (such as anger and annoyance, or excitement and joy) which, in turn, can lead to low inter-annotator agreement and unbalanced distributions among some of these categories. therefore, we adopt plutchik's wheel of emotions @xcite , which provides clearly distinct and well-defined coarse-grained categories, whose composition can be used to virtually describe all other fine-grained sets. moreover, some datasets in srl4e @xcite @xcite already use plutchik's or plutchik-based categories. emotions and srl. over the past few years, automatic systems for srl have achieved impressive performance in identifying and labeling predicateargument relations @xcite @xcite , and have long become useful tools in several downstream tasks, from question answering @xcite to machine translation @xcite . defined by @xcite as the task of answering the question "who did what to whom, where, when and how?", srl is almost a natural choice for the extraction of the semantic constituents of those events that elicit emotional states. indeed, emotional cues can be seen as particular types of predicates, and their semantic constituents as their arguments. among the currently available datasets for emotion detection, there are some that also provide this kind of more granular semantic information. in particular @xcite and @xcite released corpora that indicate multiple emotions and their corresponding emotion cues in each sentence; @xcite and @xcite indicate the cause of an emotion, with the latter providing such annotations both in english and in chinese. finally, @xcite , @xcite , @xcite and @xcite provide annotations for emotion cues, experiencers, targets and stimuli, employing, however, different sets of emotions in different domains. this means that the results of a system trained on one of these datasets cannot be compared against the results of another system trained on a different dataset, emphasizing the need for a unified framework to train and evaluate future approaches to this task. this is also evidenced by the success of existing works, e.g. @xcite for sentence-level emotion classification and @xcite for word sense disambiguation. in srl4e, not only do we aggregate the resources under the same task formulation, but we also manually correct their inconsistencies and unify the different emotion schemes. ## srl4e in this section, we introduce srl4e. we first describe the categories of emotions and the format of the semantic roles we adopt to unify the annotation scheme of the original datasets. next, we provide a short overview of the datasets included in srl4e. finally, we give a formal definition of the task. ## data analysis emotion classes distribution. depending on the dataset, the distribution of emotion classes changes drastically, as illustrated in figure 2 . for example, in elections, which contains random tweets related to an american election campaign, almost 45% of samples are tagged with disgust, as one might expect: this is because many of the tweets in question tend to discredit the opposing party; similarly, the second most used class is trust in the tweets in favor of candidates. another interesting example is gne, where the most frequent category is surprise, highlighting the sensationalistic tone typically found in newspaper headlines. it is worth noting that, in contrast to each individual dataset, our unified dataset includes a fairly balanced distribution between categories, where the only category that appears more often is joy (20%), while all the others are between 6% and 10%, approximately. other statistics. the statistics reported in borderline examples. srl4e's formulation is based on the presence of cues within sentences, which are seen as the trigger of the emotion in that context. this formulation particularly suits those domains where emotions are expressed explicitly, such as gne, ntcir and reman. however, handling cues becomes non-trivial in some situations, for example in social networks (elections and emotweet) and blog posts (blogs). in these contexts, language features numerous implicit references and ironic content, where the mere presence of an emoji or a particular punctuation mark completely changes the context. in our task formulation, the presence of a cue is a fundamental requirement even if it may be difficult to identify, as we want to be able to model multiple, sometimes opposite, emotions in the same sentence. here is an example: in this case, the sadness emotion is associated only with the first cue, which is ";/", while the joy emotion is associated with the other two. even if a cue is composed only of punctuation marks (or emojis), it may still be the only useful signal for disambiguating the emotion, or for separating the presence of multiple emotions in the same sentence. ## experiments in this section, we analyze the benefits that our unified framework can bring to a neural model, based on recent contextualized representations from a pretrained language model. ## conclusion and future work recently, the study of emotions in nlp has been gaining interest, due to their potential not only for application to downstream tasks, but also for enhancing the interpretability of automatic outputs, especially when emotions are accompanied by information about their semantic constituents, i.e., their experiencers, targets and stimuli. however, recent efforts to provide manually annotated data for emotions and their semantic constituents have been heterogeneous in their annotation scheme, making it difficult to train, evaluate, and compare novel approaches. in this paper, we aimed at addressing these issues and presented a unified framework for the semantic role labeling of emotions (srl4e). our framework collects, cleans, and unifies the annotation schemes of six datasets that provide information about emotions and their semantic roles, making it easy to train and evaluate existing and future systems. we conducted several experiments to demonstrate empirically that our unified scheme is beneficial in each subtask, namely, emotion classification and role (experiencer, target, stimulus) identification, especially in bilingual settings (english-chinese). with srl4e, we hope to stimulate future research in this complex area at the intersection of emotion detection and semantic role labeling. we release the software to reproduce the benchmark and our experiments at • ntcir 13 eca. the license is not specified and the download page is no longer online, but a snapshot can be accessed using internet archive; foot_4 a summary of the above information is reported in