Experiencers, Stimuli, or Targets: Which Semantic Roles Enable Machine Learning to Infer the Emotions? (2020)

Abstract: Emotion recognition is predominantly formulated as text classification in which textual units are assigned to an emotion from a predefined inventory (e.g., fear, joy, anger, disgust, sadness, surprise, trust, anticipation). More recently, semantic role labeling approaches have been developed to extract structures from the text to answer questions like: “who is described to feel the emotion?” (experiencer), “what causes this emotion?” (stimulus), and at which entity is it directed?” (target). Though it has been shown that jointly modeling stimulus and emotion category prediction is beneficial for both subtasks, it remains unclear which of these semantic roles enables a classifier to infer the emotion. Is it the experiencer, because the identity of a person is biased towards a particular emotion (X is always happy)? Is it a particular target (everybody loves X) or a stimulus (doing X makes everybody sad)? We answer these questions by training emotion classification models on five available datasets annotated with at least one semantic role by masking the fillers of these roles in the text in a controlled manner and find that across multiple corpora, stimuli and targets carry emotion information, while the experiencer might be considered a confounder. Further, we analyze if informing the model about the position of the role improves the classification decision. Particularly on literature corpora we find that the role information improves the emotion classification.

## introduction emotion analysis is now an established research area which finds application in a variety of different fields, including social media analysis @xcite @xcite , opinion mining @xcite , and computational literary studies @xcite @xcite . the most prominent task in emotion analysis is emotion categorization, where text receives assignments from a predefined emotion inventory, such as the fundamental emotions of fear, anger, joy, anticipation, trust, surprise, disgust, and sadness which follow theories by @xcite or @xcite . other tasks include the recognition of affect values, namely valence or arousal @xcite or analyses of event appraisal @xcite . more recently, categorization (or regression) tasks have been complemented by more fine-grained analyses, namely emotion stimulus detection and role labeling, to detect which words denote the experiencer of an emotion, the emotion cue description, or the target of an emotion. these efforts lead to computational approaches of detecting stimulus clauses @xcite @xcite and emotion role labeling and sequence labeling @xcite @xcite @xcite , with different advantages and disadvantages we discuss in @xcite . further, this work led to a rich set of corpora with annotations of different subsets of roles. an example of a sentence annotated with semantic role labels for emotion is " john manually construct a dataset following framenet's emotion predicate and annotate the stimulus as its core argument. @xcite annotate tweets for emotion cue phrases, emotion targets, and the emotion stimulus. in our previous work @xcite we publish news headlines annotated with the roles of emotion experiencer, cue, target, and stimulus. @xcite annotate sentence triples taken from literature for the same roles. a popular benchmark for emotion stimulus detection is the mandarin corpus by @xcite . @xcite annotate english and mandarin texts in a comparable way on the clause level (emotion cause analysis, eca). in this paper, we utilize role annotations to understand their influence on emotion classification. we evaluate which of the roles' contents enable an emotion classifier to infer the emotions. it is reasonable to assume that the roles' content carries different kinds of information regarding the emotion: one particular experiencer present in a corpus might always feel the same emotion; hence, be prone to a bias the model could pick up on. the target or stimulus might be independent of the experiencer and be sufficient to infer the emotion. the presence of a target might limit the set of emotions that can be triggered. finally, as some of the corpora contain cue annotations, we assume that these are the most helpful to decide on the expressed emotion, as they typically have explicit references towards concrete emotion names. ## experimental setting in the following, we describe our experiments to understand which of the datasets' annotated roles contribute to the emotion classification performance. datasets. we base our experiments on five available datasets that are annotated for at least one of the roles of an experiencer, stimulus, target, or cue. the dataset by @xcite is one of the earliest we are aware of that contains stimulus annotations. they annotate based on framenet's emotion-directed frames that have a stimulus argument in the data (we refer to their corpus as emotion-stimulus, es). similarly early work is the twitter corpus by @xcite . they also follow the emotion frame semantics definition but use data concerning the 2012 u.s. election. therefore, their resource may be considered more diverse in language but more consistent in its domain than es. more recently, @xcite published an annotation of news headlines (goodnewseveryone, gne). while they do not limit their corpus on a domain, they use a comparably narrow time window to retrieve the data and sample according to the inclusion of emotion words and popularity on social media. @xcite and @xcite use literature data, which might be considered the most challenging for emotion analysis (for eca, we use the english subset only). as ## results in the following, we discuss the results of the bi-lstm model in detail and then point to differences to those of the transformer-based approach. as expected, an important role for emotion classification is the cue. in reman, the performance drops the most when the classifier does not see the cue span and gains the most when only the cue is available. for all other corpora, the cue role is not as important, but performance still shows a drop when it is not available (without). similarly, for all datasets except eca, the performance drops when the stimulus is not shown. on the other hand, the stimulus alone is insufficient to infer the emotion with competitive performance. noteworthy here is the corpus es, in which the performance drop is particularly high. these results show that the information contained in different roles is of varying importance and depends on the data's source and domain. in the setting position, we leave all information accessible to the model but add positional indicators for the investigated role to the input for emotion classification. we see improvements in most cases, except reman, for which adding the positional information hurts the classification for all roles. this result could be because reman has very long annotation spans. both eca and es show an improvement for their annotated role (stimulus). for et, an increase in performance is shown when additional knowledge about the stimulus position is given, and for gne, a slight improvement is shown when the model is given the experiencer's position information. this might have two reasons: on one hand, it is less likely to improve upon already high values when changing the model configuration. on the other hand, and more interestingly, it might be that the contextualized embeddings compensate for missing information. interestingly for the position setting, the results are improving on all datasets, and reman gains from the cue's positional indicators. the dataset that stands out in this setting is et, for which we see a slight decrease in performance across all roles available. the only setting shows that the stimulus captures most of the emotion information for gne and et. the result for gne is due to the particularly lengthy stimuli spans that sometimes stretch over the whole instance. ## conclusion and future work our experiments show that the importance of semantic roles for emotion classification differs between datasets and roles: the stimulus and cue are critical for classification, which correspond to the direct report of a feeling and the description that triggered an emotion. this result is shown in the drop in performance when removing these roles. this information is not redundantly available outside of these arguments. it is particularly beneficial for the model's performance to have access to the position of cues and stimuli. this suggests that the classifier learns to tackle the problem differently when this information is available, especially so for eca and es -the cases in which literature has been annotated and the instances are comparably long. the bi-lstm model indicates that the experiencer role is a confounder in gne. the performance can be increased when the model does not have access to its content. similar results are observed for et, in which the target role is a confounder. however, these results should be taken with a grain of salt given that they are not confirmed while switching to the transformer-based model. the differences in results between the bi-lstm and the transformer also motivate further research, as they suggest that the contextualized representation might compensate for missing information, and is, therefore, more robust. finally, our results across both models and multiple datasets indicate that emotion classification approaches indeed benefit from semantic roles' information by adding the positional information. similarly to targeted and aspect-based sentiment analysis, this motivates future work, in which emotion classification and role labeling should be modelled jointly. in this case, it can also be interesting to investigate what happens when the positional indicators are added to all roles jointly.