Emotion-Aware, Emotion-Agnostic, or Automatic: Corpus Creation Strategies to Obtain Cognitive Event Appraisal Annotations (2021)

Abstract: Appraisal theories explain how the cognitive evaluation of an event leads to a particular emotion. In contrast to theories of basic emotions or affect (valence/arousal), this theory has not received a lot of attention in natural language processing. Yet, in psychology it has been proven powerful: Smith and Ellsworth (1985) showed that the appraisal dimensions attention, certainty, anticipated effort, pleasantness, responsibility/control and situational control discriminate between (at least) 15 emotion classes. We study different annotation strategies for these dimensions, based on the event-focused enISEAR corpus (Troiano et al., 2019). We analyze two manual annotation settings: (1) showing the text to annotate while masking the experienced emotion label; (2) revealing the emotion associated with the text. Setting 2 enables the annotators to develop a more realistic intuition of the described event, while Setting 1 is a more standard annotation procedure, purely relying on text. We evaluate these strategies in two ways: by measuring inter-annotator agreement and by fine- tuning RoBERTa to predict appraisal variables. Our results show that knowledge of the emotion increases annotators’ reliability. Further, we evaluate a purely automatic rule-based labeling strategy (inferring appraisal from annotated emotion classes). Training on automatically assigned labels leads to a competitive performance of our classifier, even when tested on manual annotations. This is an indicator that it might be possible to automatically create appraisal corpora for every domain for which emotion corpora already exist.

## introduction automatically detecting emotions in written texts consists of mapping textual units, like documents, paragraphs, or sentences, to a predefined set of emotions. common sets of classes used for this purpose rely on psychological theories such as those proposed by @xcite (anger, disgust, fear, joy, sadness, surprise) or @xcite . these theories are based on the assumption that there is a restricted number of emotions that have prototypical realizations. however, not all sets of emotions are appropriate for every domain. for instance, @xcite argue that some of the basic emotions are too strong for measuring how people feel when driving a car and, based on that, @xcite resort to joy, annoyance (instead of anger), insecurity (instead of fear), boredom, and relaxation to classify in-car utterances. @xcite model the emotional perception of poetry and opt for the categories beauty/joy, sadness, uneasiness, vitality, awe/sublime, suspense, humor, nostalgia, and annoyance, following the definition of aesthetic emotions @xcite . @xcite define a taxonomy of emotions, reaching a high coverage while maintaining inter-class relations. an alternative to the use of categorical variables are the so-called "dimensional" approaches. the most popular of them models affective experiences along the variables of dominance, valence, and arousal @xcite . feldman @xcite theorizes that emotions are interpretations of continuous affective states experiencers find themselves in. still, as @xcite note, not all emotions can be distinguished based on valence and arousal. one might argue that predicting three continuous variables instead of a richer set of categories is a simplification and can be limiting for downstream applications of emotion analysis models. @xcite particularly argue that the vad model does not capture all relevant aspects of an emotion in the context of an event. in a fight or flight situation @xcite , for instance, the decision to take one of these two actions emotion appraisals text joy attention, certainty, pleasant, sit. ctrl. i felt . . . when i knew that i was going back to florida a year earlier than i thought i would. disgust attention, certainty, effort, sit. ctrl. i felt . . . when my kitten was sick and i had to clean it up. fear attention, effort, sit. ctrl. i felt . . . when i was having a hard attach. guilt attention, certainty, effort, respons., control i felt . . . when i went on holiday and left our cat behind. sadness attention, certainty, sit. ctrl. i felt . . . when i found out one of my favourite shops had shut down. as the cognitive appraisal is a fundamental subcomponent of emotions, we deem that appraisal dimensions are useful to perform emotion recognition, and that even the prediction of appraisals themselves can contribute to computational approaches to affective states. these appraisal dimensions have only recently found application in automatic emotion analysis in text: hofmann et al. (2020) re-annotated a corpus of 1001 english emotional event descriptions @xcite for which the experienced emotion has been disclosed by the author of the description ( we test this by comparing three annotation procedures: (1) we give the annotator access only to the text but not to its emotion label; (2) we give the annotator access to the text and the emotion, and evaluate if this additional information has an impact on annotation reliability and performance of a pretrained transformer-based classifier fine-tuned on these data; and (3) we automatically infer the appraisal dimensions from existing emotion annotations, investigating the hypothesis that manual annotation might not be necessary. our main contributions are that we show that (a) appraisal annotation is more reliable when annotators have access to the emotion label of the original experiencer, hence, the event description itself does not carry sufficient information for annotation. that also means that annotating appraisals for corpora in which the original emotion is not available might be particularly challenging. (b) automatic, rule-based annotation of appraisals that leverages emotion labels is a viable alternative to human annotation, and therefore, appraisal corpora can automatically be created for domains for which emotion corpora are already available. our classifier further constitutes a novel state of the art for appraisal prediction on the data by @xcite . the data is available at https:// www.ims.uni-stuttgart.de/data/appraisalemotion . ## qualitative analysis to better understand how revealing emotions affects the annotations in experiment 1, we provide some concrete examples. in example (1) an event is described in a way which leaves open if there is responsibility, pleasantness, anticipated effort, and even if the experiencer is entirely certain about what is happening. the score is the sum of agreement changes, either improvements (+1 for each dimension) or degradation (-1). a: attention, ce: certainty, e: anticipated effort, p: pleasantness, r: responsibility, ci: circumstance. with the knowledge that the emotion is fear, it becomes clear that the situation does involve anticipated effort, is not pleasant, and that circumstance is not likely. the annotators also agree here that there is responsibility involved, which is likely an interpretation based on world-knowledge. in the situation of getting a new job (example (2)), knowing the emotion adds agreement regarding certainty, which is in line with added agreement that the person was responsible. example (3) is an instance in which the situation was already entirely clear without knowing the emotion -the experiencer participated in gossip. that is a certain, non-pleasurable situation (when recognizing this) which is under their own control. knowledge that guilt has been experienced does not add anything. example (4) shows that complex situations cause more disagreement in annotations which are not necessarily resolved by knowing the emotion. the described event is about a negative emotion felt because the experiencer did not recognize the bad mood of the daughter. annotators come to disagree about attention, certainty and responsibility. example (5) describes another situation in which a negative event is discussed. knowledge of the emotion puts a clear focus away from the sad part of the description (father dying) and puts it on something that causes anger. however, this shift does not resolve appraisal disagreements but indeed adds on top of them, with attention and anticipated effort. finally, example (6) is another long description with annotators' focus on different aspects, one on the darkness (hence, no responsibility), the other on the cycling (responsibility). here, knowledge of the emotion does not change the interpretability of the event. however, it informs on which part of the described situation the original author focused on. these examples show that emovis helps solve ambiguities when events can be associated to multiple emotions, other times it helps people give more weight to specific portions of texts. in the first case agreement tends to be reached more easily. ## conclusion & future work we analyzed how to build corpora of text annotated with appraisal variables and we evaluated how well such concepts can be modelled. by doing so, we brought together emotion analysis and a strand of emotion research in psychology which has received little attention from the computational linguistics community. we propose that in addition to wellestablished approaches to emotion analysis, like affect-oriented dimensional approaches or classification into predefined emotion inventories, psychological models of appraisals will be considered in future emotion-based studies, particularly those relying on event-oriented resources. the use of appraisals is interesting from a theoretical perspective: motivated by psychology, we leveraged the cognitive mechanisms underlying emotions, thus accounting for many complex patterns in which humans appraise an event and emotionally react to it. in this light, it is interesting in itself that our annotators were able to empathically reconstruct the event appraisals experienced by others, even without knowing their emotion. from a practical perspective, appraisal annotations are less prone to being poorly chosen for particular domains, in comparison to regular emotion classes, as the actual feeling develops based on the cognitive evaluation of an event. we also have shown that event descriptions alone might not be sufficient to properly annotate the hypothetical appraisal of the experiencer (which is, however, also an issue with traditional emotion analysis annotations and models -we cannot look into the feeler, we deal with private states). this shows and is presumably the reason that additional context (e.g., the emotion label) is required. some implications for future research and developments follow: ideally, appraisal (and emotion) annotations should stem directly from the experiencer. this is not doable in many nlp settings. for instance, when analyzing literature, it is impossible to ask fictional characters for their current event appraisal. however, we presume that settings on social media might be realistic, for instance by probing appropriate distant labeling methods, e.g., a careful choice of hashtags. if this is unfeasible, because text authors do not disclose their appraisals, the available emotion labels still represent a valuable source of information: as we have shown, they can guide the interpretation of the described events, and hence, the way in which these are post-assigned appraisal dimensions. finally, we provided evidence that even if a model is trained on automatically obtained appraisal labels, it is still capable of substantial performance. therefore, we conclude that more corpora with appraisal dimensions from different languages and domains should be created from scratch. in the meantime, one can build on top of the rich set of available emotion corpora and automatically create appraisal-annotated resources out of them.