Emotion Ratings: How Intensity, Annotation Confidence and Agreements are Entangled (2021)

Abstract: When humans judge the affective content of texts, they also implicitly assess the correctness of such judgment, that is, their confidence. We hypothesize that people’s (in)confidence that they performed well in an annotation task leads to (dis)agreements among each other. If this is true, confidence may serve as a diagnostic tool for systematic differences in annotations. To probe our assumption, we conduct a study on a subset of the Corpus of Contemporary American English, in which we ask raters to distinguish neutral sentences from emotion-bearing ones, while scoring the confidence of their answers. Confidence turns out to approximate inter-annotator disagreements. Further, we find that confidence is correlated to emotion intensity: perceiving stronger affect in text prompts annotators to more certain classification performances. This insight is relevant for modelling studies of intensity, as it opens the question wether automatic regressors or classifiers actually predict intensity, or rather human’s self-perceived confidence.

## introduction a plethora of theories exist on the matter of emotions: the intensity of affective states, their link to cognition, and their arrangement into categories are just a few of the angles from which psychology has tackled this complex phenomenon @xcite . correspondingly, in computational emotion analysis, texts have been associated to values of intensity @xcite , to cognitive components @xcite , and discrete classes @xcite . in support of these tasks, substantial research effort has been directed to resource construction, which typically relies on the participation of human judges. yet, emotions are a subjective experience. their interpretation in text varies across individuals, and this poses a major challenge in emotion analysis: it is hard to reach acceptable levels of inter-annotator agreement (iaa) @xcite @xcite . on their side, humans (roughly) know how well they can "read" emotions @xcite . they judge the affective content of texts, and at the same time, the correctness of such judgment: in other words, annotators can evaluate their own confidence with respect to their labelling decisions. this hints a possible relation between confidence and iaa. one could expect that annotators are more likely to incur inconsistencies when they feel uncertain about their answers. hence, it would be interesting to verify if self-assessed confidence approximates inter-annotator disagreements in affectrelated annotation tasks. the collection of this type of judgments is not a common practice: past research has found that self-assigned scores of confidence are predictable based on some vocal attributes of emotion speech stimuli @xcite , but this has not been done on text, to the best of our knowledge. yet another aspect involved in emotion recognition and which, at first sight, relates to confidence, is emotion intensity. it would be intuitive to assume that emotions are recognized with higher confidence if they are expressed with stronger magnitude (e.g., "the teacher exploded" > "he snapped his annoyed temper", "sadder" > "a bit sad", "ecstasy" > "joy"). still, in the sentence "we had to cheer him up; later, he was off the ground", readers have to choose what part of the text to attend to (the challenge that the speakers undertook -not intense, or the effect they had -intense) and be very confident about either choice. similar counter examples reveal that the link between the perception of emotion intensity and the self-perceived confidence is opaque, and leaves room for exploration. in this paper, we experimentally investigate the relationship between three human judgments: about the presence of emotions, about their intensity, and about the confidence of the annotation decision. leveraging such information, we aim at understanding in what cases annotators differ regarding the judgement that an emotion is expressed. our first research question is: (rq1) are (dis)agreements with respect to the presence of emotions related to confidence? second, (rq2) are judgments of intensity and confidence entangled? we bring these issues together in an annotation study based on emotion recognition, in which self-perceived confidence is a dimension to be rated. given a subset of the corpus of contemporary american english (coca) @xcite , raters distinguish emotionbearing sentences from neutral ones, while quantifying both the intensity of the emotion and their confidence on a likert scale @xcite . we find that confidence can explain systematic differences in decisions of annotators; so does intensity; and impressions on intensity and confidence are correlated. based on these results, we devise a strategy to leverage the two factors and smoothen out inter-annotator inconsistencies. ## annotation setup tasks. the first step in this study is to collect emotion assessments. 1 we are not interested in which emotion people interpret from text, but rather if they recognize any. judges read sentences and answer the question: (emo) is it emotional or neutral? for the items deemed to express an emotion, we also ask (int) how strong is it?, which enables us to obtain ratings about affective strengths on a likert scale from 1 (not intense) to 3 (very). lastly, since raters interpret emotions without an immediate first-hand experience, we have them self-evaluate their own judgments on a scale from 1 (unsure) to 3 (certain), in response to the question (conf) how confident are you about your answer to emo? as for emo, we acknowledge that the emotional content of an utterance can be inferred from many perspectives. it is possible to assess one's own emotion after reading the text, to reconstruct the affective state of the writers who produced it, to guess the reaction that they intended to elicit in the readers, and so on. to avoid confusion, we instruct annotators to consider the presence of an emotion only with respect to their personal viewpoint. we opt for an in-lab setting. raters are three female master students aged between 24 and 27, who are proficient in english, and have some annotation experience and background in computational emotion analysis. data. corpora that include emotion classes or gradations are tailored on specific domains, like self-reports @xcite , tweets @xcite and newspapers @xcite . we broaden our focus to multiple genres, and annotate sentences from the 2020 version of coca foot_1 , which includes unlabelled texts that occurred from 1990 to present in different domains, like blogs, magazines, newspapers, academic texts, spoken interactions, fictions, tv, and movie subtitles. with a corpus of this size (>1b words), considering all data points would be costly, and randomly selecting them may cause imbalance in the final annotation -i.e., a majority of neutral instances. therefore, we draw a sample biased towards emotional sentences with a combination of rules and classifier-based information. to obtain such a classifier, we fine-tune the pre-trained bert @xcite base-case model on a number of emotion analysis resources foot_2 , adding a classification layer that outputs the labels emotion or neutral. having that, we filter academic texts out of coca for their arguably impartial language, and from each of the other genres, we randomly pick 500 sentences; out of these, we sample 100 sentences balanced by class, i.e., 50 labelled as neutral by our classifier, 50 as bearing an emotion. thus, the annotators are shown 700 items, 100 per domain, with a balanced class distribution, according to the classifier. ## results to answer our research questions, we first observe annotators' agreements on emo. the highest @xcite between pairs of human judges was .43 ( at first glance, these numbers appear unsatisfactory. on the one hand, they are due to the skewed class distribution in the annotators' choices. the other, they can be traced back to the way in which the emo task was formulated: asking if a text is emotional from the readers' own point of view (e.g., it "describes an event [...] to which you would associate an emotion", see appendix) as opposed, for instance, to the writers', paves the way for more heterogeneous responses. however, a look at other iaa measures, like the absolute counts of items that were assigned to each label, leads to a more detailed picture. ## discussion and conclusion this annotation investigated if the perceived emotion (class), a perceived feature of emotion (intensity), and self-perception (confidence) are tied together -and can help understand inconsistent annotations. we found that (rq1) both intensity and confidence account for inter-annotator inconsistencies relative to binary decisions. adopting confidence as an acceptance threshold showed that higher scores lead to more uniform assessments of emotions; though not a surprising effect of confidence, this also applies to intensity. moreover, (rq2) the two variables are correlated, that is, people feel more certain about their emotion recognition performance on items with high intensity. we acknowledge that our design of emo naturally incurs the risk of inconsistent answers. however, precisely for the subjective nature of the task, the finding that disagreements decrease with high conf/int is interesting in itself: some judgments which are seemingly unsolvable can be explained by certain perceived properties of emotions (intensity) or self-perceived features (confidence). from these results we can draw some lessons. first, the correlation between confidence and intensity brings relevant implications to all those studies that focus on emotional strength. when asked to evaluate intensity, do people confound that with confidence? even more, is there a causal relation between the two? as a best practice to put safeguards in their guidelines, experimenters may ask people to tease the two variables apart. potentially, this issue concerns modelling studies as well: do classifiers for emotion intensity predict such feature, or rather the confidence with which people judge emotions? we provided reasons to look into this further. second, confidence turned out to be an important dimension of rating, because it can inform us when the annotators expect to disagree. when judgments diverge, annotators do not deem their intuition credible. hence, our finding that confidence approximates disagreements means precisely that people themselves predict their performance to differ from that of the others. concretely, all this knowledge can come in support of annotation studies. including confidence as a rating dimension may give an additional source of information about annotators' reliability. this can help experimenters to refine the guidelines in a pre-testing phase: one might want to disentan-gle cases in which annotators' disagreements are random -signaling a lack of annotators' reliability, and when, instead, they are due to consistently different ways of perceiving and reporting on confidence. in this second case, disagreements may be normalized to an extent, by post-processing the annotation results. for instance, as people seem to agree on the class emotion only if they also agree on certain degrees of conf, there might be some levels of confidence (or intensity) that one filters in/out of the final annotation labels. a similar strategy may be somewhat restrictive, since it accepts as emotional only those items on which humans' intuition fall above a pre-defined threshold. while we observed agreement on such items, it is possible to adopt more nuanced evaluation approaches and integrate information about intensity or confidence into iaa measures. as an example, disagreements between two raters can be penalized more when the one choosing the emotion label does so by perceiving extreme confidence or intensity -even though we provided evidence that these cases are rare. future work could explore this direction. in summary, we uphold that disagreements are not necessarily symptomatic of unreliability. this claim has so far not found much attention in emotion annotations, but is in line with a more general body of research dedicated to the reasons and the patterns underlying annotators' disagreements and to the ways in which their intuitions should be aggregated and evaluated @xcite @xcite @xcite . the applicability of these ideas to emotions should not come as a surprise -their assessment can derive from perceptive and metaperceptual phenomena (intensity and confidence, for instance). therefore, if emotion judgments alone might not be sufficient to measure the quality of annotations, they can be enriched and, eventually, explained by the knowledge of such phenomena.