Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024)

Abstract: The subjective perception of emotion leads to inconsistent labels from human annotators. Typically, utterances lacking majority-agreed labels are excluded when training an emotion classifier, which cause problems when encountering ambiguous emotional expressions during testing. This paper investigates three methods to handle ambiguous emotion. First, we show that incorporating utterances without majority-agreed labels as an additional class in the classifier reduces the classification performance of the other emotion classes. Then, we propose detecting utterances with ambiguous emotions as out-of-domain samples by quantifying the uncertainty in emotion classification using evidential deep learning. This approach retains the classification accuracy while effectively detects ambiguous emotion expressions. Furthermore, to obtain fine-grained distinctions among ambiguous emotions, we propose representing emotion as a distribution instead of a single class label. The task is thus re-framed from classification to distribution estimation where every individual annotation is taken into account, not just the majority opinion. The evidential uncertainty measure is extended to quantify the uncertainty in emotion distribution estimation. Experimental results on the IEMOCAP and CREMA-D datasets demonstrate the superior capability of the proposed method in terms of majority class prediction, emotion distribution estimation, and uncertainty estimation.

## introduction the inherent subjectivity of human emotion perception introduces complexity in annotating emotion datasets. multiple annotators are often involved in labelling each utterance and the majorityagreed (ma) class is usually used as the ground truth @xcite . utterances that have no majority-agreed (nma) labels (i.e., with tied votes) are typically excluded during emotion classifier training @xcite @xcite , which may cause issues when the system encounters such utterances in practical applications. this paper investigates three approaches to handling ambiguous emotion data. first, a naive method is tested which aggregates nma utterances into an additional class when training an emotion classifier. this approach proves problematic as nma utterances contain a blend of emotions, thereby confusing the classifier and undermining the classification performance. then we explore if an emotion classifier can appropriately respond with "i don't know" for ambiguous emotion data that does not fit into any predefined emotion class. this is realised by quantifying the uncertainty in emotion classification using evidential deep learning (edl) @xcite . when a classifier trained on ma data encounters an nma utterance during the test, the model should identify it as an out-of-domain (ood) sample by providing a high uncertainty score, indicating its uncertainty regarding the specific emotion class to which the nma utterance belongs. moreover, to obtain fine-grained distinctions between ambiguous emotional data, we re-frame the task from classification to distribution estimation. consider the example shown in figure 1 with the annotations assigned to three utterances. since the majority emotion classes are "angry" for both utterances (a) and (b), they will be assigned the same ground-truth label "angry" in the aforementioned classification system, which implies that they convey the same emotion content and is evidently unsuitable. on the contrary, utterance (c), though being an nma utterance, is more likely to share similar emotional content with utterance (b). therefore, in order to obtain more comprehensive representations of emotion content, we further propose representing emotion as a distribution rather than a single class label and re-framing emotion recognition as a distribution estimation problem rather than a classification problem. a novel algorithm is proposed which extends edl to estimate the underlying emotion distribution given observed human annotations and quantify the uncertainty in emotion distribution estimation. the proposed approach considers all human annotations rather than relying solely on the majority vote class. multiple evaluation metrics are adopted to evaluate the performance in terms of majority class prediction, uncertainty measure, and distribution estimation. rather than simply saying "i don't know", the proposed system demonstrates the ability to estimate the emotion distributions of the nma utterances and also offer a reliable uncertainty measure for the distribution estimation. our contributions are summarised as follows. (i) to the best of our knowledge, this paper is the first work that treats ambiguous emotion as ood and detects it by uncertainty estimation; (ii) this is the first work that applies edl to quantify uncertainty in emotion classification; (iii) imposing a single ground truth through majority voting leads to under-representation of minority views. we instead estimate the distribution over emotion classes which provides a more comprehensive representation of emotion content as well as a more inclusive representation of human opinions; (iv) a novel algorithm is proposed that extends edl to quantify uncertainty in emotion distribution estimation. ## related work human annotators often interpret the emotion of the same utterance differently due to their personal experiences and cultural backgrounds @xcite @xcite . instead of using the ma annotation as the ground truth label, some research suggests treating emo-tion classification as a multi-label task @xcite @xcite where all emotion classes assigned by any annotator are considered as correct classes and the ground truth label is presented as a multi-hot vector. the model is trained to predict the presence of each emotion class for each utterance. an issue with this approach is that it ignores the differences in strengths of different emotion classes. an alternative approach uses "soft labels" as the proxy of ground truth, which is defined as the relative frequency of occurrence of each emotion class @xcite @xcite . the kullback-leibler (kl) divergence or distance metrics between the soft labels and model predictions are used to train the model. however, soft labels, being maximum likelihood estimates (mle) of the underlying distribution based on observed samples, might not provide an accurate approximation to the unknown distribution when the number of observations (annotations) is limited. also, although adopting soft labels, those methods still focus on obtaining a "correct" label (i.e., pursuing improved classification accuracy). so far, the calibration of emotion models has not been extensively studied. in this study, we introduce a novel approach which provides not only better emotion content estimation but also a reliable measure of the model's prediction confidence. ## detecting nma as ood by quantifying emotion classification uncertainty as explained in the introduction and confirmed experimentally in section 7.1, training an emotion classifier with nma utterances grouped into an additional class degrades the classification performance. this section studies an alternative method. the emotion classifier is trained on ma utterances and nma utterances are treated as ood samples. by quantifying uncertainty in emotion classification, the model is expected to output a high uncertainty score when encountering ambiguous emotions, indicating that the utterance doesn't belong to any predefined ma class. ## evaluation metrics the proposed method is evaluated in terms of majority prediction, uncertainty estimation, ood detection, and distribution estimation. majority prediction. majority prediction for ma utterances is evaluated by classification accuracy (acc) and unweighted average recall (uar) which is the sum of class-wise accuracy divided by the number of classes. ## results this section presents experimental results of the three approaches for handling ambiguous emotion: incorporating nma as an extra class (section 7.1), detecting nma as ood (section 7.2), and representing emotion as distributions (section 7.3). the average of three runs are reported for all results.