Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression (2023)

Abstract: In automatic emotion recognition (AER), labels assigned by different human annotators to the same utterance are often inconsistent due to the inherent complexity of emotion and the subjectivity of perception. Though deterministic labels generated by averaging or voting are often used as the ground truth, it ignores the intrinsic uncertainty revealed by the inconsistent labels. This paper proposes a Bayesian approach, deep evidential emotion regression (DEER), to estimate the uncertainty in emotion attributes. Treating the emotion attribute labels of an utterance as samples drawn from an unknown Gaussian distribution, DEER places an utterance-specific normal-inverse gamma prior over the Gaussian likelihood and predicts its hyper-parameters using a deep neural network model. It enables a joint estimation of emotion attributes along with the aleatoric and epistemic uncertainties. AER experiments on the widely used MSP-Podcast and IEMOCAP datasets showed DEER produced state-of-the-art results for both the mean values and the distribution of emotion attributes.

## introduction automatic emotion recognition (aer) is the task that enables computers to predict human emotional states based on multimodal signals, such as audio, video and text. an emotional state is defined based on either categorical or dimensional theory. the categorical theory claims the existence of a small number of basic discrete emotions ( i.e. anger and happy) that are inherent in our brain and universally recognised @xcite . dimensional emotion theory characterises emotional states by a small number of roughly orthogonal fundamental continuousvalued bipolar dimensions @xcite such as valence-arousal and approach-avoidance @xcite @xcite . these dimensions are also known as emotion attributes, which allow us to model more subtle and complex emotions and are thus more common in psychological studies. as a result, aer includes a classification approach based on emotion-class-based labels and a regression approach based on attribute-based labels. this paper focuses on attribute-based aer with speech input. emotion annotation is challenging due to the inherent ambiguity of mixed emotion, the personal variations in emotion expression, the subjectivity in emotion perception, etc. most aer datasets use multiple human annotators to label each utterance, which often results in inconsistent labels, either as emotion categories or attributes. this is also a typical manifestation of the intrinsic data uncertainty, also referred to as aleatoric uncertainty @xcite , that arises from the natural complexity of emotion data. it is common to replace such inconsistent labels with deterministic labels obtained by majority voting @xcite or (weighted) averages @xcite @xcite . however, this causes a loss of data samples when a majority agreed emotion class doesn't exist @xcite @xcite and also ignores the discrepancies between annotators and the aleatoric uncertainty in emotion data. in this paper, we propose to model the uncertainty in emotion attributes with a bayesian approach based on deep evidential regression @xcite , denoted deep evidential emotion regression (deer). in deer, the inconsistent human labels of each utterance are considered as observations drawn independently from an unknown gaussian distribution. to probabilistically estimate the mean and variance of the gaussian distribution, a normal inverse-gamma (nig) prior is introduced, which places a gaussian prior over the mean and an inverse-gamma prior over the variance. the aer system is trained to predict the hyper-parameters of the nig prior for each utterance by maximising the per-observation-based marginal likelihood of each observed label under this prior. as a result, deer not only models the distribution of emotion attributes but also learns both the aleatoric uncertainty and the epistemic uncertainty @xcite without repeating the inference procedure for sampling. epistemic uncertainty, also known as model uncertainty, is associated with uncertainty in model parameters that best explain the observed data. aleatoric and epistemic uncertainty are combined to induce the total uncertainty, also called predictive uncertainty, that measures the confidence of attribute predictions. as a further improvement, a novel regulariser is proposed based on the mean and variance of the observed labels to better calibrate the uncertainty estimation. the proposed methods were evaluated on the msp-podcast and iemocap datasets. the rest of the paper is organised as follows. section 2 summarises related work. section 3 introduces the proposed deer approach. sections 4 and 5 present the experimental setup and results respectively, followed by the conclusion. ## related work there has been previous work by aer researchers to address the issue of inconsistent labels. for emotion categories, a single ground-truth label can be obtained as either a continuous-valued mean vector representing emotion intensities @xcite , or as a multi-hot vector obtained based on the existence of emotions @xcite . recently, distribution-based approaches have been proposed, which consider the labels as samples drawn from emotion distributions @xcite . for emotion attributes, annotators often assign different values to the same attribute of each utterance. @xcite proposed a multiannotator model which contains multiple heads to predict each annotator's judgement. this approach is computationally viable only when the number of annotators is relatively small. the method requires sufficient annotations from each annotator to be effective. @xcite derived confidence measures based on annotator agreement to build emotion-scoring models. @xcite proposed predicting the standard deviation of the attribute label values as an extra task in the multitask training framework. @xcite included annotator variability as a representation of uncertainty in a gaussian mixture regression model. these techniques take the variance of human annotations either as an extra target or as an extra input. more recently, bayesian deep learning has been introduced to the task, which models the uncertainty in emotion annotation without explicitly using the variance of human annotations. these include the use of gaussian processes @xcite , variational auto-encoders @xcite , bayesian neural networks @xcite , monte-carlo dropout @xcite and sequential monte-carlo methods @xcite . so far, these methods have not distinguished aleatoric uncertainty from epistemic uncertainty which are defined in the introduction. our proposed deer approach can simultaneously model these two uncertainties. in addition, our approach is more generic. it has no limits on the number of annotators, the number of annotators per utterance, and the number of annotations per annotator, and thus can cope with large crowd-sourced datasets. ## conclusions two types of uncertainty exist in aer: (i) aleatoric uncertainty arising from the inherent ambiguity of emotion and personal variations in emotion expres-sion; (ii) epistemic uncertainty associated with the estimated network parameters given the observed data. this paper proposes deer for estimating those uncertainties in emotion attributes. treating observed attribute-based annotations as samples drawn from a gaussian distribution, deer places a normal-inverse gamma (nig) prior over the gaussian likelihood. a novel training loss is proposed which combines a per-observation-based nll loss with a regulariser on both the mean and the variance of the gaussian likelihood. experiments on the msp-podcast and iemocap datasets show that deer can produce state-of-the-art results in estimating both the mean value and the distribution of emotion attributes. the use of nig, the conjugate prior to the gaussian distribution, leads to tractable analytic computation of the marginal likelihood as well as aleatoric and epistemic uncertainty associated with attribute prediction. uncertainty estimation is analysed by visualisation and a reject option. beyond the scope of aer, deer could also be applied to other tasks with subjective evaluations yielding inconsistent labels.