Learning More from Mixed Emotions: A Label Refinement Method for Emotion Recognition in Conversations (2023)

Abstract: One-hot labels are commonly employed as ground truth in Emotion Recognition in Conversations (ERC). However, this approach may not fully encompass all the emotions conveyed in a single utterance, leading to suboptimal performance. Regrettably, current ERC datasets lack comprehensive emotionally distributed labels. To address this issue, we propose the Emotion Label Refinement (EmoLR) method, which utilizes context- and speaker-sensitive information to infer mixed emotional labels. EmoLR comprises an Emotion Predictor (EP) module and a Label Refinement (LR) module. The EP module recognizes emotions and provides context/speaker states for the LR module. Subsequently, the LR module calculates the similarity between these states and ground-truth labels, generating a refined label distribution (RLD). The RLD captures a more comprehensive range of emotions than the original one-hot labels. These refined labels are then used for model training in place of the one-hot labels. Experimental results on three public conversational datasets demonstrate that our EmoLR achieves state-of-the-art performance.

## introduction emotion recognition in conversations (erc) is an important research topic with broad applications, including human-computer interaction @xcite , opinion mining @xcite , and intent recognition @xcite . unlike vanilla text emotion detection, erc models need to model context-and speakersensitive dependencies @xcite to simulate the interactive nature of conversations. recurrent neural networks (rnns) @xcite and their variants have been successfully applied for erc. recently, erc research has primarily focused on understanding the influence of internal/external factors on emotions in conversations, such as topics @xcite , commonsense @xcite , causal relations @xcite , and intent @xcite . these efforts have improved the model's understanding of the semantic structure and meaning of conversations. however, despite advancements in modeling context information @xcite , the final classification paradigm in erc remains the same: calculating a certain loss between predicted probability distribution and one-hot labels. this black-and-white learning paradigm has the following problem: according to plutchik's wheel of emotions @xcite and the hourglass model @xcite , emotions expressed in dialogue are often mixed expressions that include various basic emotions (such as anger, sadness, fear, and happiness) @xcite . each basic emotion contributes to the overall emotional expression to some extent. however, the one-hot label representation assumes emotions are independent. in real scenarios, as shown in figure 1 , the dialogue expression is normally not a single emotion, but multiple emotions are presented with a specific distribution. particularly when dealing with mixed and ambiguous emotions, one-hot vectors as labels are insufficient. they will overlook the emotional information within the utterance, which may lead to suboptimal performance of the erc models. to address the limitations of one-hot labels, several efforts have been proposed in the fields of text classification and object detection, such as label smoothing (ls) @xcite and label distribution learning (ldl) @xcite . ls partially alleviates the problem by randomly injecting noise, but it still cannot fundamentally recover the inherent emotional distribution within an utterance. ldl is capable of handling instances with multiple labels quantitatively, making it advantageous for tasks involving fuzzy labels @xcite . however, obtaining the true label distribution for structured conversation data is challenging. based on this, we propose the emotion label refinement (emolr) method based on contextand speaker-sensitive information. emolr consists of two components: the emotion predictor (ep) and the label refinement (lr). the ep module preserves context-and speaker-related representations during the emotion detection process. in the lr module, the context-and speakersensitive representations are compared with each label to estimate their correlation individually. the refined label is generated based on the correlation scores. the original one-hot label is combined with the new label distribution to reduce interference from noise in the model. the final output, after softmax activation, is used for training. the refined label, capturing the relationship between the speaker and contextual information, reflects all possible emotions to varying degrees. this distribution provides a more comprehensive representation of the utterance's emotional state compared to the ground-truth label. it enables models to learn more label-related information, leading to improved performance in dialogue emotion analysis. our main contributions are summarized as follows: ## conclusion in this paper, we propose the emolr method to adaptively generate a refined label distribution that quantitatively describes emotional intensity. rld guides the model to learn more label-related knowledge and capture more comprehensive semantic information. our proposed method influences rld through context-and speaker-sensitive states without requiring external knowledge or changing the original model structure. emolr has been proven effective on both unimodality and multimodality data through extensive experimental analysis and outperforms state-of-the-art results on three datasets. future work on emolr should address rld's noise generation with unrelated emotions and reduce interference. additionally, exploring efficient methods to encode labels in multimodal settings can shed light on mixed emotions' relevance.