M ulti EMO : An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations (2023)

Abstract: Emotion Recognition in Conversations (ERC) is an increasingly popular task in the Natural Language Processing community, which seeks to achieve accurate emotion classifications of utterances expressed by speakers during a conversation. Most existing approaches focus on modeling speaker and contextual information based on the textual modality, while the complementarity of multimodal information has not been well leveraged, few current methods have sufficiently captured the complex correlations and mapping relationships across different modalities. Furthermore, existing state-of-the-art ERC models have difficulty classifying minority and semantically similar emotion categories. To address these challenges, we propose a novel attention-based correlation-aware multimodal fusion framework named MultiEMO, which effectively integrates multimodal cues by capturing cross-modal mapping relationships across textual, audio and visual modalities based on bidirectional multi-head cross-attention layers. The difficulty of recognizing minority and semantically hard-to-distinguish emotion classes is alleviated by our proposed Sample-Weighted Focal Contrastive (SWFC) loss. Extensive experiments on two benchmark ERC datasets demonstrate that our MultiEMO framework consistently outperforms existing state-of-the-art approaches in all emotion categories on both datasets, the improvements in minority and semantically similar emotions are especially significant.

## introduction emotion recognition in conversations (erc) is an emerging task in the field of natural language processing (nlp), which aims to identify the emotion of each utterance in a conversation based on textual, audio and visual cues of the speaker. erc has attracted an enormous amount of attention from both academia and industry, due to its widespread to solve the problem of erc, numerous approaches have been proposed. the majority of existing works concentrate on modeling speaker dependencies and conversational contexts @xcite @xcite @xcite @xcite @xcite , while there still exist several unsolved challenges: (1) the complementarity of multimodal information has not been well exploited. apart from rich information contained in the textual modality, the tone and intonation of the speaker can indicate the intensity of the emotion, facial expressions of interlocutors are also able to explicitly reveal emotional tendencies @xcite . figure 1 shows an example where the complementarity of acoustic and visual signals in addition to the textual modality is essential for an accurate emotion classification. nevertheless, most existing approaches focus on the textual modality of utterances or simply utilize feature concatenation as the multimodal fusion mechanism @xcite @xcite @xcite without modeling the complicated correlations and mapping relationships across textual, audio and visual modalities, which results in an inadequate integration of multimodal cues. (2) unsatisfactory performances in minority emotion classes. existing benchmark datasets in erc, such as iemocap @xcite and meld @xcite , suffer from the problem of imbalanced classes. as illustrated in figure 2 , both meld and iemocap are class-imbalanced, especially in meld, where the majority class neutral takes up a much larger proportion than minority classes disgust and fear. current state-of-the-art approaches fail to solve the class imbalance problem and have poor perfor-mances in minority emotions. (3) the difficulty of distinguishing between semantically similar emotions. it remains to be a challenging task to correctly classify different emotions that are semantically related, such as disgust and anger in meld, since they share similar underlying cognitive, affective and physiological features, and tend to be expressed by speakers in similar contexts. to address the above problems, in this paper, we propose a novel attention-based correlationaware multimodal fusion framework named mul-tiemo. firstly, unimodal feature extraction and context modeling are performed for each modality, in which we introduce a visual feature extractor named visextnet based on a multi-task cascaded convolutional network (mtcnn) @xcite and a vggface2 @xcite . visextnet accurately captures visual cues of utterance videos by extracting emotion-rich facial expressions of interlocutors without modeling redundant scenerelated visual information. secondly, we propose a multimodal fusion model called multiattn to effectively integrate multimodal information based on bidirectional multi-head cross-attention layers @xcite , which successfully captures complex cross-modal correlations and mapping relationships across contextualized textual, audio and visual features. thirdly, in order to mitigate the difficulty of classifying minority and semantically similar emotion classes, enlightened by focal contrastive loss @xcite , a sample-weighted focal contrastive (swfc) loss is proposed, in which we assign more focus to hard-to-classify minority classes and make sample pairs with different emotion labels mutually exclusive with each other such that semantically similar emotions can be better distinguished. in addition, we utilize a soft hirschfeld-gebelein-rényi (soft-hgr) loss @xcite to maximize the correlations across multimodal-fused textual, audio and visual feature representations extracted from multiattn. finally, extensive experiments are conducted on two erc benchmark datasets, meld and iemocap. experimental results demonstrate the effectiveness and superiority of our proposed multiemo framework compared with existing state-of-the-art approaches, the improvements in minority and semantically similar emotion categories are especially remarkable. the main contributions of this work can be summarized as follows: