Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition (2023)

Abstract: The multimodal emotion recognition in conversation task aims to predict the emotion label for a given utterance with its context and multiple modalities. Existing approaches achieve good results but also suffer from the following two limitations: 1) lacking modeling of diverse dependency ranges, i.e., long, short, and independent context-specific representations and without consideration of the different recognition difficulty for each utterance; 2) consistent treatment of the contribution for various modalities. To address the above challenges, we propose the Self-adaptive Context and Modal-interaction Modeling (SCMM) framework. We first design the context representation module, which consists of three submodules to model multiple contextual representations. Thereafter, we propose the modal-interaction module, including three interaction submodules to make full use of each modality. Finally, we come up with a self-adaptive path selection module to select an appropriate path in each module and integrate the features to obtain the final representation. Extensive experiments under four settings on three multimodal datasets, including IEMOCAP, MELD, and MOSEI, demonstrate that our proposed method outperforms the state-of-the-art approaches.

## introduction emotion is a crucial part of human conversation. the emotion recognition in conversation task is to analyze each utterance in a conversation and give the corresponding emotion. this task has recently received more and more attention from researchers in both nlp and multimodal fields because of its potential applications, such as humancomputer interaction and opinion mining in social media @xcite . traditional emotion recognition in conversation paradigms is either based on unrelated utterances in a dialogue or a single modality, such * jianlong wu is the corresponding author. figure 1 : motivation of the proposed method. this is an example from the iemocap dataset that contains three different kinds of context dependencies, including long, short, and independent dependency, which is marked by red arrow, green arrow, and blue square, respectively. besides, the primary modality for the final prediction varies for different samples. as text. however, in many cases, people's emotions are elusive and cannot be delivered well by just one utterance or a single modality. as multimodality is closer to real-world application scenarios, multimodal emotion recognition in conversation is gaining increasing research attention in recent years. to identify emotions more accurately, dialoguernn @xcite first designs an rnn-based model which includes four grus to model both intra-and inter-speaker relations. dialoguegcn @xcite then uses a graph neural network to model conversations. later, mmgcn @xcite proposes a graph-based method under the additional multimodal setting. although pioneer research studies have achieved promising progress, they mainly ignore the varying difficulty of each utterance for the model to recognize and multimodal interaction in conversa-tion, which leads to the following two limitations. first, existing methods treat all samples equally without considering their specific characteristic or difficulty for recognition. for example, they lack detailed modeling of diverse dependency ranges, i.e., long, short, and independent context-specific representations for each utterance. as illustrated in figure 1 , some utterances in a conversation require a long-range dependency, while others only require a short-range dependency or can determine the emotion on their own. existing methods do not consider respectively modeling these varying dependency ranges. second, current approaches regard the contribution of each modality equally and simply concatenate the features of different modalities. however, the contribution of each modality varies and it is of great importance to investigate the correlation and interaction among different modalities. in particular, figure 1 illustrates the different contributions among modalities for different utterances, where the primary modality for recognition varies from case to case. we argue the necessity to explore the modality-specific contributions. towards the above issues, we propose the selfadaptive context and modal-interaction modeling (scmm) method for multimodal emotion recognition. first, to model different ranges of context dependency, we design the context representation module, which consists of three submodules, including global, local, and direct mapping. second, towards the different contributions of various modalities, we propose the modal-interaction module, which also contains three submodules, including full, partial, and biased interaction, to investigate the correlation among them. thereafter, faced with multiple outputs from each module, we come up with the self-adaptive path selection strategy to adaptively select an appropriate path to obtain the final representation for each utterance. we also put forward a contrastive learning loss to learn more discriminative representations. finally, we conduct extensive experiments to validate the effectiveness of our approach. our main contributions are four-fold: ## conclusion in this paper, for the task of multimodal emotion recognition, we propose the self-adaptive contextual and modal-interaction modeling method. we first come up with the context representation module with global, local modeling and direct mapping to solve the issue of long, short, and independent dependency. then the modal-interaction consists of full, partial, and bias interactions to fully investigate the correlation and potential complementarity among different modalities. then we propose the self-adaptive path selection module for better combination and cross-modal contrastive learning loss for discriminative feature learning. extensive experiments on three datasets under four settings have demonstrated the effectiveness and superiority of our proposed method.