A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation (2023)

Abstract: Emotion recognition in conversation (ERC) has attracted enormous attention for its applications in empathetic dialogue systems. However, most previous researches simply concatenate multimodal representations, leading to an accumulation of redundant information and a limited context interaction between modalities. Furthermore, they only consider simple contextual features ignoring semantic clues, resulting in an insufficient capture of the semantic coherence and consistency in conversations. To address these limitations, we propose a cross-modality context fusion and semantic refinement network (CMCF-SRNet). Specifically, we first design a cross-modal locality-constrained transformer to explore the multimodal interaction. Second, we investigate a graph-based semantic refinement transformer, which solves the limitation of insufficient semantic relationship information between utterances. Extensive experiments on two public benchmark datasets show the effectiveness of our proposed method compared with other state-of-the-art methods, indicating its potential application in emotion recognition. Our model will be available at https://github.com/zxiaohen/CMCF-SRNet .

## introduction emotion recognition in conversation (erc) plays an important role in affective dialogue systems, aiming to understand and generate empathetic responses @xcite . most studies on erc focus primarily on the textual modality @xcite . although they can be easily extended to multimodal paradigms by performing early or late fusion @xcite , it is difficult to capture contextual interactions between modalities, which limits the utilization of multiple modalities. for instance, a tensor fusion network based on the utterance-level explicit alignment learning both intra-modality and intermodality interactions via the cartesian product was * corresponding author fig. 1 . an example conversation between two speakers with corresponding emotions evoked for each utterance illustrating the importance of local context. however, these methods mostly use a simple concatenation ignoring complex interactions between modalities, resulting in leveraging context information insufficiently or the problem of data sparseness. besides, they simply consider the emotional impact of context in the whole conversation but neglect the emotional inertia of speakers and the fact that the local context may have a higher impact than long-distance utterances. as both the current utterance and the surrounding contexts are vital for the emotion perception, previous works proposed different methods including rnn-based models and graph-based models to explore contextual clues: a lstm-based model @xcite and an interactive conversational memory network icon @xcite capture interaction and history context, while dialoguernn model @xcite leverages distinct grus to capture speakers' contextual information. other popular approaches use graph-based neural networks and solve the context propagation issues in rnn-based architectures, including dia-loguegcn @xcite which first constructed the graph considering both speaker and conversation sequential information. recent approaches like dag-erc @xcite combined the advantages of conventional graph-based models and rnn-based models, a semantics gat was employed to adjust the weight of knowledge @xcite , latent correlations have been leveraged among the utterances through a multibranch graph network @xcite . meanwhile, existing gnn models use different aggregation schemes for a node to aggregate feature messages from its neighbors @xcite : graph convolutional networks use mean pooling while graph attention networks aggregate neighborhood information with trainable attention weights to capture local details @xcite . furthermore, graph networks consider global graph information during aggregation and have been used to explore the semantic relationship between regional objects and global concepts @xcite . however, the existing graph-based methods also have limitations. first, they mostly ignore the semantic similarity between context utterances leading to a lack of semantic correlation. second, these models learn node embeddings by capturing local network structure but ignore the position of the node within a broader context of the graph structure and the deep semantic features from a global view. to address this issue, we investigate a semantic graph-based transformer. in this work, we propose a cross-modality context fusion and semantic refinement network (cmcf-srnet). first, we investigate a crossmodality context fusion module to integrate textual and audio information considering the impact of the local context and the emotional inertia of speakers, achieved by a cross-modal locality-constrained attention. second, we design a semantic refinement module to extract effective semantic features and contextual information including the nearby surroundings and distant information. the main contributions can be summarized as follows: ## methodology given a dialogue @xmath0, u 2 , ..., u n }, where n denotes the number of utterances and u m is the m th utterance in the conversation, the emotion recognition in conversation task aims to predict the emotion label for each utterance in the conversation. each utterance involves two sources of data corresponding to acoustic (a) and textual (t) modalities represented as u @xmath1, u t m } where u ## conclusion in this paper, we propose a novel framework for multimodal emotion recognition which contains two innovative modules: the cross-modal locality-constrained context fusion leverages the transformer-based method to focus on localness, effectively improved the multimodal interaction. the semantic refinement module makes full use of the semantic relation information from a global view. experiments on two public datasets and the results demonstrate that our proposed cmcf-srnet is superior to the existing state-of-the-art methods. the ablation experiments prove the effectiveness of the two innovative modules. the detailed discussion shows that our proposed cmcf-srnet has satisfactory generalization ability and interpretability, indicating that it has the potential for practical use for emotion recognition.