Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction (2023)

Abstract: Emotion recognition is a crucial task for human conversation understanding. It becomes more challenging with the notion of multimodal data, e.g., language, voice, and facial expressions. As a typical solution, the global- and the local context information are exploited to predict the emotional label for every single sentence, i.e., utterance, in the dialogue. Specifically, the global representation could be captured via modeling of cross-modal interactions at the conversation level. The local one is often inferred using the temporal information of speakers or emotional shifts, which neglects vital factors at the utterance level. Additionally, most existing approaches take fused features of multiple modalities in an unified input without leveraging modality-specific representations. Motivating from these problems, we propose the Relational Temporal Graph Neural Network with Auxiliary Cross-Modality Interaction (CORECT), an novel neural network framework that effectively captures conversation-level cross-modality interactions and utterance-level temporal dependencies with the modality-specific manner for conversation understanding. Extensive experiments demonstrate the effectiveness of CORECT via its state-of-the-art results on the IEMOCAP and CMU-MOSEI datasets for the multimodal ERC task.

## introduction our social interactions and relationships are all influenced by emotions. given the transcript of a conversation and speaker information for each constituent utterance, the task of emotion recognition in conversations (erc) aims to identify the emotion expressed in each utterance from a predefined set of emotions @xcite . the multimodal nature of human communication, which involves verbal/textual, facial expressions, vocal/acoustic, bodily/postural, and symbolic/pictorial expressions, adds complexity to the task of emotion recognition in conversations (erc) @xcite . this isn't anything like i thought anything would be. this is just this… oh, sure this is standing on the beach, this is waiting, fighting. ## related works this section presents a literature review on multimodal emotion recognition (erc) and the application of graph neural networks for erc. ## methodology figure 2 illustrates the architecture of corect to tackle the multimodal erc task. it consists of main components namely relational temporal graph convolution network (rt-gcn) and pairwise cross-modal feature interaction. for a given utterance in a dialogue, the former is to learn the local-context representation via leveraging various topological relations between utterances and modalities, while the latter infers the cross-modal globalcontext representation from the whole dialogue. given a multi-speaker conversation c consisting of n utterances [u 1 , u 2 , . . . , u n ], let us denote s as the respective set of speakers. each utterance u i is associated with three modalities, including audio (a), visual (v), and textual (l), that can be represented as u a i , u v i , u l i respectively. using localand global context representations, the erc task aims to predict the label for every utterance u @xmath0, y 2 , . . . , y m ]. ## conclusion in this work, we propose corect, an novel network architecture for multimodal erc. it consists of two main components including rt-gcn and p-cm. the former helps to learn local-context representations by leveraging modality-level topological relations while the latter supports to infer cross-modal global-context representations from the entire dialogue. extensive experiments on two popular benchmark datasets, i.e., iemocap and cmu-mosei, demonstrate the effectiveness of corect, which achieves the new state-of-theart record for multimodal conversational emotion recognition. furthermore, we also provide ablation studies to investigate the contribution of various components in corect. interestingly, by analyzing the temporal aspect of conversations, we have validated that capturing the long-term dependencies, e.g., past relation, improves the performance of the multimodal emotion recognition in conversations task.