MMGCN : Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation (2021)

Abstract: Emotion recognition in conversation (ERC) is a crucial component in affective dialogue systems, which helps the system understand users’ emotions and generate empathetic responses. However, most works focus on modeling speaker and contextual information primarily on the textual modality or simply leveraging multimodal information through feature concatenation. In order to explore a more effective way of utilizing both multimodal and long-distance contextual information, we propose a new model based on multimodal fused graph convolutional network, MMGCN, in this work. MMGCN can not only make use of multimodal dependencies effectively, but also leverage speaker information to model inter-speaker and intra-speaker dependency. We evaluate our proposed model on two public benchmark datasets, IEMOCAP and MELD, and the results prove the effectiveness of MMGCN, which outperforms other SOTA methods by a significant margin under the multimodal conversation setting.

## introduction emotion is an important part of human daily communication. emotion recognition in conversation (erc) aims to automatically identify and track the emotional status of speakers during a dialogue. it has attracted increasing attention from researchers in the field of natural language processing and multimodal processing. erc has a wide range of potential applications such as assisting conversation analysis for legal trials and e-health services etc. it is also a key component for building natural humancomputer interactions that can produce emotional responses in a dialogue. the fast growing availability of conversational data on social media is one of the factors that boost the research focus on emotion recognition in conversation. different from traditional emotion recognition on isolated utterances, emotion recognition in conversation requires context modeling of individual utterances. the context can be attributed to the preceding utterances, temporality in conversation turns, or speaker related information etc. different models have been proposed to capture the contextual information in previous works, including the lstm-based model @xcite , the conversational memory network (cmn) model @xcite , interactive conversational memory network (icon) model @xcite , and dialoguernn model @xcite ) etc. in the example conversation as shown in figure 1 , the two speakers are chatting in the context of the male speaker being admitted to usc. in this chatting scene, they change topics a few times, such as the female speaker inviting the male speaker out to play and so on. but they keep coming back to the topic of usc, and then both of them express an excitement emotional status. it shows that long-distance contextual information is of great help to the prediction of speakers' emotions. however, previous models can not effectively capture both speaker and long-distance dialogue contextual information simultaneously in multi-speaker conversation scenarios. ghosal et al. @xcite , therefore, first propose the dialoguegcn model which applies graph convolutional network (gcn) to capture long-distance contextual information in a conversation. dia-loguegcn takes each utterance as a node and connects any nodes that are in the same window within a conversation. it can well model both the dialogue context and speaker information which leads to the state-of-the-art erc performance. however, like most previous models, dialoggcn only focuses on the textual modality of the conversation, ignoring effective combination of other modalities such as visual and acoustic modalities. works that consider multimodal contextual information often conduct the simple feature concatenation type of multimodal fusion. in order to effectively explore the multimodal information and at the same time capture longdistance contextual information, we propose a new multimodal fused graph convolutional network (mmgcn) model in this work. mmgcn constructs the fully connected graph in each modality, and builds edge connections between nodes corresponding to the same utterance across different modalities, so that contextual information across different modalities can interact. in addition, the speaker information is injected into mmgcn via speaker embedding. furthermore, different from dialoguegcn, which is a non-spectral domain gcn and its many optimized matrices occupy too much computing resource, we encode the multimodal graph using spectral domain gcn and extend the gcn from a single layer to deep layers. to verify the effectiveness of the proposed model, we carry out experiments on two benchmark multimodal conversation datasets, iemocap and meld. mmgcn significantly outperforms other models on both datasets. the rest of the paper is organized as follows: section 2 discusses some related works; section 3 introduces the proposed mmgcn model in details; section 4 and 5 present the experiment setups on two public benchmark datasets and the analysis of experiment results and ablation study; finally, section 6 draws some conclusions. ## method a dialogue can be defined as a sequence of utterances {u 1 , u 2 , ..., u n }, where n is the number of utterances. each utterance involves three sources of utterance-aligned data corresponding to three modalities, including acoustic (a), visual (v) and textual (t) modalities, which can be represented as follows: ## results and discussions we compare our proposed mmgcn with all the baseline models presented in section 4.5 on iemo-cap and meld datasets under the multimodal setting. in order to compare the results under the same experiment settings, we reimplement the models in the following experiments. ## conclusion in this paper, we propose an multimodal fused graph convolutional network (mmgcn) for multimodal emotion recognition in conversation (erc). mmgcn provides a more effective way of utilizing both multimodal and long-distance contextual information. it constructs a graph that captures not only intra-speaker context dependency but also inter-modality dependency. with the residual connection, mmgcn can have deep layers to further improve recognition performance. we carry out experiments on two public benchmark datasets, iemocap and meld, and the experiment results prove the effectiveness of mmgcn, which outperforms other state-of-the-art methods by a significant margin under the multimodal conversation setting.