Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimoda Emotion Recognition (2023)

Abstract: Multimodal emotion recognition aims to recognize emotions for each utterance from multiple modalities, which has received increasing attention for its application in human-machine interaction. Current graph-based methods fail to simultaneously depict global contextual features and local diverse uni-modal features in a dialogue. Furthermore, with the number of graph layers increasing, they easily fall into over-smoothing. In this paper, we propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition (Joyful), where multimodality fusion, contrastive learning, and emotion recognition are jointly optimized. Specifically, we first design a new multimodal fusion mechanism that can provide deep interaction and fusion between the global contextual and uni-modal specific features. Then, we introduce a graph contrastive learning framework with inter- and intra-view contrastive losses to learn more distinguishable representations for samples with different sentiments. Extensive experiments on three benchmark datasets indicate that Joyful achieved state-of-the-art (SOTA) performance compared with all baselines. Code is released on Github (https://anonymous.4open.science/r/MERC-7F88).

## methodology figure 2 shows an overview of joyful, which mainly consists of four components: (a) a unimodal extractor, (b) a multimodal fusion (mf) module, (c) a graph contrastive learning module, and (d) a classifier. hereafter, we give formal notations and the task definition of joyful, and introduce each component subsequently in detail. ## experiments and result analysis baselines. different merc datasets have different best system results, following cogmen, we selected sota baselines for each dataset. for iemocap-4, we selected mult @xcite , raven @xcite , mtag @xcite , pmr @xcite , cog-men and mica @xcite as our baselines. for iemocap-6, we selected mult, fe2e @xcite , diarnn @xcite , cosmic @xcite , af-can @xcite , aghmn @xcite , cogmen and rgat @xcite as our baselines. for meld, we selected diagcn @xcite , diacrn @xcite , mmgcn @xcite , unimse @xcite , cogmen and mm-dfn @xcite as baselines. for mosei, we selected mul-net @xcite , tbje @xcite , cogmen and mr @xcite . ## conclusion we proposed a joint learning model (joyful) for merc, that involves a new multimodal fusion mechanism and gcl module to effectively improve the performance of merc. the mr mechanism can extract and fuse contextual and uni-modal specific emotion features, and the gcl module can help learn more distinguishable representations. for future work, we plan to investigate the performance of using supervised gcl for joyful on unbalanced and small-scale emotional datasets.