U ni MSE : Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022)

Abstract: Multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) are key research topics for computers to understand human behaviors. From a psychological perspective, emotions are the expression of affect or feelings during a short period, while sentiments are formed and held for a longer period. However, most existing works study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. In this paper, we propose a multimodal sentiment knowledge-sharing framework (UniMSE) that unifies MSA and ERC tasks from features, labels, and models. We perform modality fusion at the syntactic and semantic levels and introduce contrastive learning between modalities and samples to better capture the difference and consistency between sentiments and emotions. Experiments on four public benchmark datasets, MOSI, MOSEI, MELD, and IEMOCAP, demonstrate the effectiveness of the proposed method and achieve consistent improvements compared with state-of-the-art methods.

## introduction with the rapid development of multimodal machine learning @xcite and dialog system @xcite , multimodal sentiment analysis (msa) and emotion recognition in conversations (erc) have become the keys for machines to perceive, recognize, and understand human behaviors and intents @xcite . multimodal data provides not only verbal information, such as textual (spoken words) features but also non-verbal information, including acoustic (prosody, rhythm, pitch) and visual (facial attributes) features. these different modalities allow the machine to make decisions from different perspectives, thereby achieving more accurate predictions @xcite . the goal of msa is to predict sentiment intensity * corresponding authors. or polarity, and erc aims to predict predefined emotion categories. there are many research directions in msa and erc, such as multimodal fusion @xcite , modal alignment @xcite , context modeling @xcite and external knowledge @xcite . however, most existing works treat msa and erc as separate tasks, ignoring the similarities and complementarities between sentiments and emotions. on the one hand, from a psychological perspective, both sentiments and emotions are experiences that result from the combined influences of the biological, cognitive, and social @xcite , and could be expressed similarly. in figure 1 , we illustrate how sentiments and emotions are relevant in the verbal or non-verbal, and could be projected into a unified embedding space. on the other hand, emotions are reflections of the perceived change in the present within a short period @xcite , while sentiments are held and formed in longer periods @xcite . in our preliminary study, we found that the video dura-tion of msa is almost twice of erc 1 , which is consistent with the above definitions. a variety of psychological literature @xcite @xcite explain the similarities and differences between sentiment and emotion. @xcite also investigates the relevance and complementarity between the two and point out that analyzing sentiment and emotion together could better understand human behaviors. based on the above motivation, we propose a multimodal sentiment knowledge-sharing framework that unified msa and erc (unimse) tasks. unimse reformulates msa and erc as a generative task to unify input, output, and task. we extract and unify audio and video features and formalize msa and erc labels into universal labels (ul) to unify sentiment and emotion. besides, previous works on multimodal fusion at multi-level textual features @xcite , like syntax and semantics, are lacking. therefore, we propose a pre-trained modality fusion layer (pmf) and embed it in transformer @xcite layers of t5 @xcite , which fuses the acoustic and visual information with different level textual features for probing richer information. last but not least, we perform inter-modal contrastive learning (cl) to minimize intra-class variance and maximize interclass variance across modalities. our contributions are summarized as follows: tsai et al. (2019a) and luo et al. (2021) leverage cross-modality and multi-scale modality representation to implement modal alignment, respectively. lastly, studies of multimodal context integrate the unimodal context, in which chauhan et al. (2019) adapts context-aware attention, ghosal et al. (2018) uses multi-modal attention, and poria et al. (2017) proposes a recurrent model with multi-level multiple attentions to capture contextual information among utterances. ## conclusion this paper provides a psychological perspective to demonstrate that jointly modeling sentiment and emotion is feasible and reasonable. we present a unified multimodal knowledge-sharing framework, unimse, to solve msa and erc tasks. unimse not only captures knowledge of sentiment and emotion, but also aligns the input features and output labels. moreover, we fuse acoustic and visual modal representation with multi-level textual features and introduce inter-modality contrastive learning. we conduct extensive experiments on four benchmark datasets and achieve sota results in all metrics. we also provide the visualization of multimodal representation, proving the relevance of sentiment and emotion in embedding space. we believe this work presents a new experimental setting that can provide a new and different perspective to the msa and erc research communities.