C on FEDE : Contrastive Feature Decomposition for Multimodal Sentiment Analysis (2023)

Abstract: Multimodal Sentiment Analysis aims to predict the sentiment of video content. Recent research suggests that multimodal sentiment analysis critically depends on learning a good representation of multimodal information, which should contain both modality-invariant representations that are consistent across modalities as well as modality-specific representations. In this paper, we propose ConFEDE, a unified learning framework that jointly performs contrastive representation learning and contrastive feature decomposition to enhance the representation of multimodal information. It decomposes each of the three modalities of a video sample, including text, video frames, and audio, into a similarity feature and a dissimilarity feature, which are learned by a contrastive relation centered around the text. We conducted extensive experiments on CH-SIMS, MOSI and MOSEI to evaluate various state-of-the-art multimodal sentiment analysis methods. Experimental results show that ConFEDE outperforms all baselines on these datasets on a range of metrics.

## introduction multimodal deep learning involves interpreting and analyzing multimodal signals together, where each modality refers to a way in which something is experienced and felt, e.g., the visual, audio, or language modality. with the widespread popularity of online social media, such as instagram, tik-tok, facebook, etc., videos containing multiple modalities have become a major information carrier, which brings new challenges to content recommendation and classification, e.g., video question answering @xcite , video captioning @xcite , and video retrieval @xcite . while traditional sentiment analysis is mainly based on language, multimodal sentiment analysis (msa) predicts the human emotion by utilizing extra information available in visual and audio modalities of the content to assist with language-based prediction. here, the text modality contains the semantic meaning of the spoken language. the visual modality extracts the facial characteristics (e.g., head orientation, facial expressions, and pose) of the speaker. the audio modality reflects the emphasis on the utterance (e.g., through pitch, bandwidth and intensity). msa has recently gained much attention in research for several reasons. on one hand, because of the abundance of social media content, commercial interests are switching from gauging user opinions/emotions from text only to more thorough multimodal analysis based on videos. on the other hand, short video platforms (e.g., tiktok, instagram) allow users to easily create multimodal content including visual information, audio, and inserted text, while these modalities are sometimes noisy or even contradicting each other in sentiments. therefore, the presence of multimodal information in addition to the text or language itself is necessary to make a thorough conclusion about the overall sentiment of a video. multimodal fusion has become essential to gaining a deeper understanding of these video scenes @xcite and has proven to be helpful in many downstream tasks. various multimodal fusion techniques have been proposed for msa, among which a basic solution is concatenating the extracted feature of each modality before performing downstream regression or classification. recent work has recognized the importance of identifying modality-invariant information across modalities and fuse them to strengthen sentiment prediction @xcite @xcite . although modality-invariant information helps reinforce the understanding of the content, there are also cases where sentiments of different modalities contradict each other. for example, when one thanks someone with phrases like "finally i can rest easy tonight" or "i can't thank you enough", it is very hard to conclude whether the sentiment is positive or negative without looking at the nonverbal cues, such as tones, facial expressions, and gestures. in fact, many sarcastic opinions are expressed by non-linguistic markers. in these cases, the overall sentiment cannot simply be judged by a majority vote among all modalities. thus, multimodal representation learning that respects both consistency and incongruity between modalities have recently shown great promise @xcite . in this paper, we propose confede, a contrastive feature decomposition framework, which integrates both modality decomposition within each sample and supervised contrastive learning across samples in a single unified contrastive learning framework. our main contributions are summarized as follows: (1) we integrate inter-sample contrastive learning and intra-sample modality decomposition into a simple unified loss function, based on a customized data sampler that allows us to sample positive/negative data pairs to perform both learning tasks. (2) we propose to decompose each modality into a similarity feature and a dissimilarity feature, and use the similarity feature of the text as an anchor to build the contrastive relation among all decomposed features. this is due to the observation that sentiment analysis is still largely centered around text and spoken language, while other modalities can provide extra information to assist with prediction. (3) based on multimodal representation learning proposed above, we further introduce a multi-task prediction loss that depends on each decomposed modality representation and enables the model to learn from both multimodal prediction and unimodal prediction. we mainly evaluated confede on ch-sims @xcite ) benchmark, which contains both unimodal and overall sentiment labels for each sample. the result shows that the proposed method significantly outperforms a wide range of stateof-the-art multimodal sentiment analysis methods. to test the capability when no unimodal labels are provided, we further conduct experiments on mosi @xcite and mosei @xcite , which contain only an overall sentiment label for each sample, which shows that our proposed method can also achieve better per-formance than state-of-the-art methods on a number of performance metrics without unimodal labels. we provide extensive ablation studies to show the effectiveness and necessity of each design component in confede. the code is released at https://github.com/xpastax/confede/ . ## related work in this section, we discuss the related work in msa and contrastive representation learning. ## methodology in this section, we introduce the overall architecture of confede followed by a detailed description of the contrastive feature decomposition process for learning multimodal representations. ## experiments we mainly evaluate confede on ch-sims @xcite , since it has unimodal labels, which can best meet the design of confede. to justify the effectiveness of confede when unimodal labels are unavailable, we further test confede on the mosi @xcite and mosei @xcite , which are two english msa datasets. however, they can not best test the performance of confede. we compare our methods with the state-of-theart baselines in compute the cosine similarity score: ## . for every sample @xmath1 retrieve the similar sample set s i 0 : retrieve the dissimilar sample set s i 1 : randomly select two samples from the last z elements of s i 0 as neighbor i ; randomly select two samples from the first z elements of s i 1 as outlier i 1 ; randomly select two samples from the last z elements of s i 1 as outlier i 2 ; ## conclusion in this paper, we propose a novel method for multimodal sentiment analysis (msa) called confede. the confede framework is based on contrastive feature decomposition, which utilizes a unified contrastive training loss to capture the consistency and difference across modalities and samples. this approach allows for the simultaneous learning of modality decomposition within each sample and su-pervised contrastive learning across samples. our proposed method is mainly evaluated on ch-sims. the result shows that the proposed method significantly outperforms many state-of-the-art multimodal sentiment analysis methods. we further conduct an extensive experiment on mosi and mosei to test the capability of confede when no unimodal label is available, where our method achieves better performance than state-of-the-art methods on a number of performance metrics.