Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion Representation (2023)

Abstract: The effectiveness of a model is heavily reliant on the quality of the fusion representation of multiple modalities in multimodal sentiment analysis. Moreover, each modality is extracted from raw input and integrated with the rest to construct a multimodal representation. Although previous methods have proposed multimodal representations and achieved promising results, most of them focus on forming positive and negative pairs, neglecting the variation in sentiment scores within the same class. Additionally, they fail to capture the significance of unimodal representations in the fusion vector. To address these limitations, we introduce a framework called Supervised Angular-based Contrastive Learning for Multimodal Sentiment Analysis. This framework aims to enhance discrimination and generalizability of the multimodal representation and overcome biases in the fusion vector’s modality. Our experimental results, along with visualizations on two widely used datasets, demonstrate the effectiveness of our approach.

## introduction through internet video-sharing platforms, individuals engage in daily exchanges of thoughts, experiences, and reviews. as a result, there is a growing interest among scholars and businesses in studying subjectivity and sentiment within these opinion videos and recordings @xcite . consequently, the field of human multimodal language analysis has emerged as a developing area of research in natural language processing. moreover, human communication inherently encompasses multiple modalities, creating a heterogeneous environment characterized by the synchronous coordination of language, expressions, and audio modalities. multimodal learning leverages diverse sources of information, such as language (text/transcripts), * corresponding author as shown in a more pivotal role compared to the visual and audio modalities. while words undoubtedly carry significant emotional information (e.g., words like "good", "great", "bad"), emotions are conveyed not only through words but also other communication channels, e.g. tone of voice or facial expressions. we introduce a fusion scheme called triplet loss for triplet modality to enhance representations of partial modalities in the fusion vector. in our study, we found that incorporating facial expressions and tone of voice is crucial in determining communication-related scores. neglecting these modalities can lead to a loss of valuable information. for instance, in previous research often focused on contrastive learning techniques, forming positive and negative representation pairs, but overlooked distinctions within the same class. as illustrated in to address this challenge, we challenge the assumption that all positive samples should be represented similarly in the feature space. in the process of sampling positive and negative pairs, we take into account the sentiment score difference between each sample. when the difference in sentiment score does not exceed a predefined threshold, we consider the input as positive. we propose a supervised angular margin-based contrastive learning approach that enhances the discriminative representation by strengthening the margin within the angular space while capturing the varying degrees of sentiment within the samples. by not assuming uniform representation for all positive/negative samples in the feature space, our method effectively handles the differences between their sentiment levels. our contributions can be summarized as follows: this method enhances the discriminative representation of samples with varying degrees of sentiment, allowing for more accurate classification. ## related work recently, multimodal research has garnered attention because of its many applications and swift expansion @xcite @xcite @xcite . in particular, multimodal sentiment analysis (msa) predicts sentiment polarity across various data sources, like text, audio, and video. early fusion involves concatenating the features from different modalities into a single feature vector @xcite ). this unified feature vector is then employed as input. late fusion entails constructing separate models for each modality and subsequently combining their outputs to obtain a final result @xcite . some works such as @xcite and @xcite have explored the utilization of hybrid fusion techniques that integrate both early and late fusion methods. in addition, mag-bert model @xcite combines bert and xlnet with a multimodal adaptation gate to incorporate multimodal nonverbal data during finetuning. besides, @xcite introduced a novel capsule network-based method, enabling the dynamic adjustment of weights between modalities. mmim @xcite preserves taskrelated information through hierarchical maximization of mutual information (mi) between pairs of unimodal inputs and the resulting fusion of multiple modalities. recently, several works @xcite @xcite have focused on uncertainly solving the missing modalities problem. contrastive learning @xcite has led to its wide adoption across various problems in the field and is widely applied in many applications @xcite . noteworthy training objectives such as n-pair loss @xcite , triplet margin loss @xcite , structured loss @xcite , and arccse @xcite , directly apply the principles of metric learning. in supervised downstream tasks, softmax-based objectives have shown promise by incorporating class centers and penalizing the distances between deep features and their respective centers. center loss @xcite , sphereface @xcite , cosface @xcite , and arcface @xcite have become popular choices in deep learning applications in computer vision and natural language processing. however, these loss functions are designed specifically for classification tasks and are not appropriate ## training objective in this section, we introduce the objective function of our model with two different optimization objectives: supervised angular margin-based contrastive learning and triplet modalities triplet loss. the model's overall training is accomplished by minimizing this objective: here, α, β is the interaction weights that determine the contribution of each regularization component to the overall loss l. l main is a main task loss, which is mean absolute error. each of these component losses is responsible for achieving the desired subspace properties. in the next two sections, we introduce two different optimization objectives: ## experimental setup in this part, we outline certain experimental aspects such as the datasets, metrics and feature extraction. ## conclusion in conclusion, our framework, supervised angularbased contrastive learning for multimodal sentiment analysis, addresses the limitations of existing methods by enhancing discrimination and generalization in the multimodal representation while mitigating biases in the fusion vector. through extensive experiments and visualizations, we have demonstrated the effectiveness of our approach in capturing the complexity of sentiment analysis. the publicly available implementation of our framework paves the way for further research and development in this field, benefiting applications such as opinion mining and social media analysis.