CLGSI : A Multimodal Sentiment Analysis Framework based on Contrastive Learning Guided by Sentiment Intensity (2024)

Abstract: Recently, contrastive learning has begun to gain popularity in multimodal sentiment analysis (MSA). However, most of existing MSA methods based on contrastive learning lacks more detailed learning of the distribution of sample pairs with different sentiment intensity differences in the contrastive learning representation space. In addition, limited research has been conducted on the fusion of each modality representation obtained by contrastive learning training.In this paper, we propose a novel framework for multimodal sentiment analysis based on Contrastive Learning Guided by Sentiment Intensity (CLGSI). Firstly, the proposed contrastive learning guided by sentiment intensity selects positive and negative sample pairs based on the difference in sentiment intensity and assigns corresponding weights accordingly.Subsequently, we propose a new multimodal representation fusion mechanism, called Global-Local-Fine-Knowledge (GLFK), which extracts common features between different modalities’ representations. At the same time, each unimodal encoder output is separately processed by a Multilayer Perceptron (MLP) to extract specific features of each modality. Finally, joint learning of the common and specific features is used to predict sentiment intensity. The effectiveness of CLGSI is assessed on two English datasets, MOSI and MOSEI, as well as one Chinese dataset, SIMS. We achieve competitive experimental results, which attest to the strong generalization performance of our approach. The code for our approach will be released in https://github.com/AZYoung233/CLGSI

## introduction sentiment is one of the most important ways for human beings to perceive the world, and it can significantly affect human behavior and decisionmaking. msa aims to comprehensively analyze *corresponding author human sentiments by integrating and examining information from diverse modalities @xcite , such as text, video and audio. this integration and analysis enables machines to better understand and interpret human sentiments. due to the rapid advancements in multimedia and computer technologies, msa has garnered significant attention within the natural language processing (nlp) community @xcite @xcite . in recent times, contrastive learning has gained popularity in the field of msa. the msa approaches based on contrastive learning involve three significant issues: 1) the selection of positive and negative sample pairs, 2) the attention given to different positive and negative samples during the learning process, and 3) the integration of modality representations obtained after contrastive learning. several researchers have proposed solutions to address these issues. mai et al. @xcite first introduced contrastive learning in msa and proposed hycon. in their method, positive and negative sample pairs are first roughly divided using labels. during training, positive and negative sample pairs are dynamically selected based on the similarity across modalities. another approach, confede, was proposed by @xcite , who argued that the text is generally more effective than audio and video in msa. thus, confede selects sample pairs to be trained during the learning process by considering text similarity, and only selects 2 positive samples and 4 negative samples for each anchor. although the aforementioned methods have yielded promising results, they do not account for differences in sentiment intensity between samples. samples with sentiment intensity of -0.2 and -0.4 are likely to be treated as negative samples pair according to the pairs selection mechanism of hycon and confede. however, they still have similarities in terms of labels and should not be pushed away in the representation space. in contrast, conki, proposed by @xcite , can alleviate this problem to some extent by selecting positive and negative sample pairs using predefined sentiment intervals (e.g., positive, weak positive, etc.) in the dataset. moreover, the majority of existing studies treat the learning of different sample pairs equally and lack a detailed learning of the distribution of sample pairs with varying sentiment intensity differences in the representation space. nevertheless, in msa, it is crucial to assign distinct attention to sample pairs with differing sentiment intensity differences during the optimization process of contrastive learning. for example, take the negative sample pair a: {y 1 =-0.4, y 2 =+1} and b: {y 1 =-0.4, y 3 =+0.6}. it is obvious that a exhibits a larger sentiment intensity difference than b. therefore, it is necessary to pay more attention to a, that is, letting the two samples in a have a greater relative distance in the representation space. in addition, the modal representations obtained by contrastive learning training in the aforementioned studies are simply concatenated and fed into the mlp, which lacks further exploration of the integration of representation information, potentially restricting the model's generalization performance. considering the aforementioned limitations, we introduce a novel multimodal sentiment analysis framework based on contrastive learning guided by sentiment intensity (clgsi). our contributions are summarized as follows: ## conclusion in this paper, we propose clgsi, a novel msa method. firstly, clgsi uses the contrastive learning guided by sentiment intensity to project different modalities into the same representation space. then, by mimicking human cognitive process, glfk is used to extract the common features be-tween different modalities' representations. at the same time, the output of each modal encoder was processed separately by mlp to extract the specific features of each modality. finally, the joint learning of common and specific features was used to predict the sentiment intensity. we validate our model on both english and chinese datasets, and the competitive results prove the good generalization performance and effectiveness of our model.