CLMLF :A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection (2022)

Abstract: Compared with unimodal data, multimodal data can provide more features to help the model analyze the sentiment of data. Previous research works rarely consider token-level feature fusion, and few works explore learning the common features related to sentiment in multimodal data to help the model fuse multimodal features. In this paper, we propose a Contrastive Learning and Multi-Layer Fusion (CLMLF) method for multimodal sentiment detection. Specifically, we first encode text and image to obtain hidden representations, and then use a multi-layer fusion module to align and fuse the token-level features of text and image. In addition to the sentiment analysis task, we also designed two contrastive learning tasks, label based contrastive learning and data based contrastive learning tasks, which will help the model learn common features related to sentiment in multimodal data. Extensive experiments conducted on three publicly available multimodal datasets demonstrate the effectiveness of our approach for multimodal sentiment detection compared with existing methods. The codes are available for use at https: //github.com/Link-Li/CLMLF

## introduction with the development of social networking platforms which have become the main platform for people to share their personal opinions. how to extract and analyze sentiments in social media data efficiently and correctly has broad applications. therefore, it has attracted attention from both academic and industrial communities @xcite . at the same time, with the increasing use of mobile internet and smartphones, more and more users are willing to post multimodal data (e.g., text, image, and video) about different topics to convey their feelings and sentiments. so multimodal sentiment analysis has become a popular research topic @xcite . as for multimodal data, the complementarity between text and image can help the model analyze the real sentiment of the multimodal data. as shown in figure 1 , detecting sentiment with only text modality or image modality may not be certain of the true intention of the tweet. such as figure 1a , if we only analyze the text modality, we will find that this is a declarative sentence that does not express sentiment. in fact, the girl's smile in the image shows that the sentiment of this tweet is positive. at the same time, in figure 1c , we can find that the ruins in the image which deepen the expression of negative sentiment in the text. for multimodal sentiment analysis, we focus on text-image sentiment analysis in social media data. in existing works, some models try to concatenate different modal feature vectors to fuse the multimodal features, such as multisentinet @xcite and hsan @xcite . @xcite proposes to use gating mechanism and attention to obtain deep multimodal contextual feature vectors. multi-view attentional network (mvan) is proposed by @xcite which introduces memory networks to realize the interaction between modalities. although the above mentioned models are relatively better than unimodal models, the inputs with different modalities are in different vector spaces. therefore, it is difficult to fuse multimodal data with a simple concatenation strategy, so the improvement is also limited. furthermore, the gating mechanism and memory network are essentially not designed for multimodal fusion. although they can help the model analyzes the sentiment in the multimodal data by storing and filtering the features in the data, it is obvious that these methods are difficult to align and fuse the features of text and image. since transformers have achieved great success in many fields, such as natural language processing and computer vision @xcite , we propose multi-layer fusion (mlf) module based on transformer-encoder. benefiting from the multi-headed self-attention in transformer, which can capture the internal correlation of data vectors. therefore, text tokens and image patches with explicit and implicit relationships will have higher attention weight allocation to each other which means the mlf module can help align and fuse the token-level text and image features better. and mlf is a multi-layer encoder, which can help improve the abstraction ability of the model and obtain deep features in multimodal data. some previous work has explored the application of contrastive learning in the multimodal field. @xcite proposes the application of contrastive learning in multilingual text-to-video search, and @xcite applies contrastive learning to learn visual representations that embraces multimodal data. however, there is little work to study the application of contrastive learning in multimodal sentiment analysis, so we propose two contrastive learning tasks, label based contrastive learning (lbcl) and data based contrastive learning (dbcl), which will help the model learn common features related to sentiment in multimodal data. for example, as shown in figure 1a and figure 1b . we can find that both tweets show positive sentiment. and we also can find there are smiling expressions in the image of the two tweets which is a common feature of those tweets. if the model can learn common features related to sentiment, it will greatly improve the performance of the model. in this paper, we propose a contrastive learning and multi-layer fusion (clmlf) method for multimodal sentiment analysis based on text and image modalities. for evaluation, clmlf is verified on three multimodal sentiment datasets, namely mvsa-single, mvsa-multiple @xcite and hfm @xcite . clmlf achieves better performance compared to several baseline models in all three datasets. through a comprehensive set of ablation experiments, case study, and visualizations, we demonstrate the advantages of clmlf for multimodal fusion foot_0 . our main contributions are summarized as follows: those two contrastive learning tasks can help the model learn common features related to sentiment in multimodal data, which improve the performance of the model. ## conclusion and future work in this paper, we propose a contrastive learning and multi-layer fusion method for multimodal sentiment detection. compared with previous works, our proposed mlf module performs multimodal feature fusion from the fine-grained token-level, which is more conducive to the fusion of local features of text and image. at the same time, we design learning tasks based on contrastive learning to help the model learn sentiment related features in the multimodal data and improve the ability of the model to extract and fuse features of multimodal data. the experimental results on public datasets demonstrate that our proposed model is competitive with strong baseline models. especially through visualization, the contrastive learning tasks and multi-layer fusion module we proposed can be ver-ified with intuitive interpretations. in future work, we will incorporate other modalities such as audio into the sentiment detection task.