Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis (2023)

Abstract: Multimodal Sentiment Analysis (MSA) has made great progress that benefits from extraordinary fusion scheme. However, there is a lack of labeled data, resulting in severe overfitting and poor generalization for supervised models applied in this field. In this paper, we propose Sentiment Knowledge Enhanced Self-supervised Learning (SKESL) to capture common sentimental patterns in unlabeled videos, which facilitates further learning on limited labeled data. Specifically, with the help of sentiment knowledge and non-verbal behavior, SKESL conducts sentiment word masking and predicts fine-grained word sentiment intensity, so as to embed sentiment information at the word level into pre-trained multimodal representation. In addition, a non-verbal injection method is also proposed to integrate non-verbal information into the word semantics. Experiments on two standard benchmarks of MSA clearly show that SKESL significantly outperforms the baseline, and achieves new State-Of-The-Art (SOTA) results.

## introduction multimodal sentiment analysis (msa) is a rapidly developing research field, which extends conventional text sentiment analysis to a multimodal setup where three modalities are present: text, audio and visual @xcite . with the abundance of user-generated opinion videos, msa has a wide range of applications in e-commerce, intelligent customer service, human-computer interaction, etc. in msa, the construction of the training dataset relies on artificial perceptual evaluation for the sentiment of opinion videos, which is a very timeconsuming and labor-intensive task, that is why video data with sentimental annotation is insufficient. as a result, supervised models applied in this field suffer from severe overfitting and poor generalization @xcite . although previous studies have used several methods to alleviate the overfitting, most of them are based on general approaches such as multi-task learning @xcite @xcite , parameter regularization @xcite and data augmentation @xcite , which neglect to consider the large number of unlabeled opinion videos that naturally exist on the internet. these opinion videos contain common sentimental patterns or compositional sentiment semantics about how the three modalities in the video are fused to express the overall sentiment, which can be leveraged to learn better sentiment representations. inspired by recent knowledge-enhanced pretraining models on text sentiment analysis @xcite @xcite , we argue that pre-training models enriched with the sentiment knowledge of words and non-verbal behavior will facilitate the characteriza-tion of the sentimental patterns in videos, thereby resulting in better performance on multimodal sentiment analysis. in this paper, we propose a sentiment knowledge enhanced self-supervised learning (skesl) method, which uses contextual and non-verbal information to predict the fine-grained sentiment intensity of a word to learn the common sentimental patterns in opinion videos, as shown in figure 1 . specifically, given a speaker video without sentiment annotation, we first use the automatic speech recognition (asr) technology to obtain the transcribed text and then mask the most sentimentally salient words in the text according to the pre-specified sentiment lexicon. a pre-trained language representation model is utilized to acquire the sequence representation of the processed text. to integrate non-verbal information into the text representations, we further propose a non-verbal information aggregation method based on the crossmodal attention mechanism to derive non-verbal information-enhanced text representations. finally, the masked word representations are exploited to predict the sentiment intensity itself. after the skesl is completed, we transfer the pre-trained model to the task of multimodal sentiment analysis, and adopt a small amount of sentiment-annotated data to fine-tune the model. to evaluate the effectiveness of skesl, we test on two benchmark datasets: cmu-mosi @xcite and cmu-mosei @xcite . experimental results demonstrate that our model outperforms both the baseline and the current state-of-the-art (sota) approach. the main contributions can be summarized as follows: recently, more pre-trained language models have been proposed, which can be roughly divided into four categories @xcite : 1) knowledge enhancement: introducing domain-specific knowledge in the process of pre-training language representation models has been shown to be effective. a representative model is ernie @xcite , which explicitly introduces knowledge graph to pretrained language models. 2) transferability: on the basis of the general pre-training language model, further post-training is performed on more specific auxiliary tasks @xcite . 3) model compression: the compressed pre-trained language model can be widely applied in resource-constrained devices and tasks requiring real-time capability. the commonly used model compression methods include knowledge distillation @xcite , quantization @xcite and pruning @xcite . 4) pre-training objectives: aiming at rich and variable text expressions, many studies have further improved the text feature on basis of general masked language modeling (mlm) and next sentence prediction (nsp) objective @xcite . for instance, spanbert @xcite masks consecutive spans randomly instead of individual tokens, while bert-wwm @xcite utilizes the whole word mask (wwm) strategy to impose the model to learn complete semantics. knowledge enhanced pre-training language models incorporating external knowledge into pretraining language models has become prevalent and has been shown to be significant. such external knowledge includes commonsense knowledge for tasks such as entity typing and relation classification @xcite @xcite , sentiment knowledge for sentiment analysis @xcite @xcite , word sense knowledge for word sense disambiguation @xcite , commonsense knowledge for commonsense reasoning and sarcasm generation @xcite , legal knowledge for legal element extraction @xcite , and biomedical knowledge for health question answering and medical inference @xcite . knowledge enhanced pre-training models for text sentiment analysis some research @xcite @xcite integrates the sentiment knowledge into the pre-training process which includes sentiment words, word polarity and aspect-sentiment pairs. the learned representation would be more sentiment-specific and appropriate for text sentiment analysis. knowledge enhanced models for multimodal sentiment analysis in msa, some works consider sentiment knowledge with explicit supervision. for example, swafn @xcite designs a sentimental words prediction objective as an auxiliary task to incorporate sentimental words knowledge. magcn @xcite also incorporates sentiment knowledge into inter-modality learning. ## methodology in this section, we describe our proposed sentiment knowledge enhanced self-supervised learning (skesl) framework for multimodal sentiment analysis, as shown in figure 2 . the framework contains sentiment word masking (swm), text representation learning, non-verbal information injection (a.k.a., multimodal fusion), and sentiment intensity prediction (sip) modules. in the subsequent subsections, we will detail the four modules. ## results and analysis in this section, we make a detailed analysis and discussion about our experimental results. ## conclusion in this paper, we highlighted the sentiment knowledge enhanced self-supervised learning in msa. we find that mining sentimental prior information from unlabeled video data can lead to better predictions on labeled data. the larger the amount of unlabeled video data and the stronger the language modeling ability, the better the performance that can be achieved.