Multimodal Sentiment Analysis with Multi-perspective Fusion Network Focusing on Sense Attentive Language (2020)

Abstract: Multimodal sentiment analysis aims to learn a joint representation of multiple features. As demonstrated by previous studies, it is shown that the language modality may contain more semantic information than that of other modalities. Based on this observation, we propose a Multi-perspective Fusion Network(MPFN) focusing on Sense Attentive Language for multimodal sentiment analysis. Different from previous studies, we use the language modality as the main part of the final joint representation, and propose a multi-stage and uni-stage fusion strategy to get the fusion representation of the multiple modalities to assist the final language-dominated multimodal representation. In our model, a Sense-Level Attention Network is proposed to dynamically learn the word representation which is guided by the fusion of the multiple modalities. As in turn, the learned language representation can also help the multi-stage and uni-stage fusion of the different modalities. In this way, the model can jointly learn a well integrated final representation focusing on the language and the interactions between the multiple modalities both on multi-stage and uni-stage. Several experiments are carried on the CMU-MOSI, the CMU-MOSEI and the YouTube public datasets. The experiments show that our model performs better or competitive results compared with the baseline models.

## introduction multimodal sentiment analysis is a task of predicting sentiment of a video, an image or a text based on multiple modal features. with the increase of short videos on the internet, such as douyin, youtube, etc., multimoal sentiment analysis can be used to analyze the opinions of the public based on the speaker's language, facial gestures and acoustic behaviors. based on the successes in video, image, audio and language processing, multimodal sentiment analysis has been studied extensively and produced impressive results in recent years @xcite @xcite @xcite . the core of the multimodal sentiment analysis is to capture a better fusion of different modalities. different methods are proposed to fuse the multimodal features and help to capture the interactions of the modalities. tensor fusion network @xcite is proposed to obtain raw unimodal representations, bimodal interactions and trimodal interactions in the form of 2d-tensor and 3d-tensor simultaneously. low-rank fusion network @xcite is then proposed to alleviate the drawback of the large amount of parameters by lowrank factor. although the above methods achieved good results, they treat all modalities equally and fuse the modalities in the same contribution. we find that language modality always contain more semantic information for sentiment analysis, that's why most of ablation experiments of previous studies @xcite @xcite show that when using features from only one modality, the model using language features performs much better than using vision features or acoustic features. in this paper, we take the assumption that the language modality contains more information than that of the vision and acoustic modalities. we regard language as the major modality and hope to use other modalities to assist the language modality to produce better performance for multimodal sentiment analysis. to this end, we propose a multi-perspective fusion network for multimodal sentiment analysis focusing on sense attentive language. our model focuses on two aspects: (1)getting rich semantic language representation through the fusion of the sense level attention of language guided by other modalities. (2)learning comprehensive multimodal fusion from multiple perspectives, as well as keeping the enhanced language representation. in order to get rich semantic information of the language modality, we incorporate a sense-level attention network into the model to obtain a more elaborate representation of the language. generally speaking, there are many words which have more than one sense and their different senses may lead to different sentiment of a text in different context. previous studies try to distinguish the ambiguities of a word from the text modality @xcite using hownet @xcite and liwc @xcite , while we hope the sense of a word can be distinguished not only by the context of the text but also by fusion of other modalities(video and acoustic). as an example shown in figure1, we hope to predict the sentiment of the language "it would make sense". as can be seen, the word "sense" in the language modality has a higher attention weight which could be guided by the "smile face of the vision modality" and "high sound audio modality", and also by the "common sense" of the word "sense", which expresses more positive sentiment. figure 1 : the sense-level attention and word-level attention of the text "it would make sense" learned by our model. the first line is the acoustic modality, the second line is the video modality. the third line and the last line are the language modality, in which the third line is the original sentence, and the last line presents the senses of word "sense". darker color means greater weight. for the effectiveness of modal fusion, the key problem is to model the gap between different modalities and to learn a better multimodal fusion. in this paper, we propose a multi-stage and uni-stage strategy to fuse the multiple modalities in order to capture the interactions between multi-stage sharing information and global information integrated from uni-stage fusion. for multi-stage fusion, we use cnn with different window sizes to capture the multimodal fusion of consecutive temporals within different windows respectively. as for uni-stage fusion, we first perform a projection operation on the concatenation of the lstm outputs of three modalities, then attention mechanism is applied to learn the different contributions of the multimodal features at each temporal and produce a summary, which is regarded as the global multimodal fusion. the main contributions of our work are as follows: compared with conventional text-based sentiment analysis, sentiment analysis with multiple modalities achieves significant improvements @xcite . one of the most challenging task in multimodal sentiment analysis is to learn a joint representation of multiple modalities. earlier work uses fusion approaches such as concatenation of multi-modality features @xcite , while recent studies propose more sophisticated fusion approaches. @xcite propose a lstm-based model to capture contextual information. @xcite propose a tensor fusion network to explicitly aggregate unimodal, bimodal and trimodal interactions. @xcite propose a low-rank fusion network to alleviate the drawback of the large amount of parameters by low-rank factor. @xcite propose a gated multimodal embedding model to learn an on-off switch to filter noisy or contradictory modalities. as the modalities can have interactions between different timestamps, several models are proposed to fuse the multiple modals from different views. zadeh et al. (2018c) recently, @xcite propose to learn joint representations based on translations between modalities. they use a cycle consistency loss to ensure that the joint representations retain maximal information from all modalities. instead of directly fusing features at holistic level, @xcite propose a strategy named 'divide, conquer and combine' for multimodal fusion. their model performs fusion hierarchically to consider both local and global interactions. @xcite propose a recurrent attended variation embedding network (raven) to model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns. @xcite introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modalityspecific generative factors to optimize for a joint generative-discriminative objective across multimodal data and labels. although previous studies propose many effective approaches, most of them treat all modalities equally during the learning of multimodal fusion, which are different from our approach. in our model, we propose a sense-level attention network to learn different word representation under different senses. with the sense-attentive word representation, we can learn enhanced language representation. in addition, we try to learn sufficient multiomodal fusion through multi-stage fusion and uni-stage fusion, as well as keeping the language representation to form our final representation. ## our model our model consists of three components: sense attentive language representation which is regared as the main representation of the multimodal fusion; multi-stage multimodal fusion which is designed to capture the interactions between the sharing information on the multi-stage; uni-stage multimodal fusion which is used to capture the global fusion information. the whole architecture of our model is shown in figure 2 . in the following sections, we will introduce the sense-level attention network in section 3.1, and describe the multi-stage multimodal fusion and the uni-stage multimodal fusion strategy in section 3.2. section 3.3 describes the final representation and model trainning. ## conclusion in this paper, we propose a novel multi-perspective fusion network focusing on sense attentive language for multimodal sentiment analysis. evaluations show that using our proposed multi-stage and uni-stage fusion strategies and using sense attentive language representation can improve performance on multimodal sentiment analysis for the cmu-mosi, cmu-mosei and youtube data. our model also achieves a new state-of-the-art in the youtube and cmu-mosei dataset on accuracy and f1 measure