SWAFN : Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis (2020)

Abstract: Multimodal sentiment analysis aims to predict sentiment of language text with the help of other modalities, such as vision and acoustic features. Previous studies focused on learning the joint representation of multiple modalities, ignoring some useful knowledge contained in language modal. In this paper, we try to incorporate sentimental words knowledge into the fusion network to guide the learning of joint representation of multimodal features. Our method consists of two components: shallow fusion part and aggregation part. For the shallow fusion part, we use crossmodal coattention mechanism to obtain bidirectional context information of each two modals to get the fused shallow representations. For the aggregation part, we design a multitask of sentimental words classification to help and guide the deep fusion of the three modalities and obtain the final sentimental words aware fusion representation. We carry out several experiments on CMU-MOSI, CMU-MOSEI and YouTube datasets. The experimental results show that introducing sentimental words prediction as a multitask can really improve the fusion representation of multiple modalities.

## introduction multimodal sentiment analysis is a task of predicting the sentiment of a video, an image or a text based on multiple modal features. based on the contributions of different modalities to each other, multimodal sentiment analysis has achieved significant results and attracted the attentions of many researchers in recent years. the main challenge of the multimodal sentiment analysis is to capture a better fusion of different modalities. previous studies have proposed different methods for the fusion in different point of views. some methods focus on the improvement of the lstm structure to learn the interactions of different modal features from the view of the uni-stage and multi-stage. @xcite propose a memory fusion network to learn both the view-specific interactions and the cross-view interactions. @xcite propose a recurrent multistage fusion network to model cross-modal interactions using multistage fusion approach. some methods focus on exploiting the expressiveness of tensors for multimodal representation. @xcite propose a tensor fusion network to explicitly model the unimodal, bimodal and trimodal interactions through a 3-fold cartesian product from modality embedding. more recently, other methods are proposed and achieve new state-of-the-art results @xcite @xcite . although previous studies achieved good results, there are still two points can be improved: (1) we find that the fusion of most of previous methods is from one direction, that is, when the two modals are fused, the representation of two modals are fused directly as a new representation, similar to the work of @xcite and @xcite . this fusion strategy ignores the long range of context information of each modality. as an example shown in figure 1 , for the fusion of language modality and vision modality, if we can capture the context information of each modality from bi-directions, we can get more sufficient fusion information. (2) few of previous studies explicitly explored the knowledge contained in the language text which can be used to help the fusion of different modalities based on the rich information existing in the language. to this end, in this paper, we propose a sentimental words aware fusion network (swafn) for multimodal sentiment analysis. more specifically, we first use lstm to encode the original features of three modalities. then we use the coattention mechanism @xcite to learn the co-dependent representation between language and other modalities separately by capturing attention contexts of each modality. we call this kind of bimodal fusion between language and other modalities as the shallow fusion part. figure 1 presents the illustration of crossmodal coattention for language and vision modalities. then, we design a sentimental words prediction task as an auxiliary task through the multitask learning mechanism to guide the aggregation of the shallow fusion of multiple modal features and obtain the final sentimental words aware deep fusion representation. the main contribution of this work are as follows: ## related work the key problem of multimodal sentiment analysis is to fill the gap of different modalities and learn the effective fusion of multimodal features. in recent years, with the successful application of neural networks in many tasks, different sophisticated fusion approaches are proposed and achieve significant results. fusion methods based on improved lstm structure. some of the previous studies propose to improve the lstm structure to learn the interactions of different modality features from the view of the same timestep and cross timestep. @xcite propose a gated multimodal embedding lstm with temporal attention model which consists of two modules, one is gated multimodal embedding aiming to alleviate the fusion difficulty when there are noisy modalities, another is lstm with temporal attention to perform word-level fusion. @xcite propose a multi-attention recurrent network, in which the lsthm (an extension of lstm) is used to store view-specific dynamics of the assigned modality and cross-view dynamics related to the assigned modality, and the multi-attention block is used to discover cross-view dynamics cross different modalities. @xcite propose a memory fusion network which employs lstm to learn view-specific interactions and an attention mechanism called the delta-memory attention network to identify the cross-view interactions. @xcite propose a recurrent multistage fusion network to model cross-modal interactions using multi-stage fusion approach. figure 2 : the whole architecture of our model. we first use coattention mechanism to learn the bidirectional long range of context information between language modality and other modalities separately. then we integrate a sentimental words classification task into the model through multitask learning mechanism to guide the learning and aggregation of multimodal fusion. although previous studies have proposed many effective multimodal fusion approaches, few studies have explored the possibility of using knowledge in language as a multi-task learning framework in multimodal sentiment analysis. in this paper, we try to design an auxiliary task to guide the model to learn sentimental words information aware multimodal representation. ## our model in this section, we will describe our model in more detail. section 3.1 introduces the crossmodal coattention, section 3.2 introduces the sentimental words prediction auxiliary task, section 3.3 introduces the sentimental words aware representation and section 3.4 describes the model training. figure 2 shows the whole architecture of our model. ## discussion in this section, we investigate the impact of different modalities on the performance of the final model. we also conduct a case study to investigate how the auxiliary task guide the learning of attention weights of sentimental words in the sentence. ## conclusion in this paper, we propose a sentimental words aware fusion network (swafn) which first applies the crossmodal coattention mechanism to learn the long range of context information and then use a sentimental words classification auxiliary task to guide and learn the sentimental words aware final multimodal fusion representation. the experimental results on several datasets show the effectiveness of our model. the results and case study also demonstrate that our proposed sentimental words classification auxiliary task is an effective way to use the external knowledge to help the model to learn more powerful multimodal representation. in the future, we will consider incorporating more external language knowledge to obtain better multimodal fused representations.