Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks (2021)

Abstract: With the popularity of smartphones, we have witnessed the rapid proliferation of multimodal posts on various social media platforms. We observe that the multimodal sentiment expression has specific global characteristics, such as the interdependencies of objects or scenes within the image. However, most previous studies only considered the representation of a single image-text post and failed to capture the global co-occurrence characteristics of the dataset. In this paper, we propose Multi-channel Graph Neural Networks with Sentiment-awareness (MGNNS) for image-text sentiment detection. Specifically, we first encode different modalities to capture hidden representations. Then, we introduce multi-channel graph neural networks to learn multimodal representations based on the global characteristics of the dataset. Finally, we implement multimodal in-depth fusion with the multi-head attention mechanism to predict the sentiment of image-text pairs. Extensive experiments conducted on three publicly available datasets demonstrate the effectiveness of our approach for multimodal sentiment detection.

## introduction the tasks of extracting and analyzing sentiments embedded in data have attracted substantial attention from both academic and industrial communities @xcite . with the increased use of smartphones and the bloom of social media such as twitter, tumblr and weibo, users can post multimodal tweets (e.g., text, image, and video) about diverse events and topics to convey their feelings and emotions. therefore, multimodal sentiment analysis has become a popular research topic in recent years @xcite . as shown in fig. 1 , sentiment is no longer expressed by a pure modality in the multimodal scenario but rather by the comtwo posts express the user's positive sentiment from multimodal data that has global characteristics, including the "have a fun/nice day" phrase, the ocean scene, and the beach scene. we focus on multimodal sentiment detection for image-text pairs in social media posts. the problem of image-text mismatch and flaws in social media data, such as informality, typos, and a lack of punctuation, pose a fundamental challenge for the effective representation of multimodal data for the sentiment detection task. to tackle this challenge, @xcite constructed different networks for multimodal sentiment analysis, such as a hierarchical semantic attentional network (hsan) and a multimodal deep semantic network (mdsn). @xcite and @xcite proposed a co-memory network (co-mem) and a multi-view attentional network (mvan) models, respectively, introducing memory networks to realize the interaction between modalities. the above methods treat each image-text post in the dataset as a single instance, and feature dependencies across instances are neglected or modeled implicitly. in fact, social media posts have specific global co-occurring characteristics, i.e., co-occurring words, objects, or scenes, which tend to share similar sentiment orientations and emotions. for example, the co-occurrences of the words "have a fun/nice day" and of the bright scenes "ocean/beach" in the two images in fig. 1 imply a strong relationship between these features and positive sentiment. how to more effectively make use of the feature co-occurrences across instances and capture the global characteristics of the data remain a great challenge. we propose a multi-channel graph neural networks model with sentiment-awareness (mgnns) for multimodal sentiment analysis that consists of three stages. our main contributions are summarized as follows: ## proposed model fig. 2 illustrates the overall architecture of our proposed mgnns model for multimodal sentiment detection that consists of three modules: the encoding module, the multi-gnn module, and the multimodal interaction module. we first encode text and image input into hidden representations. then, we introduce gnn from different channels to learn multiple modal representations. in this paper, the channels are the text-gnn (tg) module, the image-gcn-scene (igs) module, and the image-gcn-object (igo) module. finally, we realize the in-depth interactions between different modalities by multimodal multi-head attention. ## experiments we conduct experiments on three multimodal sentiment datasets from social media platforms, mvsa-single, mvsa-multiple @xcite , and tumemo @xcite , and compare our mgnns model with a number of unimodal and multimodal approaches. amount of image-text data crawled from tumblr foot_1 . the statistics of these datasets are given in appendix a; and for a fair comparison, we adopt the same data preprocessing method as that of yang @xcite we adopt the cross-entropy loss function and adam optimizer. in the process of extracting objects and scenes, we reserve the objects with the probability greater than 0.5 and the top-5 scenes, respectively. the other parameters are listed in ## conclusions this paper proposes a novel model, mgnns, that is built based on the global characteristics of the dataset for multimodal sentiment detection tasks. as far as we know, this is the first application of graph neural networks in image-text multimodal sentiment analysis. the experimental results on publicly available datasets demonstrated that our proposed model is competitive with strong baseline models. in future work, we plan to construct a model that adopts the advantages of the gnn and pretrained models such as bert, visualbert, and etc. we want to design a reasonable algorithm to characterize the quality of the objects and scenes selected from the image and further improve the representation ability of the model.