Modeling Intra- and Inter-Modal Relations: Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis (2022)

Abstract: The existing research efforts in Multimodal Sentiment Analysis (MSA) have focused on developing the expressive ability of neural networks to fuse information from different modalities. However, these approaches lack a mechanism to understand the complex relations within and across different modalities, since some sentiments may be scattered in different modalities. To this end, in this paper, we propose a novel hierarchical graph contrastive learning (HGraph-CL) framework for MSA, aiming to explore the intricate relations of intra- and inter-modal representations for sentiment extraction. Specifically, regarding the intra-modal level, we build a unimodal graph for each modality representation to account for the modality-specific sentiment implications. Based on it, a graph contrastive learning strategy is adopted to explore the potential relations based on unimodal graph augmentations. Furthermore, we construct a multimodal graph of each instance based on the unimodal graphs to grasp the sentiment relations between different modalities. Then, in light of the multimodal augmentation graphs, a graph contrastive learning strategy over the inter-modal level is proposed to ulteriorly seek the possible graph structures for precisely learning sentiment relations. This essentially allows the framework to understand the appropriate graph structures for learning intricate relations among different modalities. Experimental results on two benchmark datasets show that the proposed framework outperforms the state-of-the-art baselines in MSA.

## introduction multimodal sentiment analysis (msa) has received increasing research attention in recent years. different from textual sentiment analysis, msa generally contains three modalities: text (caption), he was the only character that was slightly interesting visual and acoustic, in which the visual and acoustic information can complement the text for identifying the sentiment score and thus preferably detecting the sentiment polarity (e.g. positive, negative, etc.). as shown in figure 1 , although the text modality expresses the positive sentiment, we can infer that the correct sentiment of this example is negative in the light of the study of visual and acoustic information. therefore, dealing with msa needs to consider learning and fusing the information from different modalities. early msa work attempted to fuse the information from different modalities by tensor-based features fusion @xcite @xcite or attention-based features fusion @xcite . furthermore, some representation learning-based approaches @xcite aim to model the consistency and the variability between modalities for extracting the sentiment cues among modalities or consider both fusion and alignment of multimodal sequential data with a graph model @xcite . despite the promising progress made by current work, they generally focus on fusing multimodal representations via class-driven supervised learning or multitask learning, which fails to understand the intricate relations within and across modalities for better sentiment extraction. as shown in figure 1 , the sentiment is scattered within each modality and across different modalities. in this paper, we study how to understand representations within and across modalities, enabling the highly correlated modal representations to be explicitly linked for learning the multimodal sentiment information. to reach this goal, in the light of developing the merit of graph structure for modeling intricate relations of representations, we first build a unimodal graph for each modality and further build a multimodal graph for each instance based on the unimodal graphs. concretely, for the intra-modal graphs, to account for the underlying relations within each modality, we construct a syntax-aware graph for the text modality based on the dependency tree of the sentence and build sequential connection graphs for visual and acoustic modality. for the inter-modal graph, we build a fully-connected inter-modal graph based on the modality-specific graphs to capture the potential relations across different modalities. then, we apply a graph attention networks @xcite architecture to model the semantic relations by means of specifying different weights to different nodes in a neighborhood, without requiring any costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. following that, we propose a hierarchical graph contrastive learning (hgraph-cl) framework to model the correlation and difference of graph information within a specific modality and further across different modalities. specifically, for the intra-modal level, inspired by @xcite , we first devise a self-supervised graph contrastive learning strategy based on the graph augmentations, aiming to explore more appropriate graph structures and derive robust graph representations for each modality. in addition, inspired by @xcite ; @xcite , we employ a supervised contrastive learning strategy to make sense of the correlation and difference between different classes, so as to capture the similarity between examples in one class and contrast them with examples in other classes. moreover, for the inter-modal level, we also perform these two contrastive learning strategies to learn the graph representations for better generalizability, transferability, and robustness in learning sentiment cues compared with pure class-driven methods. the main contributions of our work can be summarized as follows: ## method in this section, we describe the proposed hgraph-cl framework in detail. as illustrated in figure 2 , the framework mainly consists of four components: 1) feature extraction, which applies bert @xcite and bilstms @xcite to extract features from the three modalities of text, images and audio. ## analysis 5.1 ablation study to verify the impact of the hierarchical graph contrastive learning on performance, we conduct ablation experiments on the two datasets and show the results in on mosi and mosei, and makes sense for classification. on the other hand, performing graph contrastive learning at the intra-modal level only or at the inter-modal level only can improve both classification and regression on two datasets. additionally, our model with complete hierarchical graph contrastive learning can achieve the best overall performance, with a significant improvement over the model without it. the results suggest that hierarchical graph contrastive learning has a great positive impact on the classification and the regression of hgraph-cl. ## conclusion this paper proposes a novel hierarchical graph contrastive learning (hgraph-cl) framework for multimodal sentiment analysis (msa), in which graph contrastive learning is performed at intra-modal and inter-modal levels. for the graph contrastive learning strategy performed at each level, we devise a fully-supervised contrastive loss and a selfsupervised contrastive loss. the fully-supervised contrastive loss is devised to improve the learning of sentiment cues by capturing the similarity between examples in one class and the contrast among different classes. and the self-supervised contrastive loss is devised to explore a more appropriate graph structure based on the graph augmentations for making use of sentiment relations within each modality and across different modalities. experimental results on two benchmark datasets show that our method outperforms the state-of-the-art baselines in msa.