Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection (2023)

Abstract: With the popularity of social media, detecting sentiment from multimodal posts (e.g. image-text pairs) has attracted substantial attention recently. Existing works mainly focus on fusing different features but ignore the challenge of modality heterogeneity. Specifically, different modalities with inherent disparities may bring three problems: 1) introducing redundant visual features during feature fusion; 2) causing feature shift in the representation space; 3) leading to inconsistent annotations for different modal data. All these issues will increase the difficulty in understanding the sentiment of the multimodal content. In this paper, we propose a novel Multi-View Calibration Network (MVCN) to alleviate the above issues systematically. We first propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual elements. We then devise a sentiment-based congruity constraint task to calibrate the feature shift in the representation space. Finally, we introduce an adaptive loss calibration strategy to tackle inconsistent annotated labels. Extensive experiments demonstrate the competitiveness of MVCN against previous approaches and achieve state-of-the-art results on two public benchmark datasets.

## introduction multimodal sentiment detection @xcite aims to explore the sentiment embedded in the multimodal contents, such as text, images, and videos. with the growth of social media, it shows broad applications for the understanding of one's position, attitude, or opinion towards an entity, person, or topic, which has attracted substantial attention from both academic and industrial communities. in this paper, we focus on detecting the sentiment of multimodal posts in social media. as shown in figure 1 , the model is required to infer the human sentiments given the image and text pairs. previous works mainly focus on how to integrate modalities and have achieved astonishing progress. earlier works apply concatenation to fuse different features, such as @xcite ; @xcite . to promote the fusion, @xcite ; @xcite ; @xcite ; @xcite construct different modules to realize deeper multimodal interaction. however, these methods of viewing different modalities in the same light neglect the modality heterogeneity, hindering the performance of the model. although clmlf @xcite tries to apply contrastive learning to alleviate this problem, it is in coarse granularity and only considers feature level, which is insufficient to address the issue of feature shift. modality heterogeneity is mainly caused by the modality gap, which has been discussed in some related multimodal research @xcite @xcite . in multimodal sentiment detection, it is because image modality often has less information and more redundant features compared to text modality. as shown in figure 1 , only a few visual elements reveal sentiment information(e.g., people smile, ill dog), causing redundant visual elements and low information density for the sentiment. in contrast, text modality is more indicative of sentiment owing to higher information density @xcite . consequently, this heterogeneity across different modalities will bring difficulty in understanding the sentiment of multimodal content. concretely, the ignorance of modality heterogeneity will directly cause three problems: a) it takes sparse and noisy visual features into fusion and causes confusion in understanding sentimental information. b) integrating modalities of different properties will further lead to the multimodal feature shift and makes the model capture spurious correlations between multimodal features and sentiments. c) as mentioned by @xcite ; @xcite , it will affect the data annotators to vote inconsistent labels for unimodal, which leads to uncertain annotated labels and weaken the label confidence. taking mvsa-single @xcite for example, approximately 47% of the samples suffer from inconsistent annotated labels. to tackle the above problems systematically, we propose a multi-view calibration network (mvcn) from three different views: ## methodology the architecture of the proposed multi-view calibration network (mvcn) is shown in figure 2 . generally, mvcn consists of text-guided fusion module and two paralleled sub-tasks. the two paralleled sub-tasks are sentiment classification and sentiment-based congruity constraint respectively. ## experiments in this section, we first introduce the experimental setup and report the experimental results, then conduct the ablation study and visualization analysis. ## related work in this section, we introduce the background of multimodal sentiment detection and modality heterogeneity. ## conclusion in this paper, we present a multi-view calibration network (mvcn) to address the modality heterogeneity problem for text-image sentiment detection from different views. specially, we respectively introduce the text-guided fusion module to calibrate multimodal fusion and reduce the negative impacts of redundant visual elements, a sentiment-based congruity constraint task to further calibrate the feature shift in representation space and an adaptive loss calibration strategy to calibrate the training loss in terms of uncertain annotated labels. the thorough experiments show the mvcn achieves state-of-the-art performance on two benchmark datasets.