Multimodal Invariant Sentiment Representation Learning (2025)

Abstract: Multimodal Sentiment Analysis (MSA) integrates diverse modalities to overcome the limitations of unimodal data. However, existing MSA datasets commonly exhibit significant sentiment distribution imbalances and cross-modal sentiment conflicts, which hinder performance improvement. This paper shows that distributional discrepancies and sentiment conflicts can be incorporated into the model training to learn stable multimodal invariant sentiment representation. To this end, we propose a Multimodal Invariant Sentiment Representation Learning (MISR) method. Specifically, we first learn a stable and consistent multimodal joint representation in the latent space of Gaussian distribution based on distributional constraints Then, under invariance constraint, we further learn multimodal invariant sentiment representations from multiple distributional environments constructed by the joint representation and unimodal data, achieving robust and efficient MSA performance. Extensive experiments demonstrate that MISR significantly enhances MSA performance and achieves new state-of-the-art.

## introduction multimodal sentiment analysis (msa) understands human emotions by fusing information from modalities such as vision, audio, and text @xcite . msa effectively compensates for the limitations of a unimodal data by exploring the relationships between different modalities @xcite , and shows significant advantages in improving the understanding and expression of sentiment. msa research has focused on utilizing various strategies to enhance performance. early methods focused on fusion mechanisms for modality integration @xcite , followed by attention-based approaches that enhanced performance by capturing intra-and inter-modal [-3,-2] (-2,-1] (-1,0] (0,1] (1,2] (2,3] d 6hqwlphqwglvwulexwlrqv correlations @xcite @xcite . recently, decoupled representation learning has emerged as a key direction to minimize interference by separating modality information @xcite . despite achieving impressive improvements, these methods ignore the significant distribution imbalance and cross-modal sentiment conflicts present in the datasets used for model training, limiting the performance of the models. as shown in fig. 1 (a), it is evident that in the mosei dataset @xcite , sentiment values predominantly fall within the range of @xcite , accounting for 65.97% of the samples. the imbalance in the sentiment distribution of the data causes the model to bias toward predictions of high-frequency sentiments, while insufficiently learning low-frequency sentiments @xcite . due to the intrinsic uncertainty of the data @xcite , there are conflicting sentiment across modalities. sims @xcite provides sentiment categories for each modality, and we analyze their inconsistency across modalities. as shown in fig. 1 (b), significant sentiment conflicts exist within unimodal data, with 48.97% inconsistency across the three modalities. this conflict leads to contradictions between modalities, interfering with the fusion process. data distribution imbalances are difficult to avoid in real-world training, and countering them is often costly @xcite . even when addressed, cross-modal sentiment conflicts persist. inspired by the invariant risk minimization (irm) @xcite , we observe that while unimodal sentiment knowledge is a primary source for generating multimodal labels, the correlation between unimodal sentiment categories and multimodal labels is unstable (as shown in fig. 1(b) ). we wish to train the msa model to learn invariant features, stably correlated with multimodal labels (e.g., object shapes in image classification), rather than spurious features (e.g., environments) with unstable correlations @xcite . if the uncertain unimodal sentiment distribution is treated as an unstable spurious feature, how can invariant features directly related to multimodal labels be represented? we propose a reasonable assumption: multimodal sentiment representation is invariant to unimodal sentiment polarity and data distribution. for example, the multimodal sentiment polarity of "positive" remains positive, regardless of changes in unimodal sentiment or data distribution. based on this, we reformulate the msa task as learning stable and invariant sentiment representation from imbalanced multimodal data, using invariant constraints to enhance msa performance. based on the above analysis, we propose the multimodal invariant sentiment representation learning (misr). specifically, we first map data to the latent space of gaussian distributions to learn stable and consistent joint representation. then, we treat unimodal data as unstable features relevant to the task and combine them with joint representation to create multiple distribution environments. finally, based on invariant risk constraints, we learn multimodal invariant sentiment representation across these environments, thereby achieving effective msa. in the misr learning process, unfavorable factors in the data distribution are incorporated into optimization objectives for model training. data distribution imbalance provides an optimization direction for distribution constraints, while sentiment conflicts combined with joint representation construct diverse sentiment distribution environments for invariant learning. misr learns stable and invariant sentiment representation through joint distribution constraint and invariance constraint, achieving effective and robust msa performance. the main contributions are as follows: ## conclusion this paper proposes the multimodal invariant sentiment representation learning (misr) method, which learns multimodal invariant sentiment representation based on data distribution discrepancies and sentiment conflicts to achieve effective and robust msa. this paper shows that the imbalance in data distribution provides an optimization direction for distribution constraints, while sentiment conflicts combined with joint representation construct diverse sentiment distribution environments for invariant learning. misr incorporates unfavorable factors in the data distribution into optimization objectives for model training. comprehensive experiments across multiple datasets demonstrate that misr significantly outperforms existing models, providing a new research direction for achieving effective and robust msa.