Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis (2022)

Abstract: Multimodal representation learning is a challenging task in which previous work mostly focus on either uni-modality pre-training or cross-modality fusion. In fact, we regard modeling multimodal representation as building a skyscraper, where laying stable foundation and designing the main structure are equally essential. The former is like encoding robust uni-modal representation while the later is like integrating interactive information among different modalities, both of which are critical to learning an effective multimodal representation. Recently, contrastive learning has been successfully applied in representation learning, which can be utilized as the pillar of the skyscraper and benefit the model to extract the most important features contained in the multimodal data. In this paper, we propose a novel framework named MultiModal Contrastive Learning (MMCL) for multimodal representation to capture intra- and inter-modality dynamics simultaneously. Specifically, we devise uni-modal contrastive coding with an efficient uni-modal feature augmentation strategy to filter inherent noise contained in acoustic and visual modality and acquire more robust uni-modality representations. Besides, a pseudo siamese network is presented to predict representation across different modalities, which successfully captures cross-modal dynamics. Moreover, we design two contrastive learning tasks, instance- and sentiment-based contrastive learning, to promote the process of prediction and learn more interactive information related to sentiment. Extensive experiments conducted on two public datasets demonstrate that our method surpasses the state-of-the-art methods.

## introduction with the surge of user-generated videos, multimodal sentiment analysis (msa) have become a hot research field, which aims to infer people's * corresponding author. sentiment based on multimodal data including text, audio and video @xcite @xcite . to successfully understand human behaviours and interpret human intents, it is necessary to attain an effective and powerful multimodal representation for the model. however, two major challenges in learning such multimodal representation exist: the accurate extraction of uni-modal features and the heterogeneities across different modalities bring difficulty of modeling cross-modal interaction. to acquire powerful uni-modal features, @xcite presents a large-scale language model named bert for textual modality and @xcite introduce an audio representation learning method for audio modality by distilling from @xcite which targets at transferable model for visual modality. in msa, previous methods @xcite mainly utilize bert for textual modality while vague feature extractor such as covarep @xcite and facet (imotions 2017) for acoustic and visual modality, where the inherent noise contain in unimodal features may still exist. to avoid uni-modal noise interfering downstream sentiment inference task, we design uni-modal contrastive coding (umcc) which employs feature cutoff strategy inspired by @xcite and generates augmentation features to construct contrastive learning task with origin uni-modal representation. as shown in figure 1 , we then obtain robust and efficient representations for acoustic and visual modalities. to alleviate the impact of modality heterogeneity, previous msa models propose various modalities fusion methods to learn cross-modality interaction information @xcite . modality translation is a popular method to explicitly translate source modality to the target one, which directly manipulates the commonalities across modalities @xcite @xcite . however, due to the exis-tence of discrepancy modality-specific information and huge modality gap, it is undesirable and extremely difficult to project the representations from different modalities to the same one. different with these explicit modality translation methods, we propose cross-modal contrastive prediction (cmcp) composed of a pseudo siamese predictive network and two designed contrastive learning tasks to predict cross-modal representation in an implicitly contrastive way. the predictive representation efficiently capture cross-modality dynamics and concurrently preserve modality-specific features for the modalities. the novel contributions of our work can be summarized as follows: ## experiments we conduct extensive experiments on two public datasets which offer both token-aligned andunaligned data for multimodal sentiment analysis. the details about the datasets, evaluation metrics and baseline methods are provided subsequently. ## conclusion in this paper, we present a novel framework named multimodal contrastive learning (mmcl) which efficiently adopts contrastive learning in multi-modal sentiment analysis. mmcl consists of uni-modal contrastive coding (umcc) which learns robust uni-modal representations by focusing on informative features and reducing the interference of inherent noise for acoustic and visual modalities, and cross-modal contrastive prediction (cmcp) which construct a powerful pseudo siamese predictive network to learn commonalities and interactive dynamics across different modalities. besides, we propose two effective contrastive learning tasks, instance-and sentiment-based contrastive learning, to help the predictive network maintain modalityspecific dynamics and learn sentiment-related information in multimodal data, which improve the performance of the model significantly. the extensive experiments conducted on cmu-mosi and cmu-mosei demonstrate the superiority of our model and the efficacy of contrastive learning. moreover, the further visualization of losses and representation space provide comprehensive insight into our model. we believe our work can boost the creativity in contrastive learning and multimodal representation learning in the future.