U ni S - MMC : Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning (2023)

Abstract: Multimodal learning aims to imitate human beings to acquire complementary information from multiple modalities for various downstream tasks. However, traditional aggregation-based multimodal fusion methods ignore the inter-modality relationship, treat each modality equally, suffer sensor noise, and thus reduce multimodal learning performance. In this work, we propose a novel multimodal contrastive method to explore more reliable multimodal representations under the weak supervision of unimodal predicting. Specifically, we first capture task-related unimodal representations and the unimodal predictions from the introduced unimodal predicting task. Then the unimodal representations are aligned with the more effective one by the designed multimodal contrastive method under the supervision of the unimodal predictions. Experimental results with fused features on two image-text classification benchmarks UPMC-Food-101 and N24News show that our proposed Unimodality-Supervised MultiModal Contrastive UniS-MMC learning method outperforms current state-of-the-art multimodal methods. The detailed ablation study and analysis further demonstrate the advantage of our proposed method.

## introduction social media has emerged as an important avenue for communication. the content is often multimodal, e.g., via text, speech, audio, and videos. multimodal tasks that employ multiple data sources include image-text classification and emotion recognition, which could be used for specific applications in daily life, such as web search @xcite , guide robot @xcite . hence, there is a need for an effective representation strategy for multimodal content. a common way is to fuse unimodal representations. ## methodology in this section, we present our method called unis-mmc for multimodal fusion. ## conclusion in this work, we propose the unimodality-supversied multimodal contrastive (unnis-mmc), a novel method for multimodal fusion to reduce the multimodal decision bias caused by inconsistent unimodal information. based on the introduced multi-task-based multimodal learning framework, we capture the task-related unimodal representations and evaluate their potential influence on the final decision with the unimodal predictions. then we contrastively align the unimodal representation towards the relatively reliable modality under the weak supervision of unimodal predictions. this novel contrastive-based alignment method helps to capture more trustworthy multimodal representations. the experiments on four public multimodal classification datasets demonstrate the effectiveness of our proposed method.