T^2DR : A Two-Tier Deficiency-Resistant Framework for Incomplete Multimodal Learning (2025)

Abstract: Multimodal learning is garnering significant attention for its capacity to represent diverse human perceptions (e.g., linguistic, acoustic, and visual signals), achieving more natural and intuitive interactions with technology.However, the frequent occurrence of incomplete data, either within a single modality (intra-modality) or across different modalities (inter-modality), presents substantial challenges in reliable semantic interpretation and model reasoning.Furthermore, there is currently no robust representation learning mechanism capable of managing both intra-modality and inter-modality real-data deficiencies.To address this challenge, we present \text{T}^2 DR, a two-tier deficiency-resistant framework for incomplete multimodal learning, which comprises two main modules:(1) Intra-Modal Deficiency-Resistant module (IADR): To address fine-grained deficiencies, we introduce Intra-Attn to focus on the available data while avoiding excessive suppression of the missing regions.(2) Inter-Modal Deficiency-Resistant module (IEDR): To handle coarse-grained deficiencies, we propose the shared feature prediction (SFP) to leverage cross-modal shared features for preliminary data imputation. Subsequently, we apply Inter-Attn to allocate appropriate attention to each modality based on the results from the capability-aware scorer (CAS).Extensive experiments are performed on two well-known multimodal benchmarks, CMU-MOSI and CMU-MOSEI, across various missing scenarios for sentiment analysis. Experimental results show that \text{T}^2 DR significantly outperforms the SOTA models. Code is available at https://github.com/LH019/T2DR .

## introduction in modern data analysis, the significance of multimodal data is increasingly prominent. unlike unimodal data, multimodal data can capture diverse information from heterogeneous sources and * corresponding author leverage the complementary of different modalities for more sufficient information @xcite . nevertheless, the integration process is often compromised by the presence of incomplete data @xcite . this incomplete data may present as fine-grained (intra-modality), shown by slight variations or gaps within a single modality @xcite , typically arising from interruptions in data collection or improper data management. it can also appear as coarse-grained (inter-modality) @xcite , manifested as significant omissions across different modalities, commonly due to unavailable data sources or failures to acquire multimodal data synchronously. existing methods for handling both intra-modal and inter-modal incomplete data @xcite typically rely on randomly masking features derived from complete data to simulate intra-modal incompleteness, which fails to faithfully represent the real-world scenarios. therefore, addressing the issue of fine-grained and coarse-grained deficiencies is crucial for improving the precision and reliability of multimodal learning. existing methods for addressing incomplete multimodal learning are divided into three main categories: generative methods @xcite @xcite , multimodal joint learning @xcite @xcite , and knowledge distillation @xcite . generative methods such as vigan @xcite use generative adversarial networks alongside denoising autoencoders to recover and refine incomplete modalities. multimodal joint learning methods, exemplified by drfuse @xcite , extract shared information from available modalities to compensate for missing modalities. knowledge distillation approach mmanet @xcite leverages a margin-aware distillation strategy to help the student network improve its comprehension of inter-class relationships, enhancing the final shared feature representation. while existing methods have shown promising results in handling missing modalities, they often focus on the quality of the imputed features @xcite , with substantial challenges remaining. these include: (1) a lack of dynamic adjustment mechanisms during model training to resist interference from incomplete data, especially for imputed features with lower quality, and (2) a predominant focus on inter-modality incomplete data scenarios, often overlooking the more common intra-modality incomplete data, where the missingness occurs at the raw data level rather than the processed feature level. to address the above two challenges, we present a novel approach -a two-tier deficiency-resistant framework for incomplete multimodal learning, referred to t 2 dr, which focuses on deeper learning from valid data and enhancing its resistance to various deficiencies. our main contributions can be summarized as follows: ## related work to tackle deficiencies in multimodal learning, we enhance the model's adaptability by using missing modality prediction for a robust end-to-end system, weight allocation to perform dynamic supervision within each data entry, and modality fusion to leverage modality complementarity for richer semantic features. ## experimental setup benchmark datasets. we evaluate t 2 dr's performance with two multimodal datasets, each containing text, acoustic, and visual modalities, and three unimodal datasets, each focused on one modality: text, acoustic, or visual. for multimodal datasets, cmu-mosi @xcite and cmu-mosei @xcite are sourced from youtube videos, annotated with sentiment intensity scores from -3 (highly negative) to +3 (highly positive), covering positive, negative, and neutral sentiments. notably, cmu-mosei is nearly ten times larger than cmu-mosi, offering a more extensive range of speakers and topics. for unimodal datasets, we use carer @xcite ) for text, esc-50 @xcite for acoustic, and imagenet @xcite for visual. carer targets emotion recognition in text, esc-50 supports environmental sound classification, and imagenet focuses on image classification across diverse objects and scenes. ## conclusion in this paper, we propose t 2 dr, a comprehensive framework for addressing both intra-modality and inter-modality incomplete data. to address various granularities of deficiency, we introduce two weight allocation mechanisms, intra-attn and inter-attn, to provide stronger supervision of higherquality data. additionally, for coarse-grained deficiencies, we present shared feature prediction (sfp) for missing modality imputation, and capabilityaware scorer (cas) to dynamically perceive the optimal weight for each modality, enabling inter-attn-based adaptive supervision. our experiments on both unimodal and multimodal datasets have demonstrated the remarkable robustness and adaptability of t 2 dr under incomplete conditions.