Two Challenges, One Solution: Robust Multimodal Learning through Dynamic Modality Recognition and Enhancement (2025)

Abstract: Multimodal machine learning is often hindered by two critical challenges: modality missingness and modality imbalance. These challenges significantly degrade the performance of multimodal models. The majority of existing methods either require the availability of full-modality data during the training phase or necessitate explicit annotations to detect missing modalities. These dependencies severely limit the models’ applicability in the real world. To tackle these problems, we propose a Dynamic modality Recognition and Enhancement for Adaptive Multimodal fusion framework *DREAM*. Within DREAM, we innovatively employ a sample-level dynamic modality assessment mechanism to direct selective reconstruction of missing or underperforming modalities. Additionally, we introduce a soft masking fusion strategy that adaptively integrates different modalities according to their estimated contributions, enabling more accurate and robust predictions. Experimental results on three benchmark datasets consistently demonstrate that DREAM outperforms several representative baseline and state-of-the-art models, marking its robustness against modality missingness and imbalanced modality.

## introduction multimodal machine learning, inspired by humans' ability to solve problems using information from varying modalities, such as acoustic, visual, and textual cues, focuses on enabling models to effectively leverage multimodal data @xcite . it has been successfully applied across various domains, including medical @xcite , public safety @xcite , and multimodal sentiment analysis @xcite . despite its success, multimodal machine learning often faces the challenge of missing modalities * corresponding author. figure 1 : the yellow masks indicate the missing modalities at these positions. without specific designs to handle modality missingness, models may predict 'happy' when all modalities are present, but shift to 'neutral' when some modalities are missing. in real-world data, caused by sensor failures, data corruption, or privacy constraints. modality missingness is random and prevalent, impacting both the training and inference phases. in real-world scenarios, models built on the assumption that all modalities are fully available @xcite @xcite may be misguided by missing modalities @xcite , as shown in figure 1 . it suggests that those models are sensitive to missing modalities. although a wide range of methods have been proposed to address modality missingness @xcite @xcite , many of them face practical limitations. these approaches generally rely on additional information to function effectively. some approaches @xcite @xcite assume the availability of complete multimodal data during the training phase. however, such an assumption is often unrealistic in real-world scenarios. other methods depend on explicit annotations that indicate which modalities are missing for each input sample @xcite @xcite . while these annotations can guide the learning process, they impose a substantial manual workload, demanding consistent and accurate labeling across extensive datasets. moreover, these annotations may not always be reliable or even feasible in dynamic, open-world settings. therefore, these limitations make existing solutions difficult to scale or generalize. moreover, current methods for addressing modality missingness primarily focus on alleviating the performance decline resulting from incomplete data @xcite @xcite . however, they neglect a more profound and inherent issue in multimodal machine learning: modality imbalance. modality imbalance refers to the unequal learning proficiency of a model across different modalities, where the model may over-rely on certain modalities while underutilizing or mislearning others. in such scenarios, dominant modalities disproportionately influence predictions, whereas other modalities contribute minimally or even introduce noise @xcite @xcite . modality missingness and modality imbalance are two distinct problems, yet they often co-occur and may mutually reinforce each other. modality imbalance can intensify the impact of missing modalities. it makes models more vulnerable when dominant modalities are absent. conversely, missing modalities can exacerbate the imbalance by further restricting the learning of modalities with lower contributions. in this work, we propose dynamic modality recognition and enhancement for adaptive multimodal fusion framework (dream), a novel framework that addresses both missing modality and imbalanced modality dynamically without explicit labels. notably, dream does not require full-modality data during training or manual annotations to identify missing modalities like previous works do. specifically, dream dynamically estimates the actual contribution of each modality. based on these estimates, dream identifies modalities that are missing or contribute little to the prediction and selectively reconstructs and enhances them. to effectively integrate both reconstructed and observed modalities, we introduce an adaptive fusion mechanism in dream. by dynamically modulating the fusion process according to the estimated modality contributions, the framework produces more effec-tive and resilient decisions, even when dealing with missing or imbalanced modalities. our main contributions are summarized as follows: ## experiments in this section, we test dream's performance to address the following research questions. ## conclusion in this paper, we present dream, a framework that addresses both missing and imbalanced modalities. dream dynamically identifies modalities with low contributions and reconstructs them to enhance overall representation quality. dream adaptively fuses both the original and reconstructed modalities based on their estimated contributions. importantly, the entire learning process is carried out without the need for full-modality supervision or explicit annotations indicating missing modalities. experiments on three benchmark datasets demonstrate that dream achieves strong performance and robustness with missing and imbalanced modalities. the results of the ablation studies further highlight the effectiveness of dream's dynamic modality contribution recognition and its robustness to different pre-trained models.