Efficient Prompting for Continual Adaptation to Missing Modalities (2025)

Abstract: Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, performance can degrade significantly. Current methods often aggregate various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and the risk of catastrophic forgetting in continual environments where data arrives sequentially. In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task. To address this challenge efficiently, we introduce three types of prompts: modality-specific, task-aware, and task-specific prompts. These prompts enable the model to learn intra-modality, inter-modality, intra-task, and inter-task features. Furthermore, we propose a contrastive task interaction strategy to explicitly learn prompts correlating different modalities. We conduct extensive experiments on three public datasets, where our method consistently outperforms state-of-the-art approaches.

## introduction pre-trained multimodal models have shown great potential in many applications @xcite @xcite . when finetuning these pre-trained models on downstream tasks, missing modality issues often occur due to equipment failure, data corruption, privacy concerns, etc. existing methods @xcite @xcite address missing modality issues by reconstructing missing information or aligning multimodal features. however, both recovering missing features and aligning multimodal features are based on ## conclusion in this paper, we introduce the task of continual multimodal missing modality to tackle the challenges posed by missing modalities in dynamic environments. we propose a novel and efficient prompt design consisting of three distinct types of prompts, complemented by a contrastive task interaction strategy aimed at mitigating catastrophic forgetting in the multimodal domain. our approach facilitates effective learning of intra-modality, intermodality, intra-task, and inter-task features, enhancing the model's adaptability. extensive experiments and ablation studies validate the robustness and efficacy of our proposed method. given that cases of missing modalities frequently arise during data collection in real-world scenarios, we believe our approach represents a significant step towards practical applications in multimodal fields facing ongoing missing modality challenges.