Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing (2025)

Abstract: Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations, undermining their robustness and generalization in complex multimodal reasoning tasks. This paper addresses the critical challenge of superficial correlation bias in MLLMs through a novel causal mediation-based debiasing framework. Specially, we distinguishing core semantics from spurious textual and visual contexts via counterfactual examples to activate training-stage debiasing and employ a Mixture-of-Experts (MoE) architecture with dynamic routing to selectively engages modality-specific debiasing experts. Empirical evaluation on multimodal sarcasm detection and sentiment analysis tasks demonstrates that our framework significantly surpasses unimodal debiasing strategies and existing state-of-the-art models.

## introduction multimodal large language models (mllms) have demonstrated significant capabilities in integrating information from various modalities, such as vision and language, achieving notable success in multimodal tasks @xcite @xcite . by unifying modalities, mllms can capture richer semantic information than unimodal approaches, establishing them as a promising direction for complex, real-world applications @xcite . despite this progress, current mllms remain unreliable on tasks requiring nuanced semantic understanding and reasoning. in practice, they often over-rely on spurious correlations in one modality instead of truly fusing information, which leads to cause contains a lively day that felt like the world was wide awake and smiling. even the crowd felt gentle today. image: ## multimodal debiasing methods in this section, we propose methods to mitigate spurious multimodal biases, targeting both inference and training stages: 1. multimodal inference debiasing (mid): an inference-stage, plug-and-play method for external bias correction without altering model parameters. ## experimental setup in this section, we outline the experimental framework used to evaluate our proposed methods. comprehensive details regarding datasets, evaluation metrics, implementation specifics, and all comparative methods are provided in app. c. ## conclusion in this paper, we addressed the critical issue of spurious multimodal biases that impair the robustness and generalization of mllms. prior approaches lacked comprehensive multimodal analysis and training-time adaptability. we first developed a fine-grained multimodal causal framework, explicitly distinguishing spurious context from semantic content, thereby extending previous inference-time adjustments to multimodal scenarios. building upon this, we introduced mme-jd, incorporating counterfactual content into training and employing a mixture-of-experts architecture with dynamic routing for adaptive, sample-specific debiasing. extensive experiments demonstrate that mme-jd significantly surpasses simpler debiasing methods and existing benchmarks. future research will explore enhancing counterfactual generation techniques and refining router designs for enhancement. limitation despite the improvements demonstrated, our approach has several limitations that open avenues for future research: accuracy of spurious context identification: the efficacy of our framework critically depends on the precise identification of spurious contexts versus core semantic content. our current methodology, employing prompt engineering and attentionbased mechanisms for semantic extraction, faces inherent challenges. prompt engineering can inadvertently introduce new biases, while attention scores may not always reliably pinpoint genuinely spurious regions. therefore, despite the advanced nature of llms, inaccuracies in this initial extraction phase can propagate the effectiveness of subsequent debiasing efforts. future studies should integrate additional robust and objective methodologies alongside these advanced models to further enhance extraction accuracy and reliability.