Treble Counterfactual VLM s: A Causal Approach to Hallucination (2025)

Abstract: Vision-Language Models (VLMs) excel at tasks such as image captioning and visual question answering but frequently produce hallucinated outputs that deviate from the actual visual input or prompt. While prior work links hallucination to biases in data or representation, their causal origins remain unclear. We propose a causal framework to analyze and mitigate hallucination in VLMs. Our key hypothesis is that hallucinations arise from unintended direct influences of the vision or text modality that bypass the intended multi-modal fusion. To examine this, we construct a causal graph of the VLM and use counterfactual analysis to estimate the Natural Direct Effect (NDE) of each modality and their interaction. By systematically identifying and suppressing these direct effects, we encourage outputs that are more faithfully grounded in true cross-modal reasoning. Our approach consists of three steps: (1) designing structural causal graphs to distinguish correct fusion pathways from spurious modality shortcuts, (2) estimating modality-specific and cross-modal NDE using perturbed image representations, hallucinated text embeddings, and degraded visual inputs, and (3) implementing a test-time intervention module to dynamically adjust the model’s dependence on each modality. Experimental results demonstrate that our method significantly reduces hallucination while preserving task performance, providing a robust and interpretable framework for improving VLM reliability.

## introduction vision-language models (vlms) have made significant progress in multi-modal tasks such as image captioning @xcite , visual question answering, and visual reasoning @xcite @xcite @xcite . by integrating visual and textual inputs, vlms generate descriptive outputs that enhance machine understanding of multi-modal contexts @xcite . they typically comprise a vision encoder for extracting image features and a language model for generating outputs conditioned on both modalities. advances in large-scale pre-training and transformer architectures have further improved their generalization @xcite , making vlms key to ai applications. hallucination in vlms. despite strong performance, vlms are prone to hallucination @xcite : producing outputs inconsistent with the visual input or textual prompt, often introducing incorrect or fabricated information. this reduces reliability in high-stakes domains such as medical imaging @xcite , autonomous driving @xcite , and surveillance @xcite . while several factors contribute to hallucination, e.g., modality misalignment and learned biases, its root causes remain understudied, which needs systematic investigation and mitigation. existing approaches. prior work has explored various ways to understand and reduce hallucination in vlms @xcite @xcite , with different explanations and mitigations. some studies link hallucinations to biases in training data @xcite , where models latch onto spurious correlations rather than truly learning visualtext relationships. others point to overreliance on language priors @xcite , leading to text-focused outputs that overlook visual context. additional research highlights biased feature learning @xcite @xcite , which can cause certain patterns to dominate the representations and distort multi-modal reasoning. however, most approaches focus on statistical or empirical analyses and often do not differentiate vlms from llms, overlooking the distinct challenges inherent in multi-modal architectures in vlms. ## methodology building on prior work in editing vision-language model intermediate representations @xcite , we quantify the ndes of different modalities by analyzing representation shifts before and after applying modality-specific perturbations. this allows us to analyze separately the contributions of vision and text, along with their interaction, to the final model output. ## conclusion in this work, we introduced a causal framework to analyze and mitigate hallucination in vlms. by constructing structural causal graphs and estimating the natural direct effect of each modality, we identified unintended direct modality influences as a key contributor to hallucination. our proposed test-time intervention mechanism effectively reduces modality bias, ensuring that generated outputs are more accurately grounded in fused multi-modal information. empirical results across multiple benchmarks demonstrate that our method improves the reliability of vlms while maintaining task performance.