Mixed Signals: Decoding VLM s’ Reasoning and Underlying Bias in Vision-Language Conflict (2025)

Abstract: Vision-language models (VLMs) have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. However, it is not clear how these models reason over the visual and textual data together, nor how the flow of information between modalities is structured. In this paper, we examine how VLMs reason by analyzing their biases when confronted with scenarios that present conflicting image and text cues—a common occurrence in real-world applications. To uncover the extent and nature of these biases, we build upon existing benchmarks to create five datasets containing mismatched image-text pairs, covering topics in mathematics, science, and visual descriptions. Our analysis shows that VLMs favor text in simpler queries but shift toward images as query complexity increases. This bias correlates with model scale, with the difference between the percentage of image- and text-preferred responses ranging from +56.8% (image favored) to -85.1% (text favored), depending on the task and model. In addition, we explore three mitigation strategies: simple prompt modifications, modifications that explicitly instruct models on how to handle conflicting information (akin to chain-of-thought prompting), and a task decomposition strategy that analyzes each modality separately before combining their results. Our findings indicate that the effectiveness of these strategies in identifying and mitigating bias varies significantly and is closely linked to the model’s overall performance on the task and the specific modality in question. We released our dataset and code.

## introduction vision-language models (vlms) have rapidly advanced the field of artificial intelligence by effectively combining visual and textual data to achieve state-of-the-art performance on a variety of tasks 1 https://github.com/megagonlabs/modality-bias figure 1 : we investigate vlms' bias toward text versus image inputs when mismatches occur between the modalities. our observations reveal that this bias heavily depends on the task's and sample's difficulty. for example, while the model relies on textual representations to compute the roots of a degree-2 polynomial, increasing the degree to 3 shifts the reliance more toward the visual representation of the function. @xcite @xcite . however, it remains unclear how these models integrate and reason over information from multiple modalities-a capability that becomes increasingly important in applications such as retrieval-augmented generation (rag) @xcite and multi-agent systems @xcite , where data can be sourced from diverse modalities. to understand how vlms reason over multiple modalities, one approach is to investigate their behavior when confronted with conflicting information. moreover, it is important to determine whether these models exhibit a bias toward one modality over the other, when exposed to deliberately mismatched image-text pairs (see figure 1 ). for example, consider a scenario in healthcare where a chest x-ray image shows no signs of pneumonia, yet the accompanying text erroneously describes clear evidence of the disease. if the model disproportionately relies on the text, it may incorrectly diagnose pneumonia, potentially leading to inappropriate treatment recommendations and severe patient harm. in this paper, we investigate vlms' biases in the presence of conflicting multimodal cues by building upon existing datasets to create five novel benchmarks @xcite . each benchmark consists of mismatched textual and image cues, with every sample assigned a complexity score to study the impact of sample difficulty on the models bias. these benchmarks span various topics, including mathematical reasoning, science, and visual descriptions. to generate the mismatched pairs, we begin with aligned image-text pairs from the original datasets and then employ a combination of rule-based methods and manual modifications to alter the textual content, thereby introducing conflicting information. we adopt six state-of-the-art vlms spanning a diverse range of scales and conduct experiments on our constructed benchmarks. our analysis reveals that when presented with conflicting information, vlms tend to favor textual data over images in simpler scenarios. however, as query complexity increases, the models shift their bias toward the modality they perceive to be simpler-in many cases, this is the image. furthermore, we observe that the extent of this bias correlates with the model's scale and strength. finally, our findings indicate that this modality preference arises partly from the models' perception of task difficulty and partly from other internal biases, and is strongly linked to llms performance with different modalities as the input on a given task. observing the significant degree of modality bias in our experiments, we set out to explore several mitigation strategies aimed at reducing this issue. we consider three approaches to address modality bias: simple prompt modifications, introducing explicit instructions similar to chain-of-thought prompting, and a decomposition approach that analyzes each modality separately before combining outputs. our experiments show that each approach exhibits a diverse impact based on the task, the model, and-most importantly-the model's performance when given different modalities as input. in scenarios where the model shows strong perfor-mance with both modalities (or even one modality) as the input, at least one of explored mitigation strategies demonstrate reasonable performance. ## conflicting modalities in this work, our goal is to understand how vlms reason over multiple modalities when they are presented with conflicting information. we simulate realistic scenarios in which the visual and textual inputs contradict one another, forcing the models to weigh and integrate disparate cues. in this section, we outline the problem setting and describe the methodology used to construct our benchmarks. we provide the prompts used in creating the benchmark and the data statistics of created benchmarks in the appendix. ## mitigation strategies our experiments indicate that vlms exhibit a notable bias when handling conflicting multimodal information (see section 4.2). to address this challenge, we propose three distinct mitigation strategies, as summarized in figure 2 with the adopted prompts provided in the appendix. these strategies aim to detect and mitigate the bias by altering the models' processing of visual and textual inputs, thereby encouraging a more balanced integration of information. the strategies are as follows: verbalized mitigation: directly prompting the model to identify and report any mismatches or contradictions between the modalities. instead of asking the models to provide only the answer, we additionally instruct them to indicate a mismatch if a discrepancy is detected. this explicit acknowledgment of conflict helps prevent the model from defaulting to one modality. cot mitigation: taking inspiration from chainof-thought prompting, in this method, the model is guided through a three-step process. first, we instruct the model to process the image input alone; second, evaluate the textual description independently; and third, combine the outputs from both modalities. we further instruct the model to extract all relevant information from each modality, particularly when a single modality does not suffice to solve the task. finally, the model is instructed to compare the two results and either provide an answer or indicate if a mismatch is detected. decomposed mitigation: in this method, we employ a multi-stage approach wherein the vlm is run three separate times. the first run solve the task using only the image input, the second only use the text input, and the third combines the outputs from the previous two runs. in the final stage, the model is specifically instructed to highlight any inconsistencies between the outputs, and if a discrepancy is detected, to report a mismatch. ## experiments in here, we first examine the extent of bias in vlms using our created benchmarks. next, we examine how bias correlates with the models' perceived problem difficulty and conduct an error analysis of their performance. finally, we explore the impact of various mitigation strategies in reducing the bias. ## related works as llms and vlms become more capable in reasoning tasks @xcite @xcite , it is essential to investigate the interplay between modalities and how models reason across them to solve tasks. previous studies have examined various aspects of vlm reasoning capabilities by introducing benchmarks that assess their general performance @xcite , or by focusing on specific abilities such as spatial reasoning @xcite and robot navigation @xcite . despite these efforts, it remains unclear how these models reason and combine information across multiple modalities, and whether they exhibit specific biases toward any input modality. many previous works investigate the robustness of vlms @xcite . for instance, @xcite demonstrate that vlms are sensitive to the order of words, while @xcite show that they struggle to distinguish between semantically equivalent but lexically different captions. furthermore, several studies have examined vlms robustness through adversarial attacks @xcite and jailbreak strategies @xcite . the study most closely related to ours is a concurrent work @xcite that examines vlms bias toward text and image inputs in conflicting scenarios. our approach differs in three key aspects. first, the authors in @xcite generate mismatching text using gpt-4o; in contrast, we create mismatched text using rule-based and manual methods, giving us greater control. second, while @xcite report that vlms exhibit a "blind faith in text", we demonstrate that vlms can, in fact, show a stronger bias toward images in certain tasks, and the extent of their bias is heavily correlated with sample difficulty. third, whereas their work focuses on supervised fine-tuning approaches for mitigation, which can be challenging and expensive, we explore the impact of postprocessing mitigation strategies. ## conclusion we investigate bias in vlms by creating five novel benchmarks that span various tasks and domains, including mathematical reasoning, science, and visual description. our findings reveal that vlms exhibit distinct biases toward either textual or vi-sual cues depending on the task, with the nature of this bias shifting according to sample difficulty and model capabilities-favoring text in simpler scenarios and images in more complex ones. furthermore, we explore a range of mitigation strategies-including verbalized, chain-ofthought, and decomposed approaches-which have demonstrated reasonable accuracy in detecting mismatches, provided the model achieves strong performance on the original task. these results underscore the complexity of multimodal reasoning in vlms and highlight promising directions for enhancing their reliability in real-world applications.