MM -R ^3 : On (In-)Consistency of Vision-Language Models ( VLM s) (2025)

Abstract: With the advent of LLMs and variants, a flurry of research has emerged, analyzing the performance of such models across an array of tasks. While most studies focus on evaluating the capabilities of state-of-the-art (SoTA) Vision Language Models (VLMs) through task accuracy ( \textit{e.g.} , visual question answering, grounding), our work explores the related but complementary aspect of \textit{consistency} – the ability of a VLM to produce semantically similar or identical responses to semantically similar queries. We note that consistency is a fundamental prerequisite (necessary but not sufficient condition) for robustness and trust in VLMs. Armed with this perspective, we propose the MM-Rbenchmark, which allows us to analyze performance, in terms of consistency and accuracy, of SoTA VLMs on three tasks: Question Rephrasing, Image Restyling, and Context Reasoning. Our analysis reveals that consistency does not always align with accuracy, indicating that models with higher accuracy are not necessarily more consistent, and vice versa. Furthermore, we propose a simple yet effective mitigation strategy in the form of an adapter module trained to minimize inconsistency across prompts. With our proposed strategy, we are able to achieve absolute improvements of 5.7% and 12.5%, on average on widely used VLMs such as BLIP-2 and LLaVa 1.5M in terms of consistency over their existing counterparts.

## introduction vision language models (vlms) @xcite @xcite , following and often built on top of purely lingual llm @xcite , have recently emerged as incredible tools for a broad range of visual understanding tasks, spanning captioning @xcite @xcite , language grounding @xcite @xcite , visual question answering (vqa) @xcite , and many others. as the number of such models and their capabilities explode, the research community is progressively focusing on benchmarking their capabilities by developing new benchmarks and testing harnesses. notable examples include mm-bench @xcite , seed-bench @xcite , mm-vet @xcite , and others that define numerous tasks that capture a broad range of capabilities of such models (e.g., instance counting @xcite , spatial relation understanding @xcite , abductive @xcite and deductive @xcite reasoning). these benchmarks continue to shed light on abilities and limitations of vlms by analyzing their accuracy. however, despite significant progress in the anal-yses of vlms, consistency foot_0 -ability to produce semantically equivalent response with the semantically equivalent inputs, remains broadly overlooked. this is a fundamental requirement for vlms to be reliably deployable for most tasks. anecdotally, llms and, by extension, vlms are sensitive to their prompts which led to the widespread practice of prompt engineering. this is problematic as the models' outputs may vary with the phrasing of a query rather than its actual intent, which undermines their reliability. consider the example illustrated in figure 1 (mid) top: most humans would realize that while the three questions (i.e r1, r2, and r3) are superficially different, the semantic meaning is the same. hence even when the correct answer may not perhaps be apparent (i.e., "to be visible"), the same (consistent) answer should be produced. in contrast, asking models like blip-2 @xcite to answer these questions results in varied responses "to protect them from splinters","to protect the horse's legs","to make the hooves more visible" for the three questions considered. similarly, it is obvious to humans that the object being masked in figure 1 (mid) bottom is the same irrespective of the type of the mask, and that the object in question is a "bench" however, qwen-vl-chat's responses vary, indicating "a bench","a yellow object", "a green wooden slat sign" for different masks, highlighting the inconsistency in visual modifications. it may be tempting to equate accuracy and consistency, but the relationship is more intricate. while it is true that for objective visual tasks (e.g., what color is an object), high accuracy will result in high(er) consistency, current vlms are far from this high accuracy regime. further, for more subjective visual tasks (e.g., abductive reasoning), high accuracy which tends to be measured as being similar to one of the sets of answers, may not lead to high consistency. in general, one can think of consistency as a necessary but not sufficient property of an ai system and one should seek to maximize both consistency and accuracy. in this work, we first present a comprehensive analysis of sota vlms in terms of their consistency. we do so by developing mm-r 3 benchmark atop of the vqa task, where we produce both lingual rephrasings of the original questions (by leveraging gpt-3.5) and visual rephrasings of the image (through stylization) and measure both accuracy and consistency of the produced responses. we find that sota vlms while often quite competitive in accuracy can differ substantially in their consistency of responses. for example, mplug-owl2 @xcite is much more susceptible to inconsistency when image inputs are perturbed while moe-llava @xcite is more consistent in the change of the visual domain than the lingual domain. in addition, we also define an abductive task of predicting the contents of the masked region, where we find models like blip-2 and llava 1.5m @xcite are lower in accuracy but have higher consistency. overall results for vlms are illustrated in figure 1 (left). we believe these findings both benchmark the capabilities of existing models and outline future directions and developments in more consistent vlms and pre-training objectives (e.g., see efforts in @xcite @xcite ). toward the latter goal, we propose a simple adapter module based strategy that effectively improves consistency. the adapter is flexible and can be added to any existing vlm. it sits between the vlm embedding layer and the frozen llm decoder. the goal of the adapter is to modify the llm's embeddings such that they are invariant to surface form variations in the language prompt / question or the image. we test the proposed adapter in widely used vlms such as blip-2 and llava 1.5m models. the experiments show that it is able to improve performance on all three tasks in our proposed benchmark (shown in figure 1 (right)). our contributions are summarized as follows: ## consistency improvement the analyses on the mm-r 3 benchmark in section 4 show that consistency drops significantly with semantically equivalent prompts. to overcome this issue, we propose a simple strategy based on an adapter module that can be plugged into any vlm with lightweight training (figure 3 (green)). the goal of the adapter is to help vlms overcome the variability of visual and language prompts by making them invariant to surface form variability in data and ensuring a single consistent output. ## conclusion in this paper, we explore and analyze consistency of vlms and its relationship to accuracy. we do so by introducing the mm-r 3 benchmark, comprising three tasks -question rephrasing, image restyling, and context reasoning -to conduct a comprehensive analysis of sota vlms in terms of both accuracy and consistency. our analysis reveals that higher accuracy does not necessarily equate to greater consistency in models, and vice versa. in addition, we observe significant variations in the consistency of sota models, while their accuracy levels tend to be more stable across models. these findings suggest that future vlm development and objectives might benefit from a dual focus, emphasizing the optimization of consistency and the improvement of overall accuracy. furthermore, we propose an effective adapter to improve consistency performance. experiments on blip-2, llava validate improvements. acknowledgments. this work was funded, in part, by the vector institute for ai, canada cifar ai chairs, nserc canada research chair (crc), and nserc discovery grants. resources used in preparing this research were provided, in part, by the province of ontario, the government of canada through cifar, the digital research alliance of canada foot_1 , companies sponsoring the vector institute, and advanced research computing at the university of british columbia. additional hardware support was provided by john r. evans leaders fund cfi grant and compute canada under the resource allocation competition award.