Analyzing the Sensitivity of Vision Language Models in Visual Question Answering (2025)

Abstract: We can think of Visual Question Answering as a (multimodal) conversation between a human and an AI system. Here, we explore the sensitivity of Vision Language Models (VLMs) through the lens of cooperative principles of conversation proposed by Grice. Specifically, even when Grice’s maxims of conversation are flouted, humans typically do not have much difficulty in understanding the conversation even though it requires more cognitive effort. Here, we study if VLMs are capable of handling violations to Grice’s maxims in a manner that is similar to humans. Specifically, we add modifiers to human-crafted questions and analyze the response of VLMs to these modifiers. We use three state-of-the-art VLMs in our study, namely, GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-Flash on questions from the VQA v2.0 dataset. Our initial results seem to indicate that the performance of VLMs consistently diminish with the addition of modifiers which indicates our approach as a promising direction to understand the limitations of VLMs.

## introduction vision language models (vlms) @xcite @xcite that unify large language models with computer vision have made significant advances in multimodal tasks such as image captioning @xcite @xcite and visual question answering (vqa) @xcite . however, we are just beginning to understand the reasoning capabilities and more importantly, the limitations of these models @xcite . in this work, inspired by theories from cognitive science, we understand the behavior of vlms in vqa when we increase the cognitive load in comprehending questions. specifically, in grice's classical theory of cooperative principles @xcite , it is known that humans acting cooperatively in a conversation typically need to follow a set of rules commonly known as grice's maxims. these maxims make conversation more effective and ensure efficient communication. however, it is known from previous studies that even when these maxims are violated, humans can comprehend conversation easily @xcite . however, violations to grice's maxims places greater cognitive burden on the listener @xcite . in this work, we study how vlms react when grice's maxims are violated. specifically, we treat vqa as a single utterance conversation where a human is asking the ai model a question to which the ai model responds with an answer. we introduce modifiers into human-crafted questions that adds greater detail to a question. at the same time, these details typically tend to violate grice's maxims since they were not deemed to be essential when a human crafted the original question. while an ai model could benefit from the added information, processing modifiers will increase the reasoning required to answer the question. we add two types of modifiers, namely, visual and relational modifiers. the visual modifiers add more detail related to visual properties such as color, shape, etc., while relational modifiers add details related to spatial relationships. we use vlms to generate a modified question with either visual or relational modifiers. next, we verify if the modified question changes human perception. that is, if humans can answer the modified question with an answer that is equivalent to the answer to an unmodified question, this implies that the modifier does not alter the question. therefore, we would expect a vlm to be able to do a similar type of reasoning. we evaluate this on three state-of-the-art vlms, gpt-4o (ope-nai, 2024), gemini-1.5-flash @xcite and claude-3.5-sonnet @xcite on the vqa v2.0 dataset. that is, we generate modified questions from each of these vlms and evaluate the responses of each vlm to the modified questions. our initial results seem to indicate that vlms are sensitive to modifications to questions. in particular, we find that there is a consistent performance degradation in the presence of modifiers. in particular, when modifiers are added through gemini-1.5-flash, the performance degradation is more significant in all 3 vlms. ## related work following the original vqa task @xcite , several improved datasets for vqa have been developed @xcite @xcite to evaluate vqa systems. more recently, the trend has shifted towards incorporating llms within the evaluation process. for instance, @xcite uses chatgpt to automatically evaluate model outputs on a likert scale. the work in @xcite leverages llms to evaluate answers. specifically, it formulates vqa as an answer-rating task where the llm @xcite , vicuna-v1.3 @xcite and gpt-3.5-turbo) is instructed to score the correctness of a candidate answer given a set of reference answers. the work in @xcite is related to our approach where it adds question modifiers to vqa and analyzes its effect on lxmert @xcite . however, there has not been a significant amount of work that relates the reasoning of vlms in vqa grounded in principles of human cognition which is the direction we follow in this work. ## pragmatics in visual question answering grice's classical theory of cooperative principles in pragmatics is widely used to characterize hu-man conversation. specifically, grice developed principles that explain effective conversation between participants assuming that the participants have a common goal of understanding each other and therefore act cooperatively. these principles are summarized in four maxims, namely, the maxim of quality, quantity, relation and manner. the maximum of quality suggests that speakers should be as truthful as possible and only say what they believe to be true based on evidence. the maxim of quantity suggests that the right amount of information must be provided in a conversation, i.e., one should not add too much or too little information. the maxim of relation suggests that a speaker should stay relevant to the topic and the maxim of manner suggests the need to avoid ambiguity and focus on clarity. while grice's maxims characterize effective conversation, violation of grice's maxims does not mean that the conversation is incomprehensible @xcite . specifically, since participants are assumed to be acting cooperatively, if a speaker violates a maxim, then the burden of understanding falls on the listener. that is, the listener is expected to work harder (cognitively) to comprehend the intention behind utterances that violate the maxims. we use this principle as a way to understand the limitations of vlms. specifically, we think of the vqa task as a conversation that involves a single utterance between two participants, i.e., one participant is a human who asks a question to the ai model and the other participant is the ai model that needs to generate an answer. in cases where the human participant flouts grice's maxims, there is an increased burden of understanding on the ai model to produce an answer that the human can agree on. ## conclusion in this work, we studied if vlms are sensitive to modifications to questions in vqa. specifically, adding modifiers increases details in a question, but when viewed from the perspective of cooperative principles, they can violate grice's maxims. humans can accurately ignore irrelevant details to answer questions even with these violations. we studied if vlms could do the same in vqa by generating modified questions from human-crafted questions that preserve the original answer. we used 3 state-of-the-art vlms in our study and showed that in most cases, adding modifiers to questions degrades the performance of the vlm. based on these initial results, we plan to develop more detailed experiments to understand the types of modifications that vlms are better at processing. further, while our current results reveal that the performance of vlms drops in the presence of modifiers, it is not yet clear as to why such a drop occurs. in future work, we plan to analyze the reasons for why some vlms tend to perform more poorly than others in modified questions.