N eg VQA : Can Vision Language Models Understand Negation? (2025)

Abstract: Negation is a fundamental linguistic phenomenon that can entirely reverse the meaning of a sentence. As vision language models (VLMs) continue to advance and are deployed in high-stakes applications, assessing their ability to comprehend negation becomes essential. To address this, we introduce NegVQA, a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. We construct NegVQA by leveraging large language models to generate negated versions of questions from existing VQA datasets. Evaluating 20 state-of-the-art VLMs across seven model families, we find that these models struggle significantly with negation, exhibiting a substantial performance drop compared to their responses to the original questions. Furthermore, we uncover a U-shaped scaling trend, where increasing model size initially degrades performance on NegVQA before leading to improvements. Our benchmark reveals critical gaps in VLMs’ negation understanding and offers insights into future VLM development. Project page available at https://yuhui-zh15.github.io/NegVQA/.

## introduction vision language models (vlms) such as gpt-4o and claude have demonstrated remarkable capabilities in understanding and reasoning about visual content through natural language interactions @xcite . these models can answer image-based questions, generate descriptions, and engage in multi-turn dialogues about visual scenes @xcite @xcite . more recently, they have been integrated into embodied ai systems and robotics, allowing direct interaction with environments and humans in high-stakes scenarios @xcite @xcite . despite their impressive progress, vlms' ability to understand negation @xcite -a fundamental linguistic phenomenon that can completely alter the meaning of a sentence-remains poorly understood. a failure to correctly interpret negation can lead to critical errors, particularly in interactive ai systems. for instance, if a user instructs a vlm not to take a certain action or asks about something that is absent, misunderstanding negation could result in actions contrary to user intent and pose serious safety risks. to address this, we introduce negvqa, a visual question answering (vqa) benchmark designed to assess vlms' comprehension of negation. while existing vqa datasets primarily focus on affirmative questions, negvqa systematically examines negation understanding across diverse scenarios. the dataset consists of 7,379 two-choice questions, covering a range of negation types, including cases where objects are absent, attributes such as colors or sizes are negated, actions are described in terms of what is not happening, and more complex forms of negation that require deeper reasoning. to construct negvqa, we leverage large language models to generate natural negations of questions from existing vqa datasets, ensuring fluency while creating challenging evaluation cases that test both linguistic and visual understanding. we evaluate 20 state-of-the-art vlms across seven model families and find that negation remains a major challenge. despite their strong performance on standard vqa tasks, all models struggle significantly when faced with negated questions. for instance, qwen2-vl-72b @xcite , the best-performing model, achieves 92.2% accuracy on original questions but drops nearly 20 percentage points to 72.7% on negvqa. furthermore, we observe a u-shaped scaling trend, where increasing model size initially leads to worse performance on negation before eventually improving. this finding raises important questions about how where is this picture not taken at? a. office b. internet cafe vqav2 what is not the main color of the rocky area in the image? a. gray b. beige seedbench what is not parked in front of the garage? a. motorcycle b. car gqa what would not be impacted by an increase in owls? a. mice b. grass mathvista who did not write this? a. joannem amstelodami b. samuelis pufendorfii textvqa who is not the author of this book? a. gary chapman b. steven carter ocrvqa what was not glasgow underground's highest number of journeys in 2007/08? a. 14.0 b. 14.5 chartqa which is not the final stage in the life cycle of a grasshopper? a. adult b. nymph ai2d negvqa vlms process negation and how to scale up vlms to enhance negation understanding abilities. in summary, we propose negvqa, a critical diagnostic tool for evaluating negation comprehension in vlms. our study establishes baseline performance across major vlm families, reveals their significant shortcomings and uncovers scaling behaviors. these insights highlight the need to develop more robust and trustworthy vlms that can accurately handle negation, a fundamental aspect of natural language understanding. ## dataset: negvqa this section details the construction and statistics of negvqa, our benchmark for evaluating vision language models' ability to handle negation. ## results in this section, we describe our experimental setup and present our findings on vlm performance on negvqa. our evaluation highlights two key insights: current vlms exhibit significant difficulty in understanding negation, regardless of their size or architecture, and model scaling exhibits a ushaped performance trend. ## related work vision language models (vlms). vlms enable multimodal understanding by modeling p(y t |@xmath0, x) in an auto-regressive manner, where y i represents text tokens and x represents visual input. modern vlms typically comprise three key components: a visual encoder (often clip @xcite ), a language model, and a linear or mlp projector connecting them. notable examples include proprietary models such as @xcite and claude @xcite , as well as open-source models like llava @xcite and blip @xcite . these models are generally trained on image-text pairs and instruction-tuning datasets, leveraging pre-trained vision and language components. while they exhibit strong performance on various image understanding tasks @xcite @xcite and have been applied in embodied ai and robotics @xcite @xcite , their ability to handle negation remains largely unexplored. negation understanding. negation plays a fundamental role in language comprehension @xcite . most prior research has focused on evaluating language models' ability to understand negation @xcite @xcite . more recently, studies have begun assessing clip @xcite 's understanding of negation @xcite @xcite . however, to the best of our knowledge, no prior work has systematically evaluated negation comprehension in generative vlms. in this work, we introduce negvqa, the first benchmark designed to assess vlms' ability to handle negation. given the increasing deployment of vlms in real-world embodied ai systems, understanding their limitations in processing negation is crucial, as failures in user intent interpretation could lead to unintended and risky scenarios. scaling trends. scaling up models has been a dominant approach in advancing foundation models. however, most scaling studies have focused on language models @xcite @xcite . while many tasks benefit from scaling, some exhibit inverse scaling @xcite or u-shaped scaling @xcite . in this work, we analyze scaling effects in vision language models on the negation task and reveal a similar u-shaped scaling pattern. ## conclusion in this work, we present negvqa, a benchmark designed to evaluate vision language models' ability to comprehend negation. our analysis of 20 vlms highlights their significant limitations in handling negation and uncovers a u-shaped scaling pattern in performance. we envision negvqa as a valuable resource for advancing linguistically competent, safe, and trustworthy vision language models.