VISREAS : Complex Visual Reasoning with Unanswerable Questions (2024)

Abstract: Verifying a question’s validity before answering is crucial in real-world applications, where users may provide imperfect instructions. In this scenario, an ideal model should address the discrepancies in the query and convey them to the users rather than generating the best possible answer. Addressing this requirement, we introduce a new compositional visual question-answering dataset, VisReas, that consists of answerable and unanswerable visual queries formulated by traversing and perturbing commonalities and differences among objects, attributes, and relations. VisReas contains 2.07M semantically diverse queries generated automatically using Visual Genome scene graphs. The unique feature of this task, validating question answerability with respect to an image before answering, and the poor performance of state-of-the-art models inspired the design of a new modular baseline, Logic2Vision that reasons by producing and executing pseudocode without any external modules to generate the answer. Logic2Vision outperforms generative models in VisReas (+4.82% over LLaVA-1.5; +12.23% over InstructBLIP) and achieves a significant gain in performance against the classification models.

## introduction in visual question answering (vqa), validating question authenticity with the corresponding image and then reasoning over it is an important requirement in real-world application dynamics where users may make errors in judgment, leading to invalid queries. confirming a question's validity becomes pivotal to maintaining consistency, rectifying mistakes, and preventing misguided responses @xcite . following the prior vqa datasets' (goyal et al., 2017; krishna et al., 2016; 1 code and data at https://github.com/re-n-y/ visreas.git using scene graphs, we cluster objects (orange), relations, and attributes of the related objects (blue) based on the attribute of the corresponding objects (orange). then the question engine takes each template as input and traverses all possible clusters to generate the query as well as the reasoning steps. each function in the reasoning steps can return none if any object, attribute, or relation is absent in the image. @xcite ) focus on answerable questions only, a system trained solely for answerable questions may exhibit unstable behaviors when faced with unanswerable queries. for instance, a delivery robot receiving an incorrect address but a valid instruction like "place the package by the yellow door" might overlook the error unless prompted to reevaluate its decision. in contrast, presuming the correctness of the query would likely lead to unpredictable behaviors. therefore, a reliable and responsible system should be able to question the validity of the instruction it receives before acting upon it. however, aligning questions with the region of interest in the image breaks down visual reasoning task into perception (object detection and scene representation learning) and reasoning (question interpretation and inference grounded in the scene). datasets and models proposed to date have shown significant improvement in the detection task which therefore improved the perception system @xcite @xcite , but they face critical vulnerabilities due to the lack of generalities in the datasets @xcite . recent datasets @xcite @xcite encourage reasoning beyond surface-level object recognition and focus on multi-step inference. but they tend to reason about object relations (often questions revolving around single object) instead of reasoning over clusters of objects in the image that share common attributes or relations. reasoning over general sets of objects requires both identifying objects and understanding their attributes and relations.where prior scene-graph based work assumes reasoning follows from traversing a single path to generate an answer, our goal is to establish a multi-hop approach of identifying cliques with shared properties. bridging the gap in prior benchmarks, we introduce a new dataset, visreas (visual reasoning), for studying reasoning over commonalities and differences across objects. the unnatural assumption in the current vqa datasets -"a correct answer for every question" causes models to produce an answer even when the question is inapplicable and has no possible answer. to ensure that models verify the consistency of question text with the image before answering, we curate questions that have no answer given the image by altering relations and attributes among the objects. we design a question generation engine that takes the information about objects, attributes, and relations from the visual genome scene graphs @xcite and finds common features shared among multiple objects. based on this retrieved information, we generate 2.07m unique questions covering vast semantic variations. each question is paired with a scene graph and a semantic program that specifies the series of reasoning steps needed to be performed to produce the answer. our generated questions require visual reasoning abilities such as comparing, differentiating, counting, clustering objects, and performing logical reasoning. most importantly, unlike other vqa datasets, visreas enforces the vqa models to verify the information in the question with the image in each reasoning step before predicting an answer. we find existing vqa models less robust in the reasoning and unanswerable settings presented by visreas. motivated by the shortcomings of existing models, we propose a new architecture, logic2vision that has been trained to produce logical reasoning steps from the query at first and then predict answers based on the reasoning steps and the image. unlike prior generative models, logic2vision is compute and cost-efficient as it does not require any external expensive apis or modules and solely relies on the reasoning capabilities of visual language models (vlm). experiments on visreas shows that logic2vision outperforms the current fine-tuned vqa models: obtaining 66.20% (+4.82% over llava-1.5 @xcite , +12.23% over instructblip @xcite ) accuracy on visreas. in short, our contributions are twofold: ## related works recent years have witnessed tremendous progress in visual understanding. multiple attempts have been made to mitigate the systematic biases of vqa datasets @xcite @xcite , but they fall short in providing an adequate solution: some approaches operate over constrained and synthetic images @xcite , neglecting the realism and diversity natural photos provide. @xcite introduced a dataset for reasoning about semanticallydiverse natural language descriptions of images in the form of a classification task. while the dataset exhibits diverse semantic phenomena, this task rarely requires much beyond a single type of object recognition and its associated relation and attribute. unlike these datasets, visreas is open-ended and consists of both unanswerable and answerable queries based on the similarity/dissimilarity of multiple objects in the image. visreas jointly evaluates vqa models' alignment, multihop rea-soning, and verification ability which cannot be approximated by simply finding the most likely object/relation/attribute to answer the question. recent transformer-based models have @xcite @xcite achieved promising performance on visual reasoning tasks. yet, these models are prone to reproducing spurious correlations without accurately learning true causal relations @xcite @xcite . neuralsymbolic methods @xcite @xcite ) explicitly perform symbolic reasoning on the object and language representations. these models offer modularity and interpretability in the reasoning process. however, as module parameters are usually derived solely from end-task supervision, there is a potential for the program to deviate from accurately explaining the model's behavior @xcite @xcite . conversely, a recent approach to modularity leverages large language models (llm) to craft code or python programs using expensive apis @xcite @xcite . however, these approaches outsource basic aspects of the reasoning to external components rather than performing reasoning as part of the model itself. for example, prior works outsource basic cognitive abilities such as recognizing objects, counting, and even arithmetic operations. focusing on these limitations, our proposed logic2vision aims to leverage single vlm to address complex reasoning in a modular approach that shows promising performance across models of three different categories. ## visreas: visual reasoning the visreas dataset is an attempt towards better aligning model capabilities with real application circumstances. in parallel, visreas aims to develop complex compositional reasoning into the machine that involves consideration of relations among multiple objects and verification of alignment between information provided in the question and the image. in the following sections, we provide details about the visreas data generation pipeline and a comprehensive analysis of the visreas dataset. in the supplementary material, we conduct a detailed comparative study between visreas and the wellestablished gqa dataset, followed by details of the human evaluation process using mechanical turk. ## logic2vision in recent years, llms combined with code generation and chain-of-thought prompting have shown impressive performance in complex reasoning by generating intermediate reasoning steps before inferring the answer @xcite . however, these frameworks are often prone to hallucinations of llms and are too restricted in terms of reasoning they can perform and dependent on expensive external modules to execute the reasoning @xcite . to address these limitations and elicit the reasoning capability of vlms, we propose logic2vision, a two-stage vqa framework that (1) plans the necessary reasoning steps using the question and (2) executes the plan with the help of an image leveraging the sota vlm (figure 4 ). prompt: given an image, the user asks a question and assistant executes the code and logs the results step-by-step to provide an answer. user : what is the material of the fence that the black bird is sitting on? ## experiments and analysis in the subsequent sections, we conduct a comprehensive analysis of the visreas dataset and assess the performance of various benchmarks including logic2vision, gpt-4v (openai, 2023), and human participants, revealing a notable disparity from ## conclusion we introduce the visreas dataset, for real-world complex and multihop visual reasoning and compositional question answering. the dataset emphasizes object commonalities, differences, and relational aspects, necessitating validation of questiontext relevance with the image before answering. we describe the dataset curation process along with the performance of sota models from three different domains in our task. addressing the shortcomings in grounding and clustering in recent models, we propose a novel logic2vision baseline that deconstructs questions into pseudocodes and sequentially executes them using images to generate answers. we anticipate that this dataset and model will catalyze advancements in vqa research, pushing it toward complex semantic comprehension, robust reasoning, and addressing unanswerability when the provided context is not sufficient.