CLEVR _ HYP : A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images (2021)

Abstract: Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video. In this paper, we take visual understanding to a higher level where systems are challenged to answer questions that involve mentally simulating the hypothetical consequences of performing specific actions in a given scenario. Towards that end, we formulate a vision-language question answering task based on the CLEVR (Johnson et. al., 2017) dataset. We then modify the best existing VQA methods and propose baseline solvers for this task. Finally, we motivate the development of better vision-language models by providing insights about the capability of diverse architectures to perform joint reasoning over image-text modality. Our dataset setup scripts and codes will be made publicly available at https://github.com/shailaja183/clevr_hyp .

## introduction in 2014, michael jordan, in an interview @xcite said that "deep learning is good at certain problems like image classification and identifying objects in the scene, but it struggles to talk about how those objects relate to each other, or how a person/robot would interact with those objects. for example, humans can deal with inferences about the scene: what if i sit down on that?, what if i put something on top of something? etc. there exists a range of problems that are far beyond the capability of today's machines." while this interview was six years ago, and since then there has been a lot of progress in deep learning and its applications to visual understanding. additionally, a large body of visual question answering (vqa) datasets @xcite @xcite have been compiled and many models have been developed in most existing vqa datasets, scene understanding is holistic and questions are centered around information explicitly present in the image (i.e. objects, attributes and actions). as a result, advanced object detection and scene graph techniques have been quite successful in achieving good performance over these datasets. however, provided an image, humans can speculate a wide range of implicit information. for example, the purpose of various objects in a scene, speculation about events that might have happened before, consider numerous imaginary situations and predicting possible future outcomes, intentions of a subject to perform particular actions, and many more. among the above, an ability to imagine taking specific actions and simulating probable results without actually acting or experiencing is an important aspect of human cognition (figure 1 gives an example of this). thus, we believe that having autonomous systems equipped with a similar capability will further advance ai research. this is particu-larly useful for robots performing on-demand tasks in safety-critical situations or navigating through dynamic environments, where they imagine possible outcomes for various situations without executing instructions directly. motivated by the above, we propose a challenge that attempts to bridge the gap between state-ofthe-art ai and human-level cognition. the main contributions of this paper foot_0 are as follows; ## related work in this section we situate and compare our work with related areas such as implicit text generation/retrieval for a visual, visual question answering (vqa) over synthetic images, question answering (qa) involving hypothetical reasoning, and language-based manipulation in visual domains closest to clevr_hyp. ## clevr_hyp task and dataset figure 2 gives a glimpse of clevr_hyp task. we opt for synthetic dataset creation as it allows automated and controlled data generation at scale with minimal biases. more details are described below. ## baseline results in this section, we benchmark models described above on the clevr_hyp. the dataset is formulated as a classification task with exactly one correct answer, so we use standard accuracy as evaluation metric. we then analyze their performance according to question and action types. quantitative results from above experiments can be visualized in top part of the overall baseline performance for various test sets of clevr_hyp original test balanced test 2hopta test 2hopqh test bl1 bl2 bl3 bl4 bl1 bl2 bl3 bl4 bl1 bl2 bl3 bl4 bl1 bl2 bl3 into the templates that we use to convert image into a machine comprehension passage. two of our models (scene graph update and textediting image) are transparent to visualize intermediate changes in the scene after performing actions. we analyse their ability to understand actions and make appropriate changes as shown in below part of both scene graph and text-editing models do quite well on 'remove' and 'change' actions whereas struggle when new objects are added or existing objects are moved around. the observation is consistent when multiple actions are combined. therefore, actions remove+change can be performed with maximum accuracy whereas other combinations of actions accomplish relatively lower performance. it leads to the conclusion that understanding the effect of different actions are of varied complexity. most models demonstrate better performance over counting, existence and attribute query type of questions than comparison questions. the scene graph update and text-editing methods show a performance drop of 6.1% and 9.1% respectively when multiple actions are performed on the scene. however, there is less of a performance gap for models on 2hopq h compared to the test set, suggesting that models are able to better generalize with respect to multiple reasoning skills than complex actions. ## conclusion we introduce clevr_hyp, a dataset to evaluate the ability of vqa systems after hypothetical actions are performed over the given image. we create this dataset by extending the data generation framework of clevr @xcite ) that uses synthetically rendered images and templates for reasoning questions. our dataset is challenging because rather than asking to reason about objects already present in the image, it asks about what would happen in an alternative world where changes have occurred. we provide ground-truth representations for images, hypothetical actions and questions to facilitate the development of models that systematically learn to reason about underlying process. we create several baseline models to benchmark clevr_hyp and report their results. our analysis shows that the models are able to perform reasonably well (70.5%) on the limited number of actions and reasoning types, but struggle with complex scenarios. while neural models have achieved almost perfect performance on clevr and considering human performance as upperbound (98%), there is a lot of room for improvement on clevr_hyp. our future work would include relaxing constraints by allowing a larger variety of actions, attributes and reasoning types. by extending this approach further for natural images, we aim to contribute in the development of better vision+language models.