QLEVR : A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning (2022)

Abstract: Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR (John- son et al., 2017), for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual question-answering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models. Code and Dataset are available at https://github.com/zechenli03/QLEVR

## introduction visual question answering is at the locus of computer vision and natural language processing, and its objective is developing computer vision systems that can answer arbitrary natural language questions about images @xcite @xcite . this is useful across a range of applications, including medical image analysis, accessibility for visually impaired, video surveillance, art and advertisement @xcite . the complexity of visual question answering naturally depends on the complexity of the images and the complexity of the natural language questions. the task reduces to object recognition for very simple questions of the form: question: are all the cyan metallic triangular prisms on the brown plane? answer: true question: on the non-white planes on the left rear side of the black wood rectangular plane, all the cyan metallic cubes but at least 2 are larger than at most 7 cubes; is it right? answer: false figure 1 : a sample image and questions from qlevr. tasks involve attribute recognition, counting, comparing numbers, spatial relationships, and understanding of quantifiers. object recognition can of course be a very complex task on its own, depending on the types of objects, the number of possible objects to be recognized, the amount of supervision for inducing a good model, general image quality, etc. however, more complex queries such as (2) make visual question answering much harder: answering such a question in the presence of an image requires a computer vision system that not only recognizes objects, but also relations between them. clevr @xcite probes computer vision systems's ability to answer even more complex queries, such as, for instance: question (3) involves reasoning about the relation between two objects, as well as the compositional semantics of color adjectives. in addition to shapes and colors, clevr also includes questions about sizes and quantities. in this paper, we present a novel visual questionanswering dataset that goes beyond clevr in focusing specifically on quantificational language, e.g.: given the complexity of quantificational language, the rich typology of expressions of quantification across different languages, and the interest from philosophy, it is perhaps surprising that quantificational language has received relatively little attention in the nlp community (see §2), but we believe it is a crucial step in pushing the research horizons in (visual) question-answering. contributions based on a comprehensive typology of english quantifiers, we build a dataset of 100,000 synthetic images and 999,446 unique questions to these images. this is roughly the same size as or a little bigger than clevr @xcite . our questions are on average longer than previous datasets. we evaluate three baselines from @xcite , a text-only baseline based on bert @xcite , and mac (hudson and manning, 2018) on qlevr and analyze performance across quantifier types. ## related work visual question answering challenge datasets several synthetic challenge datasets for visual question answering exist: @xcite presents shapes, a predecessor to clevr and qlevr, relying also on synthetic constellations of colored geometric shapes and template-driven question generation. @xcite create a similar dataset to probe visual question answering models for knowledge of adjectival semantics. a portion of the visual question answering dataset @xcite contains synthetic cartoon imagery. @xcite present an extension of clevr that probes for hypothetical reasoning of the form: if someone removed three triangles from this image, how many would be left? malinowski and fritz (2014) combined natural images with synthetic, template-driven question generation. finally, @xcite recently created a dataset of three-image scenes to probe two-step reasoning. synthetic visual question answering datasets have several advantages over ones based on real images and questions that tend to suffer from selection biases @xcite , but of course they are limited in what can be induced from them. they are therefore mostly useful for probing the limitations of visual question answering architectures and off-the-shelf models. showing results only on synthetic data is often seen as a weakness in the literature @xcite , but synthetic data is useful for diagnosing the errors of visual question answering systems, in our case highlighting the challenges posed by quantifiers. quantifiers quantifiers have been largely ignored in the nlp community. question-answering datasets have been developed for numerical reasoning in english @xcite , and some have identified quantifier words as important sources of errors for textual entailment systems @xcite . @xcite recently focused on the two quantifier words part and whole in an error analysis for named entity recognition. ## qlevr we design a challenge dataset called qlevr (for quantificational language and elementary visual reasoning) that requires more complex reasoning than previous visual question-answering datasets. qlevr is designed to probe the visual reasoning capabilities of visual question-answering systems with respect to quantificational language, including detecting members of sets, quantifying sets, and reasoning about the relationships between sets. to this end, we automatically construct scene graphs @xcite and use these to generate synthetic images with ground-truth locations, attributes, and relationships for planes and objects. each scene graph can be queried in a number of way, and we design query templates to render natural language questions involving complex reasoning about sets of such planes and objects. we describe each of these steps in detail: image generation all images in qlevr are images of objects organized in a particular way on a desk-like surface. figure 2 shows how the images are generated. we construct a scene graph for a two-dimensional image containing areas and objects of different sizes and shapes. scene graphs determine the ground-truth locations, bounding boxes, attributes and relationships for the planes and objects in the form of a graph or tree structure. nodes are planes or objects annotated with attributes, each of which is connected to its spatially related nodes. each image contains one to five areas or geometric planes. these can be either triangular, rectangular or circular. the rest of the desk area we refer to as the white non-geometric plane. geometric planes come in two materials (marble and wood), three colors (black, gray, and brown), and random sizes. each geometric plane contains one to ten (1-10) objects, with different sizes and shapes, and the non-geometric plane contains one to twelve (1-12) objects, with different sizes and shapes. object come in seven shapes (cone, cube, cylinder, pentahedron, sphere, triangular prism, and tetrahedron), two absolute sizes (small and large), five materials (metal, rubber, leather, marble, and wood), and eight colors (blue, brown, cyan, gray, green, purple, red and yellow). the spatial relationships between planes and objects include front, back, left and right, as well as right front, right rear, left front and left rear. we render three-dimensional images of the scene graphs with @xcite . light settings and three preset camera positions were chosen at random, after validating that all objects were at least partially visible. since the depth of the scene can affect the judgment of the spatial relationship in the three-dimensional image, the desk boundary is always visible as a reference for determining the depth of the scene. minimum distances between objects and planes were kept to reduce the ambiguity of spatial relationships. see appendix b for more details. question generation quantifiers are often said to be among the most important and complex constructs of natural languages @xcite . as pointed out by by @xcite , visual questionanswering models need to master a wide range of linguistic phenomena, including negation, entailment, mutual exclusivity and so on. we add (generalized) quantifiers to this list and design a dataset to probe the ability of visual question-answering systems to handle quantifiers in combination with other linguistic phenomena. see see figure 2 for how questions are formed from scene graphs. in brief, we think of the scene graph as a model and evaluate various combinations of logical operators, including quantifiers, on the scene graph, i.e., performing a model checking @xcite procedure. we introduce the notion of a question family, defined by a set of operators and a scene graph. each question family is associated with 2-6 text templates and a set of synonyms (for shapes, colors, materials, and spatial relationships). the templates were written by hand. each question template can thus generate multiple questions. for example, the template (5) are there @xmath0 we construct a total of 671 different templates, which are randomly constructed from 11 plane templates and 61 object templates. our questions involve attribute recognition, counting, comparing numbers or attributes, spatial relationships, and understanding of quantifiers. figure 2 shows the operators built in the given question family, such as filter, relate, and at least. note that many (generalized) quantifiers are related by entailment. the question (7) are all the red cubes on the marble planes? is, assuming an image with red cubes, semantically equivalent to (8) are no red cubes not on the marble planes? the semantics of combinations of quantifiers can be derived using squares of opposition @xcite . we exploit these entailment relations in creating qlevr. some combinations of key values may generate unreasonable questions. we therefore define restrictions for each question family to avoid the generation of pragmatically odd, ill-posed or trivial questions. for example, the phrase on the marble plane where there are at least 5 red objects would be pragmatically odd if there was only one marble plane in the scene. the sentence (9) on the marble plane, do between 2 and 4 cubes have the same size as most of the cylinders? we present many examples of images and questions in the appendix, but see also figure 3 for a complex question with embedded quantifiers. dataset characteristics qlevr has 1,000,000 questions for 100,000 images, with each image having 10 questions generated from different question families. the dataset is balanced, preventing answering in the absence of the images. in addition, the answer distribution across question families is constrained by acceptance-rejection sampling. the data is randomly split, with 70% for training data, 15% for validation and 15% for heldout evaluation data (the test set). as shown in figure 4 , qlevr includes 27 different quantifiers. questions contain 1-4 quantifiers. ## experiments in this section, we evaluate the performance of baselines and near-state-of-the-art models on the qlevr dataset and perform detailed error analysis. we ran each each method three times with different random seeds and report the test set performance for the model that achieved the best performance on the validation data. ## discussion in this paper, we proposed a dataset, which we call qlevr -for quantificational language and elementary visual reasoning. qlevr probes the ability of visual question-answering systems to reason with quantificational language, including 27 different quantifiers and combinations thereof. it requires complex visual reasoning to locate the specific planes and understand various relationships between objects. we increase the semantic diversity of the questions by negating quantifiers and by using different templates for semantically equivalent questions. our analysis highlights how challenging such examples are to visual question-answering systems, and we hope that qlevr will help guide push research horizons in visual question-answering by zooming in on the challenges posed by quantificational language. one fundamental limitation is that qlevr only considers english questions, and we plan to extent it to other, typologically unrelated languages. besides, qlevr can easily be extended by adding new question families, and questions whose answers are not limited to true or false, e.g., with numbers or attributes as answer types. in addition to the three-dimensional images, we also provide two-dimensional images and scene graphs recording the ground-truth information (see figure 2 ). it is also possible to generate questions about 2d im-ages by simply modifying our question families. we hope these two datasets can be used for transfer learning for visual question answering in the future. other models on most question types. the only exception is: on quantity comparison task, bert performs slightly better than mac, showing that mac has better reasoning ability in complex scenes. questions of existence type 1 obtain better results than existence type 2 for vision-language model cnn+lstm and mac, suggesting that the position relationship between object and plane is easier to be inferred by the models than the spatial relationship between the objects. for questions of quantity comparison, mac and cnn+lstm performs on par with lstm, suggesting that the image features extracted by resnet-101 may contain little information related to counting in complex scenes.