COVR : A Test-Bed for Visually Grounded Compositional Generalization with Real Images (2021)

Abstract: While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to synthetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with real images. To create COVR, we use real images annotated with scene graphs, and propose an almost fully automatic procedure for generating question-answer pairs along with a set of context images. COVR focuses on questions that require complex reasoning, including higher-order operations such as quantification and aggregation. Due to the automatic generation process, COVR facilitates the creation of compositional splits, where models at test time need to generalize to new concepts and compositions in a zero- or few-shot setting. We construct compositional splits using COVR and demonstrate a myriad of cases where state-of-the-art pre-trained language-and-vision models struggle to compositionally generalize.

## introduction models for natural language understanding (nlu) have exhibited remarkable generalization abilities on many tasks, when the training and test data are sampled from the same distribution. but such models still lag far behind humans when asked to generalize to an unseen combination of known concepts, and struggle to learn concepts for which only few examples are provided @xcite . humans, conversely, do this effortlessly: for example, once humans learn the meaning of the quantifier "all", they can easily understand the utterance "all cheetahs have spots" if they know what "cheetahs" and "spots" mean @xcite @xcite . this ability, termed compositional generalization, is crucial for building models that generalize to new settings @xcite . in recent years, multiple benchmarks have been created, illustrating that current nlu models fail to generalize to new compositions. however, these benchmarks focused on semantic parsing, the task of mapping natural language utterances to logical forms (lake and baroni, 2018; @xcite . visual question answering (vqa) is arguably a harder task from the perspective of compositional generalization, since the model needs to learn to compositionally "execute" the meaning of the question over images, without being exposed to an explicit meaning representation. for instance, in fig. 1 , a model should learn the meaning of the quantifier "all' from the first example, and the meaning of a conjunction of clauses from the second example, and then execute both operations compositionally at test time. existing vqa datasets for testing compositional generalization typically use synthetic images and contain a limited number of visual concepts and reasoning operators @xcite ; ruis et al., 2020), or focus on generalization to unseen lexical constructions rather than unseen reasoning skills @xcite . other datasets such as gqa (hudson and manning, 2018) use real images with synthetic questions, but lack logical operators available in natural language datasets, such as quantifiers and aggregations, and contain "reasoning shortcuts", due to a lack of challenging image distractors @xcite . in this work, we present covr (compositional visual reasoning), a test-bed for visuallygrounded compositional generalization with real images. we propose a process for automatically generating complex questions over sets of images (fig. 2 ), where each example is annotated with the program corresponding to its meaning. we take images annotated with scene graphs (fig. 2a ) from gqa, visual genome @xcite , and imsitu @xcite , automatically collect both similar and distracting images for each example, and filter incorrect examples due to errors in the source scene graphs (fig. 2b , c ). we then use a template-based grammar to generate a rich set of complex questions that contain multi-step reasoning and higher-order operations on multiple images (2d). to further enhance the quality of the dataset, we manually validate the correctness of development and test examples through crowdsourcing and paraphrase the automatically generated questions into fluent english for a subset of the automatically generated dataset (2e). covr contains 262k examples based on ∼89k images, with 13.9k of the questions manually validated and paraphrased. our automatic generation process allows for the easy construction of compositional data splits, where models must generalize to new compositions, and is easily extendable with new templates and splits. we explore both the zero-shot setting, where models must generalize to new compositions, and the few-shot setting, where models need to learn new constructs from a small number of examples. we evaluate state-of-the-art pre-trained models on a wide range of compositional splits, and expose generalization weaknesses in 10 out of the 21 setups, where the generalization score we define is low (0%-70%). moreover, results show it is not trivial to characterize the conditions under which generalization occurs, and we conjecture generalization is harder when it requires that the model learn to combine complex/large structures. we encourage the community to use covr to further explore compositional splits and investigate visually-grounded compositional generalization. foot_0 ## related work prior work on grounded compositional generalization has typically tested generalization in terms of lexical and syntactic constructions, while we focus on compositions in the space of meaning by testing unseen program compositions. for example, the "structure split" in @xcite splits examples based on their surface form. as a result, identical programs are found in both the training and the test splits. the "content split" from the same work tests generalization to unseen lexical concepts. this is a different kind of skill than the one we address, where we assume the model sees all required concepts during training. c-vqa @xcite splits are based only on question-answer pairs and not on the question meaning. clevr-cogent @xcite tests for unseen combinations of attributes, thus focuses more on visual generalization. other datasets that do split samples based on their programs @xcite are using synthetic images with a small set of entities and relations (≤ 20). gqa @xcite uses real images with synthetic questions, which in theory could be used to create compositional splits. however, our work uses multiple images which allows testing reasoning operations over sets and not only over entities, such as counting, quantification, comparisons, etc. thus, the space of possible compositional splits in covr is much larger than gqa. if we anonymize lexical items in gqa programs (i.e., replace them with a single placeholder), the number of distinct programs (79) is too low to create a rich set of splits where all operators appear in the training set. in contrast, covr contains 640 anonymized programs, allowing us to create a large number of splits. moreover, our process for finding high-quality distracting images mitigates issues with reasoning shortcuts, where models solve questions due to a lack of challenging image distractors @xcite . finally, other vqa datasets with real images and questions that were generated by humans @xcite do not include a meaning representation for questions and thus cannot be easily used to create compositional splits. ## dataset creation the goal of covr is to facilitate the creation of vqa compositional splits, with questions that require a high degree of compositionality on both the textual and visual input. task definition examples in covr are (q, i, a) triples, where q is a complex question, i is a set of images, and a the expected answer. unlike most visually-grounded datasets , which contain 1-2 images, each example in covr contains up to 5 images. this allows us to (a) generate questions with higher-order operators, and (b) detect good distracting images. also, questions are annotated with programs corresponding to their meaning, which enables creating compositional splits. high-level overview fig. 2 provides an overview of the data generation process. given an image and its annotated scene graph describing the image objects and their relations, we iterate through a set of subgraphs. for example, in fig. 2a the subgraph corresponds to "a man is catching a frisbee and wearing jeans". next, we sample images with related subgraphs: (a) images that contain the same subgraph (fig. 2b ), and (b) images that contain a similar subgraph, to act as distracting images (e.g., images with a man catching a ball or a woman catching a frisbee, fig. 2c ). to ensure the quality of the distracting images, we propose models for automatic filtering and validation ( §3.2). next (fig. 2d ), we instantiate questions from a templatebased grammar by filling slots with values computed from the selected subgraphs, and automatically obtain a program for each question. last, we balance the answers and question types, and use crowdsourcing to manually validate and provide fluent paraphrases for the evaluation set and a subset of the training set (fig. 2e ). ## compositional splits our generation process lets us create questions and corresponding programs with a variety of reasoning operators. we now show how covr can be used to generate challenging compositional splits. we propose two setups: (1) zero-shot, where training questions provide examples for all required reasoning steps, but the test questions require a new composition of these reasoning steps, and (2) fewshot, where the model only sees a small number of examples for a specific reasoning type @xcite . because each question is annotated with its program, we can define binary properties over the program and answer, where a property is a binary predicate that typically defines a reasoning type in the program. for example, the property has-quant is true iff the program contains a quantifier, and has-quant-none iff it contains the quantifier none. we can create any compositional split that is a function of such properties. we list the types of properties used in this work in all compositional splits are based on the original training/validation splits from visual genome and imsitu to guarantee that no image appears in both the training set and any of the test sets. splits are created simply by removing certain questions from the train and test splits. if we do not remove any question, we get an i.i.d split. ## experiments experimental setup we consider the following baselines: (a) maj, the majority answer in the training set, and (b) majtempl: an oracle-based baseline that assumes perfect knowledge of the template from which the question was generated, and predicts the majority answer for that template. for templates that include two possible answers ("candle or lighter"), it randomly picks one. we use the volta framework @xcite to train and evaluate different pre-trained vision-and-language models. we use volta's controlled setup models (that have a similar number of parameters and pre-training data) of @xcite and vilbert @xcite . in this section we show results only for visualbert, and results for vilbert can be found in app. h, showing mostly similar scores. a vision-and-language model provides a representation for question-image pairs, (q, i). we modify the implementation to accept n images by running the model with (q, i) as input for each image @xmath0, and then passing the n computed representations through two transformer layers with a concatenated [cls] token. we pass the [cls] token representation through a classifier layer to predict the answer. the classifier layer and the added transformer layers are randomly initialized, and all parameters are fine-tuned during training. to estimate a lower bound on performance without any reasoning on images, we evaluate a textonly baseline vb text that only sees the input text (image representations are zeroed out). for the compositional splits, we evaluate vi-sualbert trained on the entire data (vb iid ), the text baseline (vb text ), and the compositionallytrained models vb 250 , vb 0 for the few-shot (@xmath1, respectively. to control the training size, we also evaluate vb iid-size , a model trained with a similar data size as the compositionally-trained model, by uniformly downsampling the training set. all models are evaluated on the same subset of the development compositional split. to focus on the generalization gap, we define a generalization score ("gen. score") that measures the proportion of the gap between vb text and our upper-bound, vb iid-size , that is closed by a model. in all compositional splits, we train the models 8 epochs and early-stop using the subset of the development set that does not contain any of the compositional properties we test on @xcite . results first, we show how models perform on paraphrased and automatically-generated questions in the i.i.d setup in next, we report results on the compositional splits. we show results on automatically-generated questions (not paraphrased), to disentangle the effect of compositional generalization from transfer to natural language. app. h reports results for the paraphrased test set, where generalization scores are lower, showing that transfer to natural language makes compositional generalization even harder. the large gap between the nearly-perfect score of has-num-3 (92%), and the low score of has-num-3-ans-3 (0%), where in both the number 3 is rarely seen in the question, and in the latter it is also rare as an answer, suggests that the model learns good number representations just from seeing numbers in the answers. other cases where the generalization scores are low are has-quant, conversely, the model generalizes well to questions with the count operator where the subgraph contains a complex sub-graph (has-count & rm/v/c) or an attribute (has-count & has-attr), and in the lexical split, where the model is tested on unseen combinations of names of nodes. a possible explanation for the above is that compositional generalization is harder when the model needs to learn to combine large/complex structures, and performs better when composing more atomic constructs. however, further characterizing the conditions under which compositional generalization occurs is an important question for future work. ## conclusion we present covr, a test-bed for visuallygrounded compositional generalization with real images. covr is created automatically except for manual validation and paraphrasing, and allows us to create a suite of compositional splits. covr can be easily extended with new templates and splits to encourage the community to further understand compositional generalization. through covr, we expose a wide range of cases where models struggle to compositionally generalize.