Attribute Diversity Determines the Systematicity Gap in VQA (2024)

Abstract: Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this work, we study the systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes. To test, we introduce a novel diagnostic dataset, CLEVR-HOPE. We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data. In particular, our experiments suggest that the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be.

## introduction systematicity, the ability to handle novel combinations of known concepts, is a type of compositional generalization @xcite . while systematicity is crucial to human intelligence @xcite , conventionally trained neural networks often struggle to generalize systematically @xcite . inspired by prior work investigating compositionality failures in language models @xcite , we study the systematicity gap in visual question answering (vqa): the drop in model performance when reasoning about a combination of properties that was held out from both the text and vision modalities at train time. as an example, let us consider material and shape as two attribute types. if a model was trained without exposure to a particular combination of attribute values, e.g., rubber sphere, then we say the model composes * project lead † joint senior authors systematically if it has high performance at test time on data that includes a rubber sphere. our work empirically demonstrates that systematicity emerges in a neural vqa model if the model is trained with diverse contexts for the attribute values in question (i.e., exposed to many mate-rial-shape combinations). the intuition for this hypothesis is simple: given many training examples of distinct combinations, the model learns how material and shape interact, and thus systematically generalizes to an unseen combination of mate-rial and shape. in contrast, a model trained on low-diversity data (i.e., only exposed to a few ma-terial-shape combinations) fails to learn rules to recombine them. using clevr-hope, a novel dataset for evaluating systematicity on a variety of held-out object attribute value pairs in a controlled setting, we measure the systematic compositionality of multimodal transformer and neurosymbolic models. we find that, while systematicity does not improve with more training data, it does improve with more diverse training data. specifically, attribute types that include more diverse combinations during training can be composed systematically. ## clevr-hope diagnostic dataset our dataset is based on clevr @xcite , a synthetic experimental setting for testing basic visual reasoning skills. clevr comprises english questions (such as "what is the color of the cube on the right side of the yellow sphere?") and corresponding 3d-rendered images of colored blocks. each block has four attribute types (size, color, material, and shape). our experiments rely on data splits that create distinct in-distribution (id) and out-of-distribution (ood) test sets. we present the clevr held-out pair evaluation (clevr-hope) dataset for testing the systematicity of vqa models. clevr-hope is a train what is the shape of the blue metallic object that is the same size as the gray block? controlled setting to test whether vqa models generalize to pairs of attribute values that were not seen during either training or fine-tuning. within clevr-hope, we refer to an unseen pair of attribute values as a held-out pair (hop). the dataset is composed of 29 sub-datasets, each for a different hop (see tab. 1 for the list of hops) . the 29 hops are selected such that there are 5 hops from each of the 6 possible pairs of attribute types, with the exception of size+material as clevr contains only 4 such combinations. each hop has its own train set and 4 test sets. for rubber cylinder, in fig. 1 , these datasets are: train: 560k image-question pairs in the training/finetuning set. the data distribution is similar to clevr, but any images or questions involving rubber cylinder have been removed. complex-iid test: test data sampled from the train distribution (i.e., rubber cylinder is filtered out). this is a standard iid test set; it's primarily used as a point of comparison for the complex-ood test split. complex-ood test: test data sampled from the clevr distribution filtered to always have (i) at least one object matching rubber cylinder, and (ii) rubber cylinder in the question. it is ood as rubber cylinder is a combination never seen visually or textually in the train data. this split requires the model to behave systematically (i.e., generalize to rubber cylinder) while reasoning over scenes with several objects. minimal-iid test: minimal image-question pairs that check whether a model can recognize pairs of attribute values, corresponding to rubber cylinder's attribute types, that were seen in the train set. the scene contains a single object, and the question only asks if there is an object matching a given pair of attributes. this split is called iid as it only contains pairs of attribute seen at train time. this split provides a point of comparison for the minimal-ood test split. minimal-ood test: minimal image-question pairs that check recognition of rubber cylinder. this split tests a model's systematicity independent of reasoning or visual clutter. if a model had some systematic behaviour which degraded when reasoning was required, then we'd expect to see a small performance gap between minimal-ood test and minimal-iid test, and a large gap between complex-ood test and complex-iid test. by construction, always returning false would yield 75% accuracy (see appx. b.1 for details). summarizing the split naming convention: "complex" splits contain clevr-like visual and textual complexity. "minimal" splits contain single-object images, and only recognition questions (e.g., "are any rubber cylinders visible?"). iid splits contain only the attribute pairs seen at train time. ood splits contain the hop in both the question, and in at least one object in the scene. appendix b includes dataset details. note, clevr-hope omits validation sets to prevent tuning for specific task @xcite ; instead, hyperparameters should be chosen using clevr. training: for each hop, we subsample the training set to test the impact the amount of training data has on performance. for 3 random seeds per hop, we finetune pretrained lxmert (lxmertp) and train lxmert from scratch (lxmert-s). we also train tensor-nmn from scratch, again for three runs, though only for the first 6 hops, combinations of {large, cyan, rubber, cylinder}. for hyperparameter selection, we perform a grid search on the original clevr dataset @xcite . for further details, see appendix c. ## related work while compositionality in vqa has been studied, prior work has focused on generalization to new question structures @xcite @xcite , task-object combinations @xcite , or questionanswer combinations @xcite , rather than new attribute combinations. systematicity has often been investigated through synthetic datasets to control for the model's exposure to particular attribute combinations. lake and baroni (2018) introduced the scan benchmark to evaluate compositionality in sequence-to-sequence models, revealing a lack of systematicity. followup @xcite and concurrent @xcite seq2seq works have shown that the conceptual diversity of the training set significantly affects systematicity-our work extends these findings to the multi-modal domain of vqa. the closest prior work is the clevr-cogent dataset: johnson et al. (2017a) created a train-test clevr split where at train time cubes and cylin-ders are restricted to limited color palettes, that are reversed at test time. they observed that model performance declined on held-out attribute combinations. but, unlike clevr-hope, clevr-cogent does not change the question distribution at train time-held-out combinations can leak by appearing in text at train time. furthermore, clevr-cogent has only a single train set with held-out color-shape combinations-whereas clevr-hope expands the set of held-out combinations to 29 train sets, covering all possible pairs of attribute types. clevr-hope also independently assesses each hop, including in a minimal setting. in combination, these improvements allow us to study the impact of train-time diversity. our results align with concurrent work on the effects of training diversity in vqa: rahimi et al. (2023) modify clevr to study the related question of productivity, concluding that increasing the diversity of question combinations increases productivity. unlike our work, they do not use a transformer architecture, instead studying mac (hudson and manning, 2018), film @xcite , and vector-nmn @xcite . additionally, as they study a fundamentally different question, their dataset only alters the question distribution-their image distribution is unchanged between train and test time. given that systematicity and productivity are both aspects of compositional generalization @xcite , the growing evidence across task settings and facets of compositionality @xcite suggests a close relationship between traintime diversity and compositional generalization as a broad phenomenon. we encourage further study, and release our code and data to facilitate these efforts. confirming our findings in newer vision-language models, quantifying them with scaling laws, exploitation in data curation pipelines, and using these effects to predict model reliability, are all important future work. ## conclusions using clevr-hope, we demonstrate that several models exhibit a degree of systematic generalization to held-out object attribute pairs. furthermore, we illustrate that the systematicity gap (the difference between in-and out-of-distribution performance) does not improve with more data, but does with more attribute diverse data-i.e., the number of attribute pairs of the same type seen at train time.