Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (2021)

Abstract: Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets (Gardneret al., 2020) quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically generate contrast sets for the visual question answering task. Our method computes the answer of perturbed questions, thus vastly reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects (e.g., spatial or relational reasoning). We demonstrate the effectiveness of our approach on the GQA dataset and its semantic scene graph image representation. We find that, despite GQA’s compositionality and carefully balanced label distribution, two high-performing models drop 13-17% in accuracy compared to the original test set. Finally, we show that our automatic perturbation can be applied to the training set to mitigate the degradation in performance, opening the door to more robust models.

## introduction nlp benchmarks typically evaluate in-distribution generalization, where test sets are drawn i.i.d from a distribution similar to the training set. recent works showed that high performance on test sets sampled in this manner is often achieved by exploiting systematic gaps, annotation artifacts, lexical cues and other heuristics, rather than learning meaningful task-related signal. as a result, figure 1 : illustration of our approach based on an example from the gqa dataset. top: qa pairs and an image annotated with bounding boxes from the scene graph. bottom: relations among the objects in the scene graph. first line at the top is the original qa pair, while the following 3 lines show our pertubated questions: replacing a single element in the question (a fence) with other options (a wall, men, an elephant), leading to a change in the output label. for each qa pair, the lxmert predicted output is shown. the out-of-domain performance of these models is often severely deteriorated @xcite @xcite @xcite @xcite . recently, @xcite and @xcite introduced the contrast sets approach to probe out-of-domain generalization. contrast sets are constructed via minimal modifications to test inputs, such that their label is modified. for example, in fig. 1 , replacing "a fence" with "a wall", changes the answer from "yes" to "no". since such perturbations introduce minimal additional semantic complexity, robust models are expected to perform similarly on the test and contrast sets. however, a range of nlp models severely degrade in performance on contrast sets, hinting that they do not generalize well @xcite . except two recent exceptions for textual datasets @xcite , contrast sets have so far been built manually, requiring extensive human effort and expertise. in this work, we propose a method for automatic generation of large contrast sets for visual question answering (vqa). we experiment with the gqa dataset @xcite . gqa includes semantic scene graphs @xcite representing the spatial relations between objects in the image, as exemplified in fig. 1 . the scene graphs, along with functional programs that represent the questions, are used to balance the dataset, thus aiming to mitigate spurious dataset correlations. we leverage the gqa scene graphs to create contrast sets, by automatically computing the answers to question perturbations, e.g., verifying that there is no wall near the puddle in fig. 1 . we create automatic contrast sets for 29k samples or ≈22% of the validation set. we manually verify the correctness of 1,106 of these samples on mechanical turk. following, we evaluate two leading models, lxmert @xcite and mac @xcite on our contrast sets, and find a 13-17% reduction in performance compared to the original validation set. finally, we show that our automatic method for contrast set construction can be used to improve performance by employing it during training. we augment the gqa training set with automatically constructed training contrast sets (adding 80k samples to the existing 943k in gqa), and observe that when trained with it, both lxmert and mac improve by about 14% on the contrast sets, while maintaining their original validation performance. our key contributions are: (1) we present an automatic method for creating contrast sets for vqa datasets with structured input representations; (2) we automatically create contrast sets for gqa, and find that for two strong models, performance on the contrast sets is lower than on the original validation set; and (3) we apply our method to augment the training data, improving both models' performance on the contrast sets. ## automatic contrast set construction to construct automatic contrast sets for gqa we first identify a large subset of questions requiring specific reasoning skills ( §2.1). using the scene graph representation, we perturb each question in a manner which changes its gold answer ( §2.2). finally, we validate the automatic process via crowdsourcing ( §2.3). ## experiments we experiment with two top-performing gqa models, mac @xcite and lxmert @xcite ), 3 to test their generalization on our automatic contrast sets, leading to various key observations. models struggle with our contrast set. training on perturbed set leads to more robust models. previous works tried to mitigate spurious datasets biases by explicitly balancing labels during dataset construction @xcite @xcite or using adversarial filtering @xcite . in this work we take an inoculation approach @xcite and augment the original gqa training set with contrast training data, resulting in a total of 1,023,607 training samples. we retrain both models on the augmented training data, and observe in ## discussion and conclusion our results suggest that both mac and lxmert under-perform when tested out of distribution. a remaining question is whether this is due to model architecture or dataset design. @xcite claim that both of these models are prone to fail on compositional generalization because they do not decompose the problem into smaller sub-tasks. our results support this claim. on the other hand, it is possible that a different dataset could prevent these models from finding shortcuts. is there a dataset that can prevent all shortcuts? our automatic method for creating contrast sets allows us to ask those questions, while we believe that future work in better training mechanisms, as suggested in @xcite and @xcite , could help in making more robust models. we proposed an automatic method for creating contrast sets for vqa datasets that use annotated scene graphs. we created contrast sets for the gqa dataset, which is designed to be compositional, balanced, and robust against statistical biases. we observed a large performance drop between the original and augmented sets. as our contrast sets can be generated cheaply, we further augmented the gqa training data with additional perturbed questions, and showed that this improves models' performance on the contrast set. our proposed method can be extended to other vqa datasets.