C ross VQA : Scalably Generating Benchmarks for Systematically Testing VQA Generalization (2021)

Abstract: One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role. In this paper, we propose a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer generation (VQAG) module that is capable of generating highly-relevant and diverse question-answer pairs with the desired dataset style. We use it to create CrossVQA, a collection of test splits for assessing VQA generalization based on the VQA2, VizWiz, and Open Images datasets. We provide an analysis of our generated datasets and demonstrate its utility by using them to evaluate several state-of-the-art VQA systems. One important finding is that the visual shifts in cross-dataset VQA matter more than the language shifts. More broadly, we present a scalable framework for systematically evaluating the machine with little human intervention.

## introduction multiple datasets have been proposed to measure the progress on visual question answering (vqa) @xcite @xcite @xcite @xcite @xcite . however, these datasets often possess biases introduced in the data collection process and by the human annotators. it has been shown that existing vqa models leverage these spurious biases and take shortcuts @xcite @xcite . as a result, the performance of those models on a specific vqa dataset can only serve as a rough proxy for the true learning of the vqa task @xcite one common remedy to this is to go beyond indomain evaluation, in which the test set exhibits some form of "distribution shifts" from the training set @xcite . the key idea is that a generalizable vqa model should be able to extrapolate, for example, from one dataset to another. one challenge that is quite unique to vqa in this setting is that the distribution shift is multi-modal. when one dataset unsatisfactorily transfers to another, it is difficult to identify how much of this is due to vision or language distribution mismatches. to complicate things even more, the frequency of objects occurring in natural images follows a long-tail distribution @xcite @xcite . lack of sufficient instances of minority classes in the test sets further complicates the estimation of generalization capabilities from one dataset to another. a possible solution to address this issue is to use an iterative, human-in-the-loop approach for dataset collection where human annotators carefully devise new test samples by incorporating visual and language distribution shifts @xcite @xcite . however, this approach is not scalable and training the human annotators, be they seasoned ai experts or non-experts, would incur huge annotation time and cost. in this work, we propose to make the process of creating distribution shifts more systematic and automatic. inspired by recent work on dynamic benchmarks that co-evolve with strong models @xcite , we propose to bring in visual question-answer generation (vqag) module in the evaluation process. more specifically, we first build a strong, controllable vqag engine that is capable of creating particular dataset-style question-answer pairs. then, we use it to generate novel image, question, answer test splits, while controlling distribution shifts in vision and language features. this is summarized in there are at least two advantages in using a vqag model to construct our crossvqa test sets: (1) we can evaluate the adaptation skills of vqa models on non-vqa datasets such as open images (oid) @xcite , which contains various image annotations but no question/answer pairs, i.e. i oid , q vqa2 and i oid , q vzwz (see we conduct extensive experiments to evaluate the utility of our proposed framework. first, we validate that our vqag module is capable of generating relevant questions and correct answers with the desired distribution shifts, which we achieve through a combination of transformer-based architectures, vision-and-language pre-training, and multiple types of control signals. we also find that, when evaluated against state-of-the-art generative models for visual question generation, our vqag substantially outperforms them in terms of accuracy, diversity, and novelty. additionally, we perform analysis and human evaluation of our crossvqa test sets that are built on vqa2, vizwiz, and open images datasets. we show that they are effective at finding and quantifying weaknesses of cross-dataset generalization abilities in the state-of-the-art vqa models. for instance, our experimental results show that vqa models drop up to 40% in absolute accuracy if there is a mismatch in image distribution. on the other hand, vqa models are found to be relatively less sensitive to a mismatch in language distribution. finally, inspired by the success of contrastive learning and multi-task learning techniques in improving generalization and robustness of multimodal tasks @xcite , we investigate whether these techniques improve the performance of vqa models on our crossvqa test sets. interestingly, we find that contrastive losses and multitask regularization do not lead to significant generalization gains on crossvqa. in summary, our key contributions are three-fold. first, we introduce the crossvqa benchmark for systematically assessing the generalization skills of vqa models, and provide analysis and experiments to support its utility. second, we describe a scalable data collection and benchmarking framework for semi-automatically constructing the proposed benchmarks using a strong and controllable visual question-answer generation (vqag) module. finally, we empirically demonstrate the superiority of our vqag module by achieving new state-of-the-art results in visual question generation. ## experiments in this section, we first evaluate the performance of our vqag model against existing state-of-theart baselines @xcite @xcite . we then demonstrate the importance of conditioning our vqag model on the proposed control signals by performing several ablation studies. next, we present crossvqa examples and several statistics based on the generated data. we finally show that crossvqa is effective at identifying the limitations of state-ofthe-art vqa models, and examine the extent to which existing adaptation techniques help in improving performance of vqa models as measured by crossvqa. ## conclusion we present a step toward scalable and systematic evaluation of vqa systems. key to our approach is an accurate and controllable vqag module that is capable of generating disentangled distribution shifts. we generate crossvqa benchmarks, a collection of test splits based on vqa2, vizwiz, and open images datasets. we validate their utility by showing that existing vqa models struggle to perform well in this evaluation scenario and identifying the image distribution mismatch as the main factor. examples of invented questions examples of invented answers q1: what hand is the man using to write with? q2: are most of the lights on or off in the living room? q3: will this woman be drinking beer? q4: what is the number on the front side of the bike? q5: in this scene how many sheep can be clearly seen? q6: what is the purpose of the number on the yellow board? q7: which sheep is the older in the picture? q8: is the fire hydrant old or new? q9: what is the first letter of the word on the blue sign? q10: what is the name of the logo on top of the keyboard? {at least 10 years, above doorway, inside the baggage, behind red car, towards bottom left side, dirt bikes, fishing boats, fork and sharp knife, riding big elephants, right side of road}