Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering (2021)

Abstract: Visual question answering (VQA) is challenging not only because the model has to handle multi-modal information, but also because it is just so hard to collect sufficient training examples — there are too many questions one can ask about an image. As a result, a VQA model trained solely on human-annotated examples could easily over-fit specific question styles or image contents that are being asked, leaving the model largely ignorant about the sheer diversity of questions. Existing methods address this issue primarily by introducing an auxiliary task such as visual grounding, cycle consistency, or debiasing. In this paper, we take a drastically different approach. We found that many of the “unknowns” to the learned VQA model are indeed “known” in the dataset implicitly. For instance, questions asking about the same object in different images are likely paraphrases; the number of detected or annotated objects in an image already provides the answer to the “how many” question, even if the question has not been annotated for that image. Building upon these insights, we present a simple data augmentation pipeline SimpleAug to turn this “known” knowledge into training examples for VQA. We show that these augmented examples can notably improve the learned VQA models’ performance, not only on the VQA-CP dataset with language prior shifts but also on the VQA v2 dataset without such shifts. Our method further opens up the door to leverage weakly-labeled or unlabeled images in a principled way to enhance VQA models. Our code and data are publicly available at https://github.com/heendung/simpleAUG .

## introduction several key bottlenecks have been identified. in particular, a machine (i.e., vqa model) learned in the conventional supervised manner using humanannotated image-question-answer (iqa) triplets is shown to overlook the image or language contents @xcite @xcite , over-fit the language bias @xcite , or struggle in capturing the diversity of human language @xcite . many recent works thus propose to augment the original vqa task with auxiliary tasks or losses such as visual grounding @xcite , de-biasing @xcite @xcite , or (cycle-)consistency @xcite to address these issues. intrigued by these findings and solutions, we investigate the bottlenecks further and argue that they may result from a more fundamental issue -there are simply not enough training examples (i.e., iqa triplets). concretely, most of the existing vqa datasets annotate each image with around ten questions, which are much fewer than what we humans can ask about an image. take the popular vqa v2 dataset @xcite , for instance, the trained vqa model can answer most of the training examples (on average, six questions per image) correctly. however, if we ask some more questions about the training images -e.g., by borrowing relevant questions from other training images -the same vqa model fails drastically, even if the model has indeed seen these images and questions during training (see figure 1 ). namely, the vqa model just has not learned enough through the human-annotated examples, leaving the model unaware of the huge amount of visual information in an image and how it can be asked via natural language. at first glance, this seems to paint a grim picture for vqa. however, in this paper we propose to take advantage of this weakness to strengthen the vqa model: we turn implicit information already in the dataset, such as unique questions and the rich contents in the training images, into explicit iqa triplets which can be directly used by vqa models via conventional supervised learning. we propose a simple data augmentation method simpleaug, which relies on (i) the original imagequestion-answer triplets in the dataset, (ii) midlevel semantic annotations available on the training images (e.g., object bounding boxes), and (iii) pretrained object detectors @xcite foot_0 . concretely, we build upon the aforementioned observations -questions annotated for one image can be valuable add-ons to other relevant imagesand design a series of mechanisms to "propagate" questions from one image to the others. more specifically, we search images that contain objects mentioned in the question and identify the answers using information provided by (ii) and (iii), such as numbers of objects, their attributes, and existences. simpleaug requires no question generation step via templates or language models @xcite , bypassing the problems of limited diversity or artifacts. besides, simpleaug is completely detached from the training phase of a vqa model and is therefore model-agnostic, making it fairly simple to use to improve vqa models. we validate simpleaug on two datasets, vqa v2 @xcite and vqa-cp @xcite . the latter is designed to evaluate vqa models' generalizability under language bias shifts. with simpleaug, we can not only achieve comparable gains to other existing methods on vqa-cp, but also boost the accuracy on vqa v2, demonstrating the applicability of our method. we note that many of the prior works designed for vqa-cp indeed degrade the accuracy on vqa v2, which does not have language bias shifts between training and test data. simpleaug further justifies that mid-level vision tasks like object detection can effectively benefit high-level vision tasks like vqa. in summary, our contributions are three-folded: • we propose simpleaug, a simple and modelagnostic data augmentation method that turns information already in the datasets into explicit iqa triplets for training vqa models. ## conclusion we proposed simpleaug, a data augmentation method for vqa that can turn information already in the datasets into explicit iqa triplets for training. simpleaug is simple but by no means trivial. first, it justifies that mid-level vision tasks like object detection can effectively benefit vqa. second, we probably will never be comprehensive enough in annotating data, and simpleaug can effectively turn what we have at hand (i.e., "knowns") to examples a vqa model wouldn't have known (i.e., "unkowns"). simpleaug can notably improve the accuracy of vqa models on both vqa v2 @xcite and vqa-cp v2 @xcite . multi-stage training. as discussed in § 4.2 and § 4.4 of the main paper, we train the vqa models with a three-stage paradigm (@xmath0: first with original triplets o, then with the simpleaug triplets a, and then with o again. in each of these three stages, we follow the same optimization procedures as we train the baseline vqa models in the first stage. we report the best results on vqa v2 validation set @xcite and vqa-cp v2 test set @xcite .