All You May Need for VQA are Image Captions (2022)

Abstract: Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation. In this paper, we propose a method that automatically derives VQA examples at volume, by leveraging the abundance of existing image-caption annotations combined with neural models for textual question generation. We show that the resulting data is of high-quality. VQA models trained on our data improve state-of-the-art zero-shot accuracy by double digits and achieve a level of robustness that lacks in the same model trained on human-annotated VQA data.

## introduction visual question answering (vqa) is a complex multimodal task that, to be successfully modeled and evaluated, requires large amounts of annotations that are not naturally produced by existing business processes, the way translation-pair annotations @xcite or image alt-text annotations @xcite are produced. at present, a main bottleneck for developing robust vqa systems that are useful for downstream applications, such as for visually-impaired people and in the medical and education domains, appears to be a lack of large image-question-answer training triplets (on the order of millions). manual annotation of such triplets is costly, time-consuming, and prone to a variety of human biases that are difficult to account for @xcite . in addition, the brittleness of vqa systems trained on such manual annotations is well-understood and documented @xcite . to address the data limitation, we turn to a potential source for creating vqa examples: image-english caption pairs @xcite . large-scale image caption datasets vq 2 a figure 1 : given an english caption (along with its corresponding image), our vq 2 a method generates high-quality question-answer pairs. these image-question-answer triplet data can be automatically produced at volume (millions of examples) and used to effectively train vqa systems. in this paper, we explore the automatic creation of millions of vqa training data using neural models for textual question generation and question answering. we refer to this method as vq 2 a, for visual question generation with question answering validation. we demonstrate that vqa models trained on such data, with no exposure to human-annotated vqa data at all, exhibit high zero-shot performance. our best models obtain 61.1% accuracy on vqa2.0, 52.1% on gqa, around 15-17 points higher than previous zero-shot state-of-the-art results, and getting close to fullysupervised performance. in addition, taking our generated examples as a test set, we provide further evidence for the brittleness of vqa systems built with human-annotated examples, as well as evidence for the robustness of vqa systems built with the automatically-induced vq 2 a data. ## textual question generation for vqa we study whether automatically producing vqa annotations from existing image-text resources can alleviate or completely replace the need for manual data annotation. we only focus on english in this paper. to this end, we follow and improve upon some of the recent directions in section 2.2 on automatic question-answer generation from text. we start with a given dataset of image-caption pairs @xmath0, cap i } n for each pair {img i , cap i }, an initial set of candidate answers {a i,j } m i @xmath1. for each such candidate answer, a question is generated by a neural model q i,@xmath2,j , cap i ). each generated question-answer pair undergoes a validation step, and, if validated, is coupled with the corresponding image img i to induce a vqa example triplet {img i , q i,j , a i,j }. we refer to this method as vq 2 a (visual question generation with question answering validation). figure 2 provides an overview of our approach. we next detail the steps in vq 2 a. ## visual question answering (vqa) to assess the effectiveness of our automatic generation of vqa annotations, we perform extrinsic question answers how many pieces of fruit are in the bowl "0" is there a refrigerator in the kitchen "yes" what color are the cabinets in the kitchen "white" is the kitchen lit or dark "lit" is there a stove in the kitchen "no" what color is the formica in the kitchen "white" what is on the door of the refrigerator "papers", "several papers" where are the papers on the refrigerator "door" ## results we report several sets of experimental results that shed light both on the accuracy and on the robustness of vqa models trained on vq 2 a data in this section, with additional results, analysis and ablation studies in appendix c. ## conclusions in this paper, we show that high-quality vqa training data can be automatically induced at scale from existing image-caption datasets. our method, vq 2 a, annotates candidate answers using syntactic parsing of the captions and then derives questions for them using neural models for question generation and question answering verification. we demonstrate that vqa models trained only on such data exhibit high zero-shot performance with new state-of-the-art results on vqa2.0 and gqa. additionally, we provide evidence for the brittleness of vqa systems built with humanannotated examples compared to the ones built with automatically-induced image-question-answer triplets using vq 2 a. for future work, we plan to explore even larger automatically-curated image-text datasets, consisting of billions of examples. in addition, we want to test the applicability of vq 2 a to languages other than english, for which human-annotated vqa data is scarce.