M a XM : Towards Multilingual Visual Question Answering (2023)

Abstract: Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.

## introduction visual question answering (vqa), the task of answering visual questions grounded in images, is key to human-machine interaction in the visual world. in particular, the natural language interface in vqa makes it easy for lay people to express their needs and benefit from its applications, including accessibility, education, and search. yet, vqa advances were mostly focused on english, therefore only applied to a privileged subset of human populations. arguably, the english language has dominated the field mainly because of the availability of english vqa benchmarks. these benchmarks are diverse, from general vqa @xcite @xcite @xcite , robust vqa @xcite , compositional visual reasoning @xcite , for the blind and the visually-impaired @xcite , scene-text understanding @xcite , to vqa that requires external, commonsense, or world knowledge @xcite @xcite . these benchmarks require considerable amount of resources to create, mostly by employing human annotators to laboriously collect and verify the questions and the answers for each image. to extend vqa to all languages in the world, we must make data creation more automatic. building on recent work on automatic data creation for english vqa from captions @xcite , in this paper we propose a translation-based framework for multilingual visual question answering (mvqa) data creation. our framework automates much of the task of generating questions and answers, thus providing a scalable path to mvqa. we apply our framework to the generation of question-answer pairs from the multilingual captions of the recently-proposed crossmodal-3600 dataset (xm3600) @xcite . combined with an efficient human annotation protocol, we construct maverics-xm3600 (maxm), a test benchmark for mvqa in 7 languages (see examples in fig. 1 ). finally, we use this novel benchmark to drive progress in mvqa modeling and measure where we stand. we leverage advances in image modeling and multilingual modeling: vit @xcite and mt5 @xcite and propose a unified, extensible, open-ended mvqa model, called simple mpt, which is competitive to stateof-the-art english vqa models that we adapt to apply in the mvqa setting (ofa @xcite and blip2 @xcite ). overall, there exists a large room for improvement. in summary, our main contributions are (i) a scalable translation-based framework for mvqa data generation based on captions (sect. 3); (ii) an efficient annotation protocol, deriving a novel test benchmark called maverics-xm3600 (maxm) in 7 diverse languages: english, french, hindi, hebrew, romanian, thai and chinese (sect. 4);michal: beer, why is this dataset better than xgqa? (iii) simple and lightweight mvqa modeling (sect. 5.2, sect. c) with strong performance; (iv) benchmarking (adaptations of) the state-of-the-art vqa models on maxm (sect. 5.3). ## multilingual vqa data creation like in many other machine learning tasks, the main bottleneck to mvqa is obtaining high-quality labeled data. the most popular data collection framework to english vqa is to ask a set of human annotators to come up with visual questions, and another set of annotator to answer them (sect. 2.2). to scale vqa to all languages, we argue that mvqa data creation must significantly reduce its use of human annotation. to this end, we study the extension of an automatic english vqa data creation method called visual question generation with question answering validation, or vq 2 a @xcite for the purpose of mvqa data creation. ## maxm: multilingual vqa benchmark in this section, we leverage the approach we presented in sect. 3 for creating a multilingual vqa test-only benchmark. we next describe our data sources, how candidate data was generated, human annotation protocol, and an analysis and a discussion of our benchmark. following the naming convention in @xcite , we call our benchmark maverics-xm3600, or maxm in short. we will release maxm to foster research on mvqa. ## conclusions we take initial steps toward multilingual vqa by proposing scalable solutions on both data creation and modeling fronts. we create a multilingual vqa benchmark in 7 diverse languages to drive modeling progress on multilingual vqa. we establish strong unified and open-ended vqa models that work well on 13 languages as well as benchmark state-of-the-art models. for future work, we would like to expand native-language question generation that is done in a limited scope and have single one for all target answers.