x GQA : Cross-Lingual Visual Question Answering (2022)

Abstract: Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and—vice versa—multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling.

## xgqa the original english gqa dataset @xcite was constructed by leveraging visual genome scene graphs @xcite . an english question engine that utilizes content (i.e. information about objects, attributes, and relations provided) and structure (a linguistic grammar that couples hundreds of structural patterns and detailed lexical semantic resources) was used to generate over 22 million diverse questions, which are visually grounded in the image scene graphs. as the questions are automatically generated using templates, they do not necessarily reflect the wide spectrum of natural language, making any assumptions on the performance in the wild difficult. each question is associated with additional metadata such as structural types: (1) verify for yes/no questions (e.g. "do you see any cats?"), ( 2 jeans?"), (3) choose for questions that present two alternatives to choose from (e.g. "is it red or blue?"), ( 4 ) logical which involve logical inference (e.g. "is the field soft and snowy"), and ( 5 ) compare for comparison questions between two or more objects (e.g. "are all the animals zebras?"). for further details regarding the metadata, we refer the reader to @xcite . dataset design. the principal objective when devising xgqa was to create a genuinely typologically diverse multimodal and multilingual evaluation benchmark for visual question answering. we utilize the balanced 3 test-dev set of gqa, which consists of 12,578 questions about 398 images. 4 due to the defined structural patterns, the formulation of the questions is simple, with an average length of 8.5 words. 5 the resulting xgqa dataset covers translations in 7 languages, each representing a distinct language family, and contains examples written in 5 different scripts (see few-shot data splits. in order to conduct crosslingual few-shot learning experiments, we provide new data splits of different sizes. we split on images and add all questions associated with the image to the respective set. the development and test sets consist of 50 and 300 images, respectively. the training splits consist of 1, 5, 10, 20, 25, and 48 images, see ## baselines to analyze the performance and current gaps on xgqa, we first evaluate the recently proposed m 3 p model, which has been pretrained on multilingual and multimodal data. however, pretraining is computationally expensive and only limited amounts of multilingual multimodal resources are available. therefore, we further propose new and more efficient approaches that (1) extend state-of-the-art multilingual language models to the multimodal domain and (2) provide multilingual capabilities to state-of-the-art multimodal models. unless noted otherwise, we follow the predominant fine-tuning strategy for gqa; a prediction head is placed on top of the output of a pretrained transformer. all possible 1853 answers of the gqa task are mapped to a class label. the question associated with an image together with the positionaware region features are passed as input to the transformer, supervised using a cross-entropy loss. foot_3 4.1 @xmath0. to extend a monolingual transformer lm to a multilingual domain, @xcite fine-tune a new word-embedding layer in the target language. inspired by this idea, we now describe how we extend the current state-of-theart monolingual multimodal transformer model oscar+ @xcite to learn new embeddings for the target languages. in the language-extension phase, we replace the embedding matrix of oscar+ with a randomly initialized embedding matrix. 7 the transformer weights are frozen while only the newly introduced embeddings are fine-tuned on unlabeled text data of the target language with the mlm objective. in the target-task phase, the original oscar+ model is fine-tuned on the english training data of gqa, where the transformer layers are fine-tuned, but the embedding layer is frozen. during inference, the embedding layer is replaced with the target language's embedding layer. oscar+ ada . we extend this by adding adapters. in the language-extension phase we follow pfeiffer et al. (2021) in order to extend the model to the target languages. similar to oscar+ emb , we train a new embedding layer. we further add language adapters at every transformer layer. given that oscar+ is trained on english text, we follow @xcite when training english language adapter modules, without replacing the embedding matrix. the transformer weights are frozen while only the newly introduced embeddings and language adapter weights are fine-tuned on unlabeled text data of the language. for the target-task phase, we propose a novel modality-split architecture (see figure 2 ) inspired by the cross-lingual transfer method of @xcite . at each transformer layer, text and image representations are passed through the pretrained multi-head attention (mha) and feed-forward (ffn) layers. both image and text representations are also passed through the pre-trained language adapters. each modality is then passed through modality-specific text and image task adapters and next through a shared multimodal alignment adapter. 8 we follow @xcite , freezing transformer, embedding and language adapter weights during training, thus fine-tuning only the task and multimodal aligner adapter weights, together with the prediction head. at inference time, the embedding layer and the language adapters are replaced with the target language weights. 4.2 @xmath1. for experiments where we extend a multilingual model to become multimodal, we utilize mbert @xcite . given that mbert is able to represent many different languages, it is not necessary to learn new embedding layers for the target languages in the language-extension phase. instead, we utilize the mbert-compatible language adapters available on adapterhub.ml @xcite ). 9 for the target-task phase, we follow oscar+ for the image representation layer, where image features are combined with their respective positional information and passed through an affine transformation layer. we experiment with the same adapter architecture from figure 2 , as described for oscar+ ada . we again freeze transformer, embedding and language adapter weights during training. however, in contrast to oscar+ * , we randomly initialize and fine-tune the affine image transformation layer. we also fine-tune the task, multimodal aligner adapter weights, and prediction head, all on the gqa task. at inference time, the embedding layer and the language adapters are replaced with the corresponding target language weights. ## results and discussion the main results are presented in ## contemporary work with the recent rise in interest in multilingual vision and language learning, contemporary work has already further analyzed and extended the proposed xgqa dataset. we provide a brief description and pointers to this work in what follows. further analysis. @xcite provide an extensive analysis of multilingual and multimodal models trained on cross-lingual visual question answering, and propose several approaches to mitigate the multilingual misalignment problem discussed in §6.1. their results suggest that standard approaches taken from text-only cross-lingual transfer scenarios @xcite do not leverage the full multilingual capability of the pretrained models. interestingly, they find that a deeper prediction head does not have any measurable impact on the model's performance in the source language, while at the same time it considerably improves zero-shot transfer results across all target languages. translated test data. @xcite propose the first benchmark for transfer learning across modalities, tasks, and languages, covering visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. they extend the xgqa dataset by providing machine translated testset questions and evaluate state-of-the-art monolingual multimodal models in a translate-test setup. in this setting, they achieve slightly better results. however, the performance remains to fall behind source language performance. the translate-test data can be found at iglue-benchmark.github.io. ## conclusion we have proposed xgqa, a first cross-lingual evaluation benchmark for the visual question answering task. xgqa extends the english gqa dataset with development and test data in 7 more typologically diverse languages, covering 5 different scripts. as additional baselines, we have further proposed new adapter-based methods to extend unimodal multilingual models to become multimodal and-viceversa-monolingual multimodal models to become multilingual. our results have indicated that 1) efficient adapter-based methods slightly outperform the pretrained multilingual multimodal model m 3 p in zero-shot scenarios, but 2) the overall zero-shot cross-lingual transfer yields harsh accuracy drops compared to the english performance for all models in comparison. further, accuracy can be partially recovered via few-shot learning, where small amounts of training data are available in the target language. however, the large gaps remain, suggesting the inherent complexity of the cross-lingual task despite it being extremely intuitive and easy to solve by (bilingual) humans. we hope that our dataset and error analysis will motivate future work on this task and, more broadly, in the exciting emerging domain of multilingual multimodal representation learning.