Delving Deeper into Cross-lingual Visual Question Answering (2023)

Abstract: Visual question answering (VQA) is one of the crucial vision-and-language tasks. Yet, existing VQA research has mostly focused on the English language, due to a lack of suitable evaluation resources. Previous work on cross-lingual VQA has reported poor zero-shot transfer performance of current multilingual multimodal Transformers with large gaps to monolingual performance, without any deeper analysis. In this work, we delve deeper into the different aspects of cross-lingual VQA, aiming to understand the impact of 1) modeling methods and choices, including architecture, inductive bias, fine-tuning; 2) learning biases: including question types and modality biases in cross-lingual setups. The key results of our analysis are: 1. We show that simple modifications to the standard training setup can substantially reduce the transfer gap to monolingual English performance, yielding +10 accuracy points over existing methods. 2. We analyze cross-lingual VQA across different question types of varying complexity for different multilingual multimodal Transformers, and identify question types that are the most difficult to improve on. 3. We provide an analysis of modality biases present in training data and models, revealing why zero-shot performance gaps remain for certain question types and languages.

## introduction the lack of multilingual resources has hindered the development and evaluation of visual question answering (vqa) methods beyond the english language until recently. a rise in interest in creating multilingual vision-and-language (v&l) resources has inspired more research in this area @xcite @xcite @xcite . large transformer-based models pretrained on images and text in multiple different languages have been proven as a viable vehicle for the development of multilingual v&l task architectures through transfer learning, but such models are still few and far between (m3p, uc2; @xcite . large decreases in task performance between monolingual and (zero-shot) cross-lingual transfer setups have been measured and reported, among other multilingual v&l tasks, in vqa @xcite . yet, the reasons for such low results in this pivotal v&l task have not been investigated in depth. in this work, we aim to shed new light on the cross-lingual performance gap of cross-lingual vqa models from multiple angles. to the best of our knowledge, we are the first to provide a comprehensive analysis of multilingual vqa, with a focus on cross-lingual transfer. we first assess and discuss the impact of modeling methods and choices on the final cross-lingual vqa performance, aiming to mitigate the present performance gap. this includes experimenting with diverse prediction head architectures, incorporating inductive bias by extending input signals, as well as more sophisticated fine-tuning strategies. we analyze cross-lingual vqa across different question types of varying complexity for different multilingual multimodal transformers, and in zero-shot and few-shot scenarios. next, we focus on the learning biases, where we investigate whether current multilingual multimodal models suffer from the so-called unimodal bias: that is, we probe if the models truly reason over both images and questions to solve the vqa task, or if they take unimodal 'shortcuts' instead, exploiting spurious correlations and artifacts of data creation. our analysis allows us to identify the most difficult question types and reveals a shortcoming of the current evaluation scheme. we find that standard approaches from text-only cross-lingual transfer scenarios @xcite do not leverage the full multilingual capabilities of the pretrained models; we mea-sure the considerably worse performance of 'standard' fine-tuning compared to a simple modified fine-tuning regime. interestingly, we report a discrepancy between monolingual and cross-lingual performance in the modified fine-tuning regime: while they do not have any substantial impact on the model performance in the source language (english), they considerably improve cross-lingual vqa capabilities, achieving gains of more than 10 absolute accuracy points over the baselines. code is available at github.com/ukplab/ eacl2023-xlingvqa. ## preliminaries the vqa task is typically framed as a classification problem with a large number of classes. for instance, in the vqa task on the standard english gqa dataset @xcite , given a pair of an image and a question, a model needs to predict a correct answer from 1,853 possible classes. gqa consists of diverse structural and semantic patterns, in which the questions are visually grounded in the image. in multilingual and cross-lingual vqa, the goal is to make similar predictions, but the questions can be posed in different target languages @xcite : e.g., the vqa task on the multilingual xgqa dataset (pfeiffer et al., 2022) relies on the same set of 1,853 classes as english gqa. we base all our analyses and experiments on the xgqa dataset, which is, due to its size and language coverage, arguably the most comprehensive evaluation resource for cross-lingual vqa to date. it has also been included in the multimodal multilingual evaluation benchmark iglue @xcite . xgqa is the multilingual extension of the english gqa dataset @xcite to 7 typologically diverse languages. in this work, we use and empirically compare two state-of-the-art pretrained multimodal multilingual transformer architectures: m3p @xcite and uc2 @xcite . 1 the standard cross-lingual zero-shot transfer setup for vqa involves fine-tuning all the weights of the pretrained model on the downstream task data in the source language only. in the few-shot setup, after the source-language fine-tuning, the model is additionally optimized on a handful of task-annotated examples in the target language @xcite . ## modeling methods motivation. recent work on vqa in cross-lingual settings @xcite benchmarked standard multimodal architectures in zero-shot and few-shot transfer scenarios on the xgqa dataset, without aiming to provide a deeper understanding of the particulars of the crosslingual vqa task. at the same time, they report large gaps of cross-lingual transfer performance when compared to monolingual english performance, suggesting that there is ample room for improvement. in this work, we aim to leverage novel insights into different aspects of the cross-lingual vqa task (e.g., analyses over different question types or classification architectures) to guide improved cross-lingual vqa methods. in particular, we assess the impact of three orthogonal directions: 1) classification architectures ( §3.1); 2) (richer) input signals ( §3.2); 3) fine-tuning strategies ( §3.3). ## analysis methods the vqa task is inherently multimodal-a model is required to reason over both images and questions in textual form to solve the task. however, as with some unimodal text-only tasks @xcite vqa models might also be prone to 'taking shortcuts', that is, exploiting spurious correlations and artifacts of data creation. in other words, the vqa model could circumvent the multimodal aspect and only focus on a single modality to solve the task @xcite . therefore, to better understand the multimodal reasoning abilities of vqa models in cross-lingual transfer, we propose several diagnostic approaches and methods that ablate the input features of the models, inspired by the diagnostic methods of @xcite and @xcite in monolingual setups. they should provide us with deeper insights into the inner workings of cross-lingual vqa models. ## experimental setup pretrained models and data. as introduced in §2, we 1) rely on two standard state-of-the-art multimodal multilingual transformers (m3p, uc2; @xcite as the underlying pretrained models, and 2) conduct all evaluations on the standard monolingual english gqa dataset, and its multilingual extension: xgqa. the gqa dataset consists of two training sets: full and balanced. the full dataset contains 113k images and 22m questions, whereas the balanced dataset consists of 1.7m data samples. the dataset also contains a balanced test-dev set with 12,578 questions and 398 images for evaluation. in xgqa, the questions are manually translated from the training details and hyperparameters. following the recommendations from @xcite , we predominantly run training on the more lightweight balanced subset of gqa. 4 we also define a total training budget of 6 epochs (less than 24 hours of training). for the self-bootstrapping procedure, this means the total training time (stage 1 + stage 2) is equal to 6 epochs. see appendix a for further details. ## results and discussion in §6.1, we discuss the results of the different modeling approaches across the three dimensions (see 4 another established yet less efficient training procedure is to train on the full gqa dataset first, then further train on the balanced dataset @xcite . this procedure can produce good results on the english evaluation dataset at the cost of a substantial increase in computation demands (∼4 days on one nvidia v100 for one model). furthermore, our initial experiments have indicated that training with the balanced set performs similarly to the previously reported baselines in the xgqa paper while using substantially less computing. we stress that we also further run experiments under the more demanding training regime @xcite with the bestperforming model configuration from our experiments. for more details, we refer the reader to §7. §3): classification architectures, input signals and fine-tuning strategies. a finer-grained analysis concerning different structural question types is provided in §6.1. finally, in §6.2 we delve deeper into the vqa models' susceptibility towards exploiting unimodal biases and artifacts of the vqa datasets, relying on model variants discussed in §4. ## additional results training with full english gqa. to validate the effectiveness of our approach in setups where more data in the source language is available, we additionally run experiments in another vqa setup: we train the best-performing method uc2 q +sb for 5 few-shot experiments. besides the zero-shot transfer scenario-which is the primary focus of this work-we also evaluate whether similar findings extend to few-shot scenarios, where a handful of annotated examples in the target language is assumed. following the standard setup of @xcite we start from the weights of the bestperforming model, already fine-tuned on english vqa data. we then further fine-tune it on the few examples in the target language. in particular, we conduct few-shot experiments with 1, 5, and 48 images. 6 following @xcite we fine-tune for 10 epochs, with a learning rate of 5e-5. the results are summarized in more comprehensive multilingual multimodal benchmarks have been developed only recently @xcite @xcite @xcite , inter alia) making it possible to evaluate multimodal models which have either been pretrained on multilingual data @xcite or extended to unseen languages @xcite . our work complements this recent line of work by delving deeper into cross-lingual visual question answering, again highlighting the inherent difficulty of multilingual multimodal learning. ## conclusion in this work, we provide an extensive analysis of the issues present in vqa-related multilingual vision-and-language learning, aiming to inspire new solutions that can improve cross-lingual vqa performance. to this end, we studied simple yet effective methods that increase previously low transfer performance and thus substantially reduce the gap to monolingual english performance. this has been achieved through more sophisticated classification architectures, fine-tuning strategies, and introducing inductive biases to input via questiontype conditioning. we also conducted further analyses and empirical comparisons, including detection of unimodal biases in training and evaluation data, fine-grained analyses across different question types, and comparisons across different multilingual transformer models and transfer scenarios. we hope that this work will spark more interest and inspire future research on cross-lingual vqa tasks in particular, as well as on multilingual multimodal learning in general.