Multilingual Multimodal Learning with Machine Translated Text (2022)

Abstract: Most vision-and-language pretraining research focuses on English tasks. However, the creation of multilingual multimodal evaluation datasets (e.g. Multi30K, xGQA, XVNLI, and MaRVL) poses a new challenge in finding high-quality training data that is both multilingual and multimodal. In this paper, we investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model. We apply it to both pretraining and fine-tuning data with a state-of-the-art model. In order to prevent models from learning from low-quality translated text, we propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning, both at pretraining and fine-tuning.

## introduction vision-and-language (v&l) pretraining is the process of learning deep contextualised cross-modal representations from large collections of imagesentence pairs @xcite @xcite . these pretrained models are an excellent backbone for transfer learning to a wide range of downstream tasks, such as visual question answering @xcite @xcite , referring expression alignment @xcite , and image-sentence retrieval @xcite . thus far, downstream evaluations have mostly focused on english tasks due to the availability of datasets, but the recent iglue benchmark @xcite makes it now possible to evaluate models on several downstream tasks across 20 languages. welche art von essen befindet sich oben links in der brotdose? 饭盒左上角是什么食物 ? ¿qué tipo de comida se encuentra en la parte superior izquierda de la lonchera? … welche art von essen befindet sich oben links in der brotdose? 饭盒左上角是什么食物 ? ¿qué tipo de comida se encuentra en la parte superior izquierda de la lonchera? english multimodal dataset, we generate a multilingual dataset using a black box translation system. we explore the utility of this approach to creating multilingual text for both downstream task fine-tuning and pretraining. success in multilingual multimodal tasks, such as those in iglue, is expected to depend on models with grounded representations that transfer across languages @xcite . for example, in the marvl dataset @xcite , models need to deal with a linguistic and cultural domain shift compared to english data. therefore, an open problem is to define pretraining strategies that induce high-quality multilingual multimodal representations. existing work has tackled this problem by either jointly training on english multimodal data and multilingual text-only data @xcite , pretraining with a private dataset of multilingual captioned images @xcite , or machine translating multimodal pretraining data @xcite . in this paper, we further investigate the potential of machine translated text for both fine-tuning and pretraining across four diverse v&l tasks. 1 the overarching motivation is that machine translation is an inexpensive approach to producing large amounts of multilingual text compared to collecting data from humans, or scraping high-quality imagecaption pairs from the web. having access to thou-sands of data points in a target language might indeed be necessary to improve cross-lingual performance in downstream tasks @xcite . as such, translating fine-tuning data into multiple languages may be a compelling approach towards downstream task success. moreover, if this can be achieved through machine translated text, it raises the question of whether we can also pretrain on many millions of multilingual translated examples. motivated by the initial experiments of @xcite , we test this hypothesis further, on more languages and more tasks, reporting more nuanced results from large-scale translated text. overall, we show that machine translation can provide inexpensive and impressive improvements when fine-tuning models for multilingual multimodal tasks. moreover, translation-based pretraining leads to significant gains in zero-shot crosslingual transfer over existing approaches. however, we find mixed results when combining this with multilingual fine-tuning. there are still opportunities to realise further benefits from machine translated text, which may be found through more compute-intensive pretraining. ## related work inspired by the success of self-supervised language model pretraining @xcite , researchers have also explored this paradigm with multimodal models @xcite . the first wave @xcite @xcite were initialised from bert and pretrained on english image-text datasets like conceptual captions @xcite and coco @xcite , where the visual modality was represented using feature vectors extracted from 10-100 automatically detected object proposals @xcite . more recent models @xcite @xcite represent the visual modality using a vision transformer @xcite , which can be end-to-end fine-tuned during pretraining, as opposed to working with pre-extracted object proposals. more related to our work are the multilingual variants of these models @xcite @xcite . the lack of large-scale multilingual multimodal datasets has resulted in different strategies to train such models. @xcite simply augment english caption data with text-only multilingual wikipedia data. in addition to this, @xcite further create code-switched multimodal data foot_1 by randomly swapping english words in conceptual captions with the corresponding translation in one of 50 other languages, obtained through the panlex dictionary. on the other hand, @xcite machine translate the conceptual captions dataset into german, french, czech, japanese, and chinese, for a total of 19.8m pretraining data points. ## the iglue benchmark the impetus of our work is the recent creation of the image-grounded language understanding evaluation (iglue; @xcite ) benchmark for evaluating multimodal models across twenty languages and four tasks, using five different datasets. specifically, the benchmark focuses on zero-and few-shot transfer, where models are fine-tuned on english data and then tested to crosslingually generalise with no or few samples in the target language for the target downstream task. the following datasets are included in iglue: xvnli is a cross-lingual visual natural language inference task @xcite , which requires models to predict the relation (entailment, contradiction, or neutral) between a premise in the form of an image, and a hypothesis in the form of a sentence. marvl focuses on multicultural reasoning over images @xcite . the task is in the same format as the english nlvr2 @xcite data, namely to judge whether a sentence is true or false for a pair of images. however, the images and the descriptions are sourced directly in the target languages. wit is a second image-text retrieval dataset based on the wikipedia-based image text dataset @xcite . wit is scraped directly from wikipedia and contains a much more diverse set of image types than the other datasets, as well as more complex and entity-centric descriptions. each of the tasks has a natural english training counterpart: snli-ve @xcite for xvnli; gqa for xgqa, nlvr2 for marvl, and english training splits of flickr30k and wit. @xcite found that current multilingual v&l models show a large gap in performance, in each of these tasks, when evaluating on non-english data. moreover, further training these models on a few examples in a target language only slightly improved their cross-lingual capabilities. ## fine-tuning with translated data as an initial experiment, we investigate the extent to which performance can be improved by finetuning on multilingual machine-translated data instead of only english data. we conduct this experiment on the marvl and xgqa datasets. the results can be seen in tables 1 and 2 , respectively. we use the m2m-100-large model @xcite to translate the nlvr2 training data into the 5 marvl languages, and the gqa training data into the 7 xgqa languages. for the model, we use the xuniter @xcite implementation from volta @xcite . xuniter extends the uniter architecture @xcite multilingually, by initializing the model from xlm-roberta @xcite and pretraining on english image captions and text-only multilingual wikipedia paragraphs. starting from the publicly-released xuniter checkpoint, we fine-tune on the machine translated training sets for each task. for a fair comparison to english-only fine-tuning, we ensure that the multilingual finetuning is based on the same number of parameter updates. in effect, this reduces the number of training epochs from 20→3 for marvl, and 5→1 for xgqa. we round number of epochs so it is close to the english-only setup. 4 this means that, in our setup, all the images are seen for the same number of times, but each unique caption will be seen fewer times in each of the target languages. using machine translated data for fine-tuning brings large improvements in performance for both marvl and xgqa. ## on pretraining with translated data the previous section showed the benefits of using machine translated data for multilingual finetuning. we now turn our attention to whether further improvements can be realised by adding multilinguality via machine translated data is useful for pretraining. this requires two components: (i) a large-scale translation pipeline and the means to deal with potential data quality issues, and (ii) a model that can exploit the machine translated training data, which we dub td-mml for translated-data multimodal multilingual learning. ## model the model we implement within our translated data for multilingual multimodal learning framework, td-mml, follows the single-stream crossmodal framework, such as uniter @xcite and xuniter @xcite . it can be seen as a translate-train version of xuniter, to which we add an additional pretraining task: visual translation language modelling @xcite . the td-mml architecture consists of a series of transformer blocks, which first concatenate the visual and language embeddings as the input, and then passes these into a multi-layer transformer to encode the contextualized representations across image-text modalities and languages. the input sequences are image-text pairs (v, x), where v are the visual features and x are the embedding sequence of the corresponding caption. the image features v in @xmath0, v 2 , . . . , v n } correspond to @xmath1. we extend the english pretraining text to also include the machine translated captions in m languages: ## experimental setup the implementation of td-mml is built in the volta framework @xcite . td-mml uses the same model configuration as the xuniter architecture @xcite , which is initialised from the xlm-r cross-lingual language model @xcite . pretraining. the dataset used for pretraining is conceptual captions @xcite , which consists of 3.3m images with their english alttext descriptions, of which we only have access to 2.7m instances due to linkrot. the images are represented using resnet-101 features extracted from @xmath2., 2015) object detector trained on the visual genome dataset @xcite . as described in section 5.1, we translate the captions in the 19 non-english iglue languages with the 1.2b-parameter m2m-100-large model @xcite ; the translations are then filtered to eliminate likely translation errors. during training, we iterate over the images and uniformly at random sample the corresponding caption in one of the twenty languages, i.e. a batch contains samples from multiple languages. we reuse the training hyperparameters of @xcite . specifically, xuniter is trained over 2.77m image-english caption pairs, while td-mml is pretrained on 52m image-multilingual caption pairs on 3×24gb titan rtx for 250,000 steps, which takes 5 days. figure 4 shows the recall@1 curves for image retrieval and text retrieval on 5k validation samples from the conceptual captions dataset as a function of the number of pretraining steps. the samples are chosen from languages that represent the original marvl languages, i.e. eng, ind, jpn, swa and cmn, where the non-english data comes from the machine translation process. model performance continues to increase in both metrics until 220,000 updates. therefore, we use this checkpoint to finetune on the downstream tasks. fine-tuning on downstream tasks. we employ a translated data procedure at fine-tuning as well, using the same data filtering steps. similar to the initial experiments in section 4, we match the number of parameter updates between the experiments with english-only data and the ones with translated multilingual data. we use the same setup as in iglue when fine-tuning on downstream tasks. ## conclusion in this paper, we investigate the role of machine translated (mt) data in multilingual multimodal learning in a controlled setup. we consider two applications of mt data, namely for augmenting the pretraining and the fine-tuning data. we find that both convey a clear immediate benefit on downstream performance for nearly all tasks in the iglue benchmark; however, we do not find an additive benefit of combining it for both pretraining and fine-tuning together. when using machine translated text, filtering out bad translations in a quick and reliable way is crucial, and we develop a simple and effective strategy for doing this. our results shed light in the importance of explicitly grounding multilingual text in the visual modality in both pretraining and fine-tuning stages.