Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021)

Abstract: Pre-trained language models have been shown to improve performance in many natural language tasks substantially. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training methods. In this paper, we combine these two approaches to learn visually-grounded cross-lingual representations. Specifically, we extend the translation language modelling (Lample and Conneau, 2019) with masked region classification and perform pre-training with three-way parallel vision & language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state-of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations.

## introduction pre-trained language models @xcite have been proven valuable tools for contextual representation extraction. many studies have shown their effectiveness in discovering linguistic structures @xcite , which is useful for a wide variety of nlp tasks @xcite @xcite . these positive results led to further exploration of (i) cross-lingual pretraining @xcite @xcite through the use of multiple mono-lingual and parallel resources, and (ii) visual pre-training where large-scale image captioning corpora are used to induce grounded vision & language representations @xcite @xcite @xcite . the latter is usually achieved by extending the masked language modelling (mlm) objective @xcite with auxiliary vision & language tasks such as masked region classification and image sentence matching. in this paper, we present the first attempt to bring together cross-lingual and visual pre-training. our visual translation language modelling (vtlm) objective combines the translation language modelling (tlm) @xcite with masked region classification (mrc) @xcite to learn grounded crosslingual representations. unlike most of the prior work that use classification or retrieval based downstream evaluation, we focus on the generative task of multimodal machine translation (mmt), where images accompany captions during translation @xcite . once pre-trained, we transfer the vtlm encoder to a transformer-based @xcite mmt and fine-tune it for the mmt task. to our knowledge, this is also the first attempt of pre-training & fine-tuning for mmt, where the current state of the art mostly relies on training multimodal sequence-to-sequence systems from scratch @xcite @xcite @xcite . our findings highlight the effectiveness of crosslingual visual pre-training: when fine-tuned on the @xmath0, our mmt model surpasses our constrained mmt baseline by about 10 bleu and 8 meteor points. the rest of the paper is organised as follows: §2 describes our pre-training and fine-tuning protocol, §3 presents our quantitative and qualitative analyses, and §4 concludes the paper with pointers for future work. ## method we propose visual translation language modelling (vtlm) objective to learn multimodal crosslingual representations. in what follows, we first describe the tlm objective @xcite and then introduce the modifications required to extend it to vtlm. transformer english modality embeddings german station [/s] [mask] [/s] very [mask] [mask] [/s] typische [/s] [mask] sehr 4 5 3 0 1 2 4 5 3 0 1 2 typical bus + + + + + + + + + + + + + + + + + + + + + + + + eine bushaltestelle image 0 1 2 3 + + + + + + + + [mask] human translation language modelling masked region classification positional embeddings lang. & vision embeddings ## conclusions we proposed a novel cross-lingual visual pretraining approach and tested its efficacy for multimodal machine translation. our pre-training approach extends the tlm framework (lample and conneau, 2019) with regional features and performs masked language modelling and masked region classification on a three-way parallel corpus. we show that this leads to substantial improvements compared to multimodal machine translation with cross-lingual pre-training only or without pre-training at all. as future work, we consider exploring more informed masking strategies for visual regions and investigating the impact of visual masking probability for the mrc pre-training task for downstream mmt performance.