LVP - M 3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation (2022)

Abstract: Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features, which has attracted considerable attention from both natural language processing and computer vision communities. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world. In other words, the multilingual multimodal machine translation (Multilingual MMT) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for multiple languages. Besides, the image modality has no language boundaries, which is superior to bridging the semantic gap between languages. To this end,we first propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.Then, an effective baseline LVP-M3 using visual prompts is proposed to support translations between different languages,which includes three stages (token encoding, language-aware visual prompt generation, and language translation). Extensive experimental results on our constructed benchmark datasets demonstrate the effectiveness of LVP-M3 method for Multilingual MMT.

## introduction multimodal machine translation (mmt) extends the conventional text-based machine translation by taking corresponding images as additional inputs @xcite to mitigate the problems of data sparsity and ambiguity @xcite when compared with purely text-based machine translation. similar to other multimodal tasks (e.g., visual question answering @xcite , image captioning @xcite (a) for mmt, we need to train different mmt models to support translations between different language pairs (e.g., "en-de" represents to translate the english to german). (b). for multilingual mmt, we only need one single model to translate the source language to different target languages. however, as shown in fig. 1 (a), previous mmt models (e.g., dccn @xcite ) can handle a single language translation pair (e.g., @xmath0, @xmath1, but training a separate model for each language pair is unaffordable considering there are thousands of languages in the world. a straightforward solution to reduce computational cost is to use one model for handling the translations of multiple languages as shown in fig. 1 @xcite . meanwhile, multilingual machine translation has been investigated for many years @xcite , but these existing methods only consider the language as the input, where the vision context has been ignored. therefore, in our work, we first propose the multilingual multimodal machine translation (multilingual mmt) task to achieve the translations for multiple languages using one single model. to eliminate the above limitations, we propose a simple and effective lvp-m 3 method, including token encoding, language-aware visual prompt generation (lvpg), and language translation. specifically, in the token encoding stage, we use the pre-trained vision encoder to extract the visual tokens. then, we follow @xcite to utilize the transformer to encode the textual tokens. in lvpg, inspired by @xcite and @xcite , a controller network in fig. 3 is leveraged to dynamically generate the parameters of the mapping network conditioned on the target language. further, the mapping network outputs the language-aware visual prompts. after that, during the language translation, following the works (e.g., vilbert @xcite ), we utilize co-transformer to generate the vision-guided language tokens. then the transformer decoder is adopted to predict the translation results. extensive experiments are conducted on our proposed benchmark datasets for lvp-m 3 . results show that our model achieves the state-of-the-art performance in all translation directions, especially outperforming the text-only multilingual model by 4.3 bleu scores on average. the contributions of this work are summarized as follows: ## datasets we introduce two multilingual mmt benchmark datasets (i.e., m 3 -multi30k, m 3 -ambigcaps) using multi30k @xcite and ambig-caps @xcite . here, we descried the details of the m 3 -multi30k and m 3 -ambigcaps. ## experiments we evaluate our proposed lvp-m 3 method on the multilingual dataset including 7 languages and 6 translation directions. in all experiments, english (en) is treated as the pivot language for multilingual mmt setting. ## conclusion in our work, we first propose the multilingual mmt task to support the multilingual multimodal machine translations between different language pairs using one single model. then, we propose an effective lvp-m 3 baseline method for the multilingual mmt task, where a language-aware prompt generation module is proposed to generate visual prompts for different target languages dynamically. comprehensive experimental results on our established multilingual mmt benchmark datasets demonstrate the effectiveness of our proposed lvp-m 3 method for multilingual mmt.