Visual Prediction Improves Zero-Shot Cross-Modal Machine Translation (2023)

Abstract: Multimodal machine translation (MMT) systems have been successfully developed in recent years for a few language pairs. However, training such models usually requires tuples of a source language text, target language text, and images. Obtaining these data involves expensive human annotations, making it difficult to develop models for unseen text-only language pairs. In this work, we propose the task of zero-shot cross-modal machine translation aiming to transfer multimodal knowledge from an existing multimodal parallel corpus into a new translation direction. We also introduce a novel MMT model with a visual prediction network to learn visual features grounded on multimodal parallel data and provide pseudo-features for text-only language pairs. With this training paradigm, our MMT model outperforms its text-only counterpart. In our extensive analyses, we show that (i) the selection of visual features is important, and (ii) training on image-aware translations and being grounded on a similar language pair are mandatory.

## introduction multimodal machine translation (mmt) aims to improve translation quality with the help of other modalities, such as images @xcite or videos @xcite . mmt models have shown promising improvement over their text-only neural machine translation (mt) counterparts, especially when it matters @xcite @xcite . while prior work has successfully developed mmt models for language pairs with available multimodal parallel corpora, incorporating visual information into language pairs with no multimodal dataset has ## zero-shot cross-modal machine translation we propose a new challenge for multimodal machine translation systems that we denote zero-shot cross-modal machine translation (figure 1 ). this task is motivated by the real-world lack and cost of multimodal parallel corpora, which inhibits the development of multimodal translation systems beyond a few, mostly indo-european, language pairs. ## proposed approach: m2kt-vpn in this section, we introduce a new mmt model, called m2kt-vpn, which aims to transfer multimodal knowledge learned from the multimodal corpus into the zero-shot language pair. m2kt-vpn comprises four modules (figure 2 ): a transformer @xcite encoder to encode a source sentence, a visual prediction network (vpn) to predict a visual feature, a fusion module to incorporate multimodal information, and a transformer decoder to generate a system output. all modules are trained simultaneously on grounded and zero-shot language pairs. ## discussion this section first provides two basic analyses of the m2kt-vpn model: model analysis and probing. we then examine various kinds of features to investigate the importance of feature selection. finally, we ran an analysis to identify the requirement for the grounded language pair. ## related work multimodal machine translation. this task has been developed along with the creation of multimodal parallel corpora. after the first multimodal parallel corpus, namely multi30k for english-german translation, emerged at the first conference of machine translation @xcite , many publicly available datasets have been proposed: the english-french version of multi30k and new test sets at 2017 @xcite , the english-czech version of multi30k @xcite , and the english-japanese version of multi30k @xcite . more recently, @xcite proposed a private expansion of multi30k, including hindi, turkish, and latvian translations. they examined a multilingual mmt model on their dataset and investigated the multilingual ability of the model. we put the step forward and investigate the zero-shot cross-modal translation capability in an mmt task. predicting a visual feature from textual modality is a well-established approach for improving multimodal machine translation systems. @xcite first divided the multimodal machine translation task into two subtasks: translation task and visual grounding task. similarly, zhou et al. ( 2018 ) employed a latent space learning task as their visual grounding task to bridge textual and visual modalities. recently, @xcite proposed to utilize the feature prediction from a visual prediction network. we make use of the model for the visually grounding task and propose to incorporate the prediction as a pseudo-visual feature with mmt models. ## conclusion in this paper, we proposed a new task, zeroshot cross-modal machine translation, aiming to evaluate mmt systems from the perspective of the cross-lingual transferability of multimodal knowledge learned from grounded language pairs into language pairs with only text data during training. our proposed mmt model shows promising results, suggesting that the vpn mitigates the modality mismatch between training and inference steps for zero-shot language pairs. the analysis shows the importance of selecting a proper visual feature and the necessity of image-aware translations, both of which should be key properties of mmt models.