Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting (2020)

Abstract: Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when images are not available at the testing time.

## introduction neural machine translation @xcite has achieved near human-level performance @xcite . however, its effectiveness strongly relies on the availability of large-scale parallel corpora. unfortunately, preparing the parallel data remains a challenge as there are more than 6,500 languages in the world, and recruiting translators with bilingual or multilingual knowledge to cover all those languages is impractical. as a result, developing methods alleviating the need of well-annotated large parallel corpora has recently attracted increasing attention in the community. these methods fall into two broad categories. the first type of methods use a third language as the pivot @xcite @xcite to enable zero-resource translation. although the progress is encouraging, pivoting with a third language still demands bilingual knowledge for collecting large-scale parallel source-pivot and pivottarget corpora. the second type of methods explore unsupervised approaches @xcite @xcite have recently achieved impressive translation quality. these methods rely only on monolingual data and back-translation @xcite . however, as discussed in @xcite , the alignment of source-target sentences is uncertain and highly subject to proper initialization. using visual content for unsupervised mt @xcite is a promising solution for pivoting and alignment based on its availability and feasibility. abundant multimodal content in various languages are available online (e.g. instagram and youtube). it is also easier to recruit monolingual annotators to describe an image than to find multilingual translators to translate sentences. importantly, visual content is eligible to improve the alignments in the language latent spaces since the physical visual perception is similar among people speaking different languages (e.g. similar "blue car" for a german and a french). based on these insights, we propose a novel unsupervised multimodal mt framework incorporating images as pseudo pivots promoting latent space alignment. in addition to use features of visual objects for multimodal back-translation, we align a shared multilingual visual-semantic embedding (vse) space via leveraging disjoint image-sentence pairs in different languages. as illustrated in figure 2 , for sentences approximately pivoted by similar images (src-img-tgt), drawing embeddings of corresponding image-sentence pairs closer results in better alignments of semantically equivalent sentences in the language latent spaces. inspired by back-translation, we further explore another pseudo pivoting strategy, which approximates multilingual sentence pairs (src-img-tgt) conditioned on a real image via captioning. instead of using annotation of images for pivoting as in @xcite , we generate sentences in two languages pivoted on the real image, and then approximately pairing them as weak supervision for training unsupervised mt system. this approach is analogous to a cross-modal version of back-translation. we make the following contributions: (1) building a unified view of employing visual content for pseudo pivoting. (2) we learn and improve the alignments in the shared multilingual multimodal embedding space for unsupervised mmt with disjoint image-text pairs in different languages. (3) our model achieves state of the art on multi30k and generalizes well to the text-only scenario. when a parallel corpus is available, the maximum likelihood estimation (mle) is usually adopted to optimize the (source to target language) nmt model by minimizing the following loss: among all encoder-decoder models, the transformer @xcite architecture recently achieves state-of-the-art translation quality. instead of using recurrent or convolutional operations, it facilitates multi-head self-attention @xcite . in this paper, we choose the transformer as the underlying architecture for both the translation and the captioning modules. unsupervised machine translation while conventional mt systems rely on the availability of a large parallel corpus, translation with zero-resource (unsupervised mt) @xcite @xcite has drawn increasing research attention. only monolingual sentences are presented at the training and validation phase, i.e., only @xmath0. successful unsupervised mt systems share several common principles. first, they require the pre-training step to initialize the model and establish strong monolingual language models properly. for example, xlm @xcite utilizes the masked language model objective in bert @xcite . mass @xcite utilizes a span-based sequence-to-sequence masking objective for language model pre-training. second, these systems transform the unsupervised problem into a weakly or self-supervised one by automatically generating pseudo sentence pairs via back-translation @xcite . the idea behind can be analogous to the cycleconsistency objective in cyclegan @xcite for image-image translation with unpaired data. specifically, let us denote by h * (y) = (x 1 , • • • , xn ) the sentence in the source language inferred from @xmath1. similarly, let us denote by g * (x) = (ŷ 1 , • • • , ŷm ) the sentence in the target language inferred from @xmath2. then the "pseudo" parallel sentences (h * (y), y) and (x, g * (x)) can be further used to train two two mt models (@xmath3: although reinforcement learning-based approaches @xcite and gumbel-softmax reparametrization @xcite have been used to handle back-propagation thorough non-differentiable "argmax" predictions. in this paper, we do not back-propagate through h * (y) and g * (x) to simplify the training process. ## unsupervised multimodal machine translation as illustrated in figure 1 , our model is composed of seven modules: two encoder-decoder pairs for translation, two decoders for captioning, and one shared visual encoder. in this section, we first detail our basic mmt model architecture and the unsupervised setup. then we introduce pseudo visual pivoting: learning multilingual vse and pivoted captioning. ## experiments and results we first describe the implementation details and the experimental setup. then we compare our approach with baselines with detailed analysis. ## conclusion we have presented a novel approach: pseudo visual pivoting for unsupervised multimodal mt. beyond features, we use visual content to improve the crosslingual alignments in the shared latent space. precisely, our model utilizes the visual space as the approximate pivot for aligning the multilingual multimodal embedding space. besides, it synthesizes image-pivoted pseudo sentences in two languages and pairs them to translate by reconstruction without parallel corpora. the experiments on multi30k show that the proposed model generalizes well and yields new state-of-the-art performance.