Generative Imagination Elevates Machine Translation (2021)

Abstract: There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the “imagined representation” to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.

## introduction visual foundation has been introduced in a novel multimodal neural machine translation (mnmt) task @xcite @xcite , which uses bilingual (or multilingual) parallel corpora annotated by images describing sentences' contents (see figure 1(a) ). the superiority of mnmt lies in its ability to use visual information to improve the quality of translation, but its effectiveness largely depends on the availability of data sets, especially the quantity and quality of annotated images. in addition, because the cost of manual image annotation is relatively high, at this stage, mnmt is mostly applied on a small and specific dataset, multi30k @xcite , and is not suitable for large-scale text-only neural machine translation (nmt) @xcite 2015; @xcite . such limitations hinder the applicability of visual information in nmt. to address the bottlenecks mentioned above, @xcite propose to build a lookup table from an image dataset and then using the searchbased method to retrieve pictures that match the source language keywords. however, the lookup table is built from multi30k, which leads to a relatively limited coverage of the pictures, and potentially introduces much irrelevant noise. it does not always find the exact image corresponding to the text, or the image may not even exist in the database. @xcite present a multitask learning framework to ground visual representation to a shared space. their architecture called "imagination" shares an encoder between a primary nmt task and an auxiliary task of ranking the vi-sual features for image retrieval. however, neither the image is explicitly generated, nor the visual feature is directly leveraged by the translation decoder, the model simply learns the visual grounded shared encoder. based on other researchers' earlier exploration, we hypothesize that the potential of vision in conventional text-only nmt has not been fully discovered. different with @xcite implicit approach, we understand "imagination" to be more like "picturing", since it is similar to humans who can visually depict figures in the mind from an utterance. our approach aims to explicitly imagine a "vague figure" (see figure 1(b) ) to guide the translation, since a picture is worth a thousand words, and imagining the picture of a sentence is the instinctive reaction of a human being who is learning bilingualism. in this paper, we propose a novel end-to-end machine translation model that is embedded in visual semantics with generative imagination (imagit) (see figure 1(b) ). given a source language sentence, imagit first encodes it and transforms the word representations into visual features through an attentive generator, which can effectively capture the semantics of both global and local levels, and the generated visual representations can be considered as semantic-equivalent reconstructions of sentences. a simple yet effective integration module is designed to aggregate the textual and visual modalities. in the final stage, the model learns to generate the target language sentence based on the joint features. to train the model in an end-to-end fashion, we apply a visual realism adversarial loss and a text-image pair-aware adversarial loss, as well as text-semantic reconstruction loss and target language translation loss based on cross-entropy. in contrast with most prior mnmt work, our proposed imagit model does not require images as input during the inference time but can leverage visual information through imagination, making it an appealing method in low-resource scenario. moreover, imagit is also flexible, accepting external parallel text data or non-parallel image captioning data. we evaluate our imagination modal on the multi30k dataset. the experiment results show that our proposed method significantly outperforms the text-only nmt baseline. the analysis demonstrates that imagination help the model complete the missing information in the sentence when we perform degradation masking, and we also see improvements in translation quality by pre-training the model with an external non-parallel image captioning dataset. to summarize, the paper has the following contributions: ## imagit model as shown in figure2, imagit embodies the encoder-decoder structure for end-to-end machine translation. between the encoder and the decoder, there is an imagination step to generate semanticequivalent visual representation. technically, our model is composed of following modules: source text encoder, generative imagination network, image captioning, multimodal aggregation and decoder for translation. we will elaborate on each of them in the rest of this section. @xcite propose the state-of-art transformer-based machine translation framework, which can be written as follows: ## conclusion this work presents generative imagination-based machine translation model (imagit), which can effectively capture the source semantics and generate semantic-consistent visual representations for imagination-guided translation. without annotated images as input, out model gains significant improvements over text-only nmt baselines and is comparable with the sota mnmt model. we analyze how imagination elevates machine translation and show improvement using external image captioning data. further work may center around introducing more parallel and non-parallel, text, and image data for different training schemes.