There are thousands of languages on earth, but visual perception is shared among peoples. Existing multimodal neural machine translation  (MNMT) methods achieve knowledge transfer by enforcing one encoder to learn shared representation across textual and visual modalities. However, the training and inference process heavily relies on well-aligned bilingual sentence - image triplets as input, which are often limited in quantity . In this paper, we hypothesize that visual imagination via synthesizing visual representation from source text could help the neural model map two languages with different symbols, thus helps the translation task. Our proposed end-to-end imagination-based machine translation model (ImagiT ) first learns to generate semantic-consistent visual representation from  source sentence, and then generate target sentence based on both text representation and imagined visual representation. Experiments demonstrate that our translation model benefits from visual imagination and significantly outperforms the text-only neural machine translation (NMT) baseline. We also conduct analyzing experiments, and the results show that imagination can help fill in missing information when performing the degradation strategy.
There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference . In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.
R
thousands of languages on earth, but visual perception is shared among peoples.
common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language?
A
methods
R
methods achieve knowledge transfer by enforcing one encoder to learn shared representation across textual and visual modalities. However, the training and inference process heavily relies on well-aligned bilingual
require triplets of bilingual sentence - image for training and tuples of source
R
triplets as input, which are often limited in quantity
for inference
R
hypothesize that visual imagination via synthesizing visual representation from source text could help the neural model map two languages with different symbols, thus helps the translation task. Our proposed end-to-end imagination-based machine translation model (ImagiT )
propose ImagiT, a novel machine translation method via visual imagination. ImagiT
D
semantic-consistent
A
the
R
generate target sentence based on both text representation and imagined visual representation.
utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time.
R
our translation model
ImagiT
R
(NMT) baseline. We also conduct analyzing experiments, and the results show that imagination can help
baselines. Further analysis reveals that the imagination process in ImagiT helps