Multimodal Transformer for Multimodal Machine Translation (2020)

Abstract: Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality. Previous works propose various incorporation methods, but most of them do not consider the relative importance of multiple modalities. Equally treating all modalities may encode too much useless information from less important modalities. In this paper, we introduce the multimodal self-attention in Transformer to solve the issues above in MMT. The proposed method learns the representation of images based on the text, which avoids encoding irrelevant information in images. Experiments and visualization analysis demonstrate that our model benefits from visual information and substantially outperforms previous works and competitive baselines in terms of various metrics.

## introduction multimodal machine translation (mmt) is a novel machine translation (mt) task which aims at designing better translation systems using context from an additional modality, usually images (see figure 1 ). it initially organized as a shared task within the first conference on machine translation @xcite @xcite . current works focus on the dataset named multi30k @xcite , a multilingual extension of flickr30k dataset with translations of the english image descriptions into different languages. previous works propose various incorporation methods. @xcite utilize global image features to initialize the encoder/decoder hidden states of rnn. @xcite model the source sentence and reconstruct the image representation jointly via multi-task learning. recently, @xcite propose a translate-and-refine approach using two-stage decoder based on transformer @xcite . @xcite put forward a latent variable model to learn the interaction between visual and textual features. however, in multimodal tasks the different modalities usually are not equally important. for example, in mmt the text is obviously more important than images. although the image carries richer information, it also contains more irrelevant content. if we directly encode the image features, it may introduce a lot of noise. to address the issues above, we propose the multimodal transformer. the proposed model does not directly encode image features. instead, the hidden representations of images are induced from the text under the guide of image-aware attention. meanwhile, we introduce a better way to incorporate information from other modality based on a graph perspective of transformer. experimental results and visualization show that our model can make good use of visual information and substantially outperforms the current state of the art. ## methodology our model is adapted from transformer and it is also an encoder-decoder architecture, consisting of stacked encoder and decoder layers. the focus of our work is to build a powerful encoder to incorporate the information from other modality. thus, we will first begin with an introduction to the incorpo-ration method. then we will detail the multimodal self-attention. the final representations of text and images are sent to the sequence decoder to generate the target text. ## conclusion in this paper, we propose the multimodal selfattention to consider the relative importance between different modalities in the mmt task. the hidden representations of less important modality (image) are induced from the important modality (text) under the guide of image-aware attention. the experiments and visualization show that our model can make good use of multimodal information and get better performance than previous works. there are various multimodal tasks where multiple modalities have different relative importance. in future work, we would like to investigate the effectiveness of our model in these tasks.