A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation (2020)

Abstract: Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images. However, dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning. To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). We then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, these representations provide an attention-based context vector for the decoder. We evaluate our proposed encoder on the Multi30K datasets. Experimental results and in-depth analysis show the superiority of our multi-modal NMT model.

## introduction multi-modal neural machine translation (nmt) @xcite has become an important research direction in machine translation, due to its research significance in multimodal deep learning and wide applications, such as translating multimedia news and web product information @xcite . it significantly extends the conventional text-based machine translation by taking images as additional inputs. the assumption behind this is that the translation is expected to be more accurate compared to purely text-based translation, since the visual context helps to resolve ambiguous multi-sense words @xcite . apparently, how to fully exploit visual information is one of the core issues in multi-modal nmt, which directly impacts the model performance. to this end, a lot of efforts have been made, roughly consisting of: (1) encoding each input image into a global feature vector, which can be used to initialize different components of multi-modal nmt models, or as additional source tokens @xcite , or to learn the joint multi-modal representation @xcite ; (2) extracting object-based image features to initialize the model, or supplement source sequences, or generate attention-based visual context @xcite ; and (3) representing each image as spatial features, which can be exploited as extra context @xcite @xcite , or a supplement to source semantics @xcite ) via an attention mechanism. despite their success, the above studies do not fully exploit the fine-grained semantic correspondences between semantic units within an input sentence-image pair. for example, as shown in figure 1 , the noun phrase "a toy car" semantically corresponds to the blue dashed region. the neglect of this important clue may be due to two big challenges: 1) how to construct a unified representation to bridge the semantic gap between two different modalities, and 2) how to achieve semantic interactions based on the unified representation. however, we believe that such semantic correspondences can be exploited to refine multimodal representation learning, since they enable the representations within one modality to incorporate cross-modal information as supplement during multi-modal semantic interactions @xcite . two boys are playing with a toy car multi-modal graph 𝒗 𝒙 𝟐 𝒗 𝒙 𝟑 𝒗 𝒙 𝟒 𝒗 𝒙 𝟓 𝒗 𝒙 𝟔 𝒗 𝒙 𝟕 𝒗 𝒙 𝟖 𝒗 𝒐 𝟑 𝒗 𝒐 𝟏 𝒗 𝒐 𝟐 𝒗 𝒙 𝟏 image text two boys are playing with a toy car in this paper, we propose a novel graph-based multi-modal fusion encoder for nmt. we first represent the input sentence and image with a unified multi-modal graph. in this graph, each node indicates a semantic unit: textual word or visual object, and two types of edges are introduced to model semantic relationships between semantic units within the same modality (intra-modal edges) and semantic correspondences between semantic units of different modalities (inter-modal edges) respectively. based on the graph, we then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions among the nodes to conduct graph encoding. particularly, during this process, we distinguish the parameters of two modalities, and sequentially conduct intraand inter-modal fusions to learn multi-modal node representations. finally, these representations can be exploited by the decoder via an attention mechanism. compared with previous models, ours is able to fully exploit semantic interactions among multimodal semantic units for nmt. overall, the major contributions of our work are listed as follows: particularly, our multi-modal nmt model significantly outperforms several competitive baselines. ## nmt with graph-based multi-modal fusion encoder our multi-modal nmt model is based on attentional encoder-decoder framework with maximizing the log likelihood of training data as the objective function. ## experiment we carry out experiments on multi-modal english⇒german (en⇒de) and english⇒french (en⇒fr) translation tasks. ## conclusion in this paper, we have proposed a novel graphbased multi-modal fusion encoder, which exploits various semantic relationships between multimodal semantic units for nmt. experiment results and analysis on the multi30k dataset demonstrate the effectiveness of our model. in the future, we plan to incorporate attributes of visual objects and dependency trees to enrich the multi-modal graphs. besides, how to introduce scene graphs into multi-modal nmt is a worthy problem to explore. finally, we will apply our model into other multi-modal tasks such as multimodal sentiment analysis.