Neural Machine Translation with Phrase-Level Universal Visual Representations (2022)

Abstract: Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them suffer from shortage of sentence-image pairs. In this paper, we propose a phrase-level retrieval-based method for MMT to get visual information for the source input from existing sentence-image data sets so that MMT can break the limitation of paired sentence-image input. Our method performs retrieval at the phrase level and hence learns visual information from pairs of source phrase and grounded region, which can mitigate data sparsity. Furthermore, our method employs the conditional variational auto-encoder to learn visual representations which can filter redundant visual information and only retain visual information related to the phrase. Experiments show that the proposed method significantly outperforms strong baselines on multiple MMT datasets, especially when the textual context is limited.

## introduction multimodal machine translation (mmt) introduces visual information into neural machine translation (nmt), which assumes that additional visual modality could improve nmt by grounding the language into a visual space @xcite . however, most existing mmt methods require additional input of images to provide visual representations, which should match with the source sentence. unfortunately, in practice it is difficult to get this kind of pairwise input of text and images which hinders the applications of mmt. what is worse, to train an mmt model, the training data still involves the target sentence besides the source sentence and the image, which is costly to collect. as a result, the mmt model is usually trained on a small multi30k @xcite data set, which limits the performance of mmt. therefore, it is necessary to utilize the separate image data set to obtain visual representations to break the constraints of pairwise input. towards this end, some researchers @xcite propose to integrate a retrieval module into nmt, which retrieve images related to the source sentence from existing sentenceimage pairs as complementary input, and then use a pre-trained convolutional neural network (cnn) to encode the images. however, such sentence-level retrieval usually suffers from sparsity as it is difficult to get the images that properly match with the source sentence. besides, visual features outputted by the cnn contain richer information (e.g., color, size, shape, texture, and background) than the source text, thus encoding them in a bundle without any filtering will introduce noise into the model. to solve these problems, we propose a novel retrieval-based method for mmt to learn phraselevel visual representations for the source sentence, which can mitigate the aforementioned problems of sparse retrieval and redundant visual representations. for the sparsity problem, our method retrieves the image at the phrase level and only refers to the grounded region in the image related with the phrase. for the redundancy problem, our method employs the conditional variational auto-encoder to force the learned representations to properly reconstruct the source phrase so that the learned representations only retain the information related to the source phrase . experiments on multi30k @xcite show that the proposed method gains significant improvements over strong baselines. when the textual context is limited, it achieves up to 85% gain over the text-only baseline on the bleu score. further analysis demonstrates that the proposed method can obtain visual information that is more related to translation quality. ## phrase-guided visual representation we use phrase-level visual representation to improve nmt. in this section, we will introduce our proposed phrase-guided visual representation. we first build a phrase-level image set, and then introduce a latent-variable model to learn a phraseguided visual representation for each image region. ## nmt with phrase-level universal visual representation in this section, we will introduce our retrievalbased mmt method. specifically, we obtain visual context through our proposed phrase-level visual retrieval, and then learn a universal visual representation for each phrase in the source sentence, which is used to improve nmt. figure 2 shows the overview of our proposed method, which is composed of four modules: source encoder, phraselevel visual retrieval module, multimodal aggregation module, and target decoder. the source encoder and target decoder are the same as the encoder and decoder of conventional text-only trans-former @xcite . therefore, we will introduce the phrase-level visual retrieval module and multimodal aggregation module in detail in the rest of this section. we denote the input source sentence as @xmath0, x 2 , ..., x n ), the ground truth target sentence as y * = (y * 1 , y * 2 , ..., y * m ) and the generated translation as @xmath1, y 2 , ..., y m ). the input source sentence x will be encoded with the source encoder to obtain source sentence representation, which is denoted as @xmath2, h 2 , ..., h n ). ## related work multimodal machine translation (mmt) aims to enhance nmt @xcite @xcite with additional visual context. since the release of multi30k @xcite dataset, researchers have proposed many mmt methods. early methods @xcite @xcite @xcite @xcite @xcite @xcite are mainly based on the rnn-based encoder-decoder architecture with attention @xcite . recent methods based on transformer @xcite achieve better performance. yao and wan (2020); yin et al. (2020); liu et al. (2021) design multimodal encoder to fuse the textual and visual information during encoding. ive et al. (2019); lin et al. (2020) enhance the decoder with deliberation networks @xcite or capsule networks @xcite to better utilize visual information during decoding. @xcite propose a cross-lingual visual pre-training method and fine-tuned for mmt. it is worth noting that some of previous works @xcite @xcite @xcite adopt regional visual information like us, which shows effectiveness compared with global visual features. the major difference between our method and theirs is that our method is a retrieval-based method, which breaks the reliance on bilingual sentence-image pairs, therefore, our method is still applicable when the input is text only (without paired images), which is unfortunately not available with those previous methods. in addition to focusing on model design, yang et al. (2020); nishihara et al. (2020) ; @xcite propose auxiliary loss to allow the model to make better use of visual information. caglayan et al. (2019); wu et al. (2021 ) conduct systematic analysis to probe the contribution of visual modality. caglayan et al. (2020); ive et al. (2021) focus on improving simultaneous machine translation with visual context. all of the above methods require a specific image as input to provide visual context, which heavily restricts their applicability. to break this bottleneck, @xcite propose target-side image retrieval to help the translation. @xcite propose a multitask learning framework imagination to decomposes the multimodal translation into learning translation and learning visually grounded representation. @xcite introduce a latent variable and estimate a joint distribution over translations and images. @xcite predict the translation with visual representation generated by a generative adversarial network (gan) @xcite . the most closely related work to our method is uvr-nmt @xcite , which breaks the reliance on bilingual sentence-image pairs. like some retrieval-enhanced mt @xcite ) methods, they build a topic-image lookup table from multi30k, and then retrieve images related to the source sentence as visual context based on the topic words. the central differences between @xcite and our method are as follows: ## conclusion in this paper, we propose a retrieval-based mmt method, which learns a phrase-level universal visual representation to improve nmt. our method not only outperforms the baseline systems and most existing mmt systems, but also breaks the restrictions on input that hinder the development of mmt in recent years. experiments and analysis demonstrate the effectiveness of our proposed method. in the future, we will explore how to apply our method to other tasks.