Multimodal Neural Machine Translation with Search Engine Based Image Retrieval (2022)

Abstract: Recently, numbers of works shows that the performance of neural machine translation (NMT) can be improved to a certain extent with using visual information. However, most of these conclusions are drawn from the analysis of experimental results based on a limited set of bilingual sentence-image pairs, such as Multi30K.In these kinds of datasets, the content of one bilingual parallel sentence pair must be well represented by a manually annotated image,which is different with the actual translation situation. we propose an open-vocabulary image retrieval methods to collect descriptive images for bilingual parallel corpus using image search engine, and we propose text-aware attentive visual encoder to filter incorrectly collected noise images. Experiment results on Multi30K and other two translation datasets show that our proposed method achieves significant improvements over strong baselines.

## introduction with the development of nmt, the role of visual information in machine translation has attracted researchers' attention @xcite @xcite . although we are still not clear about the specific role of visual information in nmt @xcite , visual information can assist nmt model to achieve better translation performance @xcite @xcite . different with those text-only nmt @xcite ., * *corresponding author in multi30k, as shown in 1, the sentences consists mostly of common and simple words, and the content of each bilingual parallel sentence pair is well represented by a single image. to address the problem, @xcite proposed to transform the existing sentence-image pairs into a topic-image lookup table, and a group of images with similar topics to the source sentence is retrieved from the topic-image lookup table. however, the topic-image lookup table is made from a limited collection of sentence-image pairs, such as multi30k and ms coco image caption dataset @xcite , their image retrieval method is difficult to deal with the out-ofvocabulary words. besides, results from @xcite can hardly prove that the performance of nmt is improved by visual information rather than the co-occurrence of images and sentences. their model may suffer problems in translating sentences with images that are not contained in the topic-image lookup table. in this paper, we propose an open-vocabulary image retrieval methods to collect images for bilingual parallel corpus using image search engine, thus addressing the problems caused by limited collection of sentence-image pairs in @xcite . in detail, to focus on the major part of the sentence, we apply the term frequency-inverse document frequency (tf-idf). instead of a single keyword, we use multiple words as search query for image retrieval to ensure that the contents of collected images are partially consistent with the given sentences. since the quality of images from search engine may be varied, we propose to apply a simple but effective attention layer, and introduce a textaware attentive visual encoder to filter incorrectly collected noise images. the proposed method is then evaluated on three translation datasets, including the multi30k english-to-german, wmt'16 english-to-german, global voices @xcite english-to-german. experiment results show that our proposed method achieves significant improvements over strong baselines. to summarize, out contributions are primarily three-fold: ## related work recently, multimodal nmt models have gradually become a hot topic in machine translation research. they use image information to improve the translation effect of nmt models through different methods. in some cases, visual features are directly used as supplementary information to the text presentation. for example, @xcite takes global visual features and local visual features as additional information for sentences. @xcite initializes the encoder hidden states or decoder hidden states through global visual features. @xcite uses an independent attention mechanism to capture visual representations. @xcite incorporates spatial visual features into the multimodal nmt model via an independent attention mechanism. on this basis, @xcite employs compact bilinear pooling to fuse two modalities. @xcite introduces image-text mutual interactions to refine their semantic representations. @xcite attempts to introduce the capsule network into multimodal nmt, they use the timestep-specific source-side context vector to guide the routing procedure. all the above work is performed on the multi30k dataset. however, some recent studies indicate that the visual features may play a less important role in the nmt model than previously thought. @xcite @xcite . such problems are mainly caused by the limitations of the multi30k dataset. @xcite presents a universal visual representation method that overcomes the shortcomings of multi30k dataset. however, all their image information still comes from multi30k, which is obviously not enough to represent complex machine translation corpus. ## background in this section, we give a simple description of the multimodal nmt model proposed by @xcite . the multimodal nmt model is composed of one text encoder, one visual encoder and one decoder with two attention mechanisms. the multimodal nmt aims to construct an end-to-end neural network to model @xmath0, i) as follows: the decoder is a conditional gru (cgru) foot_1 with two separate attention mechanisms. the text attention mechanism generates a time-dependent context vector c t based on the text hidden states c and the hidden state proposal s ′ t as follows: meanwhile, the visual attention computes a timedependent context vector i t based on the visual feature maps a and the hidden state proposal s ′ t as follows: where s ′ t is calculated by the previous hidden state s t-1 and the previously generated target word y t-1 . ## our proposed method figure 1 shows the 4 components of our proposed method, consisting of image retrieval, text-aware attentive visual encoder, rnn text encoder and translation decoder with co-attention & bi-attention. ## conclusions in this paper, inspired by problem of @xcite caused by applying limited collections of sentence-image pairs, we propose an openvocabulary image retrieval methods to collect descriptive images for bilingual parallel corpus using image search engine, and introduce text-aware attentive visual encoder to filter incorrectly collected noise images. experiment results show that our proposed method achieves significant improvements over strong baselines, especially on multi30k and global voices. further analysis shows that the effectiveness of the proposed methods in translating sentences that describe real objects and people. as one of our future work, we are going to evaluate our proposed method on some larger datasets, such as the entire wmt'16 dataset, and analyze the influence of the number of texts for the task of multimodal nmt.