Double Attention-based Multimodal Neural Machine Translation with Semantic Image Regions (2020)

Abstract: Existing studies on multimodal neural machine translation (MNMT) have mainly focused on the effect of combining visual and textual modalities to improve translations. However, it has been suggested that the visual modality is only marginally beneficial. Conventional visual attention mechanisms have been used to select the visual features from equally-sized grids generated by convolutional neural networks (CNNs), and may have had modest effects on aligning the visual concepts associated with textual objects, because the grid visual features do not capture semantic information. In contrast, we propose the application of semantic image regions for MNMT by integrating visual and textual features using two individual attention mechanisms (double attention). We conducted experiments on the Multi30k dataset and achieved an improvement of 0.5 and 0.9 BLEU points for English-German and English-French translation tasks, compared with the MNMT with grid visual features. We also demonstrated concrete improvements on translation performance benefited from semantic image regions.

## introduction neural machine translation (nmt) @xcite has achieved state-of-the-art translation performance. recently, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. many studies @xcite @xcite have been increasingly focusing on incorporating multimodal contents, particularly images, to improve translations. hence, researchers in this field have established a shared task called multimodal machine translation (mmt), which consists of translating a target sentence from a source language description into another language using information from the image described by the source sentence. the first mmt study by @xcite ) demonstrated the potential of improving the translation quality by using image. to effectively use an image, several subsequent studies @xcite @xcite incorporated global visual features extracted from the entire image by convolutional neural networks (cnns) into a source word sequence or hidden states of a recurrent neural network (rnn). furthermore, other studies started using local visual features in the context of an attention-based nmt. these features were extracted from equally-sized grids in an image by a cnn. for instance, multimodal attention @xcite has been designed for a mix of text and local visual features. additionally, double attention mechanisms @xcite have been proposed for text and local visual features, respectively. although previous studies improved the use of local visual features and the text modality, these improvements were minor. as discussed in @xcite , these local visual features may not be suitable to attention-based nmt, because the attention mechanism cannot understand complex relationships between textual objects and visual concepts. other studies utilized richer local visual features to mnmt such as dense captioning features @xcite . however, their efforts have not convincingly demonstrated that visual features can improve the translation quality. @xcite demonstrated that, when the textual context is limited, visual features can assist in generating better translations. mmt models disregard visual features because the quality of the image features or the way in which they are integrated into the model are not satisfactory. therefore, which types of visual features are suitable to mnmt, and how these features should be integrated into mnmt, still remain open questions. this paper proposes the integration of semantic image region features into a double attentionbased nmt architecture. in particular, we combine object detection with a double attention mechanism to fully exploit visual features for mnmt. as shown in figure 1 , we use the semantic im-age region features extracted by an object detection model, namely, faster r-cnn @xcite . compared with the local visual features extracted from equally-sized grids, we believe that our semantic image region features contain object attributes and relationships that are important to the source description. moreover, we expect that the model would be capable of making selective use of the extracted semantic image regions when generating a target word. to this end, we integrate semantic image region features using two attention mechanisms: one for the semantic image regions and the other one for text. code and pre-trained models are publicly available at: https://github.com/zhao-yuting/mnmtwith-semantic-regions. the main contributions of this study are as follows: in figure 2 , our model comprises three parts: the source-sentence side, source-image side, and decoder. inspired by @xcite , we integrated the visual features using an independent attention mechanism. from the source sentence @xmath0, x 2 , x 3 , • • • , x n ) to the target sentence @xmath1, y 2 , y 3 , • • • , y m ), the image-attention mechanism focuses on all semantic image regions to calculate the image context vector z t , while the text-attention mechanism computes the text context vector c t . the decoder uses a conditional gated recurrent unit (cgru) 1 with attention mechanisms to generate the current hidden state s t and target word y t . at time step t, first, a hidden state proposal ŝt is computed in cgru, as presented below, and then used to calculate the image context vector z t and text context vector c t . ## results in comparing the baselines, the doubly-attentive mnmt outperformed opennmt. because there did not exist a big difference amongst the three image feature extraction methods for the doublyattentive mnmt model, we only used resnet-101 in our model. compared with the opennmt baseline, the pro-posed model improved both bleu scores and me-teor scores for @xmath2. additionally, the results of our proposed model are significantly better than the results obtained by the baseline with a p-@xmath3.01 for both tasks. compared with the doubly-attentive mnmt (resnet-101) baseline, the proposed model also improved the bleu scores and meteor scores for both tasks. moreover, the results are significantly better than the baseline results with a @xmath4.01 for @xmath5. for comparison with @xcite , we report their results for the text-only nmt baseline, grid and global visual features for mnmt method. with the grid visual features, their results surpassed the text-only nmt baseline for @xmath6, but failed to surpass the text-only nmt baseline for @xmath7. with the global visual features, their results surpassed the text-only nmt baseline. for @xmath8, though @xcite (global) achieved higher scores than our model, the improvements were minor. in terms of relative improvement compared with the text-only nmt baseline, their results improved the bleu score by 1.8% and meteor score by 0.3%. in contrast, our model improved the bleu score by 6.6% and meteor score by 3.9%. for @xmath9, our results outperform @xcite (global) with regard to both metrics. in terms of relative improvement compared with the text-only nmt baseline, their results improved the bleu score by 1.9% and meteor score by 1.1% with the grid visual features and improved the bleu score by 3.8% and meteor score by 2.3% with the global visual features. our model improved the bleu score by 2.8% and meteor score by 1.5%. ## related work from the first shared task at wmt 2016, foot_4 many mmt studies have been conducted. existing studies have fused either global or local visual image features into mmt. ## conclusion this paper proposed a model that integrates semantic image regions with two individual attention mechanisms. we achieved significantly improved translation performance above two baselines, and verified that this improvement mainly benefited from the semantic image regions. additionally, we analyzed the advantages and shortcomings of our model by comparing examples and visualization of semantic image regions. in the future, we plan to use much finer visual information such as instance semantic segmentation to improve the quality of visual features. in addition, as english entity and image region alignment has been manually annotated to the multi30k dataset, we plan to use it as supervision to improve accuracy of the attention mechanism.