Supervised Visual Attention for Multimodal Neural Machine Translation (2020)

Abstract: This paper proposed a supervised visual attention mechanism for multimodal neural machine translation (MNMT), trained with constraints based on manual alignments between words in a sentence and their corresponding regions of an image. The proposed visual attention mechanism captures the relationship between a word and an image region more precisely than a conventional visual attention mechanism trained through MNMT in an unsupervised manner. Our experiments on English-German and German-English translation tasks using the Multi30k dataset and on English-Japanese and Japanese-English translation tasks using the Flickr30k Entities JP dataset show that a Transformer-based MNMT model can be improved by incorporating our proposed supervised visual attention mechanism and that further improvements can be achieved by combining it with a supervised cross-lingual attention mechanism (up to +1.61 BLEU, +1.7 METEOR).

## introduction as mainstream machine translation, neural machine translation (nmt) model, widely used since the early days, is the recurrent neural network (rnn)-based nmt with attention mechanism @xcite . this model achieves higher translation accuracy than conventional rnn-based nmt by using a cross-lingual attention mechanism that captures the relationship between words in source and target language sentences. in recent years, the transformer model @xcite has been attracting much attention because it achieves higher accuracy than methods using @xcite . in addition to the conventional cross-lingual attention mechanism, the transformer model introduces a self-attention mechanism that captures relationships between words in a sentence. various studies have been conducted on methods to improve nmt's performance and on one method constraining the cross-lingual attention mechanism @xcite @xcite . these cited researchers have improved translation performance by using a tool to obtain alignments between words in source and target language sentences in advance and then providing alignments as supervisions to train the cross-lingual attention mechanism. multimodal nmt (mnmt) @xcite is a machine translation task that aims to improve translation performance by using images in addition to source language sentences because input images are considered useful for disambiguation and omission completion. @xcite have proposed an mnmt model that introduces a visual attention mechanism into the transformer-based mnmt model's decoder to capture alignment between words in a sentence and image regions to utilize image features from cnns. additionally, @xcite have proposed a model that introduces a visual attention mechanism into an rnn-based mnmt model's encoder. however, these visual attention mechanisms are automatically trained through mnmt in an unsupervised manner, and, hence, they do not always capture the relationship between words in a sentence and image regions that they should. this paper proposes a supervised visual attention mechanism trained with constraints based on manual alignments between words in a sentence and their corresponding image regions to improve mnmt performance. in the mnmt's conventional visual attention mechanism, the image's region where a word should pay attention is not given as a supervision but is learned automatically through mnmt in an unsupervised manner. in our method, we prepare data that shows alignments between words in a source sentence and objects in an image as supervisions and then directly train the visual attention mechanism in the transformer-based mnmt model's encoder from the supervisions. note that when only alignments between target language words and image objects can be obtained, our method converts them to alignments between source language words and image objects by using word alignments between source and target sentences. we experimented with english-german and german-english translation using the multi30k dataset @xcite and with english-japanese and japanese-english translation using the flickr30k entities jp dataset @xcite . these experiments show that the proposed supervised visual attention mechanism improves a transformer-based mnmt model's performance (i.e., meteor and bleu scores). ## background in this section, we overview the transformer nmt model @xcite and then explain supervised cross-lingual attention for the transformer model @xcite . ## proposed method this section explains the proposed method for supervised training of visual attention mechanisms to improve mnmt translation performance. first, we describe our transformer-based mnmt model and then propose a supervised training for the visual attention mechanism. ## experiments we performed four translation experiments: english-german, german-english, english-japanese, and japanese-english. for english-german and german-english translation experiments, we used the multi30k dataset @xcite , which consists of a pair of images and their captions. 29,000 pairs of training data, and 1,014 pairs of development data. we used the '2016 test set' as test data; 1,000 test pairs. for english-japanese and japanese-english translation experiments, we used the flickr30k entities jp dataset @xcite 1 . following text pre-processing in the multi30k dataset, we applied lowercase, punctuation normalization, and the moses tokenizer foot_1 for english sentences. we used kytea foot_2 for word segmentation of japanese sentences. only pairs in which both english and japanese sentences were less than 100 words were used in training data. we used 59,516 pairs for training, 2,017 pairs for development, and 2,000 pairs for testing. as pre-processing, the image was resized to 256 × 256, and then the image's central part was cropped to 224 × 224. resnet50 @xcite was used for cnn in the image encoder. output of the final convolutional layer in resnet50 was used for image features, which consisted of 7 × 7 × 2048 features. cnn fine-tuning was not performed during training. the image encoder, source sentence encoder, and decoder were stacked with 6 layers. the number of heads was 8, and the number of embedding dimensions was 512. the size of the inner feed-forward network layer was set to 2048. we (a) reference (b) mnmt (w/o constraint) (c) mnmt+sva figure 6: visual attentions for the word "lamp" used the adam optimizer (kingma and ba, 2014), and we trained a model for 40 epochs with a minibatch size of 128. we applied byte pair encoding (bpe) (sennrich et al., 2016) for english-german and german-english experiments, in which we used 3,492 words for english vocabulary and 4,536 words for german vocabulary. during inference, target sentences were generated by the greedy method. translation performance was evaluated by bleu @xcite and meteor @xcite . we selected the epoch model with the highest bleu value for development data and evaluated performance for the test data. in experiments, we compared: (i) the text-only transformer nmt model (nmt), (ii) the unconstrained multimodal transformer nmt model (mnmt), (iii) the model with only the supervised visual attention mechanism (mnmt+sva), and (iv) the model with supervised both visual and cross-lingual attention mechanisms (mnmt+sva+sca). the supervised cross-lingual attention mechanism is the same as @xcite 's supervised attention mechanism described in section 2.2. constraints to the attention mechanism for supervision were applied to the sixth layer (i.e., the final layer) of the source language sentence encoder and the decoder. constraints were placed on one head of the visual attention mechanism and the cross-lingual attention mechanism. in equation ( 10 ), hyperparameters were λ 1 = 0.05 and λ 2 = 0.05, following @xcite . in the model with only the supervised visual attention mechanism, hyperparameters were set to λ = 0.05. note that alignments between words and image objects are tagged only in english sentences for both experimental datasets. as for english-german and english-japanese tasks, supervision matrices for the visual attention mechanism were created directly from manually annotated alignments. as for german-english and japanese-english tasks, manually annotated alignments in english sentences were first converted to alignments between german/japanese words and image objects by using word alignments obtained by mgiza, and then supervision matrices for the visual attention mechanism were created from converted alignments. ## conclusion we have proposed an mnmt supervised visual attention mechanism that generates supervisions for visual attentions from manual alignments between words in a sentence and their corresponding regions of an image and trains visual attentions of an mnmt model's encoder from the supervisions. experiments showed that our proposed supervised attention mechanism improved a transformer-based mnmt model on english-german and german-english translation tasks using the multi30k dataset and on english-japanese and japanese-english translation tasks using the flickr30k entities jp dataset. in future work, we will explore our proposed supervised visual attention mechanism's effectiveness for other transformer-based mnmt models.