Original: N EURAL M ACHINE T RANSLATION WITH U NIVERSAL V ISUAL R EPRESENTATION
Revised: N EURAL M ACHINE T RANSLATION WITH U NIVERSAL V ISUAL R EPRESENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Though visual information has been introduced for enhancing neural machine translation (NMT), its effectiveness strongly relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations.
Revised: Though visual information has been introduced for enhancing neural machine translation (NMT), its effectiveness strongly relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations.

Original: In this paper, we present a universal visual representation learned over the monolingual corpora with image annotations, which overcomes the lack of largescale bilingual sentence-image pairs, thereby extending image applicability in
Revised: In this paper, we present a universal visual representation learned over the monolingual corpora with image annotations, which overcomes the lack of largescale bilingual sentence-image pairs, thereby extending image applicability in

Original: NMT.
Revised: NMT.

Original: In detail, a group of images with similar topics to the source sentence will be retrieved from a light topic-image lookup table learned over the existing sentence-image pairs, and then is encoded as image representations by a pretrained ResNet.
Revised: In detail, a group of images with similar topics to the source sentence will be retrieved from a light topic-image lookup table learned over the existing sentence-image pairs, and then is encoded as image representations by a pretrained ResNet.

Original: An attention layer with a gated weighting is to fuse the visual information and text information as input to the decoder for predicting target translations.
Revised: An attention layer with a gated weighting is to fuse the visual information and text information as input to the decoder for predicting target translations.

Original: In particular, the proposed method enables the visual information to be integrated into large-scale text-only NMT in addition to the multimodel
Revised: In particular, the proposed method enables the visual information to be integrated into large-scale text-only NMT in addition to the multimodel

Original: NMT.
Revised: NMT.

Original: Experiments on four widely used translation datasets, including the
Revised: Experiments on four widely used translation datasets, including the

Original: WMT’16 English-to-Romanian, WMT’14 English-to-German, WMT’14 Englishto-French, and Multi30K , show that the proposed approach achieves signiﬁcant improvements over strong baselines.
Revised: WMT’16 English-to-Romanian, WMT’14 English-to-German, WMT’14 Englishto-French, and Multi30K , show that the proposed approach achieves signiﬁcant improvements over strong baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Visual information has been shown beneﬁcial in neural machine translation  (NMT) (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018).
Revised: Visual information has been introduced for neural machine translation in some previous studies (NMT) (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018; Ive et al., 2019) though the contribution of images is still an open question (Elliott, 2018; Caglayan et al., 2019).

Original: Typically, each bilingual (or multilingual) parallel sentence pair is annotated manually by one image describing the content of this sentence pair.
Revised: Typically, each bilingual (or multilingual) parallel sentence pair is annotated manually by one image describing the content of this sentence pair.

Original: The bilingual parallel corpora with manual image annotations are used to train a multimodel NMT model by an end-to-end framework, and results are reported on a speciﬁc data set,
Revised: The bilingual parallel corpora with manual image annotations are used to train a multimodel NMT model by an end-to-end framework, and results are reported on a speciﬁc data set,

Original: Multi30K (Calixto & Liu, 2017; Calixto et al., 2017).
Revised: Multi30K (Calixto & Liu, 2017; Calixto et al., 2017).

Original: One strong point of the multimodel NMT model is the ability to use visual information to improve the quality of the target translation.
Revised: One strong point of the multimodel NMT model is the ability to use visual information to improve the quality of the target translation.

Original: However, the effectiveness heavily relies on the availability of bilingual parallel sentence pairs with manual image annotations, which hinders the image applicability to the NMT.
Revised: However, the effectiveness heavily relies on the availability of bilingual parallel sentence pairs with manual image annotations, which hinders the image applicability to the NMT.

Original: As a result, the visual information is only applied to the translation task over a small and speciﬁc multimodel data set Multi30K (Elliott et al., 2016), but not to large-scale text-only NMT (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) and low-resource text-only NMT (Fadaee et al., 2017; Lample et al., 2018;
Revised: As a result, the visual information is only applied to the translation task over a small and speciﬁc multimodel data set Multi30K (Elliott et al., 2016), but not to large-scale text-only NMT (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017) and low-resource text-only NMT (Fadaee et al., 2017; Lample et al., 2018;

Original: Ma et al., 2019;
Revised: Ma et al., 2019;

Original: Zhou et al., 2019).
Revised: Zhou et al., 2019).

Original: In addition, because of the high cost of annotation, the content of one bilingual parallel sentence pair is only represented by a single image, which is weak in capturing the diversity of visual information.
Revised: In addition, because of the high cost of annotation, the content of one bilingual parallel sentence pair is only represented by a single image, which is weak in capturing the diversity of visual information.

Original: The current situation of introducing visual information results in a bottleneck in the multimodel
Revised: The current situation of introducing visual information results in a bottleneck in the multimodel

Original: NMT, and is not feasible for text-only NMT and low-resource NMT.
Revised: NMT, and is not feasible for text-only NMT and low-resource NMT.

Original: In this paper, we present a universal visual representation (VR) method relying only on image-monolingual annotations instead of the existing approach that depends on image-bilingual annotations, thus breaking the bottleneck of using visual information in NMT.
Revised: In this paper, we present a universal visual representation (VR) method relying only on image-monolingual annotations instead of the existing approach that depends on image-bilingual annotations, thus breaking the bottleneck of using visual information in NMT.

Original: In detail, we transform the existing sentence-image pairs into topic-image lookup table from a small-scale multimodel data set Multi30K .
Revised: In detail, we transform the existing sentence-image pairs into topic-image lookup table from a small-scale multimodel data set Multi30K .

Original: During the training and decoding process, a group of images with similar topic to the source sentence will be retrieved from the topic-image lookup table learned by the term frequency-inverse document frequency, and thus is encoded as image representations by a pretrained ResNet (He et al., 2016).
Revised: During the training and decoding process, a group of images with similar topic to the source sentence will be retrieved from the topic-image lookup table learned by the term frequency-inverse document frequency, and thus is encoded as image representations by a pre- trained ResNet (He et al., 2016).

Original: A simple and effective attention layer is then designed to fuse the image representations and the original source sentence representations as input to the decoder for predicting target translations.
Revised: A simple and effective attention layer is then designed to fuse the image representations and the original source sentence representations as input to the decoder for predicting target translations.

Original: In particular, the proposed approach can be easily integrated into the text-only NMT model without annotating large-scale bilingual parallel corpora.
Revised: In particular, the proposed approach can be easily integrated into the text-only NMT model without annotating large-scale bilingual parallel corpora.

Original: The proposed method was evaluated on four widely-used translation datasets, including the WMT’16 Englishto-Romanian, WMT’14 English-to-German, WMT’14 English-to-French, and Multi30K which are standard corpora for NMT and multi-modal machine translation (MMT) evaluation.
Revised: The proposed method was evaluated on four widely-used translation datasets, including the WMT’16 Englishto-Romanian, WMT’14 English-to-German, WMT’14 English-to-French, and Multi30K which are standard corpora for NMT and multi-modal machine translation (MMT) evaluation.

Original: Experiments and analysis show effectiveness.
Revised: Experiments and analysis show effectiveness.

Original: In summary, our contributions are primarily three-fold:
Revised: In summary, our contributions are primarily three-fold:

Original: We present a universal visual representation method that overcomes the shortcomings of the bilingual (or multilingual) parallel data with manual image annotations for MMT.
Revised: We present a universal visual representation method that overcomes the shortcomings of the bilingual (or multilingual) parallel data with manual image annotations for MMT.

Original: The proposed method enables the text-only NMT to use the multimodality of visual information without annotating the existing large scale bilingual parallel data.
Revised: The proposed method enables the text-only NMT to use the multimodality of visual information without annotating the existing large scale bilingual parallel data.

Original: Experiments on different scales of translation tasks veriﬁed the effectiveness and generality of the proposed approach.
Revised: Experiments on different scales of translation tasks veriﬁed the effectiveness and generality of the proposed approach.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recently, visual information plays an essential role in the image description generation (IDG) task. Typically, the neural-based IDG model aims to generate a description in the end-to-end way (Mao et al., 2014; Luong et al., 2015; Elliott et al., 2015; Venugopalan et al., 2015; Xu et al., 2015). For example, Xu et al. (2015) introduced the ﬁrst attention-based IDG model in which an attentive decoder learns to attend to different parts of an image as it generates its description in natural language.
Revised: Inspired by the studies on the image description generation (IDG) task (Mao et al., 2014; Luong et al., 2015; Elliott et al., 2015; Venugopalan et al., 2015; Xu et al., 2015a), a new shared translation task for multimodel machine translation was addressed by the machine translation community (Specia et al., 2016).

Original: The work provides a non-trivial clue that non-textual visual information is a useful feature for textual-based natural language processing tasks.

Original: Inspired by these studies on the IDG task, a new shared translation task for multimodel machine translation was addressed by the machine translation community (Specia et al., 2016).

Original: In particular, the released dataset Multi30K (Elliott et al., 2016) includes 29,000 multilingual (English, German, and French) parallel sentence pairs with image annotations (Elliott et al., 2017; Barrault et al., 2018).
Revised: In particular, the released dataset Multi30K (Elliott et al., 2016) includes 29,000 multilingual (English, German, and French) parallel sentence pairs with image annotations (Elliott et al., 2017; Barrault et al., 2018).

Original: Subsequently, there has been a rise in the number of studies (Caglayan et al., 2016; 2017; Calixto et al., 2016; Huang et al., 2016; Libovick ` y & Helcl, 2017; Helcl et al., 2018).
Revised: Subsequently, there has been a rise in the number of studies (Caglayan et al., 2016; 2017; Calixto et al., 2016; Huang et al., 2016; Libovick ` y & Helcl, 2017; Helcl et al., 2018).

Original: For example, Calixto et al.
Revised: For example, Calixto et al.

Original: (2017) proposed a doubly-attentive multi-modal NMT model to incorporate spatial visual features, improving the translation performance.
Revised: (2017) proposed a doubly-attentive multimodal NMT model to incorporate spatial visual features, improving the translation performance.

Original: Compared with spatial-visual features, Calixto & Liu (2017) further incorporated global image features as words in the source sentence and to enhance the encoder or decoder hidden state.
Revised: Compared with spatial-visual features, Calixto & Liu (2017) further incorporated global image features as words in the source sentence and to enhance the encoder or decoder hidden state.

Original: In contrast, some recent studies indicated that the visual modality is either unnecessary (Zhang et al., 2017) or only marginally beneﬁcial (Gr ¨ onroos et al., 2018).
Revised: In contrast, some recent studies indicated that the visual modality is either unnecessary (Zhang et al., 2017) or only marginally beneﬁcial (Gr¨onroos et al., 2018).

Original: More recently, Ive et al.
Revised: More recently, Ive et al.

Original: (2019) showed that visual information is only needed in particular cases, such as for ambiguous words where the textual context is not sufﬁcient.
Revised: (2019) showed that visual information is only needed in particular cases, such as for ambiguous words where the textual context is not sufﬁcient.

Original: However, these approaches only center around a small and speciﬁc Multi30K data set to build multimodel NMT model, which hinders image applicability to NMT.
Revised: However, these approaches only center around a small and speciﬁc Multi30K data set to build multimodel NMT model, which hinders image applicability to NMT.

Original: The reason would be the high cost of image annotations, resulting potentially in the image information not being adequately discovered.
Revised: The reason would be the high cost of image annotations, resulting potentially in the image information not being adequately discovered.

Original: We believe that the capacity of MMT has not yet been excavated sufﬁciently and there is still a long way to go before the potential of MMT is fully discovered.
Revised: We believe that the capacity of MMT has not yet been excavated sufﬁciently and there is still a long way to go before the potential of MMT is fully discovered.

Original: In this work, we seek to break this constraint and enable visual information to beneﬁt NMT, especially text-only NMT.
Revised: In this work, we seek to break this constraint and enable visual information to beneﬁt NMT, especially text-only NMT.

Original:  U NIVERSAL V ISUAL R ETRIEVAL
Revised:  U NIVERSAL V ISUAL R ETRIEVAL

Original: In this section, we will introduce the proposed universal visual representation method.
Revised: In this section, we will introduce the proposed universal visual representation method.

Original: Generally, the default input setting of the MMT is a sentence-image pair.
Revised: Generally, the default input setting of the MMT is a sentence-image pair.

Original: Our basic intuition is to transform the existing sentence-image pairs into topic-image lookup table 1 , which assumes the topic words in a sentence should be relevant to the paired image.
Revised: Our basic intuition is to transform the existing sentence-image pairs into topic-image lookup table 1 , which assumes the topic words in a sentence should be relevant to the paired image.

Original: Consequently, a sentence can possess a group of images by retrieving the topic-image lookup table.
Revised: Consequently, a sentence can possess a group of images by retrieving the topic-image lookup table.

Original: Topic-image Lookup Table Conversion To focus on the major part of the sentence and suppress the noise such as stopwords and low-frequency words, we design a ﬁltering method to extract the 
Revised: Topic-image Lookup Table Conversion To focus on the major part of the sentence and suppress the noise such as stopwords and low-frequency words, we design a ﬁltering method to extract the “topic” words of the sentence through the term frequency-inverse document frequency (TF-IDF)inspired by Chen et al. Speciﬁcally, given an original input sentence X = { x 1 , x 2 , . . . , x I } describe

Original: Require: Input sentences, S =
Revised: Require: Input sentences, S =

Original: { X 1 , X 2 , . . .
Revised: { X 1 , X 2 , . . .

Original: X I } and paired images E = { e 1 , e 2 , . .
Revised: X I } and paired images E = { e 1 , e 2 , . .

Original: , e I }
Revised: , e I }

Original: Ensure: Topic-image lookup table Q where each word is associated with a group of images 1: Obtain the TF-IDF dictionary F = TF-IDF( S ) 2: Transform sentence-image pair to topic-image lookup table Q = LookUp
Revised: Ensure: Topic-image lookup table Q where each word is associated with a group of images 1: Obtain the TF-IDF dictionary F = TF-IDF( S ) 2: Transform sentence-image pair to topic-image lookup table Q = LookUp

Original: ( S , E , F ) 3: procedure TF-IDF( S ) 4: for each sentence in S do 5:
Revised: ( S , E , F ) 3: procedure TF-IDF( S ) 4: for each sentence in S do 5:

Original: Filter stop-words in the sentence 6:
Revised: Filter stop-words in the sentence 6:

Original: Calculate the TF-IDF weight for each word 7: end for 8: return TF-IDF dictionary F 9: end procedure 10: procedure L OOK U P ( S , E , F ) 11: for For each pair { T i , e i } ∈ zip { S, E } do 12:
Revised: Calculate the TF-IDF weight for each word 7: end for 8: return TF-IDF dictionary F 9: end procedure 10: procedure L OOK U P ( S , E , F ) 11: for For each pair { T i , e i } ∈ zip { S, E } do 12:

Original: Rank and pick out the top- k “topic” words in the sentence according to the TF-IDF score in the dictionary F , and each sentence is reformed as T = { t 1 , t 2 , . .
Revised: Rank and pick out the top- w “topic” words in the sentence according to the TF-IDF score in the dictionary F , and each sentence is reformed as T = { t 1 , t 2 , . .

Original: . , t w } 13:
Revised: . , t w } 13:

Original: Pair the k words with the corresponding image e i 14: for For each word t j in T do 15: ift j in M and e j not in Q [ t j ] then 16:
Revised: Pair the w words with the corresponding image e i 14: for For each word t j in T do 15: if e i not in Q [ t j ] then 16:

Original: Add e j to the corresponding image set Q [ t j ]  17: end if 18: end for 19: end for 20: return Topic-image lookup table Q 21: end procedure“topic” words of the sentence through the term frequency-inverse document frequency (TF-IDF)inspired by Chen et al. Speciﬁcally, given an original input sentence X = { x 1 , x 2 , . . . , x I } of length I and its paired image e , X is ﬁrst ﬁltered by a stopword list 3 and then the sentence is treated as a document g .
Revised: Add e j to the corresponding image set Q [ t j ] for word t j 17: end if 18: end for 19: end for 20: return Topic-image lookup table Q 21: end procedure of length I and its paired image e , X is ﬁrst ﬁltered by a stopword list 3 and then the sentence is treated as a document g .

Original: We then compute TF-IDF TI i,j for each word x i in g ,TI i,j = o
Revised: We then compute TF-IDF TI i,j for each word x i in g ,

Original: [Equation]
Revised: [Equation]

Original: where o i,j represents the number of occurrences of the word x i in the input sentence g , | G | the total number of source language sentences in the training data, and | j : x i ∈ g | the number of source sentences including word x i in the training data.
Revised: where o i,j represents the number of occurrences of the word x i in the input sentence g , | G | the total number of source language sentences in the training data, and | j : x i ∈ g | the number of source sentences including word x i in the training data.

Original: We then select the top- w high TF-IDF words as the new image description T = { t 1 , t 2 , . . .
Revised: We then select the top- w high TF-IDF words as the new image description T = { t 1 , t 2 , . . .

Original: , t w } for the input sentence X .
Revised: , t w } for the input sentence X .

Original: After preprocessing, each ﬁltered sentence T is paired with an image e , and each word t i ∈ T is regarded as the topic word for image e .
Revised: After preprocessing, each ﬁltered sentence T is paired with an image e , and each word t i ∈ T is regarded as the topic word for image e .

Original: After processing the whole corpus (i.e., Multi30K ), we form a topic-image lookup table Q as described in Algorithm 1, in which each topic word t i would be paired with dozens of images.
Revised: After processing the whole corpus (i.e., Multi30K ), we form a topic-image lookup table Q as described in Algorithm 1, in which each topic word t i would be paired with dozens of images.

Original: Image Retrieval For input sentence, we ﬁrst obtain its topic words according to the text preprocessing method described above.
Revised: Image Retrieval For input sentence, we ﬁrst obtain its topic words according to the text preprocessing method described above.

Original: Then we retrieve the associated images for each topic word from the lookup table Q and group all the retrieved images together to form an image list G .
Revised: Then we retrieve the associated images for each topic word from the lookup table Q and group all the retrieved images together to form an image list G .

Original: We observe that an image might be associated with multiple topic words so that it would occur multiple times in the list G .
Revised: We observe that an image might be associated with multiple topic words so that it would occur multiple times in the list G .

Original: Therefore, we sort the images according to the frequency of occurrences in G to maintain the total number of images for each sentence at m .
Revised: Therefore, we sort the images according to the frequency of occurrences in G to maintain the total number of images for each sentence at m .

Original: Figure 1 illustrates the retrieval process 4 .
Revised: Figure 1 illustrates the retrieval process 4 .

Original: In the left block, we show six examples of sentence-image pairs in which the topic words are in boldface.
Revised: In the left block, we show six examples of sentence-image pairs in which the topic words are in boldface.

Original: Then we process the corpus using the topic-image transformation method demonstrated above and obtain the topic-image lookup table.
Revised: Then we process the corpus using the topic-image transformation method demonstrated above and obtain the topic-image lookup table.

Original: For example, the word dog is associated with 1,512 images.
Revised: For example, the word dog is associated with 1,512 images.

Original: For an input source sentence, we obtain the topic words (in boldface) using the same preprocessing.
Revised: For an input source sentence, we obtain the topic words (in boldface) using the same preprocessing.

Original: Then we retrieve the corresponding images from[Figure] the lookup table for each topic word.
Revised: Then we retrieve the corresponding images from the lookup table for each topic word.

Original: Now we have a list of images, and some images appear multiple times as they have multiple topics (like the boxed image in Figure 1).
Revised: Now we have a list of images, and some images appear multiple times as they have multiple topics (like the boxed image in Figure 1).

Original: So we sort the retrieved image list by the count of occurrence to pick out the top- m images that cover the most topics of the sentence.
Revised: So we sort the retrieved image list by the count of occurrence to pick out the top- m images that cover the most topics of the sentence.

Original:  At test time, the process of getting images is done using the image lookup table built by the training set, so we do not need to use the images from the dev and test sets in Multi30K dataset 5 .
Revised: [Figure] At test time, the process of getting images is done using the image lookup table built by the training set, so we do not need to use the images from the dev and test sets in Multi30K dataset 5 .

Original: Intuitively, we do not strictly require the manual alignment of the word (or concept) and image, but rely on the co-occurrence of topic word and image, which is simpler and more general.
Revised: Intuitively, we do not strictly require the manual alignment of the word (or concept) and image, but rely on the co-occurrence of topic word and image, which is simpler and more general.

Original: In this way, we call our method as universal visual retrieval.
Revised: In this way, we call our method as universal visual retrieval.

Original:  NMT WITH U NIVERSAL V ISUAL R EPRESENTATION
Revised:  NMT WITH U NIVERSAL V ISUAL R EPRESENTATION

Original: In this section, we introduce the proposed universal visual representation (VR) method for NMT.
Revised: In this section, we introduce the proposed universal visual representation (VR) method for NMT.

Original: The overview of the framework of our proposed method is shown in Figure 2.
Revised: The overview of the framework of our proposed method is shown in Figure 2.

Original: 4. S OURCE R EPRESENTATION FOR N EURAL M ACHINE T RANSLATION
Revised: 4. S OURCE R EPRESENTATION FOR N EURAL M ACHINE T RANSLATION

Original: In the state-of-the-art Transformer-based NMT (Vaswani et al., 2017), source information is encoded as source representation by an SAN-based encoder with multiple layers.
Revised: In the state-of-the-art Transformer-based NMT (Vaswani et al., 2017), source information is encoded as source representation by an SAN-based encoder with multiple layers.

Original: Speciﬁcally, the encoder is composed of a stack of L identical layers, each of which includes two sub-layers.
Revised: Speciﬁcally, the encoder is composed of a stack of L identical layers, each of which includes two sub-layers.

Original: The ﬁrst sublayer is a self-attention module, whereas the second is a position-wise fully connected feed-forward network.
Revised: The ﬁrst sublayer is a self-attention module, whereas the second is a position-wise fully connected feed-forward network.

Original: A residual connection (He et al., 2016) is applied between the two sub-layers, and then a layer normalization (Ba et al., 2016) is performed.
Revised: A residual connection (He et al., 2016) is applied between the two sub-layers, and then a layer normalization (Ba et al., 2016) is performed.

Original: Formally, the stack of learning the source representation is organized as follows:
Revised: Formally, the stack of learning the source representation is organized as follows:

Original: [Equation]
Revised: [Equation]

Original: where ATT l ( · ) , LN ( · ) , and FFN l ( · ) are the self-attention module, layer normalization, and the feedforward network for the l -th identical layer, respectively.
Revised: where ATT l ( · ) , LN ( · ) , and FFN l ( · ) are the self-attention module, layer normalization, and the feedforward network for the l -th identical layer, respectively.

Original: { Q l − 1 , K l − 1 , V l − 1 } are query, key, and value vectors that are transformed from the ( l -1)-th layer H l − 1 .
Revised: { Q l − 1 , K l − 1 , V l − 1 } are query, key, and value vectors that are transformed from the ( l -1)-th layer H l − 1 .

Original: For example, { Q 0 , K 0 , V 0 } are packed from the summation H 0 of the positional embeddings and word embeddings.
Revised: For example, { Q 0 , K 0 , V 0 } are packed from the summation H 0 of the positional embeddings and word embeddings.

Original: Finally, the output of the stack of L identical layers H L is the ﬁnal source sentence representation.
Revised: Finally, the output of the stack of L identical layers H L is the ﬁnal source sentence representation.

Revised: 4. A GGREGATION FOR T EXT AND I MAGE R EPRESENTATIONS

Revised: After retrieval as described in Section 3, each original sentence X = { x 1 , x 2 , . . .

Revised: , x I } is paired with m images E = { e 1 , e 2 , . . .

Revised: , e m } retrieved from the topic-image lookup table Q .

Revised: First, the

Original: [Figure]
Revised: [Figure]

Original: ec od 
Revised: ec od e

Original: [Figure]
Revised: [Figure]

Original: Aggregation
Revised: Aggregation

Original: Feed Forward
Revised: Feed Forward

Original: Add & Norm
Revised: Add & Norm

Original: Multi-headAttention
Revised: Multi-headAttention

Original: Add & Norm
Revised: Add & Norm

Original: Masked Multi-head Attention
Revised: Masked Multi-head Attention

Original: Add & Norm
Revised: Add & Norm

Original: Linear & Softmax
Revised: Linear & Softmax

Original: Output Probabilities
Revised: Output Probabilities

Original: [Figure]
Revised: Feed Forward

Original: Add & Norm
Revised: Add & Norm

Original: Multi
Revised: Multi-headAttention

Original: [Figure]
Revised: Add & Norm

Revised: Positional Encoding

Original: L×
Revised: L×

Revised: L×

Original: [Figure]
Revised: [Figure]

Revised: Encoding

Original: topic
Revised: Target

Original: [Figure] image lookup table
Revised: Output Embedding topic-image lookup table

Original: Gate & Add
Revised: Gate & Add

Original: [Figure]
Revised: [Figure]

Original: 1×
Revised: 1×

Original: M M
Revised: M M

Original: H

Original: H
Revised: H

Original: 4. A GGREGATION FOR T EXT AND I MAGE R EPRESENTATIONS

Original: After retrieval as described in Section 3, each original sentence X = { x 1 , x 2 , . . .

Original: , x I } is paired with m images E = { e 1 , e 2 , . . .

Original: , e m } retrieved from the topic-image lookup table Q .

Original: First, the source sentence X
Revised: H source sentence X

Original: = { x 1 , x 2 , . . .
Revised: = { x 1 , x 2 , . . .

Original: , x I } is fed into the encoder (Eq.2) to learn the source sentence representation H L .
Revised: , x I } is fed into the encoder (Eq.2) to learn the source sentence representation H L .

Original: Second, the images E = { e 1 , e 2 , . . .
Revised: Second, the images E = { e 1 , e 2 , . . .

Original: , e m } are the inputs to a pre-trained ResNet (He et al., 2016) followed by a feed forward layer to learn the source image representation M ∈
Revised: , e m } are the inputs to a pre-trained ResNet (He et al., 2016) followed by a feed forward layer to learn the source image representation M ∈

Original: R m × 2048 .
Revised: R m × 2048 .

Original: Then, we apply a self-attention mechanism to append the image representation to the text representation:
Revised: Then, we apply a self-attention mechanism to append the image representation to the text representation:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where { K M , V M } are packed from the learned source image representation M .
Revised: where { K M , V M } are packed from the learned source image representation M .

Original: Intuitively, NMT aims to produce a target word sequence with the same meaning as the source sentence rather than a group of images.
Revised: Intuitively, NMT aims to produce a target word sequence with the same meaning as the source sentence rather than a group of images.

Original: In other words, the image information may play an auxiliary effect during the translation prediction.
Revised: In other words, the image information may play an auxiliary effect during the translation prediction.

Original: Therefore, we compute λ ∈ [0 , 1] to weight the expected importance of source image representation for each source word:
Revised: Therefore, we compute λ ∈ [0 , 1] to weight the expected importance of source image representation for each source word:

Original: [Equation]
Revised: [Equation]

Original: where W λ and U λ are model parameters.
Revised: where W λ and U λ are model parameters.

Original: We then fuse H L and H to learn an effective source representation:
Revised: We then fuse H L and H to learn an effective source representation:

Original: [Equation]
Revised: [Equation]

Original: Finally, H is fed to the decoder to learn a dependent-time context vector for predicting target translation.
Revised: Finally, H is fed to the decoder to learn a dependent-time context vector for predicting target translation.

Original: Note that there is a single aggregation layer to fuse image and text information.
Revised: Note that there is a single aggregation layer to fuse image and text information.

Original:  E XPERIMENTS 5. D ATA
Revised:  E XPERIMENTS 5. D ATA

Original: The proposed method was evaluated on four widely-used translation datasets, including WMT’ English-to-Romanian (EN-RO), WMT’14 English-to-German (EN-DE), WMT’14 English-toFrench (EN-DE), and Multi30K which are standard corpora for NMT and MMT evaluation.
Revised: The proposed method was evaluated on four widely-used translation datasets, including WMT’ English-to-Romanian (EN-RO), WMT’14 English-to-German (EN-DE), WMT’14 English-toFrench (EN-DE), and Multi30K which are standard corpora for NMT and MMT evaluation.

Original: 1) For the EN-RO task, we experimented with the ofﬁcially provided parallel corpus: Europarl v and SETIMES2 from WMT’16 with 0.6M sentence pairs.
Revised: For the EN-RO task, we experimented with the ofﬁcially provided parallel corpus: Europarl v and SETIMES2 from WMT’16 with 0.6M sentence pairs.

Original: We used newsdev2016 as the dev set and newstest2016 as the test set.
Revised: We used newsdev2016 as the dev set and newstest2016 as the test set.

Original:  For the EN-DE translation task, 4.43M bilingual sentence pairs of the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7.
Revised: 2) For the EN-DE translation task, 4.43M bilingual sentence pairs of the WMT14 dataset were used as training data, including Common Crawl, News Commentary, and Europarl v7.

Original: The newstest and newstest2014 datasets were used as the dev set and test set, respectively.
Revised: The newstest and newstest2014 datasets were used as the dev set and test set, respectively.

Original: For the EN-FR translation task, 36M bilingual sentence pairs from the WMT14 dataset were used as training data.
Revised: For the EN-FR translation task, 36M bilingual sentence pairs from the WMT14 dataset were used as training data.

Original: Newstest12 and newstest13 were combined for validation and newstest14 was used as the test set, following the setting of Gehring et al.
Revised: Newstest12 and newstest13 were combined for validation and newstest14 was used as the test set, following the setting of Gehring et al.

Original: The Multi30K dataset contains 29K English →{ German, French } parallel sentence pairs with visual annotations.
Revised: The Multi30K dataset contains 29K English →{ German, French } parallel sentence pairs with visual annotations.

Original: The 1,014 English →{ German, French } sentence pairs visual annotations are as dev set.
Revised: The 1,014 English →{ German, French } sentence pairs visual annotations are as dev set.

Original: The test sets are test2016 and test2017 with 1,000 pairs for each.
Revised: The test sets are test2016 and test2017 with 1,000 pairs for each.

Original: 5. S YSTEM S ETTING
Revised: S YSTEM S ETTING

Original: Image Retrieval Implementation We used 29,000 sentence-image pairs from Multi30K to build the topic-image lookup table.
Revised: Image Retrieval Implementation We used 29,000 sentence-image pairs from Multi30K to build the topic-image lookup table.

Original: We segmented the sentences using the same BPE vocabulary as that for each source language.
Revised: We segmented the sentences using the same BPE vocabulary as that for each source language.

Original: We selected top-8 ( w = 8 ) high TF-IDF words, and the default number of images m was set 5.
Revised: We selected top-8 ( w = 8 ) high TF-IDF words, and the default number of images m was set 5.

Original: The detailed case study is shown in Section 6.2.
Revised: The detailed case study is shown in Section 6.2.

Original: After preprocessing, we had about 3K topic words, associated with a total of 10K images for retrieval.
Revised: After preprocessing, we had about 3K topic words, associated with a total of 10K images for retrieval.

Original: Image features were extracted from the averaged pooled features of a pre-trained ResNet50 CNN (He et al., 2016).
Revised: Image features were extracted from the averaged pooled features of a pre-trained ResNet50 CNN (He et al., 2016).

Original: This led to feature maps V ∈ R 2048 .
Revised: This led to feature maps V ∈ R 2048 .

Original: Baseline
Revised: Baseline

Original: Our baseline was text-only Transformer (Vaswani et al., 2017).
Revised: Our baseline was text-only Transformer (Vaswani et al., 2017).

Original: We used six layers for the encoder and the decoder.
Revised: We used six layers for the encoder and the decoder.

Original: The number of dimensions of all input and output layers was set toand 1024 for base and big models.
Revised: The number of dimensions of all input and output layers was set toand 1024 for base and big models.

Original: The inner feed-forward neural network layer was set to 2048.
Revised: The inner feed-forward neural network layer was set to 2048.

Original: The heads of all multi-head modules were set to eight in both encoder and decoder layers.
Revised: The heads of all multi-head modules were set to eight in both encoder and decoder layers.

Original: For Multi30K dataset, we further evaluated a multimodal baseline (denoted as MMT) where each source sentence was paired with an original image.
Revised: For Multi30K dataset, we further evaluated a multimodal baseline (denoted as MMT) where each source sentence was paired with an original image.

Original: The other settings were the same as our proposed model.
Revised: The other settings were the same as our proposed model.

Original: Model Implementation
Revised: Model Implementation

Original: The byte pair encoding algorithm was adopted, and the size of the vocabulary was set to 40,000.
Revised: The byte pair encoding algorithm was adopted, and the size of the vocabulary was set to 40,000.

Original: In each training batch, a set of sentence pairs contained approximately 4096 × 4 source tokens and 4096 × 4 target tokens.
Revised: In each training batch, a set of sentence pairs contained approximately 4096 × 4 source tokens and 4096 × 4 target tokens.

Original: During training, the value of label smoothing was set to 0.1, and the attention dropout and residual dropout were p = 0.1.
Revised: During training, the value of label smoothing was set to 0.1, and the attention dropout and residual dropout were p = 0.1.

Original: The Adam optimizer (Kingma & Ba, 2014) was used to tune the parameters of the model.
Revised: The Adam optimizer (Kingma & Ba, 2014) was used to tune the parameters of the model.

Original: The learning rate was varied under a warm-up strategy with 8,000 steps.
Revised: The learning rate was varied under a warm-up strategy with 8,000 steps.

Original: For evaluation, we validated the model with an interval of 1, batches on the dev set.
Revised: For evaluation, we validated the model with an interval of 1, batches on the dev set.

Original: For the Multi30K dataset, we trained the model up to 10,000 steps, and the training was early-stopped if dev set BLEU score did not improve for ten epochs.
Revised: For the Multi30K dataset, we trained the model up to 10,000 steps, and the training was early-stopped if dev set BLEU score did not improve for ten epochs.

Original: For the EN-DE,
Revised: For the EN-DE,

Original: EN-RO, and EN-FR tasks, following the training of 200,000 batches, the model with the highest
Revised: EN-RO, and EN-FR tasks, following the training of 200,000 batches, the model with the highest

Original: BLEU score of the dev set was selected to evaluate the test sets.
Revised: BLEU score of the dev set was selected to evaluate the test sets.

Original: During the decoding, the beam size was set to ﬁve.
Revised: During the decoding, the beam size was set to ﬁve.

Original: All models were trained and evaluated on a single V100 GPU.
Revised: All models were trained and evaluated on a single V100 GPU.

Original: Multi-bleu.perl 6 was used to compute case-sensitive 4 -gram BLEU scores for all test sets.
Revised: Multi-bleu.perl 6 was used to compute case-sensitive 4 -gram BLEU scores for all test sets.

Original: The signtest (Collins et al., 2005) is a standard statistical-signiﬁcance test.
Revised: The signtest (Collins et al., 2005) is a standard statistical-signiﬁcance test.

Original: In addition, we followed the model conﬁgurations of Vaswani et al.
Revised: In addition, we followed the model conﬁgurations of Vaswani et al.

Original: (2017) to train Big models for WMT EN-RO, EN-DE, and EN-FR translation tasks.
Revised: (2017) to train Big models for WMT EN-RO, EN-DE, and EN-FR translation tasks.

Original: All experiments were conducted with fairseq 7 (Ott et al., 2019).
Revised: All experiments were conducted with fairseq 7 (Ott et al., 2019).

Original: The analysis in Section 6 is conducted on base models.
Revised: The analysis in Section 6 is conducted on base models.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Table 1 shows the translation results for the WMT’14 EN-DE, EN-FR, and WMT’16 EN-RO textonly translation task.
Revised: Table 1 shows the translation results for the WMT’14 EN-DE, EN-FR, and WMT’16 EN-RO textonly translation task.

Original: Our implemented Transformer (base/big) models showed similar BLEU scores with the original Transformer (Vaswani et al., 2017), ensuring that the proposed method can be evaluated over strong baseline NMT systems.
Revised: Our implemented Transformer (base/big) models showed similar BLEU scores with the original Transformer (Vaswani et al., 2017), ensuring that the proposed method can be evaluated over strong baseline NMT systems.

Original: As seen, the proposed +VR signiﬁcantly outperformed 
Revised: As seen, the proposed +VR signiﬁcantly outperformed the baseline Transformer (base), demonstrating the effectiveness of modeling visual information for text-only NMT. In particular, the effectiveness was adapted to the translation tasks of the three language pairs which have different scales of training data, verifying that the proposed approach is a universal method for improving the translation performance. [Table] et al.

Original: [Table]
Revised: [Table]

Revised: Lee et al.

Original: [Table]
Revised: [Table]

Original: This work
Revised: This work

Original: [Table]
Revised: [Table]

Original: the baseline Transformer (base), demonstrating the effectiveness of modeling visual information for text-only NMT.

Original: In particular, the effectiveness was adapted to the translation tasks of the three language pairs which have different scales of training data, verifying that the proposed approach is a universal method for improving the translation performance.

Original: Our method introduced only 1.5M and 4.0M parameters for base and big transformers, respectively.
Revised: Our method introduced only 1.5M and 4.0M parameters for base and big transformers, respectively.

Original: The number is less than 3% of the baseline parameters as we used the ﬁxed image embeddings from the pre-trained ResNet feature extractor.
Revised: The number is less than 3% of the baseline parameters as we used the ﬁxed image embeddings from the pre-trained ResNet feature extractor.

Original: Besides, the training speed (steps/second) was basically the same as the baseline model 
Revised: Besides, the training time was basically the same as the baseline model (Appendix A.1)

Original: [Table]
Revised: [Table]

Original: In addition, the proposed method was also evaluated for MMT on the multimodel dataset, Multi30K .
Revised: In addition, the proposed method was also evaluated for MMT on the multimodel dataset, Multi30K .

Original: Results in Table 2 show that our model also outperformed the transformer baseline.
Revised: Results in Table 2 show that our model also outperformed the transformer baseline.

Original: Compared with the results in text-only NMT, we ﬁnd that the image presentation gave marginal contribution, which was consistent with the ﬁndings in previous work (Zhang et al., 2017; Gr¨onroos et al., 2018;
Revised: Compared with the results in text-only NMT, we ﬁnd that the image presentation gave marginal contribution, which was consistent with the ﬁndings in previous work (Zhang et al., 2017; Gr¨onroos et al., 2018;

Original: Caglayan et al., 2019).
Revised: Caglayan et al., 2019).

Original: The most plausible reason might be that the sentences in Multi30K are so simple, short, and repetitive that the source text is sufﬁcient to perform the translation (Caglayan et al., 2019; Ive et al., 2019).
Revised: The most plausible reason might be that the sentences in Multi30K are so simple, short, and repetitive that the source text is sufﬁcient to perform the translation (Caglayan et al., 2019; Ive et al., 2019).

Original: This veriﬁes our assumption of the current bottleneck of MMT due to the limitation of Multi30K and shows the necessity of our new setting of transferring multimodality into more standard and mature text-only NMT tasks. 
Revised: This veriﬁes our assumption of the current bottleneck of MMT due to the limitation of Multi30K and shows the necessity of our new setting of transferring multimodality into more standard and mature text-only NMT tasks. [Table] [Figure] [Figure]

Original:  A NALYSIS 6. T HE INFLUENCE OF GATING WEIGHT λ
Revised:  A NALYSIS 6. T HE CONTRIBUTION

Revised: THE LOOKUP TABLE

Original: In our model, the weight λ of the gated aggregation method was learned automatically to measure the importance of the visual information. To investigate its inﬂuence of the overall model performance,
Revised: The major contribution would be more effective sentence contextualized encoding for better representation from the visual clue combination instead of single image enhancement for encoding each individual sentence or word 8 .

Original: [Figure] [Figure] we compared by manually setting the weight λ into scalar values in { 0.1, 0.3, 0.5, 0.7, 0.9 } for experiments on  the EN-RO test set.
Revised: To verify the hypotheses, we conduct the following ablations: we replace the ResNet50 feature extractor with 1) ResNet101 ; 2) ResNet152 ; 3) Caption : that adopts a standard image captioning model (Xu et al., 2015b) 9 with the updated ResNet101 encoder trained on COCO dataset to encode the images;

Original: Figure 4 shows that all models withmanual λ outperformed the baseline Trans. (base), indicating theeffectiveness of image information. In contrast, they were inferior to the performance of our model.
Revised: In addition, we compare with the operations: 4) Shufﬂe : shufﬂe the image features but keep the lookup table; 5) Random Init : randomly initialize the image embedding but keep the lookup table; 6) Random Mapping : randomly retrieve unrelated images. Results are shown in Table 3.

Original: This means that the degree of dependency for image information varies for each source sentence, indicating the necessity ofautomatically learning the gating weights of image representations.
Revised: The BLEU scores of 1-5 are close to the proposed VR and outperform the baseline, which shows that the content of images would not be very important. Ablation 6) gives very pool result, which veriﬁes the necessity of the mapping especially the topic consistency.

Original: 6. I NFLUENCE OF THE NUMBER OF IMAGES
Revised: 6. I NFLUENCE OF THE NUMBER OF IMAGES

Original: To evaluate the inﬂuence of the number of  images m , we constrained m in { 0, 1, 3, 5, 7, 9, 15, 20, 30 } for experiments on the EN-RO test set, as shown in Figure 5.
Revised: To evaluate the inﬂuence of the number of paired images m , we constrained m in { 0, 1, 3, 5, 7, 9, 15, 20, 30 } for experiments on the EN-RO test set, as shown in Figure 3.

Original: When m = 0 , the model is the baseline NMT model, whose BLEU score was lower than all the models with images.
Revised: When m = 0 , the model is the baseline NMT model, whose BLEU score was lower than all the models with images.

Original: As the number of images increases, the BLEU score also increased at the beginning (from 32 . 66 to 33 . 78 ) and then slightly decreased when m exceeds 5.
Revised: As the number of images increases, the BLEU score also increased at the beginning (from 32 . 66 to 33 . 78 ) and then slightly decreased when m exceeds 5.

Revised: The reason might be that too many images for a sentence would have greater chance of noise.

Original: Therefore, we set m = 5 in our models.
Revised: Therefore, we set m = 5 in our models.

Original: [Table]
Revised: The number of sentence-image pairs to create the lookup table could also make effects.

Original: 6. V ISUAL S ENSITIVITY
Revised: We randomly split the pairs of Multi30K into the proportion in [0.1, 0.3, 0.5, 0.7, 0.9], the corresponding BLEU scores for EN-RO are [33.07, 33.44, 34.01, 34.06, 33.80].

Original: We further probed the visual sensitivity by deliberately feeding features from unrelated images to understand how sensitive our model is to visual modalities.
Revised: Furthermore, we also evaluate the performance by adding external sentence-pairs from the training set of MS COCO image caption dataset (Lin et al., 2014).

Original: Speciﬁcally, we fed ﬁve random noisy images for each source sentence to break the topic association of the sentence andimages. Tableshows the results.
Revised: The BLEU scores are 33.55 and 33.71 respectively for COCO only and

Original: In all likelihood, our model would deteriorate in performance when integrating with noisy images.
Revised: Multi30K+COCO.

Original: This result further veriﬁes the effectiveness of our image retrieval method, by which the model can beneﬁt from the assistance of relevant images with similar topics.
Revised: These results indicate that a modest number of pairs would be beneﬁcial.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work presents a universal visual representation method for neural machine translation relying on monolingual with image annotations, which breaks the restraint of heavy dependence on bilingual sentence-image pairs in the current multimodel NMT setting.
Revised: This work presents a universal visual representation method for neural machine translation relying on monolingual with image annotations, which breaks the restraint of heavy dependence on bilingual sentence-image pairs in the current multimodel NMT setting.

Original: In particular, this method enables visual information to be applied to large-scale text-only NMT through a topic-image lookup.
Revised: In particular, this method enables visual information to be applied to large-scale text-only NMT through a topic-image lookup.

Original: We hope this work sheds some light for future MMT research.
Revised: We hope this work sheds some light for future MMT research.

Original: In the future, we will try to adopt the proposed method to other tasks.
Revised: In the future, we will try to adopt the proposed method to other tasks.

Original:  A A PPENDIX 
Revised: . A A PPENDIX A. E XTRA C OMPUTATION T IME

Revised: There are mainly two extra computation cost using our method, including obtaining image data for sentences and learning image representations, which are negligible compared with training a NMT model. The time of obtaining image data for MT sentences for EN-RO dataset is approximately less than 1 minute by tensor operation in GPU. The lookup table is formed as the mapping of token (only topic words) index to image id.

Original: [Figure]
Revised: Then, the retrieval method is applied as the tensor indexing from the sentence token (only topic words) index to image ids, which is the same as the procedure of word embedding. The retrieved image ids are then sorted by frequency. Learning image representations takes only about 2 minutes for all the 29,000 images in Multi30K using 6G GPU memory for feature extraction and 8 threads of CPU for transforming images. The extracted features are formed as the “image embedding layer” with the size of (29000, 2400) for quick accessing in neural network.

Revised: A. W HY DOES THE LOOKUP TABLE WORK

Revised: The major contribution would be the more effective sentence contextualized encoding for better representation from the visual clue combination instead of single image enhancement for encoding each individual sentence or word.

Revised: According to Distributional Hypothesis (Harris, 1954) which states that “words that occur in similar contexts tend to have similar meanings”, we are inspired to extend the concept in multimodal world, “the sentences with similar meanings would be likely to pair with similar even the same images”, where the consistent images (with similar topic) could play the role of topic or type hints for similar sentence modeling.

Revised: This is also very similar to the idea of word embedding by taking each image as a “word”.

Revised: Because we use the average pooled output of ResNet, each image is represented as 2400d vector.

Revised: For all the 29,000 images, we have an embedding layer with size (29000, 2400).

Revised: The “content” of the image is just like the embedding initialization.

Revised: It indeed makes effects, but the capacity of the neural network

Revised: [Figure]

Revised: is not up to it.

Revised: In contrast, the mapping from text word to the index in the word embedding is critical.

Revised: Similarly, the mapping of sentence to image in image embedding would be essential, i.e., the similar sentences (with the same topic words) tend to map the same similar image.

Revised: To verify the hypotheses, we conduct the following ablations: we replace the ResNet50 feature extractor with 1) ResNet101 ; 2) ResNet152 ; 3) Caption : that adopts a standard image captioning model (Xu et al., 2015b) (with the updated ResNet101 encoder) trained on COCO dataset to encode the images; In addition, we compare with the operations: 4) Shufﬂe : shufﬂe the image features but keep the lookup table; 5) Random Init : randomly initialize the image embedding but keep the lookup table; 6) Random Mapping : randomly retrieve unrelated images.

Revised: Results are shown in Table 3.

Revised: The

Revised: BLEU scores of 1-5 are close to the proposed VR and outperform the baseline, which shows that the content of images would not be very important.

Revised: Ablation 6) gives very pool result, which veriﬁes the necessity of the mapping especially the topic consistency.

Revised: We believe this ﬁnding would be suggestive for the future research since most previous work focused on the content of the image itself.

Revised: As a different research line, we highlight the consistency among the mono-modality to bridge the gap of language and image modeling.

Revised: A. T HE INFLUENCE OF GATING WEIGHT λ

Revised: In our model, the weight λ of the gated aggregation method was learned automatically to measure the importance of the visual information.

Revised: We compared by manually setting the weight λ into scalar values in { 0.1, 0.3, 0.5, 0.7, 0.9 } for experiments on the EN-RO test set.

Revised: Figure 4 shows that all models with manual λ outperformed the baseline Trans.

Revised: (base), indicating the effectiveness of image information.

Revised: In contrast, they were inferior to the performance of our model.

Revised: This means that the degree of dependency for image information varies for each source sentence, indicating the necessity of automatically learning the gating weights of image representations.

Revised: A. E XAMPLES OF RETRIEVED IMAGES

Revised: [Figure]

Revised: by a silver vehicle an elderly woman pan frying food in a kitchen small boy carries a soccer ball on a field

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: (WMT)

Revised: The old system of private arbitration courts is off the table

Revised: This issue is shaping as a potential early rift with the business community

Revised: He said he then heard his friend , Hamza calling to him

Revised: The red flag has been risen

Revised: The character attempts to pass human smugglers and then border police on his way to a refugee centre in the European Union .