O N THE R ELATIONSHIP BETWEEN N EURAL M ACHINE T RANSLATION AND W ORD A LIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Prior researches suggest that attentional neural machine translation (NMT) is able to capture word alignment by attention, however, to our surprise, it almost fails for NMT models with multiple attentional layers except for those with a single layer.
This paper introduce two methods to induce word alignment from general neural machine translation models.
Experiments verify that both methods obtain much better word alignment than the method by attention.
Furthermore, based on one of the proposed method, we design a criterion to divide target words into two categories (i e. those mostly c ontributed f rom s ource “CFS” words and the other words mostly c ontributed f rom t arget “CFT” words) and analyze word alignment under these two categories in depth.
We ﬁnd that although NMT models are difﬁcult to capture word alignment for CFT words but these words do not sacriﬁce translation quality signiﬁcantly, which provides an explanation why NMT is more successful for translation yet worse for word alignment compared to statistical machine translation.
We further demonstrate that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment and translation for several NMT systems
I NTRODUCTION
Machine translation aims at modeling the semantic equivalence between a pair of source and target sentences (Koehn, 2009), and word alignment tries to model the semantic equivalence between a pair of source and target words (Och & Ney, 2003).
As a sentence consists of words, word alignment is conceptually related to machine translation and such a relation can be traced back to the birth of statistical machine translation (SMT)
(Brown et al., 1993), where word alignment is the basis of
SMT models and its accuracy is generally helpful to improve translation quality (Koehn et al., 2003;
Liu et al., 2005).
In neural machine translation (NMT), it is also important to study word alignment, because word alignment provides potential ways to understanding black-box NMT models and analyzing their translation errors (Ding et al., 2017).
Unlike SMT, NMT models do not directly depend on word alignment and prior researches implicitly extract word alignment by attention (Bahdanau et al., 2014; Mi et al., 2016;
Liu et al., 2016).
Unfortunately, this method is not general for all NMT models.
In particular, to our surprise, it can only obtain word alignment for NMT models with a single attentional layer, but fails for those with multiple attentional layers, which is the standard for state-of-the-art NMT systems such as T RANSFORMER (Vaswani et al., 2017).
In this paper, we propose two methods to induce word alignment from general NMT models and answer a fundamental question how much word alignment NMT models can learn.
The ﬁrst method explicitly builds a word alignment model between a pair of source and target word representations encoded by NMT models, and then it learns additional parameters for this word alignment model with the supervision from external aligners similar to (Mi et al., 2016) and (Liu et al., 2016).
The second method is more intuitive and ﬂexible: it is parameter-free and thus does not need retraining and external aligners.
Its key idea is to measure the prediction difference of a target word if a source word is removed, inspired by (Arras et al., 2016; Zintgraf et al., 2017).
Unlike the ﬁrst method, the second one only depends on NMT models, and it thereby facilitates better understanding and interpreting NMT models.
Experiments on an advanced NMT model show that both methods achieve much better word alignment than the method by attention.
However, word alignment obtained by our methods is still worse than statistical word aligners (Dyer et al., 2013; Och & Ney, 2003).
This raises another natural question why advanced NMT models including less word alignment knowledge but deliver better translations than SMT models based on statistical aligners, which was observed in prior researches yet without deep interpretation (Tu et al., 2016; Liu et al., 2016).
To answer this question, in the spirit of the proposed prediction difference (PD) method, we design a criterion to divide target words in the reference of a test sentence into two categories: those target words mostly contributed from source (CFS) words and the other target words mostly contributed from target (CFT) words in their histories.
1 Our experiments further demonstrate that NMT models capture good word alignment for the ﬁrst category (CFS) words with accuracy comparable to a statistical aligner, while their word alignment for the second category (CFT) words is much worse.
Thankfully, most target words in the second category (CFT) can be easily decoded by NMT models because of the strong language model effects implicitly learned by
NMT models.
Finally, we exploit the correlation between translation quality and word alignment quality for neural machine translation.
We ﬁnd that there is a weak correlation between translation and word alignment for all target words, yet a strong correlation between translation and word alignment for those target words mostly contributed from source words, after analyzing translations from several NMT systems.
This ﬁnding suggests that word alignment errors for CFS words are responsible for translation errors in some extent.
This paper makes the following contributions:
• It proposes two better methods to acquire word alignment from general NMT models.
• It explains why NMT models deliver excellent translations no matter their worse word alignment compared to statistical machine translation.
• It empirically shows a strong correlation between translation quality and word alignment accuracy for those target words mostly contributed from source words.
P RELIMINARIES 2.
N
[Equation]
Given a source sentence x = (cid:104) x 1 , · · · , x | x | (cid:105) and a target sentence y = (cid:104) y 1 , · · · , y | y | (cid:105) , NMT aims at maximizing the following conditional probabilities:
[Equation]
where y <
i = (cid:104) y 1 , . . .
, y i − 1 (cid:105) denotes a preﬁx of y with length i − 1 , and s Li is the decoding state of y i .
Generally, the conditional distribution P (cid:0) y
i | s L i (cid:1) is somehow modeled within an encoderdecoder framework.
In encoding stage, the source sentence x are encoded as a sequence of hidden vectors h by an encoder according to speciﬁc NMT models, such as a multi-layer encoder consisting of recurrent neural network (RNN), convolutional neural network (CNN), or self-attention layer.
In decoding stage, each decoding state s Li is computed by an L -layer decoder as follows:
[Equation]
where y i is the word embedding of word y i , f is a general function dependent on a speciﬁc NMT model, c li is a context vector in l th layer, computed from h and s l<i according to different NMT models.
As the dominant models, attentional NMT models deﬁne the context vector c li as a weighted sum of h , where the weight α l i = g (cid:0) s l − 1 i , s l <i , h (cid:1) is deﬁned by a similarity function g .
Due to the space limitation, we refer readers to (Bahdanau et al., 2014), (Gehring et al., 2017) and (Vaswani et al., 2017) for the details on the deﬁnitions of f and g .
2. A LIGNMENT BY A TTENTION
Since the attention weight α li,j measures the similarity between s l − 1 i and h j , it has been widely used to evaluate the word alignment between y i and x j (Bahdanau et al., 2014; Ghader & Monz, 2017).
Once an attentional NMT model has been trained, one can easily extract word alignment A from the attention weight α according to the style of maximum a posterior strategy (MAP) as follows:
[Equation]
where A i,j = 1 indicates y i aligns to x j .
For NMT models with multiple attentional layers or multiple head attentional layers as in (Vaswani et al., 2017), we sum all attention weights with respect to all layers and heads to a single α before MAP in equation 3.
M ETHODS TO I NDUCING W ORD A LIGNMENT
Although attention might obtain some word alignment as described in previous section, it is unknown whether NMT models contain more word alignment information than that obtained by attention.
In addition, the method using attention is useful to induce word alignment for attentional
NMT models, (more precisely, those models including a single attentional layer as shown in our experiments), whereas it is useless for general NMT models.
In this section, in order to induce word alignment from general NMT models, we propose two different methods, which are agnostic to speciﬁc NMT models.
3. A LIGNMENT BY E XPLICIT A LIGNMENT M ODEL
Given a source sentence x , a target sentence y , following (Liu et al., 2005)
and(Taskar et al., 2005), we explicitly deﬁne a word alignment model as follows:
[Equation]
where δ ( x j , y i , x , y ; W ) is a distance function parametrized by W .
Ideally, δ is able to include arbitrary features such as IBM model 1 similar to (Liu et al., 2005).
However, as our goal is not to achieve the best word alignment but to focus on that captured by an NMT model, we only consider these features completely learned in NMT.
Hence, we deﬁne the
[Equation]
where x j and y i are word embeddings of x j and y i learned in NMT, h j is the hidden unit of x j in the encoding network and s Li is the hidden unit of y j in the decoding network, (cid:107) denotes the concatenation of a pair of column vectors of dimension d , and W is a matrix of dimension
2 d × 2 d .
The explicit word alignment model is trained by maximizing the objective function with respect to
W :
[Equation]
where A ref ij is the reference alignment between x j and y i for a sentence pair x and y .
As the number of elements in W is up to one million (i.e., (2 × 512) 2 ), it is not feasible to train it using a small dataset with gold alignment.
Therefore, following (Mi et al., 2016; Liu et al., 2016), we run statistical word aligner such as F AST A LIGN (Dyer et al., 2013) on a large corpus and then employ resulting word alignment as the silver alignment A ref for training.
Note that our goal is to quantify word alignment learned by an NMT model, and thus we only treat W as the parameter to be learned, which differs from the joint training all parameters including those from NMT models as in (Mi et al., 2016; Liu et al., 2016).
After training, one obtains the optimized W and then easily infers word alignment for a test sentence pair (cid:104) x , y (cid:105) via the MAP strategy as deﬁned in equation 3 by setting α i,j (cid:48)
=
P ( x j (cid:48)
| y
i , y , x ; W ) .
Note that if word embeddings and hidden units learned by NMT models capture enough information for word alignment, the above method can obtain excellent word alignment.
However, because the dataset for supervision in training deﬁnitely include some data intrinsic word alignment information, it is unclear how much word alignment is only from NMT models.
Therefore, we propose the other method which is parameter-free and only dependent on NMT models themselves.
3. A LIGNMENT BY P REDICTION D IFFERENCE
The intuition to this method is that if y i aligns to x j , the relevance between y i and x j should be much higher than that between y i and any other x k with k (cid:54) =
j .
Therefore, the key to our method is that how to measure the relevance between y i and x j .
Sampling method Zintgraf et al.
(2017) propose a principled method to measure the relevance between a pair of tokens in input and output.
It is estimated by measuring how the prediction of y i in the output changes if the input token x j is unknown.
Formally, the relevance between y i and x j for a given sentence pair (cid:104) x , y (cid:105) is deﬁned as follows:
[Equation]
(7) with
[Equation]
8) where x ( j, x ) =
(cid:104) x 1 , . . .
, x j − 1 , x , x j +1 , . . .
, x | x | (cid:105) by the empirical distribution P ( x ) , which can be considered as the 1-gram language model for the source side of the training corpus.
Unlike a computer vision task in (Zintgraf et al., 2017), the size of source vocabulary in NMT is up to 30000 and thus summation over this large vocabulary is challenging in computational efﬁciency.
As a result, we only sample multiple words to approximate the expectation in equation 8 by Monte Carlo (MC) approach.
Deterministic method Inspired by the idea of dropout (Srivastava et al., 2014), we measure the relevance by disabling the connection between x j and the encoder network in a deterministic way.
Formally, R ( y i , x j , x , y ) is directly deﬁned via dropout effect on x j as follows:
[Equation]
where x ( j, 0 ) denotes the sequence by replacing x j with a word whose embedding is a zero vector.
In this way, the computation in equation 9 is much faster than the Monte Carlo sampling approach involving multiple samples.
It is worth mentioning that equation 9 resembles the Monte Carlo sampling approach with a single sample in calculation, but it is signiﬁcantly better than MC with a single sample in alignment quality and is very close to MC approach with enough samples, as to be shown in our experiments.
Note that R ( y i , x j , x , y ) ∈ [ − 1 , 1] , where R ( y i , x j , x , y )
= 1 means i th target word is totally determined by the j th source word;
R ( y i , x j , x , y )
= − 1 means i th target word and j th source word are mutual exclusive;
R ( y i , x j , x , y )
=
means j th source word do not affect generating i
th target word.
In order to obtain word alignment, after collecting R ( y i , x j , x , y ) for x j , y i , x and y , one can easily infer word alignment via the MAP strategy as deﬁned in equation 3 by setting α i,j (cid:48)
=
R ( y i , x j (cid:48) , x , y ) .
Remark The above R ( y i , x j , x , y ) in equation 7 quantiﬁes the relevance between a target word y i and a source word x j .
Similarly, one can quantify the relevance between y i and its history word y k as follows:
[Equation]
where R o indicates the relevance between two target words y i and y k with k < i , and P ( y i |
y <
i ( k, 0 ) , x ) is obtained by disabling the connection between y k and the decoder network, similar to
P (cid:0) y i | y <i , x ( j, 0 ) (cid:1) .
Unlike R ( y i , x j , x , y ) capturing word alignment information, R o ( y i , y k , x , y ) is able to capture word allocation in a target sentence and it will be used to answer a fundamental question why NMT models yields better translation yet worse word alignment compared with SMT in section of experiments.
E XPERIMENTS
In this section, we will empirically explore the following questions through experiments:
Does attention really capture word alignment for attentional NMT models?
How much word alignment do NMT models learn?
Why NMT models deliver better translation yet worse word alignment than SMT?
What is the relationship between translation quality and word alignment quality?
To answer these questions, we conduct experiments on Chinese-to-English dataset, which includes many reorderings and thereby is challenging for word alignment and translation tasks.
Translation quality is evaluated with case-insensitive 4-gram BLEU (Papineni et al., 2002), implemented by multi-bleu.perl , and alignment performance is evaluated by AER (Mihalcea & Pedersen, 2003;
Koehn, 2009).
In the following experiments and analyzes, both BLEU and AER are shown in percentage.
4. S ETTING
The training data consists of 1.8M sentence pairs from Chinese-to-English task of NIST2008 Open
Machine Translation Campaign with 40.1M Chinese words and 48.3M English words respectively.
The development set is chosen as NIST2002, and the test set is NIST2005.
To make NMT model capable of open-vocabulary translation, all the datasets are pre-processed by Byte Pair Encoding (BPE) (Sennrich et al., 2015) with 32K merging operations.
To measure word alignment quality, we report
AER on NIST05 test set, whose reference alignment was manually annotated by experts (Liu et al., 2016).
The experiments are based on an alignment baseline and three strong translation baselines from NMT and SMT
: • F AST A LIGN (Dyer et al., 2013): a simple, fast, unsupervised word aligner.
• M OSES (Koehn et al., 2007): an open source phrase based translation system with default conﬁguration, whose translation tables are derived from F AST A LIGN .
• N EMATUS
(Sennrich et al., 2017): an open source neural machine translation implementation of RNN based sequence-to-sequence model.
• T RANSFORMER (Vaswani et al., 2017): a novel neural network architecture for language understanding implemented by Zhang et al.
More details on training these systems are described in Appendix A.
4. D OES A TTENTION R EALLY C APTURE A LIGNMENT ?
To our surprise, word alignment by attention is with AER around 83 for T RANSFORMER with six attentional layers: we examined averaged alignment and that from each attentional layer, but any of their alignment is worse.
Since the bilingual corpus intrinsically includes word alignment in some extent, word alignment by attention should be better than the data intrinsic alignment if attention indeed captures alignment.
To obtain the data intrinsic word alignment, we calculate point-wise mutual information (PMI) from the bilingual corpus and then infer word alignment for each bilingual sentence by using
[Table]
AttentioncaptureslesswordalignmentthanPMIforNMTmodelswithmultipleattentionallayers.
the MAP strategy as in equation 3.
4 Referring to Table 1, alignment captured by attention of sixlayer T RANSFORMER (T RANSFORMER -L6) is obviously worse than alignment by PMI.
Hence, it is hard to conclude attention from the standard six-layer T RANSFORMER captures word alignment.
As shown in Table 1, N EMATUS using single attentional layer of RNN is able to capture better word alignment than PMI, so the architecture of multiple attentional layers in T RANSFORMER may be the reason why its attention fails to capture word alignment.
Therefore, we reset T RANSFORMER to include one attentional layer in both encoder and decoder and retrain it on the same dataset.
We ﬁnd that the T RANSFORMER with a single attentional layer (T RANSFORMER -L1) can produce much more reasonable alignment than T RANSFORMER -L6, even if its translation quality substantially decreases.
In this sense, we believe that the hidden units h j and s li in equation 2 represent the information of the entire sequence x and y instead of the information particularly emphasizing the words x j and y i , and thereby attention between h j and s li is not necessary to indicate the word alignment between x j and y i especially for multi-layer attention models.
[Equation]
Previous subsection shows alignment by attention may not be a well-deﬁned method to acquiring alignment particularly for multi-layers models, and thus we need better methods to tell how much alignment NMT models can learn.
In this subsection, we analyze performances of the two proposed methods, which includes an explicit alignment model and prediction difference based on
T RANSFORMER -L6.
[Table]
Explicit Alignment Model
As the explicit model employs the silver alignment dataset from F AST
A LIGN for training its parameters, its ﬁnal AER includes contributions from both the aligned data and the model.
We analyze the alignment performance from the model itself by comparing with
PMI (F AST A LIGN ), which is similar to PMI but calculates co-occurrence of a word pair aligned by
F AST A LIGN in a bilingual sentence.
4 As shown in Table 2, the explicit model outperforms PMI (F AST A LIGN ), which indicates that the six-layer T RANSFORMER indeed learns word alignment information.
In addition, AER by the explicit model is better than attention method over both one layer T RANSFORMER and RNN models.
One might argue that the reason why the explicit model yields excellent alignment is attributed to the supervision from silver alignment dataset.
However, with the same amount of supervision,
T RANSFORMER -L1 generates much worse alignment than T RANSFORMER -L
and it is only comparable to PMI (F AST A LIGN ) in AER, as shown in Table 3.
This suggests that supervision is
[Table]
not enough to obtain good alignment and the hidden units learned by a translation model indeed implicitly capture alignment knowledge.
Prediction Difference As shown in Table 2, prediction difference delivers better word alignment than the data intrinsic word alignment,
i
[Table] e. PMI (F AST A LIGN ) [Table] and this result gives a strong indicator that T RANSFORMER with multiple attentional layers indeed induces reasonable alignment even though the alignment is not captured by its attention.
In addition, prediction difference is able to capture better word alignment than attention in the state of the art RNN model N EMATUS .
However, both explicit model and prediction difference are worse than statistical word aligner F AST A LIGN in terms of AER.
[Table]
Prediction different can be implemented by sampling method or deterministic method.
As shown in Table 4, the alignment performance of sampling method is improving as growing of the sample size, because the accuracy of Monte Carlo approach is dependent on the number of samples.
And no matter what sample size is, the variance of AER is always ignorable.
The reason might be arg max operation in equation 3 eliminate the ﬂuctuation of probability matrix.
Although using large sample size can achieve nice alignment performance, but it will cost too much computation.
Fortunately, the deterministic method can also achieve the nice alignment performance with the same computational cost as sample size equals one in sampling method.
As a result, we employ deterministic version as the default for prediction difference in this paper.
Although explicit model achieves better word alignment result than prediction difference, it is difﬁcult to interpret and understand neural machine translation through word alignment from explicit model.
The main reason is that explicit model relies on an external aligned dataset with guidance from statistical word aligner F AST A LIGN , and thus the characteristic of its alignment result are similar to that of F AST A LIGN , leading to interpretation biased to F AST A LIGN .
On the other hand, prediction difference only relies on prediction from a neural model to deﬁne the relevance, it has been successfully used to understand and interpret a neural model (Zintgraf et al., 2017).
Therefore, in the rest of this section, we try to understand T RANSFORMER by using prediction difference in the rest of this section.
4. D OES A LIGNMENT E RROR A FFECT T RANSLATION ?
It is still unreasonable that T RANSFORMER is good at translation even though messes up with word alignment.
In order to reveal this observation, we ﬁrstly divide the target words into two categories:
• Contributing from source (CFS): the prediction of a target word can be mostly attributed to the appearance of a source side word.
• Contributing from target (CFT): the prediction of a target word can be mostly attributed to the appearance of a target side word.
Speciﬁcally, we employ prediction difference to deﬁne CFS and CFT as follows:
[Equation]
where (cid:15) ∈ [0 , 1) is a probability margin between CFS and CFT words.
[Table]
After dividing the target words into two categories of CFS words and CFT words according to the criterion deﬁned above, 5 we calculate their percentages and ﬁnd that the ratio between CFS and
CFT is about 7:3.
This result suggests that T RANSFORMER employs more information from source side than target side for translation and it is more important for T RANSFORMER to make better use of source side information.
In addition, it is shown in Table 5 that AER of CFS words by prediction difference improves to 33.95 comparable to AER of CFS by F AST A LIGN , even though AER by prediction difference on overall targets is substantially worse than AER by F AST A LIGN .
We further compare word alignment from prediction difference and that from F AST A LIGN according to the CFS words or the
CFT words and the results are shown in Table 5.
We ﬁnd that prediction difference captures good alignment for CFS words as F AST A LIGN does, but its alignment for CFT words is much worse than F AST A LIGN .
Additionally, we evaluate the effects of translation quality on both the CFS words and the CFT words by translation recall, which measures the percentages of target words in references recalled by T RANSFORMER and M OSES after decoding.
As shown in Table 5, although
T RANSFORMER induces worse alignment for the CFT words, it successfully decodes 62% of the
CFT words, which is comparable to the percentages of the CFS words.
These results demonstrate that T RANSFORMER is able to easily translate the CFT words by using target history words thanks to its strong language model effect, even if it involves noise source information due to inaccurate alignment.
Therefore, CFT words might be the reason why T RANSFORMER yields better translation yet worse word alignment compared to SMT models.
4. C ORRELATION BETWEEN T RANSLATION AND A LIGNMENT
We train six translation systems on the same dataset: they share the same model architecture but use encoder-decoder layer number ranging from 1 to 6.
As there is only a single manually aligned dataset (i.e. NIST2005), the result might be highly dependent on the speciﬁc dataset.
Inspired by (Koehn, 2004), we randomly sample 1200 datasets with replacement from NIST2005, each of which includes the same number of the sentences as NIST2005.
We report their BLEU points and AER captured by prediction difference on the 1200 sampled datasets.
Figure 1(a) shows the correlation between translation quality and alignment error rate is only about − 0.45.
This result indicates a weak correlation between translation and alignment.
Because CFT words yield worse alignment but are easy to translate as observed before, we propose to modify AER on CFS words only.
However, since CFS words deﬁned in previous section are dependent on a speciﬁc translation model whereas we employ six translation models, there will be a bias for the correlation between translation and alignment on CFS words for different translation models.
Hence, we redeﬁne CFS words using two PMI statistics based on a pair of source and target words and a pair of a target word and its history target word, 4 by replacing R and R o with these
32.5 35.0 37.5 40.0 42.5 45.0 47.5 50.0 BLEU 39.
A E R (a) Overall ( r = − 0 . 45 ) 32.5 35.0 37.5 40.0 42.5 45.0 47.5 50.0 BLEU
A E R (b) PMI-CFS ( r = − 0 . 82 ) 32.5 35.0 37.5 40.0 42.5 45.0 47.5 50.0 BLEU
A E R (c) PMI-CFT ( r = 0 . 13 ) two PMI statistics in equation 11. It is known that the redeﬁned CFS and CFT are dependent on the training data yet independent on any speciﬁc model. In order to avoid redundant concepts, they are still mentioned as CFS and CFT words as before.
In Figure 1(b), one can see that the modiﬁed AER on CFS words correlates well with BLEU points, and such a correlation is up to − 0.82.
Additionally, as shown in Figure 1(c), the correlation between translation and alignment for CFT words is around zero and even positive.
This indicates that correct alignment of CFT words is not necessary to lead to improved translation quality, because these CFT words are easily ﬁgured out by the strong language model effects encoded in NMT models.
Together with the result shown in Table 5, improving word alignment for CFS words is potential to improve translation quality for neural machine translation.
R ELATED W ORK
In NMT, there are many notable researches which mention word alignment captured by attention in some extent.
For example, (Bahdanau et al., 2014) is the ﬁrst work to show word alignment examples by using attention in an NMT model.
Tu et al.
(2016) quantitatively evaluate word alignment captured by attention and ﬁnd that its quality is much worse than statistical word aligners.
Motived by this ﬁnding, Chen et al.
(2016), Mi et al.
(2016) and Liu et al.
(2016) improve attention with the supervision from silver alignment results obtained by statistical aligners, in the hope that the improved attention leads to better word alignment and translation quality consequently.
Despite the close relation between word alignment and attention, Koehn & Knowles (2017) and Ghader & Monz (2017) discuss the differences between word alignment and attention in NMT.
All these works study word alignment for the same kind of NMT models, i.e. that with a single attentional layer.
One of our contribution is that we propose model-agnostic methods to study word alignment in a general way which deliver better word alignment quality than attention method.
Moreover, for the ﬁrst time, we further explain and quantify the relationship between translation quality and word alignment for an NMT model.
The prediction difference method in this paper actually provides an avenue to understand and interpret neural machine translation models.
Therefore, it is closely related to many works on visualizing and interpreting neural networks (Lei et al., 2016; Bach et al., 2015; Zintgraf et al., 2017).
Indeed, our method is inherited from (Zintgraf et al., 2017), and our advantage is that it is computationally efﬁcient particularly for those tasks with a large vocabulary.
In sequence-to-sequence tasks, Ding et al.
(2017) focus on model interpretability by modeling how inﬂuence propagates across hidden units in networks, which is often too restrictive and challenging to achieve as argued by (AlvarezMelis & Jaakkola, 2017).
Instead, Alvarez-Melis & Jaakkola (2017) concentrate on prediction interpretability with only oracle access to the model generating the prediction.
To achieve this effect, they propose a casual learning framework to measure the relevance between a pair of source and target words.
Our method belongs to the type of prediction interpretability similar to (AlvarezMelis & Jaakkola, 2017), but ours is a uniﬁed and parameter-free method rather than a pipeline and parameter-dependent one.
In addition, both Ding et al.
and Alvarez-Melis & Jaakkola (2017) qualitatively demonstrate interpretability by showing some sentences, while we exhibit the interpretability by quantitatively analyzing all sentences in a test set.
C ONCLUSION AND F UTURE W ORK
This paper points out that attention is insufﬁcient to induce word alignment or even surprisingly fails for general NMT models.
Therefore, it proposes two better methods to induce word alignment than the method by attention for general NMT models.
Based on one of the proposed method, it divides target words into two categories including CFS and CFT and shows that NMT models yield excellent word alignment on CFS words but worse word alignment on CFT words, which do not signiﬁcantly sacriﬁce translation quality.
This explains a fundamental question why NMT delivers better translation yet worse word alignment than its SMT counterpart.
Finally, this paper empirically demonstrates that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment quality and translation quality.
In the future, we believe that more work is interesting to analyze translation errors such as those errors for
CFT words.
In addition, we will investigate solutions to improving NMT models, in the hope of using source context and target history context in a more robust manner for better predicting CFS and CFT words.
A T RAINING DETAILS
We implemented the proposed methods to induce word alignment on T RANSFORMER , since it is the most popular NMT model nowadays.
For training M OSES , we use all 1.8M sentences from the corpus, and we train a 4-gram language model based on the target side of its training data.
For training both NMT models, only the sentences of length up to 256 tokens are used, with no more than 2 15 tokens in a batch.
The dimension of both word embeddings and hidden states are 512.
Both encoder and decoder have 6 layers by default, and adopt multi-head attention with 8 heads.
The beam size for decoding is 4, and the loss function is optimized by Adam (Kingma & Ba, 2014), where β 1 = 0 .
9 , β 2 = 0 .
98 and (cid:15) = 10 − 9 .
Particularly for the explicit alignment model, the alignment reference is produced by F AST A LIGN .
Note that M OSES and N EMATUS achieve BLEU points on NIST2005 test set comparable to those reported in recent works using the similar corpora (Tu et al., 2016;
Liu et al., 2016;
Zhou et al., 2017), and T RANSFORMER achieves much higher performance (i.e. 47 BLEU points).
This shows that our MT models are well-trained.
B C ASE S TUDY
[Figure]
[Table]
[Figure]
Reference: Translation: (a) Gold Alignment
[Figure]
著名
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
<eos
[Figure]
is
[Figure]
world
[Figure]
navigator
[Figure]
he is
[Figure]
world
[Figure]
navigator
[Figure]
zhèng hé shì shì jiè
zhù míng háng hǎi jiā 。
[Figure]
eos
[Figure]
著名 航海家 。 <eos>
Zheng he is a world famous navigator .
<eos>
Zheng he is a world famous navigator .
<eos
[Figure]
Reference: Translation:
[Figure]
Here is an example to demonstrate the results of previous experiments intuitively.
As shown in
Figure 2(a), the word ‘a’ should not be aligned to any source word.
However, in Figure 2(b) F AST
A LIGN wrongly aligned ‘a’ to ‘sh ` ı’, and in Figure 2(c) PD makes the same mistake.
Fortunately, as shown in Figure 2(d), if we only consider alignment of words in CFS, ‘a’ is superbly not aligned to any source word because it belongs to CFT.
In terms of translation, T RANSFORMER translates perfectly, yet M OSES wrongly translate ‘h´e’ into ‘and’ due to ‘h´e’ mostly means ‘and’ ignoring the context in Chinese.
Instead of depending on phrase table, as shown in Figure 2(d), T RANSFORMER successfully translate ‘h´e’ into the given name ‘he’ referring to the surname ‘Zheng’ in translation history, thanks to its more powerful language model.
C P OINTWISE M UTUAL I NFORMATION AMONG W ORDS
Pointwise mutual information (PMI) measures the relevance of two discrete random variables, which is deﬁned as
[Equation]
where C ( µ, ν ) is a function for counting occurrence of the pair ( µ, ν ) according to different scenarios, and Z is the normalizer, i.e. the total number of all possible ( µ, ν ) pairs.
In this paper, we deﬁne three types of PMI according to different deﬁnitions of C ( µ, ν ) in the three scenarios as follows.
PMI on Bilingual Data
In this scenario, a set of bilingual sentences is given.
For a given bilingual sentence (cid:104) x , y (cid:105) , C ( y i , x j ) is added by one if both y i ∈ y and x j ∈ x .
PMI on Word Aligned Bilingual Data In this scenario, a set of word aligned bilingual sentences is given.
That is, for a bilingual sentence, words in its target side may align to words in its source side.
For a given word aligned bilingual sentence ( x , y ) , C ( y i , x j ) is added by one if y i ∈ y and x j ∈ x
and y i aligns to x j .
PMI between a Word and Its History Word on Monolingual Data In this scenario, a set of monolingual sentences is given.
For a given monolingual sentence y , C ( y k , y i ) is added by one if y k ∈ y and y i ∈ y with k < i .
D D IFFERENT M ARGINS FOR D IVIDING CFS AND CFT
In equation 11, different margins will partition CFS and CFT differently.
As growing of the margin (cid:15) , the partition of CFS and CFT becomes more conﬁdent.
As shown in Table 6, more conﬁdent CFS words can achieve better alignment performance and translation performance.
But the translation recall is still similar between CFS and CFT words, despite the big gap of AER between CFS and
CFT words.
[Table]