Paper:
Prior researches suggest that attentional neural machine translation (NMT) is able to capture word alignment by attention, however, to our surprise, it almost fails for NMT models with multiple attentional layers except for those with a single layer. This paper introduce two methods to induce word alignment from general neural machine translation models. Experiments verify that both methods obtain much better word alignment than the method by attention. Furthermore, based on one of the proposed method, we design a criterion to divide target words into two categories (i.e. those mostly contributed from source "CFS" words and the other words mostly contributed from target "CFT" words), and analyze word alignment under these two categories in depth. We find that although NMT models are difficult to capture word alignment for CFT words but these words do not sacrifice translation quality significantly, which provides an explanation why NMT is more successful for translation yet worse for word alignment compared to statistical machine translation. We further demonstrate that word alignment errors for CFS words are responsible for translation errors in some extent by measuring the correlation between word alignment and translation for several NMT systems.

Review:
This paper sets out to build good bilingual word alignments from the information in an NMT system (both Transformer and RNN), where the goal is to match human-generated word-alignments as measured by AER. At least that’s how it starts. They contribute two aligners: one supervised aligner that uses NMT source and target representations as features and is trained on silver data generated by FastAlign, and one interpretability-based aligner that scores the affinity of a source-target word-pair by deleting the source word (replacing its embedding with a 0-vector) and measuring the impact on the probability of the target word. These are both shown to outperform directly extracting alignments from attention matrices by large margins. Despite the supervised aligner getting better AER, the authors proceed to quickly discard it as they dive deep on the interpretability approach, applying it also to target-target word pairs, and drawing somewhat interesting conclusions about two classes of target words: those that depend most of source context and those that depend most on target context.
Ultimately, this paper’s main contribution is its subtraction-based method for doing model interpretation. Its secondary contributions are the idea of evaluating this interpretation method empirically using human-aligned sentence pairs, and the idea of using the subtraction method on target-target pairs. The conclusion does a good job of emphasizing these contributions, but the abstract and front-matter do not. Much of the rest of the paper feels like a distraction. Overall, I believe the contributions listed above are valuable, novel and worth publishing. I can imagine using this paper’s techniques and ideas in my own research.
Specific concerns:
The front-matter mentions ‘multiple attention layers’. It would probably be a good idea to define this term carefully, as there are lots of things that could fit: multiple decoder layers with distinct attentions, multi-headed attention, etc.
In contrast to what is said in the introduction, GNMT as described in the Wu et al. 2016 paper only calculates attention once, based on the top encoder layer and the bottom decoder layer, so it doesn’t fit any definition of multiple attention layers.
Equation (1) and the following text use the variable L without defining it.
‘dominative’ -> ‘dominant’
Is there any way to generate a null alignment with Equation 3? That is, a target word that has no aligned source words? If not, that is a major advantage for FastAlign.
Similarly, what exactly are you evaluating when you evaluate FastAlign? Are you doing the standard tricks from the phrase-based days, and generating source->target and target->source models, and combining their alignments with grow-diag-final? If so, you could apply the same tricks to the NMT system to help even the playing field. Maybe this isn’t that important since the paper didn’t win up being about how to build the best possible word aligner from NMT (which I think is for the best).
I found Equations (7) and (8) to be confusing and distracting. I understand that you were inspired by Zintgraf’s method, but the subtraction-based method you landed on doesn’t seem to have much to do with the original Zintgraf et al. approach (and your method is much easier to the understand in the context of NMT than theirs). Likewise, I do not understand why you state, “we take the uniform distribution as P(x) regarding equation 8 for simplicity” - equation 9 completely redefines the LHS of equation 8, with no sum over x and no uniform distribution in sight.
The Data section of 4.1 never describes the NIST 2005 hand-aligned dataset.
The conclusions drawn at the end of 4.4 based on ‘translation recall’ are too strong. What we see is that the Transformer outperforms Moses by 2.8 onCFS, and by 3.7 on CFT. This hardly seems to support a claim that CFT words are the reason why Transformer yields better translation.
4.5 paragraph 1: there is no way to sample 12000 datasets without replacement from NIST 2005 and have the samples be the same size as NIST 2005. You must mean “with replacement”?