Accurate Word Alignment Induction from Neural Machine Translation (2020)

Abstract: Despite its original goal to jointly learn to align and translate, prior researches suggest that Transformer captures poor word alignments through its attention mechanism. In this paper, we show that attention weights do capture accurate word alignments and propose two novel word alignment induction methods Shift-Att and Shift-AET. The main idea is to induce alignments at the step when the to-be-aligned target token is the decoder input rather than the decoder output as in previous work. Shift-Att is an interpretation method that induces alignments from the attention weights of Transformer and does not require parameter update or architecture change. Shift-AET extracts alignments from an additional alignment module which is tightly integrated into Transformer and trained in isolation with supervision from symmetrized Shift-Att alignments. Experiments on three publicly available datasets demonstrate that both methods perform better than their corresponding neural baselines and Shift-AET significantly outperforms GIZA++ by 1.4-4.8 AER points.

## introduction the task of word alignment is to find lexicon translation equivalents from parallel corpus @xcite . it is one of the fundamental tasks in natural language processing (nlp) and is widely studied by the community @xcite @xcite . word alignments are useful in many scenarios, such as error analysis @xcite , the introduction of coverage and fertility models @xcite , inserting external constraints in interactive machine translation @xcite ; das weiß ich . das weiß ich . das weiß ich . source: das weiß ich . dec. input: <@xmath0. dec. output: i understand this . <@xmath1., 2020) and providing guidance for human translators in computer-aided translation @xcite . word alignment is part of the pipeline in statistical machine translation @xcite , but is not necessarily needed for neural machine translation @xcite . the attention mechanism in nmt does not functionally play the role of word alignments between the source and the target, at least not in the same way as its analog in smt. it is hard to interpret the attention activations and extract meaningful word alignments especially from transformer @xcite . as a result, the most widely used word alignment tools are still external statistical models such as fast-align @xcite and giza++ @xcite . recently, there is a resurgence of interest in the community to study word alignments for the transformer @xcite . one simple solution is naive-att, which induces word alignments from the attention weights between the encoder and decoder. the next target word is aligned with the source word that has the maximum attention weight, as shown in fig. 1 . however, such schedule only captures noisy word alignments @xcite . one of the major problems is that it induces alignment before observing the to-be-aligned target token @xcite . suppose for the same source sentence, there are two alternative translations that diverge at decoding step i, generating y i and y i which respectively correspond to different source words. presumably, the source word that is aligned to y i and y i should change correspondingly. however, this is not possible under the above method, because the alignment scores are computed before prediction of y i or y i . to alleviate this problem, some researchers modify the transformer architecture by adding alignment modules that predict the to-be-aligned target token @xcite or modify the training loss by designing an alignment loss computed with full target sentence @xcite . others argue that using only attention weights is insufficient for generating clean word alignment and propose to induce alignments with feature importance measures, such as leaveone-out measures @xcite and gradientbased measures @xcite . however, all previous work induces alignment for target word y i at step i, when y i is the decoder output. in this work, we propose to induce alignment for target word y i at step i + 1 rather than at step i as in previous work. the motivation behind this is that the hidden states in step i + 1 are computed taking word y i as the input, thus they can incorporate the information of the to-be-aligned target token y i easily. following this idea, we present shift-att and shift-aet, two simple yet effective methods for word alignment induction. our contributions are threefold: ## method in this section, we propose two novel alignment induction methods shift-att and shift-aet. both methods adapt the alignment induction with the to-be-aligned target token by computing alignment scores at the step when the target token is the decoder input. ## related work alignment induction from rnnsearch @xcite has been explored by a number of works. @xcite are the first to show word alignment example using attention in rnnsearch. @xcite further demonstrate that the rnn-based nmt system achieves comparable alignment performance to that of giza++. alignment has also been used to improve nmt performance, especially in low resource settings, by supervising the attention mechanisms of rnnsearch @xcite @xcite . there is also a number of other studies that induce word alignment from transformer. @xcite claim that attention may not capture word alignment in transformer, and propose to induce word alignment with prediction difference @xcite or gradient-based measures @xcite . @xcite modify the transformer architecture for better alignment induction by adding an extra alignment module that is restricted to attend solely on the encoder information to predict the next word. @xcite propose a multi-task learning framework to improve word alignment induction without decreasing translation quality, by supervising one attention head at the penultimate layer with giza++ alignments. although these methods are reported to improve over head average baseline, they ignore that better alignments can be induced by computing alignment scores at the decoding step when the to-be-aligned target token is the decoder input. ## conclusion in this paper, we have presented two novel methods shift-att and shift-aet for word alignment induction. both methods induce alignments at the step when the to-be-aligned target token is the decoder input rather than the decoder output as in previous work. experiments on three public alignment datasets and a downstream task prove the effectiveness of these two methods. shift-aet further extends transformer with an addi-tional alignment module, which consistently outperforms prior neural aligners and giza++, without influencing the translation quality. to the best of our knowledge, it reaches the new state-of-theart performance among all neural alignment induction methods. we leave it for future work to extend our study to more downstream tasks and systems.