End-to-End Neural Word Alignment Outperforms GIZA ++ (2020)

Abstract: Word alignment was once a core unsupervised learning task in natural language processing because of its essential role in training statistical machine translation (MT) models. Although unnecessary for training neural MT models, word alignment still plays an important role in interactive applications of neural machine translation, such as annotation transfer and lexicon injection. While statistical MT methods have been replaced by neural approaches with superior performance, the twenty-year-old GIZA++ toolkit remains a key component of state-of-the-art word alignment systems. Prior work on neural word alignment has only been able to outperform GIZA++ by using its output during training. We present the first end-to-end neural word alignment method that consistently outperforms GIZA++ on three data sets. Our approach repurposes a Transformer model trained for supervised translation to also serve as an unsupervised word alignment model in a manner that is tightly integrated and does not affect translation quality.

## introduction although word alignments are no longer necessary to train machine translation (mt) systems, they still play an important role in applications of neural mt. for example, they enable injection of an external lexicon into the inference process to enforce the use of domain-specific terminology or improve the translations of low-frequency content words @xcite . the most important application today for word alignments is to transfer text annotations from source to target @xcite @xcite . for example, if part of a source sentence is underlined, the corresponding part of its translation should be underlined as well. html tags and other markup must be transferred for published documents. although annotations could in principle be generated directly as part of the output sequence, they are instead typically transferred via word alignments because example annotations typically do not exist in mt training data. the transformer architecture provides state-ofthe-art performance for neural machine translation @xcite . the decoder has multiple layers, each with several attention heads, which makes it difficult to interpret attention activations as word alignments. as a result, the most widely used tools to infer word alignments, namely giza++ @xcite and fastalign @xcite , are still based on the statistical ibm word alignment models developed nearly thirty years ago @xcite . no previous unsupervised neural approach has matched their performance. recent work on alignment components that are integrated into neural translation models either underperform the ibm models or must use the output of ibm models during training to outperform them @xcite . this work combines key components from @xcite and @xcite and presents two novel extensions. statistical alignment methods contain an explicit bias towards contiguous word alignments in which adjacent source words are aligned to adjacent target words. this bias is expressed in statistical systems using a hidden markov model (hmm) @xcite , as well as symmetrization heuristics such as the growdiag-final algorithm @xcite . we design an auxiliary loss function that can be added to any attention-based network to encourage contiguous attention matrices. the second extension replaces heuristic symmetrization of word alignments with an activation optimization technique. after training two alignment models that translate in opposite directions, we infer a symmetrized attention matrix that jointly optimizes the likelihood of the correct output words under both models in both languages. ablation experiments highlight the effectiveness of this novel extension, which is reminiscent of agreement-based methods for statistical models @xcite @xcite . end-to-end experiments show that our system is the first to consistently yield higher alignment quality than giza++ using a fully unsupervised neural model that does not use the output of a statistical alignment model in any way. ## contiguity loss contiguous alignment connections are very common in word alignments, especially for pairs of indo-european languages. that is, if a target word at position t is aligned to a source word at position s, the next target word at position t + 1 is often aligned to s -1, s or s + 1 @xcite . our goal is to design a loss function that encourages alignments with contiguous clusters of links. the attention activations form a 2-dimensional matrix @xmath0, where n is the number of source tokens and m the number of target tokens: each entry represents a probability that specifies how much attention weight the network puts on each source word to predict the next target word. by using a convolution with a static kernel k over these attention scores, we can measure how much attention is focused on each rectangle within the two dimensional attention matrix: we use a 2 × 2 kernel @xmath1.5. therefore, ā ∈ r n×m will contain the normalized attention mass of each 2 × 2 square of the attention matrix a. the resulting values after the convolution will be in the interval [0.0, 1.0]. for each target word we select the square with the highest attention mass, encouraging a sparse distribution over source positions in ā and thus effectively training the model towards strong attention values on neighboring positions. we mask the contiguity loss such that the end of sentence symbol is not considered during this procedure. we apply a position-wise dropout of 0.1 on the attention logits before using the softmax function to obtain a, which turned out to be important to avoid getting stuck in trivial solutions during training. 3 optimizing the alignment loss especially encourages diagonal and horizontal patterns foot_3 as visualized in figure 4 . these correspond well to a large portion of patterns appearing in human alignment annotations as shown in figure 1 . ## bidirectional attention optimization a common way to extract word alignments is to train two models, one for the forward direction (source to target) and one for the backward direction (target to source). for each model, one can extract separate word alignments and symmetrize these using heuristics like grow-diagonal @xcite . however, this approach uses the hard word alignments of both directions as an input, and does not consider any other information of the forward and backward model. for attention-based neural networks it is possible to adapt attention optimization as described in section 3.4 to consider two models at the same time. the goal of attention optimization is to find attention activations that lead to the correct prediction of the target sequence for a single neural network. we extend this procedure to optimize the likelihood of the sentence pair jointly under both the forward and the backward model, with the additional bias to favor contiguous alignments. figure 5 depicts this procedure. ## experiments 6.1 data we use the same experimental setup foot_4 as described by @xcite and used by @xcite . it contains three language pairs: @xmath2, @xmath3. we learn a joint byte pair encoding (bpe) for the source and the target language with 40k merge operation @xcite . to convert from alignments between word pieces to alignments between words, we align a source word to a target word if an alignment link exists between any of its word pieces. using bpe units instead of words also improved results for giza++ (e.g., 20.9% vs. 18.9% for @xmath4. therefore, we use the exact same input data for giza++ and all our neural approaches. for training giza++ we use five iterations each for model 1, the hmm model, model 3 and model 4. ## conclusion this work presents the first end-to-end neural approach to the word alignment task which consistently outperforms giza++ in terms of alignment error rate. our approach extends a pre-trained stateof-the-art neural translation model with an additional alignment layer, which is trained in isolation without changing the parameters used for the trans-lation task. we introduce a novel auxiliary loss function to encourage contiguity in the alignment matrix and a symmetrization algorithm that jointly optimizes the alignment matrix within two models which are trained in opposite directions. in a final step the model is re-trained to leverage full target context with a guided alignment loss. our results on three language pairs are consistently superior to both giza++ and prior work on end-to-end neural alignment. as the resulting model repurposes a pre-trained translation model without changing its parameters, it can directly benefit from improvements in translation quality, e.g. by adaptation via fine-tuning.