A Bidirectional Transformer Based Alignment Model for Unsupervised Word Alignment (2021)

Abstract: Word alignment and machine translation are two closely related tasks. Neural translation models, such as RNN-based and Transformer models, employ a target-to-source attention mechanism which can provide rough word alignments, but with a rather low accuracy. High-quality word alignment can help neural machine translation in many different ways, such as missing word detection, annotation transfer and lexicon injection. Existing methods for learning word alignment include statistical word aligners (e.g. GIZA++) and recently neural word alignment models. This paper presents a bidirectional Transformer based alignment (BTBA) model for unsupervised learning of the word alignment task. Our BTBA model predicts the current target word by attending the source context and both left-side and right-side target context to produce accurate target-to-source attention (alignment). We further fine-tune the target-to-source attention in the BTBA model to obtain better alignments using a full context based optimization method and self-supervised training. We test our method on three word alignment tasks and show that our method outperforms both previous neural word alignment approaches and the popular statistical word aligner GIZA++.

## introduction neural machine translation (nmt) @xcite achieves stateof-the-art results for various translation tasks @xcite . neural translation models, such as rnn-based @xcite and transformer @xcite models, generally have an encoder-decoder structure with a target-to-source attention mechanism. the targetto-source attention in nmt can provide rough word alignments but with a rather low accuracy @xcite . high-quality word alignment can be used to help nmt in many different ways, such as detecting source words that are missing in the translation @xcite , integrating an external lexicon into nmt to improve translation for domain-specific terminology or low-frequency words @xcite , transferring word-level annotations (e.g. underline and hyperlink) from source to target for document/webpage translation @xcite . a number of approaches have been proposed to learn the word alignment task, including both statistical models @xcite and recently neural models @xcite @xcite @xcite . the popular word alignment tool giza++ @xcite is based on statistical ibm models @xcite which learn the word alignment task through unsupervised learning and do not require gold alignments from humans as training data. as deep neural networks have been successfully applied to many natural language processing (nlp) tasks, neural word alignment approaches have developed rapidly and outperformed statistical word aligners @xcite . neural word alignment approaches include both supervised and unsupervised approaches: supervised approaches @xcite ) use gold alignments from human annotators as training data and train neural models to learn word alignment through supervised learning; unsupervised approaches do not use gold human alignments for model training and mainly focus on improving the target-to-source attention in nmt models to produce better word alignment, such as performing attention optimization during inference @xcite , encouraging contiguous alignment connections @xcite or using alignments from giza++ to supervise/guide the attention in nmt models @xcite . we propose a bidirectional transformer based alignment (btba) model for unsupervised learning of the word alignment task. our btba model predicts the current target word by paying attention to the source context and both left-side and right-side target context to produce accurate targetto-source attention (alignment). compared to the original transformer translation model @xcite which computes target-to-source attention based on only the left-side target context due to left-to-right autoregressive decoding, our btba model can exploit both left-side and rightside target context to compute more accurate targetto-source attention (alignment). we further finetune the btba model to produce better alignments using a full context based optimization method and self-supervised training. we test our method on three word alignment tasks and show that our method outperforms previous neural word alignment approaches and also beats the popular statistical word aligner giza++. ## our approach we present a bidirectional transformer based alignment (btba) model for unsupervised learning of the word alignment task. motivated by bert which learns a masked language model @xcite , we randomly mask 10% of the words in the target sentence and then train our btba model to predict the masked target words by paying attention to the source context and both leftside and right-side target context. therefore, our btba model can exploit both left-side and rightside target context to compute more accurate targetto-source attention (alignment) compared to the original transformer translation model @xcite which computes the target-to-source attention based on only the left-side target context due to left-to-right autoregressive decoding. we further fine-tune the target-to-source attention in the btba model to produce better alignments using a full context based optimization method and self-supervised training. ## conclusion this paper presents a novel btba model for unsupervised learning of the word alignment task. our btba model predicts the current target word by paying attention to the source context and both left-side and right-side target context to produce accurate target-to-source attention (alignment). we further fine-tune the target-to-source attention in the btba model to obtain better alignments using a full context based optimization method and selfsupervised training. we test our method on three word alignment tasks and show that our method outperforms both previous neural alignment methods and the popular statistical word aligner giza++.