Mask-Align: Self-Supervised Neural Word Alignment (2021)

Abstract: Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks. Current unsupervised neural alignment methods focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence. In this paper, we propose Mask-Align, a self-supervised word alignment model that takes advantage of the full context on the target side. Our model masks out each target token and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates the problem of unexpected high cross-attention weights on special tokens such as periods. Experiments on four language pairs show that our model outperforms previous unsupervised neural aligners and obtains new state-of-the-art results.

## introduction word alignment is an important task of finding the correspondence between words in a sentence pair @xcite and used to be a key component of statistical machine translation (smt) @xcite . although word alignment is no longer explicitly modeled in neural machine translation (nmt) @xcite , it is often leveraged to analyze nmt models @xcite . word alignment is also used in many other scenarios such as imposing lexical constraints on the decoding process @xcite ich wurde 1968 in tokio geboren . source : i was born in in 1968 . target : ## approach figure 2 shows the architecture of our model. the model predicts each target token conditioned on the source and other target tokens and generates alignments from the attention weights between source and target (section 2.1). specifically, our approach introduces two attention variants, static-kv attention and leaky attention, to efficiently obtain attention weights for word alignment. to better utilize attention weights from two directions, we encourage agreement between two unidirectional models during both training (section 2.2) and inference (section 2.3). ## related work our work is closely related to unsupervised neural word alignment. while early unsupervised neural aligners @xcite @xcite failed to outperform their statistical counterparts such as fast-align @xcite and giza++ @xcite , recent studies have made significant progress by inducing alignments from nmt models @xcite @xcite . our work differs from prior studies in that we design a novel self-supervised model that is capable of utilizing more target context than nmt-based models to generate high quality alignments without using guided training. our work is also inspired by the success of conditional masked language models (cmlms) @xcite , which have been applied to non-autoregressive machine translation. the cmlm can leverage both previous and future context on the target side for sequence-to-sequence tasks with the masking mechanism. @xcite extend it with a disentangled context transformer that predicts every target token conditioned on arbitrary context. by taking the characteristics of word alignment into consideration, we propose to use static-kv attention to achieve masking and aligning in parallel. to the best of our knowledge, this is the first work that incorporates a cmlm into alignment models. ## conclusion we have presented a self-supervised neural alignment model mask-align. our model parallelly masks out and predicts each target token. we propose static-kv attention and leaky attention to achieve parallel computation and address the "garbage collectors" problem, respectively. experiments show that mask-align achieves new stateof-the-art results without using the guided alignment loss. in the future, we plan to extend our method to directly generate symmetrized alignments without leveraging the agreement between two unidirectional models.