Neural word alignmentmethods have received increasing attention recently. These methods usually extract word alignment from a machine translation model. However, there is a gap between translation and alignment tasks, since the target future context is available in the latter . In this paper, we propose Mask-Align, a self-supervised model specifically designed for the word alignment task . Our model parallelly masks and predicts each target token , and extracts high-quality alignments without any supervised loss. In addition, we introduce leaky attention to alleviate the problem of unexpected high attention weights on special tokens  . Experiments on four language pairs show that our model significantly outperforms all existing unsupervised neural baselines and obtains new state-of-the-art results.
Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks. Current unsupervised neural alignment methods focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence . In this paper, we propose Mask-Align, a self-supervised word alignment model that takes advantage of the full context on the target side . Our model masks out each target token and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates the problem of unexpected high cross-attention weights on special tokens such as periods . Experiments on four language pairs show that our model outperforms previous unsupervised neural aligners and obtains new state-of-the-art results.
R
Neural word alignmentmethods have received increasing attention recently. These methods usually extract word alignment from a machine translation model. However, there is a gap between translation and alignment tasks, since the target future context is available in the latter
Word alignment, which aims to align translationally equivalent words between source and target sentences, plays an important role in many natural language processing tasks. Current unsupervised neural alignment methods focus on inducing alignments from neural machine translation models, which does not leverage the full context in the target sequence
R
model specifically designed for the word alignment task
word alignment model that takes advantage of the full context on the target side
R
parallelly masks and predicts
masks out
R
, and extracts high-quality alignments without any supervised loss. In addition, we introduce leaky attention to alleviate
and predicts it conditioned on both source and the remaining target tokens. This two-step process is based on the assumption that the source token contributing most to recovering the masked target token should be aligned. We also introduce an attention variant called leaky attention, which alleviates
R
attention
cross-attention
A
such as periods
R
significantly outperforms all existing unsupervised neural baselines
outperforms previous unsupervised neural aligners