Third-Party Aligner for Neural Word Alignments (2022)

Abstract: Word alignment is to find translationally equivalent words between source and target sentences. Previous work has demonstrated that self-training can achieve competitive word alignment results. In this paper, we propose to use word alignments generated by a third-party word aligner to supervise the neural word alignment training. Specifically, source word and target word of each word pair aligned by the third-party aligner are trained to be close neighbors to each other in the contextualized embedding space when fine-tuning a pre-trained cross-lingual language model. Experiments on the benchmarks of various language pairs show that our approach can surprisingly do self-correction over the third-party supervision by finding more accurate word alignments and deleting wrong word alignments, leading to better performance than various third-party word aligners, including the currently best one. When we integrate all supervisions from various third-party aligners, we achieve state-of-the-art word alignment performances, with averagely more than two points lower alignment error rates than the best third-party aligner.We released our code at https://github.com/sdongchuanqi/Third-Party-Supervised-Aligner.

## introduction word alignment is to find the correspondence between source side and target side words in a sentence pair @xcite . it is widely applied in a variety of natural language processing (nlp) tasks, including learning translation lexicons @xcite , cross-lingual transfer @xcite @xcite @xcite , and semantic parsing @xcite . in figure 1 : gold and third-party word alignments, and cosine similarities between contextualized embeddings of subwords in a parallel sentence pair before and after fine-tuning with third-party supervision. particular, word alignment plays a key role in many neural machine translation (nmt) related methods, such as imposing lexical constraints in the decoding process @xcite , improving automatic post-editing @xcite , guiding learned attention @xcite , and automatic analysis or evaluation of nmt models @xcite @xcite @xcite . word alignment is usually inferred by giza++ @xcite or fastalign @xcite , which are based on the statistical ibm word alignment models @xcite . recently, neural methods are applied for inferring the word alignment. they use nmt-based framework to induce alignments through using attention weights or feature importance measures, and surpass the statistical word aligners such as giza++ on a variety of language pairs @xcite @xcite @xcite @xcite . inspired by the success of the large-scale crosslingual language model (clm) pre-training @xcite @xcite , the pre-trained contextualized word embeddings are also explored for the word alignment task by either extracting alignments based on the pre-trained contextualized embeddings @xcite or fine-tuning the pretrained clms by self-training to get new contextualized embeddings appropriate for extracting word alignments @xcite . based on careful design of self-training objectives, the finetuning approach achieves competitive word alignment results @xcite . in this paper, we use simple supervision instead of the self-training to fine-tune the pre-trained clms. the simple supervision is derived from a third-party word aligner. given a parallel corpus, the third-party word aligner predicts the word alignments over the corpus, which are used as the supervision signal for the fine-tuning. in particular, for each aligned word pair of a parallel sentence pair, the contextualized embeddings of the source and target words are trained to have high cosine similarity to each other in the embedding space. as illustrated by figure 1 , the cosine similarities between the source and target words of the correct word alignments are not quite high before the finetuning. the third-party word aligner can provide some correct word alignments (e.g. "that" "must" "be" associated alignments) along with wrong ones (e.g. "primary" "objective" associated alignments) as the supervision. although the supervision is not perfect, it is still helpful for driving the contextualized embeddings of the source and target words of a correct word alignment closer in the embedding space after the fine-tuning. surprisingly, with imperfect third-party supervision in fine-tuning, the heat map of the cosine similarities exhibits clearer split between the correct and wrong word alignments than not fine-tuning. wrong alignments of the third-party aligner are rectified after fine-tuning (e.g. "primary" "objective" associated alignments), and the incorrect alignment before fine-tuning (e.g. "be"associated alignment) is also rectified after finetuning. we perform experiments on word alignment benchmarks of five different language pairs. the results show that the proposed third-party supervising approach outperforms all third-party word aligners. when we integrate all supervisions from various third-party word aligners, we achieve state-of-theart performances across all benchmarks, with an average word error rate two points lower than that of the best third-party word aligner. ## approach formally, the word alignment task can be defined as finding a set of word pairs in the sentence pair ⟨s, t⟩, where s denotes the source sentence "s 1 , ..., s n ", and t denotes the corresponding target sentence "t 1 , ..., t m " parallel to s. the set of the word pairs is: @xmath0, t j ⟩|s @xmath1, t @xmath2. in each word pair ⟨s i , t j ⟩, s i and t j are translationally equivalent to each other within the context of the sentence pair. in the following, we will describe how we obtain the word alignments by fine-tuning the pre-trained clms. different to previous work that fine-tunes by self-training @xcite , we supervise the fine-tuning process with third-party word alignments. ## experiments we test the proposed third-party supervised finetuning approach on word alignment tasks of five language pairs: chinese-english (zh-en), german-english (de-en), english-french (en-fr), romanian-english (ro-en) and japanese-english (ja-en). ## conclusion we propose an approach of using a third-party aligner for neural word alignments. different to previous work based on careful design of selftraining objectives, we simply use the word alignments generated by the third-party aligners to supervise the training. although the third-party word alignments are imperfect as the supervision, we observe that the training process can do selfcorrection over the third-party word alignments by detecting more accurate word alignments and deleting wrong word alignments based on the geometry similarity in the contextualized embedding space, leading to better performances than the third-party aligners. the integration of various third-party supervisions improves the performance further, achieving state-of-the-art word alignment performance on benchmarks of multiple language pairs.