Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021)

Abstract: Word alignment over parallel corpora has a wide variety of applications, including learning translation lexicons, cross-lingual transfer of language processing tools, and automatic evaluation or analysis of translation outputs. The great majority of past work on word alignment has worked by performing unsupervised learning on parallel text. Recently, however, other work has demonstrated that pre-trained contextualized word embeddings derived from multilingually trained language models (LMs) prove an attractive alternative, achieving competitive results on the word alignment task even in the absence of explicit training on parallel data. In this paper, we examine methods to marry the two approaches: leveraging pre-trained LMs but fine-tuning them on parallel text with objectives designed to improve alignment quality, and proposing methods to effectively extract alignments from these fine-tuned models. We perform experiments on five language pairs and demonstrate that our model can consistently outperform previous state-of-the-art models of all varieties. In addition, we demonstrate that we are able to train multilingual word aligners that can obtain robust performance on different language pairs.

## introduction word alignment is a useful tool to tackle a variety of natural language processing (nlp) tasks, including learning translation lexicons @xcite , cross-lingual transfer of language processing tools @xcite @xcite @xcite , semantic parsing @xcite and speech recognition @xcite . in particular, word alignment plays a crucial role in many machine translation (mt) related methods, including guiding learned attention @xcite , incorporating lexicons during decoding @xcite , domain adaptation @xcite , unsupervised mt @xcite and automatic evaluation or analysis of translation models @xcite @xcite . however, with neural networks advancing the state of the arts in almost every field of nlp, tools developed based on the 30-yearold ibm word-based translation models @xcite , such as giza++ @xcite or fast-align @xcite , remain popular choices for word alignment tasks. one alternative to using statistical word-based translation models to learn alignments would be to instead train state-of-the-art neural machine translation (nmt) models on parallel corpora, and extract alignments therefrom, as examined by @xcite ; @xcite ; @xcite . however, these methods have two disadvantages (also shared with more traditional alignment methods): (1) they are directional and the source and target side are treated differently and (2) they cannot easily take advantage of large-scale contextualized word embeddings derived from language models (lms) multilingually trained on monolingual corpora @xcite @xcite , which have proven useful in other cross-lingual transfer settings @xcite . in the field of word alignment, @xcite have recently proposed methods to align words using multilingual contextualized embeddings and achieve good performance even in the absence of explicit training on parallel data, suggesting that these are an attractive alternative for neural word alignment. in this paper, we investigate if we can combine the best of the two lines of approaches. concretely, we leverage pre-trained lms and fine-tune them on parallel text with not only lm-based objectives, but also unsupervised objectives over the parallel corpus designed to improve alignment quality. specifically, we propose a self-training objective, which encourages aligned words to have further closer contextualized representations, and a parallel sentence identification objective, which enables the model to bring parallel sentences' representations closer to each other. in addition, we propose to effectively extract alignments from these fine-tuned models using probability thresholding or optimal transport. we perform experiments on five different language pairs and demonstrate that our model can achieve state-of-the-art performance on all of them. in analysis, we find that these approaches also generate more aligned contextualized representations after fine-tuning (see figure 1 as an example) and we can incorporate supervised signals within our paradigm. importantly, we show that it is possible to train multilingual word aligners that can obtain robust performance even in zero-shot settings, making them a valuable tool that can be used out-ofthe-box with good performance over a wide variety of language pairs. ## methods formally, the task of word alignment can be defined as: given a sentence @xmath0, • • • , x n in the source language and its corresponding parallel sentence @xmath1, • • • , y m in the target language, a word aligner needs to find a set of pairs of source and target words: in the following paragraphs, we will first illustrate how we extract alignments from contextualized word embeddings, then describe our objectives designed to improve alignment quality. ## experiments in this section, we first present our main results, then conduct several ablation studies and analyses of our models. ## related work based on the ibm translation models @xcite , many statistical word aligners have been proposed @xcite , including the current most popular tools giza++ @xcite @xcite and fast align @xcite . recently, there is a resurgence of interest in neural word alignment @xcite . based on nmt models trained on parallel corpora, researchers have proposed several methods to extract alignments from them @xcite @xcite and successfully build an end-to-end neural model that can outperform statistical tools @xcite . however, there is an inherent discrepancy between translation and word alignment: translation models are directional and the source and target side are treated differently, while word alignment is a non-directional task. therefore, certain adaptations are required for translation models to perform word alignment. another disadvantage of mt-based word aligners is that they cannot easily utilize contextualized embeddings. using learned representations to improve word alignment have been investigated @xcite . recently, pre-trained lms @xcite @xcite have proven to be useful in cross-lingual transfer @xcite . in word alignment, sabet et al. (2020) propose effective methods to extract alignments from multilingual lms without explicit training on parallel data. in this work, we propose better alignment extraction methods and combine the best of the two worlds by fine-tuning contextualized embeddings on parallel data. there are also work on supervised neural word alignment @xcite . however, supervised data are not always accessible, making their methods inapplicable in many scenarios. in this paper, we demonstrate that our model can incorporate supervised signals if available and perform semi-supervised learning, which is a more realistic and general setting. some work on bilingual lexicon induction also share similar general ideas with ours. for example, @xcite minimize the earth mover's distance to match the embedding distributions from different languages. similarly, @xcite present an algorithm to align point clouds with procrustes @xcite in wasserstein distance for unsupervised embedding alignment. ## discussion and conclusion we present a neural word aligner that achieves stateof-the-art performance on five diverse language pairs and obtains robust performance in zero-shot settings. we propose to fine-tune multilingual embeddings with objectives suitable for word alignment and develop two alignment extraction methods. we also demonstrate its applications in semisupervised settings. we hope our word aligner can be a tool that can be used out-of-the-box with good performance over various language pairs. future directions include designing better training objectives and experimenting on more language pairs. also, note that we mainly evaluate our word aligners using aer following previous work, which has certain limitations. for example, it may not be well-correlated with statistical machine translation performance @xcite and different types of alignments can be suitable for different tasks or conditions @xcite . although we have evaluated models in annotation projection and cross-lingual transfer settings, alternative metrics @xcite @xcite are also worth considering in the future.