PMI -Align: Word Alignment With Point-Wise Mutual Information Without Requiring Parallel Training Data (2023)

Abstract: Word alignment has many applications including cross-lingual annotation projection, bilingual lexicon extraction, and the evaluation or analysis of translation outputs. Recent studies show that using contextualized embeddings from pre-trained multilingual language models could give us high quality word alignments without the need of parallel training data. In this work, we propose PMI-Align which computes and uses the point-wise mutual information between source and target tokens to extract word alignments, instead of the cosine similarity or dot product which is mostly used in recent approaches. Our experiments show that our proposed PMI-Align approach could outperform the rival methods on five out of six language pairs. Although our approach requires no parallel training data, we show that this method could also benefit the approaches using parallel data to fine-tune pre-trained language models on word alignments. Our code and data are publicly available.

## introduction word alignment, as the task of finding the corresponding source and target tokens in a parallel sentence, was well-known as an essential component of statistical machine translation (smt) systems. despite the dominance of neural machine translation (nmt) in recent years, word alignment is still a notable area of research due to its usage in a wide variety of nlp applications, such as annotation projection @xcite @xcite , bilingual lexicon extraction @xcite @xcite , typological analysis @xcite , guided alignment training of nmt @xcite @xcite , and evaluation and analysis of translation 1 https://github.com/fatemeh-azadi/pmi-align outputs @xcite @xcite . for many years statistical methods such as ibm models @xcite and tools implemented based on them, namely giza++ @xcite or fast-align @xcite , were among the most popular solutions to the word alignment task. following the rise of deep neural models, several attempts have been made to extract word alignments from nmt models and their attention matrices @xcite @xcite . however, most of these methods, as well as the statistical aligners, require a sufficient amount of parallel training data to produce high quality word alignments. recently, jalili sabet et al. (2020) have shown that high quality word alignments could be achieved using pre-trained multilingual language models (lms), like @xcite and @xcite . their proposed method called simalign, extracts word alignments from similarity matrices induced from multilingual contextualized word embeddings with no need for parallel training data, which is very useful for lowresource language pairs. afterwards, dou and neubig (2021) and @xcite proposed methods called probability thresholding and optimal transport to extract alignments using the similarity matrices derived from pre-trained lms. they have also proposed some word alignment objectives to fine-tune the pre-trained models over parallel corpora. in this paper, we follow the work done by jalili @xcite to extract alignments from pre-trained lms without requiring any parallel training data and propose pmi-align. our main contribution is proposing to compute the point-wise mutual information (pmi) between source and target tokens and using the pmi matrices instead of similarity matrices made of cosine similarities between the representation vectors of each source and target tokens, to align words. we argue that our proposed pmi-based method could align better as it considers the total alignment probability of each source or target token, as well as the joint alignment probabilities (equivalent to cosine similarities). this could alleviate the so-called hubness problem @xcite in high dimensional spaces, where some token's representation is close to many others (see _went in figure 1 ). we perform experiments on six different language pairs and show that our method could surpass other alignment methods on five of them. we also conduct our experiments on different pre-trained lms to show that pmi-align could be advantageous regardless of the pre-trained model used. ## proposed method in this section, we first discuss how we define and compute the pmi matrix for each sentence pair and then we describe our alignment extraction method using the pmi matrix. ## related work statistical aligners based on ibm models @xcite , such as giza++ @xcite and fast align @xcite were the most dominant tools for word alignment until the late 2010s. with the rise of neural machine translation models, several attempts made to extract alignments from them @xcite @xcite @xcite . however, all these models need parallel training data and could not utilize pre-trained contextualized embeddings. recently, jalili sabet et al. ## conclusions this paper presents a word alignment extraction method based on the pmi matrices derived from cross-lingual contextualized embeddings, instead of just the similarity matrices. we proposed a way to compute the pmi matrix for each sentence pair and argued that using this pmi measure would be beneficial since for each source-target word pair, it considers not only their similarity to each other but also their similarity values to the other tokens of the sentence, that could mitigate the hubness problem. experimental results show that our pmi-align method could outperform the previous alignment extraction methods in five out of six language pairs, regardless of the base pre-trained language model used to derive word embeddings. although our method does not require any parallel training data, our experiments show that it could also benefit the approaches using such data to fine-tune the pretrained models for better word alignments. in future work, the proposed pmi matrix could be investigated in other cross-lingual or even monolingual applications, like the translation quality estimation or the evaluation of text generation tasks, instead of the similarity matrix.