S im A lign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings (2020)

Abstract: Word alignments are useful for tasks like statistical and neural machine translation (NMT) and cross-lingual annotation projection. Statistical word aligners perform well, as do methods that extract alignments jointly with translations in NMT. However, most approaches require parallel training data and quality decreases as less training data is available. We propose word alignment methods that require no parallel data. The key idea is to leverage multilingual word embeddings – both static and contextualized – for word alignment. Our multilingual embeddings are created from monolingual data only without relying on any parallel data or dictionaries. We find that alignments created from embeddings are superior for four and comparable for two language pairs compared to those produced by traditional statistical aligners – even with abundant parallel data; e.g., contextualized embeddings achieve a word alignment F1 for English-German that is 5 percentage points higher than eflomal, a high-quality statistical aligner, trained on 100k parallel sentences.

## introduction word alignments are essential for statistical machine translation and useful in nmt, e.g., for imposing priors on attention matrices @xcite @xcite or for decoding @xcite . further, word alignments have been successfully used in a range of tasks such as typological analysis @xcite , annotation projection @xcite @xcite and creating multilingual embeddings @xcite @xcite . ## conclusion we presented word aligners based on contextualized embeddings that outperform in four and match the performance of state-of-the-art aligners in two language pairs; e.g., for eng-deu contextualized embeddings achieve an alignment f 1 that is 5 percentage points higher than eflomal trained on 100k parallel sentences. further, we showed that alignments from static embeddings can be a viable alternative to statistical aligner when few parallel training data is available. in contrast to all prior work our methods do not require parallel data for training at all. with our proposed methods and extensions such as match, itermax and null it is easy to obtain higher precision or recall depending on the use case. future work includes modeling fertility explicitly and investigating how to incorporate parallel data into the proposed methods. zvi galil. 1986. efficient algorithms for finding maximum matching in graphs. acm computing surveys (csur), 18(1). sarthak garg, stephan peitz, udhyakumar nallasamy, and matthias paulik. 2019. jointly learning to align and translate with transformer models. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), hong kong, china. association for computational linguistics. ulrich germann. 2001. aligned hansards of the 36th parliament of canada. hamidreza ghader and christof monz. 2017. what does attention in neural machine translation pay attention to? in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), taipei, taiwan. asian federation of natural language processing. jiang guo, wanxiang che, david yarowsky, haifeng wang, and ting liu. 2016. a representation learning framework for multi-source transfer parsing. in thirtieth aaai conference on artificial intelligence. john hewitt and christopher d. manning. 2019. a structural probe for finding syntax in word representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), minneapolis, minnesota. association for computational linguistics. anh khoa ngo ho and franc ¸ois yvon. 2019. neural baselines for word alignment. in proceedings of the 16th international workshop on spoken language translation. matthias huck, diana dutka, and alexander fraser. 2019. cross-lingual annotation projection is effective for neural part-of-speech tagging. in proceedings of the sixth workshop on nlp for similar languages, varieties and dialects, pages 223-233, ann arbor, michigan. association for computational linguistics. masoud jalili sabet, heshaam faili, and gholamreza haffari. 2016. improving word alignment of rare words with word embeddings. in proceedings of coling 2016, the 26th international conference on computational linguistics: technical papers, osaka, japan. the coling 2016 organizing committee. philipp koehn. 2005. europarl: a parallel corpus for statistical machine translation. in machine translation summit, volume 5. philipp koehn, amittai axelrod, alexandra birch mayne, chris callison-burch, miles osborne, and