C omb A lign: a Tool for Obtaining High-Quality Word Alignments (2021)

Abstract: Being able to generate accurate word alignments is useful for a variety of tasks. While statistical word aligners can work well, especially when parallel training data are plentiful, multilingual embedding models have recently been shown to give good results in unsupervised scenarios. We evaluate an ensemble method for word alignment on four language pairs and demonstrate that by combining multiple tools, taking advantage of their different approaches, substantial gains can be made. This holds for settings ranging from very low-resource to high-resource. Furthermore, we introduce a new gold alignment test set for Icelandic and a new easy-to-use tool for creating manual word alignments.

## introduction word alignment, the task of finding corresponding words in a bilingual sentence pair (see figure 1 ), was a key component of statistical machine translation (smt) systems. while word alignments are not necessary for neural machine translation (nmt), various mt methods incorporating word alignment have been found to achieve significant improvements in performance. @xcite and @xcite use alignments as a prior; @xcite augment nmt systems with discrete translation lexicons that encode lowfrequency words; @xcite infer a correspondence between words in sentence pairs before encoding/decoding and, as demonstrated by @xcite , back-translated data created using smt systems, requiring word alignments, can be valuable to augment nmt systems. word alignments have also been utilized to improve automatic post-editing @xcite as well as to preserve markup in machine-translated texts @xcite . various other subfields of nlp make use of word alignments. @xcite show that by simply pipelining word alignment with unsupervised bitext mining, bilingual lexicon induction (bli) quality can be improved significantly. for bli, @xcite use an unsupervised mt pipeline, also employing word alignments. @xcite use word alignments to filter noisy parallel corpora, and @xcite include word alignment as a part of their pipeline to align monolingual comparable documents. there is a variety of word aligners available. giza++ @xcite and fast_align @xcite are easy to use implementations of the ibm models @xcite . other statistical aligners, such as eflomal @xcite , have also been shown to be fast and give competitive results. simalign @xcite takes advantage of the rising availability of contextualized embeddings and leverages them by extracting alignments from similarity matrices. in this work, we present combalign, an ensemble of these four tools (giza++, fast_align, eflomal, and simalign). as they are based on different approaches, and all able to attain a fairly high f 1 -score, it is reasonable to expect that combining their results in a sensible way could give better results than using any one of the individual systems. recently, the first reported results in smt and nmt for icelandic were published @xcite within the context of an icelandic national language technology programme @xcite . icelandic is a morphologically rich west germanic language with relatively few speakers, for which a substantial amount of language resources has been made available in recent years. however, no previous work has been conducted on word alignments for icelandic. while testing our methods on four language pairs, we focus in particular on the effects of different alignment methods on the english-icelandic (en-is) language pair. for finding the best hyperparameters for our ensemble, we thus do a grid search using an en-is development set. our main contribution is showing that it is possible to obtain high-quality word alignments using a combination of selected tools, outperforming all of the individual word alignment tools. we show this for four language pairs, with more detailed scrutiny of the results for one of them, en-is. furthermore, we: the most common statistical word alignment tools are based on the ibm models @xcite . these include fast_align @xcite , giza++ @xcite and eflomal @xcite , all used in this work. the five ibm models use lexical translation probabilities and probability distributions with the different models adding or emphasizing different features to tackle weaknesses of the other models. while fast_align builds on ibm model 2, giza++ iterates on a number of the models in sequence, as well as using an hmm model. eflomal uses a bayesian model with markov chain monte carlo inference on the ibm models. several studies on word alignments in relation to neural models have been published. @xcite show that attention can be seen as a reordering model as well as an alignment model, and @xcite investigate the differences between attention and alignment. @xcite apply stochastic gradient descent to directly optimize the attention activations towards a given target word, resulting in accurate word alignments, and @xcite extract discrete alignments from the attention probabilities learnt during regular nmt training and leverage them to optimize towards translation and alignment objectives. most of these systems require parallel data for training, but simalign (masoud et al., 2020) takes advantage of the rising availability of contextualized embeddings and leverages them by extracting alignments from similarity matrices induced from the embeddings, with no need for any external data. ensemble methods are common in nlp and, in many cases, have been shown to give more accurate results than using just one single approach. they have been used, for example, for classifying patent applications @xcite , for spellchecking @xcite , postagging @xcite and sentiment analysis @xcite . for word alignments, @xcite have previously used a union of two different alignment approaches, each producing distinct alignments. one of their aligners was an implementation of the ibm models, and the other used translation lexicons and phrase boundaries to detect alignments. their combined aligner outperformed both individual systems, and its results produced approximately 10% fewer errors than the better individual aligner. ## data for evaluation, we use gold standard word alignments for four language pairs: czech, german, french and icelandic, all paired with english (encs, en-de, en-fr and en-is, respectively). for the methods trained on parallel data, giza++, fast_align and eflomal, we use a subset of 512k sentences from europarl @xcite , except in the case of icelandic as detailed in section 3.1. further information on the test sets is given in ## methodology in order to find the best settings for each aligner, we carry out a grid search. we run giza++, fast_align and eflomal using different setups. for simalign, we use two different contextual embedding models and run them with different hyperparameters. we are thus working with five different aligners/alignment models. finally, we proceed to find the best ensemble for different levels of parallel data availability. ## experiments and results as described in section 4.1, we identified the optimal settings and post-processing heuristics for each tool using grid search on the dev-set (see ## conclusion and future work we have shown that using a very simple combination method for word alignment, it is possible to increase the accuracy substantially, both in lowand high-resource settings. we evaluated on four language pairs, en-cs, ende, en-fr and for the first time en-is, for which we manually created a new gold standard word alignment reference set. in order to do that we created and published a graphical tool for manual word alignments. while our method uses minimal data processing, some pre-processing like pos-tagging and lemmatizing may raise the accuracy even further, especially in the case of a morphologically rich language like icelandic. a comparison of typical misalignments per aligner is also likely to be beneficial, as knowing these properties might help in combining the aligners more effectively. the mbert and xlm-r models we employ through simalign give good results, but there may still be room for improvement, for instance by pretraining these models on more icelandic texts, which are scarce in the multilingual training corpus. it may also be worth considering to train a bilingual word embedding model and use that for alignment instead of, or in combination with, the other contextualized embedding models. in the paper, we reported on preliminary results from training an smt system using our word alignments. we plan to investigate whether the slightly better smt output will be more beneficial for back-translations to augment nmt systems, following @xcite . we also plan to compare bli quality using the setup in @xcite and the same setup using our alignments. furthermore, we intend to apply our alignments to training alignment-assisted nmt transformer models, by adding an alignment attention layer as described in @xcite .