Embedding-Enhanced GIZA ++: Improving Low-Resource Word Alignment Using Embeddings (2022)

Abstract: A popular natural language processing task decades ago, word alignment has been dominated until recently by GIZA++, a statistical method based on the 30-year-old IBM models. New methods that outperform GIZA++ primarily rely on large machine translation models, massively multilingual language models, or supervision from GIZA++ alignments itself. We introduce Embedding-Enhanced GIZA++, and outperform GIZA++ without any of the aforementioned factors. Taking advantage of monolingual embedding spaces of source and target language only, we exceed GIZA++’s performance in every tested scenario for three languages pairs. In the lowest-resource setting, we outperform GIZA++ by 8.5, 10.9, and 12 AER for RoEn, De-En, and En-Fr, respectively. We release our code at www.blind-review.code.

## introduction word alignment techniques were once ubiquitous in the machine translation (mt) literature, as they formed a critical part of statistical machine translation (smt) systems. since the advent of neural machine translation (nmt), word alignment is no longer a step in typical nmt training, but is still important for other tasks such as annotation transfer (e.g. @xcite , as a post-processing step of mt to reinsert markup (e.g. @xcite , and for some mapping-based unsupervised mt methods such as @xcite . giza++ @xcite , a statistical alignment model, has been the most commonly used tool for word alignment quality for 20 years and is based the ibm translation models that are yet a decade older @xcite . though a handful of neural systems have outperformed giza++, these rely on large mt models (e.g. @xcite @xcite , massively multilingual language models (e.g. @xcite @xcite , supervision from human-annotated alignments @xcite , or combinations of the above. we introduce embedding-enhanced giza++ (ee-giza++), an improvement to giza++ without any of the aforementioned factors. ee-giza++ biases giza++ to align semantically similar words from a shared embedding space. we outperform giza++ in all tested settings on three language pairs. ee-giza++ is particularly strong in comparison with giza++ when parallel training data is scarce: using only ∼500 lines of bitext, it outperforms giza++ by 10.9 aer foot_0 and 12.0 aer for de-en and fr-en, respectively. ## related work fast-align is a statistical aligner similar to giza++. it is a reparameterization of ibm model 2 @xcite . eflomal is another highly-performant non-neural aligner @xcite . we use giza++ as our base system because it commonly-used and trusted for generating high-quality alignments. numerous improvements to giza++ have been proposed (e.g. @xcite . recent work involves using neural translation models to guide or extract alignments, viewing attention as a proxy for alignment (e.g. @xcite @xcite @xcite . other aligners use massive multilingual language models with contextualized embeddings such as mbert @xcite . like us, jalili @xcite experiment with mapped monolingual embedding spaces, but exceed the giza++ baseline only when using spaces such as mbert and xlm-r @xcite . @xcite 's approach is similar to the aforementioned authors, but they improve results by finetuning mbert on auxiliary tasks. @xcite use mbert and require supervision with human-annotated alignments. @xcite use word embedding similarity to augment parallel data seen by giza++, improving alignment and downstream low-resource mt. jalili @xcite also use nearest-neighbors in a word embedding space to alter ibm model 1, but their performance does not match ours. perhaps most similar to our work, @xcite incorporate word similarity into giza++ using a feedforward neural network trained to model word similarity, with a hyperparameter to control the influence of the neural model. ## background let s be a source-language sentence of tokens (s 1 , s 2 , ..., s m ) and t be a target-language sentence (t 1 , t 2 , ..., t l ). alignments are defined as a ⊆ {(s, t) ∈ s × t } where each s, t are meaningfully related-usually, translations of one another. performance is typically measured with alignment error rate (aer; @xcite . ## experimental setup we use the same training setup as previous work foot_5 @xcite @xcite @xcite . training corpora for german-english (de-en), english-french (en-fr), and romanian-english (ro-en) are 1.9m, 1.1m, and 448k lines, and test sets are 508, 447, and 248 lines, respectively. validation sets do not exist, so we tune λ on 1 million lines of de-en. foot_6 λ is set to 10,000. we use the vecmap implementation of csls and scipy for some utility functions and softmax calculation @xcite . for pretrained word embedding spaces, we use the publicly-available wikipedia word vectors trained using fasttext from @xcite . 6 we limit vocabulary size to 200,000 and perform embedding mapping with vecmap in unsupervised mode. ## results the main results are presented in ## conclusion and future work we introduce ee-giza++, an unsupervised enhancement to giza++ that uses word embeddings for improved word alignment in low-bitext settings, without the use of nmt or massivelymultilingual language models that to-date have been the strongest competitors to giza++. ee-giza++ outperforms giza++ by 8.5, 10.9, and 12 aer in lowest-bitext scenarios for ro-en, de-en, and en-fr, respectively. future work should examine performance of ee-giza++ on a diverse set of languages with varying scripts and amounts of data available.