An Alignment-Based Approach to Semi-Supervised Bilingual Lexicon Induction with Small Parallel Corpora (2021)

Abstract: Aimed at generating a seed lexicon for use in downstream natural language tasks and unsupervised methods for bilingual lexicon induction have received much attention in the academic literature recently. While interesting and fully unsupervised settings are unrealistic; small amounts of bilingual data are usually available due to the existence of massively multilingual parallel corpora and or linguists can create small amounts of parallel data. In this work and we demonstrate an effective bootstrapping approach for semi-supervised bilingual lexicon induction that capitalizes upon the complementary strengths of two disparate methods for inducing bilingual lexicons. Whereas statistical methods are highly effective at inducing correct translation pairs for words frequently occurring in a parallel corpus and monolingual embedding spaces have the advantage of having been trained on large amounts of data and and therefore may induce accurate translations for words absent from the small corpus. By combining these relative strengths and our method achieves state-of-the-art results on 3 of 4 language pairs in the challenging VecMap test set using minimal amounts of parallel data and without the need for a translation dictionary. We release our implementation at www.blind-review.code.

## introduction unsupervised methods for machine translation (mt) and bilingual lexicon induction (bli) have received considerable attention in recent years, showing impressive performance without bilingual data for supervision. while academically interesting, small amounts of supervised data can almost always help model performance. the typical use case for unsupervised bli is to provide initial synthetic training data for a traditional supervised setup where no parallel bitext exists, such as for mt or cross-lingual information retrieval. a starting lexicon is induced in an unsupervised manner, and then serves as initial training data to the supervised model. practically, however, one struggles to identify a scenario where one would truly fail to have any parallel text whatesoever from which to gain some supervision. the christian bible, for instance, is translated into over 1600 world languages, providing multi-way parallel data for many of the world's languages that are typically considered "low-resource" @xcite . human translators can also create a small translation corpus or seed dictionary. the practical necessity of fully unsupervised scenarios for bli or mt therefore becomes hard to imagine. statistical translation/alignment models are very proficient at inducing bilingual lexicons from small amounts of parallel data. particularly when words occur frequently in the corpus, statistical models easily recover the translation. at the same time, however, the number of seed translation pairs possible to extract is limited by the vocabulary of the parallel corpus. we address a more realistic scenario: there is ample monolingual data and a small parallel corpus. we combine the strengths of statistical alignment and unsupervised mapping methods and achieve state-of-the-art results on 3 of 4 languages in the challenging vecmap dataset @xcite @xcite , trailing by only 0.1 in the 4th language pair. ## related work automatic bli has been a popular task in natural language processing for decades, beginning with statistical decipherment (e.g., @xcite @xcite @xcite . with the advent of the ability to create large monolingual vector spaces from abundant monolingual text, the focus has shifted to finding an optimal linear transformation between such monolingual embedding spaces from which a seed lexicon can be extracted using nearest neighbors search. practically, this often involves solving variations of the generalized procrustes problem (e.g., @xcite @xcite @xcite @xcite @xcite . differing metrics and heuristics can be used to extract the seed lexicon once the mapping is found. cross-domain similarity local scaling (csls) to mitigate the hubness is popular and effective @xcite . while the orthogonal variant of the procrustes problem has a simple closed-form solution, one must know in advance the pairings of words one wants to be closest after the transformation (i.e., you already know the translations). to adapt to the unsupervised or semi-supervised scenario, such mapping-based bli procedures must make a "guess" of some correct translation pairs. the solution can then iteratively refined through self-learning. the initial "guess" can come in the form of direct supervision using a bilingual training dictionary, or in an unsupervised manner, such as by identifying the nearest neighbors in a similarity matrix (e.g., @xcite or via adversarial training (e.g., @xcite . like us, @xcite also use statistical alignment within a pipeline for bli, but unlike our work, they do not use the induced alignments as seeds for monolingual embedding mapping. ## experimental settings language corpus # of words english wacky, bnc, wikipedia 2.8 b italian itwac 1.6 b german sdewac 0.9 b spanish news crawl 2007-2012 386 m finnish common crawl 2016 2.8 b ## results and analysis our main results compared with the existing literature are presented in ## conclusion motivated by the strength of statistical translation and alignment models in inducing accurate word translation pairs from small amounts of data, the breadth of training data used to train monolingual word embedding spaces, we propose a motivated semi-supervised approach for bilingual lexicon induction that demonstrates state-of-the-art results on the challenging vecmap test sets. we capitalize upon the complementary strengths of statistical alignment and embedding space mapping methods for generating translation dictionaries, combining the methods for better downstream bilingual lexicon induction performance than either achieves alone. by taking this middle ground, we achieve state-of-the-art results with as little as 5,000 sentencesan amount readily available in thousands of language pairs. we release our implementation at https://github.com/kellymarchisio/align-semisup-bli .