Improving Bilingual Lexicon Induction with Unsupervised Post-Processing of Monolingual Word Vector Spaces (2020)

Abstract: Work on projection-based induction of cross-lingual word embedding spaces (CLWEs) predominantly focuses on the improvement of the projection (i.e., mapping) mechanisms. In this work, in contrast, we show that a simple method for post-processing monolingual embedding spaces facilitates learning of the cross-lingual alignment and, in turn, substantially improves bilingual lexicon induction (BLI). The post-processing method we examine is grounded in the generalisation of first- and second-order monolingual similarities to the nth-order similarity. By post-processing monolingual spaces before the cross-lingual alignment, the method can be coupled with any projection-based method for inducing CLWE spaces. We demonstrate the effectiveness of this simple monolingual post-processing across a set of 15 typologically diverse languages (i.e., 15*14 BLI setups), and in combination with two different projection methods.

## introduction cross-lingual word embeddings (clwes) are a mainstay of modern cross-lingual nlp @xcite . clwe models induce a shared cross-lingual vector space in which words with similar meanings obtain similar vectors regardless of their language. their usefulness has been attested in tasks such as bilingual lexicon induction (bli) @xcite , information retrieval @xcite , machine translation @xcite , document classification @xcite , and many others @xcite . importantly, clwes are one of the central mechanisms for facilitating transfer of language technologies for low-resource languages, which often lack sufficient bilingual signal for obvious transfer via machine translation. lack of language re-sources is the main reason for popularity of the socalled projection-based clwe methods @xcite @xcite . these models align two independently trained monolingual word vector spaces post-hoc, using limited bilingual supervision in the form of several hundred to several thousand word translation pairs @xcite @xcite . some models even align the monolingual spaces using only identical strings @xcite or numerals @xcite . the most recent work focused on fully unsupervised clwe induction: they extract seed translation lexicons relying on topological similarities between monolingual spaces @xcite @xcite . in this work, we do not focus on projection itself: rather, we investigate a transformation of input monolingual word vector spaces that facilitates the projection and leads to higher quality clwes. regardless of the actual projection method, the quality of the input monolingual spaces has a profound impact on the induced shared cross-lingual space, and, in turn, on the quality of induced bilingual lexicons. we demonstrate that simple unsupervised post-processing of monolingual embedding spaces leads to substantial bli performance gains across a large number of language pairs. our work is inspired by observations that monolingual "embeddings capture more information than what is immediately obvious" @xcite . in other words, the information surfaced in the pretrained monolingual vector spaces may not be optimal for an application such as word-level translation (bli). we rely on a monolingual post-processing method of @xcite : a linear transformation controlled by a single parameter that adjusts the similarity order of the input embedding spaces. we demonstrate that applying this trans-formation on both monolingual spaces before any standard projection-based clwe framework yields consistent bli gains for a wide array of languages. we run a large-scale bli evaluation with 15 typologically diverse languages (i.e., 15×14 = 210 bli setups) and show that this simple monolingual postprocessing yields gains in 183/210 setups over the current state-of-the-art bli models which combine self-learning @xcite with (weak) word-level supervision @xcite . we further show that this monolingual post-processing yields improvements on other bli datasets @xcite , for different projection-based clwe models, and also for bli with 210 similar (major european) languages @xcite , indicating the importance and robustness of monolingual post-processing for bli. ## methodology projection-based clwes: preliminaries. projection-based clwe models learn a linear projection between two independently trained monolingual spaces -x (source language l s ) and z (target language l t ) -using a word translation dictionary d to guide the alignment. x d ⊂ x and z d ⊂ z denote the row-aligned subsets of x and z containing vectors of aligned words from d. x d and z d are used to learn orthogonal projections w x and w z defining the bilingual space: @xmath0. while (weakly) supervised methods start from a readily available dictionary d, fully unsupervised models automatically induce the seed dictionary d (i.e., from monolingual data). 1 furthermore, it has been empirically validated @xcite that applying an iterative self-learning procedure leads to consistent bli improvements, especially for distant languages and in low-data regimes. in a nutshell, at each self-learning iteration k, a dictionary d (k) is first used to learn the joint space k) are then used to extract the new dictionary d (k+1) . relying on mutual nearest neighbours partially removes the noise, leading to better performance. for more technical details on self-learning, we refer the reader to prior work @xcite . motivation. most existing clwe models ignore the properties of the initial monolingual spaces x and z (i.e., they are taken "as-is") and focus on improving the projection. however, monolingual postprocessing of x and z prior to learning the projections may facilitate the projection and be beneficial for iterative setups such as self-learning. this intuition is already confirmed by a number of monolingual transformations, e.g., ℓ 2 -normalisation, mean centering, or whitening/dewhitening, that are "by default" performed by toolkits such as muse @xcite and vecmap @xcite . in this work, however, we investigate a transformation to the monolingual spaces which is applied before they undergo the series of standard normalisation and centering steps. further, we investigate a line of research that leverages unsupervised post-processing of monolingual word vectors @xcite @xcite to emphasise semantic properties over syntactic aspects, typically with small gains reported on intrinsic word similarity (e.g., simlex-999 @xcite ). in this work, we empirically validate that these unsupervised post-processing techniques can also be effective in cross-lingual scenarios for lowresource bli, even when coupled with the current state-of-the-art clwe frameworks that rely on "all the bells and whistles", such as self-learning and additional vector space preprocessing. ## experimental setup we evaluate the impact of unsupervised monolingual post-processing described in §2 on bli, focusing on pairs of typologically diverse languages. 4 mean reciprocal rank (mrr) is used as the main evaluation metric, reported as mrr×100%. 5 training and test data. we exploit the training and test dictionaries compiled from panlex @xcite by @xcite : the data encompasses 15 diverse languages listed in monolingual embeddings. we use the 300-dim vectors of @xcite for all languages, pretrained on common crawl and wikipedia with fasttext @xcite . 8 all vocabularies are trimmed to the 200k most frequent words. projection-based framework. we base the induction of projection-based clwes on the wellknown vecmap framework @xcite ); 9 it shows very competitive and robust bli performance, especially for distant pairs, according to the recent comparative studies @xcite @xcite . we analyse the impact of unsupervised monolingual postprocessing from §2 by (1) feeding the original vectors x and y to vecmap (baseline), and then by (2) feeding their post-processed variants x αs and y αt (postproc). we experiment with projection model variants without and with self-learning, and with different initial dictionary sizes (5k and 1k). note that the postproc variant requires tuning of two hyper-parameters: α s and α t . due to a lack of development sets for bli experiments, we tune the two α-parameters on a single language pair (bg-ca) via crossvalidation; we grid-search over the following values: [-0.5, -0.25, -0.15, 0, 0.15, 0.25, 0.5]. we then keep them fixed to the following values: α @xmath2.25, α @xmath3.15 in all subsequent experiments. ## results and discussion main bli results averaged over each source language (l s ) are provided in the scores are averaged over experimental setups where each of the 15 languages is used as the source language l s (e.g., bg-* averages scores over 14 setups in which bulgarian (bg) is the source language). 5k and 1k denote seed dictionary sizes. the avg column shows averaged mrr scores for each model over all 15×14=210 bli setups and we also report the number of bli setups in which the postproc method improves over both baseline models. the supplemental material. we also observe performance gains with a "pure" supervised model variant (i.e., without self-learning), but for clarity, we focus our analysis on the more powerful baseline, with self-learning. we note improvements in 183/210 (seed dictionary size 5k) and 181/210 bli setups (size: 1k) over the projection-based baselines that held previous peak scores using the same data @xcite . this validates our intuition that monolingual vectors store more information which needs to be "uncovered" via monolingual post-processing. the effect of monolingual postprocessing pertains after applying other perturbations such as ℓ 2 -norm or mean centering. for some languages -e.g., fi, tr, no -we achieve gains in all bli setups with those languages as sources. what is more, we have not carefully fine-tuned α s and α t : we note that even higher scores can be achieved by finer-grained fine-tuning in the future. for instance, setting (α s , α t ) = (-0.5, 0.25) instead of (-0.25, 0.15) for tr-bg increases bli score from 37.8 to 39.5; the previous peak score with baseline was 35.1. the baseline mapping is simply obtained by setting (α s , α t ) = (0, 0), and we note that the tuned post-processing validated in our work should be considered as a tunable option for any projection-based clwe method. we further probe the robustness of unsupervised post-processing by running experiments on additional bli evaluation set of @xcite and with another mapping model: rcsls @xcite . while we again observe gains across a range of different model variants and with different seed dictionary sizes, we summarise a selection of results in ## conclusion and future work we have demonstrated a simple and effective method for improving bilingual lexicon induction (bli) with projection-based cross-lingual word embeddings. the method is based on standalone unsupervised post-processing of initial monolingual word embeddings before mapping, and as such applicable to any projection-based clwe method. we have verified the importance and robustness of this monolingual post-processing with a wide range of (dis)similar language pairs as well as in different bli setups and with different clwe methods. in future work, we will test other unsupervised post-processors, and also probe similar methods that inject external lexical knowledge into monolingual word vectors towards improved bli. we also plan to probe if similar gains still hold with recently proposed more sophisticated self-learning methods @xcite , non-linear mappingbased clwe methods @xcite . another idea is to also apply a similar principle to contextualised word representations in cross-lingual settings @xcite .