Improving Bilingual Lexicon Induction with Cross-Encoder Reranking (2022)

Abstract: Bilingual lexicon induction (BLI) with limited bilingual supervision is a crucial yet challenging task in multilingual NLP. Current state-of-the-art BLI methods rely on the induction of cross-lingual word embeddings (CLWEs) to capture cross-lingual word similarities; such CLWEs are obtained <b>1)</b> via traditional static models (e.g., VecMap), or <b>2)</b> by extracting type-level CLWEs from multilingual pretrained language models (mPLMs), or <b>3)</b> through combining the former two options. In this work, we propose a novel semi-supervised <i>post-hoc</i> reranking method termed <b>BLICEr</b> (<b>BLI</b> with <b>C</b>ross-<b>E</b>ncoder <b>R</b>eranking), applicable to any precalculated CLWE space, which improves their BLI capability. The key idea is to ‘extract’ cross-lingual lexical knowledge from mPLMs, and then combine it with the original CLWEs. This crucial step is done via <b>1)</b> creating a word similarity dataset, comprising positive word pairs (i.e., true translations) and hard negative pairs induced from the original CLWE space, and then <b>2)</b> fine-tuning an mPLM (e.g., mBERT or XLM-R) in a cross-encoder manner to predict the similarity scores. At inference, we <b>3)</b> combine the similarity score from the original CLWE space with the score from the BLI-tuned cross-encoder. BLICEr establishes new state-of-the-art results on two standard BLI benchmarks spanning a wide spectrum of diverse languages: it substantially outperforms a series of strong baselines across the board. We also validate the robustness of BLICEr with different CLWEs.

## introduction and motivation bilingual lexicon induction (bli) or word translation is one of the core tasks in multilingual nlp @xcite @xcite , with its applications spanning machine translation @xcite , language acquisition and learning @xcite , as well as supporting nlp tasks in low-resource scenarios @xcite , among others. the predominant approach to bli is based on the induction of a shared cross-lingual word embedding (clwe) semantic space: word translation is then tackled by searching for the nearest neighbour in the other language. recent bli work has largely focused on the so-called mapping-based or projection-based methods @xcite . their prime advantage is strong performance coupled with largely reduced bilingual supervision, typically spanning only 1k-5k seed word pairs @xcite . this makes them fitting for weakly supervised setups and low-resource languages. in parallel, cross-encoders (ces) have gained popularity in sentence-level nlp tasks that involve pairwise sentence comparisons. unlike the socalled embedding-based models (also called biencoders or dual-encoders) which process two sequences independently and in parallel to create their embeddings and only model their late interaction @xcite , ces take the concatenation of two sequences as input and directly predict the similarity of the two sequences @xcite . as the self-attention heads in ces can simultaneously attend to tokens from both sequences, ces are considered more power-ful sequence-pair models than embedding-based models which can only perform post-hoc comparisons in the embedding space. a large volume of evidence suggests that, under the same amount of supervision, ces typically substantially outperform embedding-based models in information retrieval @xcite , dialogue @xcite , and semantic similarity tasks @xcite , and their benefits are especially pronounced in low-data regimes with limited task supervision @xcite . motivated by the work on ces in sentence-level tasks, in this work, we propose to use ces to benefit bli. in a nutshell, we aim to expose useful word translation knowledge from multilingual pretrained language models (mplms) such as mbert or xlm-r via bli-oriented ce fine-tuning; this knowledge complements the knowledge stored in the clwes. we demonstrate that ces can be effectively leveraged with cross-lingual word pairs, learning more accurate cross-lingual word similarity scores required for bli. we present blicer (bli with cross-encoder reranking), a post-hoc reranking method for bli, illustrated in figure 1 and described in detail in §3. blicer requires no additional supervision beyond the seed dictionary used for inducing (i.e., mapping) the clwe space, and it can be combined with any existing clwe approach, boosting their bli performance. we conduct extensive bli experiments on two standard bli benchmarks spanning a diverse language sample, covering 44 translation directions and a total of 352 different bli setups. we observe large and consistent improvements brought about by blicer across the board: we report gains in 351 out of the 352 bli setups over the very recent and strong bli baseline of @xcite , establishing new state-of-the-art (sota) performance. we also empirically validate that blicer is universally useful, yielding gains with different 'clwe backbones', and run a series of insightful ablations to verify the usefulness of individual components involved in the blicer design. our code is publicly available at https: //github.com/cambridgeltl/blicer . cross-encoders. they have wide applications in text matching @xcite , semantic textual similarity @xcite , and cross-modal retrieval @xcite . they typically outperform the class of bi-encoder models @xcite , but are much more time-consuming and even prohibitively expensive to run for retrieval tasks directly @xcite . while mplms as bi-encoders for bli have been studied in very recent research @xcite , to the best of our knowledge, blicer is the first work to leverage ces for the bli task. ## methodology clwes. let @xmath0, where l x words are represented by real-valued vectors/clwes from x, and the representations of l y words are provided in y . in other words, the d-dimensional row vector x i from x correspond to the specific word w x @xmath1, and the same holds for the target language. a plethora of different methods with various data requirements and bilingual supervision can be used to induce such clwes @xcite . most commonly, due to reduced bilingual supervision requirements, the clwes are induced by (i) pretraining monolingual word embeddings independently in two languages, and then (ii) mapping them by linear @xcite @xcite or non-linear transformations @xcite , minimising the distance between the original monolingual word embedding spaces. optionally, such static clwes can be combined or enhanced with external word-level knowledge such as word translation knowledge embedded in multilingual language models @xcite @xcite . the actual similarity function f (π) is detached from the chosen clwe method to obtain x and y . following prior work @xcite , f (π) is the cross-domain similarity local scaling (csls) measure @xcite between the associated embeddings @xmath2: here, cos denotes the cosine similarity, γ x (y) is the average cosine similarity between y and its k nearest neighbours (typically @xmath3; γ y (x) is defined similarly. csls is a standard similarity function in bli which typically outperforms the 'vanilla' cosine similarity, as it mitigates the hubness problem during inference. foot_2 3.2 blicer: cross-encoder reranking method in a nutshell. the proposed blicer method is illustrated in figure 1 . the main idea is to refine the initial cross-lingual word similarity scores obtained from the original clwe space (see eq. 1). in particular, assuming the seed dictionary d s and the precalculated clwes, we first derive positive translation pairs d p ⊇ d s (true translation pairs) and hard negative pairs d n (semantically similar words that do not constitute a real translation pair). we then polarise the scores for both d p and d n word pairs: the polarisation step effectively increases semantic similarity scores between positives and decreases them for negatives. we then use the polarised scores to fine-tune any mplm (e.g., mbert or xlm-r) to transform them into bli-oriented cross-encoders: that is, we provide word pairs as input, aiming to predict the correct similarity score. finally, the mplms, now transformed into bli-focused cross-encoders, produce cross-lingual similarity scores for unseen word pairs, which work in synergy with and refine the similarity scores produced by the original cross-lingual word embeddings. in what follows, we describe the main components of the full blicer post-processing method. ## conclusion we presented blicer, a simple and effective post-hoc reranking method for improved bilingual lexicon induction (bli). blicer is applicable to any underlying cross-lingual word embedding (clwe) space. it is based on fine-tuning multilingual pretrained language models into blioriented cross-encoders with a limited amount of direct bilingual supervision (i.e., seed word translation pairs). at bli inference, the blicer output refines cross-lingual word similarities from the underlying clwe space. we conducted extensive empirical studies covering a total of 352 supervised and semi-supervised bli setups, and observe substantial gains against representative and strong bli baselines across the board. we also performed a series of ablation studies and validated the unsupervised and zero-shot capabilities of blicer. in future research we plan to experiment with other multilingual language models @xcite and their ensembles, and we will extend the work to other languages and multilingual lexical tasks. this work has been partially supported by the erc poc grant multiconvai (no. 957356) and a research donation from huawei. yl and fl are supported by grace & thomas c. h. chan cambridge international scholarship. iv is also supported by a personal royal society university research fellowship.