How Lexical is Bilingual Lexicon Induction? (2024)

Abstract: In contemporary machine learning approaches to bilingual lexicon induction (BLI), a model learns a mapping between the embedding spaces of a language pair. Recently, retrieve-and-rank approach to BLI has achieved state of the art results on the task. However, the problem remains challenging in low-resource settings, due to the paucity of data. The task is complicated by factors such as lexical variation across languages. We argue that the incorporation of additional lexical information into the recent retrieve-and-rank approach should improve lexicon induction. We demonstrate the efficacy of our proposed approach on XLING, improving over the previous state of the art by an average of 2% across all language pairs.

## introduction bilingual lexicon induction (bli) is fundamental to many downstream nlp applications, such as machine translation @xcite , cross-lingual information retrieval @xcite , document classification @xcite , dependency parsing @xcite , and language acquisition and learning @xcite . in addition, it facilitates model sharing between highresource and their aligned low-resource languages. contemporary approaches to bli involve alignment of embeddings trained on monolingual corpora into a shared vector space. a challenge of this approach is hubness -the problem of high density regions in cross-lingual embedding space where, in the alignment space, the embedding of a term in a source language is surrounded by a dense cluster of terms in the target language. these hub terms are difficult to align and are worthy of investigation. the recent cross-domain similarity local scaling (csls) addresses this by normalizing distances by the average distance of each term's embedding to its nearest neighbors @xcite we propose instead to address the hubness prob-048 lem by including simple lexical features. we start 049 with the observation that the lexical similarity of 050 a pair of languages tends to be indicated by a rela-051 tively high rank correlation of term frequency, par-052 ticularly for certain parts of speech. figure 1 shows, 053 by part of speech, the spearman's rank correlation 054 of corresponding terms in the 5k vocabularies in 055 the xling corpus @xcite . all lan-056 guage pairs have a positive rank correlation. this is 057 especially so for proper nouns (propn) and nouns respondences respectively for their training. in the semi-supervised setting, high-confidence alignments at each step are iteratively added as anchor points for subsequent training runs. results on semi-supervised bli have shown to improve by adopting a classification-based approach to iteratively refine and augment the seed translation dictionary @xcite . such an approach allows for including arbitrary features such as term frequencies and sub-word information. recent semi-supervised and supervised approaches include contrastivebli @xcite and blicer which achieve state of the art results and serve as strong baselines for our work. con-trastivebli uses a familiar bi-encoder setup with hard negative sampling and contrastive learning. two configurations for the bi-encoder are used: c1: fine-tuned bi-encoder on static fasttext @xcite embeddings c2: fine-tuned bi-encoder on multi-lingual bert @xcite . c2 involves an additional step of a procrustes mapping from c1 (300dim) to the fine-tuned bert (768-dim) embedding. the final embeddings are then a linear combination of the projected c1 and bert representation. blicer further improves performance through a reranking step using a fine-tuned cross-encoder based on xlm-roberta-large @xcite . we maintain the simple binary objective in all our the most frequent pos types. this is consistent 214 with our expectations in 1. finally blicer re-215 ports results from using a linear combination of 216 similarity scores using the cross-encoder as well as 217 the clwe backbone. for a more direct compari-218 son, we do the same with our clwe retriever (c1) 219 which helps improve model performance across the 220 board. our approach yields improved results even 221 in the absence of this additional step. 222 in figure 2, we visualize a random sample of 50 223 baseline error cases in the en-de test set corrected 224 by our letor model with (right) and without 225 (left) the additional lexical features. through the 226 re-scaling of size of points with the probability of 227 the pos matching, and transparency by frequency 228 difference between source and candidate pairs, we 229 observe how these features help the target stand 230 out better in the right panel. this illustrates how 231 our method tackles the hubness issue. while it is 232 hard to disambiguate between close candidates in 233 the embedding space, the letor model is able to 234 turn to external cues in the form of these lexical 235 features to help it make better predictions. 236 figure 3: mean absolute difference of term frequency. to hone in on how the use of these lexical features affects a model, we do a post-hoc error analysis of our model on the test set using mean absolute difference of term frequency. figure 3 shows the frequency difference in en-* pairs for the gold set and all error cases of xlm-r and letor. xlm-r consistently has higher frequency difference between source-predicted pairs. conversely, predictions from the letor model have a frequency deviation that is more in-line with the gold distribution illustrating the models' higher proclivity to choose candidates with similar frequency. ## conclusion approaches to bli have evolved to include full the target words are much closer to the source 431 words in relative frequency as shown by their ranks.