RAPO : An Adaptive Ranking Paradigm for Bilingual Lexicon Induction (2022)

Abstract: Bilingual lexicon induction induces the word translations by aligning independently trained word embeddings in two languages. Existing approaches generally focus on minimizing the distances between words in the aligned pairs, while suffering from low discriminative capability to distinguish the relative orders between positive and negative candidates. In addition, the mapping function is globally shared by all words, whose performance might be hindered by the deviations in the distributions of different languages. In this work, we propose a novel ranking-oriented induction model RAPO to learn personalized mapping function for each word. RAPO is capable of enjoying the merits from the unique characteristics of a single word and the cross-language isomorphism simultaneously. Extensive experimental results on public datasets including both rich-resource and low-resource languages demonstrate the superiority of our proposal. Our code is publicly available in https://github.com/Jlfj345wf/RAPO .

## introduction bilingual lexicon induction (bli) aims at inducing the word translations across two languages based on the monolingual corpora, which is capable of transferring valuable semantic knowledge between different languages, spawning a myriad of nlp tasks such as machine translation @xcite , semantic parsing @xcite , and document classification @xcite . the nucleus of bli is learning a desirable mapping function to align two sets of independently trained monolingual word embeddings @xcite @xcite . @xcite empirically observe that the linear projections are superior to their non-linear counterparts due to the isomorphism across different embedding spaces. sub-sequent improvements are successively proposed to advance bli task by imposing orthogonal constraints @xcite , normalizing the embeddings @xcite @xcite , reducing the noises @xcite , relaxing the hypothesis of isomorphism @xcite , and iteratively refining the seed dictionary @xcite . existing methods @xcite @xcite ) usually aim at minimizing the distance between the word from the source language and its aligned word in the target language (e.g., crow and cuervo in figure 1 ). however, bli is essentially a ranking-oriented task because for each source word, we expect to select its top k highconfidence target candidates. namely, a desirable bli model should also be capable of distinguishing the relative orders between the positive and negative candidates (e.g., crow and curevo should be distributed closer than crow and pájaro). the objective functions used by previous works solely focus on the distances between positive pairs and cannot explicitly provide such important ranking signals, leading to the low discriminative capability. in addition, conventional bli models @xcite @xcite induce the bilingual space via a shared mapping function, in which different words in the same language tend to be rotated in the same directions. however, several studies @xcite have demonstrated that the isomorphic assumption may not strictly hold true in general, and thus a global-shared mapping is not the optimal solution. as shown in figure 1 , even for two close languages like english and spanish, due to deviations in the distributions of different training corpora and insufficient training of lowfrequency word embeddings, the optimal mapping directions are slightly shifted for different words. therefore, the bli performance could be further improved if we could learn unique or personalized mapping functions for different words. @xcite first propose to achieve the personalized mappings. however, @xcite is a non-parametric model, in which the personalized mappings are unlearnable and built upon the heuristic assumptions, which might be unreliable and suffer from low generality. to address the mentioned limitations under a unified framework, we propose a novel ranking-based model with adaptive personalized offsets, dubbed rapo. different from previous works solely relying on the aligned pairs, rapo is formulated as a ranking paradigm with powerful discriminative capability by incorporating abundant unaligned negative samples. an effective dynamic negative sampling strategy is further proposed to optimize the ranking objectives. moreover, we integrate a novel personalized adapter into rapo to learn unique mapping directions for different words. a straightforward strategy is to directly learn an independent mapping matrix for each word, which is resourceconsuming and ignores the global isomorphism information. differently, our personalized adapter learns the unique offset for each word to calibrate the vanilla embedding, and then a shared mapping function is employed to induct lexicons. by organically integrating personalized offsets with shared mapping functions, rapo enjoys the merits from the unique traits of each word and the global consistency across languages. we further propose a novel householder projection as the mapping function on the basis of householder matrices @xcite , which strictly ensures the orthogonality during the model optimization. we conduct extensive experiments over multiple language pairs in the public muse benchmarks @xcite , including rich-and low-resource languages. experimental results demonstrate that our proposal consistently achieves desirable performance in both supervised and semi-supervised learning settings. our contributions are summarized as follows: ## preliminary let @xmath0, and d stands for the embedding size. @xmath1, y 1 ), ..., (x l , y l )} denotes the available aligned seeds, which can be also formulated as two matrices x d and y @xmath2. bli aims to map the source and target words from their original embedding spaces into a shared latent space, in which the mapped source word ϕ s (x i ) should be close to its matched target word ϕ t (y i ). ϕ s and ϕ t denote the mapping functions for source language and target language, respectively. for the sake of clarification, notations used in this paper are listed in appendix a. a widely adopted solution is to set the source mapping function ϕ s as a linear transformation matrix @xmath3. the objective function is defined as follows: in the inference phase, the distances between the mapped source words and the target ones are utilized as the metrics to select the top-k nearest neighbors as the translations. as discussed in the section of introduction, this popular induction paradigm suffers from two limitations. first, formula (1) only focuses on minimizing the distance between the aligned words (e.g., x i and y i ). however, bli task is essentially a ranking problem, which means that the learned mapping function should also be able to distinguish the aligned pair {x i , y i } and the defective one {x i , y j }. second, the globally shared mapping matrix ## methodology as shown in figure 2 , rapo consists of three major components. given the monolingual embeddings and training seeds, the personalized adapter first generates the adaptive offset for each word by exploiting the contextual semantic information. the vanilla embedding spaces are properly calibrated to fit the induction task. after that, we map the adapted word embeddings to a shared latent space via the novel householder projections, which is capable of ensuring the strict orthogonality and better preserving the isomorphism. finally, rapo designs the ranking objectives to distinguish the aligned pairs from the unmatched ones. rapo can be easily adapted to the supervised and semisupervised settings, demonstrating its flexibility. ## conclusion in this paper, we propose a novel model rapo for bilingual lexicon induction. different from previous works, rapo is formulated as a ranking paradigm, which is more suitable to the nature of studied tasks. two novel modules are further employed by deeply mining the unique characteristics of bli task: the householder projection to ensure the strict orthogonal mapping function, and the personalized adapter to learn unique embedding offsets for different words. extensive experimental results on 20 datasets demonstrate the superiority of our proposal.