Improving Word Translation via Two-Stage Contrastive Learning (2022)

Abstract: Word translation or bilingual lexicon induction (BLI) is a key cross-lingual task, aiming to bridge the lexical gap between different languages. In this work, we propose a robust and effective two-stage contrastive learning framework for the BLI task. At Stage C1, we propose to refine standard cross-lingual linear maps between static word embeddings (WEs) via a contrastive learning objective; we also show how to integrate it into the self-learning procedure for even more refined cross-lingual maps. In Stage C2, we conduct BLI-oriented contrastive fine-tuning of mBERT, unlocking its word translation capability. We also show that static WEs induced from the ‘C2-tuned’ mBERT complement static WEs from Stage C1. Comprehensive experiments on standard BLI datasets for diverse languages and different experimental setups demonstrate substantial gains achieved by our framework. While the BLI method from Stage C1 already yields substantial gains over all state-of-the-art BLI methods in our comparison, even stronger improvements are met with the full two-stage framework: e.g., we report gains for 112/112 BLI setups, spanning 28 language pairs.

## introduction and motivation bilingual lexicon induction (bli) or word translation is one of the seminal and long-standing tasks in multilingual nlp @xcite @xcite . its main goal is learning translation correspondences across languages, with applications of bli ranging from language learning and acquisition @xcite to machine translation @xcite @xcite and the development of language technology in low-resource languages and domains @xcite . a large body of recent bli work has focused on the so-called mappingbased methods @xcite , 2018; @xcite . 1 such methods are particularly suitable for low-resource languages and weakly supervised learning setups: they support bli with only as much as few thousand word translation pairs (e.g., 1k or at most 5k) as the only bilingual supervision @xcite . 2 unlike for many other tasks in multilingual nlp @xcite @xcite , state-of-the-art (sota) bli results are still achieved via static word embeddings (wes) @xcite . a typical modus operandi of mapping-based approaches is to first train monolingual wes independently on monolingual corpora and then map them to a shared cross-lingual space via linear (mikolov et al., 2013; 1 they are also referred to as projection-based or alignmentbased methods @xcite . 2 in the extreme, fully unsupervised mapping-based bli methods can leverage monolingual data only without any bilingual supervision @xcite @xcite @xcite . however, comparative empirical analyses @xcite show that, with all other components equal, using seed sets of only 500-1,000 translation pairs, always outperforms fully unsupervised bli methods. therefore, in this work we focus on this more pragmatic (weakly) supervised bli setup @xcite ; we assume the existence of at least 1,000 seed translations per each language pair. @xcite or non-linear mapping functions @xcite . in order to achieve even better results, many bli methods also apply a self-learning loop where training dictionaries are iteratively (and gradually) refined, and improved mappings are then learned in each iteration @xcite . however, there is still ample room for improvement, especially for lower-resource languages and dissimilar language pairs @xcite . on the other hand, another line of recent research has demonstrated that a wealth of lexical semantic information is encoded in large multilingual pretrained language models (lms) such as mbert @xcite , but 1) it is not straightforward to transform the lms into multilingual lexical encoders @xcite , 2) extract word-level information from them @xcite , and 3) word representations extracted from these lms still cannot surpass static wes in the bli task @xcite . motivated by these insights, in this work we investigate the following research questions: (rq1) can we further improve (weakly supervised) mapping-based bli methods based on static wes? (rq2) how can we extract more useful crosslingual word representations from pretrained multilingual lms such as mbert or mt5? (rq3) is it possible to boost bli by combining cross-lingual representations based on static wes and the ones extracted from multilingual lms? inspired by the wide success of contrastive learning techniques in sentence-level representation learning @xcite @xcite , we propose a twostage contrastive learning framework for effective word translation in (weakly) supervised setups; it leverages and combines multilingual knowledge from static wes and pretrained multilingual lms. stage c1 operates solely on static wes: in short, it is a mapping-based approach with self-learning, where in each step we additionally fine-tune linear maps with contrastive learning that operates on gradually refined positive examples (i.e., true translation pairs), and hard negative samples. stage c2 fine-tunes a pretrained multilingual lm (e.g., mbert), again with a contrastive learning objective, using positive examples as well as negative examples extracted from the output of c1. finally, we extract word representations from the multilingual lm fine-tuned in stage c2, and combine them with static cross-lingual wes from stage c1; the combined representations are then used for bli. we run a comprehensive set of bli experiments on the standard bli benchmark @xcite , comprising 8 diverse languages, in several setups. our results indicate large gains over state-of-the-art bli models: e.g., ≈+8 precision@1 points on average, +10 points for many language pairs, gains for 107/112 bli setups already after stage c1 (cf., rq1), and for all 112/112 bli setups after stage c2 (cf., rq2 and rq3). moreover, our findings also extend to bli for lowerresource languages from another bli benchmark @xcite . finally, as hinted in recent work @xcite , our findings validate that multilingual lexical knowledge in lms, when exposed and extracted as in our contrastive learning framework, can complement the knowledge in static cross-lingual wes (rq3), and benefit bli. we release the code and share the data at: https: //github.com/cambridgeltl/contrastivebli. ## methodology preliminaries and task formulation. in bli, we assume two vocabularies @xmath0, . . . , w x |x | } and @xmath1, . . . , w y |y| } associated with two respective languages l x and l y . we also assume that each vocabulary word is assigned its (static) typelevel word embedding (we); that is, the respective we matrices for each vocabulary are @xmath2, @xmath3. each we is a d-dim row vector, with typical values @xmath4.g., fast-text) @xcite , and @xmath5. 3 we also assume a set of seed translation pairs @xcite , where typical values for the seed dictionary size |d 0 | are 5k pairs and 1k pairs @xcite , often referred to as supervised (5k) and semisupervised or weakly supervised settings (1k) @xcite . given another test lexi- t the goal is to retrieve its correct translation from l y 's vocabulary y, and evaluate it against the gold l y translation w y g i from the pair. method in a nutshell. we propose a novel two-stage contrastive learning (cl) method, with both stages c1 and c2 realised via contrastive learning objectives (see figure 1 ). stage c1 ( §2.1) operates solely on static wes, and can be seen as a contrastive extension of mapping-based bli approaches with static wes. in practice, we blend contrastive learning with the standard sota mapping-based framework with self-learning: vecmap @xcite , with some modifications. stage c1 operates solely on static wes in exactly the same bli setup as prior work, and thus it can be evaluated independently. in stage c2 ( §2.2), we propose to leverage pretrained multilingual lms for bli: we contrastively fine-tune them for bli and extract static 'decontextualised' wes from the tuned lms. these lm-based wes can be combined with wes obtained in stage c1 ( §2.3). ## experimental setup monolingual wes and bli setup. we largely follow the standard bli setup from prior work @xcite @xcite . the main evaluation is based on the standard bli dataset from @xcite : it comprises 28 language pairs with a good balance of typologically similar and distant languages (croatian: hr, english: en, finnish: fi, french: fr, german: de, italian: it, russian: ru, turkish: tr). again following prior work, we rely on monolingual fast-text vectors trained on full wikipedias for each language @xcite , where vocabularies in each language are trimmed to the 200k most frequent words (i.e., |x |=200k and |y|=200k). the same fasttext wes are used for our stage c1 and in all baseline bli models. mbert in stage c2 operates over the same vocabularies spanning 200k word types in each language. we use 1k translation pairs (semi-supervised bli mode) or 5k pairs (supervised) as seed dictionary d 0 ; test sets span 2k pairs @xcite . with 56 bli directions in total, foot_3 this yields a total of 112 bli setups for each model in our comparison. the standard precision@1 (p@1) bli measure is reported, and we rely on csls (@xmath6. 9 training setup and hyperparameters. since standard bli datasets typically lack a validation set @xcite , following prior work @xcite we conduct hyperparameter tuning on a single, randomly selected language pair @xmath7, and apply those hyperparameter values in all other bli runs. in stage c1, when |d 0 |=5k, the hyperparameter values are n @xmath8, n @xmath9, n @xmath10, n @xmath11, n @xmath12. sgd optimiser is used, with a learning rate of 1.5 and γ=0.99. when |d 0 |=1k, the values are n @xmath13, n @xmath14, n @xmath15, n @xmath16, and n @xmath17; sgd with a learning rate of 2.0, γ=1.0. τ =1.0 and dropout is 0 in both cases, and the batch size for contrastive learning is always equal to the size of the current dictionary |d cl | (i.e., |d 0 | (5k case), or |d 0 ∪d add | which varies over iterations (1k case); see §2.1). in stage c2, n @xmath18. we use adamw @xcite with learning rate of 2e -5 and weight decay of 0.01. we fine-tune mbert for 5 epochs, with a batch size of 100; dropout rate is 0.1 and τ =0.1. unless noted otherwise, λ is fixed to 0.2. baseline models. our bli method is evaluated against four strong sota bli models from recent literature, all of them with publicly available implementations. here, we provide brief summaries: foot_5 rcsls @xcite optimises a relaxed csls loss, learns a non-orthogonal mapping, and has been established as a strong bli model in empirical comparative analyses as its objective function is directly 'bli-oriented' @xcite . vecmap's core components @xcite have been outlined in §2.1. lnmap @xcite non-linearly maps the original static wes into two latent semantic spaces learned via non-linear autoencoders, foot_6 and then learns another non-linear mapping between the latent autoencoder-based spaces. fipp @xcite , in brief, first finds common (i.e., isomorphic) geometric structures in monolingual we spaces of both languages, and then aligns the gram matrices of the wes found in those common structures. for all baselines, we have verified that the hyperparameter values suggested in their respective repositories yield (near-)optimal bli performance. unless noted otherwise, we run vecmap, lnmap, and fipp with their own self-learning procedures. 12 model variants. we denote the full two-stage bli model as c2 (mod), where mod refers to the actual model/method used to derive the shared crosslingual space used by stage c2. for instance, c2 (c1) refers to the model variant which relies on our stage c1, while c2 (rcsls) relies on rc-sls as the base method. we also evaluate bli performance of our stage c1 bli method alone. multilingual lms. we adopt mbert as the default pretrained multilingual lm in stage c2. our supplementary experiments also cover the 1280dim xlm model foot_8 @xcite and 512-dim mt5 small @xcite ). 14 for clarity, we use c2 [lm] to denote c2 (c1) obtained from different lms; when [lm] is not specified, mbert is used. we adopt a smaller batch size of 50 for c2 [xlm] considering the limit of gpu memory, and train c2 [mt5] with a larger learning rate of 6e -4 for 6 epochs, since we found it much harder to train than c2 [mbert]. ## results and discussion the main results are provided in stage c1 versus baselines. first, we note that there is not a single strongest baseline among the four sota bli methods. for instance, rcsls and vecmap are slightly better than lnmap and fipp with 5k supervision pairs, while fipp and vecmap come forth as the stronger baselines with 1k supervision. there are some score fluctuations over individual language pairs, but the average performance of all baseline models is within a relatively narrow interval: the average performance of all four baselines is within 3 p@1 points with 5k pairs (i.e., ranging from 38.22 to 41.22), and vecmap, fipp, and lnmap are within 2 points with 1k pairs. strikingly, contrastive learning in stage c1 already yields substantial gains over all four sota bli models, which is typically much higher than the detected variations between the baselines. we mark that c1 improves over all baselines in 51/56 bli setups (in the 5k case), and in all 56/56 bli setups when d 0 spans 1k pairs. the average gains performance over languages. the absolute bli scores naturally depend on the actual source and target languages: e.g., the lowest absolute performance is observed for morphologically rich (hr, ru, fi, tr) and non-indo-european languages (fi, tr). however, both c1 and c2 (c1) mode variants offer wide and substantial gains in performance for all language pairs, irrespective of the starting absolute score. this result further suggests wide applicability and robustness of our bli method. ## related work this work is related to three topics, each with a large body of work; we can thus provide only a condensed summary of the most relevant research. mapping-based bli. these bli methods are highly popular due to reduced bilingual supervision requirements; consequently, they are applicable to low-resource languages and domains, learning linear @xcite @xcite @xcite and non-linear maps @xcite @xcite , typically using self-learning in weakly supervised setups. contrastive learning in nlp aims to learn a semantic space such that embeddings of similar text inputs are close to each other, while 'repelling' dissimilar ones. it has shown promising performance on training generic sentence encoders @xcite @xcite and downstream tasks like summarisation @xcite or ner @xcite . exposing lexical knowledge from pretrained lms. extracting lexical features from off-the-shelf multilingual lms typically yields subpar performance in lexical tasks @xcite . ## conclusion we have proposed a simple yet extremely effective and robust two-stage contrastive learning framework for improving bilingual lexicon induction (bli). in stage c1, we tune cross-lingual linear mappings between static word embeddings with a contrastive objective and achieve substantial gains in 107 out of 112 bli setups on the standard bli benchmark. in stage c2, we further propose a contrastive fine-tuning procedure to harvest crosslingual lexical knowledge from multilingual pretrained language models. the representations from this process, when combined with stage c1 embeddings, have resulted in further boosts in bli performance, with large gains in all 112 setups. we have also conducted a series of finer-grained evaluations, analyses and ablation studies.