Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring (2021)

Abstract: Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings. Such methods critically rely on those embeddings having a similar structure, but it was recently shown that the separate training in different languages causes departures from this assumption. In this paper, we propose an alternative approach that does not have this limitation, while requiring a weak seed dictionary (e.g., a list of identical words) as the only form of supervision. Rather than aligning two fixed embedding spaces, our method works by fixing the target language embeddings, and learning a new set of embeddings for the source language that are aligned with them. To that end, we use an extension of skip-gram that leverages translated context words as anchor points, and incorporates self-learning and iterative restarts to reduce the dependency on the initial dictionary. Our approach outperforms conventional mapping methods on bilingual lexicon induction, and obtains competitive results in the downstream XNLI task.

## introduction cross-lingual word embeddings (clwes) represent words from two or more languages in a shared space, so that semantically similar words in different languages are close to each other. early work focused on jointly learning clwes in two languages, relying on a strong cross-lingual supervision in the form of parallel corpora @xcite or bilingual dictionaries @xcite . however, these approaches were later superseded by offline mapping methods, which separately train word embeddings in different languages and align them in an unsupervised manner through self-learning @xcite or adversarial training @xcite . despite the advantage of not requiring any parallel resources, mapping methods critically rely on the underlying embeddings having a similar structure, which is known as the isometry assumption. several authors have observed that this assumption does not generally hold, severely hindering the performance of these methods @xcite @xcite . in later work, @xcite showed that this issue arises from trying to align separately trained embeddings, as joint learning methods are not susceptible to it. in this paper, we propose an alternative approach that does not have this limitation, but can still work without any parallel resources. the core idea of our method is to fix the target language embeddings, and learn aligned embeddings for the source language from scratch. this prevents structural mismatches that result from independently training embeddings in different languages, as the learning of the source embeddings is tailored to each particular set of target embeddings. for that purpose, we use an extension of skip-gram that leverages translated context words as anchor points. so as to translate the context words, we start with a weak initial dictionary, which is iteratively improved through self-learning, and we further incorporate a restarting procedure to make our method more robust. thanks to this, our approach can effectively work without any human-crafted bilingual resources, relying on simple heuristics (automatically generated lists of numerals or identical words) or an existing unsupervised mapping method to build the initial dictionary. our experiments confirm the effectiveness of our approach, outperforming previous mapping methods on bilingual dictionary induction and obtaining competitive results on zero-shot crosslingual transfer learning on xnli. ## proposed method let x i and xi be the input and output vectors of the ith word in the source language, and y j and ỹj be their analogous in the target language. 1 in addition, let d be a bilingual dictionary, where d(i) = j denotes that the ith word in the source language is translated as the jth word in the target language. our approach first learns the target language embeddings {y i } and {ỹ i } monolingually using regular sgns. having done that, we learn the source language embeddings {x i } and {x i }, constraining them to be aligned with the target language embeddings according to the dictionary d. for that purpose, we propose an extension of algorithm 1 proposed method input: d (dictionary), csrc (src corpus), ctgt (tgt corpus) output: {xi}, {yi} (aligned src and tgt embs) hparams: t (updates), r (restarts), k (re-inductions) 1: {yi}, {ỹi} ← sgns(ctgt) ⊲ learn target embedings 2: for r ← 1 to r do ⊲ iterative restart ( §3.3) 3: {xi}, {xi} ← random init() 4: if it mod (t /k) = 0 then ⊲ self-learn ( §3.2) 8: d ← reinduce({xi}, {yi}) 9: ## experimental setup we next describe the systems explored in our experiments ( §4.1), the data and procedure used to train them ( §4.2), and the evaluation tasks ( §4.3). ## conclusions and future work our approach for learning clwes addresses the main limitations of both offline mapping and joint learning methods. different from mapping approaches, it does not suffer from structural mismatches arising from independently training embeddings in different languages, as it works by constraining the learning of the source embeddings so they are aligned with the target ones. at the same time, unlike previous joint methods, our system can work without any parallel resources, relying on numerals, identical words or an existing mapping method for the initialization. we achieve this by combining cross-lingual anchoring with self-learning and iterative restarts. while recent research on clwes has been dominated by mapping approaches, our work shows that the fundamental techniques that popularized these methods (e.g., the use of self-learning to relax the need for crosslingual supervision) can also be effective beyond this paradigm. despite its simplicity, our experiments on bli show the superiority of our method when compared to previous mapping systems. we complement these results with additional experiments on a downstream task, where our method obtains competitive results, as well as an ablation study and a systematic error analysis. we identify a striking tendency of our method to translate words identically, even if it has no notion of the words being identically spelled. thanks to this, our method is particularly strong at translating named entities, but we show that our improvements are not limited to this phenomenon. these insights confirm the value of accompanying quantitative results on bli with qualitative evaluation @xcite and/or other tasks @xcite . in the future, we would like to further explore clwe methods that go beyond the currently dominant mapping paradigm. in particular, we would like to remove the requirement of a seed dictionary altogether by using adversarial learning, and explore more elaborated context translation and dictionary re-induction schemes.