A Simple Approach to Learning Unsupervised Multilingual Embeddings (2020)

Abstract: Recent progress on unsupervised cross-lingual embeddings in the bilingual setting has given the impetus to learning a shared embedding space for several languages. A popular framework to solve the latter problem is to solve the following two sub-problems jointly: 1) learning unsupervised word alignment between several language pairs, and 2) learning how to map the monolingual embeddings of every language to shared multilingual space. In contrast, we propose a simple approach by decoupling the above two sub-problems and solving them separately, one after another, using existing techniques. We show that this proposed approach obtains surprisingly good performance in tasks such as bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual dependency parsing. When distant languages are involved, the proposed approach shows robust behavior and outperforms existing unsupervised multilingual word embedding approaches.

## introduction learning cross-lingual word representations has been the focus of many recent works @xcite @xcite . it aims at learning a shared embedding space for words across two (bilingual word embedding) or more languages (multilingual word embedding or mwe), by mapping similar words (or concepts) across different languages close to each other in a shared embedding space. such a representation is useful in various applications such as cross-lingual text classification @xcite , building bilingual lexicons @xcite , cross-lingual information retrieval @xcite , and machine translation @xcite , to name a few. @xcite showed that the geometric arrangement of word embeddings could be (approximately) preserved by linearly transforming the word embeddings from one language space to another. subsequently, several works have explored learning bilingual word embeddings in both supervised @xcite @xcite @xcite and unsupervised @xcite @xcite @xcite @xcite settings. representing word embeddings of many languages in a common shared space is desirable to allow knowledge transfer between different languages. @xcite are among the first to propose unsupervised learning of mwes. they extend the gan-based iterative refinement procedure for learning bilingual word embeddings @xcite to the multilingual setting. however, adversarial training has known concerns of optimization stability with distant language pairs @xcite . @xcite propose a joint optimization framework for learning bilingual lexicons and mappings between several pairs of languages. they obtain the bilingual lexicons using the gromov-wasserstein approach @xcite and mapping operators between languages using the rcsls algorithm @xcite . @xcite propose to learn the shared multilingual space by incrementally adding languages to it, one in each iteration. their approach is based on a reformulation of the bilingual self-learning algorithm proposed by @xcite . this work proposes a two-stage framework for learning a shared mwe space in the unsupervised setting. the two stages aim at solving the following sub-problems: a) generating bilingual lexicons between a few pairs of languages, and subsequently b) learning the mapping operators between languages in a shared multilingual space. the subproblems are separately solved using existing techniques. in contrast, existing unsupervised multilingual approaches @xcite @xcite solve the above subproblems jointly. though it appears like a simple baseline approach, the proposed framework provides the robustness and versatility often desired while learning an effective multilingual space for distant languages, which is a challenging setting for unsupervised methods @xcite @xcite . we evaluate our approach on the bilingual lexicon induction (bli) task, cross-lingual word similarity task, and two downstream multilingual tasks: document classification and dependency parsing. we summarize our findings below. ## unsupervised multilingual multi-stage framework we propose the following framework for unsupervised learning of mwes: ## experiments the proposed methods sl-geomm and gw-geomm are compared against existing unsupervised multilingual word embeddings approaches umwe @xcite and umh @xcite on various bli and downstream tasks. as a bilingual baseline, we also include stateof-the-art unsupervised bilingual word embeddings approach bilingunsup @xcite in our bli experiments. in addition to gauging the effectiveness of the proposed two-staged framework, the experiments also study the multilingual approaches' robustness, especially when distant languages are involved. the evaluated tasks are detailed below. ## conclusion we study a two-stage framework for learning unsupervised multilingual word embeddings. the two stages correspond to unsupervised generation of bilingual lexicons for a few language pairs and subsequently learning a shared latent multilingual space. we propose to solve each of them with existing techniques @xcite @xcite . though the proposed framework seems simple compared to the joint optimization methods @xcite @xcite , our main contribution has been to show that it is a strong performer. empirical results on several different benchmarks on bilingual lexicon induction, cross-lingual word similarity, multilingual document classification, and multilingual document parsing tasks show remarkably good performance and robustness of the proposed framework. the proposed framework has the flexibility to be easily employed in hybrid setups where supervision is available for a few language pairs but is unavailable for others. overall, our results encourage the development of simple multi-stage models for learning multilingual word embeddings.