Should All Cross-Lingual Embeddings Speak E nglish? (2020)

Abstract: Most of recent work in cross-lingual word embeddings is severely Anglocentric. The vast majority of lexicon induction evaluation dictionaries are between English and another language, and the English embedding space is selected by default as the hub when learning in a multilingual setting. With this work, however, we challenge these practices. First, we show that the choice of hub language can significantly impact downstream lexicon induction zero-shot POS tagging performance. Second, we both expand a standard English-centered evaluation dictionary collection to include all language pairs using triangulation, and create new dictionaries for under-represented languages. Evaluating established methods over all these language pairs sheds light into their suitability for aligning embeddings from distant languages and presents new challenges for the field. Finally, in our analysis we identify general guidelines for strong cross-lingual embedding baselines, that extend to language pairs that do not include English.

## introduction continuous vectors for representing words (embeddings) @xcite have become ubiquitous in modern, neural nlp. cross-lingual representations @xcite additionally represent words from various languages in a shared continuous space, which in turn can be used for bilingual lexicon induction (bli). bli is often the first step towards several downstream tasks such as part-of-speech (pos) tagging @xcite , parsing @xcite , document classification @xcite , and machine translation @xcite @xcite . often, such shared representations are learned with a two-step process, whether under bilingual or multilingual settings (hereinafter bwe and mwe, respectively). first, monolingual word embeddings are learned over large swaths of text. such pre-trained word embeddings, such as the fasttext wikipedia vectors @xcite , are available for many languages and are widely used. second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision @xcite , under minimal supervision e.g. using only identical strings @xcite , or even in an unsupervised fashion @xcite . both in bilingual and multilingual settings, it is common that one of the language embedding spaces is the target to which all other languages get aligned (hereinafter "the hub"). we outline the details in section 2. despite all the recent progress in learning crosslingual embeddings, we identify a major shortcoming to previous work: it is by and large english-centric. notably, most mwe approaches essentially select english as the hub during training by default, aligning all other language spaces to the english one. we argue and empirically show, however, that english is a poor hub language choice. in bwe settings, on the other hand, it is fairly uncommon to denote which of the two languages is the hub (often this is implied to be the target language). however, we experimentally find that this choice can greatly impact downstream performance, especially when aligning distant languages. this anglocentricity is even more evident at the evaluation stage. the lexica most commonly used for evaluation are the muse lexica @xcite which cover 45 languages, but with translations only from and into english. alternative evaluation dictionaries are also very english-and european-centric: @xcite report results on english-italian, @xcite on english-german and english-finnish, @xcite on spanish-english and italian-english, and @xcite between english and italian, german, finnish, spanish, and turkish. we argue that cross-lingual word embedding mapping methods should look beyond english for their evaluation benchmarks because, compared to all others, english is a language with disproportionately large available data and relatively poor inflectional morphology e.g., it lacks case, gender, and complex verbal inflection systems @xcite . these two factors allow for an overly easy evaluation setting which does not necessarily generalize to other language pairs. in light of this, equal focus should instead be devoted to evaluation over more diverse language pairs that also include morphologically rich and low-resource languages. with this work, we attempt to address these shortcomings, providing the following contributions: ## cross-lingual word embeddings and lexicon induction bilingual word embeddings in the supervised bwe setting of @xcite , given two languages @xmath0, l 2 } and their pre-trained row-aligned embeddings x 1 , x 2 , respectively, a transformation matrix m is learned such that: the set ω can potentially impose a constraint over m , such as the very popular constraint of restricting it to be orthogonal @xcite . previous work has empirically found that this simple formulation is competitive with other more complicated alternatives @xcite . the orthogonality assumption ensures that there exists a closed-form solution through singular value decomposition (svd) of x 1 x t 2 . foot_1 note that in this case only a single matrix m needs to be learned, because x 1 -m x 2 = m -1 x 1 -x 2 , while at the same time a model that minimizes x 1 -m x 2 is as expressive as one minimizing m 1 x 1 -m 2 x 2 , with half the parameters. in the minimally supervised or even the unsupervised setting, @xcite and @xcite reframe the task as an adversarial game, with a generator aiming to produce a transformation that can fool a discriminator. however, the most popular methods follow an iterative refinement approach @xcite . starting with a seed dictionary (e.g. from identical strings @xcite or numerals) an initial mapping is learned in the same manner as in the supervised setting. the initial mapping, in turn, is used to expand the seed dictionary with high confidence word translation pairs. the new dictionary is then used to learn a better mapping, and so forth the iterations continue until convergence. the same iterative approach is followed by @xcite , with one important difference that allows their model (vecmap) to handle language pairs with different alphabets: instead of identical strings, the seed dictionary is constructed based on the similarity of the monolingual similarity distributions over all words in the vocabulary. 3 multilingual word embeddings in a multilingual setting, the simplest approach is to use bwe and align all languages into a target language (the hub). in this case, for n languages @xmath1, l 2 , . . . , l n } on has to learn n -1 bilingual mappings @xcite . rather than using a single hub space, @xcite propose an incremental procedure that uses an incremental hub space (ihs): each new language is included to the multilingual space by mapping it to all languages that have already been aligned (e.g. language l 3 would be mapped to the aligned space of {l 1 , l 2 }). alternatively, all mappings could be learned jointly, taking advantage of the inter-dependencies between any two language pairs. importantly, though, there is no closed form solution for learning the joint mapping, hence a solution needs to be approximated with gradientbased methods. the main approaches are: even though the architecture and modeling approach of all mwe methods are different, they share the same conceptual traits: one of the language spaces remains invariant and all other languages are effectively mapped to it. in all cases, english is by default selected to be the hub. the only exception is the study of triplets alignments in @xcite , where spanish is used as the spanish-french-portuguese triplet hub. lexicon induction one of the most common downstream evaluation tasks for the learned cross-lingual word mappings is lexicon induction (li), the task of retrieving the most appropriate word-level translation for a query word from the mapped embedding spaces. specialized evaluation (and training) dictionaries have been created for multiple language pairs. of these, the muse dictionaries @xcite are most often used, providing word translations between english (en) and 48 other high-to mid-resource languages, as well as on all 30 pairs among 6 very similar romance and germanic languages (english, french, german, spanish, italian, portuguese). given the mapped embedding spaces, the translations are retrieved using a distance metric, with cross-lingual similarity scaling @xcite as the most commonly used in the literature. intuitively, csls decreases the scores of pairs that lie in dense areas, increasing the scores of rarer words (which are harder to align). the retrieved pairs are compared to the gold standard and evaluated using precision at k (p@k, evaluating how often the correct translation is within the k retrieved nearest neighbors of the query). throughout this work we report p@1, which is equivalent to accuracy; we provide p@5 and p@10 results in the appendix. ## new li evaluation dictionaries the typically used evaluation dictionaries cover a narrow breadth of the possible language pairs, with the majority of them focusing in pairs with english (as with the muse or @xcite dictionaries) or among high-resource european languages. @xcite , for instance, highlighted anglocentricity as an issue, creating and evaluating on 28 dictionaries between 8 languages (croatian, english, finnish, french, german, italian, russian, turkish) based on google translate. in addition, @xcite focused on the morphology dimension, creating morphologically complete dictionaries for 2 sets of 5 genetically related languages (romance: french, spanish, italian, portuguese, catalan; and slavic: polish, czech, slovak, russian, ukrainian). in contrast to these two (very valuable!) works, our method for creating dictionaries for low-resource languages ( §3.1) leverages resources that are available for about 300 languages. in addition, we propose a simple triangulation process ( §3.2), that makes it possible to create dictionaries for arbitrary language pairs, given that dictionaries into a pivot language (usually english) are available for both languages. ## lexicon induction experiments the aim of our li experiments is two-fold. first, the differences in li performance show the importance of the hub language choice with respect to each evaluation pair. second, as part of our call for moving beyond anglo-centric evaluation, we also present li results on several new language pairs using our triangulated dictionaries. ## downstream task experiments differences in bli performance do not necessarily translate to differences in other downstream tasks that use the aligned embeddings, so @xcite advocate for actual evaluation on such tasks. we extend our analysis to an example downstream task of zero-shot pos tagging using the aligned embeddings for select language pairs. we show that indeed the choice of the hub language can have dramatic impact. using universal dependencies data @xcite we train simple bi-lstm pos taggers on en and pt using the respective embeddings produced from each mat+mpsr run, and evaluate the zero-shot performance on gl and es. 17 although all taggers achieve consistent @xmath2, the zero-shot performance on the test languages, as shown in it should be noted that the best hub for pos-tagging does not always coincide with the best hub for li, e.g. the best li hub for pt-gl is es, which leads to 11 percentage points worse gl pos tagging performance than the best system. in fact, for the language pairs that we studied we observe no correlation between the two tasks performance as we vary the hub (with an average spearman's rank correlation ρ = 0.08). ## conclusion with this work we challenge the standard practice in learning cross-lingual word embeddings. we empirically show that the choice of the hub language is an important parameter that affects lexicon induction performance in both bilingual (between distant languages) and multilingual settings. more importantly, we hope that by providing new dictionaries and baseline results on several language pairs, we will stir the community towards evaluating all methods in challenging scenarios that include under-represented language pairs. towards this end, our analysis provides insights and general directions for stronger baselines for non-anglocentric crosslingual word embeddings. the problem of identifying the best hub language, despite our analysis based on the use of typological distance, remains largely unsolved. in the future, we will investigate a hub language ranking/selection model a la @xcite .