Hierarchical Mapping for Crosslingual Word Embedding Alignment (2020)

Abstract: The alignment of word embedding spaces in different languages into a common crosslingual space has recently been in vogue. Strategies that do so compute pairwise alignments and then map multiple languages to a single pivot language (most often English). These strategies, however, are biased towards the choice of the pivot language, given that language proximity and the linguistic characteristics of the target language can strongly impact the resultant crosslingual space in detriment of topologically distant languages. We present a strategy that eliminates the need for a pivot language by learning the mappings across languages in a hierarchical way. Experiments demonstrate that our strategy significantly improves vocabulary induction scores in all existing benchmarks, as well as in a new non-English–centered benchmark we built, which we make publicly available.

## introduction word embeddings have changed how we build text processing applications, given their capabilities for representing the meaning of words @xcite @xcite . traditional embedding-generation strategies create different embeddings for the same word depending on the language. even if the embeddings themselves are different across languages, their distributions tend to be consistentthe relative distances across word embeddings are preserved regardless of the language @xcite . this behavior has been exploited for crosslingual embedding generation by aligning any two monolingual embeddings spaces into one @xcite @xcite . alignment techniques have been successful in generating bilingual embedding spaces that can later be merged into a crosslingual space using a pivoting language, english being the most common choice. unfortunately, mapping one language into another suffers from a neutrality problem, as the resultant bilingual space is impacted by language-specific phenomena and corpus-specific biases of the target language @xcite . to address this issue, @xcite propose mapping any two languages into a different middle space. this mapping, however, precludes the use of a pivot language for merging multiple bilingual spaces into a crosslingual one, limiting the solution to a bilingual scenario. additionally, the pivoting strategy suffers from a generalized bias problem, as languages that are the most similar to the pivot obtain a better alignment and are therefore better represented in the crosslingual space. this is because language proximity is a key factor when learning alignments. this is evidenced by the results in @xcite , which indicate that when using english (indo-european) as a pivot, the vocabulary induction results for finnish (uralic) are about 10 points below the rest of the indo-european languages under study. if we want to incorporate all languages into the same crosslingual space regardless of their characteristics, we need to go beyond the trainbilingual/merge-by-pivoting (tb/mp) model, and instead seek solutions that can directly generate crosslingual spaces without requiring a bilingual step. this motivates the design of hceg (hierarchical crosslingual embedding generation), the hierarchical pivotless approach for generating crosslingual embedding spaces that we present in this paper. hceg addresses both the language proximity and target-space bias problems by learning a compositional mapping across multiple languages in a hierarchical fashion. this is accomplished by taking advantage of a language family tree for aggregating multiple languages into a single crosslingual space. what distinguishes hceg from tb/mp strategies is that it does not need to include the pivot language in all mapping functions. this enables the option to learn mappings between typologically similar languages, known to yield better quality mappings @xcite . the main contributions of our work include: ## related work recent interest in crosslingual word embedding generation has led to manifold strategies that can be classified into four groups @xcite : (1) mapping techniques that rely on a bilingual lexicon for mapping an already trained monolingual space into another @xcite @xcite ; a key factor for crosslingual embedding generation techniques is the amount of supervised signal needed. parallel corpora are a scarce resource-even nonexistent for some isolated or low-resource languages. thus, we focus on mapping-based strategies that can go from requiring just a bilingual lexicon @xcite to absolutely no supervised signal (artetxe 1 resources can be found at https://github.com/ ionmadrazo/hceg . @xcite . this aligns with one of the premises for our research to enable the generation of a single crosslingual embedding space for as many languages as possible. @xcite first introduced a mapping strategy for aligning two monolingual spaces that learns a linear transformation from source to target space using stochastic gradient descent. this approach was later enhanced with the use of least squares for finding the optimal solution, l2-normalizing the word embedding, or constraining the mapping matrix to be orthogonal @xcite @xcite @xcite ; enhancements that soon became standard in the area. these models, however, are affected by hubness, where some words tend to be in the neighborhood of an exceptionally large number of other words, causing problems when using nearest-neighbor as the retrieval algorithm, and neutrality, where the resultant crosslingual space is highly conditioned by the characteristics of the language used as target. hubness was addressed by a correction applied to nearest-neighbor retrieval whether using a inverted softmax @xcite or a cross-domain similarity local scaling @xcite later incorporated as part of the training loss @xcite . neutrality was noticed by @xcite , for which they proposed using two independent linear transformations so that the resulting crosslingual space is in a middle point between the two languages rather than just on the target language, and therefore not biased towards either language. other important trends in the area concentrate on (i) the search of unsupervised techniques for learning mapping functions @xcite and their versatility in dealing with low-resource languages @xcite ; (ii) the long-tail problem, where most existing crosslingual embedding generation strategies tend to under-perform @xcite ; and (iii) the formulation of more robust evaluation procedures oriented to determining the quality of generated crosslingual spaces @xcite . most existing works focus on a bilingual scenario. yet, there is an increase on the interest for designing strategies that directly consider more than two languages at training time, thus creating fully multilingual spaces that do not depend on the tb/mp model @xcite for multilingual inference. attempts to do so include the efforts by @xcite , who leverage an inverted index based on the wikipedia multilingual links to generate multilingual word representations. @xcite instead use a sentence-level neural language model for directly learning multilingual word embeddings and as a result bypassing the need for mapping functions. in the paradigm of aligning pre-trained word embeddings where we focus, @xcite propose a technique that iteratively builds a multilingual space starting from a monolingual space and incrementally incorporating languages to it. even if this strategy deviates from the traditional tb/mp model, it still preserves the idea of having a pivot language. chen and cardie (2018) separate the mapping functions into encoders and decoders, which are not language-pair dependent, unlike those in the tb/mp model. this removes the need for a pivot language, given that the multilingual space is now latent among all encoder and decoders and not centered in a specific language. the same pivot-removal effect is achieved by the strategy introduced in @xcite , which generalizes a bilingual word embedding strategy into a multilingual counterpart by inducing a mahalanobis similarity metric in the common space. these two strategies, however, still consider all languages equidistant to each other, ignoring the similarities and differences that lay among them. our work is inspired by @xcite and @xcite , in the sense that it focuses on obtaining a non-biased or neutral crosslingual space that does not need to be centered in english (or any other pivot language) as the primary source. this neutrality is obtained by a compositional mapping strategy that hierarchically combines mapping functions in order to generate a single, non-language-centered crosslingual space, enabling a better mapping for languages that are distant or non-typologically related to english. ## proposed strategy a language family tree is a natural categorization of languages that has historically been used by linguistics as a reference that encodes similarities and differences across languages @xcite . for example, based on the relative distances among languages in the tree illustrated in figure 1 , we infer that both spanish and portuguese are relatively similar to each other, given that they are part of the same italic family. at the same time, both languages are farther apart from english than each other, and are radically different with respect to finnish. a language family tree offers a natural organization that can be exploited when building crosslingual spaces that integrate typologically diverse languages. we leverage this structure in hceg, in order to generate a hierarchically compositional crosslingual word embedding space. unlike traditional tb/mp strategies that generate a single crosslingual space, the result of hceg is a set of transformation matrices that can be used to hierarchically compose the space required in each use-case. this maximizes the typological intra-similarity among languages used for generating the embedding space, while minimizing the differences across languages that can hinder the quality of the crosslingual embedding space. thus, if an external application only considers languages that are germanic, then it can just use the germanic crosslingual space generated by hceg, whereas if it needs languages beyond germanic it can utilize a higher level family, such as the indo-european. this cannot be done with the traditional tb/mp model. in this case, if an application is, for example, using only uralic languages, then it would be forced to use an english-centered crosslingual space; this would in a decrease in the quality of the crosslingual space used because of the potential bad quality of mappings between typologically different languages, such as uralic and indo-european languages @xcite . ## evaluation framework we describe below the evaluation set up used for conducting the experiments presented in section 5. ## evaluation we discuss below the results of the study conducted over 107 languages to assess hceg. ## conclusion and future work we have introduced hceg, a crosslingual space learning strategy that does not depend on a pivot language, as instead, it takes advantage of the natural hierarchy existing among languages. results from extensive studies on 107 languages demonstrate that the proposed strategy outperforms existing crosslingual space generation techniques, in terms of vocabulary induction, for both popular and not so popular languages. hceg improves the mapping quality of many low-resource languages. we noticed that this improvement mostly happens when a language has more typologically related counterparts, however. therefore, as future work, we intend to investigate other techniques that can help improve the quality of mapping for typologically isolated low-resource languages. additionally, it is important to note that the time complexity required by the proposed algorithm is n (n -1), with n being the number of languages considered. for the traditional tb/mp strategy, complexity is limited to learning from n language pairs. therefore, we plan on exploring strategies to reduce the number of language pairs that need to be learned for creating the crosslingual space. finally, we will explore different data-driven strategies for building the tree structure, such as geographical proximity or lexical overlap, which could lead to better optimized arrangements of the crosslingual space.