Multi-Adversarial Learning for Cross-Lingual Word Embeddings (2021)

Abstract: Generative adversarial networks (GANs) have succeeded in inducing cross-lingual word embeddings - maps of matching words across languages - without supervision. Despite these successes, GANs’ performance for the difficult case of distant languages is still not satisfactory. These limitations have been explained by GANs’ incorrect assumption that source and target embedding spaces are related by a single linear mapping and are approximately isomorphic. We assume instead that, especially across distant languages, the mapping is only piece-wise linear, and propose a multi-adversarial learning method. This novel method induces the seed cross-lingual dictionary through multiple mappings, each induced to fit the mapping for one subspace. Our experiments on unsupervised bilingual lexicon induction and cross-lingual document classification show that this method improves performance over previous single-mapping methods, especially for distant languages.

## introduction and background word embeddings, continuous vectorial representations of words, have become a fundamental initial step in many natural language processing (nlp) tasks for many languages. in recent years, their cross-lingual counterpart, cross-lingual word embeddings (clwe) -maps of matching words across languages-have been shown to be useful in many important cross-lingual transfer and modeling tasks such as machine translation, cross-lingual document classification and zero-shot dependency parsing @xcite @xcite @xcite . in these representations, matching words across different languages are represented by similar vectors. following the observation of @xcite that the geometric positions of similar words in two embedding spaces of different languages ap-pear to be related by a linear relation, the most common method aims to map between two pretrained monolingual embedding spaces by learning a single linear transformation matrix. due to its simple structure design and competitive performance, this approach has become the mainstream of learning clwe @xcite @xcite . initially, the linear mapping was learned by minimizing the distances between the source and target words in a seed dictionary. early work from @xcite uses a seed dictionary of fivethousand word pairs. since then, the size of the seed dictionary has been gradually reduced, from several-thousand to fifty word pairs @xcite , reaching a minimal version of only sharing numerals @xcite . more recent works on unsupervised learning have shown that mappings across embedding spaces can also be learned without any bilingual evidence @xcite @xcite @xcite . more concretely, these fully unsupervised methods usually consist of two main steps @xcite : an unsupervised step which aims to induce the seed dictionary by matching the source and target distributions, and then a pseudo-supervised refinement step based on this seed dictionary. the system proposed by @xcite ) can be considered the first successful unsupervised system for learning clwe. they first use generative adversarial networks (gans) to learn a single linear mapping to induce the seed dictionary, followed by the procrustes analysis @xcite to refine the linear mapping based on the induced seed dictionary. while this gan-based model has competitive or even better performance compared to supervised methods on typologicallysimilar language pairs, it often exhibits poor performance on typologically-distant language pairs, pairs of languages that differ drastically in word forms, morphology, word order and other properties that determine how similar the lexicon of a language is. more specifically, their initial linear mapping often fails to induce the seed dictionary for distant language pairs @xcite . later work from @xcite has proposed an unsupervised self-learning framework to make the unsupervised clwe learning more robust. their system uses similarity distribution matching to induce the seed dictionary and stochastic dictionary induction to refine the mapping iteratively. the final clwe learned by their system performs better than the gan-based system. however, their advantage appears to come from the iterative refinement with stochastic dictionary induction, according to @xcite . if we only consider the performance of a model induced only with distribution matching, gan-based models perform much better. this brings us to our first conclusions, that a gan-based model is preferable for seed dictionary induction. fully unsupervised mapping-based methods to learn clwe rely on the strong assumption that monolingual word embedding spaces are isomorphic or near-isomorphic, but this assumption is not fulfilled in practice, especially for distant language pairs @xcite . experiments by @xcite also demonstrate that the lack of isomorphism does not arise only because of the typological distance among languages, but it also depends on the quality of the monolingual embedding space. if we replace the seed dictionary learned by an unsupervised distribution matching method with a pretrained dictionary, keeping constant the refinement technique, the final system becomes more robust @xcite . all these previous results indicate that learning a better seed dictionary is a crucial step to improve unsupervised cross-lingual word embedding induction and reduce the gap between unsupervised methods and supervised methods, and that ganbased methods hold the most promise to achieve this goal. the results also indicate that a solution that can handle the full complexity of induction of cross-lingual word embeddings will show improvements in both close and distant languages. in this paper, we focus on improving the initial step of distribution matching, using gans @xcite . because the isomorphism assumption is not observed in reality, we argue that a successful gan-based model must not learn only one single linear mapping for the entire distribution, but must be able to identify mapping subspaces and learn multiple mappings. we propose a multi-adversarial learning method which learns different linear maps for different subspaces of word embeddings. ## limitations of single-linear mappings if the assumption by @xcite that similar words across source and target languages are related by a single linear relation holds exactly or even approximately , the distance between source and target embedding spaces should be (nearly) evenly minimized during the training of the initial mapping. more specifically, each source subspace should be mapped (nearly) equally well to its corresponding target space, so that the translation ability of the single linear mapping should be similar across different source subspaces. to verify this expectation, we use the ganbased system muse foot_0 to train two linear mappings (without refinement) @xcite . one mapping relates two typologically distant languages, english and chinese, and the other maps the english space to the space of french -a typologically similar language. we use pretrained fasttext embeddings. 2 we split the english space into ten subspaces by running k-means clustering. we evaluate the trained linear mappings by calculating the translation accuracy with precision at one (p@1) -how often the highest ranked translation is the correct one-for each subspace, using the translations from google translate as the gold dataset. to reduce the influence of infrequent words, we only consider the first fifty-thousand most frequent source words. as we can see in figure 1 , the distribution of accuracies of different subspaces is not uniform or even nearly so. this is true for both language pairs, but particularly for the distant languages, where the general mapping does not work at all in some subspaces. similar phenomena were also discovered by @xcite where source words are grouped into different categories. this lack of uniformity in results corroborates the appropriateness of designing a model that learns different linear mappings for different subspaces instead of only learning a single linear mapping for the entire source space. ## multi-adversarial clwe learning to learn different mappings for different source subspaces, we propose a method for training one gan for each source subspace. these multidiscriminator gans encourage the distribution of mapped word embeddings from a specific source subspace to match the distribution of word embeddings from the corresponding target subspace. the first step of our proposed method is to train a single linear mapping, as in previous approaches. this is used to find aligned subspaces. our proposed multi-discriminator gan model then learns the multi-linear mapping. this section starts with the two gan models, followed by the subspace alignment method, and then describes methods used to improve the gan training. ## clwe mapping refinement as in previous work, after gans have been used to find a mapping from source to target word embeddings, a refinement step can be used to improve this mapping. refinement involves first inducing a seed dictionary of word translations, and then refining the mapping using this seed dictionary. bidirectional seed dictionary induction using the mapping learned with adversarial training, the translations (w t 1 , w t 2 , ..., w t 10000 ) for the top tenthousand source words (w s 1 , w s 2 , ..., w s 10000 ) are retrieved and then back-translated into the source language (w s 1 , w s 2 , ..., w s 10000 ). the mutual translation pairs (w s i , w t i ) such that w s @xmath0. this guarantees that the induced seed dictionary will be bidirectional. @xcite . with the seed dictionary, the mapping can be updated using the objective in equation ( 1 ), and forced to be orthogonal using singular value decomposition (svd) @xcite . later work combines the procrustes analysis with stochastic dictionary induction @xcite and greatly improves the performance of the standard refinement @xcite . more specifically, in order to prevent local optima, after each iteration some elements of the similarity matrix are randomly dropped, so that the similarity distributions of words change randomly and the new seed dictionary for the next iteration varies. ## experiments bilingual lexicon induction (bli) has become a standard task for evaluating clwe models. however, according to @xcite and @xcite , bli performance of a given clwe model doesn't always correlate with performance in other cross-lingual downstream tasks. in this section, we evaluate our proposal on both the task of bli and the task of cross-lingual document classification (cldc). we evaluate our system both with and without refinement. since gan-based methods of learning clwe are often criticized for their instability at inducing the seed dictionary, we report the average over 10 runs for the bli without-refinement setting. we include the random restart technique for other tasks and report the result of the best model selected by the unsupervised model selection criterion. we evaluate our model both with global refinement (g-ref) and local refinement (l-ref). bli setting we use the dataset provided by @xcite for the task of bli. this dataset contains high quality dictionaries for more than 150 language pairs. for each language pair, it provides a training dictionary of 5000 words and a test dictionary of 1500 words. this dataset allows us to have a better understanding of the performance of our proposal on many different language pairs. for each language pair, we retrieve the best translations of source words in the test dictionary using csls, and we report the accuracy with precision at one (p@1). ## conclusion in this paper, we propose a multi-adversarial learning method for cross-lingual word embeddings. our system learns different linear mappings for different source subspaces instead of just learning a single one for the whole source space. the results of our experiments on bilingual lexicon induction and cross-lingual document classification on both close languages and distant languages prove that learning cross-lingual word embeddings with a multi-linear mapping improves performance over a single-linear mapping. future work will focus on learning multi-linear mappings for contextualized embeddings.