LNM ap: Departures from Isomorphic Assumption in Bilingual Lexicon Induction Through Non-Linear Mapping in Latent Space (2020)

Abstract: Most of the successful and predominant methods for Bilingual Lexicon Induction (BLI) are mapping-based, where a linear mapping function is learned with the assumption that the word embedding spaces of different languages exhibit similar geometric structures (i.e. approximately isomorphic). However, several recent studies have criticized this simplified assumption showing that it does not hold in general even for closely related languages. In this work, we propose a novel semi-supervised method to learn cross-lingual word embeddings for BLI. Our model is independent of the isomorphic assumption and uses non-linear mapping in the latent space of two independently pre-trained autoencoders. Through extensive experiments on fifteen (15) different language pairs (in both directions) comprising resource-rich and low-resource languages from two different datasets, we demonstrate that our method outperforms existing models by a good margin. Ablation studies show the importance of different model components and the necessity of non-linear mapping.

## introduction in recent years, a plethora of methods have been proposed to learn cross-lingual word embeddings (or clwe for short) from monolingual word embeddings. here words with similar meanings in different languages are represented by similar vectors, regardless of their actual language. clwe enable us to compare the meaning of words across languages, which is key to most multi-lingual applications such as bilingual lexicon induction @xcite , machine translation @xcite , or multi-lingual information retrieval @xcite . they also play a crucial role in cross-lingual knowledge transfer between languages (e.g., from resourcerich to low-resource languages) by providing a common representation space @xcite . @xcite , in their pioneering work, learn a linear mapping function to transform the source embedding space to the target language by minimizing the squared euclidean distance between the translation pairs of a seed dictionary. they assume that the similarity of geometric arrangements in the embedding spaces is the key reason for their method to succeed as they found linear mapping superior to non-linear mappings with multi-layer neural networks. subsequent studies propose to improve the model by normalizing the embeddings, imposing an orthogonality constraint on the linear mapper, modifying the objective function, and reducing the seed dictionary size @xcite @xcite . a more recent line of research attempts to eliminate the seed dictionary totally and learn the mapping in a purely unsupervised way @xcite @xcite @xcite @xcite @xcite . while not requiring any cross-lingual supervision makes these methods attractive, @xcite recently show that even the most robust unsupervised method @xcite fails for a large number of language pairs. they suggest to rethink the main motivations behind fully unsupervised methods showing that with a small seed dictionary (500-1k pairs) their semi-supervised method always outperforms the unsupervised method and does not fail for any language pair. other concurrent work @xcite also advocates for weak supervision in clwe methods. almost all mapping-based clwe methods, supervised and unsupervised alike, solve the procrustes problem in the final step or during selflearning @xcite . this restricts the transformation to be orthogonal linear mappings. however, learning an orthogonal linear mapping inherently assumes that the embedding spaces of different languages exhibit similar geometric structures (i.e., approximately isomorphic). several recent studies have questioned this strong assumption and empirically showed that the isomorphic assumption does not hold in general even for two closely related languages like english and german @xcite . in this work, we propose lnmap (latent space non-linear mapping), a novel semi-supervised approach that uses non-linear mapping in the latent space to learn clwe. it uses minimal supervision from a seed dictionary, while leveraging semantic information from the monolingual word embeddings. as shown in figure 1 , lnmap comprises two autoencoders, one for each language. the auto-encoders are first trained independently in a self-supervised way to induce the latent code space of the respective languages. then, we use a small seed dictionary to learn the non-linear mappings between the two code spaces. to guide our mapping in the latent space, we include two additional constraints: back-translation and original embedding reconstruction. crucially, our method does not enforce any strong prior constraints like the orthogonality (or isomorphic), rather it gives the model the flexibility to induce the required latent structures such that it is easier for the non-linear mappers to align them in the code space. in order to demonstrate the effectiveness and robustness of lnmap, we conduct extensive experiments on bilingual lexicon induction (bli) with fifteen (15) different language pairs (in both directions) comprising high-and low-resource languages from two different datasets for different sizes of the seed dictionary. our results show significant improvements for lnmap over the state-ofthe-art in most of the tested scenarios. it is particularly very effective for low-resource languages; for example, using 1k seed dictionary, lnmap yields about 18% absolute improvements on average over a state-of-the-art supervised method @xcite . it also outperforms the most robust unsupervised system of @xcite in most of the translation tasks. interestingly, for resource-rich language pairs, linear autoencoder performs better than non-linear ones. our ablation study reveals the collaborative nature of lnmap's different components and efficacy of its non-linear mappings in the code space. we open-source our framework at https://ntunlpsg.github.io/project/lnmap/ . ## lnmap semi-supervised framework let v ℓx ={v x 1 , ..., v xn x } and v ℓy ={v y 1 , ..., v yn y } be two sets of vocabulary consisting of n x and n y words for a source (ℓ x ) and a target (ℓ y ) language, respectively. each word v x i (resp. v y j ) has an embedding x @xmath0. y @xmath1, trained with any word embedding models, e.g., fasttext @xcite . let e ℓx ∈ r nx×d and e ℓy ∈ r ny×d be the word embedding matrices for the source and target languages, respectively. we are also given with a seed dictionary @xmath2, y 1 ), ..., (x k , y k )} with k word pairs. our objective is to learn a transformation function m such that for any v x @xmath3, m(x i ) corresponds to its translation y j , where v y @xmath4. our approach lnmap (figure 1 ) follows two sequential steps: ## experimental settings we evaluate our approach on bilingual lexicon induction, also known as word translation. ## results and analysis we present our results on low-resource and resource-rich languages from muse dataset in tables 1 and 2 , respectively, and the results on vecmap dataset in ## conclusions we have presented a novel semi-supervised framework lnmap to learn the cross-lingual mapping between two monolingual word embeddings. apart from exploiting weak supervision from a small (1k) seed dictionary, our lnmap leverages the information from monolingual word embeddings. in contrast to the existing methods that directly map word embeddings using the isomorphic assumption, our framework is independent of any such strong prior assumptions. lnmap first learns to transform the embeddings into a latent space and then uses a non-linear transformation to learn the mapping. to guide the non-linear mapping further, we include constraints for back-translation and original embedding reconstruction. extensive experiments with fifteen different language pairs comprising high-and low-resource languages show the efficacy of non-linear transformations, especially for low-resource and distant languages. comparison with existing supervised, semi-supervised, and unsupervised baselines show that lnmap learns a better mapping. with an indepth ablation study, we show that different components of lnmap works in a collaborative nature.