Geometry-aware domain adaptation for unsupervised alignment of word embeddings (2020)

Abstract: We propose a novel manifold based geometric approach for learning unsupervised alignment of word embeddings between the source and the target languages. Our approach formulates the alignment learning problem as a domain adaptation problem over the manifold of doubly stochastic matrices. This viewpoint arises from the aim to align the second order information of the two language spaces. The rich geometry of the doubly stochastic manifold allows to employ efficient Riemannian conjugate gradient algorithm for the proposed formulation. Empirically, the proposed approach outperforms state-of-the-art optimal transport based approach on the bilingual lexicon induction task across several language pairs. The performance improvement is more significant for distant language pairs.

## introduction learning bilingual word embeddings is an important problem in natural language processing @xcite @xcite , with usage in cross-lingual information retrieval @xcite ), text classification @xcite , machine translation @xcite etc. given a sourcetarget language pair, the aim is to represent the words in both languages in a common embedding space. this is usually achieved by learning a linear function that maps word embeddings of one language to the embedding space of the other language @xcite . several works have focused on learning such bilingual mapping in supervised setting, using a bilingual dictionary during the training phase @xcite @xcite . recently, unsupervised bilingual word embeddings have also been explored @xcite @xcite @xcite @xcite . learning unsupervised cross-lingual mapping may be viewed as an instance of the more general unsupervised domain adaptation problem @xcite @xcite . the latter fundamentally aims at aligning the input feature (embeddings) distributions of the source and target domains (languages). in this paper, we take this point of view and learn cross-lingual word alignment by finding alignment between the second order statistics of the source and the target language embedding space. we formulate a novel optimization problem on the set of doubly stochastic matrices. the objective function consists of matching covariances of words from source to target languages in a leastsquares sense. for optimization, we exploit the fact that the set of doubly stochastic matrices has rich geometry and forms a riemannian manifold @xcite . the riemannian optimization framework @xcite @xcite allows to propose a computationally efficient conjugate gradient algorithm @xcite . experiments show the efficacy of the proposed approach on the bilingual lexicon induction benchmark, especially on the language pairs involving distant languages. ## motivation and related work we introduce the bilingual word alignment setup followed by a discussion on domain adaptation approaches. bilingual alignment. let @xmath0, re-spectively. the aim is to learn a linear operator w : r @xmath1. in the supervised setup, a list of source words and their translations in the target language is provided. this is represented by an alignment matrix y of size n × n, where y @xmath2. a standard way to learn orthogonal w is by solving the orthogonal procrustes problem @xcite , i.e., in the unsupervised setting, y is additionally unknown apart from w. most unsupervised works @xcite @xcite tackle this challenge by learning y and w jointly. however, their performance rely on finding a good initialization candidate for the alignment matrix y @xcite @xcite . performing optimization over the set of binary matrices, @xmath3, 1} n×n , to learn the bilingual alignment matrix is computationally hard. hence, some works @xcite view the source and the target word embedding spaces as two distributions and learn y as the transformation that makes the two distributions close. this viewpoint is based on the theory of optimal transport @xcite . y is, thus, modeled as a doubly stochastic matrix: the entries in @xmath4, 1] and each row/column sums to 1. permutation matrices are extreme points in the space of doubly stochastic matrices. @xcite propose learning the doubly stochastic y as a transport map between the metric spaces of the words in the source and the target languages. they optimize the gromov-wasserstein (gw) distance, which measures how distances between pairs of words are mapped across languages. for learning y, they propose to one modeling of interest is by @xcite , who motivate a linear transformation on the features in source and target domains. in @xcite , the linear map @xmath5 ## domain adaptation based cross-lingual alignment the domain adaptation solution strategies of @xcite can be motivated directly for the cross-lingual alignment problem by dealing with word covariances instead of feature covariances. however, the cross-lingual word alignment problem additionally has a bidirectional symmetry: if y aligns x to z, then y aligns z to x. we exploit this to propose a bi-directional domain adaptation scheme based on (3). the key idea is to adapt the second order information of the source and the target languages into each other's domain. we formulate the above as follows: fro adapts the domain of x (source) into z (target). equivalently, minimizing only the first term in the objective function of (4) leads to row indices in y x aligning closely with the row indices of z. similarly, minimizing only the second term yc z y -c x 2 fro adapts z (now treated as the source domain) into x (now treated as the target domain), which means that the row indices yz and x are closely aligned. overall, minimizing both the terms of the objective function allows to learn the alignment matrix y from x to z and y from z to x simultaneously. empirically, we observe that bi-directionality acts as a self regularization, leading to optimization stability and better generalization ability. the differences of the proposed formulation (4) with respect to the gw formulation (2) are two fold. first, the formulation (2) maximizes the inner product between y c x y and c z . this inner product is sensitive to differences in the norms of y c x y and c z . the proposed approach circumvents this issue since (4) explicitly penalizes entry-wise mismatch between y c x y and c z . second, the gw algorithm for (2) is sensitive to choices of the entropic regularization parameter @xcite . in our case, no such regularization is required. most recent works that solve optimal transport problem by optimizing over doubly stochastic matrices employ the sinkhorn algorithm with entropic regularization @xcite @xcite . in contrast, we exploit the riemannian manifold structure of the set of doubly stochastic matrices (ds n ) recently studied in @xcite . ds n is endowed with a smooth fisher information metric (inner product) that makes the manifold smooth @xcite @xcite . in differential geometric terms, ds n has the structure of a riemannian submanifold. this makes computation of optimization-related ingredients, e.g., gradient and hessian of a function, projection operators, and retraction operator, straightforward. leveraging the versatile riemannian optimization framework @xcite @xcite , the constrained problem (4) is conceptually transformed to an unconstrained problem over the nonlinear manifold. consequently, most unconstrained optimization algorithms generalize well to manifolds. we solve (4) using the riemannian conjugate gradient algorithm @xcite . there exist several manifold optimization toolboxes such as manopt @xcite , pymanopt @xcite , manopt.jl @xcite , mctorch @xcite or roptlib @xcite , which have scalable off-the-shelf generic implementation of riemannian algorithms. we use manopt for our experiments, where we only need to provide the objective function (4) and its derivative with respect to y. the manifold optimization related ingredients are handled by manopt internally. the computational cost per iteration of the algorithm is o(n 2 ), which is similar to that of gw @xcite . we term our algorithm as manifold based alignment (mba) algorithm. our code is available at https://pratikjawanpuria.com/ publications/ . ## experiments we compare the proposed algorithm mba with state-of-the-art gw alignment algorithm @xcite for the bilingual induction (bli) task. both the algorithms use second order statistics (word covariance matrices) to learn the word alignment between two languages. in our experimental setup, we first learn the word alignment between the source and the target languages and then compute cross-lingual mapping by solving the procrustes problem (1). for inference of nearest neighbors, we employ the cross-domain similarity local scaling (csls) similarity score @xcite . we report precision@1 (p@1) as in @xcite for the bli task. we show results on the muse dataset @xcite , which consists of fasttext monolingual embeddings for different languages @xcite and dictionaries between several languages (but mostly with english). follow- for gw, we use the original codes shared by @xcite and follow their recommendations on tuning the entropic regularization parameter and scaling of covariance matrices c x and c z . as a practical implementation of mba, we incrementally increase n starting from 1000 to 20 000 every fixed-number of iterations. we begin by discussing the results on six closeby european languages in ## conclusion aligning the metric spaces of languages has a wide usage in cross-lingual applications. a popular approach in literature is the gromov-wasserstein (gw) alignment approach @xcite @xcite , which constructs a transport map by viewing the two embedding spaces as distributions. in contrast, we have viewed unsupervised bilingual word alignment as an instance of the more general unsupervised domain adaptation problem. in particular, our formulation allows search over the space of doubly stochastic matrices and induces bi-directional mapping between the source and target words. both are motivated solely from the language perspective. the riemannian framework allows to exploit the geometry of the doubly stochastic manifold. empirically, we observe that the proposed algorithm mba outperforms the gw algorithm for learning bilingual mapping @xcite , demonstrating the benefit of geometric optimization modeling.