Non-Linear Instance-Based Cross-Lingual Mapping for Non-Isomorphic Embedding Spaces (2020)

Abstract: We present InstaMap, an instance-based method for learning projection-based cross-lingual word embeddings. Unlike prior work, it deviates from learning a single global linear projection. InstaMap is a non-parametric model that learns a non-linear projection by iteratively: (1) finding a globally optimal rotation of the source embedding space relying on the Kabsch algorithm, and then (2) moving each point along an instance-specific translation vector estimated from the translation vectors of the point’s nearest neighbours in the training dictionary. We report performance gains with InstaMap over four representative state-of-the-art projection-based models on bilingual lexicon induction across a set of 28 diverse language pairs. We note prominent improvements, especially for more distant language pairs (i.e., languages with non-isomorphic monolingual spaces).

## introduction and motivation induction of cross-lingual word embeddings (clwes) @xcite @xcite @xcite has been one of the key mechanisms for enabling multilingual modeling of meaning and facilitating cross-lingual transfer for downstream nlp tasks. even though clwes are recently being contested in cross-lingual downstream transfer by pretrained multilingual language models @xcite @xcite @xcite , they are still paramount in word-level translation, that is, bilingual lexicon induction (bli). while earlier work focused on joint induction of multilingual embeddings from multilingual corpora, relying on word- @xcite @xcite , sentence- @xcite @xcite @xcite , or document-level @xcite @xcite alignments, most recent efforts focus on post-hoc alignment of independently trained monolingual embeddings (the so-called projection or mapping approaches) @xcite @xcite @xcite . despite some recent evidence that joint clwe induction may lead to better bilingual spaces @xcite , projection-based methods still dominate the field @xcite @xcite @xcite , inter alia) due to their conceptual attractiveness: they operate on top of vectors produced with any embedding model and need at most a few thousand word pairs of supervision @xcite . most projection-based clwe models induce bilingual spaces by orthogonally projecting one monolingual space to another. since orthogonal projections do not affect the topology of the source space, the performance of these methods is bound by the degree of isomorphism of the two monolingual spaces. yet, evidence suggests that monolingual spaces, especially those of etymologically and typologically distant languages, are far from isomorphic @xcite @xcite . what is more, unsupervised clwe models @xcite @xcite , which additionally exploit the isomorphism assumption when inducing initial translation dictionaries, have been shown to yield near-zero bli results for pairs of distant languages @xcite . following these theoretical limitations of effectiveness of orthogonal mapping between nonisomorphic spaces, @xcite and @xcite relax the orthogonality constraint and report bli improvements. these models, however, still learn only a linear transformation, i.e., an oblique projection matrix. while oblique projections may scale or skew the source space, there still exists a strong topological similarity between the original space and its oblique projection. in this work, we deviate from learning a linear projection matrix (i.e., a parametric model) and propose a non-parametric model which translates vectors by estimating instance-specific geometric translations. our method, instamap, iteratively (1) applies the kabsch algorithm (horn, 1987) on the full training dictionary to learn a globally optimal rotation of the source space w.r.t. the target space; and then (2) translates each point along the instance-specific translation vector, which we compute from the translation vectors of the point's nearest neighbours from the training dictionary. we extensively evaluate instamap on the benchmark bli dataset @xcite encompassing 28 diverse language pairs. our results show the non-linear mappings with instamap to be substantially more robust than linear projections, both orthogonal @xcite and oblique @xcite . we also show that, unlike instamap, oblique projection models -rcsls @xcite and bliss @xcite -cannot surpass the performance of the best-performing orthogonal projection model vecmap @xcite for distant languages (i.e., for low isomorphicity). finally, we report additional significant gains by applying instamap on top of vecmap. ## instance-based mapping the core idea of instamap is illustrated in figure 1 . we iteratively: (1) use the entire training dictionary to learn a single global rotation matrix and then (2) perform an instance-based computation of translation vectors. ## evaluation we evaluate instamap on bilingual lexicon induction, the standard task for evaluating clwes. ## conclusion we have proposed instamap, a simple and effective approach for improving the post-hoc cross-lingual alignment between non-isomorphic monolingual embedding spaces. unlike existing projection-based clwe induction models, which learn a global linear projection matrix, instamap couples global rotation with instance-specific translations. this way, we learn a globally non-linear projection. our experiments show that (1) in-stamap significantly outperforms four state-ofthe-art projection-based clwe models on a benchmark bli dataset with 28 language pairs and (2) that it yields largest improvements for pairs of distant languages with a lower degree of isomorphism between their respective monolingual spaces. we plan to extend this work in two directions. first, we will explore mechanisms for instance-specific translation that are more sophisticated than the aggregation of translation vectors of nearest dictionary neighbours. second, we plan to couple instancebased mapping with other informative features (e.g., character-level features) in classificationbased bli frameworks @xcite . the instamap code is available at: https://github.com/codogogo/instamap .