I so V ec: Controlling the Relative Isomorphism of Word Embedding Spaces (2022)

Abstract: The ability to extract high-quality translation dictionaries from monolingual word embedding spaces depends critically on the geometric similarity of the spaces—their degree of “isomorphism.” We address the root-cause of faulty cross-lingual mapping: that word embedding training resulted in the underlying spaces being non-isomorphic. We incorporate global measures of isomorphism directly into the skipgram loss function, successfully increasing the relative isomorphism of trained word embedding spaces and improving their ability to be mapped to a shared cross-lingual space. The result is improved bilingual lexicon induction in general data conditions, under domain mismatch, and with training algorithm dissimilarities. We release IsoVec at https://github.com/kellymarchisio/isovec.

## introduction the task of extracting a translation dictionary from word embedding spaces, called "bilingual lexicon induction" (bli), is a common task in the natural language processing literature. bilingual dictionaries are useful in their own right as linguistic resources, and automatically generated dictionaries may be particularly helpful for low-resource languages for which human-curated dictionaries are unavailable. bli is also used as an extrinsic evaluation task to assess the quality of cross-lingual spaces. if a high-quality translation dictionary can be automatically extracted from a shared embedding space, intuition says that the space is highquality and useful for downstream tasks. mapping methods are effective for closelyrelated languages with embedding spaces trained on high-quality, domain-matched data even without supervision, but critically rely on the "approximate isomorphism assumption"-that monolingual embedding spaces are geometrically similar. 1 problematically, researchers have observed that the isomorphism assumption weakens substantially as languages and domains become dissimilar, leading to failure precisely where unsupervised methods might be helpful (e.g. @xcite @xcite @xcite . existing work attributes non-isomorphism to linguistic, algorithmic, data size, or domain differences in training data for source and target languages. from @xcite , "the performance of unsupervised bdi [bli] depends heavily on... language pair, the comparability of the monolingual corpora, and the parameters of the word embedding algorithms." several authors found that unsupervised machine translation methods suffer under similar data shifts @xcite @xcite . while such factors do result in low isomorphism of spaces trained with traditional methods, we needn't resign ourselves to the mercy of the geometry a training methodology naturally produces. while multiple works post-process embeddings or map non-linearly, we control similarity explicitly during embedding training by incorporating five global metrics of isomorphism into the skip-gram loss function. our three supervised and two unsupervised losses gain some control of the relative isomorphism of word embedding spaces, compensating for data mismatch and creating spaces that are linearly mappable where previous methods failed. ## background we discuss the mathematical background used in our methods. throughout, @xmath0, respectively. we may assume seed pairs {(x 0 , y 0 ), (x 1 , y 1 ), ...(x s , y s )} are given. ## method we implement skip-gram with negative sampling on gpu using pytorch and use it to train monolingual embedding spaces for bengali (bn), ukra-nian (uk), tamil (ta), and english (en). 5 our implementation mirrors the official word2vec foot_5 release closely @xcite . we create comparison embedding spaces using the official word2vec release with default hyperparameters and map the resulting spaces from both algorithms with vecmap for bli. we report precision@1 (p@1) on the development set in ## experiments & results we pretrain english embeddings to use as reference space y. isovec trains source space x. ## conclusion & future work we present isovec, a new method for training word embeddings which directly injects global measures of embedding space isomorphism into the skipgram loss function. our three supervised and two unsupervised isomorphism loss functions successfully improve the mappability of monolingual word embedding spaces, leading to improved ability to induce bilingual lexicons. isovec also shows promise under algorithm mismatch, domain mismatch, and data size mismatch between source and target training corpora. future work could extend our work to even greater algorithmic mismatches, and in massively multilingual contextualized models. we release isovec at https://github. com/kellymarchisio/isovec .