Are All Good Word Vector Spaces Isomorphic? (2020)

Abstract: Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic. As a result, they perform poorly or fail completely on non-isomorphic spaces. Such non-isomorphism has been hypothesised to result from typological differences between languages. In this work, we ask whether non-isomorphism is also crucially a sign of degenerate word vector spaces. We present a series of experiments across diverse languages which show that variance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the monolingual resources available, and to the properties and duration of monolingual training (e.g. “under-training”).

## introduction word embeddings have been argued to reflect how language users organise concepts @xcite . the extent to which they really do so has been evaluated, e.g., using semantic word similarity and association norms @xcite , and word analogy benchmarks @xcite . if word embeddings reflect more or less languageindependent conceptual organisations, word embeddings in different languages can be expected to be near-isomorphic. researchers have exploited this to learn linear transformations between such spaces @xcite , which have been used to induce bilingual dictionaries, as well as to facilitate multilingual modeling and cross-lingual transfer @xcite . in this paper, we show that near-isomorphism arises only with sufficient amounts of training. this is of practical interest for applications of linear alignment methods for cross-lingual word embeddings. it furthermore provides us with an explanation for reported failures to align word vector spaces in different languages @xcite , which has so far been largely attributed only to inherent typological differences. in fact, the amount of data used to induce the monolingual embeddings is predictive of the quality of the aligned cross-lingual word embeddings, as evaluated on bilingual lexicon induction (bli). consider, for motivation, figure 1 ; it shows the performance of a state-of-the-art alignment method-rcsls with iterative normalisation @xcite -on mapping english embeddings onto embeddings in other languages, and its correlation (ρ = 0.72) with the size of the tokenised target language polyglot wikipedia @xcite . we investigate to what extent the amount of data available for some languages and corresponding training conditions provide a sufficient explanation for the variance in reported results; that is, whether it is the full story or not: the answer is 'almost', that is, its interplay with inherent typological differences does have a crucial impact on the 'alignability' of monolingual vector spaces. we first discuss current standard methods of quantifying the degree of near-isomorphism between word vector spaces ( §2.1). we then outline training settings that may influence isomorphism ( §2.2) and present a novel experimental protocol for learning cross-lingual word embeddings that simulates a low-resource environment, and also controls for topical skew and differences in morphological complexity ( §3). we focus on two groups of languages: 1) spanish, basque, galician, and quechua, and 2) bengali, tamil, and urdu, as these are arguably spoken in culturally related regions, but have very different morphology. our experiments, among other findings, indicate that a lowresource version of spanish is as difficult to align to english as quechua, challenging the assumption from prior work that the primary issue to resolve in cross-lingual word embedding learning is language dissimilarity (instead of, e.g., procuring additional raw data for embedding training). we also show that by controlling for different factors, we reduce the gap between aligning spanish and basque to english from 0.291 to 0.129. similarly, under these controlled circumstances, we do not observe any substantial performance difference between aligning spanish and galician to english, or between aligning bengali and tamil to english. we also investigate the learning dynamics of monolingual word embeddings and their impact on bli performance and near-isomorphism of the resulting word vector spaces ( §4), finding training duration, amount of monolingual resources, preprocessing, and self-learning all to have a large impact. the findings are verified across a set of typologically diverse languages, where we pair english with spanish, arabic, and japanese. we will release our new evaluation dictionaries and subsampled wikipedias controlling for topical skew and morphological differences to facilitate future research at: github.com/ cambridgeltl/iso-study. ## isomorphism of vector spaces studies analyzing the qualities of monolingual word vector spaces have focused on intrinsic tasks @xcite , correlations @xcite , and subspaces @xcite . in the cross-lingual setting, the most impor-tant indicator for performance has been the degree of isomorphism, that is, how (topologically) similar the structures of the two vector spaces are. ## simulating low-resource settings as low-resource languages-by definition-have only a limited amount of data available, we cannot easily control for all aspects using only a lowresource language. instead, we modify the training setup of a high-resource language to simulate a lowresource scenario. for most of our experiments, we use english (en) as the source language and modify the training setup of spanish (es). additional results where we modify the training setup of english instead are available in the appendix; they further corroborate our key findings. we choose this language pair as both are similar, i.e. indo-european, high-resource, and bli performance is typically very high. despite this high performance, unlike english, spanish is a highly inflected language. in es wikipedia sample order to inspect if similar patterns also hold across typologically more dissimilar languages, we also conduct simulation experiments with two other target languages with large wikipedias in lieu of spanish: japanese (ja, an agglutinative language) and arabic (ar, introflexive). when controlling for corpus size, we subsample the target language (i.e., spanish, japanese, or arabic) wikipedia to obtain numbers of tokens comparable to low-resource languages as illustrated in to control for topical skew, we need to sample similar documents as in low-resource languages. to maximise topical overlap, we choose lowresource languages that are spoken in similar regions as spanish and whose wikipedias might thus also focus on similar topics-specifically basque (eu), galician (gl), and quechua (qu). these four languages have very different morphology. quechua is an agglutinative language, while spanish, galician, and basque are highly inflected. basque additionally employs case marking and derivation. if non-isomorphism was entirely explained by language dissimilarity, we would expect even low-resource versions of spanish to have high bli performance with english. we repeat the same experiment with another set of languages with distinct properties but spoken in similar regions: bengali, urdu, and tamil. typological differences however, may still explain part of the difference in performance. for instance, as we cannot simulate basque by changing the typological features of spanish 5 , we instead 5 ravfogel et al. (2019) generate synthetic versions of english that differ from english in a single typological parameter. this process requires a treebank and is infeasible for all typo-make spanish, basque, galician, and quechuan "morphologically similar": we remove inflections and case marking through lemmatisation. we follow the same process for bengali, urdu, and tamil. ## conclusion we have provided a series of analyses that demonstrate together that non-isomorphism is not-as previously assumed-primarily a result of typological differences between languages, but in large part due to degenerate vector spaces and discrepancies between monolingual training regimes and data availability. through controlled experiments in simulated low-resource scenarios, also involving languages with different morphology that are spoken in culturally related regions, we found that such vector spaces mainly arise from poor conditioning during training. the study suggests that besides improving our alignment algorithms for distant languages @xcite , we should also focus on improving monolingual word vector spaces, and monolingual training conditions to unlock a true potential of cross-lingual learning.