A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space (2021)

Abstract: In cross-lingual language models, representations for many different languages live in the same space. Here, we investigate the linguistic and non-linguistic factors affecting sentence-level alignment in cross-lingual pretrained language models for 101 languages and 5,050 language pairs. Using BERT-based LaBSE and BiLSTM-based LASER as our models, and the Bible as our corpus, we compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance, as well as four intrinsic measures of vector space alignment and isomorphism. We then examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics. The results of our analyses show that word order agreement and agreement in morphological complexity are two of the strongest linguistic predictors of cross-linguality. We also note in-family training data as a stronger predictor than language-specific training data across the board. We verify some of our linguistic findings by looking at the effect of morphological segmentation on English-Inuktitut alignment, in addition to examining the effect of word order agreement on isomorphism for 66 zero-shot language pairs from a different corpus. We make the data and code for our experiments publicly available.

## introduction cross-lingual language models are polyglots insofar as they house representations for many different languages in the same space. but to what extent are they good polyglots? the answer depends, in part, on how well-aligned and isomorphic the representations are, and not all language pairs are equally well-aligned. what determines the quality of the alignment? are language pairs from the same family (e.g., spanish and french) better figure 1 : a look at some of the strongest features for predicting cross-linguality, according to their number of occurrences in best-feature regression searches across all our dependent variables (see section 6.2). recent work has looked at the typological and training-related factors affecting cross-lingual alignment in monolingual embedding space @xcite , assessed the cross-linguality of pretrained language models using probing tasks and downstream performance measures @xcite @xcite @xcite , and probed transformer models @xcite for linguistic structure (see @xcite for an overview of over 150 studies). however, a gap in the research exists regarding the following question: what are the linguistic, quasi-linguistic, and training-related factors determining the crosslinguality of sentence representations in shared embedding space, and what are the relative weights of these factors? we argue, that given the importance of alignment in multilingual model performance, gaining fundamental insight into what affects inter-language alignment and isomorphism-specifically by exploring which linguistic factors matter for inter-language alignment-will make it possible to leverage existing information on linguistic typology to improve alignment (and thereby task performance) for low-resource languages. our contributions are as follows: ## related work various studies have assessed the cross-linguality of pretrained language models. recent efforts have approached this question via performance on an array of downstream nlp tasks @xcite @xcite @xcite , and others have proposed methods for better cross-lingual alignment in light of systematic crosslingual deficiencies @xcite . our study hews closest methodologically to @xcite and @xcite , who investigate the determinants of cross-lingual isomorphism using monolingual fasttext embeddings @xcite @xcite . findings from these studies have been mixed, but some patterns emerge. @xcite and @xcite find that cross-lingual transfer works best between typologically similar language pairs, in particular between languages that share word order features. @xcite approach cross-linguality by focusing on zero-shot cross-lingual transfer in mbert, and show that each mbert layer retains language-specific information and that token overlap correlates with cross-lingual performance. @xcite home in on low-resource languages, finding that they often fail to reap the benefits of massively multilingual joint training but that their performance can be boosted by providing similar-language training data. somewhat contrary to others' results (including ours), @xcite find that lexical overlap factors in negligibly to cross-lingual transfer, while the depth of the network is integrally important. vulić et al. (2020) and dubossarsky et al. (2020 ) look at how typological features, trainingrelated factors, and measures of vector space isomorphism predict cross-lingual performance between monolingual word embeddings. @xcite find in their experiments that cross-lingual performance depends mostly on training data and regimes, while @xcite see more mixed results from their experiments: they show that linguistic typology is important, but not deterministic, for predicting cross-lingual performance. our work not only replicates these findings for monolingual spaces in the multilingual embedding space (e.g. on word order similarity, related training data, typological distance, subword overlap), but extends that work through: (1) the scale (101 languages, 5,050 language pairs in the main analysis); (2) the quantity and diversity of predictors (13 linguistic, quasi-linguistic, and training-related features); (3) the models (cross-lingual sentence encoders with different architectures); and (4) the analytic methods (a blend of prediction-based and classical statistical techniques, supplemented by performance-based case studies on extremely low-resource languages). ## bible corpus the source of the bitexts we evaluate on is the superparallel bible corpus 2 from christodouloupoulos and steedman (2014), whence we gather texts for 101 languages and bitexts for 5,050 language pairs. 3 we evaluate on the books of matthew and john in the new testament separately and average the results, as these parts are available for all 101 languages. in doing so, we avoid the pitfalls of 2 http://christos-c.com/bible/ . we use this corpus because it is massively multilingual and well-aligned. ## measures of cross-lingual alignment & isomorphism we formulate alignment metrics in two distinct ways: over language pairs and over individual languages. the latter group is computed from the former by averaging over all pairs in which a language appears. for example, to derive the average f1score for chinese, we average over the f1-scores for chinese-german, chinese-amuzgo, etc. some metrics we use are measures of vector subspace isomorphism (i.e. those examined in dubossarsky et al. 2020), while others are measures of alignment (namely those pertaining to bitext retrieval). vector spaces may be isomorphic without being well-aligned, so we quantify multilinguality in diverse ways. ## analysis 6.1 simple correlations training data we first look at simple correlations between the training data quantities and the dependent variables (measures of alignment/isomorphism). results for language pairs are given across all dependent variables for labse and laser in ## zero-shot cases the linguistic diversity of the new testament bible corpus, combined with the imperfect overlap between the languages in the corpus and those on which labse and laser were trained, implies a large number of zero-shot cases for our analysis. we can break these cases into two sub-cases. first, there are languages in the bible corpus without language-specific training data (35 languages for labse, 45 languages for laser) 17 . but it follows that there are language pairs xx-yy for which no training data is present for either xx or yy (595 pairs for labse, 990 pairs for laser), which we dub the "double zero-shot" case. simple zero-shot case for the simple zero-shot case, we use anova (analysis of variance) to investigate differences between group means within categorical variables. anovas revealed large effects (η 2 p ≈ 0.36) of basic word order on f1-score and econd-hm for labse, with borderline p-values (p ≈ 0.07), perhaps due to the small sample size (35 languages). the breakdown across word orders for zero-shot languages is given in figure 2 . a pairwise tukey post-hoc test (salkind, 2017) revealed a borderline-significant difference between svo and vso languages, surprisingly in favor of vso. there were no statistically significant effects of polysynthesis for labse or laser. interestingly, this suggests that agreement in morphological complexity may be important for cross-linguality, but morphological complexity itself is not an important factor. more work is needed to validate this conclusion. anovas also showed large (η 2 p ≈ 1) effect sizes for family and subfamily membership, though most results were not statistically significant (again, perhaps due to sample size). this suggests that phylogenetic membership still shapes cross-linguality even when training data is perfectly controlled for, which is an interesting finding. double zero-shot case interestingly, the language-pair-specific feature which stood out most in the double zero-shot case was inventory distance, an anomaly in our analyses. inventory distance was correlated with r ≈ 0.2-0.4 for 4/5 dependent variables for labse and with r ≈ 0.13-0.14 for 2/5 dependent variables for laser. however, as inventory distance quantifies phonological distance between languages, it could be confounded with surface-level information. to test this hypothesis, we regress it with character-level overlap and token-level overlap separately. for labse, effects of inventory distance remain significant (@xmath0.05) for all dependent variables when regressing with token-level overlap, and 4/5 variables when regressing with character-level overlap. we wish to verify the importance of this feature in future studies. ## case study 1: morphological segmentation of inuktitut based on the above results, we conclude that whether a language has the same polysynthesis status as another language will affect their success on a cross-lingual task. however, our observations pertain to correlation, not causality. to test this observation further, we run an experiment in which we introduce a causal intervention. if indeed polysynthesis status matters, then we hypothesize that making a language "less polysynthetic" will improve alignment with a more analytic language like english. to test this hypothesis, we examine the effect of morphological segmentation of inuktitut on the bitext retrieval task. inuktitut is a polysynthetic, indigenous language and is completely zero-shot for both our models, in that not even in-family data is provided during pretraining. the intuition behind our experiment is that by morphologically segmenting a polysynthetic language, the "polysynthesis status" of the segmented inuktitut is made closer to that of a more analytic language. if our previous findings are correct, we expect inuktitut to align better with english post-segmentation. we use the first 10,000 sentences from the nunavut hansard inuktitut-english parallel corpus @xcite as our bitext. for the inuktitut half of the corpus, we use both the "raw" version and a version that has been pre-segmented with the uqailaut morphological analyzer 18 . we then perform bitext retrieval as described in section 4.1 on both bitexts: english aligned with non-segmented inuktitut and english aligned with segmented inuktitut. results in terms of f1-score are displayed in figure 3 . for labse, we see a +28.7 (≈ 5×) f1-score increase using segmented inuktitut; for laser, we see a +0.04 (1.5×) increase. these empirical results support our earlier statistical findings on the feature same polysynthesis status. ## case study 2: word order & isomorphism to test the validity of our findings on the same word order feature, we examine whether embeddings in languages with similar word orders are more isomorphic to each other than those with substantially different word orders, sampling from a different corpus than the one we use in our main analysis. to this end, we select twelve zero-shot 19 languages from the universal declaration of human rights (udhr) parallel corpus @xcite . six of these are canonically verb-initial: k'iche', mam, chinanteco, tzotzil, mixteco, and garifuna. the other six are subject-initial: chickasaw, quechua, achuar-shiwiar, bambara, dagaare, and guarani. we hypothesize that similar-word-order language pairs will be more isomorphic, on average, than pairs of languages with disparate word orders. we compute svg and econd-hm across all 12 2 = 66 language pairs for labse and laser separately and group the results based on whether the language pairs have similar word order or different word order. the averages of these groups are given in ## conclusions we find evidence that linguistic and quasi-linguistic factors continue to play a role in determining the cross-linguality of a model even after training data is accounted for, and validate our findings with two case studies on extremely low-resource languages. our analysis points to, among other things, the importance of word order agreement (similarly to @xcite ) and morphology agreement on building aligned and isomorphic cross-lingual subspaces. we also rigorously demonstrate the importance of in-family training data in building massively multilingual models, and show moderate effects of other typological measures on crosslinguality. in the future, we are confident that these insights can be used to improve the cross-linguality of shared embedding spaces, particularly for lowresource languages.