The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction (2022)

Abstract: In an effort to study the inductive biases of language models, numerous studies have attempted to use linguistically motivated tasks as a proxy of sorts, wherein performance on these tasks would imply an inductive bias towards a specific linguistic phenomenon. In this study, we attempt to analyse the inductive biases of language models with respect to natural language phenomena, in the context of building multilingual embedding spaces.We sample corpora from 2 sources in 15 languages and train language models on pseudo-bilingual variants of each corpus, created by duplicating each corpus and shifting token indices for half the resulting corpus. We evaluate the cross-lingual capabilities of these LMs, and show that while correlations with language families tend to be weak, other corpus-level characteristics, such as type-token ratio, tend to be more strongly correlated. Finally, we show that multilingual spaces can be built, albeit less effectively, even when additional destructive perturbations are applied to the training corpora, implying that (effectively) bag-of-words models also have an inductive bias that is sufficient for inducing multilingual spaces.

## introduction a variety of proxies and analytical methods have been used to study the inductive biases of language models towards natural language. this work includes targeted syntactic evaluation @xcite , language model responses to formulaic synthetic languages @xcite , as well as attempts to correlate differences in language modeling performance to language features over a wide range of languages @xcite . in this paper, we combine two strands that have, of late, been fairly active research threads. the first of these concerns the inductive biases of language models towards languages that exhibit a specific grammar; the second addresses the inductive biases of these models towards multilingualism, which in this context refers to a model's ability to build a multilingual space (rather than distinct monolingual spaces), when trained on corpora consisting of text in multiple languages. prior work in this domain is focused on either a) quantifying language model performance across a variety of languages, or b) studying the effects of different architectural components on the quality of the induced multilingual space. we attempt to unite the two strands of research by studying transformer-based masked language models in an effort to quantify the extent to which the grammar of the language being modelled affects the model's ability to build a multilingual space. we use @xcite metrics, namely word translation and sentence retrieval, as a proxy for the utility of this space. our main findings are: ## related work language modelling there has been a considerable amount of research addressing inductive biases that language models may have towards specific grammatical patterns, or towards natural languages with specific structures. an early study by @xcite demonstrates, over 21 languages, that certain languages are harder to model than others; the authors find that model performance correlates with the richness of a language's (inflectional) morphology. later work by @xcite shows contradictory findings; the authors extend these experiments to 69 languages and find that morphological complexity does not correlate as strongly with performance as simpler factors like vocabulary size and sentence length do. other work involves studying how language modelling is affected by manually altering corpora. @xcite train rnn-based models on english, altered to display different word orders and different degrees of morphological agreement; @xcite generate corpora of natural language sentences, with constituents permuted based on boolean switches, and show that recurrent language models show little variance in performance across word orders, compared to transformers. multilingualism moving beyond monolingual language modelling, we examine the numerous works analysing what precisely multilingual language models need, in order to form an adequate multilingual space, which is quantified by measuring a model's performance on some multilingual task. @xcite show that subword overlap tends to improve multilingual alignment, though overlap is by no means necessary, as languages with different scripts can exist in the same multilingual space. @xcite show that while structurally similar languages do not necessarily need subword overlap, dissimilar languages rely heavily on overlap; they also show that wellaligned non-contextual word embedding spaces allow for better transfer. on the other hand, @xcite have somewhat contradictory results, and show that neither shared vocabulary items nor joint pre-training are essential to build a multilingual encoder. @xcite and @xcite analyse encoders from an architectural point of view. the former work shows that model depth (and not the number of attention heads) contributes to transfer performance, even when the number of parameters is kept constant. the latter points out that multilingual spaces exist because languages are forced to share parameters, and that even in the absence of shared subwords and special tokens, position embeddings play a significant role in building these spaces. @xcite go on to show that the removal of shared position embeddings is sufficient to reduce a model's multilingual performance (as measured on word translation and sentence retrieval) to approximately random. this, we show, is not universally the case. ## results we present results per language and experiment on common crawl (top) and wikipedia (bottom) in figure 1 . we begin by making a few general observations before moving on to study correlations with morphosyntactic and corpus factors. bpe makes word translation harder despite controlling for non-initial subwords, using bpe tokenisation results in a drop in translation score for all our experiments. we hypothesise that this is due to common word-initial subwords being distributionally 'overloaded'; they are more likely to appear in a wider range of contexts than whole tokens are, due to the variety in consecutive subwords. multilingualism is robust to lemmatisation perhaps somewhat unsurprisingly, lemmatisation does not significantly affect model scores, indicating that our model relies more on word order to build multilingual spaces. interestingly, removing bpe segmentation results in an increase in fails on lemmatised corpora. ## conclusion in this work, we attempted to measure the variance in the ability of masked language models to build multilingual spaces with the underlying typology of the language. in doing so, we have shown that these models are capable of building multilingual spaces even when sentences are lemmatised and scrambled at a token level, showing that multilingualism can exist even when transformers act, functionally, like bag-of-words models. this does not, however, necessarily imply the ability to effectively model language @xcite , but merely the ability to align two disjoint linguistic spaces. we have also shown that, on the one hand, the ability to build a multilingual space is only weakly correlated to language (given multiple corpora) and to language family, and that, on the other hand, certain corpus-level metrics (specifically, type-token ratios and the presence of hapax legomena) are relatively good predictors of multilingual space quality, while others (such as the number of tokens or the average sentence length) are negatively correlated. our work is not without its caveats. for one, a lot of our correlating factors muddy the waters between what is an inherent property of the language itself, and what is a property of the corpus we use. while we use texts from the same domain in all our languages, both wikipedia and common crawl are widely inconsistent across language, unless explicitly made comparable (otero and lópez, 2010). further, as discussed earlier, our scenario is not strictly realistic: first, this is a bilingual setup meant to approximate a multilingual one; second, both our languages have exactly the same structure; third, our language models are very underparame-terised relative to full-scale models. it is unlikely that our observations would hold true in a realworld scenario; given, however, that our aim was to study the inductive biases of masked language models, using full-scale models would defeat the purpose somewhat, as the sheer volume of training data would have overridden these biases. having said that, we present this work as an attempt to add to the often conflicting pool of papers attempting to shed some light on how language models acquire language.