A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank (2022)

Abstract: We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models. We inspect zero-shot performance in balanced data conditions to mitigate data size confounds, classifying pretraining languages that improve downstream performance as donors, and languages that are improved in zero-shot performance as recipients. We develop a method of quadratic time complexity in the number of languages to estimate these relations, instead of an exponential exhaustive computation of all possible combinations. We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. Our findings can inform developers of large-scale multilingual language models in choosing better pretraining configurations.

## introduction pretrained language models are setting state-of-theart results by leveraging raw texts during pretraining (plms; peters et al., 2018; devlin et al., 2019, inter alia). interestingly, when pretraining on multilingual corpora, plms seem to exhibit zero-shot cross-lingual abilities, achieving non-trivial performance on downstream examples in languages seen only during pretraining. for example, in the bottom of figure 1 , a named entity recognition model finetuned on russian is capable of predicting correctly name entity tags for texts in english, seen only during pretraining (pires et al., 2019; conneau et al., 2020b; k et al., 2020; conneau et al., 2020a; lazar et al., 2021; turc et al., 2021). previous analyses examined how several factors contribute to this emerging behavior. for example, parameter sharing and model depth are important in certain configurations (k et al., 2020; conneau et al., 2020b), as well as typological similarities figure 1 : we build a complete, directed graph over a diverse set of 22 languages. weighted edges show the improvement of bilingual lm over monolingual performance (bold edges represent larger weights). languages which consistently improve performance are termed "donors" and marked in red, while languages which benefit most are termed "recipients" (marked in blue). we show that our observations hold in several configurations on two downstream tasks. in this work, we focus on an important factor that we find missing in prior work, namely the effect that pretraining languages have on downstream zero-shot performance. in particular, we ask three major research questions: (1) does the choice of pretraining languages affect downstream crosslingual transfer, and if so, to what extent? (2) is english the optimal pretraining language, when controlling for confounding factors such as data size and domain? and finally, (3) can we choose pretraining languages to improve downstream zeroshot performance? in addressing these research questions, we aim to decouple the language from its corresponding dataset. to the best of our knowledge, prior work has conflated pretrain corpus size and its domain with other examined factors, thus skewing results towards over-represented languages, such as english or german (joshi et al., 2020). @xcite to achieve this, we first construct a linguistically-balanced pretraining corpus based on wikipedia, composed of a diverse set of 22 languages. we carefully control for the amount of data and domain distribution in each of the languages (section 3). next, since the number of pretraining configurations grows exponentially with the number of languages n represented in the dataset, it is infeasible to exhaustively test all possible configurations, much less extend it for more languages. @xcite in section 4 we propose a novel pretraining-based approach that is quadratic in the number of languages. this is achieved by training all n 2 combinations of bilingual masked language models over our corpus, thus yielding a complete directed graph (figure 1 ), where an edge l 1 → l 2 estimates how much a language l 1 contributes to zero-shot performance in language l 2 , based only on language modeling performance. in section 5, we use the graph to identify languages which generally contribute as pretraining languages (termed "donors"), and languages which often benefit from training with other languages (termed "recipients"). further, we use the graph to make observations regarding the effect of typological features on bilingual language modeling, and make available an interactive graph explorer. finally, our evaluations on two multilingual downstream tasks (part of speech tagging and named entity recognition) lead to three main conclusions (section 6): (1) the choice of pretraining languages indeed leads to differences in zero-shot performance; (2) controlling for the amount of data allotted for each language during pretraining ques-tions the primacy of english as the main pretraining language; and (3) our hypotheses regarding donors and recipient language hold in both downstream tasks, and against two additional control groups. ## metrics for pretraining-aware cross-lingual transfer in this section, we extend existing metrics for zeroshot cross-lingual transfer to account for pretraining languages. intuitively, our metrics for a model m and a given downstream task take into account three factors: (1) p , the set of languages seen during pretraining, (2) @xmath0, the source language used for finetuning, and (3) @xmath1, the target language, seen during inference. formally, we adapt the formulation of hu et al. (2020) to define a pretraining-aware bilingual zeroshot transfer score z as: foot_2 where m p,l is a model pretrained on the set of languages p and finetuned on downstream task instances in the language @xmath2, and ε(m, l) is an evaluation of model m on instances in language l in terms of the downstream metric, e.g., word-label accuracy for part of speech tagging. following, we extend the definition of zero-shot transfer score to a set of downstream test languages d ⊆ p to measure p 's aggregated effect on zeroshot performance, by averaging over all bilingual transfer combinations in d: in the following sections, we will use these metrics to evaluate how different choices for pretraining languages influence downstream performance. ## data selection we collect a pretraining dataset to test the effect of pretraining languages on cross-lingual transfer. first, we choose a set of 22 languages from 9 language families, as listed in second, we aim to balance the amount of data and control for its domain across languages, to mitigate possible confounders in our evaluations. below we outline design choices we make toward this goal. ## bilingual pretraining graph in this section, we describe a method for estimating the effect that different pretrain language combinations would have on downstream zero-shot performance. this is achieved by evaluating bilingual performance on the pretraining masked language modeling (mlm) task. we begin by describing our experimental setup, hyperparameters and hardware configuration (section 4.1). in section 4.2, we outline our estimation method, yielding a complete graph structure over our languages, amenable for future exploration and analyses (figures 1 , 2 ). in the following sections, figure 2 : bilingual finetune scores between language pairs in our balanced corpus. coordinate (i, j) represents f(l @xmath3, i.e., the performance in mrr[%] (which correlates with perplexity) of an lm pretrained on a bilingual corpus over languages (l i , l j ) and tested intrinsically on l j . the last column (marked don.) sums over each line, i.e., index i in the column represents how much language i donated to all other languages. similarly, the j'th index in the last row (marked recp.) sums over column j and represents how much language l j improved in all configurations. ## pretraining graph analysis we present several key observations based on the bilingual pretraining graph described in the previous section and summarized by the adjacency matrix in figure 2 , as well as an interactive exploration interface. in the following sections, we use these observations in our downstream evaluations. ## limitations and future work as with other works on cross-lingual transfer, our results are influenced by many hyperparameters. below we explicitly define our design choices and how they can be explored in future work. first, data scarcity in low-resource languages restricted us to small data amounts. although our experiments showed a non-trivial signal for pretraining and downstream tasks, future work may apply our framework to larger data sizes. second, for efficiency's sake, we trained relatively small models to enable us to train a large number of language configurations, while ensuring convergence in 6 languages. furthermore, we did not do any hyper-parameter tuning and used only values reported in previous work, and use only the bert architecture. future work may revisit any of these design choices to shed more light on their effect. third, similarly to other works, our data was scraped from wikipedia, and we did not account for language contamination across supposedly monolingual corpora (e.g., due to code switching). such contamination may confound with cross-lingual transfer, as was recently shown by blevins and zettlemoyer (2022). finally, our downstream analysis focused on pos tagging and ner since they were available for many languages and are common in many nlp pipelines. further experimentation can test if our results hold for more nlp tasks. ## related work to the best of our knowledge, this is the first work to control for the amount of data allocated for each language during pretraining and finetuning while evaluating on many languages. perhaps most related to our work, turc et al. (2021) challenge the primacy of english as a source language for cross-lingual transfer in various downstream tasks. their work shows that german and russian are often more effective sources. in all of their experiments, they use mbert's imbalanced pretraining corpus. blevins and zettlemoyer (2022) complement this hypothesis by showing that english pretraining data actually contains a significant amount of non-english text, which correlates with the model's transfer capabilities. wu and dredze (2020) evaluate how mbert performs on a wide set of languages, focusing on the quality of representation for low-resource languages in various downstream tasks by defining a scale from low to high resource. they show that mbert underperforms non bert monolingual baselines for low resource languages while performing well for high resource ones. while pires et al. (2019); limisiewicz and mareček (2021) show that typology plays a significant role for mbert's multilingual performance, this is not replicated in our balanced evaluation, and has lesser impact in wu et al. (2022) as well. finally, conneau et al. (2020a) introduce the transfer-interference trade-off where low resource languages benefit from multilingual training, up to a point where the overall performance on monolingual and cross-lingual benchmarks degrades. ## conclusions we explored the effect of pretraining language selection on downstream zero-shot transfer. we first choose a diverse pretraining set of 22 languages, and curate a pretraining corpus which is balanced across these languages. second, we devise an estimation technique, quadratic in the number of languages, projecting which pretraining languages will serve better in cross-lingual transfer and which specific downstream languages will do best in that setting. finally, we test our hypothesis on two downstream multilignual tasks, and show that the choice of pretraining languages indeed leads to varying downstream cross-lingual results, and that our method is a good estimation for downstream performance. taken together, our results suggest that pretraining language selection should be a factor in estimating cross-lingual transfer, and that current practices which focus on high-resource languages may be sub-optimal.