A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank
Abstract
We show that the choice of pretraining languages affects downstream cross-lingual transfer for BERT-based models.
We inspect zeroshot performance in balanced data conditions to mitigate data size confounds, classifying pretraining languages that improve downstream performance as donors , and languages that are improved in zero-shot performance as recipients .
We develop a method of quadratic time complexity in the number of languages to estimate these relations, instead of an exponential exhaustive computation of all possible combinations.
We find that our method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks.
Our findings can inform developers of largescale multilingual language models in choosing better pretraining configurations.
Introduction
Pretrained language models are setting state-of-theart results by leveraging raw texts during pretraining (PLMs; Peters et al., 2018; Devlin et al., 2019, inter alia).
Interestingly, when pretraining on multilingual corpora, PLMs seem to exhibit zero-shot cross-lingual abilities, achieving non-trivial performance on downstream examples in languages seen only during pretraining.
For example, in the bottom of Figure 1, a named entity recognition model finetuned on Russian is capable of predicting correctly name entity tags for texts in English, seen only during pretraining (Pires et al., 2019; Conneau et al., 2020b; K et al., 2020; Conneau et al., 2020a;
Lazar et al., 2021; Turc et al., 2021).
Previous analyses examined how several factors contribute to this emerging behavior.
For example, parameter sharing and model depth are important in certain configurations (K et al., 2020; Conneau et al., 2020b), as well as typological similarities between languages (Pires et al., 2019), and the choice of specific finetune languages (Turc et al., 2021).
In this work, we focus on an important factor which we find missing in prior work, namely the effect that pretraining languages have on downstream zero-shot performance.
In particular, we ask three major research questions: (1) Does the choice of pretraining languages affect downstream crosslingual transfer, and if so, to what extent?
(2) Is
English the optimal pretraining language, when controlling for confounding factors such as data size and domain?
And finally, (3) Can we choose pretraining languages to improve downstream zeroshot performance?
In addressing these research questions, we aim to decouple the language from its corresponding dataset .
To the best of our knowledge, prior work has conflated pretrain corpus size and its domain with other examined factors, thus skewing results towards over-represented languages, such as English or German (Joshi et al., 2020).
2 To achieve this, we first construct a linguistically-balanced pretraining corpus based on Wikipedia, composed of a diverse set of 22 languages.
We carefully control for the amount of data and domain distribution in each of the languages (Section 3).
Next, since the number of pretraining configurations grows exponentially with the number of languages n represented in the [-dataset. 
 Therefore,-]dataset, it is infeasible to exhaustively test all possible configuration, much less extend it for more languages. In
Section 4 we propose a novel pretraining-based approach which is quadratic in the number of languages. This is achieved by training all (cid:0) n 2 (cid:1) combinations of bilingual masked language models over our corpus, thus yielding a complete directed graph (Figure 1), where an edge l 1 → l 2 estimates how much a language l 1 contributes to zero-shot performance in language l 2 , based only on language modelling performance.
In Section 5, we use the graph to identify languages which generally contribute as pretraining languages (termed “donors”), and languages which often benefit from training with other languages (termed “recipients”). Further, we use the graph to make observations regarding the effect of typological features on bilingual language modelling, and make available an interactive graph explorer.
Finally, our evaluations on two multilingual downstream tasks (part of speech tagging and named entity recognition) lead to three main conclusions (Section 6): (1) the choice of pretraining languages indeed leads to differences in zero-shot performance; (2) controlling for the amount of data allotted for each language during pretraining ques- tions the primacy of English as the main pretraining language; and (3) our hypotheses regarding donors and recipient language hold in both downstream tasks, and against two additional control groups.
Metrics for Pretraining-Aware Cross-Lingual Transfer
In this section, we extend existing metrics for zeroshot cross-lingual transfer to account for pretraining languages.
Intuitively, our metrics for a model
M and a given downstream task take into account three factors: (1) P , the set of languages seen during pretraining, (2) s ∈ P , the source language used for finetuning, and (3) t ∈ P , the target language, seen during inference.
Formally, we adapt the formulation of Hu et al.
(2020) to define a pretraining-aware bilingual zeroshot transfer score Z as:
[Equation]
Where M P,l is a model pretrained on the set of languages P and finetuned on downstream task instances in the language l ∈ P , and ε ( M, l ) is an evaluation of model M on instances in language l in terms of the downstream metric, e.g., word-label accuracy for part of speech tagging.
Following, we extend the definition of zero-shot transfer score to a set of downstream test languages
D ⊆ P to measure P ’s aggregated effect on zeroshot performance, by averaging over all bilingual transfer combinations in D :
[Equation]
In following sections, we will use these metrics to evaluate how different choices for pretraining languages influence downstream performance.
Data Selection
We collect a pretraining dataset to test the effect of pretraining languages on cross-lingual transfer.
First, we choose a set of 22 languages fromlanguage families, as listed in Table 1.
These represent a wide variety of scripts, as well as typological and morphological features.
We note that our approach can be readily extended to other languages beyond those selected in this study.
Second, we aim to balance the amount of data and control for its domain across languages, to mitigate possible confounders in our evaluations.
Below we outline design choices we make towards this goal.
3. Data Balancing
To achieve a balanced dataset across our languages, we sample consecutive sentences from every language’s Wikipedia dump from November 2021, such that each language is represented by 10 million characters.
This amount was chosen to align all languages to the lower-resource ones (e.g., Piedmontese or Irish) which comprise approximately of 10mb.
We choose to sample texts from Wikipedia as it consists of roughly similar encyclopedic domain across languages, and is widely used for training PLMs (Devlin et al., 2019).
Can we balance the amount of information across languages?
We note that a possible confound in our study is that languages may encode different amounts of information in texts of similar character count.
This may happen due to differences in the underlying texts or in inherent language properties.
6 To estimate the amount of information in each our 10 7 character partitions, we tokenize each language partition l with the same wordpiece tokenizer, and look at the ratio between total number of tokens in l and the number of unique tokens in l , finding a good correlation across all our languages ( r = 0 . 73 ), which may indicate that our dataset is indeed balanced in terms of information.
Our intuition is that an imbalanced amount of information would lead the tokenizer to “invest” more tokens in some of the languages while neglecting the less informative ones.
Is our sample representative of the full
Wikipedia corpus in each language?
Another concern may be that our sampled corpus per language is not indicative of the full corpus for that language, which may be much larger (see Table 1).
To test this, we create three discrete length distributions.
Two length distributions for sentences (in [Table]
terms of words and tokens), and word length distribution in terms of characters.
We then compare those three distributions between our sample and the full data using Earth Movers Distance.
All means and standard deviations score below 0.001, indicating that indeed all samples are similarly distributed to their respective full corpus in terms of these metrics.
Bilingual Pretraining Graph
In this section we describe a method for estimating the effect that different pretrain language combinations would have on downstream zero-shot performance. This is achieved by evaluating bilingual performance on the pretraining masked language modelling (MLM) task.
We begin by describing our experimental setup, hyperparameters and hardware configuration (Section 4.1).
In Section 4.2, we outline our estimation method, yielding a complete graph structure over our languages, amenable for future exploration and analyses (Figures 1, 2).
In following sections, we use the graph to formulate a set of downstream cross-lingual hypotheses regarding how different languages will affect zero-shot performance, and validate these hypotheses on two downstream tasks.
4. Experimental Setup
For all evaluations discussed below, we train a
BERT model (Devlin et al., 2019) with 4 layers and 4 attention heads, an MLM task head, and an embedding dimension of 512.
7 We train a single wordpiece tokenizer (Wu et al., 2016) on our entire dataset.
We train the models with a batch size of 8 samples, with sentences truncated to 128 tokens.
Each language model was trained up to 4 epochs.
This was determined by examining the training loss on 6 diverse languages in our set and observing that they converge around 4 epochs.
A subset of 6 lan- guages were trained on 4 additional seeds to verify the stability of the results, as seen in Table 5 and
Table 6 in the Appendix. Masks were applied with default settings, generating 15% mask tokens and 10% random tokens for each input sequence (Devlin et al., 2019).
We used single GPU core (nvidia tesla M60, gtx 980, and RTX 2080Ti).
Training time varied between 80 - 120 minutes.
4. Building a Pretraining Language Graph
Intuitively, we measure MLM performance when pretraining on a pair of languages ( l 1 , l 2 ) as a proxy to the extent of how l 1 and l 2 contribute to one another in zero-shot cross-lingual transfer.
This methodology relies on two assumptions.
First, we assume that the cross-lingual zero-shot performance as defined in Equation 2 is monotonic , i.e., that adding pretraining languages will improve the average downstream performance.
This is defined formally as:
[Equation]
Following this assumption will allow us to extend our bilingual observations to a pretraining language set P of arbitrary size.
Second, we assume that MLM performance correlates with downstream task performance, which is often the assumption made when training PLMs to minimize perplexity (Peters et al., 2018; Devlin et al., 2019).
Bilingual MLM finetune score.
Formally, for every language pair s, t ∈ P , we compute the following finetune score, F :
[Equation]
Where M s,t is a model pretrained on s, t , and ε is an intrinsic evaluation metric for MLM.
9 I.e.,
F ( s, t ) estimates how much the target language t “gains” in the MLM task from additional pretraining on the source language s compared to monolingual pretraining on t .
Figure 2 depicts a weighted adjacency matrix where coordinate ( i, j ) corresponds to F ( l i → l j ) .
As shown in Figure 1, the same information can be conveyed in a complete directed weighted graph, where each node represents a language, and edges ( l 1 , l 2 ) are weighted by F ( l 1 → l 2 ) .
Language-Level donation and recipience.
Next, for each language l ∈ P we compute a Donation score, D , as an aggregate over all of its finetune scores as a source language (i.e., how much it contributed to other languages), and similarly a recipience score, R , by aggregating over all its finetune scores as a target language, to measure how much l is contributed to by other languages.
Formally:
[Equation]
We depict both donation and recipience scores as aggregate row and column vectors in Figure 2.
Thus, based on the two assumptions above, our hypothesis is that the downstream cross-lingual transfer will be proportional to the sum of recipience scores for all pretraining languages.
Formally:
[Equation]
Moreover, higher donation scores for languages in the pretrain set will result in higher scores in the downstream task.
Formally:
[Equation]
Pretraining Graph Analysis
We present several key observations based on the bilingual pretraining graph described in the previous section and summarized by the adjacency matrix in Figure 2, as well as an interactive exploration interface.
In following sections we use these observations in our downstream evaluations.
Some language combinations are detrimental.
Negative finetune scores are present in some of the target languages, e.g., between Korean (ko) and Arabic (ar), which means that initializing a language model for Arabic with weights learned for Korean hurts MLM performance on Arabic, compared to an Arabic monolingual baseline.
I.e., in these language configurations, initializing the model with another language model’s weights leads to worse performance than random initialization.
Bilingual MLM relations are not symmetric.
In fact, we observe a moderate negative correlation between F ( l 1 → l 2 ) and F ( l 2 → l 1 ) , as shown in Figure 3.
For example, for German and Finnish we get 0 .
51 =
F ( fi → de ) > F ( de → fi ) = − 0 .
I.e., Finnish initialization improves German MLM, while the inverse is detrimental for Finnish.
Monolingual performance correlates with donation score.
Perhaps expectedly, relatively worseperforming models benefit most from bilingual transfer, while better-performing monolingual models tend to be better donors, although to a lesser extent (Figure 4). Different script leads to larger variance in bilingual finetuning. However, language family does not affect it. We find that fine-tuning between languages with different scripts is a high-risk, highreward scenario.
The highest scores occur in this setting, while the probability of negative score is higher.
A shared script is a safe setting with a
[Figure]
o n o r ja ko ta te hu
ﬁ zh my ar he cy ga hi
ne en de
[Figure]
high proportion of neutral or positive donation (Figure 5a). In contrast with recent findings (Pires et al., 2019), we did not observe a statistically significant influence for the language family (Figure 5b). Finetuning as transfusion: mapping the linguistic blood-bank. The non-symmetric nature of the scores gives rise to a coarse-grained ontology loosely reminiscent of human blood types, depicted in Figure 3.
Languages which on average donate but do not receive ( D ( l ) > 0 and R ( l ) < 0 ) are denoted O type languages , while the inverse (receiving but not donating) are denoted as AB+ type . 5. Interactive Exploration To allow further exploration of our our bilingual pretraining graph, we develop a publicly available web-based interactive exploration interface.
11 We enable exploration of interactions between differ- ent linguistic features, based on The World Atlas of
Language Structures (WALS) (Dryer and Haspelmath, 2013), allowing users to filter and focus on specific traits and analyze how they affect bilingual pretraining.  Downstream Zero-Shot Performance In this section we validate our method for estimating the effect of pretraining language combinations on downstream performance. Towards that end, in Section 6.1, we construct several pretraining configurations, based on pretraining observations.
Then, in Section 6.2 we describe the multilingual datasets we use for two downstream tasks.
Finally, our results are presented in Section 6.3, showing the influence of pretraining configuration on downstream performance.
6. Choosing Pretraining Sets
We use the donation scores to identify pretraining languages projected to lead to better downstream zero-shot performance, and the recipience score to find downstream languages which will perform better languages as source (finetune) languages.
Our setup is summarized in Table 2.
Pretraining languages.
We define three sets of languages for pretraining, using the donation score while keeping the sets linguistically diverse: (1) Top donating: Japanese, Telugu, Finnish, and Russian; (2) Least donating: Nepali, Burmese, Armenian, and English.
We also include English as it is a popular source language; and (3) Random:
A randomly selected set of 4 languages: Hebrew, Irish,
French and Swedish.
Downstream languages.
To validate that lower recipience scores indeed indicate that languages are less likely to improve via cross lingual transfer, we added 6 languages to all configurations described above: 3 Top-recipient languages ( R h ): Hindi, German, and Hungarian, and 3 Least-recipient languages ( R l ): Arabic, Greek, and Tamil. Finally, we add a fourth control configuration which was pretrained only on C : = R h ∪ R l .
Hypotheses.
We hypothesize that more donating pretraining sets will increase cross-lingual transfer in downstream tasks, and that least-recipient languages will have lesser cross-lingual performance compared to top-recipient languages.
These can be formally articulated using Equations 7 and 8:
[Figure] [Figure]
[Figure]
[Equation]
6. Tasks
We evaluated all pretraining configurations detailed in Table 2 on two of XTREME’s tasks: part of speech tagging (POS) and named entity recognition (NER). Both of which commonly appear in
NLP pipelines such as CoreNLP (Manning et al., 2014) and spaCy (Honnibal and Montani, 2017).
We aim to balance the data in both tasks across different finetune languages, so as not to skew results towards higher-resource languages.
For part-of-speech tagging, XTREME borrows from universal dependencies (Nivre et al., 2020).
Since XTREME is imbalanced across languages, we truncated the data to 1000 sentences to fit the lower-resource languages, e.g., XTREME annotates POS in 909 sentences in Hungarian. For
NER, we applied a similar procedure, where XTREME’s data was taken from the Wikiann (panx) dataset (Rahimi et al., 2019) which we truncated to 5000 sentences (the data size available for Hindi NER in XTREME). Experimental setup.
We use the code and default hyperparameter default values provided by
XTREME to train the downstream tasks (Hu et al., 2020), adapted for multilingual training.
6. Results
Several key observations can be made based on the results for both POS tagging and NER across all training configurations, which are presented in Tables 3 and 4.
For each configuration P in
Top, Bottom, Random, Control we calculated zeroshot transfer scores on C , using Z P ( C ) defined by Equation 2.
Monolingual results under each pretrain set P were calculated by the average F
[Table] [Table]
[Table]
performance of each language in C :
[Equation] Where ε ( M P,l , l ) denotes the F 1 score of a model pretrained on P , finetuned on l and evaluated on l .
Pretraining configuration affects downstream cross-lingual transfer.
In both tasks, we observe a variance in results when changing the pretraining configuration, despite all of them having similar amounts of data.
This may imply that previous work has omitted an important interfering factor.
Recipience score correlates with downstream cross-lingual performance.
We evaluated zeroshot transfer for each language set R ∈
{ R l , R h } as the average zero-shot transfer scores over all pretraining configurations.
Table 4 reveals that the more recipient set outperforms the least recipient set in both tasks ( +5 . 5% in NER, +2 .
7% in POS tagging).
Multilingual pretraining can improve monolingual performance.
As seen in Table 3, the Top Donating pretraining configuration achieved a monolingual score which is slightly higher than the control group, while the Least Donating configuration underperforms all other sets.
This suggests that multilingual pretraining datasets can benefit monolingual downstream results compared to more data in a single language.
English might not be an optimal pretraining language.
Corresponding with our previous results, if donation score is indicative of a language’s contribution in pretraining, English’s relative low donation score might indicate that it is not the best language to pretrain upon.
English was also part of the bottom pretraining configuration which scored lower than top as seen in Table 3.
Further research can ascertain this finding.
Limitations and Future Work
As with other works on cross-lingual transfer, our results are influenced by many hyperparameters.
Below we explicitly define our design choices and how they can be explored in future work.
First, data scarcity in low-resource languages restricted us to small data amounts.
Although our experiments showed non-trivial signal for pretraining and downstream tasks, future work may apply our framework to larger data sizes.
Second, for efficiency sake, we trained relatively small models to enable us to train a large amount of language configurations, while ensuring convergence in 6 languages.
Furthermore, we did not do any hyper-parameter tuning and use only values reported in previous work, and use only the BERT architecture.
Future work may revisit any of these design choices to shed more light on their effect.
Third, similarly to other works, our data was scraped from Wikipedia, and we did not account for language contamination across supposedly monolingual corpora (e.g., due to code switching).
Such contamination may confound with cross-lingual transfer, as was recently shown by Blevins and
Zettlemoyer (2022).
Finally, our downstream analysis focused on
POS tagging and NER since they were available for many languages and are common in many NLP pipelines.
Further experimentation can test if our results hold for more NLP tasks.
Related Work
To the best of our knowledge, this is the first work to control for the amount of data allocated for each language during pretraining and finetuning while evaluating on many languages.
Perhaps most related to our work, Turc et al.
(2021) challenge the primacy of English as a source language for cross-lingual transfer in various downstream tasks.
Their work shows that German and
Russian are often more effective sources.
In all of their experiments they use mBERT’s imbalanced pretraining corpus.
Blevins and Zettlemoyer (2022) complement this hypothesis by showing that English pretraining data actually contains significant amount of non-English text, which correlates with the model’s transfer capabilities.
Wu and Dredze (2020) evaluate how mBERT performs on a wide set of languages, focusing on the quality of representation for low-resource languages in various downstream tasks by defining a scale from low to high resource.
They show that mBERT underperforms non BERT monolingual baselines for low resource languages while performing well for high resource ones.
While Pires et al.
; Limisiewicz and
Mareˇcek (2021) show that typology plays a significant role for mBERT’s multilingual performance, this is not replicated in our balanced evaluation, and has lesser impact in Wu et al.
(2022) as well.
Finally, Conneau et al.
(2020a) introduce the transfer-interference trade-off where low resource languages benefit from multilingual training, up to a point where the overall performance on monolingual and cross-lingual benchmarks degrades.
Conclusions
We explored the effect of pretraining language selection on downstream zero-shot transfer.
We first choose a diverse pretraining set oflanguages, and curate a pretraining corpus which is balanced across these languages.
Second, we devise an estimation technique, quadratic in the number of languages, projecting which pretraining languages will serve better in cross-lingual transfer and which specific downstream languages will do best in that setting.
Finally, we test our hypothesis on two downstream multilignual tasks, and show that the choice of pretraining languages indeed leads to varying downstream cross-lingual results, and that our method is a good estimation for downstream performance.
Taken together, our results suggest that pretraining language selection should be a factor in estimating cross-lingual transfer, and that current practices which focus on high-resource languages may be sub-optimal.
Acknowledgements
We would like to thank Roy Schwartz for his helpful comments and suggestions and the anonymous reviewers for their valuable feedback. This work was supported in part by a research gift from the Allen Institute for AI.
Tomasz Limisiewicz’s visit to the Hebrew University has been supported by grant 338521 of the Charles University Grant Agency and the Mobility Fund of Charles University.
A Appendix Full list of tokenized languages
The full list of Wikipedia language codes for languages used in our tokenizer training is: • pms, ga, ne, cy, fi, hy, my, hi, te, ta, ko, el, hu, he, zh, ar, sv, ja, fr, de, ru, en - languages that are also evaluated and trained.
Elaborated in Table 1. • af, am, ca, cs, da, es, id, is, it, mg, nl, pl, sk, sw, th, tr, ur, vi, yi - Additional languages corresponding to Afrikaans, Amharic, Catalan, Czech, Danish, Spanish, Indonesian, Icelandic, Italian, Malagasy, Dutch, Polish, Slovak, Swahili, Thai, Turkish, Urdu, Vietnamese, and Yiddish. B Transfer Distribution In the histogram of cross-lingual transfers (Figure 7), we observe that the distribution has multiple local maximums (modes).
We distinguish four main level of cross-lingual transfer described in Section 4.2 ( F ( l i → l j ) ):
• negative transfer F ( l i → l j ) < − 0 .• neutral transfer − 0 . 1 ≤ F ( l i → l j ) < 0 .• positive transfer 0 .
1 ≤ F ( l i → l j ) < 0 .•
very positive transfer 0 .
5 ≤ F ( l i → l j ) [Figure] [Table] [Table]
[Table]