When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer
Abstract
While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer, there is a lack of consensus in the community as to what shared properties between languages enable transfer on downstream tasks.
Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects.
In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zeroshot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax.
Among other things, our experiments show that the absence of sub-word overlap signiﬁcantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., ρ s = 0 .
94 on the task of NLI).
Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.
Introduction
Multilingual language models like XLM (Conneau et al., 2020a) and Multilingual-BERT (Devlin, 2019) are trained with masked-language modeling (MLM) objective on a combination of raw text from multiple languages.
Surprisingly, these models exhibit decent cross-lingual zero-shot transfer, where ﬁne-tuning on a task in a source language translates to good performance for a different language (target).
Requirements for zero-shot transfer
Recent studies have provided inconsistent explanations for properties required for zero-shot transfer (hereon, transfer).
For example, while Wu and Dredze (2019) conclude that sub-word overlap is vital for transfer, K et al.
(2020) demonstrate that it is not crucial, although they consider only English as the source language.
While Pires et al.
(2019) suggest that typological similarity (e.g., similar SVO order) is essential for transfer, other works (Kakwani et al., 2020; Conneau et al., 2020a) successfully build multilingual models for dissimilar languages.
Need for systematic analysis
A major cause of these discrepancies is a large number of varying properties (e.g., syntax, script, and vocabulary size) between languages, which make isolating crucial ingredients for transfer difﬁcult.
Some studies alleviate this issue by creating synthetic languages which differ from natural ones only in speciﬁc linguistic properties like script (K et al., 2020; Dufter and Schütze, 2020).
However, their focus is only on English as a source language, and the scale of their experiments is small (in number of tasks or pre-training corpora size), thus limiting the scope of their ﬁndings to their settings alone.
Our approach
We perform a systematic study of cross-lingual transfer on bilingual language models trained on a natural language and a systematically derived counterpart.
We choose four diverse natural languages (English, French, Arabic, and
Hindi) and create derived variants using four different transformations on structural properties such as inverting or permuting word order, altering scripts, or varying syntax (Section 3.2).
We train models on each of the resulting sixteen language pairs, and evaluate zero-shot transfer on four downstream tasks – natural language inference ( NLI ), namedentity recognition ( NER ), part-of-speech tagging ( POS ), and question-answering ( QA ).
Our experiments reveal the following:
Contrary to previous belief, the absence of subword overlap degrades transfer when languages
differ in their word order (e.g., by more thanF1 points on POS tagging, (§ 4.1)).
There is a strong correlation between token embedding alignment and zero-shot transfer across different tasks (e.g., ρ s = 0 .
94 , p < .
005 for XNLI, Fig 4).
Using pre-training corpora from similar sources for different languages (e.g., Wikipedia) boosts transfer when compared to corpora from different sources (e.g., 17 F1 points on NER, Fig 3).
To our knowledge, we are the ﬁrst study to quantitatively show that zero-shot transfer between languages is strongly correlated with token embedding alignment ( ρ s = 0 .
94 for NLI ).
We also show that the current multilingual pre-training methods (Doddapaneni et al., 2021) fall short of aligning embeddings even between simple natural and derived language pairs, leading to failure in zero-shot transfer.
Our results call for training objectives that explicitly improve alignment using either supervised (e.g., parallel corpora or bilingual dictionaries) or unsupervised data.
Related work Multilingual pre-training for Transformers
The success of monolingual Transformer language models (Devlin et al., 2019; Radford et al., 2018) has driven studies that learn a multilingual language-model (LM) on several languages.
Multilingual-BERT (M-BERT) (Devlin, 2019) is a single neural network pre-trained using the masked language-modeling (MLM) objective on a corpus of text from 104 languages.
XLM (Conneau and Lample, 2019) introduced translation languagemodeling, which performs MLM on pairs of parallel sentences, thus encouraging alignment between their representations.
These models exhibit surprising zero-shot cross-lingual transfer performance (Conneau and Lample, 2019; K et al., 2020), a setup where the model is ﬁne-tuned on a source language and evaluated on a different target language.
Analysis of cross-lingual transfer
While Pires et al.
(2019), Conneau et al.
(2020b), and K et al. (2020) showed that transfer works even without a shared vocabulary between languages, Wu and
Dredze (2019) discovered a correlation between sub-word overlap and zero-shot performance.
Conneau et al.
(2020b) and Artetxe et al.
(2020a) showed that shared parameters for languages with different scripts were crucial for transfer.
Pires et al.
(2019) and (Wu and Dredze, 2019) observed that transfer for NER and POS tagging works better between typologically similar languages.
However, a study conducted by Lin et al.
(2019) showed that there is no simple rule of thumb to gauge when transfer works between languages.
Hsu et al.
(2019) observed that changing the syntax (SOV) order of the source to match that of the target does not improve performance.
Transfer between real and synthetic Languages
K et al.
(2020) create a synthetic language by changing English’s script and ﬁnd that transfer between it and Spanish works even without common sub-words.
However, they use only English as their source language, test only on two tasks, and use a single natural-synthetic language pair.
Dufter and Schütze (2020) study transfer between English and synthetic English obtained by changing the script, word order, or model delimiters.
However, they use a small corpus ( 228 K words) compared to current standards (we use 3 orders more) and measure only embedding similarity and not zero-shot transfer.
A contemporary work (Wu et al., 2022) uses synthetic transformations to modify the GLUE dataset (Wang et al., 2018) and analyze properties required for good zero-shot transfer, but they perform their experiments only on English and do not perform token embedding alignment analysis.
We show that the latter is crucial for good transfer.
Approach
We ﬁrst provide some background on bilingual language models (Section 3.1), followed by descriptions of our transformations (Section 3.2), and our training and evaluation setup (Section 3.3).
3. Background Bilingual pre-training
The standard setup (Conneau and Lample, 2019) trains a bilingual language model ( Bi-LM ) on raw text corpora from two languages simultaneously.
Bi-LM uses the masked language-modeling loss ( L MLM ) on the corpora from the two languages ( C 1 , C 2 ) separately with no explicit cross-lingual signal:
[Equation]
A shared byte pair encoding tokenizer (Sennrich et al., 2015) is trained on C 1 + C 2 .
A single batch contains instances from both languages, but each instance belongs to a single language.
Transformation
Instance ( s )
Transformed instance ( T ( s ))
Inversion ( T inv )
Welcome to NAACL at Seattle
Seattle at NAACL to Welcome
Permutation ( T perm )
This is a conference a This conference is
Transliteration ( T trans )
I am Sam .
I am ♣ (I) ♥ (am) ♦
(Sam)
♠ (.)
♣ (I) ♥ (am)
Syntax ( T syn )
Sara (S) ate (V) apples (O) Sara (S) apples (O) ate (V)
[Table] table [Table]
Consider a bilingual model ( Bi-LM ) pre-trained on two languages, source and target . Zero-shot transfer involves ﬁnetuning Bi-LM on downstream task data from source and evaluating on test data from target . This is considered zero-shot because Bi-LM is not ﬁne-tuned on any data belonging to target .
3. Generating language variants with systematic transformations
Natural languages typically differ in several ways, like the script, word order, and syntax.
To isolate the affect of these properties on zero-shot transfer, we obtain derived language corpora (hereon, derived corpora) from original (natural) language corpora by performing sentence level transformations ( T ) which change particular properties.
For example, an “ inversion ” transformation could be used to invert each sentence in the corpus ( Welcometo 2 NAACL 3 ⇒
NAACL 3 to 2 Welcome 1 ).
Since the transformation ( T ) is applied on each sentence of the original corpus, the size of the original and the derived corpus is the same.
In the following sections, we will use the following notation:
[Equation]
Types of transformations
We consider four transformations which modify different aspects of sentences (examples in Table 1):
Inversion ( T inv ):
Invert the order of tokens in the sentence, like in Dufter and Schütze (2020).
The ﬁrst token becomes the last, and vice versa.
Permutation ( T perm ): Permute the order of tokens in a sentence uniformly at random.
For a sentence of n tokens, we sample a random ordering with probability 1 n ! .
Transliteration ( T trans ): Change the script of all tokens other than the special tokens (like [CLS] ).
This creates a derived vocabulary ( V deriv ) with a one-to-one correspondence with the original vocabulary ( V orig ).
Syntax ( T syn ): Modify a sentence to match the syntactic properties of a different natural language by re-ordering the dependents of nouns and verbs in the dependency parse.
These transformations are stochastic because of the errors in parsing and sampling over possible re-orderings (Wang and Eisner, 2016).
These transformations allow us to systematically evaluate the effect of corresponding properties on zero-shot transfer.
We also consider composed transformations ( § 4.2) which consecutively apply two transformations. We note that while real languages typically differ in more than one or two properties considered in our transformations, our methodology remains useful in isolating crucial properties that enable good transfer and can be extended to more transformations.
Transformations for downstream tasks
We obtain the downstream corpus in the derived language ( D deriv ) by applying the same transformation ( T ) used during pre-training on the original downstream corpus ( D orig ).
Unlike pre-training corpora which contain raw sentences, instances in downstream tasks contain one or more sentences with annotated labels.
For text classiﬁcation tasks like
[Figure] [Figure] NLI, we apply the transformation on each sentence in every dataset instance.
For token classiﬁcation tasks (e.g., NER, POS), any transformation which changes the order of the tokens also changes the order of the labels.
We present the mathematical speciﬁcation in Appendix A.
3. Model Training and Evaluation
We now describe our pre-training and zero-shot transfer evaluation setup.
Figure 1 provides an overview of pre-training and ﬁne-tuning, and Table 2 summarizes the evaluation metrics we use.
Pre-training
Let C orig and C deriv be the original and derived language pre-training corpora.
We train two models for each original - derived pair:
Bilingual Model ( Bi-LM ) :
A bilingual model pre-trained on the combined corpus ( C orig + C deriv ) (Figure 1a).
Monolingual Model ( Mono-LM ) :
A monolingual model trained only on C orig for the same number of steps as Bi-LM ’s. MonoLM is used as a baseline to measure zero-shot transfer of a model not pre-trained on derived .
Evaluation
Let D train orig and D dev orig be the original language training and development sets for a downstream task, and D train deriv and D dev deriv be the corresponding derived language datasets.
For evaluation, we ﬁrst ﬁne-tune the pre-trained models on a downstream training set and evaluate the resulting model on a development set (Figure 1b).
Since our goal is to investigate the extent of zero-shot transfer, we require appropriate lower and upper bounds to make informed conclusions.
To this end, we compute three metrics, all on the same development set (summarized in Table 2):
• Bilingual zero-shot transfer ( BZ ) :
This is the standard zero-shot transfer score (Conneau and Lample, 2019) which measures how well a bilingual model ﬁne-tuned on D train orig zeroshot transfers to the other language ( D dev deriv ).
• Bilingual supervised synthetic ( BS ) :
This is the supervised learning performance on the derived language obtained by ﬁne-tuning BiLM on D train deriv and evaluating it on D dev deriv .
• Monolingual zero-shot transfer ( MZ ) :
This measures the zero-shot performance of the baseline Mono-LM , which is not pre-trained on the derived language, by ﬁne-tuning MonoLM on D train orig and evaluating it on D dev deriv
BS uses ﬁne-tuning train data from the derived language and serves as an upper-bound on BZ and MZ which don’t use it.
MZ doesn’t pre-train on the derived language and serves as a lower-bound on BZ which does pre-train on it.
For easier comparison of BZ and MZ with BS (upper-bound), we report the following score differences (Table 2), which are both negative in our experiments.
[Equation]
BZ alone cannot capture the quality of the zeroshot transfer.
A large and negative ∆ ( BZ − BS ) implies that bilingual zero-shot transfer is much worse than supervised ﬁne-tuning on derived .
Concurrently, ∆ ( BZ − BS ) ≈ ∆ ( MZ − BS ) implies that Bi-LM transfers as poorly as Mono-LM .
Thus, good zeroshot transfer is characterized by ∆ ( BZ − BS ) ≈and ∆ ( BZ − BS ) (cid:29) ∆ ( MZ − BS ) .
3. Experimental Setup Languages
We choose four diverse natural languages:
English (Indo-European, Germanic),
French (Indo-European, Romance), Hindi (IndoEuropean, Indo-Iranian), and Arabic (Afro-Asiatic,
Semitic), which are represented in the multilingual
XTREME benchmark (Hu et al., 2020).
For each language, we consider four transformations (Section 3.2) to create derived counterparts, giving us 16 different original-derived pairs in total.
For the
Syntax transformation, we use Qi et al.
(2020) for parsing.
We modify the syntax of FR , HI , and AR to that of EN , and the syntax of EN to that of FR .
Datasets
For the pre-training corpus ( C orig ), we use a 500 MB (uncompressed) subset of Wikipedia ( ≈ 100M tokens) for each language.
This matches the size of WikiText-103 (Merity et al., 2016), a standard language-modeling dataset.
For downstream evaluation, we choose four tasks from the
XTREME benchmark (Hu et al., 2020).
Tablelists all the datasets and their evaluation metrics.
Implementation Details
We use a variant of
RoBERTa (Liu et al., 2019) which has 8 layers, 8 heads, and a hidden dimensionality of 512 .
We train each model on 500 K steps, a batch size of 128 , and a learning rate of 1 e - 4 with a linear warmup of 10 K steps.
We use an original language vocabulary size of 40000 for all the models and train on 8 Cloud TPU v3 cores for 32 - 48 hours.
For ﬁne-tuning, we use standard hyperparameters (Appendix F) from the XTREME benchmark and report our scores on the development sets.
Results
Our experiments reveal several interesting ﬁndings for bilingual models including the situational importance of sub-word overlap for zero-shot transfer (§ 4.1, 4.2), the effect of domain mismatch between languages (§ 4.3), and correlation of zero-shot performance with embedding alignment (§ 4.4).
We connect our ﬁndings to zero-shot transfer results between natural languages in Section 4.5.
4. Sub-word overlap is not strictly necessary for strong zero-shot transfer
Sub-word overlap is the number of common tokens between two different language corpora.
If E 1 and
E 2 are sets of tokens which appear in the two corpora, then: Sub-word overlap =
|E 1 ∩ E 2 | / |E 1 ∪
E 2 | (Pires et al., 2019).
The Transliteration transformation ( T trans ) creates original - derived language pairs that have 0% sub-word overlap (equivalently, different scripts), but follow the same word order.
Table 3 displays ∆ ( BZ − BS ) scores for T trans , averaged over four languages (Appendix B contains a breakdown).
We observe that ∆ ( BZ − BS ) ≈ {+
 for all+} tasks while ∆ ( MZ − BS ) is highly negative, implying that zero-shot transfer is strong and on par with supervised learning.
This result indicates that zero-shot transfer is possible even when languages with different scripts have similar word orders (in line with K et al.
However, it is unrealistic for natural languages to differ only in their script and not other properties (e.g., word order).
4. Absence of sub-word overlap signiﬁcantly hurts zero-shot performance when languages differ in their word-orders
To simulate a more realistic scenario, we create original and derived language pairs which differ both in their scripts ( 0% sub-word overlap) and in word order.
We achieve this by composing two transformations on the original language corpus, one of which is Transliteration ( T trans ).
We experiment with three different compositions, (a) T trans ◦ T inv , (b) T trans ◦ T perm , and (c) T trans ◦ T syn .
Here, α ◦ β means that transformation β is applied before α .
A composed transformation ( T trans ◦ [Table] [Table] β ) differs from its second constituent ( β ) in that the former produces a derived language which has 0% sub-word overlap with the original language whereas the latter has a 100% sub-word overlap.
Results
Our results (Figure 2, breakdown in Appendix C) show that zero-shot performance is signiﬁcantly hurt for composed transformations when compared to its constituents.
| ∆ ( BZ − BS ) | is much larger for T trans ◦ T inv when compared to T trans or T inv individually.
For example, for XNLI, | ∆ ( BZ − BS ) | = 19 for the composed transformation and just 2 and 3 for T trans and T inv individually.
T trans ◦ T perm is worse by ≈ 20 points on XNLI and NER, and over 40 points on POS when compared to T perm .
T trans ◦ T syn suffers lesser than the other two composed transformations, but it is still worse than T syn by 3 , 6 , and 1 point on XNLI,
NER, and POS.
In conclusion, the absence of subword overlap signiﬁcantly degrades zero-shot performance in the realistic case of languages with different word orders.
[-|   
-]
4. Data from the same domain boosts bilingual performance
Previously, we considered transformations ( T ) that modiﬁed the original pre-training corpus to get a parallel corpus, C deriv =
T ( C orig ) , such that there is a one-to-one correspondence between sentences in C orig and C deriv (we call this setting parallel ).
Since procuring large parallel corpora is expensive in practice, we consider two other settings which use different corpora for original and derived .
Setup
Consider two text corpora of the same size,
C 1orig and C 2orig .
We compare two settings: (1)
The parallel setting pre-trains a bilingual model on C 1 orig + T ( C 1 orig ) , whereas the (2) non-parallel corpus setting uses C 1orig + T ( C 2orig ) .
We consider two variants of non-parallel , (1) non-parallel (same) which uses different splits of Wikipedia data (hence, same domain), and (2) non-parallel (diff) which uses Wikipedia data for the original and common crawl data (web text) for the derived language (hence, diff erent domain).
We use the
Transliteration transformation ( T trans ) to generate the derived language corpus and report | ∆ ( BZ − BS ) | averaged over all languages in Figure 3.
Results
We observe consistently on all tasks that the parallel setting (blue bar) performs better than both the non-parallel settings. Non-parallel (same)
XNLI
NER POS        | ( B Z B S ) |  0     trans inv trans inv
XNLI NER
POS          0    59 trans perm trans perm XNLI NER POS          0 1  trans syn
[Figure]
syn performs better than non-parallel (diff) , with gains ranging between 2 points on XQuAD to 17 points on NER.
This result shows that even for original and derived language pairs which differ only in their script, having parallel pre-training corpora leads to the best zero-shot transfer.
Since largescale parallel unsupervised data is hard to procure, the best alternative is to use corpora from similar domains (Wikipedia) rather than different ones (Wikipedia v.s. web text).
4. Zero-shot performance is strongly correlated with embedding alignment
Our previous results ( § 4.2, 4.3) showed cases where zero-shot transfer between languages is poor when there is no sub-word overlap.
To investigate this further, we analyze the static word embeddings learned by bilingual models and ﬁnd that zero-shot transfer between languages is strongly correlated with the alignment between word embeddings for the original and derived languages.
Setup
The original and the derived languages have a one-to-one correspondence between their sub-word vocabularies when we use transliteration ( T trans ).
For a token embedding in the originallanguage embedding matrix, its alignment score [-is
 %-]is 100% if it retrieves the corresponding token embedding in the derived language when a nearestneighbor search is performed, and 0% otherwise.
We average the alignment score over all the tokens and call it alignment .
Results
We measure the alignment of bilingual models pre-trained on different originalderived language pairs created using transliteration , namely the composed transformations (§ 4.2), parallel , and non-parallel (§ 4.3).
We plot the alignment along with the corresponding ∆ ( BZ − BS ) scores for XNLI in Figure 4.
Results for other tasks are in Appendix E.
We observe that higher alignment is associated with lower ∆ ( BZ − BS ) , implying better zeroshot transfer.
Alignment is lower for composed transformations like T trans ◦ T inv and T trans ◦
T perm which have large and negative ∆ ( BZ − BS ) .
Alignment also explains the results in Section 4.3, with non-parallel variants having lower alignment scores than parallel , which is in line with their lower ∆ ( BZ − BS ) .
Overall, we ﬁnd a strong and signiﬁcant Spearman’s rank correlation between alignment and ∆ ( BZ − BS ) , with ρ = 0 .
94 , p < .for
XNLI, ρ = 0 .
93 , p < .
005 for NER, and ρ = 0 .
89 , p < .
01 for POS, indicating that increasing the embedding alignment between languages helps improve zero-shot transfer.
[Figure]
{+| ( B Z B S ) |   
+}
[Figure]
4. Connections to results on natural language pairs Effect of sub-word overlap
In § 4.2, we showed that when languages have different scripts ( 0% subword overlap), zero-shot transfer signiﬁcantly degrades when they additionally have different word orders.
However, the zero-shot transfer is good when languages differ only in the script and have similar or the same word order.
This is in line with anecdotal evidence in Pires et al.
(2019), where zero-shot transfer works well between English and Bulgarian (different script but same subjectverb-object order – SVO), but is poor between
English and Japanese (different script and word order – SVO v.s. SOV).
Our result also corroborates ﬁndings in Conneau et al.
(2020b) that artiﬁcially increasing sub-word overlap between natural languages (which have different word orders) improves performance (e.g., 3 points on XNLI).
Effect of token embedding alignment
In § 4.4, we showed that zero-shot transfer is strongly correlated with word embedding alignment between languages.
This explains the usefulness of recent stud- [Figure] ies which try to improve multilingual pre-training with the help of auxiliary objectives, which improve word or sentence embedding alignment.
DICT-MLM (Chaudhary et al., 2020) and RelateLM
(Khemchandani et al., 2021) require the model to predict cross-lingual synonyms as an auxiliary objective, thus indirectly improving wordembedding alignment and the zero-shot performance on multiple tasks.
Hu et al.
(2021) add an auxiliary objective that implicitly improves word embedding alignment and show that they can achieve performance similar to larger models.
Cao et al.
(2019) explicitly improve contextual word embedding alignment with the help of wordlevel alignment information in machine-translated cross-lingual sentence pairs.
Since they apply this post hoc and not during pre-training, the improvement, albeit signiﬁcant, is small ( 2 points on XNLI).
While these studies do not fully utilize word and sentence embedding alignment information, our results lead us to posit that they are a step in the right direction and that baking alignment information more explicitly into pre-training will be beneﬁcial.
Conclusion
Through a systematic study of zero-shot transfer between four diverse natural languages and their counterparts created by modifying speciﬁc properties like the script, word order, and syntax, we showed that (1) absence of sub-word overlap hurts zero-shot performance when languages differ in their word order, and (2) zero-shot performance is strongly correlated with word embedding alignment between languages.
Some recent studies have implicitly or unknowingly attempted to improve alignment and have shown slight improvements in zero-shot transfer performance.
However, our results lead us to posit that explicitly improving word embedding alignment during pre-training by using either supervised (e.g., parallel sentences and translation dictionaries) or unsupervised data will signiﬁcantly improve zero-shot transfer.
Although real languages typically differ in more ways than the set of properties considered in our transformations, our methodology is still useful to help isolate crucial properties for transfer.
Future work can experiment with more sophisticated transformations and investigate closer connections with human language pairs.
Acknowledgments
This work was funded through a grant from the
Chadha Center for Global India at Princeton University.
We thank Shunyu Yao, Vishvak Murahari,
Tianyu Gao, Sadhika Malladi, and Jens Tuyls for reading our draft and providing valuable comments.
We also thank the Google Cloud Research program for computational support in running our experiments.
A Mathematical Speciﬁcation for Transformation of Downstream Datasets
Text classiﬁcation
Text classiﬁcation tasks like news classiﬁcation or sentiment analysis typically have instances which contain a single sentence and a label.
Instances in other classiﬁcation tasks like natural language inference (NLI) (Bowman et al., 2015) contain two sentences and one label.
For such tasks, we apply the transformation ( T ) on each sentence within every instance, and leave the annotated label as is.
Therefore, for a dataset of size n which contains m sentences per instance, we have:
[Equation]
Token-classiﬁcation tasks
Tasks like namedentity recognition (NER) and part-of-speech tagging (POS tagging) have labels associated with each token in the sentence.
For these datasets, we ensure that any transformation ( T ) that changes the order of the tokens also changes the order of the corresponding labels.
We deﬁne a few quantities to express the transformation mathematically.
Let s i =
( w i 1 , . . . , w ik ) be a sentence comprised of k tokens and y i
= ( y i 1 , . . .
, y ik ) be labels corresponding to the tokens in the sentence.
We deﬁne a new transformation ( T aug ) which operates on the label augmented sentence, s augi = (( w i 1 , y i 1 ) , . . .
, ( w ik , y ik )) .
Let s augi [ j ] correspond to the j th element in the sequence, and s augi [ j ][ word ] and s augi [ j ][ label ] correspond to the word and label of the j th element.
Let T aug
( s aug i )
[ j ][ orig ] denote the index of the j th element in the transformed sequence with respect to the original sequence s augi .
Then, the new transformation T aug is such that,
T aug ( s augi )[ j ][ orig ] =
T ( s i )
[ j ][ orig ]
Let orig_j =
T aug ( s augi )[ j ][ orig ]
T aug ( s augi )[ j ][ label ] =
s augi [ orig_j ][ label ]
We transform the dataset using T aug :
[Equation]
B Zero-shot transfer results for different transformations
Table 5 in the appendix is the extended version of Table 3 in the main paper with a breakdown for all languages.
It reports ∆ ( BZ − BS ) , ∆ ( MZ − BS ) , and BZ for different languages and transformations considered.
C Composed Transformations
Table 6 in the appendix presents the breakdown of results in Figure 2 of the main paper.
It reports ∆ ( BZ − BS ) scores for composed transformations and their constituents.
D Comparing different sources for original and derived language corpora
Table 8 in the appendix contains the breakdown of results in Figure 3 of the main paper.
It reports ∆ ( BZ − BS ) for different languages on different tasks for the settings mentioned in Section 4.3.
E Alignment Correlation
We present alignment results (Section 4.4) for all
XNLI, NER, and POS in Figure 5.
We observe strong correlations between alignment and zeroshot transfer, with ρ s = 0 .
94 , p < .
005 on XNLI, ρ s = 0 .
93 , p < .
005 on NER, and ρ s = 0 .
89 , p < .
01 on POS.
We present the raw scores in Table 7.
F Hyperparameters for XTREME
• XNLI:
Learning rate – 2e-5 , maximum sequence length – 128, epochs – 5, batch size – 32. • NER: Learning rate – 2e-5 , maximum sequence length – 128, epochs – 10, batch size –
32. • POS: Learning rate – 2e-5 , maximum sequence length – 128, epochs – 10, batch size – 32. •
Tatoeba: Maximum sequence length – 128, pooling strategy – representations from the middle layer (cid:0) n 2 (cid:1) of the model.
• XQuAD:
Learning rate – 3e-5 , maximum sequence length – 384, epochs – 2, document stride – 128, warmup steps – 500, batch size – 16, weight decay – 0.0001.
[Table]
[Table]
BZ ∆ ( BZ
[Table]
BS ) BZ ∆ ( BZ − BS )
BZ ∆ ( BZ
[Table]
BS )
T trans 74.
T inv 73.
-0.7 T trans ◦ T inv 55.
T perm 68.
-11.9 T trans
[Table]
perm -27.
T syn 74.
[Table]
9 T trans
[Table]
syn 69.
[Table]
[Table]
Composing transformations always hurts ∆ ( BZ − BS ) when compared to individual transformations.
[Table]
Parallel
[Table] [Table]
[Table] [Table]
Syntax -
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Same
[Table] [Table]
[Table] [Table]
Trans
[Table]
Inv -19.
Trans
[Table]
Perm -27.
table contains raw scores for results in Section 4.4 of the main paper.
Rows are sorted in descending order based on alignment .
We observe strong correlations between alignment and zeroshot transfer, with ρ s = 0 .
94 , p < .
005 on XNLI, ρ s = 0 .
93 , p < .
005 on NER, and ρ s = 0 .
p < .on POS.
[Table] [Figure]