When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer (2022)

Abstract: While recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks, there is a lack of consensus in the community as to what shared properties between languages enable such transfer. Analyses involving pairs of natural languages are often inconclusive and contradictory since languages simultaneously differ in many linguistic aspects. In this paper, we perform a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four diverse natural languages and their counterparts constructed by modifying aspects such as the script, word order, and syntax. Among other things, our experiments show that the absence of sub-word overlap significantly affects zero-shot transfer when languages differ in their word order, and there is a strong correlation between transfer performance and word embedding alignment between languages (e.g., \rho_s=0.94 on the task of NLI). Our results call for focus in multilingual models on explicitly improving word embedding alignment between languages rather than relying on its implicit emergence.

## introduction multilingual language models like xlm @xcite and multilingual-bert @xcite are trained with masked-language modeling (mlm) objective on a combination of raw text from multiple languages. surprisingly, these models exhibit decent cross-lingual zero-shot transfer, where fine-tuning on a task in a source language translates to good performance for a different language (target). requirements for zero-shot transfer recent studies have provided inconsistent explanations for properties required for zero-shot transfer (hereon, transfer). for example, while @xcite conclude that sub-word overlap is vital for transfer, @xcite demonstrate that it is not crucial, although they consider only english as the source language. while @xcite suggest that typological similarity (e.g., similar svo order) is essential for transfer, other works @xcite successfully build multilingual models for dissimilar languages. need for systematic analysis a major cause of these discrepancies is a large number of varying properties (e.g., syntax, script, and vocabulary size) between languages, which make isolating crucial ingredients for transfer difficult. some studies alleviate this issue by creating synthetic languages which differ from natural ones only in specific linguistic properties like script @xcite . however, their focus is only on english as a source language, and the scale of their experiments is small (in number of tasks or pre-training corpora size), thus limiting the scope of their findings to their settings alone. our approach we perform a systematic study of cross-lingual transfer on bilingual language models trained on a natural language and a systematically derived counterpart. we choose four diverse natural languages (english, french, arabic, and hindi) and create derived variants using four different transformations on structural properties such as inverting or permuting word order, altering scripts, or varying syntax (section 3.2). we train models on each of the resulting sixteen language pairs, and evaluate zero-shot transfer on four downstream tasks -natural language inference (nli), namedentity recognition (ner), part-of-speech tagging (pos), and question-answering (qa). our experiments reveal the following: to our knowledge, we are the first study to quantitatively show that zero-shot transfer between languages is strongly correlated with token embedding alignment (ρ @xmath0.94 for nli). we also show that the current multilingual pre-training methods @xcite fall short of aligning embeddings even between simple natural and derived language pairs, leading to failure in zero-shot transfer. our results call for training objectives that explicitly improve alignment using either supervised (e.g., parallel corpora or bilingual dictionaries) or unsupervised data. ## related work multilingual pre-training for transformers the success of monolingual transformer language models @xcite has driven studies that learn a multilingual language-model (lm) on several languages. multilingual-bert (m-bert) @xcite ) is a single neural network pre-trained using the masked language-modeling (mlm) objective on a corpus of text from 104 languages. xlm @xcite introduced translation languagemodeling, which performs mlm on pairs of parallel sentences, thus encouraging alignment between their representations. these models exhibit surprising zero-shot cross-lingual transfer performance @xcite , a setup where the model is fine-tuned on a source language and evaluated on a different target language. @xcite , @xcite showed that transfer works even without a shared vocabulary between languages, @xcite discovered a correlation between sub-word overlap and zero-shot performance. @xcite and @xcite showed that shared parameters for languages with different scripts were crucial for transfer. @xcite and @xcite observed that transfer for ner and pos tagging works better between typologically similar languages. however, a study conducted by @xcite showed that there is no simple rule of thumb to gauge when transfer works between languages. @xcite observed that changing the syntax (sov) order of the source to match that of the target does not improve performance. ## approach we first provide some background on bilingual language models (section 3.1), followed by descriptions of our transformations (section 3.2), and our training and evaluation setup (section 3.3). ## results our experiments reveal several interesting findings for bilingual models including the situational importance of sub-word overlap for zero-shot transfer ( § 4.1, 4.2), the effect of domain mismatch between languages ( § 4.3), and correlation of zero-shot performance with embedding alignment ( § 4.4). we connect our findings to zero-shot transfer results between natural languages in section 4.5. ## conclusion through a systematic study of zero-shot transfer between four diverse natural languages and their counterparts created by modifying specific properties like the script, word order, and syntax, we showed that (1) absence of sub-word overlap hurts zero-shot performance when languages differ in their word order, and (2) zero-shot performance is strongly correlated with word embedding alignment between languages. some recent studies have implicitly or unknowingly attempted to improve alignment and have shown slight improvements in zero-shot transfer performance. however, our results lead us to posit that explicitly improving word embedding alignment during pre-training by using either supervised (e.g., parallel sentences and translation dictionaries) or unsupervised data will significantly improve zero-shot transfer. although real languages typically differ in more ways than the set of properties considered in our transformations, our methodology is still useful to help isolate crucial properties for transfer. future work can experiment with more sophisticated transformations and investigate closer connections with human language pairs. zhengxuan wu, isabel papadimitriou, and alex tamkin. 2022. oolong: investigating what makes crosslingual transfer hard with controlled studies. arxiv preprint arxiv:2202.12312.