Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability (2022)

Abstract: Pretrained multilingual models enable zero-shot learning even for unseen languages, and that performance can be further improved via adaptation prior to finetuning. However, it is unclear how the number of pretraining languages influences a model’s zero-shot learning for languages unseen during pretraining. To fill this gap, we ask the following research questions: (1) How does the number of pretraining languages influence zero-shot performance on unseen target languages? (2) Does the answer to that question change with model adaptation? (3) Do the findings for our first question change if the languages used for pretraining are all related? Our experiments on pretraining with related languages indicate that choosing a diverse set of languages is crucial. Without model adaptation, surprisingly, increasing the number of pretraining languages yields better results up to adding related languages, after which performance plateaus. In contrast, with model adaptation via continued pretraining, pretraining on a larger number of languages often gives further improvement, suggesting that model adaptation is crucial to exploit additional pretraining languages.

## introduction pretrained multilingual language models @xcite ) are now a standard approach for cross-lingual transfer in natural language processing (nlp). however, there are multiple, potentially related issues on pretraining multilingual models. @xcite find the "curse of multilinguality": for a fixed model size, zero-shot performance on target languages seen during pretraining increases with additional pretraining languages only until a certain point, after which performance decreases. @xcite al. (2020b) also report "negative interference", where monolingual models achieve better results than multilingual models, both on subsets of high-and low-resource languages. however, those findings are limited to target languages seen during pretraining. current multilingual models cover only a small subset of the world's languages. furthermore, due to data sparsity, monolingual pretrained models are not likely to obtain good results for many lowresource languages. in those cases, multilingual models can zero-shot learn for unseen languages with an above-chance performance, which can be further improved via model adaptation with targetlanguage text @xcite , even for limited amounts @xcite . however, it is poorly understood how the number of pretraining languages influences performance in those cases. does the "curse of multilinguality" or "negative interference" also impact performance on unseen target languages? and, if we want a model to be applicable to as many unseen languages as possible, how many languages should it be trained on? specifically, we ask the following research questions: (1) how does pretraining on an increasing number of languages impact zero-shot performance on unseen target languages? (2) does the effect of the number of pretraining languages change with model adaptation to target languages? (3) does the answer to the first research question change if the pretraining languages are all related to each other? we pretrain a variety of monolingual and multilingual models, which we then finetune on english and apply to three zero-shot cross-lingual downstream tasks in unseen target languages: partof-speech (pos) tagging, named entity recognition (ner), and natural language inference (nli). experimental results suggest that choosing a diverse set of pretraining languages is crucial for effective transfer. without model adaptation, increasing the number of pretraining languages im-proves accuracy on unrelated unseen target languages at first and plateaus thereafter. last, with model adaptation, additional pretraining languages beyond english generally help. we are aware of the intense computational cost of pretraining and its environmental impact @xcite . thus, our experiments in section 4 are on a relatively small scale with a fixed computational budget for each model and on relatively simple nlp tasks (pos tagging, ner, and nli), but validate our most central findings in section 5 on large publicly available pretrained models. ## cross-lingual transfer via pretraining pretrained multilingual models are a straightforward cross-lingual transfer approach: a model pretrained on multiple languages is then fine-tuned on target-task data in the source language. subsequently, the model is applied to target-task data in the target language. most commonly, the target language is part of the model's pretraining data. however, cross-lingual transfer is possible even if this is not the case, though performance tends to be lower. this paper extends prior work exploring the cross-lingual transfer abilities of pretrained models for seen target languages depending on the number of pretraining languages to unseen target languages. we now transfer via pretrained multilingual models and introduce the models and methods vetted in our experiments. ## experimental setup pretraining corpora all our models are pretrained on the conll 2017 wikipedia dump @xcite . to use equal amounts of data for all pretraining languages, we downsample all wikipedia datasets to an equal number of sequences. we standardize to the smallest corpus, hindi. the resulting pretraining corpus size is around 200mb per language. 2 we hold out 1k sequences with around 512 tokens per sequence after preprocessing as a development set to track the models' performance during pretraining. corpora for model adaptation for model adaptation (rq2), we select unseen target languages contained in both xnli @xcite and universal dependencies 2.5 @xcite : farsi (fa), hebrew (he), french (fr), vietnamese (vi), tamil (ta), and bulgarian (bg). model adaptation is typically done for low-resource languages not seen during pretraining because monolingual corpora are too small @xcite . therefore, we use the johns hopkins university bible corpus by @xcite following @xcite . 3 tasks we evaluate our pretrained models on the following downstream tasks from the xtreme dataset @xcite ): pos tagging and nli. for the former, we select 29 languages from universal dependencies v2.5 @xcite . for the latter, we use all fifteen languages in xnli @xcite . we follow the default train, validation, and test split in xtreme. models and hyperparameters following conneau et al. (2020)'s xlm-r base model, we train transformers @xcite with 12 layers, 768 units, 12 attention heads, and a maximum of 512 tokens per sequence. to accommodate all languages for pos tagging and ner, we regard seventeen of the twenty-nine languages available in xtreme as unseen, while the remaining twelve languages are pretraining languages for at least one model. ## results we now present experimental results for each rq. ## conclusion this paper explores the effect which pretraining on different numbers of languages has on unseen target languages after finetuning on english. we find: (1) if not adapting the pretrained multilingual language models to target languages, a set of diverse pretraining languages which covers the script and family of unseen target languages (e.g., 17 languages used for xlm-17) is likely sufficient; and future directions include analyzing the effect of multilingual pretraining from different perspectives such as different pretraining tasks and architectures, e.g., mt5 @xcite , and more complex tasks beyond classification or sequence tagging. ie: germanic 0.0 0.5 1.0 af de en nl ie: slavic ru bg sino-tibetan zh afro-asiatic ar he ie: indo-aryan 0.0 0.5 1.0 hi mr ur ie: romance es fr it pt ie: greek el uralic et fi hu austronesian 0.0 0.5 1.0 id turkic tr basque eu japonic ja austro-asiatic en +ru +zh +ar +hi +es +el +fi +id +tr 0.0 0.5 1.0 vi koreanic en +ru +zh +ar +hi +es +el +fi +id +tr ko dravidian en +ru +zh +ar +hi +es +el +fi +id +tr ta te ie: iranian en +ru +zh +ar +hi +es +el +fi +id +tr fa the accuracy gain is significant for seen pretraining languages, and also the languages from the same family of the pretraining languages when added.