How to Adapt Your Pretrained Multilingual Model to 1600 Languages (2021)

Abstract: Pretrained multilingual models (PMMs) enable zero-shot learning via cross-lingual transfer, performing best for languages seen during pretraining. While methods exist to improve performance for unseen languages, they have almost exclusively been evaluated using amounts of raw text only available for a small fraction of the world’s languages. In this paper, we evaluate the performance of existing methods to adapt PMMs to new languages using a resource available for close to 1600 languages: the New Testament. This is challenging for two reasons: (1) the small corpus size, and (2) the narrow domain. While performance drops for all approaches, we surprisingly still see gains of up to 17.69% accuracy for part-of-speech tagging and 6.29 F1 for NER on average over all languages as compared to XLM-R. Another unexpected finding is that continued pretraining, the simplest approach, performs best. Finally, we perform a case study to disentangle the effects of domain and size and to shed light on the influence of the finetuning source language.

## introduction pretrained multilingual models (pmms) are a straightforward way to enable zero-shot learning via cross-lingual transfer, thus eliminating the need for labeled data for the target task and language. however, downstream performance is highest for languages that are well represented in the pretraining data or linguistically similar to a well represented language. performance degrades as representation decreases, with languages not seen during pretraining generally having the worst performance. in the most extreme case, when a language's script is completely unknown to the model, zero-shot performance is effectively random. while multiple methods have been shown to improve the performance of transfer to underrepresented languages (cf. section 2.3), previous work has evaluated them using unlabeled data from sources available for a relatively small number of languages, such as wikipedia or common crawl, which cover 316 foot_0 and 160 foot_1 languages, respectively. due to this low coverage, the languages that would most benefit from these methods are precisely those which do not have the necessary amounts of monolingual data to implement them as-is. to enable the use of pmms for truly low-resource languages, where they can, e.g., assist language documentation or revitalization, it is important to understand how state-of-the-art adaptation methods act in a setting more broadly applicable to many languages. in this paper, we ask the following question: can we use the bible -a resource available for roughly 1600 languages -to improve a pmm's zero-shot performance on an unseen target language? and, if so, what adaptation method works best? we investigate the performance of xlm-r @xcite when combined with continued pretraining @xcite , vocabulary extension, @xcite , and adapters @xcite ) making the following assumptions: (1) the only text available in a target language is the new testament, and (2) no annotated training data exists in the target language. we present results on 2 downstream tasks -partof-speech (pos) tagging and named entity recognition (ner) -on a typologically diverse set of 30 languages, all of which are unseen during the pretraining of xlm-r. we find that, surprisingly, even though we use a small corpus from a narrow domain, most adaptation approaches improve over xlm-r's base performance, showing that the bible is a valuable source of data for our purposes. we further observe that in our setting the simplest adaptation method, continued pretraining, performs best for both tasks, achieving gains of up to 17.69% accuracy for pos tagging, and 6.29 f1 for ner on average across languages. additionally, we seek to disentangle the effects of two aspects of our experiments on downstream performance: the selection of the source language, and the restricted domain of the new testament. towards this, we conduct a case study focusing on three languages with cyrillic script: bashkir, chechen, and chuvash. in order to understand the effect of the choice of source language, we use a more similar language, russian, as our source of labeled data. to explore the effect of the new testament's domain, we conduct our pretraining experiments with an equivalent amount of data sampled from the wikipedia in each language. we find that changing the source language to russian increases average baseline performance by 18.96 f1, and we achieve the highest results across all settings when using both wikipedia and russian data. ## results we present results for ner and pos tagging in tables 2 and 3, respectively. we additionally provide plots of the methods' performances as compared to the xlm-r baseline in figures 2 and 3 , showing performance trends for each model. (+mlm, +tlm, +{m|t}lm), perform best, with 3.93 to 6.29 f1 improvement over xlm-r. both +extend and +adapters obtain a lower average f1 than the xlm-r baseline, which shows that they are not a good choice in our setting: either the size or the domain of the bible causes them to perform poorly. focusing on the script of the target language (cf. pos tagging our pos tagging results largely follow the same trend as those for ner, with continued pretraining methods achieving the highest increase in performance: between 15.81 and 17.61 points. also following ner and as shown in figure 3 , the largest performance gain can be seen for languages with a low baseline performance, and, as the latter increases, the benefits obtained from adaptation become smaller. however, unlike for ner, all methods show a net increase in performance, with +adapters, the lowest performing adaptation model, achieving a gain of 9.01 points. we hypothesize that a likely reason for this is the domain and style of the bible. while it may be too restrictive to significantly boost downstream ner performance, it is still a linguistically rich resource for pos tagging, a task that is less sensitive to domain in general. additionally, there is a notable outlier language, coptic, on which no model performs better than random choice (which corresponds to 6% accuracy). this is because the script of this language is almost completely unseen to xlm-r, and practically all non-whitespace subwords map to the unknown token: of the 50% of non-whitespace tokens, 95% are unknown. while +extend solves this issue, we believe that for a language with a completely unseen script the bible is not enough to learn representations which can be used in a zero-shot setting. ## case study as previously stated, using the bible as the corpus for adaptation is limiting in two ways: the extremely restricted domain as well as the small size. to separate the effects of these two aspects, we repeat our experiments with a different set of data. we sample sentences from the wikipedia of each target language to simulate a corpus of similar size to the bible which is not restricted to the bible's domain or content. to further minimize the effect of domain, we focus solely on ner, such that the domain of the data is precisely that of the target task. additionally, we seek to further investigate the effect on the downstream performance gains of these adaptation methods when the source language is more similar to the target language. to this end, we focus our case study on three languages written in cyrillic: bashkir, chechen, and chuvash. we break up the case study into 3 settings, depending on the data used. in the first setting, we change the language of our labeled training data from english to russian. while russian is not necessarily similar to the target languages or mutually intelligible, we consider it to be more similar than english; russian is written in the same script as the target languages, and there is a greater likelihood for lexical overlap and the existence of loanwords. in the second setting, we pretrain using wikipedia and in the third setting we use both wikipedia data as well as labeled russian data. to create our wikipedia training data, we extract sentences with wikiextractor @xcite and split them with moses sentencesplitter @xcite . to create a comparable training set for each language, we first calculate the total number of subword tokens found in the new testament, and sample sentences from wikipedia until we have an equivalent amount. in the setting where we use data from the new testament and labeled russian data, for +tlm and +{m|t}lm we additionally substitute the english bible with the russian bible. when using wikipedia, we omit results for +tlm and +{m|t}lm, as they rely on a parallel corpus. ## limitations one limitation of this work -and other works which involve a high number of languages -is task selection. while part-of-speech tagging and named entity recognition foot_3 are important, they are both low-level tasks largely based on sentence structure, with no requirement for higher levels of reasoning, unlike tasks such as question answering or natural language inference. while xtreme @xcite ) is a great, diverse benchmark covering these higher level tasks, the number of languages is still limited to only 40 languages, all of which have wikipedia data available. extending these benchmarks to truly low resource languages by introducing datasets for these tasks will further motivate research on these languages, and provide a more comprehensive evaluation for their progress. additionally, while the bible is currently available in some form for 1611 languages, the available text for certain languages may be different in terms of quantity and quality from the bible text we use in our experiments. therefore, although we make no language-specific assumptions, our findings may not fully generalize to all 1611 languages due to these factors. furthermore, this work focuses on analyzing the effects of adaptation methods for only a single multilingual transformer model. although we make no model-specific assumptions in our methods, the set of unseen languages differs from model to model. moreover, although we show improvements for the two tasks, we do not claim to have state-of-the-art results. in a low-resource setting, the best performance is often achieved through task-specific models. similarly, translation-based approaches, as well as few-shot learning may offer additional benefits over a zero-shot setting. we also do not perform an extensive analysis of the target languages, or an analysis of the selected source language for finetuning. a better linguistic understanding of the languages in question would allow for a better selection of source language, as well as the ability to leverage linguistic features potentially leading to better results. finally, by using a pmm, we inherit all of that model's biases. the biases captured by word embeddings are well known, and recent work has shown that contextual models are not free of biases either @xcite . the use of the bible, and religious texts in general, may further introduce additional biases. last, we acknowledge the environmental impact from the training of models on the scale of xlm-r @xcite . ## conclusion in this work, we evaluate the performance of continued pretraining, vocabulary extension, and adapters for unseen languages of xlm-r in a realistic lowresource setting. using only the new testament, we show that continued pretraining is the best per-forming adaptation approach, leading to gains of 6.29 f1 on ner and 17.69% accuracy on pos tagging. we therefore conclude that the bible can be a valuable resource for adapting pmms to unseen languages, especially when no other data exists. furthermore, we conduct a case study on three languages written in cyrillic script. changing the source language to one more similar to the target language reduces the effect of adaptation, but the performance of the adaptation methods relative to each other is preserved. changing the domain of the adaptation data to one more similar to the target task while keeping its size constant improves performance.