When Being Unseen from m BERT is just the Beginning: Handling New Languages With Multilingual Language Models (2021)

Abstract: Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly improves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages.

## introduction language models are now a new standard to build state-of-the-art natural language processing (nlp) systems. in the past year, monolingual language models have been released for more than 20 languages including @xcite @xcite @xcite . additionally, large-scale multilingual models covering more than 100 languages are now available (xlm-r by @xcite and mbert by @xcite ). still, most of the 6500+ spoken languages in the world @xcite are not covered-remaining unseen-by those models. even languages with millions of native speakers like sorani kurdish (about 7 million speakers in the middle east) or bambara (spoken by around 5 million people in mali and neighboring countries) are not covered by any available language models at the time of writing. even if training multilingual models that cover more languages and language varieties is tempting, the curse of multilinguality @xcite makes it an impractical solution, as it would require to train ever larger models. furthermore, as shown by @xcite , large-scale multilingual language models are sub-optimal for languages that are under-sampled during pretraining. in this paper, we analyze task and language adaptation experiments to get usable language modelbased representations for under-studied low resource languages. we run experiments on 15 typologically diverse languages on three nlp tasks: part-of-speech (pos) tagging, dependency parsing (dep) and named-entity recognition (ner). our results bring forth a diverse set of behaviors that we classify in three categories reflecting the abilities of pretrained multilingual language models to be used for low-resource languages. we dub those categories easy, intermediate and hard. hard languages include both stable and endangered languages, but they predominantly are languages of communities that are majorly underserved by modern nlp. hence, we direct our attention to these hard languages. for those languages, we show that the script they are written in can be a critical element in the transfer abilities of pretrained multilingual language models. transliterating them leads to large gains in performance outperforming non-contextual strong baselines. to sum up, our contributions are the following: ## background and motivation as @xcite vividly illustrate, there is a large divergence in the coverage of languages by nlp technologies. the majority of the 6500+ of the world's languages are not studied by the nlp community, since most have few or no annotated datasets, making systems' development challenging. the development of such models is a matter of high importance for the inclusion of communities, the preservation of endangered languages and more generally to support the rise of tailored nlp ecosystems for such languages @xcite @xcite . in that regard, the advent of the universal dependencies project @xcite and the wikiann dataset @xcite have greatly increased the number of covered languages by providing annotated datasets for more than 90 languages for dependency parsing and 282 languages for ner. regarding modeling approaches, the emergence of multilingual representation models, first with static word embeddings @xcite and then with language model-based contextual representations @xcite enabled transfer from high to low-resource languages, leading to significant improvements in downstream task performance @xcite . furthermore, in their most recent forms, these multilingual models process tokens at the sub-word level @xcite . as such, they work in an open vocabulary setting, only constrained by the pretraining character set. this flexibility enables such models to process any language, even those that are not part of their pretraining data. when it comes to low-resource languages, one direction is to simply train contextualized embedding models on whatever data is available. another option is to adapt/fine-tune a multilingual pretrained model to the language of interest. we briefly discuss these two options. pretraining language models on a small amount of raw data even though the amount of pretraining data seems to correlate with downstream task performance (e.g. compare bert and roberta @xcite ), several attempts have shown that training a model from scratch can be efficient even if the amount of data in that language is limited. indeed, ortiz suárez et al. (2020) showed that pretraining elmo models @xcite on less than 1gb of text leads to state-of-the-art performance while @xcite showed that pretraining a bert model on as few as 4gb of diverse enough data results in state-of-the-art performance. @xcite further demonstrated that decent performance was achievable with only 100mb of raw text data. adapting large-scale models for low-resource languages multilingual language models can be used directly on unseen languages, or they can also be adapted using unsupervised methods. for example, @xcite successfully used unsupervised model adaptation of the english bert model to early modern english for sequence labeling. instead of fine-tuning the whole model, @xcite recently showed that adapter layers @xcite can be injected into multilingual language models to provide parameter efficient task and language transfer. still, as of today, the availability of monolingual or multilingual language models is limited to approximately 120 languages, leaving many languages without access to valuable nlp technology, although some are spoken by millions of people, including bambara and sorani kurdish, or are an official language of the european union, like maltese. what can be done for unseen languages? unseen languages strongly vary in the amount of available data, in their script (many languages use non-latin scripts such as sorani kurdish and mingrelian), and in their morphological or syntactical properties (most largely differ from high-resource indo-european languages). this makes the design of a methodology to build contextualized models for such languages challenging at best. in this work, by experimenting with 15 typologically diverse unseen languages, (i) we show that there is a diversity of behavior depending on the script, the amount of available data, and the relation to the pretraining languages; (ii) focusing on the unseen languages that lag in performance compared to their easier-tohandle counterparts, we show that the script plays a critical role in the transfer abilities of multilingual language models. transliterating such languages to a script which is used by a related language seen during pretraining can lead to significant improvement in downstream performance. ## experimental setting we will refer to any languages that are not covered by pretrained language models as "unseen." we select a small portion of those languages within a large scope of language families and scripts. our selection is constrained to 15 typologically diverse languages for which we have evaluation data for at least one of our three downstream tasks. our selection includes low-resource indo-european and uralic languages, as well as members of the bantu, semitic, and turkic families. none of these 15 languages are included in the pretraining corpora of mbert. information about their scripts, language families, and amount of available raw data can be found in the appendix in ## the three categories of unseen languages for each unseen language and each task, we experiment with our three modeling approaches: (a) training a language model from scratch on the available raw data and then fine-tuning it on any available annotated data in the target language. (b) fine-tuning mbert with task-tuning directly on the target language. (c) finally, adapting mbert to the unseen language using mlm-tuning before fine-tuning it in a supervised way on the target language. we then compare all these experiments to our non-contextual strong baselines. by doing so, we can assess if language models are 3 details about optimization can be found in appendix b , @xmath0. for hard languages, mbert under-performs the baselines in all settings. interestingly we find a large diversity of behaviors across languages regarding those language model training techniques. as summarized in figure 1, we observe three clear clusters of languages. the first cluster, which we dub "easy", corresponds to the languages that do not require extra mlm-tuning for mbert to achieve good performance. mbert has the modeling abilities to process such languages without relying on raw data and can outperform strong non-contextual baselines as such. in the second cluster, the "intermediate" languages require mlm-tuning. mbert is not able to beat strong non-contextual baselines using only task-tuning, but mlm-tuning enables it to do so. finally, hard languages are those on which mbert fails to deliver any decent performance even after mlm-and task-fine-tuning. mbert simply does not have the capacity to learn and process such languages. we emphasize that our categorization of unseen languages is only based on the relative performance of mbert after fine-tuning compared to strong non-contextual baseline models. we leave for future work the analysis of the absolute performance of the model on such languages (e.g. analysing the impact of the fine-tuning data set size on mbert's downstream performance). in this section, we present our results in detail in each of these language clusters and provide insights into their linguistic properties. ## tackling hard languages with multilingual language models our intermediate uralic language results provide initial supporting evidence for our argument on the importance of having pretrained lms on languages with similar scripts, even for generally highresource language families. our hypothesis is that the script is a key element for language models to correctly process unseen languages. to test this hypothesis, we assess the ability of mbert to process an unseen language after transliterating it to another script present in the pretraining data. we experiment on six languages belonging to four language families: erzya, bruyat and meadow mari (uralic), sorani kurdish (iranian, indo-european), uyghur (turkic) and mingrelian (kartvelian). we apply the following transliterations: ## discussion and conclusion pretraining ever larger language models is a research direction that is currently receiving a lot of attention and resources from the nlp research community @xcite . still, a large majority of human languages are under-resourced making the development of monolingual language models very challenging in those settings. another path is to build large scale multilingual language models. foot_4 however, such an approach faces the inherent zipfian structure of human languages, making the training of a single model to cover all languages an unfeasible solution @xcite . reusing large scale pretrained language models for new unseen languages seems to be a more promising and reasonable solution from a cost-efficiency and environmental perspective @xcite . recently, @xcite proposed to use adapter layers @xcite to build parameter efficient multilingual language models for unseen languages. however, this solution brings no significant improvement in the supervised setting, compared to a more simple masked-language model finetuning. furthermore, developing a language agnostic adaptation method is an unreasonable wish with regard to the large typological diversity of human languages. on the other hand, the promising vocabulary adaptation technique of @xcite which leads to good dependency parsing results on unseen languages when combined with task-tuning has so far been tested only on latin script languages (singlish and maltese). we expect that it will be orthogonal to our transliteration approach, but we leave for future work the study of its applicability and efficacy on more languages and tasks. in this context, we bring empirical evidence to assess the efficiency of language models pretraining and adaptation methods on 15 low-resource and typologically diverse unseen languages. our results show that the "hard" languages are currently outof-the-scope of any currently available language models and are therefore left outside of the current nlp progress. by focusing on those, we find that this challenge is mostly due to the script. transliterating them to a script that is used by a related higher resource language on which the language model has been pretrained on leads to large improvements in downstream performance. our results shed some new light on the importance of the script in multilingual pretrained models. while previous work suggests that multilingual language models could transfer efficiently across scripts in zero-shot settings @xcite , our results show that such cross-script transfer is possible only if the model has seen related languages in the same script during pretraining. our work paves the way for a better understanding of the mechanics at play in cross-language transfer learning in low-resource scenarios. we strongly believe that our method can contribute to bootstrapping nlp resources and tools for lowresource languages, thereby favoring the emergence of nlp ecosystems for languages currently under-served by the nlp community.