Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages (2021)

Abstract: Pretrained multilingual language models have been shown to work well on many languages for a variety of downstream NLP tasks. However, these models are known to require a lot of training data. This consequently leaves out a huge percentage of the world’s languages as they are under-resourced. Furthermore, a major motivation behind these models is that lower-resource languages benefit from joint training with higher-resource languages. In this work, we challenge this assumption and present the first attempt at training a multilingual language model on only low-resource languages. We show that it is possible to train competitive multilingual language models on less than 1 GB of text. Our model, named AfriBERTa, covers 11 African languages, including the first language model for 4 of these languages. Evaluations on named entity recognition and text classification spanning 10 languages show that our model outperforms mBERT and XLM-Rin several languages and is very competitive overall. Results suggest that our “small data” approach based on similar languages may sometimes work better than joint training on large datasets with high-resource languages. Code, data and models are released at https://github.com/keleog/afriberta .

## introduction pretrained language models have risen to the fore of natural language processing (nlp), achieving impressive performance on a variety of nlp tasks. the multilingual version of these models such as xlm-r @xcite and mbert @xcite have also been shown to generalize well to many languages. however, these models are known to require a lot of training data, which is often absent for low-resource languages. also, high-resource languages usually make up a significant part of the training data, as it is hypothesized that they help boost transfer to lower-resource languages. hence, there has been no known attempt to investigate if it is possible to pretrain multilingual language models solely on low-resource languages without any transfer from higher-resource languages, despite the numerous benefits that this could provide. motivated by this gap in the literature, the goal of our work is to explore the viability of multilingual language models pretrained from scratch on low-resource languages and to understand how to pretrain such models in this setting. we introduce afriberta, a transformer-based multilingual language models trained on 11 african languages, all of which are low-resource. 1 we evaluate this model on named entity recognition (ner) and text classification downstream tasks on 10 low-resource languages. our models outperform larger models like mbert and xlm-r by up to 10 f1 points on text classification, and also outperform these models on several languages in the ner task. across all languages, we obtain very competitive performance to these larger models. in summary, our contributions are as follows: our results show that, for the first time, it is possible to pretrain a multilingual language model from scratch on only low-resource languages and obtain good performance on downstream tasks. ## related work in recent years, unsupervised learning of text representations has significantly advanced natural language processing tasks. static representations from pretrained word embeddings @xcite were improved upon by learning contextualized representations @xcite . this has been noticeably improved further by pretraining language models @xcite based on transformers @xcite . these models have also been extended to the multilingual setting where a single language model is pretrained on several languages without any explicit cross-lingual supervision @xcite . however, much of this progress has been focused on languages with relatively large amounts of data, commonly referred to as high-resource languages. there has especially been very little focus on african languages, despite the over 2000 languages spoken on the continent making up 30.1% of all living languages @xcite . this is further visible in nlp publications on these languages. in all the association for computational linguistics (acl) conferences hosted in 2019, only 0.19% author affiliations were located in africa @xcite . other works @xcite have also noted the great disparity in the coverage of languages by nlp technologies. they note that over 90% of the world's 7000+ languages are under-studied by the nlp community. there have been a few works on learning pretrained embeddings for african languages, although many of them have been static and trained on a specific language @xcite @xcite . more recently, @xcite trained a bert model on the twi language. however, they note that their model is biased to the religious domain because much of their data comes from that domain. while some african languages have been included in multilingual language models, this coverage only scratches the surface of the number of spoken african languages. furthermore, the languages always make up a minuscule percentage of the training set. for instance, amongst the 104 languages that mbert was pretrained on, only 3 are african. 2 in xlm-r, there are only 8 african languages out of the 100 languages. in terms of dataset size, the story is the same. african languages make up 4.80 gb out of about 2395 gb that xlm-r was pretrained on, representing just 0.2% of the entire dataset @xcite . in mbert, african languages make up just 0.24 gb out of the approximately 100 gb that the model was pretrained on. all of this call for an obvious need for increased representation of african languages in modern nlp tools for the over 1.3 billion speakers on the continent. 3 pretrained language models have been shown to perform well when there is a lot of data @xcite , but some works have focused on using relatively smaller amounts of data. @xcite showed that it is possible to obtain state-of-the-art result with a french bert model pretrained on small-scale diverse data. in another work, @xcite showed that training a french bert language model on 100 mb of data yields similar performance on question answering as models pretrained on larger datasets. furthermore, ortiz suárez et al. (2020) obtained state-of-the-art performance with elmo @xcite language models pretrained on less than 1 gb of wikipedia text, and @xcite show that roberta language models @xcite trained on 10 to 100 million tokens can encode most syntactic and semantic features in its learned text representations. a common theme among these works is their focus on monolingual language models. while it is possible to learn monolingual language models on smaller amounts of data, it remains to be seen if it is possible in the multilingual case. our work is the first, to the best of our knowledge, that focuses on pretraining a multilingual language model solely on low-resource languages without any transfer from higher-resource languages. amharic, gahuza (a code-mixed language containing kinyarwanda and kirundi), hausa, igbo, nigerian pidgin, somali, swahili, tigrinya and yorùbá. these languages all come from three language families: niger-congo, afro asiatic and english creole. we select these languages because they are the languages supported by the british broadcasting corporation (bbc) news, which was our main source of data. 4 we also obtain additional data from the common crawl corpus @xcite for languages available there, specifically amharic, afaan oromoo, amharic, hausa, igbo, somali and swahili. size: the total size of our pretraining corpus is 0.94 gb (108.8 million tokens). in comparison, xlm-r was pretrained on about 2395 gb (164.0 billion tokens) @xcite , and mbert was trained on roughly 100 gb (12.8 billion tokens). 5 following findings from @xcite and @xcite that more data is always better for pretrained language modelling, our small corpus makes our task even more challenging, and one can already see that our model is at a disadvantage compared to xlm-r and mbert. our corpus contains approximately 5.45 million sentences and 108.8 million tokens. for each language we pretrained on that is present in xlm-r or mbert, we compare the size of that language in our dataset to its size in the pretraining corpora of mbert and xlm-r. from the comparison details in preprocessing: we remove lines that are empty or only contain punctuation. given that there is significant overlap between the african language corpora in common crawl and the bbc news data that we crawled, we perform extensive deduplication for each language by removing exact matched sentences. we also enforce a minimum length restriction by only retaining sentences with more than 5 tokens. we observe that the quality of the dataset from common crawl is very low, confirming recent findings from @xcite . hence, we manually clean the data as much as we can by removing texts in the wrong language, while trying to throw out as little data as possible. ## discussion in this section, we discuss some other contributions of this work. at a high level, afriberta presents the first evidence that multilingual language models are viable with very little training data. this offers numerous benefits for the nlp community, especially for low-resource languages. opportunities for smaller curated datasets: our empirical results suggest that state-of-the-art nlp methods like multilingual language models can be made more accessible for low-resource languages. @xcite recently showed that web-crawled multilingual corpora available for many languages, especially low-resource ones, are usually of very low quality. they found issues such as wrong-language content, erroneous language codes and low-quality sentences. our work opens the door to competitive multilingual language models on smaller curated datasets for low-resource languages. another possible benefit of these smaller curated datasets is that they would tend to contain local content as opposed to foreign content as is in the wikipedia and other relatively larger datasets of these languages. models trained on such datasets with local content could potentially be more useful to the speakers of the languages given that they would be trained on data with local context. strength of language similarity: our work challenges the commonly held belief in the nlp community that lower-resource languages need higher-resource languages in multilingual language models. instead, we empirically demonstrate that pretraining on similar low-resource languages in a multilingual setting may sometimes be better than pretraining using high-resource and low-resource languages together. this approach should be considered in future work, especially since there have been recent findings @xcite ) that lowresource languages also experience negative interference in multilingual models. potential ethical benefits: recent works have called for more considerations of ethics and related concerns in the development of pretrained language models @xcite . these concerns have ranged from environmental and financial @xcite to societal bias @xcite we believe our work offers the potential to ad- a comparison of model and data sizes of common multilingual models is presented in while "low-resource" is commonly used in the nlp community to describe a lack of data resources, @xcite have argued that "low-resource" also includes a wide range of societal problems, including computational constraints. thus, our work embodies the broader spirit of "lowresource", as we develop more efficient models on smaller data sizes for under-served languages. improving the representation of african languages in modern nlp tools: as discussed in section 2, there is very poor representation of african languages in modern nlp tools. recently, there have been significant efforts towards closing this gap @xcite @xcite @xcite @xcite . our work follows along this path, as there is a need to build language technologies for the over 1.3 billion people on the continent. besides showing that multilingual language models are viable on low-resource african languages with small training data, we also introduce the first language models for four of these languages: kinyarwanda, kirundi, nigerian pidgin and tigrinya. these are four languages with over 50 million speakers @xcite who are active users of digital tools. however, these languages have noticeably deficient support in nlp technologies. our work represents an important step towards improving this. ## conclusion in this work, we introduced afriberta, a multilingual language model pretrained on less than 1 gb of data from 11 african languages. we show that this model is competitive with models pretrained on larger datasets and even outperforms them on some languages. our comprehensive experiments also highlight important factors to consider when pretraining multilingual language models on smaller datasets. more importantly, we highlight some practical benefits of viable language models on smaller datasets. finally, we release code, pretrained models and the dataset to stimulate further work on multilingual language models for low-resource languages.