A fro LM : A Self-Active Learning-based Multilingual Pretrained Language Model for 23 A frican Languages (2022)

Abstract: In recent years, multilingual pre-trained language models have gained prominence due to their remarkable performance on numerous downstream Natural Language Processing tasks (NLP). However, pre-training these large multilingual language models requires a lot of training data, which is not available for African Languages. Active learning is a semi-supervised learning algorithm, in which a model consistently and dynamically learns to identify the most beneficial samples to train itself on, in order to achieve better optimization and performance on downstream tasks. Furthermore, active learning effectively and practically addresses real-world data scarcity. Despite all its benefits, active learning, in the context of NLP and especially multilingual language models pretraining, has received little consideration. In this paper, we present AfroLM , a multilingual language model pretrained from scratch on 23 African languages (the largest effort to date) using our novel self-active learning framework. Pretrained on a dataset significantly (14x) smaller than existing baselines, AfroLM outperforms many multilingual pretrained language models (AfriBERTa, XLMR-base, mBERT) on various NLP downstream tasks (NER, text classification, and sentiment analysis). Additional out-of-domain sentiment analysis experiments show that AfroLM is able to generalize well across various domains. We release the code source, and our datasets used in our framework at https://github.com/bonaventuredossou/MLM_AL .

## introduction with the appearance of transformer models @xcite , the field of natural language processing (nlp) has seen the emergence of powerful multilingual pre-trained language models (mplms), such as mbert @xcite , xlm-roberta (xml-r) @xcite , and mt5 @xcite . these prominent models have helped achieve state-of-the-art (sota) performance in many downstream nlp tasks such as named entity recognition (ner) @xcite @xcite ), text classification @xcite , and sentiment analysis @xcite @xcite . however they usually require a large amount of unlabeled text corpora for good performance: mbert was trained on wikipedia (2,500m words) and bookcorpus @xcite (800m words) across 104 languages -5 of which are african; mt5 supports 101 languages (13 african) and xlm-r supports 100 languages (8 african), and were trained on mc4 @xcite and commoncrawl data @xcite , respectively. this requirement for large-scale datasets contrasts sharply with the scarcity of available text corpora for african languages, which has pushed them into low-resource settings and largely excluded them from the pretraining phase of these large pre-trained models @xcite . this exclusion, leads very often to a poor performance on languages unseen during pre-training @xcite which eventually leads to inability to carry out the required nlp task. active learning is a semi-supervised machine learning algorithm that makes use of only a few initial training data points to achieve better performance of a given model m. the optimization is done by iteratively training m, and using another model n, usually referred to as the oracle, to choose new training samples that will help m find better configurations while improving its performance (e.g., prediction accuracy). this makes active learning a prevalent paradigm to cope with data scarcity. the efficiency of active learning (i.e. its ability to produce better performance despite being trained on a smaller training data) has been proven in tasks such as biological sequence design @xcite , chemical sampling (smith a study of active learning in the context of nlp has been carried out by @xcite . in their study, it is shown that active learning with db networks coupled with uncertainty measures and acquisition function outperforms several i.i.d baselines. they showed that with only 20% of samples labeled, their approach reached an accuracy of 98-99% on the named entity recognition (ner) task, while i.i.d tasks required 50% of labelled data to achieve comparable performance. in their study on clinical texts, @xcite also proved that active learning algorithms outperformed other learning methods. @xcite on their works with bert model(s) (for n different languages, there were n different bert-based models) went further by showing that active learning works with a balanced and unbalanced dataset. they also showed that the different active learning methods performed relatively the same. in our work, we fixed @xmath0. in our framework, we give m the ability to query itself, and use the knowledge acquired during each active learning round to construct new data points (from existing ones) that will be used for the next active learning round. we considered a diverse set of 23 african languages spread across the african continent. the selected languages are spoken in the south, central, east, and western regions of africa. the languages cover four language families: afro-asiatic (e.g., amharic, hausa, afan omoro), niger-congo (nc) (e.g., yorùbá, bambara, fon), english-creole (naija) and nilo-saharan (luo) (see appendix a for details). for each language, a dataset was collected from the news domain, which encompassed many topics such as health, politics, society, sport, environment, etc. our primary contribution to this work is our proposal of a self-active learning framework in which we pre-train the biggest multilingual african language model (for the number of languages covered) to date, and we show that our setup is very data-efficient and provides improvements on downstream nlp tasks such as ner, text classification, and sentiment analysis (even on outof-domain experiments). ## related works on mplms for african languages language adaptive fine-tuning (laft) is one of the best approaches to adapt mplms to a new language. this entails fine-tuning an mplm on monolingual texts of the said language with the same pre-training objective. however, this cannot be efficiently applied to african languages facing data-scarcity. (alabi et al., 2022b) proposed a new adaptation method called multilingual adaptive fine-tuning (maft), as an approach to adapt mplms to many african languages with a single model. their results show that maft is competitive to laft while providing a single model rather than many models that are specific for individual languages. nevertheless, alabi et al. (2022b)'s approach still works under the assumption that one does not need to train a model from scratch for languages in the low-resource settings, as they could benefit from high-resource languages. we find that this is not always the case. @xcite introduced afriberta, a multilingual language model trained on less than 1gb of data from 11 african languages. training afriberta from scratch showcased how african languages can benefit from being included in the pre-training stage of mplms. afriberta produced competitive results compared to existing mplms (e.g., mbert, xlm-r), and outperformed them on text classification and ner tasks. rather than relying on high-resource languages for transfer-learning, afriberta leverages the linguistic similarity between languages with low-resource settings to produce promising results. @xcite empirically demonstrates that this is more beneficial to these languages and is crucial in assessing the viability of language models pretrained on small datasets. in the next section, we will describe our selfactive learning framework, and the core details of our approach. ## self-active learning framework in this section, we describe our self-active learning framework (figure 1 ). in algorithm 1, we present a single active learning loop. in our current work, our model is trained only with a masked language modeling (mlm) objective @xcite @xcite . we plan to further incorporate translation language modeling (tlm) objective to improve translations of low-resource languages with relatively few thousands of data points foot_0 . this will be useful for both supervised and unsupervised translation @xcite . we used a shared sentence piece vocabulary with 250, 000 bpe codes. the subword shared vocabulary intends to improve alignment in the embedding space across languages (see languages description in appendix a and corpora details in the motivations behind the randomness in the selection of the training and held-out datasets are: as extensively detailed in algorithm 1, at each round we randomly select m sentences per language, from the held-out dataset of the language. for a language, to generate a new sentence s ′ , given an original sentence s, we proceed as follows (more details can be found in algorithm 1): the process described above will produce m new data points that will be added to the language dataset. the new dataset obtained is used to re-train the model from scratch at the next active learning round. ## experiments, results and discussion experiments: we use the xlm-roberta (xlm-r) architecture in our experiments based on previous works utilizing the model to achieve state-of-the-art performance in various downstream tasks. following the work and results of @xcite , we trained xlm-r-based models ## future works and conclusion in conclusion, we propose afrolm, a self-active learning-based multilingual language model supporting 23 african languages; the largest to date. our language datasets are collected from the news domain and span across different parts of the african continent. our experimental results on nlp downstream tasks (ner, text classification, and out-of-domain sentiment analysis), prove the data-efficiency of afrolm (as it has been trained on a dataset 14x smaller than its competitors), and its competitiveness as it outperforms many mplms (afriberta, mbert, xlmr-base) while being very competitive to afroxlmr-base. we