SERENGETI : Massively Multilingual Language Models for A frica (2023)

Abstract: Multilingual pretrained language models (mPLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. To date, only ~31 out of ~2,000 African languages are covered in existing language models. We ameliorate this limitation by developing SERENGETI, a set of massively multilingual language model that covers 517 African languages and language varieties. We evaluate our novel models on eight natural language understanding tasks across 20 datasets, comparing to 4 mPLMs that cover 4-23 African languages. SERENGETI outperforms other models on 11 datasets across the eights tasks, achieving 82.27 average F_1. We also perform analyses of errors from our models, which allows us to investigate the influence of language genealogy and linguistic similarity when the models are applied under zero-shot settings. We will publicly release our models for research. Anonymous link

## introduction pretraining nlp models with a language modeling objective has gained popularity as a precursor to task-specific finetuning @xcite . pretrained models like bert @xcite , elmo @xcite , roberta @xcite , gpt @xcite @xcite , and bart @xcite have advanced the state of the art in a wide variety of tasks, demonstrating how these models acquire valuable, generalizable linguistic information during the pretraining process. however, training language-specific models is possible for only a few languages which have large amounts of data. a popular alternative has been pretrained multilingual language models (mplm) such as mbert (devlin 1 https://github.com/ubc-nlp/serengeti ⋆ authors contributed equally. african countries our data comes from. the language varieties are represented as colored pie shapes within each country. we zero in on south africa, lesotho, swaziland, and senegal to show detail. we provide a larger map in appendix a.1. et al., 2019) and xml-r @xcite . mplms are trained on large amounts of unlabelled data from multiple languages so that low resource languages may benefit from shared vocabulary and other linguistic information from high-resource and similar languages in the model. the vast majority of the world's ∼ 7, 000 languages today remain uncovered by mplms, however. african languages are no exception. although there are few mplms that support a small number of african languages @xcite @xcite @xcite , these cover only a total of 31 languages. this is grossly inadequate considering that africa is believed to be home to ∼ 2, 000 languages @xcite . each of these languages encapsulates unique features that are essential in preserving linguistic diversity. the same way every species embodies essential value to the natural ecosystem, each language plays a crucial role in the linguistic ecosystem. that is, each language encodes knowledge about people, their traditions, wisdom, and environment, as well as how it is that they interact with the sum of the concepts in their own culture @xcite . this in turn allows people and communities to preserve and transmit their knowledge, values, unique modes of thinking, meaning and expression, history, culture, traditions, and memory to next generations, while participating in society and constructing their future @xcite . language technology plays an important role in building inclusive knowledge societies, providing access to education and information, supporting freedom of expression, cultural and linguistic diversity, and further stimulating innovation. this technology thus has great impact on multiple domains, including education, government, health, recreation, among others. this motivates adequate representation of african languages in the ongoing technological revolution. this is also likely to connect africa to the rest of the world. building technologies for african languages may also aid languages that may be at risk of falling into a state of disuse at an alarming rate, thus hopefully preventing subsequent language death that may become inevitable @xcite . developing lms that represent a large number of african languages is therefore very crucial for achieving progress in afrocentric nlp @xcite and indeed in addressing issues related to representation bias in artificial intelligence and linguistic diversity -two research themes of international relevance @xcite . motivated by this call for afrocentric nlp, we introduce serengeti. serengeti is a massively multilingual language model exploiting a large manually-curated dataset for 517 african languages and language varieties. these languages belong to 14 language families and are written in 5 different scripts. in addition to these african languages, serengeti is also pretrained on the top 10 most spoken languages globally. we also introduce afronlu, an extensive benchmark exploiting 20 different datasets across 28 different languages and language varieties for various nlp tasks. for even richer evaluation, we also apply our models to an african language identification task covering all the 517 languages in our pretraining. to the best of our knowledge, afronlu is the most extensive and inclusive evaluation benchmark proposed to date for african nlp. our contributions in this work are as follows: the rest of the paper is organized as follows: in section 2 we discuss related work. we describe genealogical information in section 3. next, we give a detailed description of serengeti in section 4. in section 5 we describe afronlu, the benchmark we create. we present performance of serengeti in section 6 and compare it to other mplms. we conclude in section 7, and outline a number of limitations and use cases for our work in section 8 and section 9. ## related work afrocentric nlp. an afrocentric approach to technology development is crucial for african languages. an afrocentric approach will mean that what technologies to build and how to build, evaluate, and deploy them arises from the needs of local african communities @xcite . we provide more details in section b in the appendix. african language models. here, we briefly describe language models covering any number of african languages. since we develop encoder-only models in this work, we will restrict our discussion to this category of models. we provide information about the african languages covered by these models in from various sources @xcite @xcite . it uses a self-active learning framework and achieves sota on ner, sentiment analysis, and text classification. afro-xlm-r @xcite uses language adaptation on the 17 most-resourced african languages and three other high-resource foreign languages widely used in africa (i.e., english, french, and arabic) simultaneously to provide a single model for crosslingual transfer learning. authors show that afro-xlm-r has competitive results with afriberta and xlm-r on ner, topic classification, and news classification. kinyabert (nzeyimana and niyongabo rubungo, 2022) uses a two-tier bert architecture that involves a morphological analyzer and explicitly represents morphological information for kinyawanda-a morphologically rich african language. authors show that kinyabert achieves good convergence and accuracy, and is robust on multiple downstream tasks. mbert @xcite is a multilingual variant of bert trained on 104 languages including four african languages. xlm-r @xcite uses a transformer based architecture and obtains sota on cross-lingual classification, sequence labeling, and question answering on 100 languages including eight african languages. ## genealogy of african languages genealogical or genetic classification groups languages based on their historical and evolutionary relationships. genetically related languages are often classified into similar families in a hierarchical tree like structure that shows the level of similarity between the languages. languages with a higher degree of similarity belong to the same class while languages with a lower degree of similarity are further subdivided into different classes and subclasses. two closely related languages can therefore be viewed as sisters of the same parent language/ancestor-they are languages that evolved over time and/or space from an older parent language @xcite . typological classification differs from geneological classification in that the former is based on grammatical features or types @xcite . for instance, a typological classification would group tone languages together, or split languages based on their morphological structure into, for instance, isolating or agglutinating languages. despite this difference, languages that belong to the same family often share similar typological information @xcite . for example, most benue-congo languages are tone languages @xcite . in the case of african languages, where typological information is scarcely available @xcite , utilizing genetic classes may be a useful way to determine typological information. if the typological information of one language in a group is known, we may make a sensible assumption that other languages in that group perhaps share similar features with minor variations. we use geneological classification information in evaluating serengeti's behaviour. specifically, we investigate the relationship between language similarity and model performance in zero-shot scenarios for south african languages in some datasets in our benchmark. we use classification information from ethnologue @xcite in all our analyses. we provide a broad overview of the families in our models under six broad ancestors in section d in the appendix. ## afronlu benchmark our goal is to evaluate our models extensively, and so we combine all available datasets we could acquire to create an evaluation benchmark that we refer to as afronlu. afronlu is composed of seven different tasks, covering both token and sentence level tasks, across 18 different datasets. the benchmark covers a total of 32 different languages and language varieties. in addition we evaluate our best model (serengeti) on an african language identification (lid) task covering all the 517 languages in our pretraining collection. for lid, we use two datasets to test serengeti. this puts afronlu at a total of 20 different datasets and eight different tasks. to the best of our knowledge, our evaluation benchmark is the most extensive compared to previous published research. we provide detailed statistics of the datasets comprising afronlu in we now describe each of the downstream tasks in afronlu. ## experimental setup and evaluation we evaluate serengeti on eight task clusters in the benchmark, and report results on our test set in the standard deviation. we also evaluate seren-geti on language identification and show results on afrolid in for multilingual datasets in each task, we show evaluation results per language, comparing the performance of various models in table e.4 in the appendix. task afrolid serengeti dev 96.14 ⋆ 97.64 ±0.02 test 95.95 ⋆ 97.41 ±0.02 ## conclusion we reported our efforts to develop serengeti, a suite of three massively multilingual language models for african nlp. serengeti outperforms 4 mplms on 11 datasets across 8 tasks. we provide extensive evaluations of model outputs, including zero-shot performance of the mplms. we also offer broad linguistically-motivated analyses of model performance.