Adapting Pre-trained Language Models to A frican Languages via Multilingual Adaptive Fine-Tuning (2022)

Abstract: Multilingual pre-trained language models (PLMs) have demonstrated impressive performance on several downstream tasks for both high-resourced and low-resourced languages. However, there is still a large performance drop for languages unseen during pre-training, especially African languages. One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts of a language using the pre-training objective. However, adapting to target language individually takes large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language. In this paper, we perform multilingual adaptive fine-tuning on 17 most-resourced African languages and three other high-resource languages widely spoken on the African continent to encourage cross-lingual transfer learning. To further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non-African writing scripts before MAFT, thus reducing the model size by around 50%. Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment classification) shows that our approach is competitive to applying LAFT on individual languages while requiring significantly less disk space. Additionally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods.

## introduction recent advances in the development of multilingual pre-trained language models (plms) like mbert @xcite , xlm-r @xcite , and @xcite have led to significant performance gains on a wide range of cross-lingual * * equal contribution. transfer tasks. due to the curse of multilinguality @xcite ) -a trade-off between language coverage and model capacity -and non-availability of pre-training corpora for many low-resource languages, multilingual plms are often trained on about 100 languages. despite the limitations of language coverage, multilingual plms have been shown to transfer to several low-resource languages unseen during pretraining. although, there is still a large performance gap compared to languages seen during pre-training. one of the most effective approaches to adapt to a new language is language adaptive fine-tuning (laft) -fine-tuning a multilingual plm on monolingual texts in the target language using the same pre-training objective. this has been shown to lead to big gains on many cross-lingual transfer tasks @xcite , and low-resource languages @xcite , including african languages @xcite . nevertheless, adapting a model to each target language individually takes large disk space, and limits the cross-lingual transfer abilities of the resulting models because they have been specialized to individual languages @xcite . an orthogonal approach to improve the coverage of low-resource languages is to include them in the pretraining data. an example for this approach is afrib-erta @xcite , which was trained from scratch on 11 african languages. a downside of this approach is that it is resource intensive in terms of data and compute. another alternative approach is parameter efficient fine-tuning like adapters @xcite and sparse fine-tuning @xcite , where the model is adapted to new languages by using a sparse network trained on a small monolingual corpus. similar to laft, it requires adaptation for every new target language. although it takes little disk space, all target language-specific parameters need to be stored. in this paper, we propose multilingual adaptive finetuning (maft), a language adaptation to multiple languages at once. we perform language adaptation on the 17 most-resourced african languages (afrikaans, amharic, hausa, igbo, malagasy, chichewa, oromo, naija, kinyarwanda, kirundi, shona, somali, sesotho, swahili, isixhosa, yorùbá, isizulu) and three other high-resource language widely spoken on the continent (english, french, and arabic) simultaneously to provide a single model for cross-lingual transfer learning for african languages. to further specialize the multilingual plm, we follow the approach of @xcite and remove vocabulary tokens from the embedding layer that correspond to non-latin and non-ge'ez (used by amharic) scripts before maft, thus effectively reducing the model size by 50%. our evaluation on two multilingual plms (afrib-erta and xlm-r) and three nlp tasks (ner, news topic classification and sentiment classification) shows that our approach is competitive to performing laft on the individual languages, with the benefit of having a single model instead of a separate model for each of the target languages. also, we show that our adapted plm improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods like adapters @xcite and sparse finetuning @xcite . as an additional contribution, and in order to cover more diverse african languages in our evaluation, we create a new evaluation corpus, antc -african news topic classification -for lingala, somali, naija, malagasy, and isizulu from pre-defined news categories of voa, bbc, global voices, and isolezwe newspapers. to further the research on nlp for african languages, we make our code and data publicly available. foot_0 additionally, our models are available via huggingface. foot_1 2 related work multilingual plms for african languages the success of multilingual plms such as mbert @xcite and xlm-r @xcite for cross-lingual transfer in many natural language understanding tasks has encouraged the continuous development of multilingual models @xcite @xcite @xcite . most of these models cover 50 to 110 languages and only few african languages are represented due to lack of large monolingual corpora on the web. to address this under-representation, regional multilingual plms have been trained from scratch such as afriberta @xcite or adapted from existing multilingual plm through laft @xcite @xcite . afriberta is a relatively small multilingual plm (126m parameters) trained using the roberta architecture and pre-training objective on 11 african languages. however, it lacks coverage of languages from the southern region of the african continent, specifically the southern-bantu languages. in our work, we extend to those languages since only a few of them have large (>100mb size) monolingual corpus. we also do not specialize to a single language but apply maft which allows multilingual adaptation and preserves downstream performance on both highresource and low-resource languages. adaptation of multilingual plms it is not unusual for a new multilingual plm to be initialized from an existing model. for example, @xcite trained infoxlm by initializing the weights from xlm-r before training the model on a joint monolingual and translation corpus. although they make use of a new training objective during adaptation. similarly, @xcite extended the languages covered by mbart @xcite ) from 25 to 50 by first modifying the vocabulary and initializing the model weights of the original mbart before fine-tuning it on a combination of monolingual texts from the original 25 languages in addition to 25 new languages. despite increasing the number of languages covered by their model, they did not observe a significant performance drop on downstream tasks. we take inspiration from these works for applying maft on african languages, but we do not modify the training objective during adaptation nor increase the vocabulary. compressing plms one of the most effective methods for creating smaller plms is distillation where a small student model is trained to reproduce the behaviour of a larger teacher model. this has been applied to many english plms @xcite @xcite and a few multilingual plms @xcite . however, it often leads to a drop in performance compared to the teacher plm. an alternative approach that does not lead to a drop in performance has been proposed by @xcite for multilingual plm. they removed unused vocabulary tokens from the embedding layer. this simple method significantly reduces the number of embedding parameters thus reducing the overall model size since the embedding layer contributes the most to the total number of model parameters. in our paper, we combine maft with the method proposed by @xcite to reduce the overall size of the resulting multilingual plm for african languages. this is crucial especially because people from under-represented communities in africa may not have access to powerful gpus in order to fine-tune large plms. also, google colab foot_2 (freeversion), which is widely used by individuals from under-represented communities without access to other compute resources, cannot run large models like e.g. xlm-r. hence, it is important to provide smaller models that still achieve competitive downstream performance to these communities. ## pre-trained language models for our experiments, we make use of different multilingual plms that have been trained using a masked language model objective on large collections of monolingual texts from several languages. ## multilingual adaptive fine-tuning we introduce maft as an approach to adapt a multilingual plm to a new set of languages. adapting plms has been shown to be effective when adapting to a new domain @xcite or language (pfeiffer et al., 2020; @xcite @xcite . while previous work on multilingual adaptation has mostly focused on autoregressive sequence-to-sequence models such as mbart @xcite , in this work, we adapt non-autoregressive masked plms on monolingual corpora covering 20 languages. crucially, during adaptation we use the same objective that was also used during pre-training. on a combination of the monolingual corpora used for afrimt5 adaptation by @xcite . details for each of the monolingual corpora and languages are provided in appendix a.1. hyper-parameters for maft the plms were trained for 3 epochs with a learning rate of 5e-5 using huggingface transformers @xcite . we use of a batch size of 32 for afriberta and a batch size 10 for the other plms. ## cross-lingual transfer learning the previous section demonstrates the applicability of maft in the fully-supervised transfer learning setting. here, we demonstrate that our maft approach is also very effective in the zero-shot cross-lingual transfer setting using parameter-efficient fine-tuning methods. ## xlm-r-base (lt-sft on news domain) here, we investigate how we can improve the cross-lingual transfer abilities of our adapted plm -afroxlmr-base by training language adapters on the same domain as the target task. for our experiments, we use the masakhaner dataset, which is based on the news domain. we compare the performance of language adapters trained on wikipedia and news domains. in addition to adapters, we experiment with another parameter-efficient method based on lottery-ticket hypothesis @xcite i.e. lt-sft @xcite . for the adapter approach, we make use of the mad-x approach @xcite ) -an adapter-based framework that enables cross-lingual transfer to arbitrary languages by learning modular language and task representations. however, the evaluation data in the target languages should have the same task and label configuration as the source language. specifically, we make use of mad-x 2.0 @xcite where the last adapter layers are dropped, which has been shown to improve performance. the setup is as follows: (1) we train language adapters via masked language modelling (mlm) individually on source and target languages, the corpora used are described in appendix a.2; (2) we train a task adapter by finetuning on the target task using labelled data in a source language. (3) during inference, task and language adapters are stacked together by substituting the source language adapter with a target language adapter. we also make use of the lottery ticket sparse fine-tuning (lt-sft) approach @xcite , a parameter-efficient fine-tuning approach that has been shown to give competitive or better performance than the mad-x 2.0 approach. the lt-sft approach is based on the lottery ticket hypothesis (lth) that states that each neural model contains a sub-network (a "winning ticket") that, if trained again in isolation, can reach or even surpass the performance of the original model. the lth is originally a compression approach, the authors of lt-sft re-purposed the approach for cross-lingual adaptation by finding sparse sub-networks for tasks and languages, that will later be composed together for zero-shot adaptation, similar to adapters. for additonal details we refer to @xcite . ## conclusion in this work, we proposed and studied maft as an approach to adapt multilingual plms to many african languages with a single model. we evaluated our approach on 3 different nlp downstream tasks and additionally contribute novel news topic classification 13 https://adapterhub.ml/ 14 https://huggingface.co/cambridgeltl dataset for 4 african languages. our results show that maft is competitive to laft while providing a single model compared to many models specialized for individual languages. we went further to show that combining vocabulary reduction and maft leads to a 50% reduction in the parameter size of a xlm-r while still being competitive to applying laft on individual languages. we hope that future work improves vocabulary reduction to provide even smaller models with strong performance on distant and low-resource languages. to further research on nlp for african languages and reproducibility, we have uploaded our language adapters, language sfts, afroxlmr-base, afroxlmr-small, and afroxlmr-mini models to the huggingface model hub 15 . language source size (mb) no. of sentences afrikaans (afr) mc4 (subset) (xue et al., 2021) 752.2mb 3,697,430 amharic (amh) mc4 (subset), and voa 1,300mb 2,913,801 arabic (ara) mc4 (subset) 1,300mb 3,939,375 english (eng) mc4 (subset), and voa 2,200mb 8,626,571 french (fra) mc4 (subset), and voa 960mb 4,731,196 hausa (hau) mc4 (all), and voa 594.1mb 3,290,382 igbo (ibo) mc4 (all), and afriberta corpus (ogueji et al., 2021) 287.5mb 1,534,825 malagasy (mlg) mc4 (all) 639.6mb 3,304,459 chichewa (nya) mc4 (all), chichewa news corpus (siminyu et al., 2021) 373.8mb 2,203,040 oromo (orm) afriberta corpus, and voa 67.3mb 490,399 naija (pcm) afriberta corpus 54.8mb 166,842 rwanda-rundi (kin/run) afriberta corpus, kinnews & kirnews (niyongabo et al., 2020), and voa 84mb 303,838 chishona (sna) mc4 (all), and voa 545.2mb 2,693,028 somali (som) mc4 (all), and voa 1,000mb 3,480,960 sesotho (sot) mc4 (all) 234mb 1,107,565 kiswahili (swa) mc4 (all) 823.5mb 4,220,346 isixhosa (xho) mc4 (all), and isolezwe newspaper 178.4mb 832,954 yorùbá (yor) mc4 (all), alaroye news, asejere news, awikonko news, bbc, and von 179.3mb 897,299 isizulu (zul) mc4 (all), and isolezwe newspaper 700.7mb 3,252,035