M ULTILINGUAL L ANGUAGE M ODEL A DAPTIVE F INET UNING : A S TUDY ON A FRICAN L ANGUAGES
A BSTRACT
Multilingual pre-trained language models (PLMs) have demonstrated impressive performance on several downstream tasks on both high resourced and lowresourced languages.
However, there is still a large performance drop for languages unseen during pre-training, especially African languages.
One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts of a language using the same pre-training objective.
However, African languages with large monolingual texts are few, and adapting to each of them individually takes large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language.
In this paper, we perform multilingual adaptive fine-tuning (MAFT) on 17 most-resourced African languages and three other high-resource languages widely spoken on the African continent – English, French, and Arabic to encourage cross-lingual transfer learning.
Additionally, to further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non-African writing scripts before MAFT, thus reducing the model size by around 50%.
Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment classification) shows that our approach is competitive to applying LAFT on individual languages while requiring significantly less disk space. Finally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods.
I NTRODUCTION
Recent advances in the development of multilingual pre-trained language models (PLMs) like mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and RemBERT (Chung et al., 2021) have led to significant performance gains on a wide range of cross-lingual transfer tasks.
Due to the curse of multilinguality (Conneau et al., 2020) – a trade-off between language coverage and model capacity – and non-availability of pre-training corpora for many low-resource languages, multilingual PLMs are often trained on about 100 languages.
Despite the limitations of language coverage, multilingual PLMs have been shown to transfer to several low-resource languages unseen during pre-training.
Although, there is still a large performance gap compared to languages seen during pre-training.
One of the most effective approaches to adapt to a new language is language adaptive fine-tuning (LAFT) — fine-tuning a multilingual PLM on monolingual texts in the target language using the same pre-training objective.
This has been shown to lead to big gains on many cross-lingual transfer tasks (Pfeiffer et al., 2020), and low-resource languages (Muller et al., 2021; Chau & Smith, 2021), including African languages (Alabi et al., 2020; Adelani et al., 2021).
Nevertheless, African languages with large monolingual texts are few and adapting a model to each of them individually takes ∗ Equal contribution. large disk space, and limits their cross-lingual transfer abilities because they have been specialized for one language.
An orthogonal approach to improve the coverage of low-resource languages is to include them in the pre-training data.
An example for this approach is AfriBERTa (Ogueji et al., 2021), which trains on 11 African languages from scratch.
A downside of this approach is that it is resource intensive in terms of data and GPU compute.
Another alternative approach is parameter efficient fine-tuning like Adapters (Pfeiffer et al., 2020) and sparse fine-tuning (Ansell et al., 2021), with the drawback that their cross-lingual transfer ability is dependent on having a source language with the same task and label configuration (e.g number of labels and label names) as the target language.
This is however not true in many settings.
In this paper, we propose multilingual adaptive fine-tuning (MAFT) .
We perform language adaptation on the 17 most-resourced African languages and three other high-resource language widely spoken on the continent – English, French, and Arabic – simultaneously to provide a single model for cross-lingual transfer learning for African languages.
To further specialize the multilingual PLM, we follow the approach of Abdaoui et al.
(2020) to remove vocabulary tokens from the embedding layer that corresponds to non-Latin and non-Ge’ez (used by Amharic) scripts before MAFT, thus effectively reducing the model size by 50%.
Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification and sentiment classification) shows that our approach is competitive to performing LAFT on the individual languages, with the benefit of having a single model instead of a separate model for each of the target languages.
Also, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods like sparse fine-tuning (Ansell et al., 2021).
As an additional contribution, and in order to cover more diverse African languages in our evaluation, we create a new evaluation corpus, ANTC — A frican N ews T opic C lassification for Lingala, Nigerian-Pidgin, Somali, and isiZulu from pre-defined news categories of VOA, BBC and Isolezwe newspapers.
To further the research on AfricaNLP, we will make our code 1 , models 2 and data publicly available.
R ELATED W ORKS
Multilingual PLMs for African languages.
The success of multilingual pre-trained language models (PLMs) such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) for crosslingual transfer in many natural language understanding tasks has encouraged the continuous development of multilingual models (Luo et al., 2021; Chi et al., 2021;
Ouyang et al., 2021; Chung et al., 2021;
He et al., 2021).
Most of these models cover 50 to 110 languages and only few African languages are represented due to lack of large monolingual corpora on the web.
To address this under-representation, regional multilingual PLMs have been trained from scratch such as AfriBERTa (Ogueji et al., 2021) or adapted from existing multilingual PLM through LAFT (Alabi et al., 2020; Pfeiffer et al., 2020; Muller et al., 2021; Adelani et al., 2021).
AfriBERTa is a smaller multilingual PLM (126M parameters) trained using the RoBERTa architecture on 11 African languages.
However, it lacks coverage of languages from the southern region of the African continent, specifically the southern-Bantu languages.
In our work, we extend to those languages since only a few of them have large ( > 100MB size) monolingual corpus.
We apply MAFT that allows multilingual adaptation and preserves downstream performance on both high-resource and low-resource languages.
Adaptation of multilingual PLMs.
It is not unusual for a new multilingual PLM to be initialized from an existing model.
For example, Chi et al.
(2021) trained InfoXLM by initializing the weights from XLM-R before training the model on a joint monolingual and translation corpus.
Although they make use of a new training objective during adaptation.
Similarly, Tang et al.
(2020) extended the languages covered by mBART (Liu et al., 2020b) from 25 to 50 by first modifying the vocabulary and initializing the model weights of the original mBART before fine-tuning it on a combination of monolingual texts from the original 25 languages in addition to 25 new languages.
Despite increasing the number of languages covered by their model, they did not observe a significant performance drop on downstream tasks.
We take inspiration from these works for applying MAFT on African languages, but we do not modify the training objective during adaptation nor increase the vocabulary.
Compressing PLMs.
One of the most effective methods for creating smaller PLMs is distillation where a small student model is trained to reproduce the behaviour of a larger teacher model.
This has been applied to many English PLMs (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020; Liu et al., 2020a) and a few multilingual PLMs (Wang et al., 2020; 2021).
However, it often leads to a drop in performance compared to the teacher PLM.
An alternative approach that does not lead to a drop in performance has been proposed by Abdaoui et al.
(2020) for multilingual PLM. They remove unused vocabulary tokens from the embedding layer.
This simple method significantly reduces embedding parameters thus reducing the overall model parameter size since the embedding layer contributes the most to the total number of model parameters.
In our paper, we combine MAFT with the method proposed by Abdaoui et al.
(2020) to reduce the overall size of the resulting multilingual PLM for
African languages.
This is crucial because especially people from under-represented communities in Africa may not have access to powerful GPUs in order to run large PLMs.
Also, Google Colab(free-version), which is widely used by individuals from under-represented communities without access to other compute resources, cannot run large models like e.g. XLM-R. Hence, it is important to provide models for these communities that have strong downstream task performance and are small.
Evaluation corpora in African languages.
One of the challenges of developing (multilingual)
PLMs for African languages is the lack of evaluation corpora.
There have been many efforts by communities like Masakhane to address this issue ( ∀ et al., 2020; Adelani et al., 2021).
We only find two major evaluation benchmark datasets that cover a wide range of African languages: one for named entity recognition (NER) (Adelani et al., 2021) and one for sentiment classification (Muhammad et al., 2022).
In addition, there are also several news topic classification datasets (Hedderich et al., 2020; Niyongabo et al., 2020; Azime & Mohammed, 2021) but they are only available for few African languages.
Our work contributes novel news topic classification datasets (i.e ANTC) for African languages: Lingala, Nigerian-Pidgin, Somali, and isiZulu.
DATA 3. A DAPTATION C ORPORA
We perform MAFT on 17 African languages (Afrikaans, Amharic, Hausa, Igbo, Malagasy,
Chichewa, Oromo, Naija, Kinyarwanda, Kirundi, Shona, Somali, Sesotho, Swahili, isiXhosa,
Yor ` ub ´ a, and isiZulu) covering the major African language families and 3 high resource languages (Arabic, French, and English) widely spoken in Africa.
We selected the African languages based on the availability of a sizeable amount of monolingual texts.
We obtain the monolingual texts from three major sources: the mT5 pre-training corpus which is based on Common Crawl Corpus 4 (Xue et al., 2021), British Broadcasting Corporation (BBC) News, Voice of America News 5 (Palen-Michel et al., 2022), and some other news websites based in Africa.
Table 6 provides a summary of the monolingual data, including their sizes and sources.
We pre-processed the data by removing lines that consist of numbers or punctuation only, and lines with less than 6 tokens.
3. E VALUATION T ASKS
We run our experiments on two sentence level classification tasks: news topic classification and sentiment classification, and one token level classification task: NER.
We evaluate our models on
English as well as diverse African languages with different linguistic characteristics.
3.2. E XISTING DATASETS
NER.
For the NER task we evaluate on the MasakhaNER dataset (Adelani et al., 2021), a manually annotated dataset covering 10 African languages (Amharic, Hausa, Igbo, Kinyarwanda, Luganda,
Luo, Nigerian Pidgin, Swahili, Wolof, and Yor`ub´a) with texts from the news domain.
For English, we use data from the CoNLL 2003 NER task (Tjong Kim Sang & De Meulder, 2003) also containing texts from the news domain.
For isiXhosa, we use the data from Eiselen (2016).
News topic classification.
We use existing news topic datasets for Amharic (Azime & Mohammed, 2021), English – AG News corpus – (Zhang et al., 2015), Kinyarwanda – KINNEWS – (Niyongabo et al., 2020), Swahili – new classification dataset– (David, 2020), and both Yor`ub´a and Hausa (Hedderich et al., 2020).
For dataset without a development set, we randomly sample 5% of their training instances and used them as a development set.
Sentiment classification.
We use the NaijaSenti multilingual Twitter sentiment analysis corpus (Muhammad et al., 2022).
This is a large code-mixed and monolingual sentiment analysis dataset, manually annotated for 4 Nigerian languages:
Hausa, Igbo, Yor`ub´a and Pidgin.
Additionally, we evaluate on the Amharic, and English Twitter sentiment datasets by Yimam et al.
(2020) and Rosenthal et al.
(2017), respectively.
For all the datasets above, we only make use of the positive, negative and neutral tweets.
3.2. N EWLY C REATED DATASETS
News topic classification We created a novel dataset, ANTC — A frican N ews T opic C lassification for 4 African languages.
We obtained data from three different news sources: VOA, BBC 6 and isolezwe 7 .
From the VOA data we created datasets for Lingala and Somali.
We obtained the topics from data released by Palen-Michel et al.
and used the provided urls to get the news category from the websites.
For pidgin and isiZulu, we scrapped news topic from the respective news website (BBC Pidgin and isolezwe respectively) directly base on their category.
We noticed that some news topics are not mutually exclusive to their categories, therefore, we filtered such topics with multiple labels.
Also, we ensured that each category has at least 200 samples. The categories include but not limited to, Africa, Entertainment, Health, and Politics.
The pre-processed datasets were divided into training, development, and test sets using stratified sampling with a ratio of 70:10:20.
Appendix A.2 has more details about the dataset size and news topic information.
M ODELS AND M ETHODS 4. P RE - TRAINED M ASKED L ANGUAGE M ODELS
For our experiments, we make use of different multilingual PLMs that have been trained using only the masked language model objective on large collections of monolingual texts from several languages.
Table 1 shows the number of parameters as well as the African languages covered by each of the models.
XLM-R (Conneau et al., 2020) has been pre-trained on 100 languages including eight African languages.
We make use of the XLM-R-base model with 270M parameters for MAFT because it was easier to adapt to more languages due to its smaller size compared to XLM-R-large. We additionally evaluated on XLM-R-large to compare its performance to the MAFT -adapted models that are of smaller sizes.
AfriBERTa (Ogueji et al., 2021) has been pre-trained only on African languages.
Despite its smaller parameter size (110M), it gives competitive performance to XLM-R on African language datasets (Adelani et al., 2021; Hedderich et al., 2020).
XLM-R-miniLM (Wang et al., 2020) is a distilled version of XLM-R with only 117M parameters.
[Table]
4. M ULTILINGUAL A DAPTIVE F INE - TUNING (MAFT)
We introduce MAFT as an approach to adapt a multi-lingual PLM to a new set of languages.
Adapting PLMs has been shown to be effective when adapting to a new domain (Gururangan et al., 2020) or language (Pfeiffer et al., 2020; Alabi et al., 2020; Adelani et al., 2021).
While previous work on multilingual adaptation has mostly focused on autoregressive sequence-to-sequence models such as mBART (Tang et al., 2020), in this work [Table] we adapt non-autoregressive masked PLMs on monolingual corpora covering 20 languages.
Crucially, during adaptation we use the training objective that was also used during pre-training.
The models resulting from MAFT were then fine-tuned on supervised NLP downstream tasks.
We only applied MAFT to smaller models (XLM-R-base, AfriBERTa, and XLM-R-miniLM), since one of our goals is to reduce model size, but XLM-R-large requires a lot of compute resources and the training is slower.
We call the resulting model after applying
MAFT to XLM-R-base as AfroXLMR-base , and AfroXLMR-mini when MAFT is applied to XLMR-miniLM.
For adaptation, we make use of the monolingual corpus used for AfriMT5 adaptation in Adelani et al. Details of the monolingual corpus and languages are in Appendix A.1. 4. P RE - TRAINED LM V OCABULARY R EDUCTION
Multilingual PLMs come with various parameter sizes, the larger ones having more than hundred million parameters, which makes fine-tuning and deploying such models a challenge due to resource constraints.
One of the major factors that contributes to the parameter size of these models is the embedding matrix whose size is a function of the vocabulary size of the model.
While a large vocabulary size is essential for a multilingual PLM trained on hundreds of languages, some of the tokens in the vocabulary can be removed when they are irrelevant to the domain or language considered in the downstream task, thus reducing the vocabulary size of the model.
Inspired by Abdaoui et al.
(2020) we experiment with reducing the vocabulary size of the XLM-R-base model before adapting via MAFT.
Although, there are two possibilities vocabulary reduction in our setting: (1) removal of tokens before MAFT or (2) removal of tokens after MAFT .
From our preliminary experiments, we find approach (1) to work better.
We call the resulting model, AfroXLMR-small .
To perform vocabulary reduction, we concatenate the monolingual texts from the 20 languages we which to adapt to.
Then, we apply sentencepiece to the concatenated texts using the original XLM-R-base tokenizer.
The frequency of all tokens in the resulting corpus is computed and we select the top-k most frequent tokens.
We assume that the top-k most frequent tokens should be representative of the vocabulary of the whole 20 languages.
We chose k = 70 . 000 which covers 99.8% of our multilingual adaptation corpus, respectively.
In addition, we include the top [Table]
tokens from the original XLM-R-base tokenizer in the new vocabulary to include frequent tokens that were not present in the new top-k tokens.
We note that our assumption above may not hold in the case of some very distant and low-resourced languages as well as when there are domain differences between the corpora used during adaptation and fine-tuning.
We leave the investigation of alternative approaches for vocabulary compression for future work.
[Table]
on the test sets after 50 epochs averaged over 5 runs.
Results are for all 4 tags in the dataset: PER, ORG, LOC, DATE/MISC.
For LAFT, we multiplied the size of XLM-R-base by the number of languages as LAFT results in a single model per language.
R ESULTS 5. B ASELINE R ESULTS
For the baseline models (top rows in Tables 2, 3, and 4), we directly fine-tune on each of the downstream tasks in the target language: NER, news topic classification and sentiment analysis.
Effect of languages seen during pre-training.
For NER and sentiment analysis we find XLMR-large to give the best overall.
We attribute this to the fact that it has a larger model capacity compared to the other PLMs.
Similarly, we find AfriBERTa and XLM-R-base to give better results on languages they have been pre-trained on, and in most cases AfriBERTa tends to perform better than XLM-R-base on languages they are both pre-trained on, for example amh , hau , and swa .
However, when the languages are unseen by AfriBERTa (e.g. eng , wol , lin , lug , luo , xho , zul ), it performs much worse than XLM-R-base and in some cases even worse than the XLM-RminiLM.
This shows that it may be better adapting to an new African language from a PLM that has seen numerous languages than one built on a subset of African languages.
LAFT is a strong baseline.
Here, we applied LAFT to the XLM-R-base model (last row in Tables 2, 3, and 4).
As our results show, applying LAFT on each language individually provides a significant improvement in performance across all languages and tasks we evaluated on.
Sometimes, the improvement is very large, for example, ( +7 . 4 ) F1 on Amharic NER and ( +9 . 5 ) F1 for Zulu newstopic classification.
The only exception is for English since XLM-R has already seen large amounts of English text during pre-training.
Additionally, LAFT tends to hurt the performance especially when training on a smaller corpus Pfeiffer et al.
5. M ULTILINGUAL A DAPTIVE F INE - TUNING
While LAFT provides an upper bound on downstream performance for most languages, our new approach is often competitive to LAFT.
On average, the difference on NER, news topic and sentiment classification is ( − 0 . 7 ), ( +0 . 1 ), and ( − 0 . 3 ) F1, respectively.
Crucially, compared to LAFT, MAFT results in a single adapted model which can be applied to many languages while LAFT results in a new model for each language.
Below, we discuss our results in more detail.
PLMs pre-trained on many languages benefit the most from MAFT.
We found all the PLMs to improve after we applied MAFT.
The improvement is the largest for the XLM-R-miniLM where, the performance improved by ( +4 . 6 ) F1 for NER, and ( +4 . 9 ) F1 for news topic classification.
Although, the improvement was lower for sentiment classification ( +0 . 8 ).
Applying MAFT on XLM-R-base gave the overall best result.
On average, there is an improvement of ( +2 . 6 ), ( +3 . 0 ), and ( +1 . 5 ) on
NER, news topic and sentiment classification, respectively.
The main advantage of MAFT is that it
[Table]
[Table] , and “yor” from Muhammad et al. (2022). For LAFT, we multiplied the size of XLM-R-base by the number of languages as LAFT results in a single model per language. allows us to use the same model for many African languages and cross-lingual transfer tasks instead of many models specialized to individual languages.
This significantly reduces the required disk space to store the models, without sacrificing performance.
Interestingly, there is no strong benefit of applying MAFT to AfriBERTa.
In most cases the improvement is ( < 0 . 4 ).
We speculate that this is probably due AfriBERTa’s tokenizer having a limited coverage.
We leave a more detailed investigation of this for future work.
More efficient models using vocabulary reduction.
Applying vocabulary reduction helps to reduce the model size by more than 50% before applying MAFT.
We find a slight reduction in performance as we remove more vocabulary tokens.
Average performance of XLM-R-base-v70k reduces by ( − 1 . 6 ), ( − 1 . 3 ) and ( − 0 . 6 ) F1 for NER, news topic, and sentiment classification compared to the XLM-R-base+LAFT.
Despite, the reduction in performance compared to XLM-R-base+LAFT, they are still better than XLM-R-miniLM, which has a similar model size, with or without MAFT.
We also find that their performance is better than that of the PLMs that have not undergone any adaptation.
We find the largest reduction in performance on amh which makes use of the Ge’ez script while other languages make use of Latin.
The vocabulary reduction impact the number of amh subwords that are covered by our tokenizer .
In summary, we recommend XLM-R-base+MAFT (i.e AfroXLMR-base) for all languages on which we evaluated, including high-resource languages like English, French and Arabic. If there are GPU resource constraints, we recommend using XLM-R-base-v70k+MAFT (i.e AfroXLMR-small).
[Table]
5. C ROSS - LINGUAL T RANSFER WITH L OTTERY T ICKET S PARSE F INE - TUNING
Lastly, we show that our adapted model obtained through MAFT improves the zero-shot transfer performance for NER. For this experiment, we make use of the adapted model for XLM-R-base which we call AfroXLMR-base for short. We make use of the Lottery Ticket Sparse Fine-tuning (LT-SFT) approach (Ansell et al., 2021), a parameter-efficient fine-tuning approach that has been shown to give a better performance than the MAD-X Adapter approach (Pfeiffer et al., 2020; 2021).
The LT-SFT approach is based on the Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019) that states that each neural model contains a sub-network (a “winning ticket”) that, if trained again in isolation, can reach or even surpass the performance of the original model. The LTH is originally a compression approach, Ansell et al. (2021) re-purposed the approach for cross-lingual adaptation by finding a sparse sub-networks for a task and a language, that will later be composed together for zero-shot adaptation, similar to Adapters. For our experiments, we followed the same setting as Ansell et al.
(2021) that adapted English
CoNLL03 to African languages (using MasakhaNER dataset) for the NER task using mBERT.
However, we adapted the same CoNLL03 dataset to MasakhaNER using XLMR-base and AfroXLMRbase.
For the training of the sparse fine-tunings (SFT) for African languages, we make use of the monolingual texts from the news domain since it matches the domain of the evaluation data.
Unlike,
Ansell et al.
(2021) that trained SFT on monolingual data from Wikipedia domain except for luo and pcm where the dataset is absent, we show that the domain used for training language SFT is also very important.
For a fair comparison, we reproduced Ansell et al.
(2021) results by training
LT-SFT on mBERT, XLM-R-base and AfroXLMR-base on target languages with the news domain corpus.
Although, we still make use of the pre-trained English language SFT 10 for mBERT and
XLM-R-base trained on the Wikipedia domain.
For the AfroXLMR-base, we make use of the same
English SFT as XLM-R-base because the PLM is already good for English language.
Table 5 shows the result of LT-SFT, we compare the performance of LT-SFT to fully supervised setting, where we fine-tune XLM-R-base on the training dataset of each of the languages, and evaluate on the test-set.
We find that LT-SFT using news domain for African languages produce much better performance ( +3 . 7 ) than LT-SFT trained largely on the wikipedia domain.
This shows that the domain of the data matters.
We also, find that training LT-SFT on XLM-R-base gives better performance than mBERT on all languages.
Overall, the best performance is obtained by training LT-SFT on AfroXLMR-base, and sometimes give better performance than the fully-supervised setting (e.g as observed in kin and lug , wol yor languages).
This shows that the MAFT approach is effective since the technique provides a better PLM that parameter-efficient methods can benefit from.
CONCLUSION
In this work, we proposed and studied MAFT as an approach to adapt multilingual PLMs to many
African languages with a single model.
We evaluated our approach on 3 different NLP downstream tasks and additionally contribute novel news topic classification dataset for 4 African languages.
Our results show that MAFT is competitive to LAFT while providing a single model compared to many models specialized for individual languages.
We went further to show that combining vocabulary reduction and MAFT leads to a 50% reduction in the parameter size of a XLM-R while still being competitive to applying LAFT on individual languages.
We hope that future work improves vocabulary reduction to provide even smaller models with strong performance on distant and low-resource languages.
To further research on AfricaNLP and reproducibility, we are releasing language SFTs,
AfroXLMR-base, AfroXLMR-small, and AfroXLMR-mini to the HuggingFace Model Hub A A PPENDIX A. M ONOLINGUAL C ORPUS FOR P RE - TRAINING [Table] A. N EWS CLASSIFICATION DATASETS
[Table]