Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023)

Abstract: With the prominence of large pretrained language models, low-resource languages are rarely modelled monolingually and become victims of the “curse of multilinguality” in massively multilingual models. Recently, AfriBERTa showed that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. Here we extend this direction, focusing on the use of related languages. We propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. We test our hypothesis on the Niger-Congo family and its Bantu and Volta-Niger sub-families, pretraining models with data solely from Niger-Congo languages and finetuning on 4 downstream tasks: NER, part-of-speech tagging, sentiment analysis and text classification. We find that models trained on genetically related languages achieve equal performance on downstream tasks in low-resource languages despite using less training data. We recommend selecting training data based on language-relatedness when pretraining language models for low-resource languages.

## introduction since the introduction of the large pretrained language models @xcite , low-resource languages have not had the opportunity to be treated in the same way as high-resource languages such as english, french or mandarin chinese. massively multilingual models trained using a mixture of high and low-resource languages such as mbert @xcite , xlm-roberta @xcite or mt5, @xcite have been proposed as a solution. yet these do not work as well on low-resource languages as they do on high-resource languages due to the "curse of multilinguality" @xcite , where an increase of languages in a model leads to capacity dilution, negatively affecting performance for all languages. this makes massively multilingual models sub-optimal solutions for such languages. the quality of the training data for low-resource languages seems to differ greatly to that of highresource languages @xcite . the afriberta models @xcite demonstrate the considerable success of pretrained representations when trained with a 'small' (1gb), highquality dataset focused on eleven languages of a single continent -africa. afriberta large outperforms the larger, massively multilingual models on named-entity-recognition (ner) and text classification for various african languages. while this continental approach is promising, it uses a mixture of different language families that are not genetically related. here, we propose using language relatedness in lieu of general geographic proximity of languages to pretrain transformer models. we test this hypothesis by grouping training data by language family and then testing on four tasks: ner, part-of-speech tagging (pos tagging), sentiment analysis and text classification. new models trained range from 100 to 600 mb of training data, in contrast to 1gb of data for afriberta and 2395 gb for xlm-r. we find that the smallest models trained on the most closely-related languages perform as well as models trained with up to 10 times the amount of data (afriberta). in this paper we: ## related work despite a long history of work on individual nlp tasks on african languages @xcite @xcite @xcite @xcite @xcite , the lack of freely available and aggregated models made it difficult for languages to build off of each other. the lack of adequate training data in lowresource languages, including african languages, led to multilingual pretraining transformer models such as mbert @xcite , xlm-roberta @xcite and mt5 @xcite using multilingual resources such as wikipedia and the common crawl corpus. in contrast, the "small data" approach, introduced with the release of the afriberta models @xcite advocates for pretraining models with small amounts of data solely in lowresource languages. the afriberta large model outperforms xlm-r and mbert on text classification and ner for a few african languages. this is likely due to the lack of inclusion of a range of african language data and the use of unclean, crawled datasets in the original training data for the large models. our proposal to use small, high-quality data draws on the finding that small data perform competitively given the right quality of data @xcite . our work asks how far we can extend this small data approach by seeing whether large uncurated datasets can be outperformed or at least equalled by small, carefully selected high-quality datasets. ## results results for our experiments are listed in figure 2 and tables 3 to 8 . given that datasets have data for languages in different families and branches, we select relevant models for comparison here and leave the full set of the results in the appendix. ## discussion in this work we see that when pretraining multilingual models with closely related languages, the resulting finetuned models work just as well as models finetuned on a wider variety of languages. sentence embeddings show that the more closely related the languages in the training data, the better the model's ability to differentiate language families. we do not see one model consistently outperforming others. however, we do see multilingual models of closely related languages work for those languages downstream and generalise better to unseen languages within the family. bantuberta works very well for pos tagging of bantu languages and voltaberta for sentiment analysis of volta-niger languages. continued pretraining of afriberta with closely related languages gives the best text classification result on average. this "small data" combined with language similarity approach demonstrates that it is possible to maintain performance with fewer resources, possibly at the expense of using different models for different downstream tasks. ## conclusion in this paper, we have pretrained several multilingual transformer models exclusively with lowresource languages. we have shown that the grouping of closely-related languages in training data can match or improve performance across several downstream tasks despite the reduction in training data used. we have also demonstrated that for very low-resource languages, we can exploit language similarity to improve performance of nlp tasks on these languages with models trained on similar languages only.