Better Quality Pre-training Data and T5 Models for A frican Languages (2023)

Abstract: In this study, we highlight the importance of enhancing the quality of pretraining data in multilingual language models. Existing web crawls have demonstrated quality issues, particularly in the context of low-resource languages. Consequently, we introduce a new multilingual pretraining corpus for 16 African languages, designed by carefully auditing existing pretraining corpora to understand and rectify prevalent quality issues. To compile this dataset, we undertake a rigorous examination of current data sources for thirteen languages within one of the most extensive multilingual web crawls, mC4, and extract cleaner data through meticulous auditing and improved web crawling strategies. Subsequently, we pretrain a new T5-based model on this dataset and evaluate its performance on multiple downstream tasks. Our model demonstrates better downstream effectiveness over existing pretrained models across four NLP tasks, underscoring the critical role data quality plays in pretraining language models in low-resource scenarios. Specifically, on cross-lingual QA evaluation, our new model is more than twice as effective as multilingual T5. All code, data and models are publicly available at https://github.com/castorini/AfriTeVa-keji.

## introduction as language models have scaled up in size and multilingual capability in recent years, commensurate effort has followed to curate pretraining data @xcite to support this growth and improve the alignment of language models. earlier multilingual models such as mbert @xcite and xlm-r @xcite were trained on monolingual data from wikipedia and/or other large-scale web crawls which included only a few african languages. the introduction of mc4 @xcite , a document-level dataset spanning 101 languages helped alleviate this cover-age gap. 1 however, previous work @xcite has shown that mc4 and other existing largescale pretraining corpora have numerous quality issues, particularly for the low-resource african languages they contain. against this backdrop, indigenous efforts to build language resources for african languages have converged to two approaches: (1) small highquality data (e.g., 1gb) pretraining where most data are from the clean or verified sources like news domain @xcite . (2) large aggregation of all available data (e.g., 15 -42 gb) from noisy or unverified sources like cc-100 @xcite , and mc4, combined with high-quality sources like news corpora @xcite @xcite . this tradeoff between quantity and quality is forced by the unavailability of large, quality pretraining data for african languages. motivated by this need, we introduce a new multilingual pretraining corpus in 20 african languages. we draw from @xcite 's audit of existing pretraining corpora to understand prevailing quality issues. for mc4, they cite a high ratio both of sentences in incorrect languages (15.98% average) and nonlinguistic content (11.40% average). we trace these issues to the quality of data sources used in mc4 for the languages in our study and design heuristics to effectively extract clean monolingual text. more notably, we demonstrate how large-scale web crawls and document-level datasets, such as mc4, can be enhanced through meticulous auditing of their document sources i.e., base urls (e.g., www.voahausa.com ). interestingly, for numerous credible sources, mc4 encompasses fewer documents than what is actually available. we conduct our own web crawl of these sources, collecting more documents than what is present in mc4 for the respective languages. we consolidate the result of our efforts (cleaning and crawling) with data from other sources, notably wikipedia, and include four high-resource languages -arabic, english, french & portuguese. to evaluate the quality of our new corpus, we pretrain a new t5-based lm on the collected dataset and benchmark its performance on multiple downstream tasks. our model demonstrates improved effectiveness over existing pretrained lms further highlighting the importance of carefully curated datasets for pretraining language models in low-resource scenarios. our model was significantly better than the baseline mt5 models across four different downstream tasks. specifically, on cross-lingual qa evaluation, our new model achieves more than double the performance of multilingual t5. ## wura dataset we present wura, foot_2 a multilingual dataset comprising 16 african languages and 4 high-resource languages popularly spoken on the african continent -arabic, english, french, and portuguese. the curation of wura was carried out in a threepart process: -(i) auditing and cleaning mc4 (ii) crawling indigenous websites and (iii) combination with existing language resources. ## afriteva v2 large model we also pre-train a large variant of afriteva v2 using the same configuration of the t5-large model except for the vocabulary size which we set to be 150, 000, similar to the configuration of afriteva v2 (base) as detailed in subsection 3.1. we present the effectiveness of scaling to a large model size on summarization and news topic classification tasks in appendix c. 4 6 related work absence of a large monolingual corpus has always been the major challenge of leveraging the benefits of self-supervised pretraining for building representation and language models for african languages. the most available corpus are mostly from religious corpus like bible @xcite or jw300 @xcite , wikipedia and common crawl archive. the latter often has significant quality issues @xcite . earlier works on building word representation models for african languages showed the importance of developing fasttext embeddings with small high-quality data @xcite over pretrained fasttext embeddings developed from 4 due to space constraint, we include results in appendix. the current models with impressive performance on african languages simply aggregate both lowquality data and high-quality data for pretraining @xcite . the quality of these models implies that there must be significant portions of the data that are of good quality. to this end, we systematically and rigorously filtered these low-quality data from mc4 corpus for african languages, similar to the oscar dataset approach. 5 to the best of our knowledge, no previous work has done this. oscar dataset only has few documents for african languages e.g., 37.2mb for afrikaans dataset while our filtered corpus has more than 4.5 gb. ## conclusion in this work, we look to address the lack of large, quality pretraining dataset for african languages. while previous works have highlighted quality issues in existing pretraining dataset such as mc4, we demonstrate how these datasets can be enhanced by auditing their document sources and incorporating rigorous data filtering methods. to highlight the effectiveness of our approach and the relevance of this new dataset, we train a new t5 model, afriteva v2, on our dataset. our experiments show significant improvements across existing nlp benchmarks for african languages underscoring the impact of qualitative pretraining data in training language models.