Does Corpus Quality Really Matter for Low-Resource Languages? (2022)

Abstract: The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl. While prior work has identified major issues on the quality of these datasets (Kreutzer et al., 2021), it is not clear how this impacts downstream performance. Taking representation learning in Basque as a case study, we explore tailored crawling (manually identifying and scraping websites with high-quality content) as an alternative to filtering CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque portion of popular multilingual corpora like CC100 and mC4, yet it has a much higher quality according to native annotators. For instance, 66% of documents are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and CC100. Nevertheless, we obtain similar results on downstream NLU tasks regardless of the corpus used for pre-training. Our work suggests that NLU performance in low-resource languages is not primarily constrained by the quality of the data, and other factors like corpus size and domain coverage can play a more important role.

## introduction large-scale pre-training has resulted in a paradigm shift in nlp @xcite . while recent progress has been primarily driven by scaling up on model size and compute, both data quantity and quality have been shown to play a critical role @xcite . nevertheless, existing efforts on data curation have primarily focused on english, and recent work on multilingual pre-training has relied on automatically filtered versions of commoncrawl. for instance, xlm-r was trained on cc100 @xcite , mt5 was trained on mc4 @xcite , and xglm was trained on cc100-xl @xcite , which were all obtained by running language identification on several commoncrawl snapshots and filtering through language-agnostic approaches. unfortunately, @xcite identified major issues on the quality of such multilingual datasets, ranging from language identification errors to boilerplate and non-linguistic content. however, the practical impact of these issues has not been studied, and it is unclear the extent to which higher-quality data could lead to better performance in low-resource languages. in this paper, we take representation learning in basque as a case study, and explore tailored crawling (i.e., manually identifying and scraping websites with high-quality content) as an alternative to filtering commoncrawl. we introduce euscrawl, a new corpus for basque comprising 12.5m documents from 33 websites with creative commons content. euscrawl is similar in size to the basque portion of cc100 and mc4, but it has substantially less issues and a higher perceived quality according to our blind audit with native annotators. however, we find that this improvement does not carry over to downstream nlu tasks, as masked language models pre-trained on either corpora obtain similar results on 5 benchmarks. our results suggests that data quantity and domain coverage play a more important role, prompting for methods to exploit more diverse sources of data in low-resource languages. this paper makes the following contributions: (i) we release euscrawl, a high-quality corpus for basque comprising 12.5m documents and 423m tokens; 1 (ii) we manually assess the quality of euscrawl in comparison with mc4 and cc100, finding that it has substantially less issues and a higher perceived quality according to native annotators; (iii) we compare masked language models pretrained on euscrawl, mc4, cc100 and wikipedia 2 on 5 nlu tasks, finding that they all perform similarly with the exception of wikipedia; and (iv) size tokens docs source mc4 @xcite 4,387 mib 1,004m 30,098k filtered commoncrawl cc100 @xcite 2,027 mib 416m 16,761k filtered commoncrawl wikipedia 313 mib 66m 2,685k wikipedia dump euscrawl (ours) 2,149 mib 423m 12,528k tailored crawling (see ## experimental setup we next detail the corpora compared in our experiments ( §2.1), and the qualitative and downstream evaluation settings ( §2.2 and §2.3). ## conclusions taking basque as a case study, our work gives further evidence that commoncrawl-derived corpora have major quality issues in low-resource languages. at the same time, we show that ad-hoc crawling websites with high-quality content can be an effective alternative to collect data in such languages. our resulting corpus euscrawl has a higher quality than mc4 and cc100 according to our manual data audit, while being similar in size. nevertheless, this improvement in quality does not carry over to downstream performance on nlu tasks, where we find evidence that data quantity and domain coverage are more important factors. our work leaves important lessons for future efforts on low-resource languages. first of all, we find that, even if commoncrawl derived multilingual corpora do have major quality issues as raised by prior work @xcite , these issues do not have a significant impact in nlu tasks. this suggests that investing on bigger and more diverse datasets might be more fruitful than addressing such quality issues in low-resource settings. given that the amount of written text in such languages is ultimately limited, we believe that developing effective cross-lingual transfer methods to exploit multilingual data is a promising future direction. having said that, it should be noted that our study is limited to nlu tasks in a single language. it is possible that data quality plays a more important role in generation tasks, which we leave for future work to study. in addition, we think that it would be valuable to conduct similar studies in other languages to corroborate our findings. finally, we note that prior work on basque nlp has often relied on private resources @xcite . our work sets a new state-of-the-art on a diverse set of nlu benchmarks, and it does so using public data alone. by releasing our corpus, we hope to facilitate future work in basque nlp, and encourage open and reproducible science using public resources.