Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2022)

Abstract: With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, Web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

## introduction access to multilingual datasets for nlp research has vastly improved over the past years. a variety of web-derived collections for hundreds of languages is available for anyone to download, such as paracrawl @xcite , wikimatrix @xcite , ccaligned @xcite , oscar @xcite , and several others. these have in turn enabled a variety of highly multilingual models, like mt5 @xcite , m2m-100 @xcite , and m4 @xcite . curating such datasets relies on the web sites giving clues about the language of their contents (e.g., a language identifier in the url) and on automatic language classification (langid). it is commonly known that these automatically crawled and filtered datasets tend to have overall lower quality than hand-curated collections @xcite , but their quality is rarely measured directly, and is rather judged through the improvements they bring to downstream applications @xcite . building nlp technologies with automatically crawled datasets is promising. this is especially true for low-resource languages, because data scarcity is one of the major bottlenecks for deep learning approaches. however, there is a problem: there exists very little research on evaluating both data collections and automatic crawling and filtering tools for low-resource languages. as a result, although many low-resource languages are covered by the latest multilingual crawl data releases, their quality and thus usability is unknown. to shed light on the quality of data crawls for the lowest resource languages, we perform a manual data audit for 230 per-language subsets of five major crawled multilingual datasets: foot_1 ccaligned @xcite , paracrawl @xcite , wiki-matrix @xcite , oscar @xcite , and mc4 @xcite . we propose solutions for effective, low-effort data auditing (section 4), including an error taxonomy. our quantitative analysis reveals surprisingly low amounts of valid in-language data, and identifies systematic issues across datasets and languages. in addition, we find that a large number of datasets is labeled with nontransparent or incorrect language codes (section 5). this leads us to reflect on the potential harm of low-quality data releases for low-resource languages (section 6), and provide a set of recommendations for future multilingual data releases (section 7). as the scale of ml research grows, it becomes increasingly difficult to validate automatically collected and curated datasets @xcite @xcite . several works have focused on advancing methodologies and best practices to address these challenges. @xcite introduced data statements, a documentary framework for nlp datasets that seeks to provide a universal minimum bar for dataset description. similar work has been done on systematizing documentation in other areas in data science and machine learning, including work focusing on online news @xcite , data ethics @xcite , and data exploration @xcite , as well as generalist work such as @xcite . data quality is also implicitly documented by successes of filtering methods. there is a large literature on filtering data for various nlp tasks, for example, @xcite , @xcite , @xcite , junczys-dowmunt (2018), and @xcite closest to our work is the analysis of a highly multilingual (non-publicly available) web-crawl and langid-related quality issues by @xcite . they perform a brief analysis of the quality of oscar with the focus only on the presence of in-language content. @xcite automatically documented and analyzed the contents and sources of c4 @xcite , the english counterpart of mc4, which surfaced parallel monolingual ccaligned paracrawl v7.1 wikimatrix oscar mc4 #languages 137 41 85 166 101 source cc 2013-2020 selected web sites wikipedia cc 11/2018 cc all filtering level document sentence sentence document document langid fasttext cld2 fasttext fasttext cld3 alignment laser vec/hun/bleu-align laser --evaluation ted-6 wmt-5 ted-45 pos/dep-5 xtreme ## multilingual corpora ccaligned (el-kishky et al., 2020) is a parallel dataset built off 68 cc snapshots. documents are aligned if they are in the same language according to fasttext langid @xcite , and have the same url but for a differing language code. these alignments are refined with cross-lingual laser embeddings @xcite . for sentence-level data, they split on newlines and align with laser, but perform no further filtering. human annotators evaluated the quality of document alignments for six languages (de, zh, ar, ro, et, my) selected for their different scripts and amount of retrieved documents, reporting precision of over 90%. the quality of the extracted parallel sentences was evaluated in a machine translation (mt) task on six european (da, cr, sl, sk, lt, et) languages of the ted corpus @xcite , where it compared favorably to systems built on crawled sentences from wikimatrix and paracrawl v6. multilingual c4 (mc4) @xcite is a document-level dataset used for training the mt5 language model. it consists of monolingual 2 http://commoncrawl.org/ . oscar @xcite ) is a set of monolingual corpora extracted from cc snapshots, specifically from the plain text wet format distributed by cc which removes all the html tags and converts the text to utf-8. it is deduplicated and follows the approach by @xcite of using fasttext langid @xcite on a line-level. 5 no other filtering was applied. for five languages (bg, ca, da, fi, id), oscar was used by its original authors to train language models which were then evaluated on parsing and pos tagging @xcite . oscar has also been used in independent studies to train monolingual or multilingual language models (ar, as, bn, de, el, fr, gu, he, hi, kn, ml, mr, nl, or, pa, ro, ta, te) and subsequently evaluate them on various downstream tasks (antoun et al., 2021; paracrawl v7.1 is a parallel dataset with 41 language pairs primarily aligned with english (39 out of 41) and mined using the parallel-data-crawling tool bitextor @xcite which includes downloading documents, preprocessing and normalization, aligning documents and segments, and filtering noisy data via bicleaner. 6 paracrawl focuses on european languages, but also includes 9 lower-resource, non-european language pairs in v7.1. sentence alignment and sentence pair filtering choices were optimized for five languages (mt, et, hu, cs, de) by training and evaluating mt models on the resulting parallel sentences. an earlier version (v5) was shown to improve translation quality on wmt benchmarks for cs, de, fi, lv, ro. wikimatrix @xcite is a public dataset containing 135m parallel sentences in 1620 language pairs (85 languages) mined from wikipedia. out of the 135m parallel sentences, 34m are aligned with english. the text is extracted from wikipedia pages, split into sentences, and duplicate sentences are removed. fasttext langid is used before identifying bitext with laser's distance-based mining approach. the margin threshold is optimized by training and evaluating downstream mt models on four wmt benchmarks (de-en, de-fr, cs-de, cs-fr). the final dataset is used to train translation models that are then evaluated by automatically measuring the quality of their translations against human translations of ted talks in 45 languages, with highest quality for translations between english and, for example, pt, es, da, and lowest for sr, ja, mr, zh tw. in the audit we focus on language pairs with english on one side. ## auditing data quality none of the above datasets has been evaluated for quality on the sentence level (exception: several languages in paracrawl v3), and downstream evaluations are centered around a small fraction of higher-resource languages. this is insufficient 6 https://github.com/bitextor/bicleaner . for drawing conclusions about the quality of individual or aligned sentences, and about the entirety of languages. in addition, there might be a publication bias preventing negative results with any of the above corpora with lower quality being published. to close this gap, we conduct a human data quality audit focused on the lowest-resource and most under-evaluated languages, but also covering mid-and high-resource languages for comparison. ## dataset mis-labeling standardized and unambiguous representations of language codes are important for practical data use and exchange. the standard used by most academic and industry applications is bcp-47 (phillips and davis, 2005), which builds off the two-letter iso639-2 codes and three-letter iso639-3 codes, but also allows for adding subtags for scripts (e.g., hindi in latin script: hi-latn) or regional varieties (e.g., french spoken in canada: fr-ca). it would enhance transparency and interoperability if adopted consistently, especially with growing language diversity in nlp. we find a variety of errors and inconsistencies in language code usage, ranging from serious mislabelings to small transgressions against standard conventions. for this analysis, we also include the jw300 (agić and vulić, 2019) dataset, a multilingual dataset crawled from jw.org. in summary, we find 8 nonstandard codes in ccaligned, 3 in oscar, 1 in mc4, 1 in wikimatrix, and 70 in jw300, for 83 in total. this does not include the 59 codes affected by superset issues. full details are given in appendix a. inconsistent language codes one common issue is simply using nonstandard or invented codes. for example, ccaligned uses only two-letter codes, so when the bcp-47 code for a language is three letters it is either shortened (e.g., @xmath0. similarly, oscar contains data labeled as als (bcp-47 for tosk albanian) that is actually in gsw (allemannic). 9 twenty-two additional language codes in jw300 have similar issues, including 12 codes that start with jw but are not javanese. false sign languages twelve percent (48/417) of jw300 carry language codes for sign languages. instead of sign language transcripts they are texts in another high-resource language, mostly english or spanish-for example, the en-zsl (zambian sign language) data is actually english-english parallel data (copies), details in appendix a. this was likely caused by videos with sign language interpretation embedded on the crawled web sites. 10 mysterious supersets when datasets contain language codes that are supersets of other language codes, it is difficult to determine which particular language the text contains. wikimatrix has serbian (sr), croatian (hr), bosnian (bs), and serbo-croatian (sh)-their superset. 11 the issue of codes that are supersets of others is common enough to include a small table dedicated to it (appendix ## future work and recommendations of the five multilingual corpora evaluated, we consistently found severe issues with quality, especially in the lower-resource languages. we rated samples of 205 languages, and found that 87 of them had under 50% usable data, with a full 15 languages at 0% in-language. we furthermore found consistent issues with mislabeled data and nonstandard language codes, particularly in the jw300 dataset, and identified 83 affected corpora, at least 48 of which were entirely spurious (section 5). while there might have been anecdotal evidence of insufficient quality for some of the datasets, the majority of these quality issues had not been reported, nor been investigated in depth. these issues might go unnoticed for languages that are not represented in the evaluation of the crawling methods, and cause harm in downstream applications @xcite . there are a variety of ways to improve both the ease and accuracy of human evaluation, as well a few classes of issues we ignored in this paper, like close dialects. ideally we would like to build a standard suite of automatic metrics for datasets, but more research is necessary to determine what the appropriate metrics would be. one important area missing from our analyses, however, is the estimated portion of a dataset which has been generated by mt @xcite , lm systems, or bots/templates, as for example in the analysis of c4 @xcite . the information captured in machine-generated content might still be useful for modeling, but might falsely overrepresent typical generation patterns and introduce linguistic errors or unnatural artifacts. we therefore strongly recommend looking at samples of any dataset before using it or releasing it to the public. as we have shown, one does not need to be proficient in a language to see when there are serious quality issues, and a quick scan of 100 sentences can be sufficient to detect major problems. moreover, going through and annotating a small sample of data can bring actionable insights about new ways to filter or use it. if data quality issues are found, a wide variety of techniques can be explored, like filtering on length-ratio, langid, tf-idf wordlists @xcite , or dictionaries @xcite ; to neural approaches like lm scoring @xcite @xcite . unfortunately, none of these provides a quick and easy fix, especially for low-resource languages-data cleaning is no trivial task! noisy datasets are by no means useless, at least if they contain some desirable content. therefore an alternative to filtering can be documentation @xcite . this can take the form of a per-language quality score and notes about known issues, a datasheet @xcite or nutrition label @xcite . however, we suggest researchers not release corpora with near-zero in-language content, as this may give the mistaken impression of usable resources. finally, we encourage the community to continue conducting evaluations and audits of public datasets-similar to system comparison papers.