Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021)

Abstract: Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.

## introduction models pretrained on unlabeled text corpora are the backbone of many modern nlp systems @xcite @xcite . this paradigm incentivizes the use of ever larger corpora @xcite , with the biggest models now training on a substantial fraction of the publicly-available internet @xcite . of course, as with all machine learning systems, the data such models are trained on has a large impact on their behavior. for structured, task-specific nlp datasets, best practices have emerged around documenting the collection process, composition, intended uses, and other provenance machine or human authored social biases documentation levels for commoncrawl-based datasets medical or health data data contamination demographic identities utterance date ## corpus-level statistics understanding the provenance of the texts that comprise a dataset is fundamental to understanding the dataset itself, so we begin our analysis of the metadata of c4.en by characterizing the prevalence of ## discussion & recommendations our analyses of c4.en and associated corpora revealed several surprising findings. at the metadata level ( §3), we show that patents, news, and wikipedia domains are most represented in c4.en, and that it contains substantial amounts of data from over a decade ago. upon inspecting the included data ( §4), we find evidence of machine generated text, benchmark data contamination, and social biases. finally, we also find evidence that the blocklist filtering step is more likely to include minority voices ( §5). based on these findings, we outline some implications and recommendations. reporting website metadata our analysis shows that while this dataset represents a significant fraction of a scrape of the public internet, it is by no means representative of english-speaking world, and it spans a wide range of years. when building a dataset from a scrape of the web, reporting the domains the text is scraped from is integral to understanding the dataset; the data collection process can lead to a significantly different distribution of internet domains than one would expect. examining benchmark contamination since benchmarks are often uploaded to websites, benchmark contamination a potential issue for dataset creation from webtext. @xcite raised this issue when introducing gpt-3, as they acknowledged that a bug in their filtering caused some benchmark contamination, found after finishing their training. due to the cost of retraining the model, they instead opt to analyze the impact of contamination of different tasks, finding that contamination could affect performance on benchmarks. our observations support dynamically collecting data with the human-in-the-loop approach @xcite that might reduce contamination of future benchmarks since (i) pretaining data is infrequently collected, and (ii) annotator-written examples for a given task are less likely to be (previously) crawled from the web. ## related work bert @xcite was trained on bookscorpus @xcite and englishlanguage wikipedia. it was soon improved with additional data (roberta; liu et al., 2019): a portion of cc-news (nagel, 2016), openwebtext @xcite , and stories (trinh and le, 2018). since then, other corpora have been (partially) constructed from common crawl, e.g., pile @xcite , ccnet @xcite , and mc4 @xcite . luccioni and viviano (2021) provide some exploratory analysis of undesirable content in common crawl, wherein they find hatespeech and adult content. one of the largest language models, gpt-3 @xcite , was trained on a mixture of filtered common crawl (60% of gpt-3's data), webtext2 (22%; @xcite , books1 and books2 (8% each; @xcite , and english-language wikipedia (3%). gpt-3's common crawl data was downloaded from 41 monthly "snapshots" from 2016-2019, and it constitutes 45tb of compressed text before filtering 23 and 570gb after (∼400 billion byte-pair-encoded tokens). since analyzing pretraining corpora is challenging due to their size, their documentation is of-ten missing @xcite . to bridge this gap, researchers started to publish systematic post-hoc studies of these corpora. gehman et al. (2020) provide an in-depth analysis with respect to toxicity and fake news of openwebtext. @xcite recruited 51 volunteers speaking 70 languages to judge whether five publicly available multilingual webcrawled corpora @xcite @xcite @xcite ) contain text in languages they report, as well as their quality. jo and gebru (2020) discuss parallels between creating historical archives and the curation of machine learning datasets including pretraining corpora. @xcite introduce a "framework for dataset development transparency that supports decisionmaking and accountability" that could be used for developing pretraining corpora. the masakhane organization advocates for participatory research @xcite , a set of methodologies that includes all necessary agents, e.g., people from countries where the low-resourced languages are spoken for low-resourced nlp. ## conclusion we present some of the first documentation and analyses of c4.en, a web-scale unlabeled dataset originally introduced by @xcite . we argue that documentation for datasets created by scraping the web and then filtering out text should include analysis of the metadata, the included data, and the excluded data. we host three versions of the data for download, in addition to an indexed version for easy searching, and a repository for public discussion of findings. 24