What’s in the Box? An Analysis of Undesirable Content in the C ommon C rawl Corpus (2021)

Abstract: Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.

## introduction in recent years, much of the progress in natural language processing (nlp) research has been largely driven by transformer-based language models, which have pushed forward the state-of-theart in tasks such as question answering @xcite and natural language inference @xcite . however, these increasingly complex models also require increasingly large amounts of data to train them, which is often a combination of curated, high-quality datasets such as encyclopedic articles and books and non-curated content from the web @xcite . this second category of large, non-curated dataset is becoming increasingly popular as they are required to train large language models. the current largest dataset used for training neural language models, the common crawl, is a non-curated corpus consisting of multilingual snapshots of the web. new versions of the common crawl are released monthly, with each version containing 200 to 300 tb of textual content scraped via automatic web crawling. this dwarfs other commonly used corpora such as english-language wikipedia, which adds up to roughly 5.6 tb of data, and the bookcorpus, which only represents around 6 gb @xcite . the common crawl has been used to train many of the recent neural language models in recent years, including the gpt model series @xcite , bert @xcite and fast-text @xcite and, given its size, often represents the majority of data used to train these architectures. in the current article, we present an initial analysis of the common crawl, highlighting the presence of several types of explicit and abusive content even after filtering. we discuss our findings and, given the potential downstream impact of this content on language models, we discuss the importance of ensuring that the corpora we use for training language models are extracted more mindfully and with more emphasis on their quality and propose avenues of research to achieve this goal. ## related work in recent years, a growing body of research in nlp has unearthed biases in common language models @xcite @xcite @xcite . this work has raised important questions regarding the impact of these embedded biases on downstream decision-making, given the increasing usage of these models in various applications. consequently, much work has also been dedicated to creating standardized diagnostic tests to detect these biases @xcite @xcite and to remove them @xcite @xcite , although the extent to which this is possible is still under debate @xcite . in fact, research has found that "the biases found in internet-scale language models like gpt-2 are representative of the data on which the model was trained" @xcite , which can be directly linked to the presence of hate speech on the internet @xcite . however, given the importance of this research, comparatively little attention has been dedicated to analyzing the corpora used to train language models. this is understandable because frequently used datasets such as the common crawl contain truly massive amounts of data, making it challenging to mine it for meaningful insights. in fact, a recent survey on automatic web page classification has deemed the task difficult not only due to the complexity and heterogeneity of web content, but also due its the high computational cost, suggesting that machine learning (ml) approaches have much to contribute to it @xcite . while certain notable endeavors have indeed analyzed specific aspects of corpora such as the common crawl @xcite and wikipedia @xcite , they have only scratched the surface of what these bodies of text contain. for instance, recent work has found that the common crawl contained over 300,000 documents from unreliable news sites and banned subreddit pages containing hate speech and racism @xcite , while complementary research has shown that individual training examples can be extracted by querying language models @xcite , together illustrating that the presence of questionable content is a significant issue for statistical language models. in the current work, we endeavor to understand the content and quality of the common crawl as a first step towards establishing more consistent approaches to filtering and refining it. ## analyzing the common crawl given its size, both downloading and analyzing the common crawl are time-consuming and costly endeavors. the most recent version of the common crawl, dating from november/december 2020, has 2.6 billion web pages in raw text format, saved in 'shards' each containing of tens of thousands of pages. given our hardware constraints, we chose to focus on a subset of the corpus, randomly sampling 1% of the files it contains, which after filtering by language amounts to roughly 115 gb of textual content or 5,835,339 web pages in total, which we analyzed in terms of hate speech, adult content, and efficacy of perplexity-based filtering foot_0 . in this work, we focus on detecting sexually-explicit and hate speech, since they represent common examples of "undesirable" content that can be generally seen as inappropriate for a language model to generate in most situations. we acknowledge that desirable model behaviour is application specific, and believe our findings can extend to any other "undesirable" topic that might be present in available language corpora. we present our results in the sections below.