A bout M e: Using Self-Descriptions in Webpages to Document the Effects of E nglish Pretraining Data Filters (2024)

Abstract: Large language models’ (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage are under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten “quality” and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.

## introduction large language models (llms) are sometimes described to be general-purpose (e.g. @xcite , and are increasingly incorporated into realworld applications. however, their behavior can reflect a limited set of human knowledge and perspectives @xcite @xcite . since the composition of pretraining data has been shown to impact model behavior @xcite @xcite , documentation of this data facilitates informed and appropriate application of models @xcite . in our work, we argue that it is additionally important to examine how data is transformed prior to pretraining, and document the implications of these "i am a published journalist and film reviewer living in sydney, studying journalism and writing. i love all things with a story." ## extracting social dimensions from about pages sociolinguists conceptualize language as a performance of one's social identity, or membership in a social group @xcite . websites' about pages capture aspects of their creators' social identities that they deem salient and significant enough to mention in a summary (figure 1 ). thus, these self-descriptions can help delineate meaningful differences in language varieties and use. we extract social aspects that are present across large sets of pages using automated methods, some of which we contribute as novel approaches. we do not examine attributes such as race or gender, as these are less commonly explicitly stated on about pages and may raise the risk of mismeasurement ( §8). figure 2 : examples of about web pages' topical interests annotated with cluster centers' top three representative words, obtained using an inverse transformation of cluster centroids and overlaid on a umap of pages. appendix c lists all 50 topical clusters. ## pretraining data filters raw data is transformed into pretraining data for llms through a variety of steps @xcite , including deduplication @xcite , decontamination (e.g. @xcite , and explicit content filtering (e.g. openai, 2023). we focus on analyzing the effects of "quality" filtering and english langid. the former is motivated by the subjectivity of how quality should be defined, and the latter by ongoing uncertainty around whether langid is robust to a wide range of language varieties @xcite . since web text contains noisy content (eisenstein, 2013), removing or downsampling "low quality" text is common in llm development ( we critically examine ten document-level quality and english filters that are sufficiently documented in prior work ( model-based quality. we experiment with quality filters that score text based on their similarity to some chosen "high quality" reference corpora. we name these filters based on the reference corpora used to train them: wikiwebbooks, openweb, wiki, and wikirefs ( heuristic-based quality. another quality filtering approach for web text applies rule-based heuristics @xcite . we examine 19 document-level heuristics and thresholds from gopher @xcite . these heuristics remove documents that do not meet thresholds pertaining to document and word length, textual repetition, and frequencies of symbols and common english words (appendix b). english langid. our data is pre-filtered to documents that fasttext langid scores as likely english @xcite . we also investigate whether the range of scores we observe with fasttext generalize to other model-based measurements of english used in llm development ( in this section, we overview the effects of data filters on sampled webpages grouped by social dimensions identified from their about pages. broadly, we examine the degree of consensus among filters when scoring pages, and identify themes that characterize their behavior. within some dimensions, we also investigate whether filtering rates reflect systemic differences in power and status among social groups @xcite . past llms have chosen a range of score cutoffs and sampling mechanisms to reduce undesirable text ( ## related work our measurements of self-descriptions are most related to prior work studying online selfpresentation. online biographies are a particularly rich source of social identity markers. for example, @xcite extracts personal identifiers, e.g. farm wife or umass amherst '20, from twitter profile bios, and show that on aggregate, identifiers in these bios align with users' sociodemographic backgrounds. others have extracted identifiers by splitting short form content by delimiters, e.g. 22yo | she/@xmath0, she/they} @xcite , or matching syntactic patterns, e.g. person is x @xcite . as one of the self-description analysis tools we employ, we contribute a novel and more flexible roberta-based approach in §2.4 for extracting social role identifiers. though a nascent research area, analyses of pretraining data curation decisions have also been the focus of other recent literature @xcite . for example, @xcite showed that discarding too much pretraining data using the pile's quality filter can lead to worse downstream task performance. our work is closest in spirit to @xcite , who use a dataset of high school newspapers to show that text from wealthier, more educated, and urban areas are more likely to be considered high quality by gpt-3's model-based quality filter. similarly, concurrent work by @xcite found that image-text clip-filtering for visual language models excludes data from lgbtq+ people, older women, and younger men at higher rates. we also critically examine social aspects of quality filtering at scale, but across a range of text filters. ## conclusion in our work, we examine how ten "quality" and english langid filters used during llm development affect web text created by a range of individuals and organizations with different topical interests, social roles, and geographic locations. to obtain this information, we use a new dataset of webpages that contain website creators' self-described social identities. overall, our framework allows for model developers and practitioners to better understand whether and how their choice of filtering approach may affect the resulting composition of web data in unintended ways. though some practices may seem tried and true for building powerful llms, we encourage future work to continue investigating, documenting, and mitigating their caveats and tradeoffs.