Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022)

Abstract: Language models increasingly rely on massive web crawls for diverse text data. However, these sources are rife with undesirable content. As such, resources like Wikipedia, books, and news often serve as anchors for automatically selecting web text most suitable for language modeling, a process typically referred to as quality filtering. Using a new dataset of U.S. high school newspaper articles—written by students from across the country—we investigate whose language is preferred by the quality filter used for GPT-3. We find that newspapers from larger schools, located in wealthier, educated, and urban zones (ZIP codes) are more likely to be classified as high quality. We also show that this quality measurement is unaligned with other sensible metrics, such as factuality or literary acclaim. We argue that privileging any corpus as high quality entails a language ideology, and more care is needed to construct training corpora for language models, with better transparency and justification for the inclusion or exclusion of various texts.

## introduction the language models central to modern nlp are trained on large internet corpora, typically gathered from community resources (e.g., wikipedia; @xcite or web crawls (e.g., webtext, common crawl; @xcite . the selection of texts impacts every research or deployed nlp system that builds on these models. yet there is rarely any clear justification given for why various texts were included. web dumps like common crawl offer the promise of more diverse text than what is available in curated resources. however, much of the web consists of frequently replicated boilerplate (e.g., privacy policies), code (e.g., html and javascript), pornography, hate speech, and more. automated approaches, typically referred to as quality filters, are often applied in an effort to re-move this undesirable content from training data. 1 these filters include code removers @xcite , heuristics @xcite , stopwords @xcite , and classifiers @xcite . although quality filtering is often treated as a relatively neutral preprocessing step, it necessarily implies a value judgment: which data is assumed to be of sufficiently high quality to be included in the training corpus? more concretely, when a quality filter is a classifier trained on instances assumed to be of high (and low) quality, the selection of those examples will impact the language model and any downstream technology that uses it. many filters use wikipedia, books, and newswire to represent high quality text. but what texts are excluded as a result? because natural language varies with social and demographic variables @xcite @xcite @xcite , inter alia), we can also ask whose language will be excluded. we begin with a summary of the handful of data sources used to construct training corpora for many language models and assumed to be of high quality ( §2). the systematic authorship biases in these datasets motivate the study that follows, in which we replicate the quality filter from @xcite . we apply this filter to a new dataset of u.s. high school newspapers, augmented with demographic data from the u.s. census and the national center for education statistics ( §3). we demonstrate that the filter has strong topical and stylistic preferences, and favors text from authors who originate from regions with better educational attainment, urban centers, larger schools, and higher valued homes. in sociolinguistics, the term language ideology refers to common (but often unspoken) presuppositions, beliefs, or reflections about language that justify its social use and structure @xcite . our analysis helps to characterize the language ideology encoded in the quality filter used by @xcite , a representative of a wider set of filtering methods. we also observe in §4 that the filter is unaligned with other plausible notions of quality: factuality ratings for news sources, standardized test scores, and literary awards. of course, these institutions entail their own language ideologies. we argue that when constructing a corpus, one cannot avoid adopting some language ideology; appropriate choices will depend on the goals of the work, and one language ideology may conflict with another. in short, there is no truly general-purpose corpus. our code and analysis are publicly available. foot_1 ## motivation: data sources across the many language models recently reported in the literature, the same small group of datasets have been routinely used as training corpora-wikipedia, collections of books, and popular online articles ( §a.1). these data are often treated as exemplars of high quality text @xcite @xcite @xcite . although these datasets include text from many sources, extensive research suggests that the voices they represent are drawn from a relatively small, biased sample of the population, over-representing authors from hegemonic social positions. wikipedia wikipedia serves as a backbone for language models because of its scale, ease of use, permissive license, and goal of providing comprehensive coverage of human knowledge. however, although anyone can edit wikipedia content, not everyone does. in practice, there are significant biases in wikipedia authorship, content, and perspectives. for instance, despite efforts by wikimedia, the site has been unable to resolve a persistent gender imbalance among its editors @xcite . this imbalance is reflected in who gets written about, and how @xcite @xcite . there is also a pervasive urban bias; editors are less likely to come from rural areas, and coverage of these areas in wikipedia tends to be more limited @xcite . although coverage in english wikipedia is not limited to those places where english is a majority language, an anglo-american perspective dominates coverage. 3 lastly, a relatively small number of people are responsible for most of the content @xcite . wikipedia is thus less representative of language of the population than one might expect given its size and design. books language models are also frequently trained on book corpora. bert @xcite used the toronto bookcorpus @xcite , which consists of 7,185 self-published novels, a dataset criticized for copyright violation, imbalanced representation, and lack of documentation @xcite . @xcite and the pile @xcite both use much larger corpora of books (although the former do not identify the source of this data). however, the pile's books (also called books3) are not a random selection. rather, they appear to be drawn from a torrent file containing hundreds of thousands of copyrighted ebooks. books3 deserves a more thorough investigation, but preliminary analyses reveal that the most prevalent authors in the corpus are prolific american and british writers, especially of romance, mystery, and children's books (e.g., danielle steel). this pattern should be considered against the backdrop of the american book publishing industry, which has been widely criticized as homogeneous (lee & low books, 2020. foot_3 ) news and other popular internet content @xcite scrape text from the websites featured in popular reddit submissions (i.e., those that received at least three upvotes) to construct the training data for gpt-2. as the original corpus is unavailable, we analyze its open-source replica, openwebtext @xcite . we do not expect the corpus to represent a wide range of language variation; reddit users are mostly male, younger, and liberal-leaning, which influences the types of content shared and upvoted on the platform @xcite . indeed, we find that 1% of the 311k unique top-level domains in openwebtext contribute 75% of documents in the corpus ( §a.2). the most common websites in openwebtext are internationally circulating british and american news outlets (e.g., bbc, new york times), blogging platforms (e.g., tumblr, blogspot), sports content (e.g., espn, sb-nation), and tech news (e.g., techcrunch, wired). as expected, these links tend to appear on the most highly trafficked subreddits (e.g., /r/politics, /r/worldnews, /r/news). these news sources are likely dominated by formal writing styles from a relatively homogeneous set of authors @xcite . the adherence to slowly evolving style guides expresses specific linguistic standards @xcite and even geopolitical interests @xcite , which encourage rules on language use that can reinforce gender norms and racial hierarchies @xcite . researchers find a striking lack of diversity in newsrooms and newspaper leadership. 5 this may be compounded by the economic hardships aspiring journalists must incur, foot_6 which act as a filter for who can afford to be employed in the news industry. summary these descriptive findings suggest that a disproportionate amount of text in the core data sources of existing language models is written by authors from select, relatively powerful social positions. such text sources appear to favor privileged segments of the english-speaking population, including men; white populations; communities of higher socio-economic status; and people harboring american and western european historical, geopolitical, and cultural perspectives. the resulting corpora tend to be less inclusive of the voices of women and members of marginalized groups. a likely implication may be that alternative perspectives, including those of people from rural areas; non-dominant gender, sexual, or racial identities; and counter-hegemonic vantage points, may be less likely to be included, and thus less likely to influence models trained on this data. although formal, streamlined content like news or wikipedia articles may seem like desirable sources for high quality content, not all writing styles or substantive topics that might be relevant to language technologies and their user communities are represented in the resulting corpora. when deployed, however, many of the technologies using language models trained on these mainstream data will face language that-despite being less formal, professional, or carefully edited-is no less high quality and is essential to the communicative lives of the people who use it. ## measuring the language ideology of the gpt-quality filter empirically evaluating the full distribution of authors in the data sources from §2 is difficult, due to their size and the lack of metadata about each document's authors. we instead curate a new dataset of u.s. high school newspaper articles that varies both topically and along demographic variables that can be resolved using zip codes. although we do not directly consider individual authors of these articles, this dataset is useful, in that it can be associated with extensive metadata at the level of individual newspapers. we then analyze the behavior of a (replicated) quality filter on text from this dataset and discuss its implications. ## alignment with other notions of quality the gpt-3 quality filter purports to judge the quality of text, something that people also do, using a variety of different criteria. in this section, we consider three forms of human evaluations: factuality judgements, human-graded standardized test essays, and institutional book awards. how well does the behavior of the gpt-3 quality filter map onto these notions of quality? ## discussion the above sections have demonstrated that automated filtering of text to build language modeling corpora may lead to counterintuitive or undesirable exclusion of sources. because of the variety of use cases for language models and the broad range of text that could be appropriate for certain tasks, we suggest that there is no simple, universal standard for what should be considered high quality text. indeed, there is a long history of privileging some people's spoken language as better or more "correct" than others. researchers and practitioners of nlp who are aware of this history have the option to be intentional in their design of systems that, however implicitly, risk excluding the language of underprivileged identities or communities. some amount of selection in building corpora is inevitable. it is not possible to collect a uniform random sample of all written utterances. however, our findings suggest that current selection methods are, for many purposes, flawed. future work into alternative filtering criteria could be paired with investigations into the unintended consequences of their assumptions. we do not believe that there is likely to be a single solution to this challenge. indeed, the text that is best suited for training a model may depend on the application of that model. at a minimum, however, the nlp community could more carefully consider and clearly document the inclusion criteria for text. nlp practitioners could also be explicit about their reasons for using certain sources, even if those reasons are related to availability or empirical performance. a collection of tests could also be deployed (and improved over time) to give a clear understanding of the implications of different choices of filters. more generally, we echo calls in the literature for more thoughtful and inclusive data collection @xcite @xcite . strategies could include, but are not limited to a) intentionally curating data from people and viewpoints that are not otherwise well represented; b) including a greater diversity of genres; c) adopting more nuanced or intentional exclusion criteria; d) conducting more thorough interrogation of what text is being excluded; e) developing standard checks for prominent biases in inclusion; and/or f) abandoning the notion of a general-purpose corpus. language ideologies have an important, but often unacknowledged, influence on the development of nlp technologies @xcite . for example, an ideology that distinguishes between standard and non-standard language variations sur-faces in text normalization tasks @xcite , which tend to strip documents of pragmatic nuance @xcite and social signals @xcite . language on the internet has been historically treated as a noisy variant of english, even though lexical variation on the internet is highly communicative of social signals @xcite and varies considerably along demographic variables @xcite and community membership @xcite . language ideologies also surface in tools for toxicity detection; for example, the classification behavior of the perspective api (a popular hate speech detector) aligns with the attitudes of conservative, white, female annotators, who tend to perceive african-american dialects as more toxic @xcite . critiques of laissez-faire data collection we provide empirical evidence that laissez-faire data collection (i.e., filtering large web data sources) leads to data homogeneity @xcite . as an alternative to laissez-faire collection, jo and gebru (2020) recommend drawing on institutional archival practices. however, we note that language ideologies are also prevalent (and may not be explicit) in institutional archives, which, for example, have preferred colonizing perspectives over colonized ones when documenting historical events @xcite . other quality filters other definitions of text quality are used to create pretraining datasets, some of which do not rely on the datasets from §2. however, all techniques adopt language ideologies of what constitutes high quality text. bad-word filtering, which removes documents that contain certain stop-words, disproportionately excludes language about and by non-dominant groups @xcite . filtering internet content for popularity @xcite leads to data homogeneity based on the characteristics of viral media and the composition of userbases in online forums ( §2). even lightweight filters @xcite put more emphasis on features like document length over factuality when determining what makes a document high quality. any filtering method requires transparent justification and recognition of tradeoffs. ## conclusion using a new dataset of u.s. school newspapers, we find that the conventional, automated valuation of wikipedia, newswire, books, and popular internet content as reference for high quality text implicitly favors content written by authors from larger schools in wealthier, educated, urban areas of the united states. adopting this language ideology for text data selection leads to implicit-yet systematic and as-yet undocumented-inequalities in terms of whose language is more likely to be included in training corpora. although no single action will solve this complicated issue, data curators and researchers could be more intentional about curating text from underrepresented authors and groups, gathering sources from multiple genres and writing styles, and documenting their curation procedures and possible sources of exclusion.