A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity (2024)

Abstract: Pretraining data design is critically under-documented and often guided by empirically unsupported intuitions. We pretrain models on data curated (1) at different collection times, (2) with varying toxicity and quality filters, and (3) with different domain compositions. First, we find that temporal shift between evaluation data and pretraining data leads to performance degradation, which is not overcome by finetuning. Second, we measure the effect of quality and toxicity filters, showing a trade-off between performance on standard benchmarks and risk of toxic generations. We also find that the effects of different types of filtering are not predictable from text domain characteristics. Third, we empirically validate that heterogeneous data sources, like books and web, are beneficial and warrant greater prioritization. To date, these experiments constitute the single largest publicly documented empirical study of the effects of pretraining data. Spanning 28 unique 1.5 billion parameter models pretrained from scratch, these findings validate, quantify, and expose many undocumented intuitions about text pretraining, which ultimately support more informed data-centric decisions in model development.

## introduction the strong performance @xcite @xcite , and emergent abilities @xcite of modern language models (lms) depend on selfsupervised pretraining on massive text datasets. all model developers implicitly or explicitly decide the composition of these datasets: what data sources to include, whether to filter for attributes such as quality and toxicity, and when to gather new documents. while many of the most prominent models do not document their curation procedures @xcite , or only document which procedures they used @xcite @xcite , they rarely document why they chose those protocols or what effect those had. this documentation debt leaves practitioners to be guided by intuitions and precedents, neither thoroughly evaluated @xcite . given the outsized and fundamental role of pretraining data in modern lms, we believe this practice has detracted from responsible data use and hampered effective model development @xcite @xcite . among the small number of general-purpose lms dominating community use and discussion, the prevailing focus has been on the scale of pretraining data and number of optimization steps @xcite @xcite . in this work, we select three common data design decisions and systematically test how they affect model performance-specifically, we choose the time of collection, content filtering strategy (toxicity/quality), and domain composition. we study the impacts in two ways. first, we present observational measurements of the effect of existing quality and toxicity filtering methods (section 3). we document how these filters affect a range of characteristics in two major pretraining datasets, c4 @xcite and the pile @xcite . second, we rigorously evaluate these dataset decisions on downstream tasks by evaluating decoder-only autoregressive 1.5b-parameter lms each pretrained on a dataset modified along one dimension of time, toxicity, quality, or domain composition. we summarize our findings and recommendations to model developers as follows: dataset age (section 4). performance degrades when the age of evaluation data does not match pretraining data, both when evaluation data is newer and older. this phenomenon is not eliminated by substantial finetuning and is exacerbated in larger models. the effect can meaningfully complicate comparisons between new and old models: for example, frozen benchmark datasets give a subtle advantage to models pretrained on older data. quality and toxicity filters (section 5). though defining document quality and toxicity is difficult, most language models use heuristics for filtering documents by quality and/or toxicity (see appendix, ## methodology we measure how pretraining data curation choices affect downstream performance. figure 1 illustrates our approach: each experiment starts with a pretraining dataset, applies a filter that removes documents, pretrains a language model on the curated dataset, and finetunes and evaluates the model on downstream tasks. ## impact of data curation on data characteristics we first present observational statistics on the pretraining datasets themselves. we find substantial interactions between curation choices. for more details on these features, see appendix e. ## impact of dataset age on pretrained models while models are frequently and cheaply updated with new finetuning data, the expense of pretraining means the nlp community has relied on relatively few static pretrained models that are rarely updated or exchanged. bert, roberta, gpt-2, and t5 variants, all pretrained prior to 2020, constitute the majority (estimated at ~58% as of april 16, 2023) of all models downloaded on hugging face. prior work demonstrates that language use changes over time @xcite and that temporal misalignment between finetuning and evaluation datasets correlates with degraded performance, visible across settings and domains @xcite @xcite . in contrast, we examine the effect of temporal misalignment between pretraining data and evaluation. in evaluating the impact of pretraining time across data domains, we can quantify the impact this design choice has on nlp broadly. we pretrain four autoregressive language models on versions of c4 @xcite , beginning with common crawl data and removing all data that was scraped after the cutoff year. following @xcite , we measure the effect of temporal misalignment by using evaluation tasks that have training and test sets split by year (from news, twitter, and science domains). after pretraining, we finetune each model on each dataset's training-year split separately, then evaluate on every test-year split. full details and results are in appendix d.2 and appendix f.1, respectively. we estimate the effects of temporal misalignment between pretraining and evaluation (appendix, figure 9 ). since all models were finetuned on the training sets of the evaluation tasks, we show that temporal misalignment during pretraining persists even with temporally-relevant finetuning data. performance degradation strongly correlates with pretraining misalignment and its effects are non-trivial. @xcite formalize a definition for temporal degradation (td), which measures the performance change observed from a one year difference between the finetuning and evaluation years. we generalize td to also measure the effect of a one year difference between pretraining time and evaluation time, as described in appendix d.2. furthermore, we quantify the strength of the relationship between performance difference and temporal difference using pearson correlation. in pretraining misalignment is not overcome by significant finetuning. the temporal degradation due to pretraining suggests models pretrained on data from the same timeframe as target evaluations will have advantages over models trained on much older or newer data. notably, this effect is observed for models which are finetuned on the full temporally-relevant training sets. this suggests that even substantial finetuning cannot overcome pretraining data that is temporally misaligned. pretraining misalignment effects are asymmetric and have implications for nlp evaluations. figure 3 summarizes temporal results, each of which is associated with an evaluation dataset, a pretraining year, a finetuning year, and an evaluation year. each result is compared to a baseline of the lowest performance for that dataset's evaluation year (across any pretraining and finetuning year combination). zero on the y-axis represents the lowest performance, and points are plotted based on their percentage improvement over their corresponding baseline. we observe performance degradation regardless of whether the pretraining data was collected before or after the evaluation data. while we would not expect a 2019 model to perform well on questions about covid, we also find that 2022 models perform less well on obama-era evaluations than earlier models. 60% 50% 40% 30% 20% 10% 0% 10% figure 3 also shows performance degradation is asymmetric: it is steeper when the evaluation year is after the pretraining year (blue bars) as opposed to the reverse (red bars). this finding suggests that both models and evaluations become stale: older models perform less well than newer models on new evaluations and newer models will perform less well on older evaluations. there are many possible causes; it could be because a lower fraction of the training dataset comes from the years that are relevant to the time-dependent task, or it might be because the facts change in pretraining data from later years. regardless of the cause, this phenomenon has implications for nlp experiments comparing models pretrained at different times. for instance, newer evaluation sets may appear much more difficult than old evaluation sets when applied to established, but less fresh, models. similarly, older evaluations may underestimate the capabilities of newer models. temporal degradation is greater for larger models. we find more temporal degradation for lm-xl (1.5b parameters) than for lm-small (20m parameters). as shown in small models. this suggests that larger models may have a greater sensitivity to temporal information than smaller models, which may not have the capacity to take advantage of subtle temporal features at all. full results for lm-small experiments are provided in appendix f.1. ## impact of quality & toxicity filters on pretrained models most modern large language models use some form of quality and/or toxicity filtering for their pretraining datasets (appendix, quality filters significantly improve performance across nearly all tasks, despite reducing training data quantity and variety. quality filters improve nearly all downstream tasks: toxicity identification by 2% (figure 4 , bottom) and most qa task categories by 1-6% ( the effect of quality filtering varies by dataset. in section 3, books, wikipedia, and web data are classified as highest quality. when we remove the highest estimated quality documents, wikipedia and web qa tasks are the most hurt, suggesting that these domains are not affected as much by the absence of the lowest quality data as the presence of the highest quality data. unexpectedly, both the quality and inverse quality filters led to models with higher toxic generation tendencies @xcite . different segments of data along this classifier's quality spectrum can have strong but varied effects on different domains. "quality" is a complex, situational concept, for which it is unlikely that one measure is sufficient. toxicity filtering leads to a trade-off between toxicity identification and toxic generation goals. we find that filtering using a toxicity classifier leads to a trade-off: models trained from heavily filtered pretraining datasets have the least toxic generation but also the worst toxicity identification (figure 4 , top). similarly, ## impact of domain composition on pretrained models as shown in common crawl, openweb, and books have the strongest positive effects on downstream performance. figure 5 shows that average downstream performance degrades the most when we remove web-based domains like cc, books, and openweb, corroborating recent findings by @xcite . these sources improve performance on challenging common sense and contrast sets tasks. while cc is the largest chunk of text in the pile, books and openweb are smaller but more diverse and high quality (see section 3). these results suggest that a combination of heterogeneity and quality is more impactful than raw dataset size. domain heterogeneity is often more beneficial than targeted data, even for targeted evaluations. performance degrades when we remove domains with close alignment between the pretraining and downstream data sources: removing pubmed hurts the biomed qa evaluations, dropping wikipedia hurts the wikipedia benchmarks, and removing web content hurts web evaluations. however, removing the large heterogeneous domains can have even more effect than the targeted domains. for instance, removing cc from pretraining reduces performance on downstream academic qa tasks to a much greater extent than removing the academic domain. we speculate that cc, openweb and books cover many topics, so removing the academic-specific category of sources does not remove all relevant academic information. the best performing models pretrain on all data sources. despite the importance of data heterogeneity, the best mean performance still comes from models that train on all, or nearly all, available data. the exceptions are the removal of targeted source domains like the pile's code or academic (advanced science and math journals) domains. these are both large but perhaps not well matched with the qa evaluation sets, which do not require coding skills or scientific rigour beyond that found on wikipedia and from web-based sources. this finding suggests that both the quantity and diversity of open source data remain a bottleneck for current pretraining methods. web and books domains cause the biggest trade-off between toxic identification and generation. section 5 identifies a trade-off: better performance on qa and toxicity identification comes at the cost of more toxic generation. ## discussion model developers often neglect to share empirical insights, maintaining a knowledge gap often referred to as "documentation debt" @xcite . as a result, pretraining dataset curation is frequently guided by intuitions or precedents that have not been thoroughly evaluated @xcite . our results show that choices made in pretraining curation affect models in ways that are not easily erased by subsequent finetuning. we urge both model producers and users to think of dataset curation policies as a form of hyperparameter, much like learning rates or network dimensions. exhaustive search methods that work for single scalar values will not, however, scale to curation policies that affect terabytes of data. in this section, we distill our findings into a few specific recommendations. age of the pretraining corpus. model creators must choose between model staleness and expensive continuous data collection and training. even with sufficient compute, newer data can add a "presentist" bias when evaluating retrospective tasks. our findings suggest the temporal properties of pretraining corpora are increasingly essential to consider for larger models, for more novel tasks (less finetuning data), and for instruction tuning models. for instance, @xcite suggests that finetuning on new information not represented during pretraining can encourage model hallucination. retrieval augmentation may offer some mitigation but presents its own challenges. we recommend model creators report the temporal distribution of pretraining data, which is not currently standard practice @xcite @xcite ; cohere ai, 2023). users should be able to predict otherwise unforeseen performance degradations on much newer datasets, or be aware of the potential side effects of finetuning models on information not covered in pretraining.