Data, Data Everywhere: A Guide for Pretraining Dataset Construction (2024)

Abstract: The impressive capabilities of recent language models can be largely attributed to the multi-trillion token pretraining datasets that they are trained on. However, model developers fail to disclose their construction methodology which has lead to a lack of open information on how to develop effective pretraining sets. To address this issue, we perform the first systematic study across the entire pipeline of pretraining set construction. First, we run ablations on existing techniques for pretraining set development to identify which methods translate to the largest gains in model accuracy on downstream evaluations. Then, we categorize the most widely used data source, web crawl snapshots, across the attributes of toxicity, quality, type of speech, and domain. Finally, we show how such attribute information can be used to further refine and improve the quality of a pretraining set. These findings constitute an actionable set of steps that practitioners can use to develop high quality pretraining sets.

## introduction recent language models (lms) @xcite @xcite have shown very strong capabilities on a number of evaluation areas. in comparison to previously developed lms @xcite @xcite @xcite , these newly released models generally follow the same architectural details, based on the transformer @xcite . rather, with emphasis being placed on the size and quality of the pretraining dataset @xcite , the improved capabilities of lms are largely due to self-supervised pretraining on ever larger, higher quality datasets. it is clear that the pretraining set is crucial to model success, but the question on how to effectively create one has yet to be openly answered. * correspondence to: jupinderp@nvidia.com most leading models @xcite @xcite do not divulge what methods were used to go from raw data sources to a final pre-training set. other models document only small sections of their process @xcite @xcite and lack information on why or how the chosen decisions were made. the scarcity of open knowledge in this area hinders the general community from contributing to the advancement of model capabilities @xcite . the steps in pretraining set construction are shown in figure 1 : the pipeline starts with a collection of text data sources, removes ill-formed and duplicate documents during data curation, further filters out low-quality documents via data selection, and finally assigns sampling weights to determine the prevalence of each data source during training. recent works @xcite @xcite have started to elucidate strategies for effective pretraining set development. however, they all focus solely on the step of data curation and analyze only a small number of mostly english sources. in this paper, we provide insights across all steps of pretraining set development for a set of over 2t tokens composed of english, multilingual, and source code documents. we compare existing methods through a series of ablations at each step of the development pipeline in figure 1 to quantify which techniques do and do not realize improvements in downstream evaluations. for the best identified method, we highlight various design decisions that impact performance. additionally, previous studies on web crawl are conducted across a small number of snapshots and are limited to the characteristics of toxicity and quality @xcite . despite web crawl documents constituting the majority of examples in pretraining sets @xcite @xcite , we still do not thoroughly understand their composition. we close this gap by conducting a large-scale analysis on over 90 common crawl web snapshots for the attributes of domain, quality, toxicity, and type of speech. we then show how such data attributes can aid in pretraining set construction to further improve model capabilities. by sharing this information, we provide an actionable series of steps that can be used to construct highly performant pretraining sets. concretely, our contributions are as follows: ## experimental setup we ablate a singular part of the development pipeline and train a lm on the resulting pretraining set to understand how various methods affect performance on downstream benchmarks. our experimental setup is detailed below. ## related work data curation, which is the identification, organization, storage and cleaning of datasets @xcite @xcite , has been the most wellstudied aspect in pretraining set development. early models, like bert @xcite and xlnet @xcite , focused their data curation efforts on obtaining examples from high quality sources. in conjunction with the creation of larger collections of datasets such as c4 @xcite , the pile @xcite , and big-science roots @xcite ), heuristic and classifier based filters were used in data curation to remove ill-formed and useless documents @xcite @xcite . additional studies within data curation found that data deduplication @xcite @xcite further improved model capabilities by preventing overtraining on a small set of similar examples. data selection and data sampling play major roles in pretraining set construction. data selection methods @xcite @xcite remove low quality documents to retain examples that more closely align with a predetermined high quality source. moore-lewis selection @xcite proposed the initial approach, with recent extensions by cynical data selection (axelrod, 2017) and dsir (xie et al., 2023b) which both better estimate the probability that a given example belongs to a high quality domain. data sampling techniques either use a learned model @xcite @xcite or a heuristic function @xcite @xcite to define sampling weights for each data source. learned techniques, such as doremi @xcite , use the loss of a model across the data sources to define sampling weights while heuristic functions often use the size of a data source to explicitly define weights @xcite or fit a probability distribution @xcite . the data attributes of toxicity and quality have been used to further refine pretraining sets @xcite . toxicity classifiers @xcite that remove highly toxic examples reduce the number of toxic generations from lms, but also negatively impact the model's other capabilities @xcite . quality classifiers @xcite @xcite which remove documents such as machine generated texts @xcite or hate speech and sexually explicit content (luccioni and viviano, 2021) greatly improve model capabilities. (longpre et al., 2023) extensively investigate the impact that toxicity, quality, and age of data have on model accuracy. ## conclusion we present the first comprehensive study on pretraining set development conducted at the scale of modern day lms and pretraining set sizes. through a series of ablations, we identify helpful methods to use at each step of the pretraining development pipeline. we then analyze most currently available web crawl snapshots across the attribute labels of toxicity, quality, domain, and type of speech to better understand the composition of the most widely used data source in current pretraining corpora. these attribute labels are then shown to provide significant improvement in model abilities when incorporated within data selection and data sampling methods. we hope that the open transmission of this knowledge spurns more rapid advancements in the capabilities of lms.