Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research (2024)

Abstract: Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

## introduction language models are now central to tackling myriad natural language processing tasks, including few-shot learning, summarization, question answering, and more. increasingly, the most powerful language models are built by a few organizations who withhold most model development details @xcite @xcite . in particular, the composition of language model pretraining data is often vaguely described, even in cases where the model itself is released for public use, such as llama 2 @xcite . this hinders understanding of the effects of pretraining corpus composition on model capabilities and limitations, with impacts on scientific progress as well as on the public who interfaces with these models. our aim is to increase participation in scientific research of language models through open corpora: to support broader participation and inquiry in these lines of research, we present data for open language models' appetite (dolma), an open corpus of three trillion tokens designed to support language model pretraining research. we source much of our data from sources similar to those present in past work, including a mix of web text from common crawl, scientific research from semantic scholar, code from github, public domain books, social media posts from reddit, and encyclopedic materials from wikipedia. compared to other publicly-available pretraining corpora, dolma offers a larger pool of tokens at comparable quality while maintaining diverse data composition. in summary, our contributions are two-fold: through this toolkit, practitioners can not only re1 we follow the definition of "token" as a subword obtained using a tokenizer (such as llama's or gpt-neox's), which is distinct from "word", as in a unit of text as defined by the unicode text segmentation standard. open corpora for language model pretraining. we recognize prior efforts to curate, document, and release open corpora to support language model pretraining research. however, limitations in these prior corpora have motivated us to curate a new dataset: while this manuscript was under review, several other open corpora for language modeling have been released, including fineweb @xcite , zyda @xcite , and the datasets used to train llm360 amber @xcite , llm360 k2 (llm360 team, 2024), and map-neo @xcite models. ## data design goals we present the design goals of dolma and discuss how these goals guided our decision-making during data curation. in sharing these, we hope to inform users of dolma's strengths and limitations while also reinforcing practice around such disclosures in dataset curation research (see curation rationales in @xcite and motivation questions in @xcite ). be consistent with prior language model pretraining recipes. by matching data sources and methods used to create other language modeling corpora, to the extent they are known, we enable the broader research community to use our artifacts to study (and scrutinize) language models being developed today, even those developed behind closed doors. in this reproduction effort, we follow established practices to the extent they are known. notably, this also means scoping dolma to english-only text to better leverage known curation practices and maximize generalizability of scientific work on dolma to existing language models. still, there remain myriad data curation decisions for which there is no single clear recipe from prior work, both when best practice isn't known as well as when implementations differ in subtle ways. in such cases, we prioritize decisions that maximize performance of language models trained on dolma over a diverse suite of tasks and datasets (see §4.2). large scale data to train large models. hoffmann et al. @xcite suggested that one can train computeoptimal models by maintaining a fixed ratio between language model size (in parameters) and a minimum number of training tokens. recent works that follow these "scaling laws," such as llama 2, show that there is still room for performance improvement by increasing the number of training tokens. we aim for a sufficiently large corpus-2-3t tokens-to allow further study of the relationship between model and dataset size. ## curating dolma-web in this section, we describe the web subset of dolma, which consists of 2.28t tokens derived from common crawl, foot_2 a collection of over 250 billion pages that were crawled since 2007. common crawl is organized in snapshots, each corresponding to a full crawl over its seed urls; as of feb 2024, there are 97 snapshots. we used 25 snapshots between 2020-05 to 2023-06. ## curating dolma-social in this section, we describe the social media subset of dolma, which consists of 80b tokens derived from reddit data. ## assembling other data sources in this section, we briefly summarize additional highquality sources that were used to derive dolma. more details on collection and processing in datasheet §n. c4 for curated web content similar to data recipes for llama and llama 2, we supplement our web subset with c4 @xcite . we further refine this data by reprocessing it through our full web pipeline (excluding url deduplication) ( §5) which removed additional content, including more low-quality and duplicated texts, and performed pii masking. semantic scholar for academic literature the pes2o dataset (soldaini and lo, 2023) is a collection of approximately 40 million open-access academic papers that have been cleaned, filtered, deduplicated, and formatted for pretraining language models. it is derived from the semantic scholar open research corpus (s2orc; @xcite . as this dataset has been created for language modeling purposes, we use it as-is. project gutenberg for books project gutenberg is a repository of over 70 thousand public domain books. we collected project gutenberg's archive in april 2023. we use english language books, which we filter using the same approach described in §5.1. we deduplicate this dataset based on book title exact match. wikipedia and wikibooks for encyclopedic content this dataset was derived by march 2023 wikimedia dumps. we use the "english" and "simple" editions of wikipedia and wikibooks as base for the encyclopedic subset of dolma. sources were processed using wikiextractor @xcite . we remove any document with 25 or fewer utf-8-segmented words, as we found shorter pages to either be the result of short, templated pages (e.g., pages containing only a few words and an information box) or xml parsing errors. by design, this dataset does not contain duplicated documents. ## training a language model on dolma as a final validation step of the dolma pipeline, we train, evaluate and release a decoder-only, autoregressive language model which we call olmo-1b. we present zero-shot experimental results of olmo-1b on a range of downstream tasks demonstrating comparable quality to other released language models of comparable size.