C raw4 LLM : Efficient Web Crawling for LLM Pretraining (2025)

Abstract: Web crawl is a main source of large language models’ (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper presents Craw4LLM, an efficient web crawling method that explores the web graph based on the preference of LLM pretraining. Specifically, it leverages the influence of a webpage in LLM pretraining as the priority score of the web crawler’s scheduler, replacing the standard graph-connectivity-based priority. Our experiments on a web graph containing 900 million webpages from a commercial search engine’s index demonstrate the efficiency of Craw4LLM in obtaining high-quality pretraining data. With just 21% URLs crawled, LLMs pretrained on Craw4LLM data reach the same downstream performances of previous crawls, significantly reducing the crawling waste and alleviating the burdens on websites. Our code is publicly available at https://github.com/cxcscmu/Craw4LLM.

## introduction massive in size and diverse in topics, web data usually serve as the primary source of pretraining data for large language models (llms), providing an extensive and heterogeneous corpus that captures a wide spectrum of human knowledge and real-world information @xcite @xcite . pretraining datasets are typically built from large-scale web crawls such as common crawl @xcite , which may contain tbs of data spanning billions of webpages @xcite . despite their vast scale, most of the data collected from web crawls are not used in the pretraining of llms. existing work often discards over 90% of the raw data collected from the web @xcite @xcite ., * work done while visiting carnegie mellon university. 2024), highlighting the inefficiency of current web crawlers in collecting llm pretraining data. common web crawlers like common crawl prioritize pages based on graph connectivity metrics like pagerank @xcite or harmonic centrality @xcite , which favor documents with a high number of inlinks (indegree) @xcite rather than those most relevant for pretraining. this misalignment not only leads to waste in computational resources during excessive data processing for llm developers, but also incentivizes overcrawling, which burdens website operators with redundant traffic and increases ethical and legal risks related to fair use of data and copyright @xcite . to bridge this gap, we propose web crawling for llm pretraining (craw4llm). instead of relying on traditional graph-connectivity-based signals, craw4llm improves crawling efficiency by prioritizing webpages based on their influence on llm pretraining. specifically, during each crawling iteration, all newly discovered documents are scored with a pretraining influence scorer derived from data-filtering pipelines for pretraining @xcite , and docu- figure 2 : correlations between pretraining influence scores from dclm fasttext @xcite and pagerank to indegrees, on randomly sampled clueweb22-b documents @xcite . spearman correlation coefficients are reported in parentheses. ## methodology in this section, we introduce web data crawling for llm pretraining (craw4llm), an efficient crawling method that integrates llm pretraining preference into the crawler. the algorithm of craw4llm is presented in algorithm 1. similar to traditional crawlers @xcite , craw4llm starts with a set of seed urls. for each unvisited outlink of them, craw4llm assigns a score using a pretraining-oriented url scoring function score_url(•; m), where m is a pretraining influence scorer which rates a docu-algorithm 1 craw4llm algorithm input: seed urls u seed , number of pages to be crawled n , number of pages to be crawled in each iteration n, pretraining influence scorer m(•) output: crawled page set p 1: initialize url and score priority queue q ← ∅ 2: initialize crawled page set p ← ∅ 3: initialize visited url set v ← u seed 4: uc ← u seed 5: while |p| ≤ n do 6: pc ← fetchpages(uc) 7: merge pc into p 8: uout ← extracturls(pc) 9: if v / ∈ v then 11: enqueue(q, v, score_url(v; m)) 12: add(v, v) 13: uc ← dequeue(q, n) 16: end while 17: return p ment's influence for pretraining. m can be derived from data classification models for pretraining data, which have been used to decide whether a document should be retained in or filtered out from the raw dataset @xcite . formally, given a pretraining influence scorer m, the score s of a url u is calculated as in contrast, traditional crawlers typically rely on graph connectivity metrics, such as pagerank @xcite and harmonic centrality @xcite , which basically assign higher priority to pages with higher indegrees @xcite . as shown in figure 2 (a), the indegrees of webpages exhibit a poor correlation with the scores assigned by the dclm fasttext classifier, a pretraining influence scorer for identifying high-quality pretraining data @xcite . this confirms that graph connectivity-based crawlers are inefficient in crawling pretraining data. by incorporating a pretraining influence scorer, craw4llm traverses the web graph in a way @xcite (down) is used as the pretraining influence scorer for craw4llm, and to select documents for oracle and crawl-then-select runs (section 3). all models are pretrained on 1× data (20m documents, 32.9b tokens). the evaluation metric is centered accuracy (0 = random guess) @xcite . best/2nd best in the last two groups are bolded/underlined. see appendix d for detailed results. ## experimental methodology in this section, we introduce our experimental setup, with details on the crawler implementation and llm training provided in appendix a and b. ## evaluation results in this section, we first present the overall performance of craw4llm (section 4.1), followed by further analysis (section 4.2). ## conclusion this paper presents craw4llm, a step toward more efficient and responsible web crawling for llm pretraining. by prioritizing documents based on the pretraining needs, our method improves crawling efficiency and reduces unnecessary crawling, easing the burden on web hosts. while fair use of web data remains a critical challenge, we hope that craw4llm can help mitigate these concerns and promote more compliant and sustainable practices in obtaining pretraining data for llms.