A uto C lean: LLM s Can Prepare Their Training Corpus (2025)

Abstract: Recent studies highlight the reliance of Large Language Models (LLMs) on high-quality, diverse data for optimal performance. The data sourced from the Internet often aggregated into datasets like the Common Crawl corpus, presents significant quality variability and necessitates extensive cleaning. Moreover, specific domain knowledge is usually presented in HTML, but there is a lack of effective methods to clean them into the training corpus automatically. Traditional cleaning methods involve either labor-intensive human teams that lack scalability or static heuristics that lead to suboptimal outcomes and are unable to be applied to specific target domains. In this paper, inspired by the recent progress in employing LLMs as versatile agents for diverse tasks, we take the initiative to explore the potential of these agents in automating data-cleaning methodologies. By configuring LLMs as an agent team that imitates the human data-cleaning team, we can automatically generate cleaning rules that traditionally require the involvement of data-cleaning experts. These rules are developed using a limited number of data samples and can then be applied broadly to substantial portions of raw data from the same domain. We demonstrate the efficiency and effectiveness of on both pre-train scale corpora such as Common Crawl and specific target websites. Both automatic and human evaluations of the quality of the cleaned content highlight the feasibility of using LLMs to prepare their training corpus.

## introduction the rapid advancement of large language models (llms) has opened pathways toward agi, with capabilities like programming @xcite and instruction-following @xcite . however, these models face a critical bottleneck: the scarcity of high-quality training data. according to scaling laws @xcite , larger models demand more extensive datasets. training data for llms primarily comes from two sources: (1) vast web-scale datasets like common crawl, which contain a huge corpus but also includes substantial noise; and (2) specific domain repositories rich in specialized knowledge. however, both sources lack effective automated methods for extracting clean, high-quality text, particularly from dynamic or noisy web content in html format. existing data cleaning methods, such as cc-net @xcite , pile @xcite , and refinedweb @xcite , rely on fixed pipelines that incorporate deduplication, classification, and filtering. while these approaches have achieved some success, they are constrained by static, heuristic-driven processes, which often struggle to handle diverse or complex data. additionally, these pipelines require an extensive volume of original data to operate effectively. we introduce autoclean, a novel approach that leverages llms as autonomous agents for intelligent and scalable data cleaning. specifically, au-toclean assumes that web pages within the same domain share a consistent structure. therefore, a set of heuristic rules generated by llms can effectively handle all web pages in that domain. by analyzing domain-specific web pages, autoclean generates cleaning rules tailored to each domain and then applies these rules across all pages within it. the entire process comprises several stages, including web page clustering, html processing, text processing, and quality inspection. each step will be explained in detail in the methods section. we conduct comprehensive experiments and analyses to show the effectiveness and efficiency of autoclean. we instantiate autoclean with gpt-3.5 @xcite as the agents and apply the rules generated by these agents to raw data from both common crawl and certain websites. both automatic evaluation and human evaluation demon- inspection inspection inspection raw data samples intermediate result subdomain intermediate intermediate result samples intermediate result subdomain intermediate intermediate result samples intermediate result subdomain intermediate intermediate result web page clustering html process text process html process text process deduplication autoclean pipeline overview strate superior data cleaning performance compared to previous heuristic methods. we provide a demo to show the effectiveness of autoclean. to summarize, our contributions are as follows: ## related work two lines of work are related to this paper: data cleaning methods, and agent automation. ## method the autoclean method starts by generating cleaning rules based on a small sample of web pages. these rules are then applied to clean the entire dataset. importantly, the rules are created from a few random web pages but can be applied to all pages within the same domain. this allows for efficient and cost-effective cleaning of large-scale datasets without requiring agents. below, we outline the stages of rule generation. ## experiments in this section, we present the experiments. we conducted an experiment using autoclean on common crawl and compared it with traditional original html process text process deduplication 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 meta score meta scores at each step meta score figure 7: meta scores for intermediate results after different stages. original autoclean trafilatura cleaning method 0.0 0.2 0.4 0.6 0.8 1.0 meta score 0.489 0.900 0.571 0.211 0.476 0.324 average meta scores of different cleaning results chn eng figure 8: the meta scores on www.gushiwen.cn (chn) and huggingface.co (eng). ## domain specific data acquisition autoclean offers unique advantages in acquiring specific types of data from targeted websites on any scale. unlike traditional methods such as cc-net @xcite , which depend on largescale corpus and rely on selecting segments with the lowest ppl, autoclean can effectively clean a domain at any scale, including smaller datasets where low-ppl segments may lack reliability. for demonstration, we applied autoclean on www.gushiwen.com and huggingface.co. a visual example is shown in figure 9 , where autoclean extracts high-quality paths (green) while removing low-quality paths (red) during the html process stage. additional text cleaning tools-short line deletion (bold strikethrough) and truncated sentence removal (blue underline)-were selected by the programmer agent. identical lines (magenta) were deduplicated, effectively removing navigation bars and web components. figure 11 illustrates the high-quality/low-quality paths generated for www.gushiwen.cn , and figure 10 highlights the cleaning contribution of each stage, showing that the html process handles most junk content, followed by text processing tools. only 14.2% of the original corpus was retained after cleaning. we evaluated the cleaned data using the method from the previous section. as shown in figure 8 , autoclean achieved the highest quality scores on both www.gushiwen.cn and huggingface.co, significantly outperforming trafilatura @xcite , even at the scale of 100 web pages. a demo is provided to demonstrate autoclean on any domain. ## conclusion in this paper, we present autoclean, a framework designed for automatic data cleaning by utilizing llms as agents. autoclean generates a comprehensive set of cleaning rules using agents for each domain, thereby ensuring scalability, flexibility, and effectiveness. future research directions include augmenting the autoclean intelligence level to support more sophisticated data cleaning processes and distilling the capability of llms into smaller models to ensure effectiveness.