FinerWeb-10BT : Refining Web Data with LLM -Based Line-Level Filtering (2025)

Abstract: Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.

## introduction in recent years, the size of large language models (llms) and their training datasets has expanded tremendously, as companies and researchers strive to build increasingly capable models. in fact, if current trends continue, we may run out of humangenerated text data within a decade @xcite . this has led to a growing interest in data quality over quantity: rather than only expanding datasets, researchers are exploring ways * these authors contributed equally. to achieve high performance with smaller, cleaner datasets. recent studies suggest that removing low-quality text from training data can improve model performance, even when the overall size of the dataset is reduced @xcite . furthermore, training state-of-the-art (sota) language models requires significant computational resources, which are expensive and, depending on the power source, can contribute to climate change. for example, the carbon emissions from training gpt-3 have been estimated at 552 tco 2 e @xcite , while meta reports that training the 405 billion parameter llama 3.1 emitted 8,930 tco 2 e (meta-llama, 2024). smaller, but higher quality datasets will speed up training and, thus, high-quality data are necessary to train not only better models but also greener ones. while several publicly available datasets are used for training llms, many recent datasets are still cleaned using simple heuristic filters, which often leave substantial amounts of low-quality text while potentially discarding clean text. machinelearning techniques offer a promising alternative, as they enable models to identify patterns related to data quality. however, labeling data to train such models is a tedious and time-consuming process. in this paper, we address these issues by investigating the following research questions (rqs): rq1: how well can an llm identify low-quality content missed by heuristic filters? rq2: does llm-based quality filtering of training datasets improve model performance? to examine these questions, we analyze fineweb, a dataset that claims to provide "the finest text data at scale" @xcite . using gpt-4o mini (openai, 2024a), we label a 20,000-document sample from fineweb, classifying each line as either clean or belonging to one of several low-quality categories, such as copyright notice, programming code, or formatting elements. instead of defining a label taxonomy ourselves, we allow the model to generate its own labels as needed, resulting in 547 unique low-quality labels. after refining these labels, we group them into nine broader categories for easier classification. next, we train a deberta-v3 @xcite classifier using the labeled data to scale the filtering process. this classifier allows us to automatically detect low-quality content in a larger 10b-token sample of fineweb. finally, we evaluate the impact of llm-based filtering by training gpt-2 models @xcite on both the filtered and unfiltered datasets. we release our quality-annotated dataset, finerweb-10bt, available at https://huggingface.co/datasets/turkunlp/finerweb10bt. the code to replicate our experiments is also provided at https://github.com/turkunlp/finerweb-10bt . ## background a recent survey by @xcite discusses the many steps involved in selecting data for training llms, including language filtering, deduplication, removal of toxic or explicit content, and heuristic-based data quality filtering. our focus here is on the latter two-data filtering and heuristic approaches-using an llm-driven approach to refine data quality more precisely. as @xcite note, there is no universal standard for "high-quality" data. in this work, we define it as human-written, continuous english text from the main content of a website, reflecting natural language use across diverse contexts and domains. examples include core text from interviews, forum posts, news articles, blogs, and recipes. in contrast, low-quality content includes recurring elements like navigational menus, copyright notices, programming code, and metadata. given that llms require vast amounts of text data for training, the internet has become a primary source for these data. since 2008, common-crawl has collected a corpus of approximately 10 petabytes of web content @xcite . despite its size, commoncrawl is neither a complete nor fully representative sample of the internet, but it serves as a foundational source for building refined datasets used in llm training. here, we focus on three major datasets sourced from commoncrawl: c4 @xcite , refinedweb @xcite , and fineweb. these datasets use different preprocessing techniques to filter out unwanted material, each with its strengths and weaknesses. we discuss these datasets because their preprocessing methods are well-documented, which allows us to make meaningful comparisons. all three datasets extract plaintext from html documents. c4 uses the wet files provided by commoncrawl, which come with pre-extracted plaintext, whereas refinedweb and fineweb use trafilatura 1 to extract text directly from html. although trafilatura and similar tools remove much of the unwanted noise, further preprocessing is often required. for instance, @xcite note that "many documents remain interlaced with undesirable lines" despite using trafilatura. deduplication and language filtering are also important aspects of document cleaning but we do not focus on them in this paper, as they are specialized techniques not directly related to linelevel text quality. existing filtering methods can be grouped into three levels based on their precision: document level, line level, and character level. by far the most common method is document level filtering, which removes entire documents based on simple rules. examples include filtering documents with phrases like "lorem ipsum", documents with fewer than three sentences, or documents with excessive repetition. line level filtering targets specific lines within documents, removing lines that contain terms like "javascript", consist solely of numbers, or fall below a certain length threshold. character level filtering is less common and is only applied in one of the three datasets: in c4, citation markers commonly found in wikipedia, such as "[1]" and "[citation needed]", are removed. document level heuristic filtering is efficient for quickly removing large volumes of low-quality data, but it can result in the loss of substantial high-quality text. in contrast, line and character level filtering provide more precision by targeting specific content but they require significantly more computational resources at scale. simple heuristics, such as removing lines that contain the word "javascript" can be hit or miss, sometimes discarding valuable data along with the low-quality content. given the vast size of datasets like com-moncrawl, creating a simple filtering system that only removes undesirable content without impacting valuable data is nearly impossible. the filters that are used are also often dataset and language specific. for example, fineweb applies a heuristic that removes documents where "the fraction of lines shorter than 30 characters @xmath0.67" @xcite , but this threshold was determined through extensive manual testing and is specific to that dataset. an ideal quality filter would work across languages and datasets, avoiding trial-and-error by focusing on actual text quality rather than proxies like line length or keywords. it should also be efficient, removing only low-quality content while keeping valuable data intact. llms bring us closer to this goal: rather than using heuristics, they assess text quality directly, enabling granular filtering, even within mostly clean documents. since llms are effective at producing fluent and readable text, they are likely well suited to identifying high-quality text across different languages and datasets. however, it should be noted that while sota llms are fluent in english and other high-resource languages, their performance in low-resource languages is consistently worse @xcite . in this study, we only analyze english documents, and care should be taken before generalizing the results to other languages or multilingual datasets. the use of llms for quality filtering is a relatively new approach, and best practices are still emerging. for instance, @xcite utilize llama 2 to assess the quality of web documents for training llama 3, but details of their methodology are vague. the recent trend of withholding full training datasets for sota models has made it difficult to understand the extent to which llms are currently used in data preprocessing @xcite . other efforts, such as those by @xcite , involve ranking documents based on quality using gpt-3.5, evaluating factors such as style, educational value, and factuality. similarly, llama 3 was used to create the finewededu dataset by evaluating educational content quality, and gunasekar et al. (2023) employ gpt-4 to annotate code datasets based on educational value. our approach differs from prior work by focusing on general-purpose data quality improvements rather than curating specialized datasets. we aim to broadly enhance training data quality through llm-driven filtering that removes lowquality lines with minimal manual intervention. this allows us to assess how automated filtering can improve training data and, ultimately, model performance in foundation model training. ## methods our data source is fineweb @xcite , a 15-trillion-token collection of english text sourced from commoncrawl and preprocessed with standard heuristics. the preprocessing includes steps such as length thresholds, string matching, language and url filtering, and deduplication. despite these measures, the authors of fineweb acknowledge that the dataset could benefit from further refinement. for more details on the preprocessing steps, see the original paper @xcite . in our study, we use a 10b-token ( 15 million documents) sample from fineweb, fineweb-10bt foot_1 . our preprocessing pipeline consists of several steps. first, we use gpt-4o mini (openai, 2024a) to label a sample of 20,000 documents from fineweb at the line level. the model is tasked with generating descriptive labels for each line, categorizing them as either high-quality (clean) or into low-quality categories. this labeling process is data-driven, allowing the model to create a dynamic labeling scheme rather than relying on predefined categories. previous research has shown that llms can be used to annotate data and create label taxonomies @xcite . next, we use openai's o1-preview model (openai, 2024c) to group the numerous labels generated by gpt-4o mini into a smaller, more manageable set. this forms the basis of a classification system, which we use to train a small encoder-based classifier. this classifier scales the labeling process by assigning quality scores throughout the fineweb-10bt dataset, enabling line-level filtering of low-quality content. to evaluate our filtering, we train gpt-2 models @xcite on both the cleaned and original versions of fineweb-10bt. we compare model performances using the hellaswag benchmark @xcite , a widely used test for commonsense reasoning in language models. this allows us to assess whether the filtering improves training data quality and model performance. given the complexity of internet text data (laippala et al., 2023), defining low-quality categories in advance is challenging. our data-driven approach, by contrast, allows the llm to dynamically create labels based on the content it encounters, rather than relying on fixed categories. we believe this approach enables a more flexible and detailed analysis of low-quality content in fineweb compared to rule-based methods or predefined categorizations. ## discussion the labels generated by gpt-4o mini reveal both the quantity and types of low-quality lines that remain in fineweb. the largest categories include lines with grammatical errors, poor formatting, and incomplete sentences, along with recurring items like time stamps, legal jargon, and promotional content. while these elements do not necessarily reduce dataset quality (a good language model should recognize items like copyright notices or phone numbers), our evaluation shows that reducing their prevalence improves both accuracy and training efficiency. these findings suggest that more precise control over the types and proportions of low-quality data included could further benefit model performance. even when simplified to binary classification, our llm-driven approach clearly outperforms heuristic methods in enhancing dataset quality. specifically, our evaluation on gpt-2 using hellaswag shows that with less but cleaner data, the model achieves comparable or even slightly better accuracy. while gpt-2 is small relative to sota models, our results provide strong evidence that llm-based data filtering can reduce training time and save energy. although we tested our method on a small, english-only dataset, this datadriven approach to quality filtering is easily adaptable to other datasets and languages, although lowresource language may suffer from worse llm performance. using an llm as a judge of text quality introduces some bias, as the model's training data and design choices influence the resulting labels. for example, mature sota llms have strong in-built safety features that prevent them from generating harmful or offensive content. in our case, we observe that gpt-4o mini sometimes labels mild expletives, such as "shut up", as toxic, reflecting an overly sensitive filter for offensive language. as described in sections 4.1 and 4.2 we made some manual adjustments to the llm labeling to account for such biases. also, the line between lowquality and high-quality is naturally vague, which introduces noise into the data. in future work, we plan to experiment with different models and adjust our prompts to further improve this filtering approach. ## conclusion in this paper, we propose a novel approach to improving the quality of large-scale language model training datasets through fine-grained, line-level filtering with large language models (llms). we first used gpt-4o mini to label a sample from the fineweb dataset, generating detailed labels that captured low-quality content often overlooked by heuristic filters, addressing our first research question (rq1). these labels were grouped into broader categories using openai's o1-preview model, followed by training a deberta-v3 classifier to scale the filtering across fineweb-10bt. our experiments demonstrate that this llmdriven filtering pipeline improves model performance (addressing rq2), as gpt-2 models trained on the filtered dataset achieved higher hellaswag accuracy with up to 25% less data than those trained on the original fineweb-10bt dataset. these findings suggest that traditional heuristic filters may not be sufficient and that more sophisticated data preprocessing methods are necessary, especially as we face challenges like data scarcity and environmental concerns. our approach con-tributes to the emerging field of llm-based data preprocessing, offering a promising avenue for improving training efficiency and model performance. in future work, we plan to refine our pipeline by broadening the labeling scheme to provide a more comprehensive description of document contents. we will also experiment with more nuanced filtering approaches, moving beyond simple scorebased thresholds, and compare against baselines such as random data reduction to further validate our filtering method. we also plan to test llamastyle models and other architectures to see how our findings scale to newer llms. further evaluations and statistical testing will help strengthen the reliability of our results. finally, we plan to extend our method to other datasets and languages.