S caling F ilter: Assessing Data Quality through Inverse Utilization of Scaling Laws (2024)

Abstract: High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity. In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process. An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws. Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks. To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations. Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.

## introduction the success of large language models (llms) is significantly influenced by the quality and quantity of the pre-training corpus. researchers have developed various data curation pipelines to enhance dataset quality, focusing on raw web crawling, text extraction, repetition and toxic content removal, and, notably, quality filtering @xcite @xcite . quality filters aim to extract high-quality data from a noisy raw corpus, thereby improving the figure 1 : in scalingfilter, we assess the quality of text documents by their scaling characteristics with language models in different sizes. language model's performance without increasing training costs. existing filters are broadly classified into two categories: reference-dependent and reference-free approaches. reference-dependent methods, such as binary classification @xcite @xcite and dsir @xcite , filter out low-quality data by comparing it with high-quality seed datasets. while effective, these methods inevitably introduce biases present in the reference data, such as specific writing styles or topics, thereby limiting the diversity and representativeness of training corpus @xcite . in contrast, reference-free methods, such as perplexity gating @xcite , assess data quality using predefined metrics like perplexity scores from pre-trained models. these methods mitigate the biases introduced by reference datasets but encounter challenges due to the indirect relationship between absolute perplexity and document quality. this indirect relationship inadvertently favors data with simple and repetitive content. although such content is easier for models to predict, it contributes minimally to learning diversity and complexity @xcite . to address these issues, we introduce a simple yet effective quality filtering approach named scal-ingfilter, which inversely leverages recent scaling laws in generative modeling to assess data quality. the core idea is to analyze the perplexity differences between two pre-trained models on the same data and assess the data quality based on these dif-ferences. we find a positive correlation between data quality and perplexity differences by inversely deriving chinchilla scaling law @xcite . in other words, given a pair of pre-trained models of different sizes, documents with higher perplexity differences indicate higher quality. scalingfilter involves utilizing the difference between two separate models for data quality assessment, effectively addressing the bias issue induced by relying on a single model trained on the reference data. this approach also mitigates the problem of selecting simple and repetitive texts that arise from overfitting to the perplexity metric, thereby enhancing data diversity and complexity. furthermore, scalingfilter offers a theoretical analysis for using perplexity differences as a quality indicator for data filtering by inversely deriving model scaling laws. our experiments demonstrate that scalingfilter is superior to existing methods in improving filtered data quality while preserving data diversity. specifically, we employ a pair of meta-models with sizes of 124m and 774m parameters to assess the perplexity difference for each document in the raw dataset, and then select the high-quality ones using a top-k strategy. we train a 1.3b model from scratch using filtered high-quality data. we then evaluate its zero-shot performance on downstream tasks and assess the semantic diversity of the filtered dataset. the results demonstrate that scalingfilter outperforms the unfiltered baseline and previous stateof-the-art (sota) quality filtering methods. specifically, compared to the unfiltered baseline, scal-ingfilter achieves a +3.09% improvement in downstream accuracy and a +2.23 increase in semantic diversity. when compared with perplexity gating @xcite , scal-ingfilter achieves a +1.12% improvement in performance and a +4.7 increase in semantic diversity. in summary, the contributions of this work are threefold: ## methodology existing quality filtering methods depend on either a reference high-quality dataset or the absolute perplexity scores of documents from a single language model, which can introduce potential bias or result in inferior performance. in this section, we will elaborate on the principles of scalingfilter through mathematical derivation. the core concept of scalingfilter lies in estimating the quality of data samples by inversely applying the scaling law. specifically, the scaling law reveals a power-law decrease in loss with increasing model size or data size @xcite @xcite . ultimately, the scaling law yields the optimal model/data scaling-up allocation strategy. in other words, under the same computational budget (tflops), it determines the optimal ratio of model size to the number of training tokens to achieve the lowest loss, represented by a model scaling exponent a and a data scaling exponent b. extensive experiments comparing multiple datasets with known quality differences revealed that high-quality data increases the model scaling exponent a @xcite . specifically, the experiments compared the early and final versions of in-house data together with openwebtext2 @xcite , revealing that the final version of openwebtext2 results in the highest a, while the early version with the poorest quality leads to the lowest a. intuitively, a higher value of a accelerates the rate at which the loss decreases as the model parameters increase. this positive relationship will be demonstrated later. such an observation suggests that high-quality data enhances logical clarity and decreases predictive difficulty after adequate training. consequently, scaling up the model size becomes more beneficial when the compute budget is increased @xcite @xcite . low quality data high quality data by inversely applying this principle, scalingfilter estimates data quality based on the rate of loss decrease in models with a fixed parameter difference, thereby separating high-quality data from the raw dataset. to proceed, we will first define the quality factor, which is the magnitude of loss reduction. then, starting from the formula of the scaling law, we will demonstrate the positive correlation between the quality factor and the model scaling exponent a. furthermore, based on the positive correlation between a and data quality observed in @xcite , we can ultimately prove the positive correlation between the quality factor and data quality. quality factor. we begin with defining the quality factor, which we will later demonstrate to have a positive correlation with data quality. we denote the smaller meta-model as p and the larger one as q. both meta-models share the same architecture and are trained on the same dataset, with the only difference being the parameter counts: n p for p and n q for q, with n @xmath0. let x i be a given text sample, and denote the quality factor of this sample as d i . then, we have: quality factor is positively correlated with data quality. next, we will introduce the expression of the scaling law @xcite @xcite and transform it into a form involving the model scaling exponent a which, as we introduced in the overview, is known to have a positive relationship with data quality @xcite . given the number of model parameters n and training tokens d, the expected model loss l is formulated @xcite as: consider setting η . = α + β, then l can be presented as: we focus on the relationship between expected loss l and model scaling exponent a as well as model size n , and thus denote l as l(a, n ). it's obvious that l decreases as n increases. we further prove in appendix a.2.1 that at a specific n 0 , the slope of the tangent to the l -n curve decreases as a increases (i.e., the larger the a, the steeper the tangent, as illustrated in figure 2a that l 2 is steeper than l 1 ). due to this monotonic relationship, we can infer the value of a from the slope of the tangent: for a given n 0 , a steeper tangent (greater absolute value of the slope) indicates a larger a, that is: furthermore, we prove in appendix a.2.2 that the above conclusion can be extended from the tangent slope at a given n 0 to the slope of the secant line for any given ∆n (i.e. k i in figure 2a ). letting ∆@xmath1, the slope of the secant line is always negative, and a is positively correlated with the negative slope of the secant line. since the quality factor d is also positively correlated with the negative slope of the secant line, it follows that d is positively correlated with a: this conclusion aligns with our practical comparative tests. as shown in figure 2b , we calculated the average loss of gpt-2 models of different sizes on several datasets with recognized quality levels: wikipedia, openwebtext, and books3 represent high-quality data, while unfiltered commoncrawl represents low-quality data, based on a random sample of 10k documents from each dataset. the results align closely with the theoretical estimates shown in figure 2a , where the high-quality data shows a steeper secant (k 2 > k 1 ) compared to low-quality data. it's worth noting that a single case might deviate from the training data distribution of the meta-models, leading to higher absolute perplexity for various model sizes. thus, relying solely on single perplexity as a quality criterion can result in misjudgments. however, high-quality data follows a law where perplexity decreases more with an increase in model parameters, indicating a greater perplexity difference (i.e., quality factor). selecting high-quality data with quality factor. we have demonstrated that the quality factor can directly characterize data quality above, so it's straightforward to directly use it to select highquality data from a noisy unfiltered dataset. we call this simple yet effective method as scalingfilter, as illustrated in figure 1 . consider a unfiltered set of documents s, containing both high and lowquality documents. for each sample x @xmath2, we calculate the quality factor d i for it. as derived previously, samples with higher d i are of better quality. the top-k samples are then selected based on the desired cleaning rate to form the resulting dataset. ## experiments in this section, we will demonstrate the effectiveness of scalingfilter through extensive experiments. specifically, language models trained on data filtered by scalingfilter consistently achieved superior performance across various downstream tasks, compared to the unfiltered baseline and other common quality filtering approaches, highlighting the higher quality of the data. furthermore, by measuring the semantic diversity of the filtered dataset, we found that scalingfilter effectively preserved the diversity present in the original dataset. ## conclusion we have presented scalingfilter for data quality filtering in a reference-free manner. starting from the scaling law, we demonstrate that the perplexity difference across agnate models of different sizes (i.e. meta-models) correlates with data quality positively. we select samples with higher perplexity difference (i.e. quality factors) to form the pretraining dataset. by eliminating the bias brought by reference datasets, our method achieves better downstream performance over several strong baselines while preserving data diversity.