Scaling Parameter-Constrained Language Models with Quality Data (2024)

Abstract: Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization.In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation – effective training tokens – which we posit to be a critical determinant of performance for parameter-constrained language models.Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text:(i) text diversity and (ii) syntheticity as measured by a teacher model.We pretrained over 200 models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores.We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyze it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.

## introduction recent advancements in language model (lm) development have been significantly influenced by the exploration of scaling laws, which articulate the relationship between training loss, dataset size, and the number of model parameters @xcite @xcite . these scaling laws have been instrumental in predicting the computational resources necessary for training increasingly large models and have provided a framework for understanding how model performance scales with data and parameters @xcite . however, these laws primarily focus on the quantity of data and model size, often underestimating the critical role of data quality in model generalization. in this work, we challenge the prevailing focus 1 on merely increasing data volume and model size by emphasizing the importance of data quality, particularly in scenarios constrained by the number of model parameters. we argue that for sub-billion parameter models, the quality of data-or what we term as effective training tokens -plays a more decisive role in model performance than previously recognized. this perspective shifts the paradigm from a quantity-centric view to a quality-centric approach in the development of language models. further, we provide qualitative measures of standard data refinement techniques including data sampling @xcite @xcite ) and text synthesis @xcite , applied to a pretraining corpus such as re-finedweb @xcite . this helps to formulate the relationship between the diversity and syntheticity of pretraining data in order to compute the number of effective training tokens, which evaluate the impact of data quality in terms of model size and the token number. further, we conduct extensive experiments across eight different benchmarks to evaluate the impact of data refinement techniques which allow us to significantly outperform models trained on randomly selected data samples, across a spectrum of model sizes ranging from 25 million to 1.5 billion parameters. by integrating the notion of effective token size into the scaling law formulation, we extend the existing scaling law formulation to better capture the nuances of data quality. our results underscore the pivotal role of high-quality data in training efficient and powerful language models, particularly in parameter-constrained settings. the contributions of this paper are as follows: our finding underscores the potential of data quality, rather than sheer quantity, to enhance model performance. ## background chinchilla scaling law @xcite provides a predictive framework for estimating model training loss, considering the number of training tokens and model parameters. initially designed to identify optimal compute settings for extensive pretraining-a costly and time-consuming endeavor-these laws are crucial for optimizing computational resources. recent studies by abbas et al. (2023); liu et al. (2024); goyal et al. (2024) on the other hand, data refinement can be categorized into non-transformative and transformative types @xcite . non-transformative refinements involve selective curation of data samples without altering their core characteristics. in contrast, transformative refinements generate new text data, rearranging and introducing new tokens, thus impacting training token distributions and data quality. this significantly affects the effective number of training tokens used in model training. in non-transformative refinements, data deduplication is essential for preventing model generalization issues by removing duplicate documents @xcite @xcite . this process not only reduces the number of training tokens but also enhances the quality and effectiveness of the remaining tokens, improving model performance @xcite . data selection, another non-transformative method, involves choosing an optimal data subset from a larger corpus for model training. both approaches aim to enhance model performance, reduce computational costs, and maintain evaluation metric integrity @xcite . transformative refinements, such as synthetic data generation through instructional prompts, are becoming popular @xcite @xcite . this approach creates new data to fill existing dataset gaps or introduce new learning scenarios. integrating synthetic data into large-scale pretraining has significantly improved model robustness and generalization @xcite @xcite . synthetic data generation allows for controlled training dataset expansion, ensuring exposure to diverse inputs and scenarios @xcite . generally, data refinements are crucial in shaping the training landscapes of modern machine learning models, directly influencing training token distribution and quality, thereby enhancing training efficiency and effectiveness in line with scaling laws @xcite . ## formulating data quality here we adopt two popular metrics to measuring text quality that are easy to compute on large-scale pretraining data, which is an important consideration when measuring data quality of pretraining sets. diversity: following @xcite , we utilize the compression ratio, which has been demonstrated to be effective for large-scale pretraining datasets and correlates well with other diversity metrics (figure 4 ). past metrics generally quantify the number of repeated substrings across outputs. among these, the token-type ratio is calculated by dividing the count of unique tokens by the total number of tokens in a text. to capture the lexical dynamics across varying text lengths, the moving average token type ratios (mattrs) were introduced, providing a robust measure that is insensitive to text length @xcite . this metric focuses on the frequency of individual word repetition within text segments and does not account for longer repeated sequences. to address longer sequences, the concept of token-type ratio has been expanded through the introduction of n-gram diversity, as explored in high compression ratios suggest greater redundancy, indicating lower diversity within the text data. therefore, diversity is defined as dr(d), where higher means more diverse text. syntheticity: we estimate the syntheticity of data points in our dataset using the perplexity metric, which is calculated with a teacher-model, i.e. llama-2 7b chat @xcite foot_1 . this model choice is strategic because teacher models are known for their robust performance across a variety of benchmarks and their alignment with safety choices, making them reliable for general evaluations without needing to tailor them to specific downstream tasks. perplexity, in this context, measures how well the teacher model predicts a sequence of subword tokens, with lower values indicating higher predictability and, by extension, higher syntheticity. a low perplexity score suggests that the data point is well-represented by the model's learned patterns, which could indirectly indicate that it is more relevant or useful for similar tasks or applications. hence syntheticity is inversely proportional to perplexity and is then defined as follows: the formula above calculates the inverse of the exponential of the negative average log-likelihood of predicting each subword token in the document d, given all previous tokens. this quantifies how expected the tokens are, given the model's current knowledge state, thus providing a direct measure of how typical or atypical the sequence is within the context of the teacher model. ## scaling law with data quality we propose to modify the third approach of the chinchilla scaling law @xcite which originally models the losses in training large language models with the functional form e + a n α + b d β with the constants: (@xmath0.89, @xmath1.3, α = 0.345, @xmath2, β = 0.452) in this work, we model the zero-shot accuracy on common sense reasoning as we postulate that the score provides an indication on how much reasoning ability a given data d could possibly instill. to incorporate data quality into this framework, we propose to use a quality term q to provide a quality-adjusted number of training tokens (d q ), combining eq. 1 and eq. 2: this revision integrates the quality-adjusted number of training tokens (d q ) into the accuracy function, allowing for a more nuanced understanding of how data quality impacts model training and performance. ## data refinement: a case study we explore two prevalent data refinement techniques aimed at enhancing data quality: data selection and data synthesis. these methods have become standard practices in the preparation of pretraining datasets, significantly influencing text diversity and syntheticity and downstream performance as shown in various studies @xcite . to put them in context, we present a comparative analysis in figure 2 , which displays the relationship between effective token counts d q and the total number of tokens d. it clearly demonstrates that data synthesis has a more substantial impact on increasing the effective token count compared to data selection and the use of original datasets. this underscores the value of synthesis in optimizing data quality for model training. ## experimental setup network and training details. for all experiments, we pretrain the decoder-only transformer using causal language modeling objectives on selected datasets, where model weights were randomly initialized. we evaluated with the language models of sizes {25, 50, 75, 125, 350, 500}m and 1.5b parameters which allowed us to explore how the accuracy values are the reference values. right: impact of scaling factor q on both diversity and syntheticity. interestingly, we found that diversity needs to be reduced while syntheticity needs to be increased for scaling factor to go up, which can then improve overall accuracy. we include the constant values in ## discussions by over 200 training runs, we re-estimate all the constants which we show in in particular, we discuss the scaling factor q and how it can be applied to pretraining scenarios. ## conclusion and future works in this paper, we revisited traditional scaling laws in language modeling that often overlook the critical impact of data quality on model generalization. we introduced the concept of effective training tokens, emphasizing its significance in enhancing model performance, particularly for models with constrained parameters, in order to offer a more precise understanding of data quality's role in model scaling. our findings highlight the pivotal role of data quality and pave the way for developing more efficient and compact language models suitable for on-device applications.