S ample M ix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity (2025)

Abstract: Existing pretraining data mixing methods for large language models (LLMs) typically follow a domain-wise methodology, a top-down process that first determines domain weights and then performs uniform data sampling across each domain. However, these approaches neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. Further, uniform sampling within domains ignores fine-grained sample-specific features, potentially leading to suboptimal data distribution. To address these shortcomings, we propose a novel sample-wise data mixture approach based on a bottom-up paradigm. This method performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample, thereby dynamically determining the optimal domain distribution. Comprehensive experiments across multiple downstream tasks and perplexity assessments demonstrate that SampleMix surpasses existing domain-based methods. Meanwhile, SampleMix requires 1.4x to 2.1x fewer training steps to achieve the baselines’ performance, highlighting the substantial potential of SampleMix to optimize pre-training data.

## introduction the mixture proportions of pretraining data, which greatly affect the language model performance, have received increasing attention from researchers and practitioners. in the early years, heuristicbased methods were widely employed to assign domain weights using manually devised rules, such as upsampling high-quality datasets (e.g., wikipedia) multiple times @xcite . afterwards, models like glam @xcite and palm @xcite established mixture weights based on the performance 9.93% 100.00% 100.00% 99.93% 47.03% 67.61% 8.87% 12.45% 100.00% 98.45% 32.90% 57.53% 8.98% 12.63% 99.97% 100.00% 33.03% 57.84% 17.84% 17.78% 99.96% 98.83% 100.00% 52.62% 11.80% 14.59% 99.77% 98.78% 30.03% 100.00% metrics of trained smaller models. more recently, learning-based methods have been proposed, involving the training of small proxy models across domains to generate optimal domain weights @xcite . these existing methods follow a domain-wise methodology, a topdown process that first determines the proportion of each domain and then samples uniformly from the selected domain. despite achieving advancements, these approaches present two key issues: to address these limitations, we propose a novel sample-wise data mixture approach with a bottomup paradigm. instead of defining domain proportions upfront, we first perform global sampling across the dataset based on sample quality and diversity, dynamically determining domain distributions. this allows for more precise control over the overall quality and diversity of the dataset. to implement this, we individually assess the quality and diversity of each sample and assign corresponding sampling weights based on these evaluations. given a target token budget, we then sample each example according to its weight to construct the optimal training dataset. also, our approach offers the additional advantage of dynamically adapting to varying token budgets, enabling the determination of optimal data proportions for each specific budget. in contrast, the vast majority of existing works rely on static data proportions, which do not adjust to different token budget constraints. the contributions of this paper are: ## related work we have covered research on data mixture in § 1, related work related to our technical designs is mainly introduced in the following. data quality heuristic rules, such as thresholds on word repetitions and perplexity, are commonly used to filter out low-quality data @xcite @xcite . earlier model-based methods employ binary classifiers to distinguish high-quality from low-quality data @xcite . recent approaches incorporated more sophisticated models. @xcite investigated four qualities-writing style, re-quired expertise, facts & trivia, and educational value respectively. however, most methods rely on relatively coarse criteria and do not fully leverage the multi-dimensional property of data quality. diversity traditional deduplication methods struggle to capture more complex semantic similarities @xcite . to better handle semantic redundancy, @xcite applies k-means clustering in the embedding space to identify and remove redundant data. @xcite builds on this approach by using semdedup as a preprocessing step before applying ssl prototypes @xcite . @xcite balances common and rare samples and ensures diversity by data clustering. ## conclusion we have presented samplemix, a sample-wise pre-training data mixing strategy by coordinating data quality and diversity. extensive experiments demonstrate that samplemix outperforms existing domain-wise methods, achieving comparable accuracy with 1.9x fewer training steps. in the future, we are interested in incorporating automatic evaluation metrics derived from the model's perspective to complement the current manually designed measures, and exploring code data mixing.