Preference Curriculum: LLM s Should Always Be Pretrained on Their Preferred Data (2025)

Abstract: Large language models (LLMs) generally utilize a consistent data distribution throughout the pretraining process. However, as the model’s capability improves, it is intuitive that its data preferences dynamically change, indicating the need for pretraining with different data at various training stages. To achieve it, we propose the Perplexity Difference (PD) based Preference Curriculum learning (PDPC) framework, which always perceives and uses the data preferred by LLMs to train and boost them. First, we introduce the PD metric to quantify the difference in how challenging a sample is for weak versus strong models. Samples with high PD are more challenging for weak models to learn and are more suitable to be arranged in the later stage of pretraining. Second, we propose the preference function to approximate and predict the data preference of the LLM at any training step, so as to complete the arrangement of the dataset offline and ensure continuous training without interruption. Experimental results on 1.3B and 3B models demonstrate that PDPC significantly surpasses baselines. Notably, the 3B model trained on 1T tokens achieves an increased average accuracy of over 8.1% across MMLU and CMMLU.

## introduction large language models (llms) have shown impressive performance on various tasks after being pretrained on vast amounts of data @xcite @xcite . as llms undergo extensive pretraining, their capabilities steadily improve, which influences their performance and data preferences @xcite . existing methods of uniformly sampling data throughout the pretraining process are suboptimal because they overlook the model's evolving data preferences figure 1 : pd-based preference curriculum framework. @xcite @xcite . recently, some research has shifted the focus to data influence on model capability during pretraining @xcite @xcite . a typical method named mates considers the changing data influence on models but requires interrupting the training process to select more preferred data based on the model's current state, which disrupts training continuity and stability @xcite . a similar issue arises in jest @xcite . in this paper, we introduce perplexity @xcite difference (pd) to quantify the difference in how challenging a sample is for llms' final and early checkpoints and use pd to gain a deeper understanding of model characteristics. we further propose a novel pd-based preference curriculum learning framework, pdpc, as shown in figure 1 . it perceives llms' data preference at any training step and uses the preferred data to continuously pretrain llms without interruption, thereby boosting their performance. to this end, pdpc needs to solve three major challenges: tinuity and stability in pretraining, we propose an offline processing method to approximate the ideal dynamic preference adjustment. first, we calculate pd for all samples offline using both the fully trained checkpoint and an early checkpoint of the experimental model. then, we develop a preference function to predict the model's preference for data with specific pd characteristics, enabling the pre-organization of data according to the model's preferences at various training stages. because the data is fully arranged offline, the training of the experimental model proceeds without interruption. calculating pd for the entire dataset is prohibitively expensive. typically, the sizes of experimental models and the pretraining dataset are quite large. training an experimental model using the entire dataset in a random sampling setting is very costly. moreover, calculating pd values offline necessitates inferring the entire dataset twice-using both early and final checkpoints-which is extremely costly. essentially, the differences between various model states can be captured by the disparity in trained flops: the early checkpoint corresponds to fewer trained flops, whereas the final checkpoint corresponds to more. naturally, we can approximate the flops difference by using two reference models (rms) with fewer parameters compared to the experimental model, pretraining them on the same data scale. the smaller and larger rm play the roles of the early checkpoint and the final checkpoint, respectively. this approach significantly reduces both pretraining and inference costs. how to use pd to orderly arrange pretraining data. data with high pd are challenging for weak models but well-suited for strong models due to the increased capacity. conversely, data with low pd are less sensitive to model capability differences and can be understood by both strong and weak models. from the perspective of flops, the early stages of training an experimental model can be seen as involving a weak, smaller model, while later stages involve a strong, larger model. thus, placing high-pd data in the later training stages can enhance the model's ability to fit these challenging samples, whereas low-pd data can be trained earlier since they are less sensitive to model capability. this establishes a natural curriculum learning principle for the pretraining: begin with low-pd data and progress to high-pd data. following the principle, we propose an s-shape function that effectively models preferences throughout training. further, to maintain diversity in each batch, we use the concentration of low-pd data rather than pd itself as the output of the preference function. by overcoming the challenges, pdpc can always perceive and use the data preferred by llms to pretrain and boost them. notably, pdpc only arranges the given data without performing data selection. in summary, our work has the following contributions: (1) we propose pd to measure the difference in the fit of strong and weak models to samples, pointing out that high-pd data is challenging for weak models and is suitable to be arranged in the later pretraining stage. (2) we propose a novel pdpc framework that always perceives and uses the data preferred by llms to train and boost them, and ensures uninterrupted continuous training, which serves as the last data preprocessing step. (3) experimental results show significant performance improvements over baselines. notably, the 3b model trained on 1t tokens with pdpc demonstrates an average accuracy increase of 4.1% across all benchmarks and 8.1% across mmlu and cmmlu. ## pd-based preference curriculum in this section, we propose pdpc, which always perceives and uses the data preferred by llms to pretrain and boost them. notably, pdpc only organizes the given data without performing selection, which can serve as the final data preprocessing step before pretraining. ## related works data preprocessing is crucial in llm pretraining, ensuring dataset quality and integrity. traditional methods use expert-crafted rules to filter lowquality data and remove duplicates @xcite @xcite @xcite . enhanced approaches leverage target data sources or proxy models for curation @xcite @xcite . automated data selection using classifiers is gaining traction; for example, @xcite employed logistic regression to evaluate data quality, and other studies have developed sophisticated scoring mechanisms @xcite . qurating @xcite uses multiple raters to assess data contributions. curriculum learning (cl) complements these efforts by organizing training data from simple to complex, improving learning efficiency and generalization @xcite . in nlp, cl enhances models, such as in word embeddings @xcite and neural machine translation @xcite . recently, cl's application in llm pretraining is also growing @xcite . ## conclusion in this paper, we propose pdpc to address the limitations of consistent data distribution in pretraining llms. pdpc perceives the models' preferences and utilizes different, model-preferred data as the models' capabilities improve to enhance their performance. we introduce pd as a data metric and incorporate the preference function based on pd to predict data preferences, enabling the offline organization of data and ensuring uninterrupted pretraining. experiments show that pdpc significantly outperforms the baselines, with the 3b model achieving an average improvement of 8.1% over random on mmlu and cmmlu. exploration of additional pd partitions this study primarily focuses on the scenario where @xmath0, analyzing concentration mixing curves and systematically blending two subsets with higher and lower pd in accordance with training progression. however, we have not yet explored dividing the training data into more than two subsets to assess whether further performance enhancements are attainable. in future research, we plan to investigate cases where @xmath1. iterative update of learning curves we determine the s-shaped learning curve through functional exploration and use it as the basis for arranging the data sequence to train the model. in fact, we can also start from the newly trained model, re-explore new learning curves, and iteratively update our curriculum learning path. optimizing the learning curve through multiple iterations could be one of our future research directions. the model's pretraining, we conduct annealing experiments on checkpoints from different stages to explore the model's preferences for data mixing ratios. based on these preferences, we fit the model's preference function to guide the next round of pretraining. this process is iterated until the model's performance converges.