FRAME : Boosting LLM s with A Four-Quadrant Multi-Stage Pretraining Strategy (2025)

Abstract: Large language models (LLMs) have significantly advanced human language understanding and generation, with pretraining data quality and organization being crucial to their performance. Multi-stage pretraining is a promising approach, but existing methods often lack quantitative criteria for data partitioning and instead rely on intuitive heuristics. In this paper, we propose the novel Four-quadRAnt Multi-stage prEtraining strategy (FRAME), guided by the established principle of organizing the pretraining process into four stages to achieve significant loss reductions four times. This principle is grounded in two key findings: first, training on high Perplexity (PPL) data followed by low PPL data, and second, training on low PPL difference (PD) data followed by high PD data, both causing the loss to drop significantly twice and performance enhancements. By partitioning data into four quadrants and strategically organizing them, FRAME achieves a remarkable 16.8% average improvement over random across MMLU and CMMLU for the 3B model, effectively boosting LLM performance.

## introduction llms have significantly advanced human language understanding and generation @xcite @xcite . the quality and organization of pretraining data are crucial as they directly impact llm performance. studies show that llms require different data at various pretraining stages @xcite , offering key insights for developing more efficient pretraining data organization strategies. recent studies partition the pretraining process into multiple stages, allowing models to learn from data with distinct characteristics at each stage, which enhances pretraining efficiency @xcite @xcite . for * equal contribution. figure 1 : few-shot downstream performance on various benchmarks with respect to pretraining iterations, for 3b models trained on 1t tokens. the average performance is based on 8 downstream tasks. frame achieves 15.3% improvement over random across mmlu, and 18.2% on cmmlu. in this paper, we propose a novel four-quadrant multi-stage pretraining strategy (frame) to boost llms' performance. this strategy is systematically guided by the principle of organizing the pretraining process into four stages to achieve significant loss drops four times. the principle is derived from two key findings based on quantifiable data metrics. specifically, we intuitively use the metric perplexity (ppl) to partition the data into two parts: high ppl and low ppl. our first key finding reveals that training on high ppl data first, followed by low pretraining dataset d low high high q3 q4 q1 q2 pd ppl ppl high ppl data for stage 1 low ppl data for stage 2 pd low pd data for stage 1 high pd data for stage 2 two-stage pretraining four quadrant multi-stage pretraining strategy finding 1 training on data with ppl from high to low leads to two significant loss drops. training on data with pd from low to high leads to two significant loss drops. finding 2 achieve four significant loss drops. frame @xmath0, leads to the loss dropping significantly twice and boosts model performance. inspired by pdpc @xcite , we also introduce the ppl difference (pd) between strong and weak models as another metric for data partitioning. our second key finding shows that training on low pd data first, followed by high pd data, similarly causes the pretraining loss to drop significantly twice, thereby boosting model performance. based on these two major findings, we establish the principle of organizing the pretraining process into four stages to achieve significant loss reductions four times. specifically, we partition the data into four quadrants based on ppl and pd: quadrant 1 (q 1 ) with low pd and low ppl, quadrant 2 (q 2 ) with high pd and low ppl, quadrant 3 (q 3 ) with low pd and high ppl, and quadrant 4 (q 4 ) with high pd and high ppl. through further analysis and experiments, we determine that the optimal strategy is to reorganize the pretraining data in the @xmath1. to ensure smooth data transitions between stages, we also implement a smoothing process for gradual stage transitions. all data is processed offline to ensure the continuity of model training is not disrupted. practical evidence shows that frame results in loss dropping significantly four times, enhances the models' emergent abilities, and boosts their performance. as shown in figure 1 , frame achieves a significant average improvement of 15.3% on mmlu and 18.2% on cmmlu over random selection, respectively. in summary, our contributions are as follows: (1) we identify two key findings: first, training on high ppl data followed by low ppl data, and second, training on low pd data followed by high pd data, both causing two significant loss reductions and performance enhancements. (2) guided by the principle of continuous loss reduction, we pro-pose frame, a novel strategy that partitions data into four quadrants and organizes the pretraining process into four stages, resulting in loss dropping significantly four times and boosting model performance. (3) experiments on 3b model, trained on 1t tokens, demonstrate a 16.8% improvement over uniform sampling in average performance across mmlu and cmmlu. ## four quadrant multi-stage pretraining strategy in this section, we first evaluate the effectiveness of ppl and pd as metrics for data organization in two-stage training (sections 2.1 and 2.2). based on these insights, we propose frame with both metrics(section 2.3), as shown in figure 2 . ## related works multi-stage pretraining has emerged as a pivotal strategy in the development of llms, enabling models to better capture and utilize diverse data characteristics by dividing the training process into distinct phases @xcite @xcite . @xcite propose a multi-stage pretraining method that leverages various granularities of information, significantly boosting model performance. @xcite explore the use of multi-stage prompting to improve translation tasks, demonstrating its effectiveness in enhancing downstream applications. in llm pretraining, data preprocessing is key to ensuring dataset quality @xcite . traditional methods use expert rules to filter low-quality data @xcite @xcite @xcite and remove duplicates @xcite @xcite @xcite . these approaches enhance quality but may lack semantic depth. to improve semantic selection, strategies involve using targeted sources or proxy models @xcite @xcite @xcite . classifiers automate selection, like the logistic regression model used by @xcite , with others employing complex scoring @xcite . qurating @xcite uses multiple raters for nuanced evaluation. current methods focus on selection but overlook aligning data characteristics with learning stages, missing opportunities in data organization and sequencing to boost pretraining effectiveness. ## conclusion in this study, we propose the four-quadrant multi-stage pretraining strategy (frame), a novel approach designed to boost the performance of llms by systematically organizing the pretraining process into four stages. guided by the principle of achieving significant loss reductions four times, frame employs a strategic partitioning of pretraining data into four quadrants based on ppl and pd. the experimental results, showing a 16.8% average improvement across mmlu and cmmlu over random sampling, underscore the effectiveness of frame in optimizing pretraining data organization.