Staged Knowledge Distillation Through Least-to-Most Prompting: Optimizing Teacher Guidance via Difficulty-Aware Training (2025)

Abstract: Knowledge distillation (KD) enables the compression of large language models (LLMs) by transferring knowledge from a high-capacity teacher model to a resource-efficient student model, maintaining competitive performance for tasks such as instruction following. However, conventional white-box KD methods often suffer from training-inference mismatches and suboptimal performance due to the asymmetric nature of Kullback-Leibler divergence (KLD) and reliance on computationally expensive student-generated outputs. To address these challenges, we propose Least-to-Most Prompting Knowledge Distillation (L2M-KD), a novel white-box KD method grounded in curriculum learning (CL) and adaptive loss design. L2M-KD employs a two-pronged approach: (1) a CL strategy that ranks training samples by difficulty using Rouge-L scores, partitioning them into easy-to-hard subsets across multiple stages, and (2) an adaptive KD loss that transitions from KLD to skew KLD, dynamically adjusting teacher guidance to mitigate mode-averaging and over-smoothing. Extensive experiments on instruction-following tasks demonstrate that L2M-KD outperforms existing white-box KD methods, achieving superior student model performance with reduced computational overhead by leveraging ground-truth outputs exclusively. Our findings underscore the efficacy of difficulty-aware training and adaptive teacher guidance, offering a computationally efficient and robust approach to LLM compression.

## introduction large language models (llms) have achieved significant progress in text generation, language understanding, and inference, driven by increased parameter scales and high-quality training data (ouyang @xcite and skl (on-policy) @xcite , based on average rouge-l scores across five evaluation sets. left: gpt-2-1.5b as the teacher with gpt-2 (125m, 340m, 760m) as student models. right: opt-2.7b as the teacher with opt (125m, 350m, 1.3b) as student models. @xcite . however, their substantial computational and memory requirements during inference limit practical deployment, particularly in resourceconstrained settings like edge computing @xcite . this has spurred demand for smaller, efficient language models (lms) that maintain competitive performance in tasks such as text generation @xcite and tool learning @xcite . knowledge distillation (kd) @xcite , a technique that transfers knowledge from a high-capacity teacher model to a smaller student model, has become a cornerstone for compressing llms into small lms, as evidenced by models like llama 3.2 (meta, 2024) and @xcite . kd methods for llms are broadly classified into black-box and white-box approaches @xcite . black-box kd, which leverages only teacher predictions @xcite , gained traction due to the proprietary nature of models like gpt-4o @xcite and claude 3.5 @xcite . however, the emergence of opensource llms, such as deepseek-v3 @xcite and qwen 2.5 @xcite , with performance rivaling proprietary models @xcite , has shifted focus toward white-box kd. white-box kd offers enhanced student performance and greater control over the distillation process. existing white-box kd methods primarily optimize the divergence function d(• ∥ •) in the kd loss (eq. 2), with kullback-leibler divergence (kld) @xcite as a standard. yet, kld's asymmetric nature often induces modeaveraging @xcite , prompting alternatives like reverse kld @xcite , jensen-shannon divergence (jsd) @xcite , and skew kld @xcite . these variants, while effective in specific contexts, lack generalizability across tasks and datasets, resulting in inconsistent performance @xcite . data curation strategies further influence kd efficacy, with options including ground-truth outputs (gtos) @xcite , teacher-generated outputs (tgos) @xcite , and studentgenerated outputs (sgos) @xcite . gtos, though widely used, can cause training-inference mismatches due to the student's limited capacity, while sgo-based onpolicy methods @xcite mitigate this but incur significant computational costs and risk teacher misguidance @xcite . thus, a central challenge in white-box kd for llms is to design an effective kd loss and leverage gtos exclusively to boost student performance without excessive computational overhead. despite the prevalence of kld and gtos in white-box kd, their full potential remains underexploited, particularly for llms. kl's modeaveraging tendencies and the computational ineffi-ciencies of sgo-based methods highlight the need for a refined approach. we posit that structuring the training process and dynamically adapting the kd loss can enhance the effectiveness of kld and gtos, improving student model performance while sidestepping the limitations of existing methods. empirical evidence suggests that student models benefit from progressive training on samples of increasing difficulty, paired with tailored teacher guidance @xcite . drawing inspiration from curriculum learning (cl) @xcite and the educational strategy of least-to-most prompting @xcite , where guidance scales with task complexity, we propose a staged kd method. this approach mirrors human learning by starting with simpler tasks and incrementally introducing challenges, adjusting the kd loss to provide optimal teacher guidance at each stage. our motivation is rooted in the observation that such a structured process can stabilize knowledge transfer, mitigate mode-averaging, and maximize the utility of gtos, offering a computationally efficient alternative to on-policy methods. we present least-to-most prompting knowledge distillation (l2m-kd), a novel white-box kd method that combines a cl framework with an adaptive kd loss to optimize teacher guidance based on training difficulty (see fig. 2 ). l2m-kd operates in two key phases: (1) a curriculum learning framework ranks training samples by difficulty using rouge-l scores between gtos and sgos, partitioning the dataset into easy-to-hard subsets across multiple stages; (2) an adaptive kd loss transitions from kld to skew kld (skl) @xcite , increasing teacher prompting as difficulty rises, following the least-to-most prompting principle. our contributions are threefold: ## related work white-box kd for llms leverages teacher model internals, outperforming black-box kd @xcite . methods focus on kd loss functions like kld @xcite , reverse kld @xcite , jsd @xcite , and skew kld @xcite , alongside data strategies using tgos @xcite , sgos @xcite , or mixtures @xcite to address mode-averaging and training-inference mismatches (section 2.1). however, optimal combinations remain challenging. curriculum learning (cl) in kd, primarily in computer vision @xcite , is underexplored for llms. confucius @xcite applies cl to black-box kd for tool learning, but general white-box kd lacks such frameworks, which l2m-kd addresses (section 3.2). ## conclusion in this work, we introduce l2m-kd, a novel whitebox kd method designed to address critical challenges in compressing llms for auto-regressive tasks. by integrating a cl strategy with an adaptive kd loss function, l2m-kd optimizes knowledge transfer from a high-capacity teacher to a resource-efficient student model. the method comprises two core components: (1) a cl strategy that ranks training samples by difficulty using rouge-l scores, enabling a staged training process from easy to hard subsets inspired by least-to-most prompting, and (2) an adaptive kd loss that transitions from kld to skl, dynamically adjusting teacher guidance to mitigate mode-averaging and oversmoothing. extensive evaluations on instructionfollowing tasks demonstrate that l2m-kd achieves superior student model performance compared to existing white-box kd methods, while exclusively utilizing gtos to minimize computational costs. these results highlight the effectiveness of difficulty-aware training and adaptive loss design in enhancing kd efficiency and robustness. our work provides a scalable and practical approach to llm compression, paving the way for broader deployment in resource-constrained environments. future research will explore the generalizability of l2m-kd across diverse model architectures and tasks, as well as its efficacy for larger student models.