On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025)

Abstract: Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. Despite the recent traction of KD research, its effectiveness for smaller language models (LMs) and the mechanisms driving knowledge transfer remain underexplored. In this work, we present the first large-scale empirical and statistical analysis of KD across models ranging from 0.5B to 7B parameters on 14 complex reasoning tasks in a zero-shot setting. Our findings reveal that KD can improve the average performance of smaller models by up to 10%, with a peak task specific gain of 22%, while providing only marginal benefits ( \sim 1.3 %) for larger models. Surprisingly, teacher performance has a minimal impact on student outcomes, while teacher task expertise impacts KD effectiveness. A correlation study indicates that smaller LMs benefit more from KD, whereas larger LMs show diminished gains. Additionally, we uncover a misalignment between improvements in student performance and reasoning fidelity, suggesting that while KD enhances accuracy, it does not always maintain the structured decision-making processes of the teacher. Our ablation study further highlights the importance of teacher signals and logit smoothing in influencing students’ performance after distillation. Overall, our study offers a comprehensive empirical and statistical assessment of KD, highlighting both its benefits and trade-offs when distilling knowledge from larger to smaller LMs.

## introduction the rapid advancement of pre-trained language models (lms) has led to the development of largescale language models that achieve state-of-theart performance across various nlp tasks (dubey et al., 2024; yang et al., 2024; deepseek-ai et al., * these authors contributed equally to this work. 2024). however, deploying these large models presents significant challenges due to their high computational and memory requirements @xcite . model compression has emerged as a crucial technique to mitigate these challenges by reducing model size while preserving performance. among various compression techniques, knowledge distillation (kd) has gained significant attention as it enables a smaller student model to learn from a larger teacher model, maintaining strong performance with reduced resource demands @xcite . kd plays a pivotal role in making large language models (llms) more accessible, facilitating their deployment in resourceconstrained environments. several knowledge distillation approaches have been proposed to improve the training of student model and enhance generalization. @xcite introduced the concept of soft target distillation, where the student learns from the softened output probabilities of the teacher model. subsequent studies extended this idea to sequence-level kd for language models, such as seqkd @xcite , which aligns the student model with the teacher's output distributions. more recent variants, including minillm @xcite , leverage reverse kullback-leibler (kl) divergence to improve the learning of student model for generative language models. despite these advancements, a comprehensive understanding of kd techniques for llms remains unexplored. current research primarily focuses on task-specific distillation setups, often overlooking broader implications across diverse tasks. moreover, the distribution mismatch between training and inference remains a persistent challenge, particularly in autoregressive language models @xcite , necessitating further exploration of large-scale kd strategies. in this paper, we conduct a large-scale empirical and statistical analysis to evaluate the effectiveness of knowledge distillation for small lms ranging qwen2.5-7b (c) performance on instruction following tasks figure 1 : performance of different qwen-2.5 student models across various (a) mathematical and (b) commonsense reasoning (c) instruction following tasks without and with distillation from larger qwen models. we highlight the zero-shot accuracy for the supervised fine-tuned (sft) model and the models distilled with different kd methods. the results are elaborated in tables 7, 8 and 9 in appendix b. similar results for llama student models are highlighted in figure 7 of appendix b. from 0.5b to 7b parameters across 19 complex reasoning and instruction following tasks in a zeroshot setting. we investigate the impact of kd on the generalizability of small lms to explore different aspects of knowledge distillation. we summarize the key findings from our study as follows 1 : despite varying improvement patterns, statistical analysis shows that different kd methods yield similar post-distillation performance. a spearman rank test reveals that a stronger teacher does not necessarily lead to a better student. however, the teacher's task expertise is crucial -distilling from a task-unaware teacher can degrade student performance by up to 40%. understanding effectiveness of kd. while kd enhances model compression and transfer learning, its inner workings remain underexplored. conventional kd methods often fail to preserve explainability, reducing trust in distilled models @xcite . studies challenge the assumption that high student-teacher fidelity ensures better generalization, revealing that students often struggle to mimic teachers due to optimization and dataset constraints @xcite . kd can also transfer biases and adversarial vulnerabilities alongside useful knowledge @xcite . attention and fidelity mechanisms are crucial, with research showing that diverse teacher attention maps enhance student generalization more effectively than rigid mimicry @xcite . however, these studies primarily focus on vision tasks. similar investigations are needed for language-based tasks to understand how kd influences reasoning, linguistic structures, and emergent capabilities of llms. how our work is different? unlike prior works that primarily focus on task-specific knowledge distillation or performance benchmarking on a limited set of tasks, this study provides a large-scale empirical and statistical analysis of kd across diverse reasoning benchmarks with small lms of varying sizes and capabilities. while most existing studies overlook explainability of kd, our work systematically examines the key factors influencing its effectiveness, providing valuable practical insights into the process. ## methodology here, we briefly describe three state-of-the-art knowledge distillation methods used in our study: sequence-level knowledge distillation (seqkd), reverse kl knowledge distillation (revkd), and generalized knowledge distillation (gkd). each method varies in its approach to training the student model and its associated loss objectives. sequence-level knowledge distillation. seqkd @xcite extends traditional kd @xcite by aligning the student's output sequences with those of the teacher. unlike standard kd, which operates at the token level, se-qkd forces the student model to generate entire sequences that match the teacher's outputs. this is particularly useful for sequence generation tasks, such as machine translation and text summarization. the loss function for seqkd is defined as: reverse kl knowledge distillation. @xcite proposed a novel kd strategy by minimizing the reverse kl divergence, instead of the standard forward kl divergence used in traditional kd. the motivation is to avoid overestimating lowprobability regions of the teacher's distribution, which is crucial for generative language models. the student model is trained using the following reverse kl loss: ## experimental setup and evaluate on hellaswag @xcite , winogrande @xcite , arc @xcite , obqa @xcite , boolq @xcite , piqa @xcite , and siqa @xcite . for instruction following, we fine-tune using dolly-15k @xcite and evaluate on dolly, selfinst @xcite (denoted by 'self' throughout the paper), vicuna @xcite , sni @xcite and uni @xcite . detailed dataset descriptions and splits are provided in appendix a and we use a batch-size of 16, learning rate of 3 × 10 -4 and max-length of 256 across all models and methods. we fine-tune the models for 3 and 4 epochs for commonsense15k and math10k datasets, respectively. following @xcite , for zero-shot evaluation we set τ = 1 (default, for temperature ablation experiments highlighted in figure 6 we use τ ∈ {1, 2, 5}), top-@xmath1, @xmath2. measures for quantifying teacher-student agreement and fidelity. in this paper, along with the performance measurement of student models, we use teacher-student agreement and reasoning fidelity scores to understand alignment between teacher and student post-kd. agreement quantifies how often student replicates the teacher's outputs and is measured using top-1 agreement (fraction of matching predictions). however, this metric only relies on the final answer and does not measure the quality of the intermediate reasoning steps. reasoning fidelity, on the other hand, captures how well the student mirrors the teacher's reasoning process rather than just final predictions. we use bleu @xcite score between teacher and student reasoning outputs to compute fidelity. ## experimental results in this section, we elaborate the empirical and statistical results obtained in our study. ## conclusion this paper elaborated the impact of kd on small lms, considering factors like teacher performance, student size, and distillation methods across mathematical and commonsense reasoning tasks. results showed that kd significantly benefits smaller models, but its effectiveness diminishes with increasing model size. teacher domain adaptation played a more critical role than teacher performance in kd success, particularly for structured reasoning tasks. surprisingly, higher teacher-student agreement did not always correlate with better student performance, especially in complex reasoning tasks where strong students often deviated from teacher outputs. these findings underscore the need for task-aware kd strategies and adaptive distillation techniques tailored to student learning dynamics. future research should explore alternative kd objectives, self-distillation mechanisms, and refined teacher-student alignment strategies to improve both performance and reasoning fidelity.