Exploring the Limits of Model Compression in LLM s: A Knowledge Distillation Study on QA Tasks (2025)

Abstract: Large Language Models (LLMs) have shown outstanding performance across a range of NLP tasks, but their computational demands hinder deployment in real-world, resource-constrained environments. This work investigates the extent to which LLMs can be compressed using knowledge distillation (KD) while maintaining strong performance on question answering (QA) tasks. We evaluate student models distilled from the Pythia and Qwen2.5 families on two QA benchmarks, SQuAD and MLQA, under zero-shot and one-shot prompting conditions. Results show that student models retain over 90% of their teacher models’ performance while reducing parameter counts by up to 57.1%. Furthermore, one-shot prompting yields additional performance gains over zero-shot setups for both model families. These findings underscore the trade-off between model efficiency and task performance, demonstrating that KD, combined with minimal prompting, can yield compact yet capable QA systems suitable for real-world applications.

## introduction in recent years, the increasing capabilities of large language models (llms) have significantly advanced the field of nlp, enabling state-of-the-art results in tasks ranging from question answering (qa) @xcite to summarization @xcite and translation @xcite . however, their considerable memory and compute demands hinder practical deployment, particularly in resource-constrained environments @xcite . these challenges have fueled interest in model compression techniques, which aim to reduce model size while preserving the performance of large models. ## related work kd has proven effective across a variety of nlp tasks, including classification and sequence generation, with notable examples such as distil-bert @xcite and tinybert @xcite . for autoregressive models, the dis-tillm framework @xcite proposed skew kullback-leibler divergence and off-policy sampling strategies to improve stability and distillation quality. our work builds directly on distillm by applying its distillation methodology to the pythia @xcite and qwen2.5 @xcite model families. in parallel, few-shot prompting, a form of in-context learning (icl), allows models to gen-eralize to new tasks by conditioning on a small number of examples in the input prompt @xcite . recent work has begun to explore the intersection of kd and icl. notably, @xcite introduced in-context learning distillation to transfer few-shot capabilities from larger to smaller models, focusing on meta-trained models evaluated on multitask and instruction-tuned benchmarks, crossfit @xcite and lama @xcite . similarly, @xcite examined kd for few-shot intent classification. our study differs in two key ways: (1) we focus on extractive qa tasks, rather than multitask or classification settings, and (2) we explicitly evaluate performance under zero-shot and one-shot prompting, without relying on instruction tuning or prompt mixture training. this setup allows us to isolate how prompt structure and model size interact, offering new insights into the compressionperformance trade-off in qa settings. ## methodology we conduct a scaling analysis using distilled models of varying sizes and assess their performance under different prompting strategies. ## results and discussion in this section, we present the empirical results from our experiments, analyzing the effects of model compression and prompting strategies on qa performance. model size vs. performance: figures 1 and 2 illustrate how model performance scales with size across the qwen2.5 and pythia model families under both zero-shot and one-shot prompting conditions. we observe that larger student models, such as qwen2.5-3b and pythia-1.4b, consistently retain over 90% of their teacher models' performance in both em and rouge-l metrics. in particular, the distilled qwen2.5-3b closely matches or even slightly exceeds the performance of the qwen2.5-7b teacher in zero-shot settings, suggesting that moderate compression can enhance generalization. as the model size decreases, performance degrades more noticeably, especially for smaller models like qwen2.5-0.5b and pythia-70m. this trend is consistent across both zero-shot and one-shot settings, suggesting that maintaining strong taskspecific capabilities under compression becomes increasingly challenging at lower parameter scales. nevertheless, moderately sized student models keep a good balance between efficiency and performance, making them good candidates for resourceconstrained applications. all experiments were run with five different random seeds, with results showing low variance and high reproducibility across runs (see appendix a.3 for detailed variance analysis). distillation vs. fine-tuning: distilled models generally outperform their fine-tuned counterparts. for instance, the distilled qwen2.5-3b and pythia-1.4b models achieve substantially higher scores compared to their fine-tuned counterparts, narrowing the gap to their teacher models. in contrast, finetuned students show larger performance degradation, particularly at smaller scales (e.g., qwen2.5-0.5b and pythia-70m). these results demonstrate that kd not only compresses models effectively but also enhances their ability to generalize from prompts, making it a more reliable strategy for scaling down llms without significant task-specific performance loss. ## conclusion this study offers a detailed investigation into compressing llms through kd, with a focus on retaining their icl abilities. evaluating models from the pythia and qwen2.5 families on both squad and mlqa datasets, we find that distilled student models can retain over 90% of their teacher models' performance while achieving substantial re-ductions in parameter size. one-shot prompting further increases performance, particularly in multilingual settings like mlqa (german), highlighting the usefulness of few-shot learning in compressed models. overall, our findings suggest that kd, when combined with few-shot prompting, offers a promising direction for building compact, generalizable, and cost-efficient language models suitable for real-world deployment. while our study focuses on extractive qa tasks, the observed trends, particularly the strong performance retention of distilled models under minimal few-shot prompting, may generalize to other nlp tasks that can be framed as qa problems. recent work on instruction-tuned models such as unifiedqa @xcite and flan @xcite has shown that a wide range of tasks, including sentiment analysis, text classification, and natural language inference, can be effectively reformulated using a qa-style input-output format. moreover, both the pythia and qwen2.5 model families have demonstrated strong baseline performance on non-qa benchmarks, such as lam-bada, piqa, and winogrande for pythia @xcite , and mmlu, bbh, and arc-c for qwen2.5 @xcite , highlighting their versatility beyond qa. therefore, our findings may hold relevance for broader instruction-style applications, especially under few-shot or zero-shot settings. future work could explore whether similar compression and prompting strategies yield consistent gains across diverse task families.