Knowledge Distillation for Language Models (2025)

Abstract: Knowledge distillation (KD) aims to transfer the knowledge of a teacher (usually a large model) to a student (usually a small one). In this tutorial, our goal is to provide participants with a comprehensive understanding of the techniques and applications of KD for language models. After introducing the basic concepts including intermediate-layer matching and prediction matching, we will present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation. For applications, we will focus on KD for large language models (LLMs), covering topics ranging from LLM sequence compression to LLM self-distillation. The target audience is expected to know the basics of machine learning and NLP, but do not have to be familiar with the details of math derivation and neural models

## introduction recent advances in deep learning have largely changed the field of natural language processing (nlp). in particular, large language models (llm) have been the cornerstone of nlp research, and they are now tightly integrated into our daily lives. despite the success of llms in a wide range of applications, they may be cumbersome to use due to their high memory and computational overhead. this calls for an increasing need to make these models more efficient and accessible, so a broader range of users can benefit from llms. researchers have been working on reducing the computational cost of running llms in various ways. for example, model pruning is a technique that removes "low-impact" parameters of a network to reduce the memory usage @xcite @xcite . alternatively, quantization aims to reduce the number of bits used to represent the parameters without severely deteriorating the performance @xcite . in this tutorial proposal, we will focus on knowledge distillation @xcite , which aims at transferring knowledge from a teacher (typically a large model) to a student (known as the student). it has gained increasing attention in the nlp community, driven by the demands of compressing the ever-growing and high-performing language models. after an introduction and overview, we will start the tutorial with the basics of kd, mainly falling into the following two categories: intermediatelayer matching and prediction matching. the former refers to the distillation of intermediate layers, including activated features @xcite @xcite and attention weights @xcite ; we will also discuss relational learning, which distills the relative structures of features (e.g., transformations) instead of the absolute feature values @xcite . for the prediction matching, we will present the classic cross-entropy approach, with an emphasis on its multi-modality issue foot_0 @xcite @xcite : when the student model's capacity is not large enough, it is unable to learn the multi-modal distribution predicted by the large teacher, oftentimes resulting in severe model collapse and mode issues. we will discuss different divergence-based methods @xcite to mitigate this issue. then, we will move on to the second part of the tutorial, where we present two selected topics on advanced kd techniques: reinforcement learning (rl)-based kd and multi-teacher kd. reinforcement learning has been gaining increasing attention in recent years, due to its success in training llms, showing great success in aligning the model with human preference as well as mitigating exposure bias @xcite . in here, we will dive into rl in the context of knowledge distillation, where the key challenge is to derive a reward function based on the teacher model @xcite . we will also discuss multi-teacher kd, where the student model learns from multiple teachers, each having its own expertise. this ties closely to the multi-modality problem that we have posed in the first part of our tutorial, where the knowledge is too diverse for the student to learn. we present a solution to this based on the ensemble-then-distill framework, where an ensemble process is applied before distillation @xcite . this allows the student to learn high-quality, consolidated knowledge instead of conflicting knowledge from different teachers. the last part of our tutorial will focus on kd with large language models (llms). we start by presenting interesting phenomena observed in llm distillation, such as the effect of teacher intervention @xcite and emerging chain-ofthought abilities in small models @xcite . then, we will showcase how kd can be used to compress the prompts @xcite @xcite and the reasoning process @xcite to speed up inference. we will move on to self-distillation, where llms are able to reflect upon its own generations and learn skills such as instruction following @xcite , reasoning @xcite and summarization @xcite . finally, we will walk through modern distilled systems, including alpaca @xcite , vicuna @xcite , and deepseek's distilled models @xcite to gain a sense of practical use of kd techniques. we conclude the tutorial by showing surprising and interesting applications related to kd, including quantization @xcite , speculative decoding @xcite , and non-autoregressive translation @xcite . overall, this tutorial will lay a solid foundation of knowledge distillation for language models, with highlights of both machine learning challenges and cutting-edge applications. ## target audience the tutorial targets a diverse audience, including machine learning and nlp researchers, as well as practitioners. we expect the audience to have a brief knowledge of deep learning (e.g., cross-entropy loss and back-propagation training) and nlp (e.g., autoregressive text generation and large language models). however, the audience do not have to be familiar with the details (e.g., derivative calculations, transformer attention formulas); only a general impression would suffice. the audience does not have to have heard of knowledge distillation. we will teach the foundations before moving on to cutting-edge algorithms and applications. break [10min] part iv: kd applications for llms [45min] • empirical findings in llm distillation • llm sequence compression • llm self-distillation for performance improvement • sota distilled systems • other interesting kd applications part v: conclusion, future directions, and qa [15min] 4 presenters yuqiao wen is currently a third-year phd student at the department of computing science, university of alberta, after having his msc in 2022 and bsc in 2020. yuqiao's research lies in developing efficient methods for large language models and making them more accessible for everyone; he has a focus on machine learning problems in knowledge distillation such as label bias and exposure bias. he has published a number of papers at top-tier venues such as aaai, acl, and iclr, including one winning an area chair's award. he was a co-presenter of a three-hour tutorial at the amii upper bound conference, which attracted several thousand attendees. freda shi is a first-year assistant professor in the david r. cheriton school of computer science at the university of waterloo and a faculty member at the vector institute. her research interests are in computational linguistics, natural language processing, and cognitive sciences. she has been working on knowledge distillation for syntactic analysis and multilingualism, with relevant papers published at acl and iclr. her work has been recognized with a google phd fellowship, a facebook fellowship finalist award, and best paper nominations at acl 2019, 2021, and 2024. she has served as an area chair for conferences such as acl, emnlp, and colm, and as a program committee member or a reviewer for leading journals and conferences in computational linguistics and machine learning, including tacl, tpami, acl, coling, emnlp, naacl, iclr, icml, and neurips. lili mou is a sixth-year assistant professor at the department of computing science, university of alberta. his main research interest lies in developing novel machine learning methods for nlp tasks; successful examples include tree-based convolutional neural networks, edit-based unsupervised text generation, and an ensemble-then-distill framework for multi-teacher kd. he regularly serves as a senior program committee member or an area chair for ai and nlp conferences, and is an action editor for acl rolling review. he is an amii fellow and a canada cifar ai chair, and has received a aaai new faculty highlight award; he also received an acl best paper nomination (2019) and acl area chair's award (2024). lili has been a co-organizer of the workshop on efficient speech and natural language processing, co-located with neurips during 2021-2023. he presented two conference tutorials at emnlp-ijcnlp 2019 and acl 2020.