P rompt KD : Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024)

Abstract: Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, research on KD for generative language models like LLMs is relatively sparse, and the approach of distilling student-friendly knowledge, which has shown promising performance in KD for classification models, remains unexplored in generative language models. To explore this approach, we propose PromptKD, a simple yet effective method that utilizes prompt tuning - for the first time in KD - to enable generative language models to transfer student-friendly knowledge. Unlike previous works in classification that require fine-tuning the entire teacher model for extracting student-friendly knowledge, PromptKD achieves similar effects by adding a small number of prompt tokens and tuning only the prompt with student guidance. Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts. Further analysis suggests that distilling student-friendly knowledge alleviates exposure bias effectively throughout the entire training process, leading to performance enhancements.

## introduction with the massive improvement of generative language models, such as the emerging abilities @xcite observed in large language models (llms), there is a growing need for model compression research to efficiently deploy models in various tasks @xcite . however, among notable compression methods such as knowledge distillation (kd; @xcite @xcite , pruning @xcite , and quantization (tao 1 project page: https://promptkd.github.io a v e r a g e r o u g e -l s t u d e n t ( # p a r a m s ) s f t k d s e q k d m i n i l l m g k d p r o m p t k d et al., 2022), kd has not been successfully applied to generative language models. since most kd methods are devised with models like bert @xcite for classification tasks, the challenge arises when attempting to directly apply these kd methods to generative language models, which have different architectures and are designed for tasks other than classification. while there have been some methods proposed for generative language models, such as supervised kd @xcite or seqkd @xcite , they tend to be naive approaches. even recently proposed works @xcite , like previous research, have focused on distribution discrepancy metrics or pseudo-targets. therefore, despite the rapid advancement of llms in recent times, the drawback is that they are not designed with the extension to llms in mind. moreover, attempts to distill student-friendly knowledge in a generative language model have yet to be explored. recent kd studies @xcite @xcite for classification tasks aim to distill such knowledge. this idea emerges because previous works extract knowledge from fixed teacher without knowing the student's capacity, and the observation (cho and hariharan, 2019) that larger teacher models do not necessarily improve student performance. however, there hasn't been any exploration of applying these ideas to generative language models. since the capacity gap between teacher and student persists in kd for generative language models, it is reasonable to expect that distilling student-friendly knowledge would be beneficial. to address this issues, we propose promptkd, which utilizes prompts in generative language models to distill student-friendly knowledge. extracting student-friendly knowledge from the teacher requires modifying the teacher, as in previous studies @xcite . however, modifying a large teacher model can incur significant computational costs. promptkd addresses this concern by exploiting prompt tuning. by appending prompt tokens to the beginning of the input, we can efficiently fine-tune the teacher model with notably fewer parameters. while there are other parameter-efficient fine-tuning methods such as prefix-tuning (li and @xcite and @xcite , they suffer from the disadvantage that the number of parameters to be trained increases proportionally with the number of layers. moreover, there is an observation @xcite that prompt tuning shows similar performance to full-parameter fine-tuning as the model size increases, making prompt tuning a more reasonable choice. promptkd learns prompts that stimulate the teacher to distill student-friendly knowledge with guidance from the student. additionally, it employs regularization loss during the early stages of training to prevent significant divergence from the original teacher when appending prompts, ensuring stable training. for evaluation, we measure the instructionfollowing performance @xcite , aiming to cover a variety of tasks that generative language models can perform. compared to the existing baseline, promptkd achieves state-of-the-art performance by adding prompt parameters equivalent to only 0.0007% of the teacher parameters, as depicted in figure 1 . additionally, the analysis of exposure bias suggests that remarkable alleviation of exposure bias through student-friendly knowledge is likely the cause of performance improvement. lastly, we explore the student-friendly knowledge in promptkd and confirm the necessity of regularization loss and the importance of prompt initialization through ablation studies. to summarize, our contribution is four-fold: kd for text classification knowledge distillation (kd) @xcite is a model compression technique where the knowledge of a teacher model is transferred to improve the performance of a student model. most kd research has been focused on text classification tasks. it has evolved from simple approaches @xcite ) that match the class distributions between teacher and student to more complex methods @xcite @xcite ) that involve matching hidden states or attention matrices between models. recently, concerns have been raised about the observation (cho and hariharan, 2019) that larger teacher models do not necessarily produce better students and the issue of teachers distilling knowledge while being unaware of the student's capacity. kd for text generation for text generation, @xcite minimizes the kl divergence between the next token prediction distributions of the teacher and student at each time step. in addition, some research @xcite focus on the sentences inputted to the teacher and student during the distillation process. for example, @xcite uses sentences generated by the teacher as pseudo-targets instead of ground truth. moreover, black-box kd methods @xcite that use inference-only black-box llms as teachers and augment existing data before training are proposed. recently, agarwal et al. (2024); gu et al. (2024) explored discrepancy metrics between model distributions and used sentences generated by the student as pseudo-targets to minimize exposure bias. however, there have been no attempts yet to distill student-friendly knowledge while the teacher is aware of the student's capacity. although liang et al. (2023b) incorporates task-aware filters into both teacher and student to transfer knowledge, its scalability is limited due to the addition of filters at each layer for layer distillation. crucially, it encourages knowledge to be task-specific, making it diverge from what we aim to explore in this paper. prompt tuning after @xcite demonstrates that pre-trained language models can perform specific tasks by prepending text prompts to input, many studies have tried to either manually craft @xcite or automatically discover @xcite @xcite such hard prompts, which are discrete tokens. subsequently, research @xcite emerged to advance prompts into the form of soft prompts composed of embeddings, making prompt updates via back-propagation easier and resulting in better performance compared to hard prompts. presently, prompt tuning @xcite has become a prominent parameter-efficient fine-tuning technique. although @xcite uses hard prompts to generate input data for knowledge extraction, we are pioneering the use of prompts for parameterefficient fine-tuning in kd research. ## promptkd promptkd is devised in the instructionfollowing @xcite setting for application to generative language models. we formulate instruction-following as a condi-tional text generation task, where the request @xmath0, x 2 , . . . , x n } sampled from the data distribution p x comprises instruction and input to describe the task. then, given the request x as a condition, the model generates a response @xmath1, y 2 , . . . , y t }. for prompt tuning, soft prompts @xmath2, p 2 , . . . , p m }, where p i is an embedding vector of the same dimension as the token embedding, are initialized with text and prepended to the input request x. formally, given the request x, the teacher model distribution conditioned on the prompt p is denoted as p(y|p, x) (here we suppress the teacher's model parameter since it is fixed), and the student's model distribution parameterized by θ is denoted as q θ (y|x), where only the student model parameters θ and the prompt p are trainable. the training process consists of 3 steps per iteration, as shown in figure 2 . first, generating input data used for knowledge distillation (pseudo-target generation). then, updating the prompt based on guidance from the student and teacher models to facilitate adaptive teaching (prompt tuning for adaptive teaching). finally, distilling student-friendly knowledge to the student using the updated prompt (student-friendly knowledge distillation). ## conclusions in this work, we have pioneered the exploration of extracting and distilling student-friendly knowledge for generative language models. to achieve this, we have proposed a novel method called promptkd, which leverages prompt tuning in knowledge distillation for the first time. owing to the memory-efficient nature of prompts and the advantage of replacing full-parameter fine-tuning, particularly beneficial for larger models like llms, promptkd has proven to be an efficient approach. through extensive experiments, promptkd has achieved state-of-the-art performance, confirming the effectiveness of student-friendly knowledge in generation tasks. specifically, it has been revealed that this student-friendly knowledge is extracted from a modified teacher, which outputs a distribution similar to that of the student while maintaining the generation performance. moreover, through exposure bias analysis, we have demonstrated that promptkd successfully alleviates exposure bias throughout the training process.