Personalized Distillation: Empowering Open-Sourced LLM s with Adaptive Learning for Code Generation (2023)

Abstract: With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are increasing interests in distilling the capabilies of close-sourced LLMs to smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT to generate a set of instructions and answers, for the student model to learn. However, such standard distillation approach neglects the merits and conditions of the student model. Inspired by modern teaching principles, we design a personalised distillation process, in which the student attempts to solve a task first, then the teacher provides an adaptive refinement for the student to improve. Instead of feeding the student with teacher’s prior, personalised distillation enables personalised learning for the student model, as it only learns on examples it makes mistakes upon and learns to improve its own solution. On code generation, personalised distillation consistently outperforms standard distillation with only one third of the data. With only 2.5-3K personalised examples that incur a data-collection cost of 4-6$, we boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to achieve 45.8% pass@1 on HumanEval.

## introduction recently, powerful close-sourced large langauge models (llms) including chatgpt, gpt-4 have become predominant, accumulating over 170 million users within 5 month of its launch. such closesourced llms demonstrate strong performance in a wide range of tasks, from improving writing proficiency to code generation. however, due to their closed-source nature, concerns have been raised regarding factors such as the availability of these services, high associated costs, concerns on ethics and safety, and potential data privacy implications, all of which limit their seamless integration into real-world applications. in light of these concerns, a natural question arises: can we distill the remarkable abilities exhibited by closed-source llms into smaller open-source llms? researchers have explored such distillation idea @xcite @xcite , by querying chatgpt to generate task instruction and solution pairs, and using the collected data to finetune a student model. however, this standard distillation approach fits different student models to the same data distribution (teacher's prior), disregarding their unique abilities and capacity. in education domain, personalised learning which provides customized learning experience that adapts to student's learning progress and capacity, has proven highly effective and widely adopted @xcite . inspired by such finding, we hypothesize that personalised learning is also beneficial for model distillation. in this work, we propose personalised distillation and empirically evaluate its effectiveness in the domain of code generation. similar to standard distillation, we first employ chatgpt to generate task instructions accompanied by unit test cases. then we follow three steps for personalized distillation as shown in figure 1 . first, we let the student model attempt to solve the task. then, we evaluate the student's attempt with unit test cases and get execution feedback. if the execution feedback contains errors, in the final step we prompt the teacher model (chatgpt) to refine the student's attempt. such data collection process makes the learning experience both interactive -as the student participates to make attempts, and personalised -both the input (tasks) and output (refinement data) are customised to the student. essentially, personalised labeled data help the student to refine its own policy, rather than adopting a new prior of the teacher. with the personalized code data as target outwe collect 10k standard distillation examples and around 2.5-3k personalised examples for pretraining. through zero-shot evaluation on hu-maneval @xcite and mbpp @xcite , we observe that all persd variants consistently outperform their counterparts which use standard distillation. this compelling result strongly validates our hypothesis regarding the advantages of personalized distillation. ablation studies further reinforce our hypothesis, uncovering intriguing properties such as the benefits of multi-round personalized distillation and the ability of our models to leverage execution feedback for self-correction. notably, personalised distillation boosts the state-of-the-art open-sourced pretrain model starcoder @xcite significantlyby 12.2% to achieve 45.8 in pass@1 and 82.3 in pass@100 on humaneval.