Lion: Adversarial Distillation of Proprietary Large Language Models (2023)

Abstract: The practice of transferring knowledge from a sophisticated, proprietary large language model (LLM) to a compact, open-source LLM has garnered considerable attention. Previous works have focused on a unidirectional knowledge distillation way by aligning the responses of the student model with those of the teacher model to a set of instructions. Nevertheless, they overlooked the possibility of incorporating any “feedback”–identifying challenging instructions where the student model’s performance falls short–to boost the student model’s proficiency iteratively. To this end, we propose a novel adversarial distillation framework for a more efficient knowledge transfer. Leveraging the versatile role adaptability of LLMs, we prompt the teacher model to identify “hard” instructions and generate new “hard” instructions for the student model, creating a three-stage adversarial loop of imitation, discrimination, and generation. By applying this adversarial framework, we successfully transfer knowledge from ChatGPT to a student model (named Lion), using a mere 70k training data. Our results show that Lion-13B not only achieves comparable open-ended generation capabilities to ChatGPT but surpasses conventional state-of-the-art (SOTA) instruction-tuned models like Vicuna-13B by 55.4% in challenging zero-shot reasoning benchmarks such as BIG-Bench Hard (BBH) and 16.7% on AGIEval.

## introduction large language models (llms) capable of following natural language instructions have exhibited tremendous success in generalizing zero-shot to new tasks @xcite . due to various concerns, the most advanced llms, such as chatgpt @xcite and @xcite ) that boasting billions of parameters, are figure 1 : an illustration of the distinction between our approach and earlier ones. previous methods facilitate a one-way knowledge transfer from the teacher to the student (solid arrow). our approach, however, incorporates an innovative step (dashed arrow) that completes a loop: it enables the feedback"-identifying the student model's weaknesses-to be relayed back to the teacher, in order to foster tailored learning. however, previous works employ a unidirectional approach to knowledge transfer (solid arrow in figure 1 ), where the teacher imparts knowledge to the student without considering any "feedback". to better illustrate this using a tangible classroom scenario, the "feedback" refers to identifying the "hard" examples or problems where the student's performance falls short. this feedback guarantees that the teacher can provide bespoke training that centers on "hard" examples, thereby paving the way for more effective and tailored learning experiences for the student. inspired by adversarial knowledge distillation (akd), which aims to iteratively improve the student model's performance by learning from generated hard samples @xcite @xcite , we propose an adversarial framework for distilling a proprietary llm into a compact student model. nevertheless, these akd methodologies necessitate accessibility to the weights or gradients of the teacher model, which cannot be directly adapted to our setting. to circumvent this problem, we leverage the unparalleled role adaptability of llms, which can be effectively employed through a diverse range of prompts @xcite . in particular, we prompt the proprietary teacher llm to serve as a "referee" to discriminate hard instructions where there exists a significant performance discrepancy between the teacher's and student's responses, and serve as a "generator" to produce new instructions that emulate the data distributions corresponding to the discriminated hard instructions. our framework, as depicted in figure 2 , consists of three stages in an iteration: 1) an imitation stage to align the student's response with the teacher's response; 2) a discrimination stage to identify hard instructions; 3) a generation stage to produce new hard instructions for escalating the challenges presented to the student model. in essence, our adversarial framework forms a positive feedback loop that efficiently bootstraps the student model's proficiency. to verify the efficiency and efficacy of our method, we apply our akd framework to transfer the knowledge of chatgpt 2 onto an open-source foundation llm, known as llama @xcite . we select alpaca's training data (generated from only 175 manually selected seed instructions) as the initial training instructions and execute three iterations of akd, resulting in a total of 70k data that our model is trained on. we've christened our model as lion, drawing inspiration from the art of "distillation". by conducting extensive exper-iments on open-ended generation and reasoning datasets, which include a total of 40 sub-tasks, our lion-13b showcases superior performance surpassing instruction-tuned baseline models such as vicuna @xcite . our main contributions are as follows: ## methodology harnessing the learned knowledge of a sophisticated teacher model t (x; θ t ) where the parameter θ t is inaccessible, our goal is to craft a more lightweight student model s(x; θ s ). ideally, a student model is optimal if the expectation of model discrepancy (which indicates the prediction differences between teacher t and student s) on the uniform data distribution is minimized. inspired by the success of adversarial knowledge distillation (akd) @xcite @xcite , we turn to optimize an upper bound of the expectation -the expectation of the model discrepancy on "hard samples", where the teacher t and the student s have a relatively large performance gap. these "hard samples" are inclined to dominate the expectation of the model discrepancy. thus, the overall expected model discrepancy can be effectively and efficiently reduced by optimizing the student model s on these "hard samples". the underlying rationale is rather straightforward and can be analogized to a realworld educational scenario: continuously concentrating on the "hard" knowledge that the student finds challenging to grasp is the most effective manner of enhancing a student's proficiency. however, in the process of training the student model s, hard samples will be mastered by the student and converted into easy samples. hence we need a mechanism to continuously generate hard samples, which can be achieved by an adversarial framework. the whole framework of our adversarial knowledge distillation is depicted in figure 2 , which contains three stages in an iteration: 1) an imitation stage to align the student's response with the teacher's response; 2) a discrimination stage to identify hard samples; 3) a generation stage to produce new hard samples for escalating the challenges presented to the student model. ## analyses 6.1 ablation studies the threshold τ for distinguishing between hard and easy instructions we systematically explored τ ranging from 0.0 to 2.0 and documented its influence on average performance across three datasets. the ratio r of generated hard and easy instructions we change the ratio of generated hard instructions to generated easy instructions from 1:0 (all hard) to 0:1 (all easy) and investigate its impact on average performance across three datasets. it can be seen from ## conclusion this paper presents an innovative adversarial knowledge distillation framework for distilling a proprietary llm into a compact, open-source student model. while previous methodologies have concentrated on unidirectional knowledge transfer, our approach seeks to integrate "feedback" into the learning process. leveraging the versatile role adaptability of llms, we prompt the proprietary model to identify "hard" instructions and generate new "hard" instructions for the student model, creating a three-stage adversarial loop of imitation, discrimination, and generation. this approach al-lows us to refine the student model's performance iteratively, efficiently bootstrapping its proficiency. we aspire that our model, named lion, may serve as a baseline to reflect the performance of chatgpt, especially the open-source instruction-following language model baseline for our community.