ELAD : Explanation-Guided Large Language Models Active Distillation (2024)

Abstract: The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve the efficiency of sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in reasoning explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model’s reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLMs knowledge distillation.

## introduction the advancement of llms @xcite @xcite has significantly impacted the natural language processing (nlp) domain, showcasing excellent in-context learning and complex reasoning capabilities. yet, the deployment of these models is hindered by their extensive parameter count, leading to significant computational demands and financial burdens. for instance, deploying llms with 100-200 billion parameters would require a cluster of nvidia a100 gpus, where each gpu costs @xmath025 per hour, which may lead to a staggering $18, 000 monthly if operated non-stop foot_0 . this financial barrier makes it impractical for many institutions and research labs to adopt llms widely, especially in resource-constrained environments @xcite , limiting their audience and application scope. additionally, relying on api calls to access pre-trained llms, e.g., gpt-4, also presents its challenges, including high usage fees, inability to run models locally for customization or fine-tuning, potential data transmission issues, and privacy concerns @xcite . recent research on llms knowledge distillation @xcite enables smaller models to achieve performance similar to llms by transferring reasoning capabilities to them, making them more computationally efficient. @xcite , @xcite demonstrate the training of smaller models using pseudo-labels generated by llms, wherein llms act as "teachers" to supervise the fine-tuning of these "student" models. recent works @xcite @xcite @xcite @xcite focus on multi-task fine-tuning of student models. they utilize chain-of-thought (cot) reasoning @xcite to generate both explanations and final answers generated by llms as pseudolabels to jointly supervise small model fine-tuning. however, a major issue with existing fully supervised learning methods is that they do not sense whether the knowledge has been sufficiently distilled into the small model. insufficient distillation can lead to suboptimal performance of the small model, while excessive distillation may incur unnecessarily high costs. to address this challenge, we take llms as an agent that guides small language models toward progressive improvement. throughout this process, the llms (teacher) can sense the weaknesses of the small language model (student) and customize its teaching accordingly. formally, we propose an explanation-guided llms active distillation (elad) framework that significantly enhances active learning through the explanations generated by language models. in each iteration, the framework encompasses a student reasoning task and a teacher reasoning task: the student model identifies samples it struggles to predict accurately and reasonably; subsequently, for these selected samples, the teacher model reviews the student's explanations, correcting any erroneous reasoning. however, first, it is nontrivial to tackle the student task. current sample selection methods typically focus on finding samples with the wrong predictions @xcite @xcite , but even if the prediction is correct, the reasoning process can be wrong or flawed. the selection of samples with bad reasoning goes beyond it and is yet to be well explored, which requires the student model to faithfully self-inspect its step-by-step explanation of its prediction and locate the flaw. to address this, we propose a novel explanation-guided sample selection method that identifies the samples that trouble its reasoning by exploiting explanation stepwise uncertainties. second, how the teacher senses and corrects the flaws in student model reasoning is also a challenging problem. merely prompting teacher and student to generate their respective explanations separately and compare their difference is problematic because a prediction could be led by different reasoning processes and different explanations (e.g., rashomon effect @xcite ). we need the teacher model to check the student model's explanation, locate the problem within its reasoning, and correct it, which is not well explored. to accomplish this, we propose a customized llmannotated explanation revision technique. it entails sequentially prompting the llm with the explanation from the small model and then asking it to assess whether the current step is reasonable or necessitates revision. we evaluate our framework across six reasoning benchmarks, comparing it against existing sample selection methods for active learning and llms explanation and answer generation methods. our findings indicate that the proposed framework notably enhances annotating efficiency. we summarize our main contributions as follows: a) an explanation-guided llms active distillation framework that enhances active learning, guided by explanations from small models, during the distillation of llms to smaller models. b) an explanation-guided sample selection method that identifies the samples that trouble the reasoning of language model by exploiting explanation stepwise uncertainties. c) a customized llm-annotated explanation revision technique that allows llm to teach customized knowledge by guiding the llm to pinpoint and correct inaccuracies in the reasoning steps of small models. d) extensive experiments demonstrate that the proposed framework can improve annotating efficiency. ## problem setup in the context of knowledge distillation of llm t , we address the problem of efficiently annotating an unlabeled question set u. this involves strategically selecting a subset q from u, where q ⊂ u and |q| < |u|. our study's primary goal is to enhance the performance of a smaller model s through fine-tuning with a training dataset @xmath1, âi , ri )} |q| , where the llm annotates an answer â and a corresponding explanation r for each question @xmath2, forming d. the objective is to achieve a performance with s that is comparable to that of t while minimizing the size of the annotated dataset q. this approach aims to maximize the efficiency of the small model and minimize the amount of annotated data required from the llm. ## methodology in this section, we first present the overview of our proposed explanation-guided llms active distillation framework. we then proceed to present a novel explanation-guided sample selection method. lastly, we present a customized llm-annotated explanation revision method. ## experiments datasets the experiments are conducted on six well-known benchmark datasets across 3 different reasoning tasks: gsm8k @xcite and aqua @xcite for arithmetic reasoning tasks, anli @xcite ) and e-snli @xcite for natural language inference (nli) tasks, and strategyqa @xcite and commonsenseqa @xcite for common sense reasoning task. further details are provided in the appendix a. evaluation metric to assess question-answering performance for the above reasoning tasks, we calculate the accuracy based on the final answers given by the student small model. setup we use gpt-3.5-turbo as a teacher via openai api. we use llama-2-7b @xcite as our student model. further implementation details are provided in the appendix a. ## conclusion this paper introduced the explanation-guided llms active distillation framework to address the challenges of deploying llms due to the high memory and computational demands. our proposed framework provides an efficient distillation of llms, incorporating an active learning method. this includes an explanation-guided sample selection technique and a customized llm-annotated explanation revision method tailored for active learning. extensive experiments on various reasoning datasets demonstrate the effectiveness of our approach in enhancing the distillation efficiency.