GKT : A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment (2024)

Abstract: The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands. Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users. Furthermore, present techniques for expediting inference and reducing costs operate independently. To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. This approach leverages a larger LLM as a ”teacher” to create guidance prompts, paired with a smaller ”student” model to finalize responses. Remarkably, GKT requires no fine-tuning and doesn’t necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can be seamlessly integrated into cloud-edge collaboration architectures, and is versatile enough for plug-and-play application across various models. It excels in both efficiency and affordability, epitomizing a ”cheap and cheerful” solution. GKT achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement of 14.00 % along with a 7.73 times speed-up in CSQA. When utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost. The results highlight substantial enhancements in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the performance of using either the student or teacher models in isolation.

## introduction the swift advancement of large language models (llms) has dramatically pushed the frontiers of ai technology. llms, with their vast number of parameters, are exceptionally adept at comprehending human intentions, offering high-quality reasoning, and responses @xcite @xcite . however, the immense size of these models is a double-edged sword. while it improves model performance, it also leads to slower inference times and higher computational costs. as the demand for llm usage increases, relying solely on llms for auto-regressive inference actually demands an overwhelming amount of computational resources and time, posing a substantial deployment challenges for cloud services and resource-constrained devices. consequently, many recent studies @xcite @xcite have taken steps to improve the inference efficiency of llms. one prevalent and widely adopted approach is knowledge distillation @xcite @xcite @xcite . the majority of knowledge distillation frameworks utilize large language models as "teacher" models to generate training samples. these samples are then used to train more compact "student" language models, effectively teaching them to mimic the performance and capabilities of teacher model. however, this process still demands a carefully crafted data generation mechanism and the subsequent training of the student model. despite being smaller in size, to achieve satisfactory results in general tasks, many student models still maintain a considerable parameter size, often around 7b @xcite and 13b @xcite . this is in line with the observations by @xcite , who noted that the emergent abilities of llms for language understanding typically become evident when the model size exceeds 10 billion parameters. feature kd sd gkt accelerates inference ✓ ✓ ✓ preserves model architecture ✓ ✓ ✓ eliminates additional fine-tuning ✗ ✓ ✓ allows custom settings ✗ ✗ ✓ enables cloud-edge collaboration ✗ ✗ ✓ allows different vocabulary ✗ ✗ ✓ to solve this limitation, another research line focuses on speculative decoding @xcite @xcite , which involves using a more efficient, smaller model to generate token predictions, which the larger target model then evaluates. if a token prediction is accepted, it's used; if not, it's discarded, and the target model generates a new token. the method is shown to significantly speed up inference without needing changes to the model's architecture or training procedures. however, users often have diverse generation requirements, such as the desire to adjust generation parameters like temperature and top_p. while speculative decoding can ensure consistency with the output of the final large model, employing batch generation to save time during high concurrent access can impede the abil-ity to meet user-specific generation settings. considering the limitations of these methods, we were inspired by the common human experience of "getting started is the hardest part." and sociological studies @xcite showing that effective prompts provided by teachers in classrooms can significantly improve student performance in exams. thus, we propose a novel knowledge transfer framework: guidance-based knowledge transfer (gkt). our framework involves two steps: firstly, using an llm as a teacher model to generate guidance prompts from concurrent user inputs through batch generation. secondly, a smaller lm acts as the student model, which simply completes the answers based on the guidance prompts, allowing for user-customized generation settings. gkt reduces the burden of llm inference, thereby speeding up response generation. unlike knowledge distillation, our framework does not require generating dataset from teacher model and fine-tuning the student model. it also doesn't necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. gkt can also be seamlessly integrated into cloud-edge collaboration architectures as shown in figure 1 . gkt deploys the larger teacher model on remote cloud servers and the smaller student model on lightweight mobile devices, such as smartphones. this setup allows for brief guidance prompts to be easily transmitted to mobile devices, significantly reducing data transmission costs. in cloud-edge collaboration, users can also perform simple personalized generation settings on their mobile devices. the finally, our study follows the philosophy that "no such thing as bad student. only bad teacher." we conducted various experiments to explore the optimal guidance strategies for the teacher model, seeking answers to questions such as how much and what type of help a teacher should provide to maximize the student model's accuracy while minimizing the teacher model's inference time. the results demonstrate that gkt not only achieves a significant accuracy improvement of 14.18% on the gsm8k dataset, but also enhances speed by 10.72 ×. furthermore, on the csqa dataset, it records a noteworthy accuracy increase of 14.00% along with a 7.73 × acceleration in inference speed. ## method the schematic representation of our guidance-based knowledge transfer (gkt) framework is depicted in figure 2 . gkt encompasses a two-step process: guidance generation and response completion. during the guidance generation phase, a large language model serves as the "teacher" model. this model processes concurrent user inputs and employs batch generation to craft guidance prompts. subsequently, in the response completion stage, a smaller language model functions as the "student" model. this model offers flexibility in generation settings, allowing for user customization. the guidance prompt created in the first stage is then fed into this smaller model, facilitating the completion of the response with enhanced efficiency. we will now elaborate on each stage: guidance generation given concurrent user inputs @xmath0, q 2 , ..., q n }, where q i represents the input question from user i, the teacher model m t batch generates the guidance prompts: here, g represents the batch-generated guidance, g i is the guidance for user i, and f(•) denotes the projection operation that generates the guidance prompt from the generated text. in this paper, we explore different projection operations including: ## experimental settings datasets in this paper we use two challenge but widely used dataset: gsm8k @xcite for arithmetic reasoning and csqa @xcite for commonsense reasoning. gsm8k focus on arithmetic reasoning which is a collection of grade school math word problems, each requiring 2 to 8 steps to solve. the solutions mainly involve a series of basic arithmetic calculations to arrive at the final answer. csqa focuses on commonsense question answering which includes multiple-choice questions that require commonsense knowledge for answering. the detailed dataset statistics can be found in appendix a. in all our experiments, we use the same prompt settings in manual-cot @xcite otherwise stated. the full prompt can be found in appendix e hyperparameter all experiments were conducted on an nvidia a800 gpu. detailed hyperparameter settings for the experiments are provided in appendix b. in our experiments, we tested various teacher models, including flan-t5-xl @xcite , bloom-7b @xcite , llama2-70b @xcite and llama2-13b. correspondingly, the student models used were flan-t5large, bloom-3b, llama2-13b and llama2-7b. ## results and exploration in this section, we delve into the empirical results of our comprehensive analysis using the gkt framework. the essence of this exploration lies in quantifying the effectiveness of gkt in enhancing the accuracy of student models while ensuring computational efficiency. we first report the overall results for the gkt on the gsm8k and csqa datasets. then, we explore various dimensions of knowledge transfer including the optimal guidance generation methods, the intriguing dynamics between different types of teacher and student models, the influence that a teacher can exert on student and the optimal guidance length. ## related work the aspect most relevant to our work is knowledge distillation @xcite @xcite @xcite . knowledge distillation involves condensing the expertise from larger models into more compact versions, thereby ensuring these smaller models retain high efficiency while still achieving impressive performance. @xcite introduces symbolic chain-of-thought distillation (scotd), which trains a smaller "student" language model using the outputs (reasoning chains) of a larger "teacher" model. the teacher model first generates multiple reasoning chains for a given task, showcasing step-by-step problem-solving. the student model then are fine-tuned on these examples, essen- tunstall et al. (2023) presents a distillation framework named distilled direct preference optimization (ddpo) result in a 7b model named zephyr. the method comprises three steps: (1) distilled supervised fine-tuning (2) ai feedback through preferences and (3) distilled direct preference optimization. different from knowledge distillation, our work focuses on leveraging the knowledge of larger models to enhance the overall efficiency and performance. the proposed gkt framework circumvents the usual requirements of producing a distillation dataset from a teacher model or fine-tuning the student model, streamlining the knowledge transfer process. another line of study that is related to our work is speculative decoding @xcite . speculative decoding uses two models: the original target model and a much smaller approximate model. the smaller model handles autoregressive sampling, while the larger assesses the output. simple tokens are generated by the smaller model, with complex tokens handled by the larger. based on speculative decoding, @xcite proposed retrieval-based speculative decoding (rest) which combines speculative decoding with retrieval techniques. instead of using a smaller lm for draft generation, rest bypasses the need for an additional small lm by retrieving draft tokens from a pre-built datastore containing contextcontinuation pairs. these drafts are then verified by a large lm. recently, @xcite proposed skeleton-of-thought (sot). sot first guides llms to first create a concise "skeleton" of an answer and then fill in each point of the skeleton in parallel, speeding up the response process. @xcite introduces super in-context learning (supericl), a method enhancing the performance of llms by integrating them with smaller, locally fine-tuned models. these smaller models, acting as plug-ins, provide task-specific knowledge and predictions. the process involves fine-tuning a small model on task-specific data, using it to generate predictions and confidence scores for in-context examples, and combining these with the llm's general language understanding. while supericl also integrates llms and lms, it primarily focuses on enhancing model performance without considering the efficiency of the framework. this focus on performance enhancement can even negatively impact overall efficiency. whereas, gkt focuses on finding the optimal balance between efficiency and effectiveness ## conclusion we introduce the innovative guidance-based knowledge transfer (gkt) framework, designed to leverage the knowledge of larger models to enhance the efficiency and performance of smaller models, while maintaining the flexibility of personalized generation settings, allowing users to freely adjust the generation parameters. the unique collaborative framework of gkt seamlessly integrates into cloud-edge architectures, deploying smaller models on edge devices to minimize data transmission delays and expedite response generation. our results demonstrate remarkable improvements: a maximum accuracy increase of 14.18% and a 10.72× speed-up on the gsm8k dataset, and a 14.00% accuracy enhancement with a 7.73× speed increase on the csqa dataset. moreover, when utilizing chatgpt as teacher model and llama2-70b as the student model, we can achieve 95.00% of chatgpt's performance at 52% of the cost. gkt signify major strides in performance metrics, combining accuracy with computational speed-ups -all wrapped up in a "cheap and cheerful" package.