PL a D : Preference-based Large Language Model Distillation with Pseudo-Preference Pairs (2024)

Abstract: Large Language Models (LLMs) have exhibited impressive capabilities in various tasks, yet their vast parameter sizes restrict their applicability in resource-constrained settings. Knowledge distillation (KD) offers a viable solution by transferring expertise from large teacher models to compact student models. However, traditional KD techniques face specific challenges when applied to LLMs, including restricted access to LLM outputs, significant teacher-student capacity gaps, and the inherited mis-calibration issue. In this work, we present PLaD, a novel preference-based LLM distillation framework. PLaD exploits the teacher-student capacity discrepancy to generate pseudo-preference pairs where teacher outputs are preferred over student outputs. Then, PLaD leverages a ranking loss to re-calibrate the student’s estimation of sequence likelihood, which steers the student’s focus towards understanding the relative quality of outputs instead of simply imitating the teacher. PLaD bypasses the need for access to teacher LLM’s internal states, tackles the student’s expressivity limitations, and mitigates the student mis-calibration issue. Through extensive experiments on two sequence generation tasks and with various LLMs, we demonstrate the effectiveness of our proposed PLaD framework.

## introduction large language models (llms) have shown remarkable abilities across a wide range of tasks @xcite . however, their huge parameter sizes and computational requirements pose significant challenges 1 work conducted during an internship at google. 2 now in google deepmind. for practical deployment, especially in environments with limited resources. knowledge distillation (kd) has emerged as a technique for addressing these challenges by transferring insights from a large, sophisticated teacher model to a compact student model with reduced memory footprints and inference costs. the seminal work @xcite proposes to train a student model to match the output class distribution of the teacher model. @xcite further extends this idea to the sequence level and teaches the student to directly produce teachers' decoded sequences. another line of work @xcite seeks to align the student model's intermediate-layer representations with the teacher's. all these approaches employ a teacher-forcing strategy, training the student to fully match the outputs or representations of the teacher model. applying conventional kd methods to llms presents several significant challenges. first, those llm teachers are typically only available through api calls. the absence of direct access to the full output logits or internal states of llm teachers hinders the implementation of traditional distillation techniques. second, the capacity gap between the student model and llm teachers becomes significantly larger compared to the previous instances when a relatively smaller teacher model was employed. this disparity exacerbates the student model's limited ability to fully match the teacher llm's output distribution. third, as llms increase in size, they often encounter a mis-calibration issue @xcite where sequences that are highly likely according to the model don't necessarily exhibit high quality for target tasks. consequently, when a student model is trained to mimic these outputs from the teacher llm, it inherits this mis-calibration, leading to suboptimal performance. although some recent studies have enhanced the standard teacherforcing kd paradigm with improved loss functions @xcite @xcite or learning strategies @xcite , these advancements have not yet fully addressed the above challenges, leaving efficient and effective llm distillation as an open research question. in this work, we present preference-based large language model distillation (plad), a novel framework for distilling llm knowledge with preference data. plad is developed based on the following observation: sequences decoded by the teacher model typically surpass the output sequences of the student in quality. by sampling outputs from both the teacher and student, plad generates pseudo-preference pairs and calculates a ranking loss that re-calibrates sequence likelihood on the student side. this innovation acknowledges the complex teacherstudent interaction dynamics within the realm of llms and thus shifts the student's learning focus towards understanding the relative quality of different outputs. without strictly adhering to teacher forcing, we address the student's inherent limitations in expressivity. moreover, the introduction of calibration loss directly ties the quality of generation to its likelihood, allowing for a targeted optimization of output quality through calibration. this strategy also skillfully bypasses the requirement for teacher model internal access and presents an annotation-free method to construct preference pairs based on the inherent capacity gap between teacher and student models. plad is also flexible enough to be applied in scenarios where additional reward models or ranking metrics are available. this versatility makes it a powerful framework for llm distillation. we evaluate plad on anthropic helpful dialogue generation @xcite and reddit tl;dr summarization @xcite . with two different model families llama-2 @xcite and gpt-neo @xcite . the student model learned by our plad framework can outperform the one learned with other state-of-the-art kd methods with respect to the win rate of model generations compared to the target sequences. we also show that plad are universally applicable across model families: from palm2-l @xcite to t5-large @xcite . contributions. the major contributions of this work are summarized as follows: (1) we propose plad, a novel framework that distills llms with preference data; (2) we present a metric-free approach to construct pseudopreference pairs without human annotations; (3) we facilitate the llm distillation with an explicit calibration objective and improve the student model's generation capability; (4) we conduct comprehensive experiments on multiple tasks with different-sized teacher models to demonstrate the effectiveness of plad. learning from preference data. the pivotal reinforcement learning from human feedback (rlhf) framework @xcite first uses preference data to fit a reward model and then fine-tunes the lm to maximize the given reward using reinforcement learning algorithms. in practice, however, using rl to directly finetune lms are challenging, incurring significant computational costs and requiring extensive hyperparameter tuning. to mitigate this issue, dpo @xcite proposes to directly train the policy lm using the pairwise logistic loss without explicitly fitting a reward model. slic @xcite and rrhf @xcite adopt the pairwise hinge loss to train the policy lm. rso @xcite further introduces the statistical rejection sampling method to improve dpo by addressing the distribution drift issue. more recently, lipo @xcite leverages a ranked list of responses (either rated by humans for ranked by models @xcite ) for llm alignment. all these studies require either human annotations or external reward models to obtain the preference data and focus on aligning a single model with human preference. additionally, some works have attempted to explain the distillation mechanisms from multiple perspectives @xcite @xcite @xcite , but none have yet bridged the preference learning and knowledge distillation, while it is a natural approach given the capacity gap between teacher and student models. in this work, we focus on distilling a teacher model into a student model with self-supervised preference pairs. llm distillation. recent efforts in the distillation of large language models (llms) have introduced innovative approaches to refine the knowledge transfer process. @xcite employ a task-aware, layer-wise distillation method which effectively tackles the challenges of tasking student models mimicking the hidden representation of a much larger teacher, but it requires access to the teacher model's intermediate layers, whereas our research focuses on the scenarios where only the teacher model's final sequence-level output is available, which is common in the context of llms. @xcite seek to bridge the capacity gap between student models and teacher models by deploying the mixture of experts (moe) architecture on the student side, thereby increasing its capacity. we alternatively address the capacity gap by shifting from traditional teacher-forcing distillation to a preference-based distillation, leveraging synthetic preference data and avoid memory footprint overhead increased by complex architectures of the student model. another line of works leverages additional knowledge to improve llm distillation. for instance, "distillation step-by-step" by @xcite enriches the student model training by integrating llm output rationales, aiming for a deeper understanding and replication of the teacher model's reasoning pathways. similarly, @xcite advance the methodology by focusing on llm output chains of thought (cot), engaging in per token distribution matching to capture nuanced decision-making processes. @xcite take a specialized approach towards reasoning tasks, proposing a dual-student model framework where one student model is dedicated to problem decomposition and the other to solving the identified subproblems, facilitating a more segmented yet comprehensive distillation strategy. additionally, @xcite also introduce a non-teacher forcing distillation approach, where the teacher output is perturbed to get a proxy teacher with a distribution closer to the ground truth. despite these advancements, none of these works incorporate preference data into the distillation process, highlighting a key contribution of this study. ## conclusions and future work in this work, we introduce plad, a novel preference-based llm distillation framework that leverages pseudo preference pairs to efficiently transfer knowledge from a large llm teacher to a compact student model. by focusing on the relative ranking of outputs, we allow the student model to learn in a way that is both resource-efficient and aligned with the qualitative nuances of language generation tasks. our experiments demonstrate that plad learns a student model that retains a high level of performance, measured by traditional metrics like rouge, and shows significant improvements in terms of generation quality measured by win rate. the ablation studies further underscore the importance of our design choices, particularly the use of pseudo-preference pairs over real reward pairs and the implementation of length normalization techniques. as llms continue to expand their role in various applications, methods like plad that optimize for efficiency and quality without requiring extensive computational resources will become increasingly vital. we hope our work will pave the way for more sustainable and accessible ai language systems in the future.