da DPO : Distribution-Aware DPO for Distilling Conversational Abilities (2025)

Abstract: Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation (KD) with Direct Preference Optimization (DPO) has emerged as a promising approach to enhance the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on “black-box” KD, which only uses the teacher’s responses, overlooking the rich distributional information within the teacher’s probability distribution. This paper addresses this gap by introducing daDPO (Distillation-Aware DPO), a novel framework that integrates the teacher’s distributional information into DPO distillation while preserving theoretical guarantees. Our framework offers a unified objective that enhances both preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller models within the same LLM family. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate), and allows Qwen2.5-1.5B to occasionally outperform its 7b teacher model (14.0% win rate).

## introduction large language models (llms) have demonstrated remarkable success across various applications @xcite @xcite @xcite @xcite . however, a key challenge in their deployment is the sharp decline in conversational ability as model size decreases. as * zhang and wang contributed equally to this work: zhang leads experiments and refined the methodology, while wang originates the idea and leads the theoretical proofs. all authors contributed to the writing, discussion, and overall development of the paper. the increasing availability of powerful opensource llms presents a unique opportunity to explore white-box distillation with broad practical implications. for instance, small models of various sizes can be developed through pruning or quantization of llm, followed by knowledge distillation @xcite . by doing so, one can avoid the need to build small models by from scratch, significantly reducing development costs. white-box distillation is potentially useful as shown in knowledge distillation literature for text classification @xcite , image classification @xcite , llm @xcite , however, there is still a research gap in leveraging white-box distillation to enhance the conversational ability of small language models. specifically, existing studies primarily focus on either supervised knowledge distillation @xcite or online reinforcement learning @xcite . the former is suboptimal for aligning conversational ability @xcite , whereas the latter-though adaptable for alignment (e.g., by training the reward model) -is more computationally expensive and challenging to optimize compared to dpo @xcite . to address this gap, we focus on the critical research question: "how can we adapt dpo-based distillation to incorporate the teacher's distribution for enhancing the student's conversational ability?" our preliminary results suggest that directly combining the kullback-leibler (kl) divergence between the student's and teacher's distributions-a common metric in knowledge distillation-into the final dpo loss does not yield optimal results. instead, we begin by incorporating the kl metric into the rl objective. our goal is to maximize the reward that favors the teacher's response over the student's response, while also regulating the student's distribution (over its own response) more closely with that of the teacher. we then derive a new dpo-style loss for distribution-aware dpo (termed dadpo) from this rl objective. additionally, we explore further considerations, such as the role of the reference model in dpo, the size of the teacher's model, and the possibility to distill across llm families. our contributions: 1) novel integration framework: dadpo offers a unified objective that simultaneously facilitates preference optimization and "white-box" kd; 2) theoretical analysis: we pro-vide theoretical analysis and show how we should balance between training stability and the "whitebox" kd objective. finally, 3) empirical validation: we evaluate the effectiveness of dadpo with various scenarios and empirical analysis. language model distillation knowledge distillation (kd) is a technique for training a smaller model (the student) by leveraging knowledge from a larger model (the teacher). in the era of llms, kd is widely used to reduce the computational demands of large models by developing smaller llms under the guidance of a larger teacher model. recent kd methods for llms can be broadly categorized into two types: black-box kd, where only the teacher-generated texts are accessible, and white-box kd, where the teacher model's output distribution or intermediate hidden states are also available. most existing kd approaches for llms fall under black-box kd, including @xcite @xcite @xcite @xcite . white-box kd methods have been comparably less explored @xcite . our work also focuses on the white-box kd setting. however, instead of using online rl like @xcite , we aim to enhance conversational capabilities through a simple yet effective method based on the dpo framework. ## preliminaries this paper investigates the problem of enhancing a student model π st under the guidance of a teacher model π te . unlike @xcite , we consider the white-box setting, where the teacher model's distribution, π te , is fully accessible. specifically, for any given context x and token t in the vocabulary space, we can directly obtain the probability π te (t|x) from the teacher model. ## conclusion in this work, we developed distribution aware dpo (dadpo), a novel framework that integrates the rich distributional information of white-box teacher llms into the direct preference optimization (dpo) distillation process. our approach offers several advantages, including more stable "white-box" distillation, the potential for cross-llm family distillation, and the ability to maintain dpo's simplicity and theoretical guarantees. our experiments on pruned models (vicuna-7b) and smaller models in the qwen2.5 series demonstrated that dadpo consistently outperforms existing baselines, achieving significant improvements in conversational performance without compromising downstream task capabilities (compared to ddpo). this work provides a flexible and theoretically grounded approach to knowledge distillation, enabling more efficient deployment of smaller, yet highly capable, language models in resourceconstrained environments.