Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models (2025)

Abstract: Aligning small language models (SLMs) with human values typically involves distilling preference knowledge from large language models (LLMs). However, existing distillation methods model preference knowledge in teacher LLMs by comparing pairwise responses, overlooking the extent of difference between responses. This limitation hinders student SLMs from capturing the nuanced preferences for multiple responses. In this paper, we propose a Preference-Aligned Distillation (PAD) framework, which models teacher’s preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals. Our insight in developing PAD is rooted in the demonstration that language models can serve as reward functions, reflecting their intrinsic preferences. Based on this, PAD comprises three key steps: (1) sampling diverse responses using high-temperature; (2) computing rewards for both teacher and student to construct their intrinsic preference; and (3) training the student’s intrinsic preference distribution to align with the teacher’s. Experiments on four mainstream alignment benchmarks demonstrate that PAD consistently and significantly outperforms existing approaches, achieving over 20% improvement on AlpacaEval 2 and Arena-Hard, indicating superior alignment with human preferences. Notably, on MT-Bench, using the Gemma model family, the student trained by PAD surpasses its teacher, further validating the effectiveness of our PAD.

## introduction recently, small language models (slms) have demonstrated impressive performance across a variety of tasks @xcite ## background this section reviews two topics: 1) preference modeling in preference learning theory, and 2) the generation process of language models under the reinforcement learning framework. preference modeling given a prompt @xmath0, the language model π generates pairs of responses (y 1 , y 2 ) ∼ π(y | x). a possible preference can be denoted as y 1 ≻ y 2 | x, where y 1 and y 2 represent the preferred and dispreferred responses. preferences are assumed to be generated based on a reward model r(y | x), which assigns a continuous reward r to each response y. for simplicity, we omit x and use r(y) to denote r(y | x). the pairwise preference probability p(y 1 ≻ y 2 | x) can be modeled using the bradley-terry (bt) framework @xcite as follows: text generation as a markov decision process (mdp) the text generation process can be modeled as an mdp, which is represented by the triple (s, v, u) 2 , where the state space s represents all possible partially generated sequences, and the action space v corresponds to the vocabulary in the language model. at each step t, an action y @xmath1. ## self-derived log-likelihood rewards this section introduces how we derive a reward function from language models without any reference model, providing the theoretical foundation for the framework proposed in the next section. ## pad: preference-aligned distillation this section outlines our pad, which involves three key training phases ( §4.1-4.3), followed by the introduction of a preference decomposition strategy to accelerate the training process ( §4.4). ## conclusion in this paper, we introduced the preference-aligned distillation (pad) framework, which models the teacher's preference knowledge as a probability distribution over all potential preferences. this supervisory signal enables the student model to capture subtle distinctions between responses. experimental results on the gemma-2 and llama-3 model families show that pad outperforms both traditional knowledge distillation and existing preference distillation methods across four benchmark tasks, highlighting its capacity for learning in-depth human preferences.