Debiasing Online Preference Learning via Preference Feature Preservation (2025)

Abstract: Recent preference learning frameworks for large language models (LLMs) simplify human preferences with binary pairwise comparisons and scalar rewards. This simplification could make LLMs’ responses biased to mostly preferred features, and would be exacerbated during the iterations of online preference learning steps. To address these challenges, we propose a novel framework coined PFP (Preference Feature Preservation). The key idea of PFP is maintaining the distribution of human preference features and utilizing such rich signals throughout the online preference learning process. Specifically, PFP first extract preference features from offline pairwise human preference data and trains a feature classifier. Then, using trained classifier and the distribution preserving optimization, PFP maps appropriate preference features for a new input instruction during online learning. Lastly, PFP trains LLM using the existing preference learning method, by incorporating the preference feature into system prompts and enabling LLM to explicitly handle various human preferences. Our experiments demonstrate that PFP successfully mitigates the bias in preference features during online learning, and hence achieves superior performance compared to previous preference learning methods on standard benchmarks to evaluate LLM alignment.

## introduction aligning large language models (llms) using human feedback, particularly by learning from human preferences, yields remarkable successes in various nlp tasks and real-world applications such as coding assistants and chatbots @xcite @xcite . to improve the alignment of llms, various preference learning algorithms, such as reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite , have been explored. a common assumption across these works is that human preference is provided in a binary pair-wise comparison @xcite . this approach enables easy modeling of human preference using the scalar reward such as the bradley-terry (bt) model @xcite . however, human preference is determined by several underlying features @xcite , and hence such simplification has critical limitations and fails to capture the complexity of human preferences. for example, even though human preference can be varied depending on the preference feature, the most dominant preference feature would be only considered to determine the binary human preference label. this issue becomes even more problematic in online preference learning scenarios, which progressively improves the alignment of llms by iterating the generation of preference data and learning from them @xcite @xcite . during online preference learning, llm will generate responses biased toward specific preference features, and the preference annotators, such as the external reward model @xcite , will provide positive feedback on this. as such iterations go on, the bias of llm accumulates (see fig. 1(a) ), and hence it results in the reduced diversity and quality of llm's responses. contribution. to address these challenges, we propose a novel online preference learning framework called pfp (preference feature preservation). our approach is to ensure that the distribution of preference features remains consistent throughout the online preference learning process. here, the key idea is to explicitly extract preference features of each input instruction and handle them using system prompts of llms (see with intent. specifically, pfp first estimates the initial distribution of preference features of the given human preference dataset, by inferring which features mainly determine binary human preferences. we then train a preference feature classifier, which maps each input instruction to appropriate preference features with additional optimization for the distribution preservation, during the online learning process. finally, pfp trains llm using the existing preference learning framework, by converting the mapped preference features of each generated data into the system prompts of llms. we demonstrate the effectiveness of the proposed pfp by applying it to align recent opensourced llms with the commonly used preference dataset, ultrafeedback @xcite . the experimental results demonstrate that pfp successfully prevents llms being biased to specific preference features during online learning, and results in the improved alignment in various aspects (fig. 1(c )). for example, our framework achieves 7.66% increase in alpacaeval 2.0 length-controlled win rate compared to the sft model. also, pfp achives 2.11% larger increase compared to iterative dpo, an online preference learning method with external reward model. in addition, unlike other baselines for online preference learning, pfp successfully improves harmlessness and honestness of the responses @xcite through multiple iterations. more interestingly, pfp exibits additional advantage that reduces the occurrence of length bias during online preference learning, despite not being specifically designed to address this. overall, these results demonstrate that pfp is highly effective and practical for real-world appli-cations, and underscoring the importance of debiasing to learn human preference for llm alignment. ## preliminary let the llm policy be denoted as π θ , which can generate output sequence (i.e. response) y, given input sequences composed of system prompt s and instruction x, i.e., y ∼ π θ (s, x). here, the system prompt s is usually considered to be fixed regardless of the input instruction x. for convenience, we assume that s is always included as the input of π θ and hence omit s in the equations in the below parts. next, we assume that we have the labeled preference dataset, @xmath0, y l , y w )}, where y l and y w are the dis-preferred and preferred responses for the corresponding instruction x, respectively. rlhf and dpo. to train π θ with d for the alignment, rlhf first introduces the reward model r(x, y) which can convert human preference data into scalar values. specifically, the reward model r(x, y) is often modeled with the bradley-terry (bt) model @xcite , and then it can yield the probability p(y w ≻ y l | x) that response y w is preferred over y l as follow: as the optimal reward function r(x, y) is not accessible, a parameterized reward model r ϕ (x, y) is usually introduced by optimizing its parameters with the maximum-likelihood objective on the preference dataset. with this reward model, rlhf optimizes llm π to maximize this reward with the additional regularization of the kl divergence between the current policy and the reference policies (π ref ) to prevent reward over-optimization: to remove the necessity of the reward model in rlhf, dpo proposed a method that is mathematically equivalent to the original rlhf objective and can directly optimize the internal reward modeled by llm π itself, by maximizing the weighted likelihood gap between y w and y l : online preference learning and spa. in the online preference learning scenario, we have unlabeled instruction datasets x @xmath1, @xmath2, ..., t where x t ∩ x t ′ = ∅ when t ̸ = t ′ . for t-th iteration, the preference dataset d @xmath3, y l , y w )|@xmath4, i.e., y 1 , y 2 ∼ π t-1 (x), and (2) judging the preference between them. then, llm policy π t which is initialized with π t-1 is trained with d t using the existing preference learning method. one representative approach is iterative dpo @xcite , where the external reward model is used for the preference judgments and π t is trained with d t using dpo. since choosing the proper reward model is non-trivial in our problem, we adopt spa @xcite as the online preference learning algorithm. spa conducts preference labeling using the implicit reward derived from the dpo's objective function (eq. 1), unlike the other online preference learning methods using the external reward model: ## pfp: debiased llm alignment via preference feature preservation in this section, we present pfp ( preference feature preservation) to align llm by reducing the bias during online preference learning. our main idea is to explicitly extract preference features of input instruction, and handle them using system prompts of llm. the overview of pfp is presented in fig. 2 . ## conclusion we propose pfp, a novel framework that explicitly preserves preference features during the online preference learning to mitigate potential bias. we demonstrate that incorporating preference features from human feedback into system prompts and preserving the feature distribution over each iteration of online learning effective in preventing bias. this approach not only aligns human preferences more effectively than the existing methods but also eliminates length bias and undesired preference feature biases, while uniformly improving various factors such as harmlessness and honestness.