BAPO : Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization (2024)

Abstract: While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups.

## introduction large language models (llms) @xcite have been successfully aligned with human preferences across various applications, ranging from summarization tasks to enhancing reasoning capabilities @xcite @xcite . this alignment process involves collecting human feedback by presenting pairs of responses generated from the same user prompt and asking users to choose their preferred response @xcite @xcite . the llms learn from this preference data to produce responses that better match human preferences, effectively addressing the challenge of converting complex human expectations into tangible training objectives @xcite @xcite . known as preference optimization, this approach has become essential in the final stages of llm training @xcite @xcite . however, the common assumption in preference optimization is that all users share a uniform set of general preferences @xcite @xcite , leading llms to align with an average of these preferences, as derived from collective feedback data @xcite @xcite . while effective for broadly accepted preferences like helpfulness and harmlessness, this approach does not account for the diversity of individual preferences in realworld scenarios @xcite @xcite . for example, given the same context, one user might prefer a humorous response, while another might prefer a concise one. this reliance on averaged preferences often fails to capture the unique preferences of each user. this is known as the condorcet paradox @xcite in social choice theory, where no single response consistently satisfies all users, leading to non-transitive preferences @xcite . recent studies have begun to tackle this challenge by fine-tuning instruction-tuned llms for personalized alignment @xcite @xcite . although these personalized preference optimization approaches enable support for diverse user preferences @xcite @xcite , the impact of learning to meet personalized preferences on previously acquired knowledge @xcite , such as global knowledge @xcite and general alignment @xcite , remains underexplored. in this work, we systematically analyze how personalizing llms according to diverse user preferences impacts their global knowledge and general alignment. our findings reveal that the extent of knowledge loss heavily depends on these preferences, often inducing significant declines how much water should i put into the melon to make up a watermelon? in specific areas of knowledge. this suggests that the conventional kullback-leibler (kl) constraints between the policy model and the reference model @xcite , which is based solely on the tokens appearing in preferred or dispreferred responses @xcite @xcite , fail to prevent the forgetting that occurs during personalized preference optimization. to address this issue, we start by analyzing the initial responses, referred to as base responses, of instruction-tuned models to the given prompts and observe how their likelihood of producing these responses changes over training steps. we discover that personalizing preferences to enhance the distinction between preferred and dispreferred responses not only diminishes the likelihood of producing the dispreferred responses, but also lowers the likelihood of generating these base responses. we hypothesize that aligning the reference and policy models, especially focusing on the tokens that appear in the base response, is essential for preserving global knowledge and ensuring general alignment. to this end, we introduce a novel preference optimization method named as base-anchored preference optimization (bapo). bapo aims to maintain the likelihood that the policy model will produce a base response originating from the reference model during personalized preference optimization. our main contributions are summarized as follows: in this section, we first introduce preference optimization for llm alignment. next, we examine how personalized preferences impact the existing knowledge of instruction-tuned llms. we suggest that the typical kl-constraint in preference optimization is not effective in preventing forgetting. ## proposed method: bapo in this section, we introduce base-anchored preference optimization (bapo). our primary motivation is to use the initial response from the instruction-tuned model before it undergoes personalized preference optimization. by anchoring the chosen (𝑦 𝑤 ) base (𝑦 𝑏 ) rejected (𝑦 𝑙 ) figure 5: average difference in reference model (π ref ) and policy model (π θ ) log probabilities for chosen, base, and rejected responses during personalization across four domain preferences in dsp datasets. ## discussion we demonstrate that bapo benefits from the separation between general alignment and personalized preferences. however, preserving the likelihood of the base response may sometimes hinder alignment with personalized preferences. for example, if a user prefers humorous but potentially harmful responses, and the base response is harmless, maintaining the base likelihood can conflict with achieving the user's preferred style. thus, while we distinguish between 'global knowledge & general alignment' and 'personalized alignment,' their interaction is influenced by both the model's existing knowledge and the new, targeted personalization. our experiments show that this separation is advantageous for bapo, as personalized preferences-whether in specific response styles (p soups datasets) or specific domains (dsp datasets)-tend not to overlap significantly with the model's general knowledge. nevertheless, the challenge arises when preserving the base response's likelihood (y b ) compromises the ability to cater to personalized preferences. this situation parallels the stability-plasticity dilemma in conventional continual learning @xcite , where the relationship between old and new knowledge influences the potential for forgetting. ## related work learning from human feedback often employed as the final stage of instruction tuning @xcite @xcite , learning from human feedback refines the policy language model to generate responses that align with human preferences @xcite . this process usually involves collecting human preferences for pairs of candidate responses to differentiate between those that are preferred and those that are dispreferred. the two main approaches used are reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite . rlhf involves training a separate reward model that is then utilized in the subsequent reinforcement learning phase @xcite . in contrast, dpo does not rely on a reward model but directly establishes a mapping between the reward function and the optimization objective @xcite . our work specifically addresses the challenge of accommodating personalized preferences within the dpo framework. we focus on maintaining the likelihood of base responses from the reference model during the personalization process, ensuring the preservation of global knowledge and general alignment. forgetting in llm fine-tuning the issue of forgetting previously acquired knowledge during the fine-tuning on heterogeneous data has been extensively discussed @xcite , highlighting a fundamental trade-off between preserving old knowledge and acquiring new knowledge @xcite . in the context of llms, recent research has shown that the knowledge from pre-training can be compromised by supervised fine-tuning (sft) on instruction data @xcite . a similar issue, known as alignment tax, occurs in preference optimization @xcite . while prior research has addressed the forgetting induced by the llm finetuning @xcite , the effects of accommodating specific user preferences and the impact of their heterogeneity remain largely unexplored. in our study, we investigate how personalized preference optimization affects both global knowledge and general alignment. personalized preference in llm the use of a single scalar reward to represent user preferences presents a significant limitation when users have diverse and conflicting preferences @xcite . to address this issue, some studies have explored clustering users who presumably share the same reward @xcite . others have considered defining a reward function with multiple objective dimensions @xcite to achieve pareto optimality among them @xcite @xcite . additionally, some approaches involve merging model parameters trained for each dimension to accommodate the diverse combinations expressed by those dimension @xcite . in our study, we focus on the impact of preference heterogeneity on forgetting during personalized preference optimization, assuming that each user has a specific, definitive type of preference. ## conclusion this study explores the degree and nature of forgetting caused by personalized preference optimization in instruction-tuned llms. our findings indicate a reduced likelihood of generating original responses, alongside a decrease in the generation of dispreferred responses. to address this, we introduce base-anchored preference optimization (bapo), a method that anchors the likelihood of base responses during the preference optimization process. this approach effectively preserves global knowledge and general alignment while successfully accommodating personalized preferences. we have conducted extensive experiments to validate the efficacy of bapo and its benefits. limitations while bapo effectively preserves existing knowledge by leveraging the base response, it is important to note that if the base model is biased, the fine-tuned personalized model may also exhibit a similar bias. this consideration is crucial for machine learning practitioners. in our experiments, we utilized q-lora for fine-tuning. although we conducted an ablation study varying model capacity by adjusting the lora rank, a fullfinetune might show different tendencies. nonetheless, using lora fine-tuning for personalization is a common approach in llm context. regarding computational costs, although bapo requires the use of a base response, potentially increasing memory and computational demands during training, most of these costs can be mitigated by pregenerating the base responses and caching them as offline datasets. concerning the anchoring strength hyperparameter, λ, the extent and scope of forgetting may vary based on the type of knowledge and the personalized preference types. ideally, the λ value should be adaptively assigned, but this aspect is left for future research.