COPR : Continual Human Preference Learning via Optimal Policy Regularization (2025)

Abstract: Reinforcement Learning from Human Feedback (RLHF) is effective for aligning Large Language Models (LLMs) with human preferences. However, RLHF’s complex process limits its ability to continually learn human feedback, making it impractical for real-world applications where the deployed model continuously receives feedback from users. The non-RL-based method, such as Direct Preference Optimization (DPO), is not primitively favorable for Continual Learning (CL). We observe that when combined with Experiment Relay (ER) for CL, DPO tends to significantly widen the gap in the probability of human-preferred and dispreferred responses. Consequently, this diminishes the diversity in model generation, potentially leading to model collapse. To overcome the above challenges, we propose the Continual Optimal Policy Regularization (COPR), a novel non-RL offline method to convert the historical optimal policies into optimization constraints when continually learning new preferences. We first derive a moderate reward function from the pairwise ranking loss and then use the moderate reward to calculate a new sampling distribution to construct novel learning objectives and constraints. We also provide formal proof of the learnability of COPR. The experimental results show that COPR outperforms strong CL baselines on our proposed benchmark, in terms of reward-based, GPT-4 evaluations and human assessment.

## introduction the rapid evolution of artificial intelligence, particularly in natural language processing (nlp), has driven the adoption of large language models (llms) across diverse applications. these models hold immense potential for aligning machinegenerated outputs with human preferences @xcite @xcite , enabling safer and more effective human-computer interactions. however, real-world deployment scenarios demand more than static alignment; llms must continually adapt to evolving user preferences * b.liang and r.xu are corresponding authors. as new tasks and domains emerge @xcite . this capability, known as continual alignment @xcite , is critical for practical applications such as ai-driven assistants, where user feedback is dynamic and incremental. the development of robust continual alignment methods can transform how ai systems learn and respond, unlocking their ability to provide longterm, adaptive assistance. despite the promise of continual alignment, existing methodologies face significant challenges. reinforcement learning from human feedback (rlhf) @xcite , a widely adopted approach for aligning llms with human preferences, relies on a multi-stage process involving supervised fine-tuning, reward model training, and reinforcement learning optimization. this pipeline, while effective, is computationally intensive and inflexible for continual learning (cl) @xcite . offline methods like direct preference optimization (dpo) @xcite eliminate the need for reinforcement learning but suffer from issues such as overfitting preference data @xcite and catastrophic forgetting of past knowledge. in cl scenarios, these methods often struggle to maintain model output diversity, resulting in preference collapse @xcite , where minority preferences are virtually disregarded. this ultimately leads to degraded performance on historical tasks and reduced adaptability to new ones. to address these challenges, we introduce continual optimal policy regularization (copr), a novel framework that redefines how llms adapt to evolving human preferences. copr bridges the gap between efficiency and effectiveness in continual preference learning by introducing a principled method to preserve historical knowledge while integrating new preferences. at its core, copr leverages the theoretical optimal policy theory @xcite , utilizing historical optimal policies as constraints to guide the learning of new tasks. this ensures that the model does not overly prioritize new preferences at the expense of forgetting old ones. by deriving a moderate reward function (mrf) from bradley-terry model @xcite , copr maintains a balanced optimization process that avoids excessive determinism or preference collapse @xcite , ensuring stable and diverse model outputs. the simplicity and theoretical grounding of copr make it a robust solution for continual alignment. copr's methodology incorporates several innovative components. first, it parameterizes the sampling distribution of historical and new policies to construct optimization objectives that maintain diversity and prevent catastrophic forgetting. second, it employs the mrf to regulate the learning process, ensuring that the relative probabilities of preferred and dispreferred responses remain within a reasonable range. third, copr adopts a lagrangian dual optimization framework to balance the learning of new preferences with the retention of past ones. these innovations collectively address the critical limitations of existing methods, enabling copr to outperform strong baselines in various cl settings. in summary, our main contributions are as follows: ## continual preference optimization our method is based on the optimal policy in eq. to prevent widening the gap log π * t (y w |x)log π * t (y l |x) → ∞ like dpo, we determine a mrf. after determining the reward, we need to bypass the intractable partition function to get a feasible learning objective. for this purpose, we propose to parameterize the sampling distribution. the sampling distribution of historical policies can be regarded as optimization constraints for learning new preferences, providing a natural advantage for cl over maximum @xcite . the optimal policy π * t is utilized as the current policy's fitting objective and the learning constraints of the next πi (@xmath0. ## experiments in this section, we conduct extensive experiments on our proposed benchmark to evaluate copr and baselines, we observe that: 1) compared to strong baselines, copr showcases distinct advantages in continual learning of human preferences. particularly, copr achieves comparable performance to the upper bound of continual learning of human preferences, namely, iterated rlhf @xcite , without retraining and reinforcement learning (section 4.2). 2) the policy regularization technique exhibits a notable effect on the continual learning of human preferences in the ablative experiment (section 4.3). ## conclusion we propose continual optimal policy regularization (copr), a scalable and efficient framework for continual alignment in large language models (llms). copr addresses key challenges such as catastrophic forgetting and preference collapse by leveraging historical optimal policies and a moderate reward function (mrf) to balance the retention of past preferences and the integration of new ones. through extensive experiments on taskincremental and domain-incremental benchmarks, copr demonstrates superior performance and stability compared to existing methods. this work provides a practical solution for dynamically aligning llms to evolving human preferences, paving the way for more adaptive and reliable ai systems in real-world applications.