Reverse Preference Optimization for Complex Instruction Following (2025)

Abstract: Instruction following (IF) is a critical capability for large language models (LLMs). However, handling complex instructions with multiple constraints remains challenging. Previous methods typically select preference pairs based on the number of constraints they satisfy, introducing noise where chosen examples may fail to follow some constraints and rejected examples may excel in certain respects over the chosen ones. To address the challenge of aligning with multiple preferences, we propose a simple yet effective method called Reverse Preference Optimization (RPO). It mitigates noise in preference pairs by dynamically reversing the constraints within the instruction to ensure the chosen response is perfect, alleviating the burden of extensive sampling and filtering to collect perfect responses. Besides, reversal also enlarges the gap between chosen and rejected responses, thereby clarifying the optimization direction and making it more robust to noise. We evaluate RPO on two multi-turn IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively. Moreover, RPO scales effectively across model sizes (8B to 70B parameters), with the 70B RPO model surpassing GPT-4o.

## introduction large language models (llms) @xcite @xcite have demonstrated impressive capabilities in various tasks such as code, math, and text generation. among all abilities, instruction following @xcite has emerged as a key ability for conversational intelligence as it reflects how well llms can align with human preferences. in a realworld scenario, user intention typically involves diverse and multiple preferences, posing a chalany two responses that differ in at least one aspect belong to the above six types (case 1 ⃝ to 6 ⃝). abcdef are six constraints that need to be followed, and the color in gray indicates this response failed to follow them. c: and r: represent the adherence of chosen and rejected responses to different constraints, respectively. lenge to accurately meet the expectations of various users. however, as the number of preferences that need to be followed increases, aligning multiple preferences simultaneously becomes a challenging problem. previous methods typically construct preference pairs for alignment and distinguish chosen and rejected responses based on their total scores (the number of constraints that were successfully followed). in a multi-preference alignment scenario, this method presents two main drawbacks: firstly, using the total score difference as the gap between two responses may not accurately reflect the true differences between them, potentially underestimating their variations. taking case 1 ⃝ in figure 1 as an example, although the two responses have the same total score, they exhibit differences in adherence to 6 constraints. secondly, selecting preference pairs in a compositional setting may introduce noise. in cases 3 ⃝ and 4 we notice that the above issues can be mitigated if the chosen response is perfect (case 6 ⃝). however, perfect responses are hard to obtain. it requires a large amount of sampling or refinement, and it may not be feasible to sample a perfect response for a weak model. besides, pursuing a perfect response is not an economical and scalable solution. as the number or difficulty of constraints increases, the difficulty of sampling a perfect response increases dramatically, and there are always cases where perfect responses can not be obtained in a limited sample time. to address these issues, we propose reverse preference optimization (rpo) to efficiently align with multiple preferences. the idea of rpo emerged from two key observations: 1). learning a constraint has no fundamental difference with learning its opposite. 2). for a clear and unambiguous constraint, a response either adheres to it or deviates from it. specifically, rpo dynamically reverses the constraints within the instructions to ensure that the chosen response fully meets all constraints. in this way, any response can be easily transformed into perfect ones for the new instruction, eliminating labor-intensive sampling and filtering to pursue perfect responses. besides, it also amplifies the gap between chosen and rejected responses, thus clarifying the optimization direction and enhancing robustness to noise. we summarize our contributions as follows: ## preliminary reinforcement learning from human feedback (rlhf) @xcite the goal of traditional rlhf encompasses two primary goals: maximizing the rewards of responses and minimizing the deviation (kl divergence) of the aligned policy model π θ from the initial reference (sft) model π ref . therefore, this rl objective can be formulated as: direct preference optimization (dpo) @xcite re-parameter the rl object and build the connection between the reward model r and the target policy model π θ . by incorporating the reward model into the bradley-terry (bt) ranking objective (bradley and @xcite the optimization objective is formulated as: ## conclusion in this paper, we explore the noise issue of aligning with multiple preferences. we introduce rpo, a simple and efficient method that dynamically reverses the unsatisfied constraints within the instruction to mitigate noise. in this way, rpo ensures that the chosen responses are perfect and strictly dominate the rejected ones. as rpo does not require a perfect response as the chosen example, it also alleviates the burden of sampling extensive response candidates to construct preference pairs. experimental results on two multi-turn instructionfollowing benchmarks demonstrate that rpo significantly outperforms dpo baseline by 4.6 and 2.5 points on sysbench and multi-if, respectively. besides, rpo can also scale to the 70b model and surpass gpt-4o with the 70b model. detailed analysis also shows rpo's superiority in learning efficiency and sample efficiency. we hope that rpo can serve as a valuable foundation for tasks that require aligning with multiple preferences.