Paper: Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment (ICLR-2024)

Reviewer comment (Theory):
The authors mention that P3O enjoys the invariance property while PPO does not. Why is the invariance property important? The authors should elaborate more on how this invariance property of P3O helps improve its performance in LLM alignment.

Author response:
The invariance property is crucial in the context of P3O and its application in Language Model (LM) alignment. To illustrate this, consider two responses, $y_1$ and $y_2$, with $y_1$ being preferred 73.1% of the time by human evaluators. When minimizing the Bradley-Terry Loss, the difference in their ratings $r_1 - r_2$ is 1, regardless of whether $r_1=1001, r_2=1000$, or $r_1=1, r_2=0$. Algorithms lacking invariance to reward translation, like PPO, might be misled by the scaling into overvaluing response $y_2$ as excellent choices, even though it may not be an optimal choice. Although clipping rewards can mitigate this to some extent, it’s not a comprehensive solution.

In practice, We often observe initial reward drops in PPO, possibly due to the value function's difficulty in accurately capturing the scale of the rewards. P3O addresses these challenges by being immune to shifts in reward and reducing the complexity of approximating additional functions, such as the value function in PPO.

Reviewer comment (Evaluation):
The authors should give a more detailed comparison between P3O and DPO. They both satisfy the invariance property. Why does P3O perform better in the KL-reward trade-off?

Author response:
For the comparison between P3O and DPO, both of which satisfy the invariance property, the key difference lies in their approach to optimizing the policy. While both algorithms use gradients directed along $\nabla\log\frac{\pi(y_1)}{\pi(y_2)}$, the weighting of these gradients differs. DPO’s weight term is derived from a specific supervised loss function, while P3O’s weight terms stem from the derivation of policy gradient. We hypothesize why DPO fell short in KL-reward tradeoff is because DPO directly minimizes an alternate “distance”, represented by DPO loss, between the current and the goal policy. While in the middle of this optimization trajectory, intermediate policies might not directly optimize the reward like policy gradient methods do. In contrast, P3O ensures strict policy improvement, a characteristic of policy gradient-like algorithms, leading to its superior performance in this trade-off.

Reviewer comment (Evaluation):
Why should the KL-reward trade-off be the performance metric for LLM alignment, not the reward? Is this a standard criterion in the literature?

Author response:
Regarding the choice of the KL-Reward frontier as a performance metric for LLM alignment, it stems from the need to balance preference optimization and maintaining linguistic coherence. While reward maximization is a common goal in reinforcement learning, in language generation, a sole focus on reward can lead to policy that generates nonsense content, for reference see [Scaling laws for reward model overoptimization](https://proceedings.mlr.press/v202/gao23h.html). The KL-Reward metric effectively captures the algorithm's ability to optimize user preferences (represented by the reward) while preserving the original policy's language generation capabilities (represented as KL), thus is more suitable than reward in our case.

Besides using the KL-Reward frontier to evaluate the generation quality, we also employ GPT-4 evaluation to consolidate our findings. Previous [studies](https://arxiv.org/abs/2304.00723) have shown that GPT-4’s assessments correlate strongly with human evaluations, sometimes even exceeding the consistency found in human-to-human evaluations. The result confirms that P3O can align better than other baselines.

Reviewer comment (Evaluation):
Compared with DPO, P3O still needs to learn a reward function, which increases the complexity of the overall algorithm pipeline. At the same time, I can tell from the GPT4 automatic evaluation the improvement is marginally better than DPO.

Author response:
Regarding the complexity of learning a reward function in P3O as compared to DPO, we acknowledge this concern. While having to train a reward model does add complexity, it also provides a more robust and adaptive framework for policy optimization. This is evidenced by our results, where P3O achieves approximately a 55% preferred rate over DPO evaluated by GPT-4.

Reviewer comment (Experiments):
I feel the overall experiments are good, but at the same time, the authors might miss some baselines. For example, given the same reward function learned from the dataset, it would also be good to compare with RAFT, since it is also a very simple algorithm for policy improvement.

Author response:
We appreciate this insightful recommendation to include RAFT in our comparisons. We agree that benchmarking P3O against RAFT would provide a more comprehensive evaluation of our algorithm. We are committed to conducting this experiment. Currently, our comparisons include three significant algorithms – SFT, PPO, and DPO – which we believe provide a strong baseline for evaluating P3O's performance.

Reviewer comment (Theory):
Reward Equivalence: Since there are no constraints (Definition 1 should hold for all prompts, $\delta(x)$ needs to be an increasing function or constant), Definition 1 holds for each prompt. For me, saying two reward functions are equivalent implies the reward of one reward function magnifies (either positively or negatively) the reward of another function for all prompts, but Definition 1 does not guarantee it.

Author response:
Thank you for raising this important point regarding reward equivalence. Regarding the function $\delta$,  it can be an arbitrary function that takes in a prompt $x$ and returns a scalar. To offer further clarification, let’s consider two reward functions: $r$ and $r’$. These two reward functions have the same **Bradley-Terry Loss** on the response pair $(x,y_w,y_l)$ if and only if $r(y_w|x) - r(y_l|x)$ and $r’(y_w|x) - r’(y_l|x)$ are equal. The quantity $r(y_w|x) - r(y_l|x)$ represent the relative preference of $y_w$ over $y_l$. This indicates that $r$ and $r’$ can differ in constant on the prompt $x$. However, this doesn’t mean $\delta$ should be a constant or increasing function (we can’t define what is an increasing function since $x$ is a sequence, not scalar), since the constant in which $r$ and $r’$ differ can vary for different prompts.

For instance, consider two query and responses pairs $(x_1,y_{1w}, y_{1l}), (x_2,y_{2w},y_{2l})$. These two rewards are equivalent: $r(x_1,y_{1w}) = 1, r(x_1, y_{1l})= 0, r(x_2,y_{2w}) = 2, r(x_2,y_{2l}) = 1$ and $r’(x_1,y_{1w}) = 2, r’(x_1, y_{1l})= 1, r’(x_2,y_{2w}) = 1, r’(x_2,y_{2l}) = 0$. $\delta$ in this example corresponds to $\delta(x_1) = 1, \delta(x_2) = -1$, which varies between prompts.

Your mention of reward function magnification is insightful. However, scaling a reward function (e.g. $r’ = 2 * r$) doesn’t yield the same Bradley-Terry Loss, so it won’t fit our original definition.

Reviewer comment (Theory):
Relative feedback: As the relative feedback is derived from the difference in rewards (as shown in Figure 1), it implies access to the rewards. First, it is not clear in which scenarios one can access the true rewards in LLMs. Second, it is unclear why not directly train the RL model using the available rewards. Methods designed for dueling bandits or RLHF are generally useful when it is hard to get the reward but easier to get pairwise preferences.

Author response:
Thank you for your question about relative feedback and reward access. We do have a **trained reward model**, following the previously proposed [RLHF pipeline](https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html). Note that the reward model is trained from pairwise comparison data, where a human chooses a preferred response from two options for the same query. The reward learning is facilitated by the Bradley-Terry Loss function.

Regarding your question about training the RL model directly using the rewards, approaches like PPO which are not reward equivalence, can exhibit instability and are vulnerable to noise in the reward model. This is particularly problematic when the scale of the reward can be misleading. For instance, consider two responses, $y_1$ and $y_2$, with $y_1$ being preferred 73.1% of the time by human evaluators. When minimizing the Bradley-Terry Loss, the difference in their ratings $r_1 - r_2$ is 1, regardless of whether $r_1=1001, r_2=1000$, or $r_1=1, r_2=0$. Algorithms lacking invariance to reward translation might be misled by the scaling into overvaluing response $y_2$ as excellent choices, even though it may not be an optimal choice. 

P3O, our proposed algorithm, addresses this issue by focusing on relative feedback. It can discern the relative preference of $y_1$ over $y_2$ and prioritize generating $y_1$ accordingly. This approach provides a more nuanced and accurate reflection of human preferences, avoiding the pitfalls of absolute reward scaling.