Paper: Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
I still have questions regarding regret-based preference model. I agree with the authors that the regret-based preference model makes more sense when we consider the hand-engineered example in section 2. However, when we talk about data collection with a real human, the human labeler would have a preference over two trajectory segments. This implies the existence of underlying dense rewards that explain the human's preferences. In such cases, I feel that the key issue lies in the hand-engineered reward is incorrect (i.e., reward design issue) in your example, rather than in the issues of the reward-based preference model.

Author response:
**The Regret Preference Model**

The regret preference model states that human *judgements* are better captured by an optimal advantage function, and does not make any claims about what underlying reward functions humans use. Going back to the example in Section 2, we did not mean to claim that the human would intend to use a sparse reward function – a user providing preferences preferring movements towards a goal could have a number of different underlying reward functions, many of which could yield the same optimal policy and optimal advantage function. 

Though the focus of our paper is not the validity of the regret preference model, for which we would refer the reader to [1], we do provide a number of additional insights on the regret preference model.

1. *Theoretical*: In Appendix A, we prove that the optimal advantage is just a shaped version of the reward function that results in the same optimal policy. The reviewer suggests that humans use dense rewards to make decisions, and Corollary 1 shows exactly that the advantage is a dense reward function.

2. *Empirical*: In the new Section 4.4, we show that CPL works on real human data, indicating that the regret-model holds on real human data. 

Again we would encourage the reader to examine [1], as it contains a number of additional examples (like stochastic environments) and experimental results which support the validity of the regret-preference model.

Reviewer comment (Experiments):
When we consider experiments with real humans and apply the reward-based preference model, could it also perform effectively? Is it possible that the learned reward captures the regret information to a large extent? Please correct me if I have misunderstood.

Author response:
**The Regret Preference Model**

The regret preference model states that human *judgements* are better captured by an optimal advantage function, and does not make any claims about what underlying reward functions humans use. Going back to the example in Section 2, we did not mean to claim that the human would intend to use a sparse reward function – a user providing preferences preferring movements towards a goal could have a number of different underlying reward functions, many of which could yield the same optimal policy and optimal advantage function. 

Though the focus of our paper is not the validity of the regret preference model, for which we would refer the reader to [1], we do provide a number of additional insights on the regret preference model.

1. *Theoretical*: In Appendix A, we prove that the optimal advantage is just a shaped version of the reward function that results in the same optimal policy. The reviewer suggests that humans use dense rewards to make decisions, and Corollary 1 shows exactly that the advantage is a dense reward function.

2. *Empirical*: In the new Section 4.4, we show that CPL works on real human data, indicating that the regret-model holds on real human data. 

Again we would encourage the reader to examine [1], as it contains a number of additional examples (like stochastic environments) and experimental results which support the validity of the regret-preference model. 

If the reviewer was referring to the ability of a reward model to be applied to unlabeled data, we show in our D4RL experiments that CPL can also take advantage of unlabeled data by using a logistic-regression based relabeling model.

Reviewer comment (Experiments):
Despite considering the model complexity of CPL, the results are promising. In terms of feedback efficiency, does CPL require more human preference data compared to the conventional two-phase PbRL framework in order to perform well? This is especially relevant considering the Metaworld tasks in the experiments, where obtaining dense data could be challenging if collected from real humans.

Author response:
**Does CPL require more human preference data in comparison to two-phase PBRL**

To demonstrate that CPL is effective with limited real-human preference data we perform additional evaluations on the PBRL benchmarks from [2]. These benchmarks use only 100 or 500 real human preferences for expert or replay datasets from D4RL respectively. 

We showed in Section 4.3 that CPL performs better with dense preference data. To overcome this, we use a logistic regression model to predict which of two segments a user prefers from the preference data, and use it to relabel a larger offline RL dataset. Critically, this logistic regression model is not a reward or advantage model, as it predicts a single value for the *entire* segment.

We find that CPL performs similarly to baselines on 2/4 datasets, outperforms in 1, and performs worse in 1. Yet, CPL remains far simpler.

Reviewer comment (Experiments):
In the experiments, the authors pretrain the model with behavior cloning. To what extent does this pretraining phase impact the model's final performance? Does P-IQL also have this pretraining phase?

Author response:
**Pretraining**


We have included results on Metaworld in state without any pre-training, and find that CPL performs similarly – no pretraining performs better in some cases and pretraining performs better in others. This ablation has been included in Appendix C and is copied below as well.

|                   | Bin Picking     | Button Press    | Door Open       | Drawer Open     | Plate Slide     | Sweep Into      |
|-------------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| CPL               | 80.0 $\pm$ 2.5 | 24.5 $\pm$ 2.1 | 80.0 $\pm$ 6.8 | 83.6 $\pm$ 1.6 | 61.1 $\pm$ 3.0 | 70.4 $\pm$ 3.0 |
| CPL (No Pretrain) | 81.0 $\pm$ 1.5 | 30.0 $\pm$ 2.7 | 76.8 $\pm$ 2.1 | 88.7 $\pm$ 1.9 | 50.8 $\pm$ 3.5 | 80.0 $\pm$ 2.5 |


This ablation shows that the use of the conservative regularizer in CPL can help it learn effectively even without pre-training. Note that our results on D4RL (Table 3) also do not use any pre-training.  


We did not use pre training for P-IQL for two reasons:


1. P-IQL uses a weighted BC objective ($\min \mathbb{E} [ e^{Q-V} \log \pi]$ and thus already has a loss function very similar to BC. 
2. In IQL value learning is completely detached from policy learning, and thus pretraining $\pi$ does not improve estimates of $Q$ and $V$.

Reviewer comment (Theory):
Since the CPL loss function has a super elegant form, is it possible to derive finite sample analysis for learning a near-optimal policy like [1]?  

[1] Zhu et al., Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons, ICML 2023

Author response:
We would like to thank the reviewer for their time in reviewing our work. We are happy to hear that our algorithm was “novel and elegant”, and that it’s “motivation… is clear”.

As the reviewer has pointed out no weaknesses, we will just mention the following revisions to the draft:

1. Additional PPO baseline. We have modified Table 1 with an additional PPO baseline. Note that as PPO is an online RL method, it requires 4x the amount of state-action data as CPL plus online interactions with the MDP. Despite that, it still underperforms in many settings and exhibits low variance. 

2. Additional Benchmark with Real Human Feedback. We have added additional experiments on the D4RL benchmark using only 100 to 500 real-human queries. 

3. Additional Theoretical revisions. We have heavily modified the appendix and Section 3.3 including a new proposition on CPLs regularization with finite data.

4. Additional ablations on pretraining in Appendix C.

Reviewer comment (Experiments):
In my experience, learning a "good" reward model and then doing RL always outperforms offline RL algorithms. The authors only compare it with IQL and not with methods with reward models to highlight more.

Author response:
The reviewer asked us to compare CPL to methods that learn a reward model. 

1. We would graciously like to point out that P-IQL does indeed learn a reward model first, which could then be applied with any RL Algorithm. We chose to use IQL since it is generally a very  high-performing offline RL method [1] and has been used as a standard in many prior RLHF for control works [2, 3].

2. We have additionally compared CPL to PPO with a learned reward function based on comments from Reviewer WpDS.  We would like to point out that this comparison is *unfair* to CPL, as PPO requires additional online data to estimate the policy gradient. While in bandits settings (like LLMs) this can be done without access to the MDP, in sequential settings (like control, or 2+ steps of dialogue) we need to interact with the MDP to get full trajectories, which can be expensive (for ex running a robot in the real world). Despite the fact that PPO uses online data and 4x the number of state-action pairs, it is highly unstable in comparison to CPL and often underperforms it by a large margin. We have included full results in Table 1. 

3. Our results are consistent with other recent works [3, 4] which have also shown that learning an explicit reward model is unnecessary to achieve good performance.

Reviewer comment (Experiments):
The authors claim that the method is generic but then it is only applied to MetaWorld benchmark. The RLHF scenario is much more interesting in aligning language models with human feedback.

Author response:
The reviewer mentioned that RLHF domains with LLMs would be interesting to explore. We whole-heartedly agree that RLHF with LLMs is a great possible application of CPL. Our work, however is about settings with *sequential* data  in particular, which in dialogue would constitute multiple turns of interaction with a human user. To our knowledge there were no such publicly available RLHF datasets with 2+ turns of dialogue at the time of submission. We believe such an investigation is a large undertaking, enough to warrant its own publication.

We would kindly like to push back against the notion that robotics domains are less interesting. We chose to evaluate in robotics domains precisely because they capture the sequential nature of CPL, and show that our method applies to general MDPs, instead of the  simpler contextual bandit setting used in standard RLHF with LLMs.

Reviewer comment (Experiments):
How does CPL compare with RLHF for language models scenario?

Author response:
The reviewer mentioned that RLHF domains with LLMs would be interesting to explore. We whole-heartedly agree that RLHF with LLMs is a great possible application of CPL. Our work, however is about settings with *sequential* data  in particular, which in dialogue would constitute multiple turns of interaction with a human user. To our knowledge there were no such publicly available RLHF datasets with 2+ turns of dialogue at the time of submission. We believe such an investigation is a large undertaking, enough to warrant its own publication.

Reviewer comment (Experiments):
How does CPL compare with other baselines, which may or may not have reward models?

Author response:
The reviewer asked us to compare CPL to methods that learn a reward model. 

1. We would graciously like to point out that P-IQL does indeed learn a reward model first, which could then be applied with any RL Algorithm. We chose to use IQL since it is generally a very  high-performing offline RL method [1] and has been used as a standard in many prior RLHF for control works [2, 3].

2. We have additionally compared CPL to PPO with a learned reward function based on comments from Reviewer WpDS.  We would like to point out that this comparison is *unfair* to CPL, as PPO requires additional online data to estimate the policy gradient. While in bandits settings (like LLMs) this can be done without access to the MDP, in sequential settings (like control, or 2+ steps of dialogue) we need to interact with the MDP to get full trajectories, which can be expensive (for ex running a robot in the real world). Despite the fact that PPO uses online data and 4x the number of state-action pairs, it is highly unstable in comparison to CPL and often underperforms it by a large margin. We have included full results in Table 1. 

3. Our results are consistent with other recent works [3, 4] which have also shown that learning an explicit reward model is unnecessary to achieve good performance.