Paper: Query-Policy Misalignment in Preference-Based Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The focus is predominantly on off-policy PbRL methods, with limited exploration of on-policy PbRL methods, which naturally select on-policy segments to query preferences.

Author response:
- As we discussed in "Conclusion and Discussion" on Page 9, although on-policy PbRL methods naturally select on-policy segments, they suffer from severe sample inefficiency issue as compared to off-policy PbRL methods. Due to the prolonged policy learning process, the query segments generated by slowly learned policies are often not very informative for rapid reward model learning, which in turn causes poor feedback efficiency in on-policy PbRL methods. 
- We add the performance curve of PrePPO (on-policy PbRL method) [1] to Figure 5 and 6 on Page 8. We can see that PrePPO's performance is notably inferior to current SOTA off-policy PbRL methods.
- All the recent SOTA PbRL methods, including ours, are built upon off-policy framework. We discover that the query-policy misalignment can potentially harm feedback efficiency, and thus enforces query selection to align with the policy induced distribution $d^{\pi}$. Note that this is different from conventional on-policy RL, as our treatment is conducted on the query selection part (i.e., selecting proper query segments that are already in the replay buffer), rather than collecting on-policy samples for policy learning as in typical on-policy RL methods.

Reviewer comment (Experiments):
The paper should compare to Liu et al. Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. NeurIPS ‘22, which is the current SoTA for PbRL.

Author response:
- We appreciate the reviewer's helpful reminder! The performance of Meta-Reward-Net (MRN) and its comparison to QPA has been added in Figure 12, Appendix D.2. In most tasks, our method QPA consistently outperforms MRN. Essentially, MRN takes a good idea of adopting the performance of the Q-function as the learning target to formulate a bi-level optimization problem, which is orthogonal to our methods. Our technique *policy-alignment query selection* can be easily incorporated into MRN to further improve feedback efficiency (MRN + policy-alignment query selection).

Reviewer comment (Presentation):
I thought the Figure 1, 2, 3 could be improved and explained better, both in the text, and in the figure, and in the caption.

Author response:
- We thank the reviewer for the suggestion. We have added more explanations on Page 2-4. Furthermore, we have provided very detailed experiment settings and descriptions in Appendix C.

Reviewer comment (Experiments):
Could the proposed methods be adapted or combined with on-policy PbRL techniques to achieve even better results?

Author response:
- Our proposed method focuses on the query selection and value learning from the **replay buffer**. In contrast, in on-policy PbRL, these processes occur within **on-policy rollout trajectories**, where the policy-aligned query selection and hybrid experience are no longer applicable. As we have explained in the previous response as well as the additional results of on-policy method PrePPO, the low sample efficiency and prolonged policy learning process in on-policy PbRL can hurt feedback efficiency. By contrast, our proposed method has achieved SOTA performance.

Reviewer comment (Evaluation):
Are there specific scenarios or domains where the proposed method may not be as effective?

Author response:
- When the reward function is highly non-smooth in the state-action space and drastically different in various regions, hard exploration might be needed to learn the reward model effectively. In such case, relying solely on policy-aligned query selection might lead to getting trapped in local solutions.

Reviewer comment (Experiments):
How does the system handle scenarios where human feedback might be inconsistent or contradictory?

Author response:
- We have added new experiments with human feedback, which could potentially contain inconsistencies. The results are presented in Appendix E, we find our proposed QPA still achieves reasonable/good performance.

Reviewer comment (Experiments):
Query-policy misalignment not conclusively proven: While the paper proposes that query-policy misalignment is an interesting hypothesis to cause a problem in PbRL, there is no comprehensive evaluation or solid evidence to confirm this proposal.

Author response:
- We really appreciate this valuable comment. We have added Figure 11 and Appendix D.1 to confirm this proposal. As all the recent off-policy PbRL methods are built upon PEBBLE, we compare the two methods: PEBBLE and PEBBLE + policy-aligned query selection in locomotion tasks. The policy-aligned query selection is proposed in Section 5.1 to address the query-policy misalignment.
    - We compute the log-likelihood of current policy $\pi$ using the queried segments at each query time. Figure 11(a) shows that the segments queried by PEBBLE exhibit a low log-likelihood of $\pi$, indicating that these segments fall outside the distribution of the current policy $\pi$. This demonstrates the existence of the query-policy misalignment.
    - Figure 11(a) shows that, if incorporating policy-aligned query selection into PEBBLE to confine the queries to the local policy-aligned buffer, there will be a substantial increase in the log-likelihood of $\pi$, which means the query-policy misalignment can be addressed.
    - Figure 11(b) shows that the performance of PEBBLE + policy-aligned query selection significantly surpasses that of PEBBLE.
- Overall, Figure 11 demonstrates that: (1) the query-policy misalignment issue does exist in typical PbRL methods and does cause feedback inefficiency; (2) using policy-aligend query selection to address the query-policy misalignment can result in a significant improvement in feedback efficiency.

Reviewer comment (Experiments):
Lack of real human experiments: The testing and validation conducted in the paper relied on a scripted annotator which is not representative of real-world users, which could limit the generalizability and applicability of the approach.

Author response:
- The use of scripted ground truth reward is to evaluate and compare these PbRL methods quantitatively, unbiasedly and quickly, which is a common practice in existing PbRL literature [1-6].
- We have included additional real human experiments in Appendix E to showcase the applicability and effectiveness of our method. Using feedback from real humans, our method also notably improves feedback efficiency compared to PEBBLE.

Reviewer comment (Experiments):
The paper could be strengthened by testing the proposed method under real-world conditions, using actual human annotators instead of an oracle, to test the generalizability of their results.

Author response:
- The use of scripted ground truth reward is to evaluate and compare these PbRL methods quantitatively, unbiasedly and quickly, which is a common practice in existing PbRL literature [1-6].
- We have included additional real human experiments in Appendix E to showcase the applicability and effectiveness of our method. Using feedback from real humans, our method also notably improves feedback efficiency compared to PEBBLE.

Thanks a lot for this good suggestion! 

As suggested, we have added more experimental details of human labeling process in Appendix E on Page 24-25 in our latest version paper. Specifically, we have provided the main Python code for segment video rendering, labeling instructions, and human preference collection on Page 24-25. The user interface is shown in added Figure 25. The human feedback frequency and total feedback remain consistent with the experimental setup outlined in Table 2. The supplementary material includes the videos of agent training processes in human experiments.

We sincerely thank the time and effort you have engaged in the review and discussion phase!

Reviewer comment (Reproducibility):
Providing more implementation details or pseudocode for the proposed method would add value to the paper and make it easier for others to understand, replicate, and build upon the proposed method.

Author response:
- We thank the reviewer for this kind suggestion. We have provided the pseudocode in Appendix B and implementation details in Appendix C on Page 13-16. These details encompass the implementation framework, query selection scheme, data augmentation explanation, and hyperparameter settings.