Paper: Hindsight PRIORs for Reward Learning from Human Preferences (ICLR-2024)

Reviewer comment (Evaluation):
In the Drawer Open / 4000 task in Figure 2, the success rate of the proposed approach plateaus after 400k steps and is eventually surpassed by some other baselines, which might imply that the hindsight prior eventually hurts the performance of the learned policy as more preference labels become available.

Author response:
On the performance plateau on Drawer Open domain : 
We showcase all of our experiments on 5 seeds. Upon inspecting the performance of individual seeds, we find that the performance drop is mainly due to a single seed <Seed value>. We would like to highlight that we strictly follow prior works for setting up our experiments (i.e. number of seeds etc). Further, while our reviewer presents an interesting hypothesis, the other experiments (8 of 9 domains) do not seem to “plateau”. However, our reviewer is correct in pointing out that PRIOR-obsession can be harmful, but we make that point to highlight the insufficiency of the PRIORs to capture human preference and the need of human feedback data. We believe that our reviewer’s comments indicate the importance of balancing the loss terms especially as the training progresses, which is an important concern for much of Machine Learning that presents auxiliary learning objectives.

Reviewer comment (Evaluation):
In Section 5.3, the authors find that a large coefficient on the prior loss leads to a collapse of the learned policy, indicating that the prior is assistive for policy learning only up to a certain point (e.g., relatively small preference data or small prior coefficient) and may eventually hurt performance if those limits are exceeded.

Author response:
On the performance plateau on Drawer Open domain : 
We showcase all of our experiments on 5 seeds. Upon inspecting the performance of individual seeds, we find that the performance drop is mainly due to a single seed <Seed value>. We would like to highlight that we strictly follow prior works for setting up our experiments (i.e. number of seeds etc). Further, while our reviewer presents an interesting hypothesis, the other experiments (8 of 9 domains) do not seem to “plateau”. However, our reviewer is correct in pointing out that PRIOR-obsession can be harmful, but we make that point to highlight the insufficiency of the PRIORs to capture human preference and the need of human feedback data. We believe that our reviewer’s comments indicate the importance of balancing the loss terms especially as the training progresses, which is an important concern for much of Machine Learning that presents auxiliary learning objectives.

Reviewer comment (Presentation):
The left two figures of Figure 3 are hard to read due to having 8 line plots on each small figure, which reduces clarity.

Author response:
We have taken our reviewer's suggestions to clarify H as "cross-entropy", and updated our equation (4) to use a more consistent notation. Furthermore, we have improved the figure captions to be more helpful to our readers in parsing figure 3 as suggested by our reviewer.

Reviewer comment (Writing):
The explanation of the hindsight prior is confusing and needs elaboration and clarification; specifically: (a) The transformer uses H attention heads, but H does not appear in the definition of the attention matrix A. (b) Equation (4) uses the notation $R_{target}$ and $\hat{R}$, but it is unclear how it is related to $\hat{\mathbf{r}}_{\psi}$ and $\mathbf{r}_{target}$ in the preceding line.

Author response:
On the explanation to H term : 
In Equation 3, H is the entropy term used by TWM (their Equation 4) to learn environment dynamics by encouraging similarity between the predicted latent next state and the true latent next state. They overload the notation of H to mean entropy (for entropy regularization) and cross entropy for latent state predictor. We borrow the notation from TWM work for consistency with their work. This objective is separate from the cross-entropy objective used to learn the reward function in PbRL.

TMW paper: Robine, J., Höftmann, M., Uelwer, T., & Harmeling, S. (2022, December). Transformer-based World Models Are Happy With 100k Interactions. In International Conference on Representation Learning 2023.  

R2: We have taken our reviewer's suggestions to clarify H as "cross-entropy", and updated our equation (4) to use a more consistent notation. Furthermore, we have improved the figure captions to be more helpful to our readers in parsing figure 3 as suggested by our reviewer.

Reviewer comment (Experiments):
It would be beneficial to note exactly how many trajectory labels such a framework requires. This would help detail whether such a framework would be feasible with actual end-users. Further, including actual tests utilizing this framework with human end-users would provide further evidence that PRIOR works well.

Author response:
On the amount of trajectory labels required by PRIOR :
We would like to highlight that the number of steps in Figures 2, 3, and 4 are the number of policy steps (x-axis). Following the backbone PEBBLE, at each step policy learning (SAC update) happens, every few RL steps the reward learning step is executed (for example, 5000 steps for Metaworld) along with query for new human feedback data, and every few RL steps the world model update happens (2000 steps for all experiments). The total amount of feedback is given by the end of policy training. Additional details on reward update schedules are available in Appendix E.1 for our reviewers. Therefore, the world model training steps are “in-between” RL episodes. Since the feedback schedule is based on the RL training steps, the plots highlight feedback efficiency and performance improvements shown by PRIOR. However, the training of the world model indeed poses an additional compute requirement but the additional world model training’s wall-clock time is insignificant compared to other costs like RL training, human feedback time (in a real world setup) etc. We recall that our choice of the world model backbone (TWM) is designed to be sample efficient as is shown by authors of TWM work on Atari benchmark. Moreover, such a cost can be easily reduced as the world model can be trained parallelly with RL training (and the two processes need to be synced only at certain episodes). Finally, we implemented a “serialized” version which performs each step i.e. policy learning, reward update, world model update in a sequential manner and found that the wall-clock time difference was no more than an hour when compared to baseline PEBBLE.

On User Study : 
We thank the reviewer for their suggestion on a human subject study with PRIOR. While additional experiments (as our reviewer highlights) can be helpful in understanding PRIOR’s gains, we believe that our current set of experiments thoroughly investigate Hindsight PRIOR. Specifically, we conduct experiments using “mistake oracle” with high degree of mistake value (i.e. oracle feedback is flipped with some $\epsilon$) and showcase the robustness of PRIOR over baseline methods to different frequencies of mistake. While a user study may help us find the exact number of feedback, our aim is to present PRIOR as a general approach and highlight the significant performance improvements over baselines.