Paper: Mixing Corrupted Preferences for Robust and Feedback-Efficient Preference-Based Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
In section 3, a strong assumption is made regarding the state and action space. Although the authors briefly mentioned the limitations in the future work section, a more comprehensive discussion of the implications of this assumption is necessary.

Author response:
In considering the assumption of convex states and actions, we concluded that it is not a crucial requirement for our method. The potential lack of real-world convexity in mixed trajectories does not negatively impact network learning. This is similar to the original mixup approach, which interpolates two image-label pairs that might not always be realistic. Despite this, mixup has shown considerable robustness against adversarial examples and incorrect labels, largely due to vicinal risk minimization.  
The primary benefit of mixup is its capacity to generalize and regularize learning process, prioritizing the overall data characteristics over the strict realism of individual augmented samples based on convexity. Moreover, studies involving the augmentation of virtual trajectories have shown significant performance gains without explicitly checking the validity of generated rollouts [1, 2]. Consequently, we have decided to remove the assumptions related to convexity and validity from the description of our method.  

References  
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.  
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

Reviewer comment (Experiments):
The experimental design could benefit from a more thorough comparison with state-of-the-art PbRL methods, particularly SURF, which also employs data augmentation for PbRL.

Author response:
Thanks for the insightful comments. We have expanded our experimental comparison to include all environments on SURF, as detailed in Appendix D.4 (Figure 12-14). Furthermore, we have implemented two additional novel preference-based RL methods, RUNE [1] and MRN [2], which were only referenced in the Related Works section of our previous manuscript. We have also updated the tables detailing the hyperparameters for these methods in Appendix C (Table 3-5).  

References  
[1] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.  
[2] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.

Reviewer comment (Novelty):
It may be pertinent to consider the paper on RL from diverse humans [1] as a closely related work that approaches the issue of corrupted human preferences in a distinct manner. This comparison would be crucial for evaluating the robustness in handling corrupted preferences. However, the authors may not be aware of this work, and they are not the first to address the issue of corrupted preferences.

Author response:
We appreciate the suggestion to consider the paper on RL from diverse human preferences [1] as a related work addressing corrupted human preferences. We have included the paper in the Related Works section. However, we encountered a challenge in conducting a direct comparison due to the unavailability of their method's code. We have reached out to the authors for access to their code. We are hopeful for a positive response and plan to update our manuscript with additional experiments upon receiving it. Meanwhile, to ensure a comprehensive comparison, we have re-implemented other publicly accessible methods [SURF, RUNE and MRN], as mentioned in our response to W2.  

References  
[1] Xue, W., An, B., Yan, S., & Xu, Z. (2023). Reinforcement Learning from Diverse Human Preferences. arXiv preprint arXiv:2301.11774.

Reviewer comment (Experiments):
While conducting real human experiments for PbRL in complex control tasks is challenging, the work could be further strengthened with user studies conducted with a diverse group of non-expert human labelers.

Author response:
We understand the value of including real human annotations in our study to further justify the effectiveness of MCP. However, we faced significant challenges, including the absence of a standardized protocol for human annotation and difficulties in maintaining control over various variables for fair comparison.  
To ensure a rigorous and fair evaluation, we followed the practices common in numerous established preference-based RL studies [1-4]. These studies typically utilize the B-Pref benchmark [5] for evaluations. The B-Pref benchmark is renowned for its effectiveness in simulating human irrationality, making it a robust and controlled environment for methodological comparisons. This benchmark allows us to assess the performance of MCP in a consistent and reliable manner, compensating for the absence of direct human-in-the-loop analysis.  

References  
[1] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.  
[2] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.   
[3] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.  
[4] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.  
[5] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021a. URL https://openreview.net/ forum?id=ps95-mkHF_.

Reviewer comment (Writing):
What are the potential negative impacts of corrupted preference? What might the newly generated trajectory look like? How do they mitigate the negative impacts? Could you provide some concrete examples for illustration? It is a bit vague to me.

Author response:
The negative impact of corrupted preference data is that it can lead to reward estimator that misinterprets user intentions, causing the agent to act inappropriately. Mixing preference data makes it more difficult to merely memorize the corrupted data, acting as a form of regularization [1]. The second benefit is that MCP improves data efficiency by creating unlimited new data, leading to a smoother reward distribution and enhanced generalization. Lastly, it also mitigates overconfidence problem [2], which can hinder the reward estimator learning. Without mixup, training with binary preference labels using binary cross entropy loss might cause significant reward divergence between two trajectories. This happens due to the tendency to assign overly low rewards to non-preferred trajectories (minimizing the likelihood of selecting non-preferred trajectories) and overly high rewards to preferred trajectories (maximizing the likelihood of selecting preferred trajectories). MCP adjusts for this imbalance in trajectory valuation. To evaluate MCP's confidence calibration, we additionally report the reliability diagram and expected calibration error [3] in Appendix D.6 and Figure 16.  

References  
[1] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018, February). mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations.  
[2] Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.  
[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.  

To clarify the concept of newly generated trajectories, we have added a visual example in Figure 7 of Appendix B. This illustration shows how two trajectories are mixed. Each trajectory segment consists of state-action sequences of length of $H$. A new trajectory is formed by linearly interpolating these sequences component-wise, based on the mixup ratio $\lambda$. Consequently, each component (state-action) of the mixed trajectory is a linear combination of corresponding components from the two original trajectories at the same indices.

Reviewer comment (Evaluation):
Could you briefly explain in which cases the proposed MCP approach may not work effectively?

Author response:
As mentioned in the conclusion section, the current implementation of MCP is designed for continuous action spaces. While MCP's reward learning is not inherently limited by the type of action space, experimental validation remains important. We recognize that its effectiveness in scenarios with discrete action spaces remains to be explored. This area will be focus of our future research efforts.

Reviewer comment (Experiments):
No analysis with actual humans-in-the-loop. Given that the main goal of MCP is to increase the robustness of PbRL, verifying whether MCP is indeed beneficial is an important ablation.

Author response:
### **W1. No analysis with actual human-in-the-loop**  

We understand the value of including real human annotations in our analysis to further justify the efficacy of MCP. However, we encountered challenges, such as the absence of a publicly verified human annotation protocol and difficulties in controlling other variables for a fair comparison.  
To ensure a rigorous and fair evaluation, we followed the practices of numerous established preference-based RL studies [1-4]. These studies typically rely on the B-Pref benchmark [5] for evaluation. The B-Pref benchmark is widely recognized for its ability to simulate human irrationality effectively. We believe that this benchmark provides a suitable and controlled environments for comparing methodologies. It allows us to assess the performance of MCP in a consistent and reliable manner, even without direct human-in-the-loop analysis.  

References  
[1] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.  
[2] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.  
[3] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.  
[4] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.  
[5] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021a. URL https://openreview.net/ forum?id=ps95-mkHF_.

Reviewer comment (Theory):
The assumption of convex states and actions seems pretty strong, and it is hard to gauge whether it has an actual effect on training.

Author response:
### **W2. Necessity of convexity assumption + Q1. Proportion of invalid state-action pairs**  

In considering the assumption of convex states and actions, we concluded that it is not a necessary condition for our method. While the mixed trajectory might not always adhere to real-world convexity, this doesn't negatively impact the network learning. This is similar to the original mixup approach, which interpolates two pairs of image and label that might not represent feasible real-world scenarios. Despite this, mixup has shown considerable robustness against adversarial examples and incorrect labels, largely due to vicinal risk minimization.  
The primary benefit of mixup is its capacity to generalize and regularize learning, focusing on the broader data rather than the precise validity of augmented samples based on convexity. Studies that have augmented virtual trajectories also reported significant performance gains without explicitly checking the validity of generated rollouts [1, 2]. Hence, we have decided to remove the assumptions regarding convexity and validity from the proposed method section to address these concerns.  
Nonetheless, we agree that checking the proportion of valid values is also beneficial. We tried to manually set mixed states in the simulator and input them with mixed actions. Unfortunately, the DMControl environments do not offer this functionality. However, in our exploration with the Metaworld environments, specifically the Button Press scenario, we manually set one million mixed transitions from the replay buffer. We found that all mixed actions were feasible at their respective mixed states. This feasibility arises because all actions in Metaworld environments are unconditionally available, and the mixed values fall within the min-max range of spaces.  

References  
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.  
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

Reviewer comment (Theory):
For W2, the authors removed the requirement for convex state and actions. It remains conceivable that the reward is trained with unfeasible trajectories, but this does not seem to negatively affect the performance of MCP.

Author response:
### **W2. Necessity of convexity assumption + Q1. Proportion of invalid state-action pairs**  

In considering the assumption of convex states and actions, we concluded that it is not a necessary condition for our method. While the mixed trajectory might not always adhere to real-world convexity, this doesn't negatively impact the network learning. This is similar to the original mixup approach, which interpolates two pairs of image and label that might not represent feasible real-world scenarios. Despite this, mixup has shown considerable robustness against adversarial examples and incorrect labels, largely due to vicinal risk minimization.  
The primary benefit of mixup is its capacity to generalize and regularize learning, focusing on the broader data rather than the precise validity of augmented samples based on convexity. Studies that have augmented virtual trajectories also reported significant performance gains without explicitly checking the validity of generated rollouts [1, 2]. Hence, we have decided to remove the assumptions regarding convexity and validity from the proposed method section to address these concerns.  
Nonetheless, we agree that checking the proportion of valid values is also beneficial. We tried to manually set mixed states in the simulator and input them with mixed actions. Unfortunately, the DMControl environments do not offer this functionality. However, in our exploration with the Metaworld environments, specifically the Button Press scenario, we manually set one million mixed transitions from the replay buffer. We found that all mixed actions were feasible at their respective mixed states. This feasibility arises because all actions in Metaworld environments are unconditionally available, and the mixed values fall within the min-max range of spaces.  

References  
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.  
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

Reviewer comment (Theory):
The main weakness of the method is it's not clear how exactly this method improves the performance, given that it's not clear what exactly the linear combination of state and action would mean and how using them for learning rewards could improve the robustness. Mixed-up values in each dimension could lie in *valid* state and action space because their value would lie between the min and max value of spaces, but as a whole, it's very unlikely that the state and action would be valid state and actions. For instance, linear combination of two proprioceptive states of 7-dof arm is very likely to lead to physically infeasible joint positions, and there's no guarantee that it'll be meaningfully connected to linear combination of labels. Without thorough investigation and support in this point, it's very difficult to justify the usage of method in many cases.

Author response:
### **Q1. How mixup improves performance and robustness?**  

The negative impact of corrupted preference data is that it can lead to reward estimator that misinterprets user intentions, causing the agent to act inappropriately. Mixing preference data makes it more difficult to merely memorize the corrupted data, acting as a form of regularization [1]. The second benefit is that MCP improves data efficiency by creating unlimited new data, leading to a smoother reward distribution and enhanced generalization. Lastly, it also mitigates overconfidence problem [2], which can hinder the reward estimator learning. Without mixup, training with binary preference labels using binary cross entropy loss might cause significant reward divergence between two trajectories. This happens due to the tendency to assign overly low rewards to non-preferred trajectories (minimizing the likelihood of selecting non-preferred trajectories) and overly high rewards to preferred  trajectories (maximizing the likelihood of selecting preferred trajectories). MCP adjusts for this imbalance in trajectory valuation. To evaluate MCP's confidence calibration, we additionally report the reliability diagram and expected calibration error [3] in Appendix D.6 and Figure 16.  

References  
[1] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018, February). mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations.  
[2] Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.  
[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.  
___  

### **Q2. Invalidity concerns about mixed state-action**  

We acknowledge that mixup-augmented data might lead to infeasible state-action pairs in complex real-world problems. However, our findings indicate that these interpolations, even if not entirely feasible in real world, do not hinder the reward estimator learning. This is similar to how the original mixup works by interpolating images and labels, which might not always result in realistic images. Still, the mixup has been effective in enhancing robustness against adversarial examples and incorrect labels, as well as improving data efficiency through vicinal risk minimization.  
The key aspect of mixup lies in its generalization for the network's learning process, as mentioned in the previous question. This prevents the network from merely memorizing the whole data, rather than the strict validity of augmented samples. Furthermore, other studies that create virtual trajectories without verifying their real-world validity have also reported significant performance enhancements [1, 2]. In response to the reviewer's concerns, we have removed the convex and validity assumption in the proposed method section.

References  
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.  
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

Reviewer comment (Evaluation):
Also the main weakness is that a lot of experimental results are not statistically significant, and the improvements over PEBBLE and SURF are very weak. At least the number of seed should be significantly increased and also other metrics from [rliable](https://github.com/google-research/rliable) could be considered.

Author response:
### **Q3. Statistical significance**  

Thanks for the suggestion. To address the concerns about statistical significance, we have now incorporated metrics from rliable library [1]. We have included these additional metrics in our analysis, including the interquartile mean and optimality gap, which can be found in Appendix D.4 and Figure 14. This approach provides a more comprehensive assessment of our method's performance compared to PEBBLE and SURF.  
In our pursuit of statistically significant comparison, we conducted ten independent runs per each experiment, as is standard in studies like PEBBLE [2] and MRN [3]. This number of runs is double that used by SURF [4] and RUNE [5]. Note that in the field of deep RL, 3-10 runs are typical due to computational constraints. Also, the metrics used in rliable are designed to address such challenge, providing reliable estimates even with a limited number of runs.

References  
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare.  
Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.  
[2] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.  
[3] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.  
[4] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.  
[5] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.