Paper: Query-Efficient Offline Preference-Based Reinforcement Learning via In-Dataset Exploration (ICLR-2024)

Reviewer comment (Presentation):
_W1_: Though the individual sections are really well written, I had trouble following the paper. To mitigate this, I would suggest better delineating the article in the introduction, highlighting key take-aways in each section, adding a summary of Pessimistic Value Iteration (which OPRIDE builds on) to the main paper, and providing more examples. The mathematical analysis is very general (this is a strength of the paper), but providing examples of how it is instantiated for linear MDP with Bradley-Terry preferences would aid in comprehension.

Author response:
We thank the Reviewer for pointing this out. As suggested, we highlighted key takeaways in each section, added a summary of Pessimistic Value Iteration to the main paper, and provided more examples in the revised version.

Reviewer comment (Experiments):
_W2_: There is still a question whether the same gains observed in MetaWorld and AntMaze apply to other environments such as mujoco-gym and perhaps from human preferences too. (Preference Transformer tested against all these environments)

Author response:
To investigate whether our method is generally applicable to other environments, we conduct experiments on three mujoco tasks, including hopper, walker2d, and halfcheetah. As shown in Table 1, our method outperforms the baseline method PT, especially on the hopper tasks.


| | OPRIDE | PT |
| :---: | :---: | :---: |
| hopper-medium-v2 | **47.5$\pm$2.3** | 36.9$\pm$2.1 |
| hopper-medium-expert-v2 | **72.1$\pm$1.5** | 68.0$\pm$2.6 |
| walker2d-medium-v2 | 71.8$\pm$2.4 | 71.7$\pm$2.6 |
| walker2d-medium-expert-v2 | 108.5$\pm$0.2 | 109.4$\pm$0.3 |
| halfcheetah-medium-v2 | 42.4$\pm$0.2 | 42.1$\pm$0.1 |
| halfcheetah-medium-expert-v2 | **88.8$\pm$0.8** | 81.9$\pm$0.1 |
| Average | **71.8** | 68.3 |

Table 1. Additional experimental results on Mujoco tasks.

Reviewer comment (Writing):
_Q2_: What is $T$ on the theoretical guarantees section? Do you mean $K$ the number of queries?

Author response:
Thanks for pointing this out. This is a typo, and it should be $K$ rather than $T$ in the theoretical guarantee section. Yes, $K$ means the number of queries. We have corrected this in the revised version.

Reviewer comment (Evaluation):
_Q3_: In figure 4, for `push-v2`, both baselines beat OPRIDE at K>15. Why is that? Is there a downside to OPRIDE as the number of queries goes up?

Author response:
This is because push-v2 is a relatively simple task, and the performance saturates with a few queries. To verify this,
We evaluate the performance of the algorithms with 30 queries on push-v2, and the result is shown in Figure 4. We can see that OPRIDE performs well compared with other methods when the number of queries goes up.

Reviewer comment (Writing):
When introducing equation 5, explain that SubOpt is the sub-optimality measure, it will help readers later on.

Author response:
We appreciate the valuable suggestions of the reviewer, and we have corrected them in the revised version.

Reviewer comment (Writing):
In equation 6, mention that $o$ is the ground-truth human preference.

Author response:
We appreciate the valuable suggestions of the reviewer, and we have corrected them in the revised version.

Reviewer comment (Writing):
In section 3.2 $N$ is meant to use the number of reward functions, but $N$ was earlier used to indicate the size of the offline dataset.

Author response:
We appreciate the valuable suggestions of the reviewer, and we have corrected them in the revised version.

Reviewer comment (Presentation):
In "Answer to Question 1", _tables 3 and 2_, should be _tables 1 and 2_ instead.

Author response:
We appreciate the valuable suggestions of the reviewer, and we have corrected them in the revised version.

Reviewer comment (Theory):
The motivation for the augmentation with a truncated Gaussian is unclear.

Author response:
**A for Q3 & Q4:**
The motivation for using weighted queries is explained in W1.

We clip the range of the weight to avoid large weights that can lead to high variance in the learning process, as commonly used in practice [3].
We conduct additional ablation studies to demonstrate the importance of weight clipping in the chain MDP experiment shown in Figure 10 (a) in Appendix G. 

Using other random variables is also possible, but we found that Gaussian weights perform better in practice. Please refer to Appendix G's Figure 10 (b) for an ablation study.

Reviewer comment (Experiments):
The chain experiment is not described well in the text or the appendix.

Author response:
**A for W2:**
We thank the reviewer for pointing this out, and we gave a more detailed description of the chain MDP experiment in Appendix E in the revised version.

Reviewer comment (Theory):
The justification for the use of projected MLE, or even the definition of projected MLE, is missing from the method section.

Author response:
**A for W3:**
By assumption, the parameter $\theta$ for the reward function is bounded, but an MLE estimator can generally be unbounded. Therefore, we project the MLE estimator to the space with bounded norms, ensuring the estimation is in the confidence set [1,2]. We have made this motivation more clear in the revised version.

Reviewer comment (Writing):
There are a substantial number of typos and errors in the writing.

Author response:
**A for W4:**
We thank the reviewer for pointing it out, and we have substantially revised the manuscript.

Reviewer comment (Writing):
It is not clear what the counterfactual trajectories are.

Author response:
**A for W5  Q8:**
The counterfactual trajectories in Figure 2(c) refer to the two colored trajectories that are out of the dataset. We can not infer the preference between these two out-of-dataset trajectories if weighted queries are not allowed. 

As explained in W1, by adding weights to trajectories, we can learn better credit assignments. 
We can use the former 2-stage task in W1 as an example. 
Without weighted queries, we can not infer the preference between the counterfactual trajectories $\tau'\_1$ and $\tau'\_2$, where $\tau'\_1$ acts as $\tau\_1$ at step 1 and acts $\tau\_2$ at step 2, and $\tau'\_2$ acts as $\tau\_2$ at step 1 and acts $\tau\_1$ at step 2. 
With weighted queries, we can identify the preference at each step.
Therefore, weighted queries help reason about preferences between 
 the counterfactual trajectories $\tau'\_1$ and $\tau'\_2$ without generating new trajectories.