Paper: Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR-2024)

Reviewer comment (Experiments):
The main weakness is the limitation to control/locomotion tasks. More real world tasks/other modalities (such as text), and also larger architectures are required to make this eco-system more attractive.

Author response:
We would like to clarify that Uni-RLHF primarily focus on RLHF research for **decision making (locomotion/manipulation/navigation…)**, building upon the settings established by a series of mainstream research works such as Preference-PPO[1], Pebble[2]. These works concentrate on decision-making tasks too, and we have developed a labeling platform and benchmark that can accommodate a broader range of decision making tasks. 

Same as many widely used annotation tools, e.g. PixelAnnotationTool[3] for vision domain and Curve[4] for time series domain, Uni-RLHF also focuses on specific areas and hopes to build a robust eco-system for decision-making.  Therefore, we are able to focus on solving specific problems in decision tasks annotation, such as a robust environment access methods, feedback format encoding, and so on. Also, we modify relevant description of introduction to avoid confusion and add the statement that we specialise in decision making domain. (**See red text in the updated manuscript, page 1 and 7**)

Based on the above core design concepts, we will also continue to maintain and update the platform to support more real-world problem environments/datasets and more modalities.

Reviewer comment (Experiments):
The benchmark only compares offline RL, thus for example the de facto online strategy for RLHF (PPO) is not ablated.

Author response:
The reason we mainly use Offline RL is that we want to provide researchers with reusable labelled datasets to easily evaluate algorithms without additional labelling cost on a uniform benchmark. While decision making tasks in Online RL needs to learn from scratch, the experimental samples and annotations will be completely different from the algorithms, sampling methods, etc., which makes it difficult to reuse.

As **section 3.1** mentioned, Uni-RLHF is able to support the online training mode. We add a more detailed implementation of the online asynchronous annotation process in the revision. (**see Appendix E and Fig.8, page 21**)

To better show the effectiveness of the online mode of Uni-RLHF, we add an extra online RLHF experiment in the revision (**See Appendix E and Fig.9, page 23**). **After totally 200 queries of human annotation, walker can master the continuous multiple front flip fluently**. Through our experiments we demonstrated that Uni-RLHF can access online training to allow agent to learn novel behaviours that are difficult to design reward.

Reviewer comment (Experiments):
The different query samplers are not ablated.

Author response:
Offline RLHF experimental results with crowdsourced annotation are based on the random sampler implementation because of limited resource. However, we provide several ways of sampler definition in our platform to facilitate flexible selection by users and researchers, and **our focus is to provide an open and easily extensible platform to customise the appropriate query according to specific tasks.** Therefore we hope to open up better tools and remain some open research problems to RLHF community like how to choose the sampler. 

To better validate the role of different sampler, we collected new crowdsourcing labels and conducted the additional experiments to demonstrate the performance of different sampling methods. In each experiment, we collect **100 annotation labels** (comparative feedback) using a different sampler separately and use IQL-MLP-CrowdSource method for simplicity. **See Appendix D.3 (page 19)** of our revision for detailed experimental setup and results. 

| Env Name | Random Sampler | Disagreement Sampler | Entropy Sampler |
| --- | --- | --- | --- |
| antmaze-medium-diverse-v2 | $46.95 ± 11.91$ | **64.43 ± 6.57** | **62.69 ± 7.51** |
| hopper-medium-replay-v2 | $42.75±5.27$ | **63.28 ± 8.58** | 44.53 ± 6.83 |


We observe that suitable label sampler could improve the task performance,  this is also consistent with a similar conclusion by OPRL[5]. More ablation experiments are in process and we will continue to update the results in rebuttal phases.

Reviewer comment (Experiments):
Only the comparative feedback is ablated. It would have been interested to compare the quality of the reward models.

Author response:
Thank you for your valuable suggestions! In the current version we think that different feedback types are suitable for different task settings, e.g., comparative feedback is suitable for reward modelling, while attribute feedback for more complex multi-objective tasks and visual feedback for extracting visual representations, and it is very interesting to use a combination or to compare different annotation types in the same task, which we incorporate in our future work. (**See Appendix H, Some challenges and potential future direction**)

In the meantime, We have added further analysis and visualisation of the reward model in the revision (**See Appendix D.2, page 19**). We experimentally demonstrated that the reward model trained by human feedback can be well fitted to trends of the hand-designed reward function (**high relevance**) and aligned with human intent (**high prediction accuracy**).

Reviewer comment (Experiments):
How did you select the hyperparameters?

Author response:
We did not carefully select or adjust the hyperparameters for feedback model training, and we only used one set of hyperparameters for each structure (mlp, cnn, transformer).  It also shows that our training method is clean and robust. Full hyperparameters and experimental setups are shown in Appendix B. ****As for RL policy training, we keep the same hyperparameters as original Clean Offline RL[7] implementation for all experiments and do not further fine-tune them.  It's possible that tuning hyperparameters on the human-annoted datasets could bring benefits, but we believe it's orthogonal to our main contribution, which could be one of our future works.

Reviewer comment (Experiments):
Have you applied any strategy to refine the dataset quality?

Author response:
Yes, we want to open source high-quality research datasets and have taken the following approaches to improve annotation quality:

- **Optimisation of Task Instructions:** We refer to **Appendix J** for full instructions, all instructions were read by annotators unfamiliar with RL before being officially labeled and then optimized. And in gaming environments such as atari, we also let the annotators try it out.
- **Annotation Example:** For each task, we provide five well-annotated samples for the annotator to learn, including the how to annotate, annotation results and detailed explanations.
- **Validation Set:** We incorporate a small amount of expert validation set and discard labels that fall below the accuracy threshold.
- **Sampling Inspection:** We sample the existing annotation for inspection during the annotation process, reject the wrong samples, summarise the overall problem and optimise the task description.

Reviewer comment (Experiments):
Uni-RLHF's large-scale crowdsourced feedback datasets may contain noise and bias, which can affect the performance of RL algorithms trained on these datasets.

Author response:
We would like to clarify that noise and bias can't be avoided in pratical problems and annotations,  that's what we're focusing on. We hope to estimate human irrationality and bias in realistic annotators to futher study the RL algorithms in more realistic scenario. 

Several RLHF benchmarks[1][2] utilize predefined hand-engineered reward functions from the environment as scripted teachers, which can be considered a noise-free environment. Experimental results in **Table.2** show that **our crowdsourced human feedback can achieve competitive performance or outperform compared to noise-free scripted teachers**. 

In the data collection process, we try to minimise extreme noise and refine the dataset quality including continuous optimisation of task instructions, annotation example and sampling inspection.

We have added further analysis and visualisation of the trained reward model (**See Appendix D.2, page 19, revised version paper**) and experimentally demonstrated that the reward model trained by human feedback can be well fitted to trends of the hand-designed reward function (**high relevance**) and aligned with human intent (**high prediction accuracy**).

Reviewer comment (Experiments):
The system's offline RLHF baselines may not be optimized for specific applications, which can limit their usefulness in practical settings.

Author response:
We agree with the reviewer that the addition of more specific applications can increase the usefulness of the Uni-RLHF, which is also in line with our core motivation. Due to the high cost of labelling, previous research has mostly been in more ideal scenarios such as synthetic labels,  however, we provide annotation tools for multi-user annotation and a large number of  crowdsourced labels and this will facilitate the RLHF community to research in more realistic settings. Our main goal is to demonstrate the feasibility of a complete annotation pipeline and an open source reusable dataset. As for offline RLHF baseline, we only utilize the collected crowdsourced feedback datasets to verify the reliability of the Uni-RLHF system, which is not the core contribution of this work. We hope that more researchers will access their real-world problems in the platform to annotate and further optimise the framework based on this codebase.

Reviewer comment (Evaluation):
The system's reliance on human feedback may introduce additional costs and delays in the RL development process, compared to purely synthetic feedback.

Author response:
It is worth noting that in the most of RLHF application scenarios, **we do not have access to synthetic feedback. Purely synthetic labels exist only in ideal research simulation environments**, as a wide range of researchers would struggle to afford the cost of crowdsourcing labels in all experiments. The process of generating synthetic labels is as follows: 1) utilize predefined hand-engineered reward functions as scripted teachers, 2) score of trajectories by scripted teachers to obtain synthetic labels. In application scenarios, RLHF is mostly suitable for environments where manual design rewards are difficult, when additional costs and delays are unavoidable.

Take SMARTS domain as an example, designing reward functions is extremely difficult, requiring extensive expertise and multiple rounds of training attempts for iteration, the final training reward (marked as oracle) consists of five reward items of widely varying magnitude: Distance, Near Goal, Mistake Event, Lane-Selection and Time Penalty. Adjusting the reward function also requires a significant amount of cost, whereas crowdsourcing feedback annotations enable simply to achieve competitive performance to oracle and more consistent with human intentions. **(See Section 4.1.3, page 8 and Appendix D.1, page 18 for details)**

Reviewer comment (Reproducibility):
The claim for a "standardized feedback encoding format along with corresponding training methodologies" in Section 3.2 might be overly ambitious. Most importantly, the training methodologies are only provided in the appendix for two out of the five feedback types. Comparative feedback interface also does not allow the user to indicate the strength of preference, as done e.g. in the Anthropic's dataset for language model alignment [1]. The extension of the methodology in Appendix E.1 to this type seems straightforward (introduce y=(0.75,0.25), for example), and it could be beneficial to mention this.

Author response:
**[Q1: The claim for a "standardized feedback encoding format along with corresponding training methodologies" might be overly ambitious.]**

Thank you for your valuable suggestion! After careful consideration, we focus on proposing a set of feedback encoding format which are easy to use. We then train them with possible training methodologies based on adapting and improving existing literature, like comparative and attribute feedback in the previous paper version. So we change the statement to **standardized feedback encoding format along with possible training methodologies** in the revision.  (**See Section 3.2, page 5**)

Based on this core idea, we complement the revision with **new crowdsourced labels of keypoint feedback and corresponding experiments** (**See Appendix D.5, page 20**)**,** so the dataset and benchmark contains comparative feedback, attribute feedback and keypoint feedback now. And we provide **more possible training methodologies** **for the all 5/5 feedback types**. (**See Appendix G, page 5**) We will continue to research and expand the platform and datasets for diverse human feedback.

**[Q2: Comparative feedback interface also does not allow the user to indicate the strength of preference]**

We agree with the reviewer that it might be very meaningful to further allow the user to indicate the strength of preference. We have added this option to the platform design through a slider like Anthropic's dataset[1]. We've also updated the description and cited the paper in the **Appendix G.1, page 23:**

> It is worth noting that $y_\text{comp}$ can simply be extended to a finer-grained division of labels including the degree to which A is better than B. For example,  $y_\text{comp} = (0.75, 0.25)$ represents that A is better than B with a smaller degree than (1, 0). Also, the expansion method can simply be trained along the Eq. (4) without any changes.
>

Reviewer comment (Experiments):
Related to the previous point, the paper provides too few details on the way Atari experiments were performed. Appendix G and Section 4.1.1 imply that comparative feedback was collected, but Figure 5 (d) in the appendix -- that visual feedback was used instead. The highlighting in Table 8 should be explained. My guess is that the best of the ST and CS labels is highlighted.

Author response:
**[Q3: the paper provides too few details on the way Atari experiments were performed]**

> Related to the previous point, the paper provides too few details on the way Atari experiments were performed. Appendix G and Section 4.1.1 imply that comparative feedback was collected, but Figure 5 (d) in the appendix -- that visual feedback was used instead.
> 

Sorry for the confusion. We begin by clarifying that our atari experiments use comparative feedback for all, which is intended as an implementation of the CNN reward structure. In Figure 5 (d) in the appendix, we demonstrate that we can use the atari environment offline dataset for visual feedback annotation in the Uni-RLHF platform, but we do not provide the annotated dataset for visual feedback and the corresponding experiments due to limited resource in the current version, and we will continue to explore the performance gain of visual feedback in our future work.

> The highlighting in Table 8 should be explained. My guess is that the best of the ST and CS labels is highlighted.
> 

Yes, you are absolutely correct in your understanding! We use highlighting to mark the better one of the ST and CS labels. And, we increase the importance of atari experiments and provide clearer experimental details In the revised main text. (**See Section 4.1.2 Atari Experiments, page 8**)

Reviewer comment (Presentation):
Star annotations in Table 2 are incomplete. As I understand it, higher is better in the entire table. If that is the case, stars that should mark that the method performs better than the oracle are missing in several places: (hopper-m, CQL, CS-TFM), or (antmaze-u-d, IQL, CS-MLP). There are more missing stars in the table.

Author response:
**[Q4: About some confusion in full results of Table 2]**

We sincerely thank the reviewer for efforts to improve the quality of our papers, In revision, we have carefully examined the data and marks throughout the table, redesigned some of the annotation (blue highlight and *) and re-analysed experimental findings. (**Table 2, page 8**)

> Star annotations in Table 2 are incomplete.
> 

Sorry for the confusion. In the updated revision, we use * to mark the method that works best of all (usually Oracle). Since multiple algorithms are close in effect in many datasets, we take the best performing as the baseline, and if standard deviations in the rest of the algorithms intersect, they are similarly labelled as *.

Reviewer comment (Presentation):
Blue annotations in Table 2 are also significantly wrong, if I understand them correctly. Blue should mark the methods where crowdsourced labels (CS) show better results than the synthetic labels (ST) for the corresponding method. Then, for example, (walker2d-m-r, IQL, CS-MLP) should not be colored in blue, since (walker2d-m-l, IQL, ST-MLP) shows a better score (109.8 > 109.4). I counted 10 instances of such mislabeling in the IQL group alone. This is especially important since the paper claims that "the performance of models trained using crowd-sourced labels (CS) tends to slightly outperform those trained with synthetic labels (ST) in most environments, with significant improvements observed in some cases." Once the blue labels are revised, it is questionable whether this claim still holds. Other offline RL approaches (CQL and TD3BC) show that CS labels are better more consistently, but these approaches perform worse than IQL, so the results for them are correspondingly less important.

Author response:
**[Q4: About some confusion in full results of Table 2]**

We sincerely thank the reviewer for efforts to improve the quality of our papers, In revision, we have carefully examined the data and marks throughout the table, redesigned some of the annotation (blue highlight and *) and re-analysed experimental findings. (**Table 2, page 8**)

> Blue annotations in Table 2 are also significantly wrong.
> 

We apologise for the unclear description. In our definition, blue is used to highlighted which of the CS-MLP and CS-TFM methods is better and we've carefully checked in the revised version.

> About performance comparison using ST and CS.
> 

Thanks for the correction. We modify the description in the revision to argue that CS can achieve competitive performance with ST in IQL backbone, even script teacher (ST) can be regarded as an expert with some degree. And other offline RL approaches (CQL and TD3BC) show that CS labels are better more consistently. We obtain a similar conclusion in atari domain (image observation and discrete action space).

Reviewer comment (Experiments):
Some CS-TFM results are missing from Table 2. I did not find the explanation for this, maybe I missed it? It would be helpful to see these results in the final version, to better assess the claim of the paper that "CS-TFM exhibits superior stability and average performance compared to CS-MLP".

Author response:
**[Q4: About some confusion in full results of Table 2]**

We sincerely thank the reviewer for efforts to improve the quality of our papers, In revision, we have carefully examined the data and marks throughout the table, redesigned some of the annotation (blue highlight and *) and re-analysed experimental findings. (**Table 2, page 8**)

> Some CS-TFM results are missing from Table 2.
> 

We filled in all the missing results in Table 2 and reanalysed them based on complete results.

Reviewer comment (Evaluation):
For the SMARTS experiment in Section 4.1.2, the paper claims that "the best experimental performance can be obtained simply by crowdsourcing annotations, compared to carefully designed reward functions or scripted teachers." I would say that from the three tasks used one cannot conclude that carefully designed rewards perform worse than crowdsourced annotations. In table 3, the former outperforms the latter on two out of three methods. The claim seems to be made based on average success rate, which only differs by 0.01, less than one standard deviation.

Author response:
[**Q5: For the SMARTS experiment in Section 4.1.2, the paper claims that "the best experimental performance can be obtained simply by crowdsourcing annotations, compared to carefully designed reward functions or scripted teachers." I would say that from the three tasks used one cannot conclude that carefully designed rewards perform worse than crowdsourced annotations. In table 3, the former outperforms the latter on two out of three methods. The claim seems to be made based on average success rate, which only differs by 0.01, less than one standard deviation**.]

We apologise for the overly ambitious summary and provide more experimental phenomena for in-depth analysis. We provide the design process for the reward items in **Table.5, page 16**, and more evaluation and visualisation of multi indicators in **Appendix D.1**. We find that simple human feedback annotations still work well on the evaluation of more metrics, which is consistent with human intentions. We modified the summary in **Section 4.1.3** to “We empirically demonstrate that we can achieve competitive performance simply by crowdsourced annotations, compared to carefully designed reward functions or scripted teachers.”

Reviewer comment (Experiments):
In Figure 4, speed and torso height are plotted against time. Every 200 steps, the speed attribute value changes (1 to 0.1 and back), so we see the respective changes in speed on the first plot. The relative strength of the "torso height" attribute, however, does not change (stays at 0.1), and the torso height parameters do not change much between the changes. It would be more interesting to see the results where the strength of torso height also changes, so that we can see that it influences the plot.

Author response:
**[Q6: In Figure 4, speed and torso height are plotted against time. Every 200 steps, the speed attribute value changes (1 to 0.1 and back), so we see the respective changes in speed on the first plot. The relative strength of the "torso height" attribute, however, does not change (stays at 0.1), and the torso height parameters do not change much between the changes.]** 

Thanks for pointing this out, as you understand, we hope to show with this experiment that by training jointly on multiple attributes strength model, we can allow agents to adjust attributes during motion. Changes in speed follow changes in target attributes very well and are a very good example. And we notice that the instruction-following changes in torso height are not significant enough. In fig.4, 400 steps (0.6→1) and 600 steps (1→0.1) showed changes in line with expectations, while the other changes showed less obvious changes. We've uploaded the full video on our homepage, where the walker switches postures multiple times as the person commands it, which is very interesting, and the task is difficult to design for manual rewards.

In the rebuttal phase, We checked and repeated experiments carefully. we found that this may be due to the limited generalisation and alignment capabilities of the context policy of TD3BC baseline, and we expect researchers to extend the baseline (e.g. more stable attribute aligned model or stronger backbone) for an in-depth study based on the attribute feedback.

Reviewer comment (Experiments):
Generally speaking, RLHF is most useful in domains where only the humans understand the true reward. This is the case, for example, with "helpfulness" of an LLM, or with "humanness" of the gait of a walking robot. An important evaluation, then, is to see whether the human evaluators prefer the policies trained with an RLHF approach in terms of these hard-to-define measures. This paper, however, does not present such an evaluation. "Humanness" comparisons are collected as shown in Section 4.2, but it is never compared to a policy that is trained without taking humanness into account.

Author response:
**[Q7: An important evaluation, then, is to see whether the human evaluators prefer the policies trained with an RLHF approach in terms of these hard-to-define measures.]** 

Thanks for your constructive suggestion. We strongly recognise that the RLHF should primarily address the evaluation of factors that are difficult to quantify, we also raise it in our future work in the **appendix J, page 22**. 

In the meantime, we add the following experiments:

- **Human evaluation:** we conducted human evaluation experiments for the agents trained by attribute feedback. We first collected 3 trajectories with different humanness, which were set to [0.1, 0.6, 1] and invited five human evaluators. The human evaluators performed a blind selection to judge which video is most human-like and which video is least human-like. If the result is consistent with the preset labels, it means that the agents can follow the humanness indicator well. Finally, all people correctly chose the highest humanness trajectory, and only one person chose the lowest humanness trajectory incorrectly. The experimental results confirm that the agents through RLHF training are able to follow the abstraction metrics well. (**See appendix D.4, page 22 and full videos on the homepage**)
- **Complex behavioural learning:** we add an extra online RLHF experiment in the revision (**See Appendix E and Fig.9, page 23**). **After totally 200 queries of human annotation, walker can master the continuous multiple front flip fluently**. Through our experiments we demonstrated that Uni-RLHF can access online training to allow agent to learn novel behaviours that are difficult to design reward.