Paper: Deep Reinforcement Learning from Weak Hierarchical Preference Feedback (ICLR-2024)

Reviewer comment (Writing):
The paper is not very clearly written and it left me with a few unclear points which are important for the proposed problem setting. The authors start by talking about human feedback in reinforcement learning, but it is not clear from the method description what feedback is provided by humans. Is it the order of factors? Or is it feedback in the form of comparisons? What exactly is the problem setting? For example, what are the assumptions about the relationship between the reward factors and the true reward?

Author response:
**Weakness 1: Lack of clarity**
- The general setting we consider here is one where at each time step, we receive a state observation, take an action, and then receive the next state as well as a list of reward factors. This setting is the same as that commonly used for reward engineering. Our setting is described in detail in Section 3.1 of the submitted manuscript.
- Given this setting, HERON asks the human overseer to assign rankings over the order of the set of reward functions, $z_1, …, z_t.$ A concrete example would be (wait time > queue length > number of emergency stops) in traffic light control. This is described in the weak preference elicitation paragraph. These rankings are then used for learning.
- We make the assumption that a decision tree over the reward factors will be able to approximate the true reward. Using this assumption has two main benefits: (1) constructing the decision tree requires less human input than the typical linear combination that reward engineering employs and (2) decision trees are highly flexible, and can therefore provide a good approximation of the true reward in most practical scenarios. Our experimental results back such an assumption.

Reviewer comment (Theory):
Assuming that humans provide the comparisons in the form of weak preferences, it is not clear to me how it can be ensured that comparisons are in the form described on page 4 (fixed precise threshold, always correct). Also, usually in RLHF the humans would provide comparisons of full trajectories because this is a more reliable feedback compared to other types of feedback. Assuming that the comparisons are coming from the algorithm itself, it seems to me that the method is just a form of reward engineering that allows one to find the coefficients by ensuring certain rules (the importance order of factors) are respected.

Author response:
**Weakness 2: reliable comparisons**
- In HERON, the comparisons are coming from the algorithm itself, and there is no human involved in the preference elicitation stage (see Algorithm 1). 
- Indeed, HERON is a new reward design algorithm, but we believe that it is a novel and significant contribution: no algorithms have been proposed to use the type of rankings that HERON uses, and HERON is a principled method that can outperform classical reward engineering approaches, only utilizing the rank of the reward factors and gaining robustness.

Reviewer comment (Theory):
One limitation of the method is that it requires reward factors which are hand crafted. In many real world applications of RL there has been a tendency of moving away from the hand crafted rewards and learning reward functions directly from the observations which are represented in the raw form (e.g., images or text). I would like to hear a discussion on the applicability of the method to the various realistic domains.

Author response:
**Weakness 3: Reliance on reward factors**
- Thank you for bringing this point up. Indeed, HERON is designed for the classical setting where a set of reward factors is available. We believe that such a setting can still be useful, as demonstrated by our experiments in code generation, robotics, and traffic light control, all of which are practical environments where learning directly from human feedback would be extremely costly.
- Moreover, we believe that using reward factors could boost emerging applications which require reward learning directly from the input, such as LLM finetuning. For example, in LLM finetuning, we could use tools such as grammar checkers and unit tests as the reward factors to allow HERON to be used. Such tools could even be used in conjunction with human feedback.

Reviewer comment (Experiments):
I have some questions and concerns about the experiments. For example, in all three environments the authors say that the ground truth reward is a linear combination of the factors. In this case it is not clear to me why one would need a non-linear function (MLP) for the learnt reward. Then, regarding the baselines, a classical baseline would be a (potentially linear) reward learnt as a function of factors on the basis of trajectory comparisons from human annotators. This is the most classical RLHF setting and it would be informative to compare the proposed method to it.

Author response:
**Weakness 4: Experiments and comparison to RLHF**
- A key advantage of including the trajectory as the input of the reward model is that doing so allows us to take the trajectory context into consideration, which reward engineering does not allow for. For example, in the code generation task, we would like the reward model to give a higher score to a program if it is almost correct compared to when it is completely incorrect. This can only be done by using the trajectory as the reward model input. Also, for code generation we fine tune a pre-trained model to obtain our reward model, allowing us to transfer domain knowledge obtained through pre-training into the reward model. 
- However, we believe that the main novelty and contribution of our work comes from how we use the reward factors to elicit preference labels. Once we have these labels, any technique can be used to learn the reward function, including linear regression with the weak reward factors as inputs. 
- To address your second point, we remark that our algorithm is not meant for the setting where a human annotator is used to compare each trajectory. Therefore we do not include RLHF in our main experiments. However, we do include a comparison of HERON to RLHF in Appendix F.

Reviewer comment (Evaluation):
I didn't understand why GT line is worse than other methods in Figure 3(c).

Author response:
**Minor Point: Why GT performs worse than other methods**
- We believe that GT performs worse than other methods due to reward shaping [1]; although the GT reward is optimal, it may not always provide enough information during training, therefore resulting in worse performance.

Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." Icml. Vol. 99. 1999.

Reviewer comment (Experiments):
Section 4.3 argues that the proposed method is quite flexible. It would be nice to hear a discussion of how this compares to the studied baselines such a reward engineering.

Author response:
**Minor Point 2: Flexibility compared to Reward Engineering**
- Thank you for the suggestion. We believe that reward engineering and HERON provide two different types of flexibility: reward engineering will be able to better model rewards that are linear in the reward factors, while HERON will be better suited to model highly non-linear rewards.
- In our experiments, we find that HERON, although only using rankings over reward factors, is still quite flexible. This indicates that the fine-grained control RLHF allows may be redundant. HERON exploits this phenomena, allowing for reinforcement learning with less feedback. We can increase the flexibility of HERON by changing the margin parameter on each reward factor, therefore decreasing or increasing its significance in HERON’s preference elicitation process. We have added such discussion to the conclusion of the paper.

Reviewer comment (Novelty):
I would like the authors to elaborate on the problem setting and assumptions (in particular relationship between the factors and true rewards). Also, I would like to understand better the principal difference of the proposed method to reward engineering and to see comparison to a more traditional reward learning scenario from RLHF.

Author response:
**Question 1: Problem setting description**
- We include a description of our problem in Section 3.1 of the submitted manuscript.
- We do not make any strong assumption on the ground truth reward, but we do expect that the hierarchical comparison rule created by HERON will be aligned with the comparison rule induced by the ground truth reward. We believe that this assumption is reasonable, as the human overseer has ranked the reward factors by importance.

**Question 2: Distinction from reward engineering.** 
- We describe classical reward engineering and its drawbacks in paragraphs 2 and 3 of our introduction. 
- Concretely, reward engineering assumes the ground-truth reward can be formed as a simple combination of rewards factors, an assumption that may suffer from large approximation bias. In addition, reward engineering usually requires a large tuning cost, as the optimal reward weights are not necessarily intuitive to human overseers and therefore must be chosen through trial and error.
- In contrast, HERON does not require a human to assign a weight to each reward factor, and instead can learn a reward from a simple ranking of reward factors. In addition, the decision tree used by HERON can be highly non-linear, possibly allowing for more complex rewards to be learned. Finally, HERON allows for the context (trajectory information) to be taken into account during the reward modeling process, which reward engineering does not. 

**Question 3: Comparison to RLHF**
- We emphasize that the goal of this work is not to replace RLHF, and that HERON and RLHF consider two different settings. We include a comparison to RLHF based learning in Appendix F, where we further examine the difference between HERON and RLHF. 
- In this setting we find that HERON can outperform RLHF, as the reward factors are quite informative. We remark that HERON and RLHF are two different algorithms, although they have a similar structure. HERON requires more information than RLHF (in the form of reward factors) but less information than reward engineering.

Reviewer comment (Experiments):
Marginal Improvement: HERON demonstrates only marginal or no substantial improvement over straightforward heuristic baselines in many environments, reducing confidence in its efficacy.

Author response:
**Weakness 1: Marginal Improvements**
- We would like to point out that the improvements over comparable baselines are not marginal. In robotics, HERON results in a 7.8% improvement compared to reward engineering. For traffic light control, HERON beats all other baselines by 33%. For Coding, HERON can beat the CodeRL method by 3.9% as measured by Pass@K and by 5.2% as measured by filtered Pass@K, while greatly reducing the tuning cost. These improvements are displayed in Tables 1, 2, and 3 and Figure 3c of the submitted manuscript.
- Moreover, we would like to point out that our method is designed to learn from rankings, greatly reducing the tuning cost of reward design. Therefore the goal of our experiments is not necessarily to beat carefully tuned rewards, but to show that we can match them while only using rankings over reward factors.

Reviewer comment (Experiments):
Unconvincing Comparison: Directly comparing HERON to standard RLHF methods, which often operate in settings with unavailable or human-annotated reward factors, may not provide a fair or meaningful benchmark.

Author response:
**Weakness 2: Unconvincing Comparison**
- We believe there is a misunderstanding here. In our main experiments, we do not compare our method to RLHF techniques as RLHF considers a completely different setting than our paper considers (see section 3.1 for our problem description). Instead, RLHF serves as the motivation for our HERON algorithm, as mentioned in the introduction. HERON is designed as an alternative to reward engineering.
- We primarily compare our method with (1) policies trained with the ground truth reward (2) heuristic algorithms that receive the same amount of information of HERON (see the description of baselines on page 6) and (3) State-of-the-art reward engineering approaches for the Coding task. All of these algorithms either receive more or an equal amount of information in the reward design process than HERON, and are therefore meaningful baselines.

Reviewer comment (Experiments):
Performance Comparison: How does the proposed method compare to existing state-of-the-art approaches in terms of performance, and what are the key differentiators? Comparative analysis is crucial to demonstrate the novel contributions of the proposed method. Evaluating its performance against existing methods (such as PEBBLE or Meta-RewardNet in control domain) highlights its strengths and areas where it excels. I think that comparison with existing RLFH methods is important since this paper targets the same problem in RL. (If this paper targeted only NLP domain or focused on the contribution in code generation, I think it is not essential.)

Author response:
**Question 1: Comparison with RLHF Baselines**
- As we previously mentioned, our work is inspired by RLHF, but is not designed for the RLHF setting. Therefore it would not be meaningful to compare it to methods like PEBBLE and MetaReward-Net, which we consider to be orthogonal directions of work. However, we do include a comparison to RLHF based learning in Appendix F, where we further examine the difference between HERON and RLHF.

Reviewer comment (Writing):
Clarity: It would be helpful to make the setting clearer. My understanding is that the input to the reward model training process is no longer preference data ($ \{ (\tau_w, \tau_l), \dots \} $), but an ordered list of reward factors. In other words, the reward model training part is not changed, but only the input to the reward model training is changed.

Author response:
**Weakness 2: Lack of clarity on the setting.**
- We believe there is a misunderstanding here. The general setting we consider is one where at each time step, we receive a state observation, take an action, and then receive the next state as well as a list of reward factors. This setting is the same as that commonly used for reward engineering. Our setting is described in section 3.1 of the submitted manuscript.
- With regards to the reward modeling setting, the only difference with RLHF comes from how we obtain the preference labels. The input to the reward model is still the trajectory segments, but now the preference label is obtained from HERON’s decision tree over the reward factors. This forms the key novelty of our algorithm. Otherwise all aspects remain unchanged from standard reward learning in RLHF. These details can be found on page 4 of the submitted manuscript .

Reviewer comment (Writing):
In Alg 1, the condition in while is not syntactically correct ($l \leq n \ \mu = 0$).

Author response:
**Minor comments.**
Thank you for pointing out the error in Algorithm 1 and suggesting improvements for Figure 3a. We have fixed the algorithm in the updated paper and edited Figure 3a to make the curves more visible.

Reviewer comment (Presentation):
You may move the legend in Fig. 3 out of the plot so readers can see the plot clearly.

Author response:
**Minor comments.**
Thank you for pointing out the error in Algorithm 1 and suggesting improvements for Figure 3a. We have fixed the algorithm in the updated paper and edited Figure 3a to make the curves more visible.

Reviewer comment (Theory):
Lack of ability in personalization. Unlike the standard preference-based learning framework, the ability to obtain personalized rewards is limited. Humans only can specify the ranking between objectives and it is hard to control the tradeoff between objectives. For example, let's consider a case where there are two objectives. Assume that there are two human annotators with the same ranking but the first human wanna get a reward function that emphasizes the first objective more. In this case, this framework can't provide a different reward for two human annotators (because the ranking is still the same). It would be nice if the authors could discuss this limitation.

Author response:
**Weakness 2: Lack of ability in personalization.**
- This is an interesting point. Beyond changing the ranking, we can increase the flexibility of HERON by tuning the margin parameters in the HERON framework. For example, if we would like to increase the emphasis on a given reward factor, we can decrease the size of the margin for that reward factor. In this way, a greater percentage of the decisions would be taken according to the reward factor we wish to emphasize. In the updated version, we have added discussion about how to increase flexibility in section 4.3, writing “To further increase the flexibility of HERON, we could tune the margin parameter of each reward factor, therefore impacting the fraction of decisions made at each level of the decision tree.”
- In addition, in the case where we have two factors, we can randomly drop the comparison based on the second factor is we would like to increase the dependence on the first factor.
- Finally we remark that HERON has two additional benefits for enhancing the capability of reward modeling, which the linear combination of the reward factors does not have:
(1) HERON allows us to take advantage of the contextual/state information contained in the trajectories by predicting the reward based on the trajectory and (2) for certain applications, e.g. code generation, it allows us to use pre-trained models as the initialization of the reward, and therefore vast amounts of domain knowledge (e.g., semantic and syntactic knowledge of code) can be transferred to the learnt reward.

**For personalization, tuning margin parameters makes sense but why not adjust weights (over multiple objectives) directly instead?**
We remark that HERON and reward engineering are two different design approaches, and which one works better will depend on the problem. However, when domain knowledge such as the importance ranking of the reward factors are available, HERON provides a much more intuitive way to do reward modeling than heuristic reward engineering. Note that the importance ranking of reward factors does not necessarily help determine their reward engineering weights, as the reward factors could be correlated, the scale of the reward factors may be different, and the reward factor scales could even change throughout training. HERON allows us to bypass or mitigate all three of these concerns.

Reviewer comment (Experiments):
Learning curves on robotics environments. It would be nice to include the learning curves on robotics environments in the main draft or appendix.

Author response:
**Question 1: Learning curves in robotics environments**
- We have included the curves in Appendix K of the updated paper. From these curves, we observe that HERON matches or beats the performance of reward engineering while being much more stable.