Paper: PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback (ICLR-2024)

Reviewer comment (Experiments):
I am not very familiar with the literature of the benchmarks used in this area but I go through the papers along the line and confirm that the experiments conducted by the authors are commonly adopted in the literature so I assume that the experiments are satisfactory (which may need to be further evaluated by other expert reviewers).

Author response:
Thank you for your feedback regarding the benchmarks used in our experiments. We appreciate your efforts in reviewing relevant literature to understand the context of our work. We remark that, for this work, our main contribution (as you mentioned in strengths) is to provide a unified novel theoretical framework (A-PARL) to address the issues of the existing literature. To support that, we did a rigorous theoritical analysis of the policy alignment problem, which is missing from existing literature (Deep RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), Surf (Park et. al 2022)) etc. Hence, we performed experiments on the three domains as a proof of concept of our algorithm and theoritical analysis.

Reviewer comment (Experiments):
Could you comment on the generality of the proposed framework? In particular, I am interested in whether this framework can be extended to handle the alignment of language model because the reward model quality is the bottleneck in many cases and most of the existing approaches use only a fixed reward model.

Author response:
This is an excellent point. Indeed, the proposed framework holds promise for extension to the alignment of large language models (LLMs). A significant benefit in the context of LLMs is the deterministic nature of transition dynamics at the lower level. This occurs because we append the new token with the input segment (current state) to generate the next token. This deterministic aspect simplifies and strengthens the framework's application in LLM alignment. We acknowledge this as a valuable direction for future research.

Reviewer comment (Writing):
The objective of the paper is somewhat confusing. Would the authors please clarify the cause of the misalignment that is considered, and what role the human agent plays?

Author response:
**Response to Weakness 1** Thanks for this opportunity to clarify. The misalignment primarily arises in RL due to the misspecified reward function because true reward is usually unknown in the majority of the practical scenarios. A human feedback helps to learn the true reward function as also highlighted in (Christiano et. al 2017, Lee et. al 2021, Park et. al 2022). However, interestingly, the existing approaches of learning from human feedback such as (Deep RLHF, PEBBLE, SURF) suffer from misalignment for not considering the interdependence between the reward and policy learning leading to sub-optimality (as highlighted in our paper, Sec 3.1, 3.2, Figure 1). Incorporating this dependence through a bilevel formulation helps in mitigating the performance gap of prior algorithms, as shown in Figure 2. 

Hope this clears the confusion. Please let us know if there are any further confusions regarding this point, we are happy to discuss and elaborate further.

Reviewer comment (Theory):
Is there a fixed RL problem, but the agent can only get some feedback from a human who partially knows the real reward, and such feedback changes over time as the human learns more about the reward?

Author response:
**Response to Weakness 2.1:** Yes, our formulation considers a fixed RL problem. The problem deals with the setting (similar setting to Deep RLHF, Pebble, Surf etc.) where there is a ***fixed ground truth reward function*** $r_{\nu^*}$ ($\nu^*$ is the optimal reward parameter unknown to RL agent).  The human feedback corresponds to the optimal reward function which is received by the RL agent as preferences on querying the human with relevant trajectory pairs. 

In our work, to learn optimal $\nu^*$ from the human feedback, the existing state-of-the-art methods ignore the dependence on the optimal parameter of the policy $\theta^*$. We highlight this drawback and propose to resolve it via formulating a novel bilevel formulation where we iteratively update the parametrized reward function $\nu_t$ at the outer objective (shown in equation 8, 14) by collecting human feedback on trajectories generated by solving the inner optimal policy (shown in equation 15). This is the exact setting of our formulation and is important to note that the true $\nu^*$ doesn't change over time and is fixed, as also followed in previous baselines.

Reviewer comment (Writing):
More precisely, the authors used $\nu$ to parameterize the upper level (evaluating the lower-level policy). What does each $\nu$ represent?

Author response:
**Response to Weakness 2.2** In our formulation, $\nu$ represents the parameter for the reward model $r_{\nu}$ and the objective is to learn the ground truth reward parameter $\nu^*$.

Reviewer comment (Experiments):
Experiments are not detailed. Descriptions and simulation setup are missing. For instance, in figure 2, what precisely do 'walker', 'door open', or 'button press' aim to do? How does the misalignment occur? What does the oracle represent? What are the upper level objective functions $G(\nu, \theta^*(\nu))$ for these tasks?

Author response:
**Response to Weakness 2.3** Thank you for raising this concern. Since we have followed the exact similar enviromental configuration and setup used in previous benchmarks, we cited the reference (Christiano et. al, Lee at. al, Park et. al) without specific descriptions. But we agree with the reviewer that providing a detailed description will definitely be helpful for the readers and hence we have updated [in Appendix K](https://openreview.net/pdf?id=ByR3NdDSZB) of our paper with relevant details.

Reviewer comment (Writing):
On top of page 4, the author mentioned $\nu \in \mathbb{R}^n$. What does $n$ represent here?

Author response:
**Response to Question 1:** Here $n$ reprsents the dimensionality of the parameter $\nu$ used to parametrize the reward function.

Reviewer comment (Writing):
In the paper, the authors used $H_l$ to represent the horizon of the lower-level trajectory, and $H_u$ to represent the horizon of the lower-level trajectory. Why are there two different horizons? Are trajectories used for the upper level resampled?

Author response:
**Response to Question 2:** To be more general (in-terms of the analysis) we used different horizon length for the upper and inner level objectives, however the results hold true when H_u = H_l = H.

Reviewer comment (Writing):
On page 7, in assumption 4, what do $\mu, \hat{l}, \hat{\mu}$ represent?

Author response:
**Response to Question 3** Here, $\mu$ represents the PL (Polyak- Lojasiewicz) constant for the value function $V_s(\nu, \theta)$ with $\mu > 0$. Then,  $\hat{l}, \hat{\mu}$ represents the bound on the eigen values of the Hessian of the value function (common for various standard policy parametrizations) which we later discuss in Appendix Section I (page 41).

Reviewer comment (Theory):
In Lemma 1, does eq 21 hold for any choice of convex function $f$ or a specific $f$? Also, should $L_{\pi}$ be $L_2$?

Author response:
**Response to Question 4** Equation 21 holds for the standard choice of convex functions $f$ used in f-divergence [1] and the specific choice of functions results in the type of divergence we want to study (for ex: Total variation, Hellinger, etc.). Thanks for pointing the typo, yes $L_{\pi} = L_2$, updated in the draft.

Reviewer comment (Writing):
Suggestion: Through Lemma 2 and Thm 1, there are too many constant symbols such as $L_1, L_5, L_6$. It might be a good idea to combine some of them together in the main text to make the statement cleaner.

Author response:
**Response to Suggestion 1** Thanks for the very helpful suggestion and we agree that using too many coefficients can confuse the reader and will combine some of them with unified notation for easier read in the updated draft.

Reviewer comment (Experiments):
Empirical evaluations could have been more thorough. In Figure 2, only three domains were considered (2 manipulation / 1 locomotion), which is much smaller than the ones in the baseline papers. For example, in the PEBBLE/SURF papers, they conducted experiments in 9 domains (6 manipulation / 3 locomotion). I am curious if A-PARL still outperforms the baselines in other domains.

Author response:
**Response to Weakness 1** : We agree that it is possible to conduct more experiments on the remaining baselines. However, for this work, our primary focus is to provide a unified novel theoretical framework (A-PARL) and a rigorous theoritical analysis of the policy alignment problem, which is missing from existing literature (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). Hence, we performed experiments on three domains (with different level of hardness) as proof of concept of our algorithm and theoretical analysis.

***New Experiment Results*:** However, we remark that our performance is transferable to other environments as well. As mentioned by the reviewers, we added additional hard environments like Cheetah (DMC) and compared the performance of A-PARL with baselines [in Figure 6 in Appendix K (marked in blue)](https://openreview.net/pdf?id=ByR3NdDSZB). We observe that A-PARL outperforms existing baselines in both environments. 



**Note**: Due to time constraints, we have only added Cheetah (Locomotion, DMSuite) for now, but we are running further experiments on more hard environments (like visual-Walker, Sweep Into) and will try our best to add them in the final revised draft.

Reviewer comment (Reproducibility):
Some part of the algorithm is unclear. For example, it seems that gradient/Jacobian/Hessian estimation (Eq 15-17) requires 'on-policy' samples for their computations. Still, I was first assuming the lower-level policy optimization is done using an off-policy RL algorithm, as in the previous work (e.g. PEBBLE). Then, is my understanding correct that upper-level optimization is done using on-policy samples while the lower-level optimization is done in an off-policy way? Does lower-level optimization also use on-policy updates (Algorithm 1 line 5 seems to perform on-policy updates)? If so, I am curious how the on-policy algorithm could have shown better sample efficiency than the off-policy baselines.

Author response:
**Response to Weakness 2** This is a good point. We note that in our bilevel formulation,  the inner level is the standard RL policy optimization problem and can be solved using any standard RL algorithm either on-policy or off-policy. For the sake of analysis and establishing a connection to the most basic setting (as in RLHF (Christiano et. al 2021)), our policy learning update in Algorithm 1 is an on-policy. For the experiments and to have fair comparisons with Pebble and Surf, we use the same backbone (SAC which is off policy) with similar configurations and hyperparameters so that the results are comparable. 

We remark that irrespective of the algorithm we use to solve the inner-level RL problem, a key innovation in our approach lies in incorporating the gradient $\nabla_{\nu} \theta^*(\nu)$ (equation 12) in the outer objective (equation 8, 14), which helps to learn the true optimal reward and close the gap in the existing literature.

Reviewer comment (Experiments):
Lack of ablation studies. For example, to see the efficiency of A-PARL more properly, it would be great to see the results using varying the number of feedbacks. In the current form of the paper, it is even unclear how many feedbacks were used or how frequently the feedback was collected in the experiments.

Author response:
**Response to Weakness 3:** In the current setting, we compare with the exact number of default feedbacks: *Walker (max_feedback = $100$), Door Open (max_feedback = $1000$) and Button Press (max_feedback = $20000$) as typically used in (RLHF, Pebble, SURF etc.)* for the three environments mentioned. 

***New Ablation Study:*** As requested by the reviewer, we did additional experiments by varying the number of feedbacks as shown [in  Figure 7 in Appendix K.1 in the updated draft](https://openreview.net/pdf?id=ByR3NdDSZB). It shows that more human feedback results in better performance as expected. We will add a much more detailed version of this ablation study with respect to human feedback in the final version of the paper.

Reviewer comment (Theory):
Another concern is the scalability of the algorithm. Given that the method requires Hessians, it seems the complexity of the method is quadratic w.r.t. the number of parameters of the policy network. The method may not be applicable to the setting where the network size is large (e.g. millions of parameters).

Author response:
**Response to Weakness 4** : This is a very good point, thanks for raising this concern. We agree that our algorithm requires second-order estimates and would suffer in very large scale settings. But since our formulation is a first step towards rigorously formulating the policy alignment problem, we stick to the most fundamental and basic algorithmic setting to emphasize on the formulation and unified algorithm. However, we want to highlight that with the recent developments of scalable libraries such as Torchopt (Hypergrad), it is comparatively easier to estimate these terms approximately. Furthermore, scaling our approach to large scale is a valid scope of future research on which we are currently working on as a follow up to this work.

Reviewer comment (Theory):
Though it was interesting to see the corrected gradient updates for the upper-level optimization, it was hard to interpret the terms directly. Could you provide some intuitive interpretation of the additional terms in the upper-level gradient ($\tilde M^K(\nu_t)$)?

Author response:
**Response to Question 1** : Thank you for raising this comment. To understand the inutition behind the additional term in the upper-level gradient $U_{\nu}(\tau)\cdot \sum_{t =0}^{H_u-1} [\widetilde{M}^K(\nu_t) \nabla_{\theta} f_h(\theta^K(\nu_t))]$ arising in Equation 8, 14, let's take a step back to the term $\nabla_{\nu} \log \pi_{\theta^*(\nu)}(a_h|s_h)$ in equation 12 which eventually results in this additional term $\tilde{M^K}(\nu_{t})$ (implicit gradient). 

Note that the term $\nabla_{\nu} \log \pi_{\theta^*(\nu)}(a_h|s_h)$ is extremely important and gives a direct connection between the reward parameter $\nu$ and the optimal policy parameter $\theta^*(\nu)$. Intuitively, it provides information about the performance of the optimal agent policy under the specified reward. Utilization of this connection was missing from the existing literature. Thus, with this additional term, we perform a new proposed update in the outer objective which respects the inner-agent's performance under that specified reward in-addition to maximizing the likelihood of human preference (for RLHF application), which is the unique aspect of our formulation.

Reviewer comment (Novelty):
Without the $U_\nu(\tau) \sum \tilde M^K(\nu_t) \nabla_\theta f_h(\theta^K(\nu_t))$ term in Eq (14), does A-PARL reduce to the existing method or not?

Author response:
**Response to Question2** : You are correct, if we ignore the first term $U_{\nu}(\tau)\cdot \sum_{t =0}^{H_u-1} [\widetilde{M}^K(\nu_t) \nabla_{\theta} f_h(\theta^K(\nu_t))]$ inside the expectation in equation 14, A-PARL will reduce to standard RLHF frameworks (Christiano et. al 2017, Lee et. al 2021, Park et. al 2022 etc.) and thus our formulation generalizes the existing approaches and captures a unique missing aspect of existing formulations in the literature.

Reviewer comment (Reproducibility):
What values of K and N were used in Algorithm 1 for the experiments?

Author response:
Thank you for this comment. For the experiments we choose the default settings as used in (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). The exact values are for Walker (N = $100$), Door Open (N = $1000$) and Button Press (N = $20,000$)  for the human feedback. As requested by some other reviewers, we have also varied the feedback count and performed additional experiments, results are updated [in Appendix K (marked in blue)](https://openreview.net/pdf?id=ByR3NdDSZB). 

Regarding the value of $K$, as you correctly pointed out,  it represents the number of gradient updates for the inner policy optimization loop i.e., how much we optimize the policy for the given reward. Interestingly, we are able to derive a precise value of $K = t$ ($t$ : outer iteration count) from our analysis (equations 58, 59 section F in Appendix) and use the same in experiments. As pointed out by the reviewer, it's also very intuitive and represents that as the reward becomes more and more closer towards the optimal, we increase the value of $K$ i.e more accurately solve the inner problem. We remark that selecting any other value of $K$ would result in a tradeoff in terms of optimality of the inner level problem. 


Thanks for the insightful questions and we will add the discussion to our revised draft.

Reviewer comment (Experiments):
Could you elaborate on how these parameters were tuned and the impact of selecting different values?

Author response:
Thank you for this comment. For the experiments we choose the default settings as used in (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). The exact values are for Walker (N = $100$), Door Open (N = $1000$) and Button Press (N = $20,000$)  for the human feedback. As requested by some other reviewers, we have also varied the feedback count and performed additional experiments, results are updated [in Appendix K (marked in blue)](https://openreview.net/pdf?id=ByR3NdDSZB). 

Regarding the value of $K$, as you correctly pointed out,  it represents the number of gradient updates for the inner policy optimization loop i.e., how much we optimize the policy for the given reward. Interestingly, we are able to derive a precise value of $K = t$ ($t$ : outer iteration count) from our analysis (equations 58, 59 section F in Appendix) and use the same in experiments. As pointed out by the reviewer, it's also very intuitive and represents that as the reward becomes more and more closer towards the optimal, we increase the value of $K$ i.e more accurately solve the inner problem. We remark that selecting any other value of $K$ would result in a tradeoff in terms of optimality of the inner level problem. 


Thanks for the insightful questions and we will add the discussion to our revised draft.

Reviewer comment (Experiments):
Specifically, how does a larger K value, which implies more accurate solving of the lower problem but at the cost of additional environment trajectories, affect the overall outcomes? What trade-offs are observed with varying K values?

Author response:
Thank you for this comment. For the experiments we choose the default settings as used in (RLHF (Christiano et. al 2017), Pebble (Lee et. al 2021), SURF (Park et. al 2022) etc.). The exact values are for Walker (N = $100$), Door Open (N = $1000$) and Button Press (N = $20,000$)  for the human feedback. As requested by some other reviewers, we have also varied the feedback count and performed additional experiments, results are updated [in Appendix K (marked in blue)](https://openreview.net/pdf?id=ByR3NdDSZB). 

Regarding the value of $K$, as you correctly pointed out,  it represents the number of gradient updates for the inner policy optimization loop i.e., how much we optimize the policy for the given reward. Interestingly, we are able to derive a precise value of $K = t$ ($t$ : outer iteration count) from our analysis (equations 58, 59 section F in Appendix) and use the same in experiments. As pointed out by the reviewer, it's also very intuitive and represents that as the reward becomes more and more closer towards the optimal, we increase the value of $K$ i.e more accurately solve the inner problem. We remark that selecting any other value of $K$ would result in a tradeoff in terms of optimality of the inner level problem. 


Thanks for the insightful questions and we will add the discussion to our revised draft.