Paper: Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR-2024)

Reviewer comment (Theory):
(Worst-case complexity) All of the reported results on trajectory-preference feedback come with the huge caveat that the rate may be exponential in the worst-case, as $r_{\text{max}}$ can be as large as $H$.

Author response:
**Worst-case Sample Complexity (in Weaknesses)**

We admit that $\kappa$ will scale with $\exp(H)$ in the worst case. This quantity occurs in **all the existing theoretical works** on PbRL [1-3] and seems inevitable for MLE estimators. That said, in the **sparse-reward setting** which is very common in practice, $r _ {max}$ is typically $O(1)$ and thus $\kappa$ will also be negligible. In addition, to circumvent the scaling of $\kappa$, we also consider **action-based setting** in the paper where we only scale with $\exp(B _ {adv})$, which is guaranteed to be not greater than $r _ {max}$ and often much smaller in practice [4].

Reviewer comment (Evaluation):
The claim of improved results with respect to prior works (e.g., Pacchiano et al., 2021) is a little overstated, as previous works target regret minimization instead of sample complexity, making the comparison arguably spurious.

Author response:
**General comments (in Weaknesses)**

We agree that [1] focuses on regret minimization while our goal is to improve sample complexities in terms of PAC-learnability. We will make it more clear in our final version.

Reviewer comment (Theory):
The paper’s Preliminaries section states that "it is necessary to make structural assumptions about the reward," but it is unclear whether this means the problem is not learnable without the linear reward assumption or just that *some* structural assumption is needed; providing a lower bound would clarify this.

Author response:
**Requirement of Linear Reward Parametrization**

We agree that “it is necessary to make structural assumptions about the reward” is a bit informal. We were not trying to make a formal statement here. We will change the wording into a “we make structural assumptions as in other works [1,3]”.  

In a nutshell, here originally, we just wanted to convey that **we need some structural assumptions for generalization, and we chose linear models as these structural assumptions**. For example, in the literature on standard supervised learning, the former statement (necessary to assume structure) is often formalized as VC dimension needs to be finite to ensure learnability.  Yeah, but to say formally in our context, we need to establish certain lower bounds as you mention. Hence, we will just remove it.

Reviewer comment (Theory):
The necessity of assuming an underlying reward function for PbRL is not addressed; it is worth questioning whether the problem can instead be formulated as directly maximizing human feedback.

Author response:
**Direct Maximization**

This is a good point. Direct maximization of human preference is possible and has been studied theoretically as the well-known dueling bandits problem [8-10]. However, in practice, in PbRL, people usually first train a reward model and then use the reward model for downstream learning [11-15]. We think this is because after learning a reward model, people can use the reward model to score an arbitrary number of trajectories or even reuse the model in other related tasks, thus reducing a large amount of human cost. Therefore, due to the wide application of such a framework, we also assume an underlying reward model in this paper.

Reviewer comment (Novelty):
The action-based preference feedback appears related to inverse RL, where an expert is queried for the optimal action given a state rather than for a preference between actions; it would be valuable to relate these findings to inverse RL literature and address whether the problem is easier or harder than inverse RL.

Author response:
**Relation to Inverse RL**

While there are some similarities as you mention, we think **PbRL is still very different from inverse RL**. Most importantly, in inverse RL, the actions are sampled from an expert policy directly and thus we only need to learn a reward function that matches such expert behaviors. However, in PbRL, the pair of actions is not sampled from an expert and can possibly be undesirable.

Reviewer comment (Novelty):
The novelty of the sample complexity analysis compared to prior work is not explicitly discussed; the authors should clarify if there are any novel techniques they would like to highlight.

Author response:
**Technical Novelty** 

The novelty of our techniques mainly lies in **the design and analysis of the reward-agnostic trajectory collection procedure (line 4-7 in Algorithm 1)**.  We will try to emphasize more in the final version. More specifically, our analysis deals with the following challenges that are not solved in the literature:

1. **First, almost all the reward-free exploration literature is focused on learning the transition dynamics of the MDPs. Our problem is completely different because our main task is to learn the reward function from human preferences**. Consequently, our iterative collection procedure (line 4-7 in Algorithm 1) also differs from the reward-free exploration literature. For example, [17] is the closest work to ours and studies the reward-free exploration in linear MDPs. They utilize a variant of UCB-LSVI, which is quite different from our collection procedure. Correspondingly, we can not directly use the existing analysis either and need to come up with new proofs. 

2. **Second, our framework is also different from the existing online PbRL [6-7]**. In their algorithms, the reward-learning process and exploration phase are not decoupled and the analysis heavily relies on the utilization of **optimism**. In contrast, our algorithm **does not use optimism**, and thus their analysis techniques also do not apply here.

3. **Third, our analysis of  REGIME-lin differs from the one in the literature on linear MDPs**. This is because while the style of our algorithm is akin to **multi-policy evaluation** in linear MDPs, the current literature of linear MDPs mainly focuses on **Q-learning-style algorithms for policy optimization** [16-17]. 

More specifically, in REGIME-lin, we want to **evaluate all the policies in a policy class under a certain reward function**. This brings a new challenge that does not occur in [16-17]: our learning difficulty will scale with the size of the policy class $\Pi$. This forces us to find and analyze **an appropriate policy class that is able to approximate the optimal policy while retaining a relatively small size**. In addition, a key component of our analysis is the **new concentration inequality** in Lemma 13, which also differs from the existing concentration lemma in [16-17] and may be of independent interest.

Reviewer comment (Presentation):
I think that the presentation can be improved. I know that space limitations are tight, however, removing margins around equations and sections will make the paper much harder to read.

Author response:
Thanks for the suggestions! We will add more margins around equations in the final version. In addition, as far as we know, the existing theoretical online PbRL works [1-2] focus on regret minimization and only an offline PbRL paper [3] studies PAC algorithm. The PAC sample complexity of [1-2] is derived from the corresponding regret bound. We will try to add the table in the final version.

Reviewer comment (Presentation):
A table which compares this paper's results with related work would also help the reader to place this work into the existing literature. This would be particularly helpful since much of the related work does not study PAC algorithms, but only regret minimization, and it is difficult to figure out the current state-of-the-art and open questions in this area from just reading the paper.

Author response:
Thanks for the suggestions! We will add more margins around equations in the final version. In addition, as far as we know, the existing theoretical online PbRL works [1-2] focus on regret minimization and only an offline PbRL paper [3] studies PAC algorithm. The PAC sample complexity of [1-2] is derived from the corresponding regret bound. We will try to add the table in the final version.

Reviewer comment (Evaluation):
How computationally expensive is step 1 (e.g., line 5 in Algorithm 1, line 7 in Algorithm 2, line 5 in Algorithm 3)?

Author response:
The main computational obstacle lies in line 5 of Algorithm 1. **To implement the algorithm in practice, gradient ascent can be applied here to solve the optimization problem**. More specifically, assume that $\pi ^ 0$ and $\pi ^ 1$ are parameterized by $\nu ^ 0$ and $\nu ^ 1$ (for example, they might be parametrized by a neural network). Then the gradient of the objective $f(\nu ^ 0,\nu ^ 1) := \Vert \widehat{\phi}(\pi ^ 0) - \widehat{\phi}(\pi ^ 1) \Vert _ {{\widehat{\Sigma}} ^ {-1} _ n}$ with respect to $\nu ^ 0$ (gradient of $\nu ^ 1$ can be similarly computed) can be computed as:

$$\frac{\partial f(\nu ^ 0 , \nu ^ 1)}{\partial \nu ^ 0} = 2 \frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0} \cdot {\widehat{\Sigma}} ^ {-1} _ n \cdot (\widehat{\phi}(\pi ^ 0) - \widehat{\phi}(\pi ^ 1)).$$

Here $\widehat{\phi} (\pi ^ 0)$ and $\widehat{\phi}(\pi ^ 1)$ can be estimated efficiently by simulating $\pi ^0 $ and $\pi ^ 1$ in $\widehat{P}$. For $\frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0}$, we show in Section 4 that each coordinate of $\widehat{\phi}(\pi^0)$ is indeed a value function with respect to a certain reward function under $\widehat{P}$. Therefore we can apply the techniques in policy gradient literature such as REINFORCE to estimate $\frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0}$ efficiently. We will incorporate this explanation in our paper.

Reviewer comment (Evaluation):
Intuitively, one would think that adaptive trajectory selection which queries human feedback after every selection would be advantageous compared to first selecting all trajectories without observing intermediate human feedback. In other words, you would expect *adaptive* experimental design to perform better than *offline* experimental design (even when it comes to PAC learning). Your results however suggest that "offline" trajectory selection does not hurt performance. Can you provide any reasons or intuition for this?

Author response:
We agree the phenomenon you mentioned has indeed been shown in the literature of standard reward-free exploration RL [3-4], where the sample complexity of the reward-free algorithms is only slightly worse than reward-aware algorithms [5-6]. **However, in PbML, we won’t necessarily see this phenomenon**. 


1. **First, reward learning in PbRL is much more difficult than reward learning in standard RL** in the sense that rewards in standard RL could be just learned from regression over $r$ onto a pair of $(s,a)$ at single-time point; but, rewards in PbRL need to be learned from exploratory data over **the whole trajectories** (not a pair of $(s,a)$).  Due to the difficulty of reward learning, it is unclear whether a reward-aware adaptive data collection process leads to better sample complexity in PbRL **because learned awards could be very inaccurate before exploring the whole trajectory very well**. That’s why our algorithm has a better sample complexity than [1], even in terms of not only the sample complexity of human feedback $N_{hum}$ but also the sample complexity of transitions $N_{tra}$, as we mention at the end of Section 3. 


2. **Second, compared to standard RL, we need to make a distinction between sample complexity in terms of human feedback $N_{hum}$ and the sample complexity of trajectories $N_{tra}$**. If we implement adaptive experimental designs depending on learned rewards, this means we need to query human feedback (to accurately learn rewards) while we explore over trajectories. This results in larger sample complexity in terms of $N_{hum}$ while we might be able to successfully decrease $N_{tra}$, which may be undesirable because in practice human feedback is often more expensive than trajectories.

Reviewer comment (Theory):
Hereto related, do you think that your approach of seperating trajectory selection and human feedback stages is applicable/useful for regret minimization as well? (Beyond trivially reducing your $(\varepsilon, \delta)$-PAC guarantees to regret bounds via some explore-then-commit strategy).

Author response:
We think this generalization is possible. For example, we can collect trajectories for reward learning and dynamics learning separately in different frequencies. When the dynamics is more complicated, we can query human feedback and update the reward model **less often** to wait for the learning of the dynamics. This can potentially reduce the human feedback complexity.

Reviewer comment (Experiments):
Since the motivation of this work derives from prominent practical problems, the results seem to be limited to guide the use of PbRL of RLHF in real-world problems. Although the results show a reduction of the number of human preference data compared to previous work by collecting a exploratory dataset in advance, it is doubtful whether the algorithmic designs are computational efficient to be implemented to improve the RLHF. Some experiment results (even on toy examples) are appreciated to show the effectiveness of decoupling the human labeling process and online exploration stage. The significance of the results cannot be fully verified without the proof of practical usage of these algorithmic designs, because the proposed algorithms may be computational inefficient, thus explaining the theoretical advantage of such algorithms rarely helps to explain the advantage of current PbRL used in practice.

Author response:
Thank you for the valuable feedback! Here are our responses.

- **Practical Insights and Computational Efficiency**

1. We first want to clarify that this work mainly focuses on the **statistical complexity** of PbRL and attempts to explain the advantages of the current practical PbRL framework from the perspective of sample complexity. For practical insights, we agree that empirical experiments can further verify our claims. However, **we do not think theoretical analysis cannot provide insights to explain the advantages of current PbRL in practice**. Instead, we believe that by studying the theoretical properties of some simplified cases (such as the linear reward parametrization setting in this paper), we are able to dig out the mathematical explanation behind the phenomenon in practice, and make actual algorithms more sample efficient.  As a matter of fact, from the sample complexity comparison against existing works, our paper provides **the following insights into practical algorithm design**: 1) **decoupling the interaction with the environment and the collection of human feedback could lead to reduced sample complexity in theory** 2) if we want to use fewer human feedback, we need to **make the query dataset as diverse as possible such that it can cover the feature space**. 

We do not have empirical evidence, but we would like to point out that **a lot of important theoretical works also focus on theoretical analysis and lack empirical evaluation**, from standard RL [1-5] to PbRL [6-8]. Among them, the algorithms in [3-8] are not computationally efficient either. In particular, as far as we know, [6-7] are the only existing online PbRL algorithms with provable guarantees, and their algorithms are **even harder to implement than ours in practice** because the optimization problems in their algorithms have more constraints and are much more complicated.

2. **Second, for computational efficiency, the majority of computational cost lies in line 5 in Algorithm 1. To implement the algorithm in practice, gradient ascent can be applied here to solve the optimization problem**. More specifically, assume that $\pi ^ 0$ and $\pi ^ 1$ are parameterized by $\nu ^ 0$ and $\nu ^ 1$ (for example, they might be parametrized by a neural network). Then the gradient of the objective $f(\nu ^ 0,\nu ^ 1) := \Vert \widehat{\phi}(\pi ^ 0) - \widehat{\phi}(\pi ^ 1) \Vert _ {{\widehat{\Sigma}} ^ {-1} _ n}$ with respect to $\nu ^ 0$ (gradient of $\nu ^ 1$ can be similarly computed) can be computed as:

\begin{align*}
\frac{\partial f(\nu ^ 0 , \nu ^ 1)}{\partial \nu ^ 0} = 2 \frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0} \cdot {\widehat{\Sigma}} ^ {-1} _ n \cdot (\widehat{\phi}(\pi ^ 0) - \widehat{\phi}(\pi ^ 1))
\end{align*}

Here $\widehat{\phi} (\pi ^ 0)$ and $\widehat{\phi}(\pi ^ 1)$ can be estimated efficiently by simulating $\pi ^0 $ and $\pi ^ 1$ in $\widehat{P}$. For $\frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0}$, we show in Section 4 that each coordinate of $\widehat{\phi}(\pi^0)$ is indeed a value function with respect to a certain reward function under $\widehat{P}$. Therefore we can apply the techniques in policy gradient literature such as REINFORCE to estimate $\frac{\partial \widehat{\phi}(\pi ^ 0)}{\partial \nu ^ 0}$ efficiently. We will incorporate this explanation in our paper. 

3. **Third, although our analysis posits linear models, we can potentially extend our algorithms to the one with deep neural networks**. This is roughly done after learning representations with deep neural networks and treating it as a feature $\phi$.  This type of extension is commonly used in the literature of standard RL [9,10]. Since the focus in our paper is to establish theoretically provably efficient algorithms without any heuristics, we explain this possible extension in the section of future works.

Reviewer comment (Novelty):
The setting looks like a combination of reward-free exploration and online PbRL, which are both standard in reinforcement learning and extensively studied. Therefore, it is hard to evaluate the technical contributions as a theoretical work. The subroutine to collect exploratory data with linear reward functions (using whether reward-free exploration oracle or linear reward-free exploration subroutine) used for human preferences, the MLE estimation of the underlying reward model, and the planning of the optimal policy seem to be slight modification of standard algorithms in the literature, as long as the proving tools. I recommend to explicitly explain the technical contributions in the paper.

Author response:
- **Technical Contributions**

Most importantly, the novelty of our algorithm mainly lies in the **reward-agnostic trajectory collection** procedure (line 4-7 in Algorithm 1). For the other steps in the framework, we follow the common practice in empirical works and thus they are the same as the practical work. Our reward-agnostic trajectory collection procedure takes inspiration from the existing works in reward-free exploration RL and online PbRL. However, it is **not a simple combination of the existing algorithms**. More specifically, our analysis has the following challenges that are not solved in the literature. We will incorporate this explanation in the final version. 

1. **First, almost all the reward-free exploration literature is focused on learning the transition dynamics of the MDPs. Our problem is completely different because our main task is to learn the reward function from human preferences**. Consequently, our iterative collection procedure (line 4-7 in Algorithm 1) also differs from the reward-free exploration literature. For example, [2] is the closest work to ours and studies the reward-free exploration in linear MDPs. They utilize a variant of UCB-LSVI, which is quite different from our collection procedure. Correspondingly, we can not directly use the existing analysis either and need to come up with new proofs. 

2. **Second, our framework is also different from the existing online PbRL [6-7]**. In their algorithms, the reward-learning process and exploration phase are not decoupled and the analysis heavily relies on the utilization of **optimism**. In contrast, our algorithm **does not use optimism** and thus their analysis techniques also do not apply here. 

3. Third, our analysis of  REGIME-lin differs from the one in the literature on linear MDPs. This is because while the style of our algorithm is akin to **multi-policy evaluation** in linear MDPs, the current literature of linear MDPs mainly focuses on **Q-learning-style algorithms for policy optimization** [1,2]. 

More specifically, in REGIME-lin, we want to **evaluate all the policies in a policy class under a certain reward function**. This brings a new challenge that does not occur in [1,2]: our learning difficulty will scale with the size of the policy class $\Pi$. This forces us to find and analyze **an appropriate policy class that is able to approximate the optimal policy while retaining a relatively small size**. In addition, a key component of our analysis is the **new concentration inequality** in Lemma 13, which also differs from the existing concentration lemma in [1,2] and may be of independent interest.

Reviewer comment (Theory):
The previous work Pacchiano et al. studied the online PbRL with a linear reward function. It seems that the sample complexity of this in the reward-free exploratory stage is better than that of the previous work, which is not reward-free setting. What causes this gap between current work and previous work?

Author response:
- **Reasons of the Sample Complexity Improvement**

As pointed out in the paper, we believe this is because [6] **couples the reward learning process with dynamics learning**. Consequently, part of the human feedback is also utilized to learn the dynamics and thus wasted. In contrast, our algorithm **decouples these two learning processes** and human feedback is only utilized to learn the reward, leading to more efficient human feedback complexity.