Paper: Provable Offline Preference-Based Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The most substantial weakness of the paper is that the related work focuses on other, theoretical work, but not practical. For example, the idea of return-based utility approximation is not novel (e.g., Preference-Based Policy Learning, Akrour, 2011 - see Wirth 2017 for more examples). This is relevant because, first, it allows the reader to determine which algorithms fit under the proposed framework, and second, it clarifies the differences between the proposed algorithm and other works.

Author response:
We mainly discuss theoretical works in the related works because our paper also focuses on the theory of offline RLHF. That said, **our algorithm is still quite different from the practical algorithms presented in [1]**. Although they also use MLE to estimate the reward model, they typically run RL algorithms **with respect to the learned MLE rewards** (i.e., $\hat{r}$ in Algorithm 1) without any pessimism. The lower bound in [Theorem 3.9, 2] has shown that this kind of algorithm can fail in the worst cases. 

In contrast, we **use pessimism** to circumvent such failure. More specifically, we **construct a confidence set of the reward model and then evaluate each policy with the most pessimistic reward model inside the confidence set**. This is the key difference between our algorithm and existing commonly used algorithms in [1], and this modification is crucial to overcome their limit.

Reviewer comment (Novelty):
Considering the algorithm is mentioned as one of the main contributions, but likelihood-based estimation of the return utility is already known, the novelty of the algorithm is a bit limited.

Author response:
We mainly discuss theoretical works in the related works because our paper also focuses on the theory of offline RLHF. That said, **our algorithm is still quite different from the practical algorithms presented in [1]**. Although they also use MLE to estimate the reward model, they typically run RL algorithms **with respect to the learned MLE rewards** (i.e., $\hat{r}$ in Algorithm 1) without any pessimism. The lower bound in [Theorem 3.9, 2] has shown that this kind of algorithm can fail in the worst cases. 

In contrast, we **use pessimism** to circumvent such failure. More specifically, we **construct a confidence set of the reward model and then evaluate each policy with the most pessimistic reward model inside the confidence set**. This is the key difference between our algorithm and existing commonly used algorithms in [1], and this modification is crucial to overcome their limit.

Reviewer comment (Reproducibility):
Some relevant information is left to the appendix – mostly the feasible implementation and the comparison to Zu 2023. Especially some details concerning the feasible implementation are relevant for practitioners.

Author response:
Thanks for pointing it out! We will move the key points in Appendix to the main paper in the final version.

Reviewer comment (Writing):
Remark 3 is not explained.

Author response:
We will clarify it as follows. In line 4 of Algorithm 1, we need to compute $\mathbb{E} _ {\tau \sim \mu _ {ref}}[r (\tau)]$, which might be computationally inefficient since the space of all trajectories is large. In this case, we can sample $N_0$ trajectories $(\tau ^ i) _ {i = 1} ^ {N _ 0}$ from $\mu _ {ref}$ and then simply use $ \frac{1}{N _ 0}\sum _ {i=1} ^ {N _ 0} r(\tau ^ i)$ as a surrogate of $\mathbb{E} _ {\tau \sim \mu _ {ref}}[r (\tau)]$. From Azuma-Hoeffding inequality, this will only incur an error scaling with $\frac{1}{N _ 0}$. Therefore as long as we choose $N _ 0 = O(1 / \epsilon ^2)$, this error can be neglected while line 4 becomes more computationally efficient.

Reviewer comment (Theory):
The statement "cannot be relaxed to the per-step concentrability without additional assumptions, such as on the reward structure" is not obvious. It is unclear why it follows from the Theorems that state-action rewards do not enable this.

Author response:
Theorem 2 shows that there exists **a set of MDPs whose rewards are defined per step (i.e., $r(\tau) = \sum _ {h=1} ^ H r _ h (s _ h, a _ h)$)** and **an offline dataset which satisfies the per-step concentrability** such that  **no offline algorithm** is able to learn a near-optimal policy. This implies that with only per-step concentrability, we are not able to find a high-quality policy, even if we assume the rewards of the MDPs are defined per step. In contrast, with our trajectory-wise concentrability coefficient in Definition 2, we are able to learn a near-optimal policy for any MDPs, no matter whether the reward is defined over trajectories or defined per step. We will add this explanation in the final version.

Reviewer comment (Writing):
Missing reference for the claim: "..per-step concentrability coefficient Cst commonly used in the general offline RL literature."

Author response:
We are sorry about the missing references. We list the references here and will add them in the main paper:
Per-step concentrability coefficient: [3-7], Optimal policy: [8]

Reviewer comment (Writing):
Definition 1 is a bit hard to understand, because g is never defined.

Author response:
We are sorry about the unclearness of the definition of $g$. $g$ is a function mapping from $(\mathcal{T} \times \mathcal{T})$ (recall that $\mathcal{T} = (\mathcal{S} \times \mathcal{A}) ^ H$ is the set of all possible trajectories) to $\mathbb{R}^2$ and thus $g(\cdot|\tau _ 1, \tau _ 2)$ is a two-dimensional vector. It doesn’t need to be a distribution, i.e., $g(\cdot|\tau _ 1, \tau _ 2)$ is not necessarily positive and $\Vert g(\cdot|\tau _ 1, \tau _ 2) \Vert _1$ doesn’t need to be 1. Similar functions are elaborated in the definitions in Appendix E. We will revise this in the final version.

Reviewer comment (Writing):
The slack parameter in Sec. 4.1 has the wrong symbol.

Author response:
Thanks for pointing them out! We will correct them in the final version.

Reviewer comment (Writing):
Typo in Algorithm 2: "Distributionally robust plnanning."

Author response:
Thanks for pointing them out! We will correct them in the final version.

Reviewer comment (Experiments):
This paper is fully focusing on theory. It would be nice to also have some empirical results.

Author response:
We are currently considering adding an empirical case study in future work. That said, **we indeed have some discussions about the practical implementation of our algorithm in Appendix B**. In summary, we use the Lagrangian multiplier to convert our algorithm into a maximin optimization problem and demonstrate how to compute the gradient of the objective function so that we can use gradient ascent-descent to solve it. Given that gradient descent-ascent has shown superior performance in standard offline RL problems [1], we believe that the above approach can also work well in practice.

Reviewer comment (Theory):
It seems to me that the link function in Eq.1 is known? Would it work if the link function is unknown?

Author response:
Yes, we assume the link function is known. When the link function is unknown, we may consider a link function class $\mathcal{F}$. Then we can generalize our analysis by implementing MLE over **the joint function class** $(\mathcal{F}, \mathcal{G} _ r)$, i.e., $(\hat {r}, \hat{\Phi}) = \arg\max _ {r \in \mathcal{G} _ r, \Phi\in\mathcal{F}} \sum _ {n=1}^N \log P _ {r,\Phi}(o=o^{n}\mid\tau^{n,1},\tau^{n,0})$ where $P_{r,\Phi}(o= 1\mid\tau^1, \tau ^ 0) := \Phi( r(\tau ^ 1) - r (\tau ^ 0))$. We leave its formal analysis to our future work.

Reviewer comment (Theory):
For unknown transition kernels, would the result extend to the case where the transition kernels' data are sampled separately from the preference data?

Author response:
Yes, the analysis still holds. We reuse the dataset in order to reduce sample complexity, but you can definitely collect a different dataset for transition learning as long as the dataset has a bounded single-policy concentrability coefficient (Definition 3).

Reviewer comment (Reproducibility):
There are no implementation details about the algorithms. These algorithms are theoretically sound but there is no intuition for the reader on how to implement them or construct a practical algorithm.

Author response:
We would like to clarify that this is mainly a **theoretical** work that strives to find sample efficient offline RLHF algorithms with **provable** guarantees. That said, we would like to point out that we indeed **have discussions about the practical implementation of our algorithm in Remark 1 in the main text and Appendix B**. In summary, we use the **Lagrangian multiplier** to convert our algorithm into a **maximin optimization problem** and demonstrate how to compute the gradient of the objective function so that we can use **gradient ascent-descent** to solve it. Given this type of gradient descent-ascent has shown superior performance in standard offline RL problems [1], we believe that the above approach can also work well in practice in our preference-based RL setting. We will emphasize this part more in the final version.

Reviewer comment (Experiments):
There are no experiments either. So there is no way to check how this method empirically scales.

Author response:
We would like to mention that **there are many practical and impactful works about offline RL in top machine learning conferences without any experiments**. First, for example, in the literature on offline model-free RL, [2] proposed the first provably efficient algorithm with general function approximation. While this paper does not have any experiments, it is later implemented in another follow-up paper [3]. It is reported that the proposed algorithm achieves SOTA and beats many offline RL baselines, such as CQL, COMBO ICL, and TD3+BC. 

Similarly, in the literature on model-based RL, [4] proposed the first probably efficient algorithm with general function approximation. While this paper does not have any experiments again, later, it is implemented in another follow-up paper [1], and it is reported that it achieves SOTA and beats many baselines, such as again MOReL, CQL, IQL, and TD3+BC. **In this way, even with no experiments, theoretically backed algorithms have been empirically impactful.**

Reviewer comment (Writing):
Several terms in the paper are not well explained. For example, $\epsilon$-bracketing: what do you mean by g(.|tau1, tau2)? Is g a probability measure, or a preference?

Author response:
We are sorry about the unclearness of the definition of $g$. $g$ is a function mapping from $(\mathcal{T} \times \mathcal{T})$ (recall that $\mathcal{T} = (\mathcal{S} \times \mathcal{A}) ^ H$ is the set of all possible trajectories) to $\mathbb{R}^2$ and thus $g(\cdot|\tau _ 1, \tau _ 2)$ is a two-dimensional vector. It doesn’t need to be a distribution, i.e., $g(\cdot|\tau _ 1, \tau _ 2)$ is not necessarily positive and $\Vert g(\cdot|\tau _ 1, \tau _ 2) \Vert _1$ doesn’t need to be 1. Similar functions are elaborated in the definitions in Appendix E. We will revise this in the final version.

Reviewer comment (Theory):
Proposition 1: For epsilon being arbitrarily small, and the bounds B and R being arbitrarily large, log N_G_r can be arbitrarily large making the sample bounds weak.

Author response:
**We would like to state that our bounds are not weak compared to standard RL literature**. First, as you notice, the log bracket number increases **logarithmically** as the accuracy $\epsilon$ decreases. However, as we will later see in Theorem 1, we just set $\epsilon=1/N$, which results in an additional $\log(N)$ term. This term is negligible compared to the leading term $1/N$.  Second, the log dependence on $B$ and $R$ is very standard in the literature on statistical machine learning in general, as seen in Section 4,5 in [5] and Definition 8.1 in a standard RL textbook [6].

Reviewer comment (Theory):
In Section 4.1, it is mentioned the distance between r and r* is computed as total variation distance or l1 norm. Aren’t r and r* scalars?

Author response:
No, $r$ and $r ^ * $ are reward **functions**. They map a trajectory $\tau$ to a scalar. Thus, here $r$ ($r^*$) can be viewed as a vector whose $\tau$-th entry corresponds to the value of $r(\tau)$ ($r^*(\tau)$) for each $\tau \in \mathcal{T}$.

Reviewer comment (Theory):
Does C_tr (per trajectory concentration coefficient) not depend on \mu_1? If so, why?

Author response:
No, it doesn’t depend on $\mu _ 1$ because when we reduce  $C _ r$ to $C _ {tr}$, we take $\mu _ {ref} = \mu _ 1$ in $C _ r(\mathcal{G} _ r, \pi _ {tar}, \mu _ {ref})$ and then $\mu _ 1$ is canceled out  by $\mu _ {ref}$.

Reviewer comment (Theory):
How does C_r(G_r, \pi_tar, \u_1) reduce to sqrt(Ctr)?

Author response:
Note that in this reduction we take $\mu _ {ref}$ to be $\mu _ 1$. From Cauchy-Schwartz inequality, we have 
$$ \sqrt{  \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2 \big] } \geq  \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)|\big].$$

This implies that
$$C^2_r(\mathcal{G} _ r, \pi _ {tar}, \mu _ 1) \leq \sup _ {r} \frac{ \mathbb{E} _ {\tau ^ 0 \sim \pi _ {tar}, \tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2\big]}{ \mathbb{E} _ {\tau ^ 0 \sim \mu _ 0,\tau ^ 1 \sim \mu _ 1}\big[|r ^ *(\tau ^ 0) - r ^ *(\tau ^ 1) - r(\tau ^ 0) + r(\tau ^ 1)| ^ 2\big]}$$

Since $\tau _ 0$ and $\tau _ 1$ are independent, we have
$$C ^ 2 _ r(\mathcal{G} _ r, \pi _ {tar}, \mu _ 1) \leq \sup _ {r} \max _ {\tau _ 0, \tau _ 1 }\frac{ d ^ {\pi _ {tar}}(\tau _ 0) \mu _ 1(\tau _ 1)}{ \mu _ 0(\tau _ 0) \mu _ 1(\tau _ 1)} =\max _ {\tau} \frac{d ^ {\pi _ {tar}}(\tau)}{ \mu _ 0(\tau)}.$$

Reviewer comment (Writing):
It looks like pi, mu and d are used interchangeably. I recommend correcting this.

Author response:
We are sorry for the unclearness. For $\pi$, we use $\tau \sim \pi$ to denote the distribution of $\tau$ when executing $\pi$ in the MDP. When $\tau$ follows a more general distribution $\mu$, i.e., when the distribution may not be induced by a policy $\pi$, we use $ \tau \sim \mu$. For $d$, we only use $d ^ {\pi} (\tau)$ to denote the probability of $\tau$ under policy $\pi$. We will further clarify these in the final version.