Paper: Demonstration-Regularized RL (ICLR-2024)

Reviewer comment (Theory):
The lower bound in Theorem 2 turns unfortunately vacuous in the limit of $\gamma \rightarrow 0$.

Author response:
A fixed $\gamma$ enables us to establish a general lower bound by constraining the space of policies we aim to learn. Thus, the minimax lower bound with respect to the space of all policies is lower-bounded by the minimax lower bound with respect to the space of restricted policies. Therefore, choosing any fixed $\gamma$ (for example, $\gamma = 1/N$) allows us to deduce a valid and reasonable lower bound.

Reviewer comment (Theory):
It is unclear why the class of linear policies at the third line of Section 3.2 is considered to be not learnable. In fact, under this choice [1] proves in their Theorem 5 that it is possible to recover an $\epsilon$-suboptimal policy compared to the expert with behavioural cloning.

Author response:
It is important to emphasize that the behavior cloning defined in our paper aims not to find an $\varepsilon$-optimal policy but to reconstruct the initial behavioral policy in trajectory KL-distance. To make the algorithm implementable, defining a hypothesis class with policies that are differentiable or at least continuous with respect to the learnable parameters becomes necessary. In the setting of purely greedy policies, the mentioned linear class of policies lacks these properties, thus preventing the efficient minimization of the log-loss. We will add an additional discussion and a reference to [1].

Reviewer comment (Theory):
Corollary 3 requires the expert to be $\mathcal{O}(\epsilon)$ optimal but I would expect that, given the reward knowledge, it should be possible to prove a sample complexity bound without the assumption on the $\mathcal{O}(\epsilon)$ optimality of the expert. To see this think to the case of any BPI algorithm which requires no expert at all to learn an $\epsilon$-optimal policy.

Author response:
We acknowledge that the assumption of nearly optimal experts might be considered strong. If it is known a priori that the expert is far from optimal, a viable approach is to simply set $\lambda=0$ and employ a stopping rule from UCBVI-BPI (Ménard et al. (2021)).  

However, it's important to emphasize that the primary goal of this paper is to provide an analysis of an algorithm actively used in practice, where these assumptions are deemed reasonable.

Reviewer comment (Writing):
I think that Lemma 11 should be referred as the standard performance difference lemma.

Author response:
We agree that Lemma 11 is already known in literature, so we will add the corresponding reference.

Reviewer comment (Writing):
Just before Corollary 3, it is said that "UCBVI-Ent+ algorithm for regularized BPI. It is a modification of the algorithm UCBVI-Ent by Tiapkin et al. (2023) with improvement sample complexity". However, it is not explained which is the crucial difference between the two algorithms. In particular, also the settings are different because UCBVI-Ent+ uses reward information while UCBVI-Ent can be used only for maximum entropy exploration and not to solve Regularized MDPs but this difference is not explained in the main text.

Author response:
We add a separate comment (available to all reviewers) regarding the key differences between UCBVI-Ent and UCBVI-Ent+.

Reviewer comment (Theory):
Is it possible to prove a bound which does not require the assumption that the expert is $\epsilon$ optimal?

Author response:
We believe that it is possible (in the setup of RL with demonstrations) since a usual BPI algorithm allows us to provide an optimal policy without any additional data. However, the resulting algorithm using  data from a highly suboptimal expert may be significantly different from the proposed one. So we focus on a simple and implementable approach.

Reviewer comment (Theory):
Why the regularization is needed in the tabular case but not in the linear one? I am referring to Section 3.2.

Author response:
Regularization in the linear setting is unnecessary because we add constraints on the parameters of the class. Alternatively, one can introduce ridge regularization with an appropriate regularization parameter and make no direct restrictions on the regression coefficients. In fact, these two approaches are nearly equivalent.

Reviewer comment (Theory):
How can UCBVI-Ent+ achieve $\mathcal{O}(\epsilon^{-1})$ sample complexity according to Theorem 5 while UCBVI-Ent achieves a worst sample complexity of $\mathcal{O}(\epsilon^{-2})$?

Author response:
We have included a separate comment detailing the distinctions between UCBVI-Ent and UCBVI-Ent+. The critical feature that enables UCBVI-Ent+ to achieve $O(\varepsilon^{-1})$ sample complexity is the exploitation of the strong convexity of the regularizer. However, to implement this effectively, it is necessary to introduce additional time randomization to the policy played during the episodes. We believe that this is a novel aspect of our work on the algorithmic side.

Reviewer comment (Writing):
What are the definitions of $\pi^{t,(h)}$ and $\tilde{\pi}^t$ in Algorithm 3?

Author response:
The definition of $\pi^{t, (h’)}$ is given after sampling rules (9) in Appendix D.2: $\pi^{t, (h’)}_h$ is equal to an regularized-greedy optimistic policy given by optimistic planning for all steps $h \not = h’$ and $\pi^{t, (h’)}_h$ is a greedy with respect to a difference between upper and lower estimate for optimal Q-value for a step $h = h’$. In particular, this randomization is crucial to achieve $\mathcal{O(\varepsilon^{-1})}$ sample complexity. Regarding $\tilde{pi}$, it is a misprint and the final policy should be $\bar{\pi}^t$, thank you for showing it to us.

Reviewer comment (Theory):
In the setting of Demonstration Regularized RLHF is it necessary to have the coefficients defined in equation 3 in the bound?

Author response:
As it is stated in Corollary 4, to achieve $O(\varepsilon) $ quality it is enough to have either large enough $N^{\text{E}}$ (independent on a coefficient defined in (3)) or large enough $N^{\text{RM}}$ depending on the coefficient defined in (3). If we assume only large enough $N^{\text{RM}}$, then the dependence on $C_r$ seems unavoidable due to the  lower bounds in Theorem 3 by Zhan et al. (2023a).

Reviewer comment (Novelty):
The contributions from the algorithm design part seem not significant. The algorithm is a combination of imitation learning and regularized RL.

Author response:
We are particularly interested in algorithms that are scalable and easily implementable. The simplicity of our algorithm enables its straightforward implementation beyond tabular and linear settings. Our primary focus in this work is to analyze the existing practical pipeline and demonstrate its provable efficiency.

On the other hand, our algorithm, UCBVI-Ent+, introduces a novel time step randomization to control the widths of the confidence intervals, as described in a separate comment available to all reviewers. In our view, this represents a significant algorithmic contribution, enabling the achievement of optimal complexity bounds.

Reviewer comment (Theory):
The results highly depends on the performance of the expert policy. However, in real life applications, obtaining expert demonstrations is usually expensive, and there might be far less offline demonstrations that that considered in this paper. Specifically, from Corollary 3, it seems that the benefit occurs when $N^E>H^3SA$, which is close to the typical sample complexity in standard RL, which might be too much in real applications.

Author response:
Regarding Corollary 3, it is important to emphasize that with significantly less expert data, reconstructing the expert policy with a small error becomes impossible, as indicated in the lower bound (Theorem 2). The application of behavior cloning techniques makes no sense with such a limited amount of data. However, with a reasonable amount of expert data, our approach demonstrates convergence to the optimal policy, surpassing the behavior cloning outcome.

In the RLHF setup, it's crucial to highlight that even a relatively small amount of expert data results in a straightforward and practical algorithm. This algorithm delivers a policy which is close  to the optimal one when provided with a reasonably large preference dataset. In contrast to existing theoretical algorithms for RLHF, our approach distinguishes itself by not necessitating the solution of complex min-max problems to achieve pessimism. Instead, it aligns more closely with approaches actively applied in practical settings.

Reviewer comment (Experiments):
It would be great to include some empirical studies, although this is purely a theory paper.

Author response:
We thank reviewer zApY for the positive and encouraging feedback. We will add numerical results for our tabular algorithm.

Reviewer comment (Experiments):
Can you provide some empirical results to validate the theoretical findings of this paper?

Author response:
We thank reviewer zApY for the positive and encouraging feedback. We will add numerical results for our tabular algorithm.

Reviewer comment (Presentation):
The paper can be better organized, there are too many references pointing towards the appendix.

Author response:
We would like to acknowledge reviewer XPKe that we included the description of algorithm UCBVI-Ent+ in the main text as well as a discussion on its difference with UCBVI-Ent and RL-Explore-Ent. Additionally, we reduced the number of references towards statements in the Appendix by including the sample complexity result for UCBVI-Ent+ and LSVI-UCB-Ent in the main text. All the changes are highlighted in red.

Reviewer comment (Theory):
I am not an expert in RL theory, I am a bit confused by the results presented in Corollary3 and Theorem6, where the bound presented in Corollary3 depends on $N^E$, whereas the bound in Theorem6 does not. Intuitively, I suppose the sample complexity would eventually depend on the size of the expert dataset. Can authors explain this?

Author response:
- Why Theorem 6 does not depend on the size of the expert dataset $N^{\mathrm{E}}$?

Theorem 6 is devoted to the sample complexity of best policy identification for KL-regularized MDPs with respect to any reference policy and any regularization coefficient $\lambda > 0$. To specialize these bounds to Corollary 3, we need to take a behavior-cloning policy $\pi^{\mathrm{BC}}$ as a reference one and choose the regularization parameter  $\lambda$ depending on the number of expert trajectories $N^{\mathrm{E}}$ (choice $\lambda^\star$ from Theorem 3) and the desired accuracy $\varepsilon$ as $\mathcal{O}(N^{\mathrm{E}} \varepsilon / (SAH))$ (as described in Corollary 3).