Paper: Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR-2024)

Reviewer comment (Presentation):
One question about Figure 2: it is hard to differentiate the curves generated by different approaches, thus difficult to see the message from this figure. I suggest the authors improve the presentation of Figure 2.

Author response:
Thanks for your review! We would definitely improve the presentation of Figure 2 to make it clearer to tell the difference between two approaches.

Reviewer comment (Novelty):
The novelty of the proposed method is limited. The algorithm is a combination of natural actor-critic and hybrid-RL [1], and it needs to be clarified what is the technical challenge in combining them.

Author response:
### What is the technical challenge in combining Hybrid RL with natural actor critic?
The main technical challenge of this paper is to achieve best-of-both-world type guarantees where we can still achieve meaningful results where either offline or online assumptions fail. This can be particularly meaningful in real-world problems where it is hard to verify the assumptions. To do this, we propose a novel HPE subroutine in Algorithm 2, where we optimize simultaneously for TD error and monte carlo regression loss and we need to carefully balance the statistical error term from the TD error and monte carlo regression such that the error of the other term does not blow up when the realizability of one term is not met. In fact, the TD error and the MC regression error interleave with each other and our analysis uses one to supervise the other (e.g., MC regression error is used to bound the Bellman residual under the offline distribution). The entirety of Appendix B is dedicated to this proof.

We emphasize that, to the best of our knowledge, and despite being simple, our hybrid PE algorithm 2 is new. Prior work either just focuses on pure on-policy algorithms or pure offline TD-style algorithms. There is no provable work that attempts to combine them together. By combining these two components together into a single objective in a principled manner, we managed to get the best of both worlds style results.

Reviewer comment (Theory):
I am also concerned with the claim that the proposed method achieves the best of both worlds. While it is nice to have guarantees without the Bellman completeness assumption, other strong assumptions are still required. The realizability for any $Q^\pi$ is a strong assumption, and the value-based method usually only requires the realizability of $Q^*$. More importantly, the assumption on NPG coverage is also strong since it is an all-policy coverage assumption. Offline RL only needs single-policy coverage assumption as in Definition 3, and it can be proved that $C_{off,\pi^e}^2 \leq C_{npg,\pi^e}$ [1,2]. This invalidates the claim that the proposed method achieves the best of both worlds since when the offline assumption does not hold, the online guarantee is lost, too. I expect an exploration mechanism so that we can still achieve low regret when the offline data has insufficient coverage.

Author response:
### The assumption for online RL such as realizability of Q^{\pi} and the definition of $C_{NPG}$ is too strong and invalidates the claim of best of both worlds.
We would like to first note that $Q^{\pi}$ realizability is a very standard assumption in the Policy Gradient literature. For example, prior works (including [1,2,3]) all use this Q^\pi realizable assumption (indeed, assumptions in [1,3] are stronger than Q^\pi being realizable).

Furthermore, our definition of $C_{NPG}$ is only ***single-policy*** concentration with respect to a single comparator policy $\pi_e$ and it can be shown that our definition of $C_{NPG}$ is weaker than the single-policy density ratio $\|\frac{d^{\pi^e}}{\nu}\|_{\infty}$ where $\nu$ is the reset distribution. The notion of a good reset distribution is also a standard assumption in PG literature [4,5]. In particular, Page 124 Figure 0.1 of [5] shows that without a good reset distribution, PG methods can easily get stuck in a local optimum. 

There are indeed two cases where online guarantees hold while offline guarantees fail. Firstly, when the offline distribution is not admissible (generated by some policy) we can no longer show $C_{off, \pi_e}^2 < C_{npg, \pi_e}$, and it is possible that $C_{off, \pi_e}^2$ is unbounded while $C_{npg, \pi_e}$ is bounded. Secondly, and more importantly as shown in our simulation experiments in Section 6, when the offline distribution is admissible but the Bellman completeness assumption does not hold, the online guarantee of our algorithm is still meaningful.

Reviewer comment (Theory):
What is the technical challenge in combining natural actor critic and hybrid RL?

Author response:
### What is the technical challenge in combining Hybrid RL with natural actor critic?
The main technical challenge of this paper is to achieve best-of-both-world type guarantees where we can still achieve meaningful results where either offline or online assumptions fail. This can be particularly meaningful in real-world problems where it is hard to verify the assumptions. To do this, we propose a novel HPE subroutine in Algorithm 2, where we optimize simultaneously for TD error and monte carlo regression loss and we need to carefully balance the statistical error term from the TD error and monte carlo regression such that the error of the other term does not blow up when the realizability of one term is not met. In fact, the TD error and the MC regression error interleave with each other and our analysis uses one to supervise the other (e.g., MC regression error is used to bound the Bellman residual under the offline distribution). The entirety of Appendix B is dedicated to this proof.

We emphasize that, to the best of our knowledge, and despite being simple, our hybrid PE algorithm 2 is new. Prior work either just focuses on pure on-policy algorithms or pure offline TD-style algorithms. There is no provable work that attempts to combine them together. By combining these two components together into a single objective in a principled manner, we managed to get the best of both worlds style results.

Reviewer comment (Theory):
When the offline guarantee does not hold, how can we still retain the on-policy guarantee?

Author response:
### The assumption for online RL such as realizability of Q^{\pi} and the definition of $C_{NPG}$ is too strong and invalidates the claim of best of both worlds.
We would like to first note that $Q^{\pi}$ realizability is a very standard assumption in the Policy Gradient literature. For example, prior works (including [1,2,3]) all use this Q^\pi realizable assumption (indeed, assumptions in [1,3] are stronger than Q^\pi being realizable).

Furthermore, our definition of $C_{NPG}$ is only ***single-policy*** concentration with respect to a single comparator policy $\pi_e$ and it can be shown that our definition of $C_{NPG}$ is weaker than the single-policy density ratio $\|\frac{d^{\pi^e}}{\nu}\|_{\infty}$ where $\nu$ is the reset distribution. The notion of a good reset distribution is also a standard assumption in PG literature [4,5]. In particular, Page 124 Figure 0.1 of [5] shows that without a good reset distribution, PG methods can easily get stuck in a local optimum. 

There are indeed two cases where online guarantees hold while offline guarantees fail. Firstly, when the offline distribution is not admissible (generated by some policy) we can no longer show $C_{off, \pi_e}^2 < C_{npg, \pi_e}$, and it is possible that $C_{off, \pi_e}^2$ is unbounded while $C_{npg, \pi_e}$ is bounded. Secondly, and more importantly as shown in our simulation experiments in Section 6, when the offline distribution is admissible but the Bellman completeness assumption does not hold, the online guarantee of our algorithm is still meaningful.

Reviewer comment (Evaluation):
Can you provide more insights on the superior performance of the proposed method on the comblock environment?

Author response:
### Intuition of the superior performance of HNPG in the comblock environment.
Compared with pure online RL such as TRPO, HNPG is better in the use of offline data to help explorations in hard exploration environments such as comblock. Comparing state-of-the-art hybrid RL approaches such as RLPD, they rely heavily on Bellman completeness by doing bootstrapping for both online and offline data. In HNPG, the dependence on Bellman completeness is relaxed by doing Monte Carlo regression for online data thus working better for the real image observations where Bellman completeness does not necessarily hold.

Reviewer comment (Theory):
In Algorithm 3, HNPG solves the squared loss regression with a linear critic. The requirement of a linear critic is probably mainly for theoretical analysis and could be relaxed to more general function classes (e.g., neural function approximation). A further discussion (possibly with some experiments) could make the proposed algorithm more impactful.

Author response:
We would like to clarify that the linear critic in Algorithm 3 is practical for the following two reasons: 1) the feature used in Algorithm 3 is $\nabla \ln \pi(a|s)$ which is equal to $\nabla \pi(a|s)/\pi(a|s)$ where $\nabla \pi(a|s)$ is the neural tangent kernel, i.e., define $\phi(s,a) := \nabla \pi(a|s)$, the kernel $k(z,z’) = \phi(z)^T \phi(z’)$ is called a neural tangent kernel ([1] and Lemma 8.1.1 of [2]). Therefore, the linear regression in Algorithm 3 is actually neural tangent kernel linear regression, and the neural tangent kernel has been shown to be expressive. 2) Practical implementations of NPG like TRPO uses fisher information matrix to precondition the gradient and it can shown that it is equivalent to first do regression with $\nabla \ln \pi(a|s)$ feature.

Reviewer comment (Writing):
The concept and the terminology of “NPG coverage” in Definition 2 could be further elaborated on. Specifically, Definition 2 is defined w.r.t. some comparator policy, which is not necessarily related to NPG.

Author response:
Our definition of NPG only requires coverage of a single policy instead of requiring coverage of all policies (as commonly used in the NPG literature such as [4,5]). It can be shown that our definition of $C_{NPG}=\max_\pi |\frac{d^{\pi^*}}{d^\pi}|_{\infty}$ 

is weaker than the single-policy density ratio of the reset distribution $\|\frac{d^{\pi^*}}{\nu}\|_{\infty}$ except a factor of $\frac{1}{1-\gamma}$. 

We also note that positivity of reset distribution, while being sufficient is not necessary for bounded $C_{NPG}$. In cases where we do not have positivity of the reset distribution, $C_{NPG}$ could be bounded when the MDP dynamics is favorable. As an example, a typical case where $C_{NPG}$ holds is Ergodic MDP (for example, see Assumption 2 from [6]) which implies that the probability of any policy to visit any reachable state is strictly positive.

Reviewer comment (Theory):
To calculate the loss in Eq. (1), one would need to obtain an unbiased estimate $y$, which could be obtained via Monte Carlo sampling. On the other hand, it would probably be better to get $y$ with the help of bootstrapping. Would this still preserve a similar sub-optimality guarantee as in Theorem 1?

Author response:
Using bootstrapping to get y typically requires the assumption of Bellman completeness (e.g., algorithms like TD which use bootstrapping often require very specific assumptions to guarantee to succeed), which is not needed in standard NPG analysis. In order to achieve online RL guarantees when the assumption of Bellman completeness does not hold, we choose to use Monte Carlo estimation for y.

Reviewer comment (Theory):
Regarding Definition 2, how to ensure a finite $C_{npg}$ without the assumption on the positivity of the reset distribution?

Author response:
Our definition of NPG only requires coverage of a single policy instead of requiring coverage of all policies (as commonly used in the NPG literature such as [4,5]). It can be shown that our definition of $C_{NPG}=\max_\pi |\frac{d^{\pi^*}}{d^\pi}|_{\infty}$ 

is weaker than the single-policy density ratio of the reset distribution $\|\frac{d^{\pi^*}}{\nu}\|_{\infty}$ except a factor of $\frac{1}{1-\gamma}$. 

We also note that positivity of reset distribution, while being sufficient is not necessary for bounded $C_{NPG}$. In cases where we do not have positivity of the reset distribution, $C_{NPG}$ could be bounded when the MDP dynamics is favorable. As an example, a typical case where $C_{NPG}$ holds is Ergodic MDP (for example, see Assumption 2 from [6]) which implies that the probability of any policy to visit any reachable state is strictly positive.