Paper: PILOT: An $\mathcal{O}(1/K)$-Convergent Approach for Policy Evaluation with Nonlinear Function Approximation (ICLR-2024)

Reviewer comment (Theory):
What is K in the paragraph before the questions “ can we develop … ” on page 2? Suppose this is the K defined after the question. In that case, the phrasing “ more than O(K) number of iterations to achieve the convergence ” seems to suggest that the algorithm terminates with the exact value function after O(K) iterations, which is probably not the case. Please clarify.

Author response:
Thanks for your comments and questions. Here, we clarify that $K$ is the number of iterations an algorithm runs. We will revise the paper and define $K$ earlier, ensuring that it is clearly and unambiguously explained before its first significant use in the text. 

Additionally, in the revision, we will rewrite and refine the sentence to avoid similar future confusion. In the revision, the sentence "...more than $O(K)$ number of iterations to achieve convergence" will be changed to "...achieving convergence rate slower than $O(1/K)$."

Reviewer comment (Writing):
Point (ii), paragraph (2) of the literature review: “our algorithms only require the stepsizes to be sufficiently small, which is easier to tune in practice” — this is not very clear, what does sufficiently small mean? In the previous paragraph a method with O(1/M) step size is criticized, so the authors need to be more accurate here.

Author response:
Thanks for your comments. Note that many existing works in the reinforcement learning (RL) policy evaluation (PE) literature require sophisticated problem instance information for setting step sizes. For example, setting the step-size in the SREDA algorithm in [Luo et al. 2020] requires the knowledge of the condition number $\kappa$, which is hard to estimate in practice. In comparison, our PILOT and PILOT+ algorithms only require the step size to be a sufficiently small constant and do not need the exact values of the parameters $L_f$ and $\mu$. 
 
Specifically, our Theorems 1 and 2 demonstrate that, as long as the step-sizes $\alpha$ and $\beta$ are smaller than the upper bounds stated in these theorems, the $\mathcal{O}(1/K)$ convergence rate of our algorithms is achieved. The phrase "sufficiently small" just means smaller than the upper bounds we provided in the theorems. Our approach does *not* need the exact knowledge of problem instance information, making the step-size setting more convenient in practice.

Lastly, regarding the algorithm in [Wai, et al, 2019], their best convergence results only hold for a small step-size that is $\mathcal{O}(1/M)$, where $M$ denotes the size of the dataset. That is, the step-size setting is *dependent* on the dataset size $M$. This is problematic for RL problems with a large state-action transition dataset size $M$. In contrast, the choices of step sizes in our paper are **independent** of the size of the dataset. 

In this revision, to address your concerns and to avoid similar confusion in the future, we revise this sentence and clearly define what constitutes "sufficiently small" in our paper.

Reviewer comment (Theory):
Paragraph 1, Section 3: The notation $\pi: \mathcal{S} \rightarrow \mathcal{A}$ is only for deterministic policies. Please introduce randomized policies properly.

Author response:
Thank you for pointing out this confusion. In the revised paper, we will change our notation to precisely define policies that could be either deterministic or randomized. The revised notation is as follows: $\pi(a|s): \mathcal{S}\times \mathcal{A} \rightarrow [0,1]$, where $\pi(a|s)$ is the probability distribution of taking an action $a$ from the action space $\mathcal{A}$ given a state $s$ in the state space $\mathcal{S}$. The deterministic policy corresponds to the special case where the probability values can only be 0 or 1.

Reviewer comment (Presentation):
Algorithm 1 refers to Eq. (4) and Eq. (5) which appear a page later.

Author response:
Thank you for pointing out this issue. This is caused by the floating of the algorithm environment in LaTeX, which was overlooked in our initial submission. In the revised paper, we have fixed this problem and placed Eq. (4), Eq. (5), and the algorithm on the same page.

Reviewer comment (Presentation):
Why do the authors use arrows+name for Figures 1-4 and not just a legend?

Author response:
Thanks for your question. The reason of using arrows+name is to create an immediate visual association between the algorithm and the specific experimental result in the figure.

Reviewer comment (Theory):
Since the paper claims the algorithms they propose achieve the first $O(1/K)$ convergence rate ($K$ is the number of iterations) with constant step-sizes for PE with nonlinear function approximation, it is better to give some brief statistical intuition of achieving this result.

Author response:
Thanks for your suggestions. The intuitions of our method achieving the $\mathcal{O}(1/K)$ convergence rate for reinforcement learning (RL) policy evaluation (PE) with nonlinear function approximation are summarized as follows:

**1) Variance Reduction:** The key idea of our approach to achieve a fast convergence is to utilize the state-of-the-art recursive path-integrated variance reduction (VR) technique. Specifically, in RL environments, there is inherent randomness due to the stochastic nature of actions and rewards. Our recursive path-integrated VR helps stabilize the learning process by reducing the impact of this randomness. It effectively mitigates the noise introduced by the stochasticity in RL by periodically evaluating full gradients and using fresh information by using previous iterates. Our proposed VR technique makes the optimization trajectory much smoother. 

We would also like to highlight that similar VR techniques have been adopted in traditional simple minimization optimization problems (e.g., [Fang et al., 2018] and [Wang et al., 2019]). These works showed that recursive path-integrated VR techniques achieve an $O(1/K)$ convergence rate for simple minimization. In this paper, we show that a similar recursive path-integrated VR technique can be applied to the MSPBE-based RL PE problem, which has a far more complex min-max structure.

**2) Constant Step-Sizes and Convergence:** The use of constant step-sizes also plays an essential in achieving an $\mathcal{O}(1/K)$ convergence rate because they allow for consistent and steady improvement of the objective function value. However, using constant step-sizes in stochastic environments is challenging due to the fluctuating nature of stochastic gradients, which often necessitates diminishing step-sizes to avoid oscillations or even divergence. Thanks to the smoothed trajectory resulting from our proposed recursive path-integrated VR technique, we show that the use of constant step-sizes becomes possible without compromising the rate of convergence.

Reviewer comment (Evaluation):
The paper proposes a new metric for convergence performance, and gives the abundant explanation for using this concept, but it is hard to be applied to other performance metric, for example, $|\widehat{V}^{\pi}(s) - V^{\pi}(s)|$, where $\widehat{V}^{\pi}$ is the estimate you get from a policy evaluation problem.

Author response:
Thank you for your valuable feedback regarding the convergence metric $M^{(k)}$ used in our paper. We appreciate your suggested performance metric, i.e., the absolute error between the estimated and actual value functions in policy evaluation, $\left|\widehat{V}^\pi(s)-V^\pi(s)\right|$. However, evaluating $\left|\widehat{V}^\pi(s)-V^\pi(s)\right|$ is intractable in our RL PE problem, because the MSPBE-based PE problem is a non-convex min-max problem, which implies that computing $\left|\widehat{V}^\pi(s)-V^\pi(s)\right|$ is NP-hard. Therefore, just as most first-order methods for non-convex optimization in the literature, convergence to a stationary point is our algorithm design goal for MSPBE-based RL PE, hence we use our proposed performance metric.


Specifically, our proposed metric $M^{(k)}$ is designed to provide a comprehensive measure of convergence in the complex context of non-convex optimization with RL policy evaluation. It comprises two components: the first term $\|J(\theta)\|^2$ in Eq. (6) measures the convergence of the primal variable $\theta$ to a first-order stationary point, reflecting the accuracy of the policy parameterization. In the second term, we can directly assess the dual solution distance to the unique maximizer under any given primal solution, i.e., $\|\omega - \omega^*(\theta)\|$ thanks to the strong concavity.

In summary, although the suggested metric $\left|\widehat{V}^\pi(s)-V^\pi(s)\right|$ is a direct measure of the value function error, it is computationally intractable to use in our convergence and sample complexity analysis. On the other hand, our proposed new metric allows us to prove a $\mathcal{O}(1/K)$ convergence rate for our PILOT algorithms, which is the first such result in the literture. We hope this explanation clarifies the rationale behind our choice of convergence metric and its relation to other performance measures in the field.

Reviewer comment (Theory):
From Corollary 1, the sample complexity of PILOT is $O(\sqrt{M}\kappa^3\epsilon^{-1} +M)$, where $M$ is the number of state-action pairs sampled by the evaluated policy, and $\epsilon$ measures the error under the new metric of convergence performance in this paper. Intuitively, to get a accurate estimate of the $V^{\pi}$, we need a sufficient number of samples (a big $M$). However, it seems that this $\epsilon$ has nothing to do with $M$. Additionally, I have noticed that the Assumption 1 needs $M$ to be sufficiently large. Can authors give the detailed justification of this explanation? I would like to see how $M$ will affect the accuracy of the policy evaluation.

Author response:
Thanks for your comments. However, it appears that there are some misunderstandings regarding the notations $\epsilon$ and $M$, as well as the terminology "sample complexity." Here, we would like to further clarify as follows:

1. **The dataset size $M$:** The reviewer is correct that the larger the value of $M$, the more accurate policy evaluation one can obtain for the value function $V^{\pi}$. However, in the RL PE literature (e.g., [Wai et al. (2019), Qiu et al. (2020), Du et al. (2017)] and also in our work), the goal is *not* to determine how large $M$ should be. Rather, the RL PE problem setting assumes that the dataset of the $M$-step trajectory is already sampled and provided, i.e., the $M$-value is **fixed** and independent of the convergence error $\epsilon$ (more on $\epsilon$ later). In other words, $M$ is *not* a variable to control in RL PE, and it is given as a fixed parameter.
 
2. **Justification of $M$ needs to be sufficiently large in Assumption 1:** Although $M$ is a fixed parameter in the RL PE problem, it needs to be sufficiently large so that designing RL PE algorithms with *fast convergence* is possible. That is, a large $M$ is needed in RL PE to allow the development of fast-convergent PE algorithms (e.g., the $\mathcal{O}(1/K)$ convergence rate in our work). Specifically, many convergence analyses for RL PE algorithms in the literature, including ours, require the strong concavity condition in Assumption 1 to hold. To see why a suffciently large $M$ allows in Assumption 1 to hold, recall that ${\textbf{D}}=\mathbb{E} _{s}\left[\nabla _{\theta} V _{\theta}(s) \nabla _{\theta} V _{\theta}(s)^{\top}\right] \in \mathbb{R}^{d \times d}$, which implies that $\mathbf{D}$ is positive semidefnite. Further, as the number of random samples $M$ increases, $\mathbf{D}$ becomes more and more likely to be full-rank, i.e., $\mathbf{D}$ becomes positive definite. That is, as $M$ becomes sufficiently large, one can always find a $\mu >0$ such that $\mu \leq \lambda _{\min }\left({\textbf{D}}\right)$. This $\mu$ further implies the strong concavity relation stated in Assumption 1.

3. **The convergence error $\epsilon$:** We want to clarify that, in the RL PE literature, $\epsilon$ does *not* represent the accuracy of PE, which is affected by $M$. Rather, $\epsilon$ is a tolerable convergence error under a given convergence metric. In our paper, $\epsilon$ is the tolerable error of our proposed convergence metric in Eq. (6), which captures the stationarity gap of the primal solution $\theta$ and the optimality distance of the dual solution $\omega$. Therefore, $\epsilon$ is a system parameter that is independent of the dataset size $M$. Also, it is important to recognize that **a fast convergence to a PE solution does not necessarily mean that this solution has a high PE accuracy**. However, due to the non-convex nature of the MSPBE-based min-max PE with nonlinear function approximation, convergence to a stationary point is typically the algorithm design goal one would pursue.

4. **The notion of sample complexity:** In the RL PE literature, sample complexity does *not* mean the dataset size $M$ we need. Rather, sample complexity is defined as follows (also see the formal definition in our response to your next comment): Given a *fixed* tolerable convergence error $\epsilon$ and a *fixed* $M$-sized dataset, **how many times of sampling from the dataset** an algorithm needs for gradient estimations, so that the algorithm can converge to within the $\epsilon$-error. In this paper, we show that the number of sampling from the $M$-sized dataset (i.e., sample complexity) is bounded by $O(\sqrt{M} \kappa^3 \epsilon^{-1}+M)$, which is the state-of-the-art result in the literature.

We sincerely hope that the above-detailed explanations can clarify the reviewers' doubts regarding $M$, $\epsilon$, and the notion of sample complexity.

Reviewer comment (Experiments):
The only weakness is the experimental section, which contains rather simple scenarios. At the same time, for the paper which is primarily theoretical, the proposed illustration is sufficient.

Author response:
Thank you for your constructive feedback on the experimental section of our paper. We appreciate your recognition of the primarily theoretical focus of our work and your understanding of the scope of our current experimental validation. **In the appendix of this paper, we have included further experiments on the Navigation tasks, where the state space has 30 dimensions.** We hope that these additional larger-scale experiments could strengthen the empirical evidence supporting our theoretical findings and provide deeper insights into the practical applicability of our proposed algorithms.

Reviewer comment (Experiments):
If time permits, I would suggest the authors to complement the experimental section.

Author response:
Thank you for your constructive feedback on the experimental section of our paper. We appreciate your recognition of the primarily theoretical focus of our work and your understanding of the scope of our current experimental validation. **In the appendix of this paper, we have included further experiments on the Navigation tasks, where the state space has 30 dimensions.** We hope that these additional larger-scale experiments could strengthen the empirical evidence supporting our theoretical findings and provide deeper insights into the practical applicability of our proposed algorithms.

Reviewer comment (Novelty):
The authors could also complement the bibliography on linear policy evaluation methods (TD(0) with modifications) by the following papers:  
1. Li, Tianjiao, Guanghui Lan, and Ashwin Pananjady. "Accelerated and instance-optimal policy evaluation with linear function approximation." arXiv preprint arXiv:2112.13109 (2021) - https://arxiv.org/abs/2112.13109  
2. Patil, Gandharv, et al. "Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation." International Conference on Artificial Intelligence and Statistics. PMLR, 2023. - https://arxiv.org/abs/2210.05918  
The first one concerns instance-optimal guarantees for Polyak-Ruppert averaged iterates of TD(0). The authors also use SPIDER-type variance reduction. The second paper concerns TD(0) with realizable step size.

Author response:
Thank you for suggesting additional references to complement the bibliography of our paper. In this rebuttal period, we have carefully read these two papers. The relationships and differences between these two papers and our work are summarized as follows:


[R4] investigates the finite-time behavior of the widely-used temporal difference (TD) learning algorithm when combined with the tail-averaging technique and [R4] achieves $\mathcal{O}(1/K)$ convergence rate. Tail averaging is a technique that averages the model parameters over the last few iterations (the "tail" of the optimization process). This method can lead to a more stable and generalizable model because it effectively smooths out the noise and variance in the parameter updates toward the end of the training. Similarly, [R3] provided instance-optimal guarantees for Polyak-Ruppert averaged iterates of TD(0). However, it's worth noting that these two works only focused on the **linear function approximation** setting, as a special case of the **nonlinear function approximation** setting considered in our paper. Moreover, as RL applications become increasingly sophisticated, most recent RL algorithms utilize **nonlinear function approximation** (e.g., deep neural network (DNN)) to model the value function. This motivates us to develop a new PILOT method, which is able to achieve a fast convergence speed for RL policy evaluation with nonlinear function approximation.

We will incorporate the references suggested by the reviewer into the related work section and add the above discussions in our paper. These additions will undoubtedly enhance the breadth and depth of our paper, providing a more comprehensive overview of the state-of-the-art in policy evaluation methods.

Reviewer comment (Theory):
I am also a bit surprised by the fact that the mixing time of the original chain $(s_1,a_1,s_2,\ldots)$ does not pop up explicitly in the bounds. This would be a typical behavior for the optimization problems with dependent data. What is the explanation?

Author response:
Thanks for your comments. We agree with the reviwer that the samples of the $M$-step trajectory $(s_1,s_1,\ldots,s_M,a_M,s_{M+1})$ is Markovian. However, due to the challenge of handling dependent data, in the literature, many policy evaluation methods with linear/nonlinear function approximations and temporal differences (e.g., [Wai et al. (2019), Qiu et al. (2020), Du et al. (2017)]) ignored the Markovian data dependence and still used the finite-sum empirical loss is an approximation of the expected loss function for PE. As a result, mixing time does not appear in the convergence bounds in these works. Moreover, as discussed in the footnote on Page 5 in our paper, although the finite-sum empirical loss is an approximation of the expected loss function, it has been shown in [Chen et al. (2021)] that, under the conditions of bounded instantaneous loss and bounded derivatives (satisfied for most applications in practice), the approximation error of using empirical loss for Markovian data is small with high probability (cf.Lemma~2 [Chen et al. (2021]). Thus, the empirical loss has been widely used as a proxy for the expected loss of Markovian data in the literature [Liu et al. (2015); Wai et al. (2019); Du et al. (2017)].

Meanwhile, we do agree with the reviewer that a more thorough analysis incorporating the effects of the mixing time of the Markovian data will provide a more complete and deeper understanding of PE algorithms' convergence behavior, especially in complex environments where the dependency structure of the data cannot be ignored. How to develop variance-reduction-based accelerated PE algorithms with nonlinear function approximation under Markovian data is an important open problem, which deserves an independent paper dedicated to this topic. This problem will be left as an important future research, and we thank the reviewer for bringing up this direction.