Paper: On the Global Convergence of Natural Actor-Critic with Neural Network Parametrization (ICLR-2024)

Reviewer comment (Novelty):
The authors claim in the introduction that existing works that study actor-critic with neural network function approximation only provide asymptotic results. However, the reviewer found the following paper, which is not compared with in this paper: Cayci, Semih, Niao He, and R. Srikant. "Finite-time analysis of entropy-regularized neural natural actor-critic algorithm." arXiv preprint arXiv:2206.00833 (2022). The reviewer requests clarification on how the result in this submission differentiates from the one in this paper from 2022.

Author response:
With regards to the paper Cayci et.al (2022), thank you for pointing it out. Since the paper is on archive, it did not come across in our literature review. However, having gone through it, we can see that in this paper, the actor and critic are both restricted to a two layer neural network, this is in contrast to our work which allows for a multi layer actor and critic. Additionally, note that this work achieves a sample complexity of $\epsilon^{-6}$ which is worse than our result or $\epsilon^{-4}$ even in this restricted setting. We will add this paper in our related works.

Reviewer comment (Novelty):
The authors consider the flexibility of allowing non-i.i.d. samples in their analysis as a contribution of this paper, but the following paper, which the authors have already cited, also allows Markovian data: Xu, Tengyu, Zhe Wang, and Yingbin Liang. "Improving sample complexity bounds for (natural) actor-critic algorithms." Advances in Neural Information Processing Systems 33 (2020): 4358-4369. The reviewer requests clarification on how the analysis technique used in this paper differs from the one in that paper, noting that the cited result is only for linear function approximation and questioning whether this makes a big difference in the part of analysis pertaining to concentration over non-i.i.d. Markovian data.

Author response:
Secondly with respect to the the non-i.i.d analysis in Xu et al.(2020), the ergodicity assumption is used to bound the difference between the estimate of the advantage function obtained form the Markovian data and that which would have been obtained from i.i.d data. This can be seen in Page 23 equation 25. For our case we account for the Markovian dependence in the critic samples by splitting the critic optimization as given in Equation (25), here the corresponding term is $\epsilon^{3}_{k,j}$. This term is bounded using the theory in Bertail and Porter (2019). Additionally, since we calculate the natural gradient update through the optimization given in equation (29), we account for the Markovian dependence of its samples through the result of Doan (2022). Note that by using this optimization we avoid having to invert our estimate of the fisher information matrix as is the case for Xu et al.(2020).

Reviewer comment (Reproducibility):
What is the architecture of the neural network in Algorithm 1? How does this function class look like? The authors stated $L$ layers and $m$ neurons per layer in Section 4, but the reviewer asks whether there are any other requirements or specifications and requests a paragraph stating this more rigorously.

Author response:
With regards to the architecture of the network, the results we have used from Zhu (2019), were proven for  a fully connected feed-forward neural network with ReLU activation layers. This result also holds for a smooth activation function (the author states that the results can be made sharper for a smooth activation function), this result would also apply for a conventional neural network as well as a residual neural network. For our case we can assume the case of a fully connected neural network with $L$ layers, at least $m$ neurons per layer and with an ReLU activation function for each layer.

Reviewer comment (Theory):
How does the size of the neural network function class or $\Theta$ affect the variance portion of the final error bound? The current bound does not show any dependence on it. The reviewer requests an explanation of why the upper bound of the expectation of a supremum over $\Theta$ does not need to depend on the complexity of $\Theta$ in Equation (110). Alternatively, if the Rad quantity in (110) actually depends on $\Theta$, the reviewer asks that the authors explicitly present in their theorem how $C_k$ in Equation (111) depends on $\Theta$, as is common in existing theoretical results. The reviewer stresses that this is important because taking a covering with a supremum usually compromises the sharpness of the bound, and comparisons with other rates might be unfair if this aspect is not shown.

Author response:
With regards to Equation (110), the set $\Theta$ here does not represent the class of all possible neural networks but only the two neural networks corresponding to the parameters $\theta_{k,j}^2$  and $\theta_{k,j}^3$, this is a typo on our part, we will rename it to $\Theta^{'}$ and state that it is the set of only two parameters $\theta_{k,j}^2$ and $\theta_{k,j}^3$. This is because we only need this bound for the two mentioned parameters as shown in Equation (112) and (113). We will make that change in the final version. With regards to the property of the function class, Bertail and Porter mentions that the function class has to be uniformly bounded. This condition is satisfies for the set $\Theta^{'}$ as it has only two elements and the state action pairs are defined on a bounded space.

Reviewer comment (Theory):
The reviewer seeks confirmation of the following understanding: $\alpha m$ can be $\Theta(\frac{1}{\mathrm{poly}(n,L)})$, so the second line of Equation (17) can be viewed as $\frac{1}{K(1-\gamma)}\sum_{k=1}^K\sum_{j=0}^{J-1}(1 - \frac{1}{n^2})^n$. Since $(1 - \frac{1}{n^2})^n$ is monotonically increasing and converges to a constant, the reviewer asks whether this term is on the order of $\frac{J}{1-\gamma}$ and if this is the case, notes that it is non-decaying.

Author response:
Thank you for pointing out this result, this was an over-site on our part. The term that you mentioned $\left(1-\frac{1}{n^{2}}\right)^{n}$ does indeed converge to $1$ and hence the term on the right you have pointed out will be non-decaying. However, we can resolve this by a minor modification. In lines  $7-11$ of Algorithm 1 we perform stochastic gradient descent with data sample of size $n$ and also putting the number of iterations equal to $n$. We did not consider the edge case which you have mentioned where the error term becomes non-decaying. However this can be remedied with a minor modification. Instead of lines  $7-11$ of Algorithm 1 we will have the following.
  * Sample and store $n$ state action pairs and the resulting rewards using policy $\pi^{\lambda_{k}}$
  * Then randomly sample the tuple $(s_{i},a_{i},r(s_{i},a_{i}),s_{i+1})$ from the stored data, calculate $y_{i}$ and preform the gradient descent step as in line 9 and 10 of Algorithm 1 respectively. This random sampling and update will be performed for $T_{k,j}$ iterations.

This will result in the superscript in the term to be $T_{k,j}$ instead of $n$. This can be seen from Equation 13.2 on page 34 of Zhu (2019) (please note that for some reason, the supplementary part of this paper is not available on the PMLR website, but is available on the archive version). Now using the result of Theorem 2 of  Zhu (2019), for $T_{k,j} = \Theta\left(poly(n,L)\cdot{\log\frac{1}{\epsilon}}\right)$ the term inside the summation will be upper bounded by epsilon.

Reviewer comment (Writing):
What is $\mu_k$ in Theorem 1? The reviewer points out that the loss function in Equation (9) seems to be just MSE.

Author response:
The term $\mu_{k}$ represents the strong convexity parameter in Equation (9), since the function in Equation (9) is the expectation of a square loss function, we have to consider its strong convexity parameter.

Reviewer comment (Theory):
The reviewer requests clarification of the following in Remark 3:  
> This term is present due to the fact that the optimization for the critic step is non-convex, hence convergence can only be guaranteed with a high probability. We show in the Appendix F that if the critic is represented by a two layer neural network with ReLU activation, using the convex reformulation as laid out in Mishkin et al. (2022), a deterministic upper bound on the error can be obtained.  
The reviewer notes that high probability bounds are normally due to randomness in the samples and asks why the authors attribute it to non-convexity. Furthermore, when everything is convex, the reviewer asks why a deterministic result can be obtained when the samples from Line 5 and 10 of Algorithm 1 are still random.

Author response:
The high probability term is due to the random initialization of the parameter estimates. (Mishkin 2022) showed that the optimization for a 2 layer neural network with ReLU activation function can be transformed into a convex problem. For this optimization the parameters are initialized. We did not use Algorithm 1 for this case. We used Algorithm 2 (the optimization is carried out in Algorithm 3 where in line 3 the parameters are initialised to zero) in Appendix E.

Reviewer comment (Writing):
The notation $n.J$ in Algorithm 1 and Theorem 1 seems odd. The reviewer asks whether the authors meant to write $n\cdot J$, and notes that the same issue appears in other instances of dots.

Author response:
We apologize for the error. We will correct the notation and use  $\cdot$ instead of .

Reviewer comment (Writing):
The authors did not give a definition for the notation r’(s,a), which caused some confusion. According to the description of Definition 2, r’(s,a) can be understood as a realization of R(\cdot | s,a), which is the same as r(s,a).

Author response:
We will define $r^{'}(s,a)$ as the realisation of the random reward at the state action pair. We will also replace $r^{'}(s,a)$ in Equation (27) and replace $r(s,a)$ with $r^{'}(s,a)$ in Equation (28). Additionally, in Algorithm 1, $r(s,a)$ will be replaced by  $r^{'}(s,a)$.

Reviewer comment (Theory):
Page 3, the definition of Bellman operator in (3) uses one realization of the reward r’(s,a) to define the Bellman operator. This makes (T^\pi Q)(s,a) a random variable, so T^\pi Q^\pi = Q^\pi does not hold. The authors could consider the setting when the reward is deterministic, or redefine the Bellman operator with r’(s,a) replaced by its conditional expectation. If T^\pi Q is not deterministic, \eps_{approx} in Assumption 4 could be as large as O(1) according to the variance of the reward, making Theorem 1 meaningless. In page 8 eqn (24), Lemma 2 of Munos (2003) is also for deterministic reward.

Author response:
We apologise for the error in the definition of $T^{\pi}Q(s,a)$. It should have been defined as $T^{\pi}Q^{\pi}(s,a) = \mathbb{E}(r(s,a)) + {\gamma}{\int}Q(s^{'},\pi(s^{'}))P(ds^{'}|s,a)$, where $\mathbb{E}(r(s,a))$ is the expected reward given the state action pair $(s,a)$. This would make $T^{\pi}Q(s,a)$ a fixed value and not a random variable. With regards to our comment about about the result in Munos (2003), we referenced that result as a result  similar to that was used in Faharmand (2010), which we used for our analysis. As for the result we used (in Equations 74-77), it depends upon the property $T^{\pi}Q^{\pi}(s,a)=Q^{\pi}(s,a)$. This property will hold even for a random reward function. This is shown as follows.

    \begin{eqnarray}
        Q^{\pi}(s,a) &=& \mathbb{E}\left[ \sum_{t=0}^{\infty}{\gamma}^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a\right] \\\\
                    &=& \mathbb{E}(r(s,a))  + \gamma \mathbb{E} \left[\sum_{t=1}^{\infty} {\gamma}^{t-1}r(s_{1},a_{1})|s_{0}=s,a_{0}=a \right] \\\\
                     &=&  \mathbb{E}(r(s,a))  + {\gamma}{\int}Q(s^{'},\pi(s^{'}))P(ds^{'}|s,a) \\\\
                     &=&  T^{\pi}Q^{\pi}
    \end{eqnarray}

Reviewer comment (Theory):
Page 4 before (8), the authors use the property of compatible function approximation, which is an extremely strong assumption for neural network parametrization. According to Sutton, this means \frac{\partial Q}{\partial \theta} = \frac{\partial \pi}{\partial \lambda} / \pi, and Sutton points out that “Q being linear in the features may be the only way to satisfy this condition”. This assumption is too strong to make the work practical, and there is no critic parametrization until eqn (10). A normal version of the policy gradient theorem might be sufficient.

Author response:
With regards to the reference of Sutton (1999), we would like to point out that before it states that a linear parameterization of the critic is required, in the line just above this statement (at the bottom of page 3), the  author assumes that the policy is being parameterised using a linear function. It is for this reason that a linear parametrization of the critic is required. In our case where we assume that the policy parametrization is smooth and not necessarily linear, restricting the critic to be a linear function would also require us to restrict the policy class to be linear. This is not something that is done typically in practice. As for the piratical feasibility of using a neural network for the critic, it is shown in Bharadwaj (2022), natural actor critic with a neural parameterization does converge.

Reviewer comment (Theory):
Theorem 1 (and Lemma 7) lacks any description or assumption on \mu_k. The constants for big O in the theorem depend on \mu_k and hence on K, making the theorem meaningless. A positive lower bound for \mu_k is suggested. Note that \mu_k is also the lower bound for the eigenvalue of the Fisher information matrix (7), which guarantees that F is invertible.

Author response:
Thank you for pointing this out. We agree that we should have added an additional assumption in order to ensure the constant appearing on the loss function is independent ok $K$. Thus the additional assumption will be as follows.
    * There exists a constant $\\mu$ such that for any policy $\pi_{\lambda}, \lambda \in \Lambda$ we have 
   
        \begin{eqnarray}
         \mathbb{E}_{s,a \sim \zeta^{\lambda}} \left(\nabla{\log}(\pi(a|s))\nabla{\log}(\pi(a|s))^{T}\right)  \succeq  
         \mu{\cdot}I_d
         \end{eqnarray}

        Where  $\zeta^{\lambda}$ is the stationary state distribution corresponding to the policy $\pi_{\lambda}$, $I_d$ is the identity matrix of dimension $d$ which is the dimensionality of  $\Lambda$.

Reviewer comment (Theory):
Lemma 6 contains the phrase “with high probability,” yet according to its description there is no randomness in (50). (50) left is not an expectation but a conditional expectation only with respect to the RL problem. Randomness comes from the initialization of the neural network, etc., but the authors give no description of the network (except the width). Inequality (50) could only hold when a very explicit setting for everything is given. Details are needed to justify the application of the referenced theorems.

Author response:
The randomness in Equation (50) is as you suggested because of the random initialization of the neural network parameters during the critic step. With regards to the architecture of the network, the results we have used from Zhu (2019), which  were proven for a fully connected feed-forward neural network with ReLU activation layers. The results also holds for a smooth activation function (the author states that the results can be made sharper for a smooth activation function). The result would also apply for a conventional neural network as well as a residual neural network. For our case we can assume the case of a fully connected neural network with $L$ layers, at least $m$ neurons per layer and with an ReLU activation function for each layer.

Reviewer comment (Novelty):
Page 2 Related works: actor-critic methods. Papers like “Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost” (NIPS2019) and “Single Timescale Actor-Critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees” (JMLR2023) are also related.

Author response:
Thank you for pointing out these works. We will incorporate them in the related works, the reason we did not focus on the case of Linear Quadratic regulators is because we were focusing on existing literature where linear critic is used. A linear quadratic regulator would be a further simplification and hence not our focus.

Reviewer comment (Theory):
Page 3, why does the stationary distribution \rho^\pi_\nu depend on the initial distribution \nu, especially when Assumption 2 holds?

Author response:
They stationary distribution as you have pointed out does indeed not depend upon the starting distribution if the Markov chain is ergodic, we will make the required change.  However, we do not know of any rigorous proof to show that the visitation distribution will not depend upon the initial distribution.

Reviewer comment (Writing):
Page 4, please give the full name of DQN when it first appears.

Author response:
We will rename it to Deep Q Networks algorithm.

Reviewer comment (Theory):
Page 6 after (15), in Xu and Gu (2020), the assumption described in the paper cannot be found.

Author response:
This is a typo. The paper were referring to here was Liu et al. (2020). The assumption we referred to can be seen in Assumption 4.4 on page 7.

Reviewer comment (Novelty):
The contribution of this paper needs to be highlighted: it seems that the result can be provided by combining the NTK analysis and the global convergence of NAC (Xu et al., 2020a). Since NTK analysis views the neural networks as a kernel method, the analysis is just let the width of $m$ be very large and then does the error analysis given the linear function.

Author response:
* With regards to your comment about combining NTK analysis with  for the work of Xu et al. (2020a), we would like to  __point out a key deficiency in that approach__. In Xu et al. (2020a), while the policy class is non linear, the critic class has been restricted to a linear function. Now note that in Sutton(1999) on page 5, it is stated that a critic is restricted to be linear only when the policy class is also parameterized by a linear function for the compatible function approximation to hold. Thus in order for the compatible function approximation to hold, in the limit of the of the infinite width, both the actor and critic should converge to a linear kernel. If such a result is achieved, we will not recover the result of  Xu et al. (2020), where the policy is still non-linear and the critic is linear. As  Xu et al. (2020a) itself notes, the term $\zeta^{actor}_{approx}$ (which is the term Sutton (1999) references) can be said to be small for an  overparameterized neural policy according to Wang (2019). This is mentioned in page 8 just before Theorem 3. However, Wang (2019) states that __both__ actor and critic are represented by a neural network for this error to be small, something that is not possible in  Xu et al. (2020a) as the critic has to be linear. 
    
    __However, results achieving convergence for the natural actor critic in the infinite width limit have been proven and our result improves upon them.__
    
    Indeed, Wang (2019), mentioned in Xu et al. (2020a) is such a work. It establishes an upper bound only for the case where both the actor and critic are restricted to a two layer network with ReLU activation.
    If you look at the convergence result in Theorem A.4. We see that while the iteration complexity of the result is $\mathcal{O}(T^{-\frac{1}{2}})$ the sample complexity required for the to achieve an $\epsilon$ error bound is $\epsilon^{-8}$. additionally, we see that the extra error due to neural network architecture, note the $\mathcal{O}(m^{-\frac{1}{8}})$ term in Theorem A.4, means that the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{8}})$. For our result, we achieve a sample complexity of $\mathcal{O}(\epsilon^{-\frac{1}{4}})$ and the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{2}})$. 

    Similarly for the result in Fu et al. (2020), while the requirement for a two layer actor and critic is extended to a multi layer actor and critic, the sample complexity even in the asymptotic limit is $\mathcal{O}(\epsilon^{-\frac{1}{6}})$ as can be seen in Theorem C.5. Additionally the minimum width in the neural network required is of the order $\mathcal{O}(T^{-\frac{1}{6}})$, where $T$ is the number of iterations of the algorithm. Since the iteration complexity is given by $\mathcal{O}(\epsilon^{-\frac{1}{2}})$, the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{12}})$. Thus both in terms of sample complexity as well as the minimum width of the neural network, our result improves upon it. 

     We would also like to point out that both these works assume that sample can independently sampled from the stationary state distribution of a given policy, something which is not possible in practice. Our work does not have this restriction. Additionally, both prior works require the use of energy based policies, our analysis is more general, which includes energy based policies but is not restricted to them. Finally, we would like to point out that these results also do not have any empirical studies.

Reviewer comment (Novelty):
The authors are encouraged to highlight the contribution of this paper based on the current well-developed NTK theory and NAC theory.

Author response:
* As per the reviewers suggestion, we will add more references related to NTK theory.

Reviewer comment (Novelty):
The authors claim that they are the first to show the sample complexity of natural actor critic algorithms with neural networks. However, to my understanding, the asymptotic converge result in Wang et al, 2019 and Fu et al, 2020 can be converted to an upper bound on the sample complexity directly. Given that, the contribution in this study is rather incremental.

Author response:
* With regard to the two papers mentioned, we respectfully disagree with the reviewer.  Let us consider them individually. 
    
    1. First, Wang (2019), establishes an upper bound only for the case where both the actor and critic are restricted to a two-layer network with ReLU activation. Furthermore, if we look at the convergence result for the neural policy gradient in Theorem A.4. We see that while the iteration complexity of the result is $\mathcal{O}(T^{-\frac{1}{2}})$ the sample complexity required for the to achieve an $\epsilon$ error bound is $\epsilon^{-8}$. Additionally, we see that the extra error due to neural network architecture, note the $\mathcal{O}(m^{-\frac{1}{8}})$ term in Theorem A.4, which means that the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{8}})$. For our result, we achieve a sample complexity of $\mathcal{O}(\epsilon^{-\frac{1}{4}})$ and the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{2}})$.   
   2. Secondly,  in Fu et al. (2020), while the requirement for a two layer actor and critic is extended to a multi layer actor and critic, the sample complexity even in the asymptotic limit is $\mathcal{O}(\epsilon^{-\frac{1}{6}})$, this can be seen in the discussion below theorem C.5. Also, the minimum width in the neural network required is of the order $\mathcal{O}(T^{-\frac{1}{6}})$, where $T$ is the number of iterations of the algorithm. Since the iteration complexity is given by $\mathcal{O}(\epsilon^{-\frac{1}{2}})$, the minimum width in the neural network required is of the order $\mathcal{O}(\epsilon^{-\frac{1}{12}})$. Thus both in terms of sample complexity as well as the minimum width of the neural network, our result improves upon the results mentioned. 

__Other novelties in our work__ We would also like to point out that both these works assume that sample can independently sampled from the stationary state distribution of a given policy, something which is not possible in practice. Our work does not have this restriction. Also, both prior works require the use of energy based policies, our analysis is more general, which includes energy based policies but is not restricted to them. Finally, we would like to point out that these results also do not have any empirical studies.

Reviewer comment (Novelty):
Missing reference: Agarwal, Alekh, et al. "On the theory of policy gradient methods: Optimality, approximation, and distribution shift." The Journal of Machine Learning Research 22.1 (2021): 4431-4506.

Author response:
* It seems we have cited the version of this paper published in COLT and not the one in JMLR. We will cite the JMLR version in the final version of our paper.

Reviewer comment (Reproducibility):
The paper does not present a detailed description on how to parameterize the Q function and the policy with neural network in the main part of the paper. A detailed description or a hyper-link to it is helpful.

Author response:
* With regards to the architecture of the network, the results we have used from Zhu (2019), which  were proven for a fully connected feed-forward neural network with ReLU activation layers. The results also holds for a smooth activation function (the author states that the results can be made sharper for a smooth activation function). The results in Zhu (2019) would also apply for a conventional neural network as well as a residual neural network. For our case we can assume the case of a fully connected neural network with $L$ layers, at least $m$ neurons per layer and with an ReLU activation function for each layer.

Reviewer comment (Novelty):
The analysis and the error rate of the neural network seems to lie in the regime of neural tangent kernel. Papers on neural tangent kernel should be cited properly. Here are a few examples: Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural tangent kernel: Convergence and generalization in neural networks." Advances in neural information processing systems 31 (2018).

Author response:
* We will add the stated reference in the related works section of our final paper.