Original: S INGLE -T IMESCALE A CTOR -C RITIC P ROVABLY F INDS G LOBALLY
Revised: S INGLE -T IMESCALE A CTOR -C RITIC P ROVABLY F INDS G LOBALLY

Original: O PTIMAL P OLICY
Revised: O PTIMAL P OLICY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms.
Revised: We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms.

Original: While most existing works on actor-critic employ bi-level or two-timescale updates, we focus on the more practical single-timescale setting, where the actor and critic are updated simultaneously.
Revised: While most existing works on actor-critic employ bi-level or two-timescale updates, we focus on the more practical single-timescale setting, where the actor and critic are updated simultaneously.

Original: Speciﬁcally, in each iteration, the critic update is obtained by applying the Bellman evaluation operator only once while the actor is updated in the policy gradient direction computed using the critic.
Revised: Speciﬁcally, in each iteration, the critic update is obtained by applying the Bellman evaluation operator only once while the actor is updated in the policy gradient direction computed using the critic.

Original: Moreover, we consider two function approximation settings where both the actor and critic are represented by linear or deep neural networks.
Revised: Moreover, we consider two function approximation settings where both the actor and critic are represented by linear or deep neural networks.

Original: For both cases, we prove that the actor sequence converges to a globally optimal policy at a sublinear
Revised: For both cases, we prove that the actor sequence converges to a globally optimal policy at a sublinear

Original: O
Revised: O

Original: ( K − 1 / 2 ) rate, where K is the number of iterations.
Revised: ( K − 1 / 2 ) rate, where K is the number of iterations.

Original: To the best of our knowledge, we establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the ﬁrst time.
Revised: To the best of our knowledge, we establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the ﬁrst time.

Original: Moreover, under the broader scope of policy optimization with nonlinear function approximation, we prove that actorcritic with deep neural network ﬁnds the globally optimal policy at a sublinear rate for the ﬁrst time.
Revised: Moreover, under the broader scope of policy optimization with nonlinear function approximation, we prove that actorcritic with deep neural network ﬁnds the globally optimal policy at a sublinear rate for the ﬁrst time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In reinforcement learning (RL) (Sutton et al., 1998), the agent aims to make sequential decisions that maximize the expected total reward through interacting with the environment and learning from the experiences, where the environment is modeled as a Markov Decision Process (MDP) (Puterman, 2014).
Revised: In reinforcement learning (RL) (Sutton et al., 1998), the agent aims to make sequential decisions that maximize the expected total reward through interacting with the environment and learning from the experiences, where the environment is modeled as a Markov Decision Process (MDP) (Puterman, 2014).

Original: To learn a policy that achieves the highest possible total reward in expectation, the actor-critic method (Konda and Tsitsiklis, 2000) is among the most commonly used algorithms.
Revised: To learn a policy that achieves the highest possible total reward in expectation, the actor-critic method (Konda and Tsitsiklis, 2000) is among the most commonly used algorithms.

Original: In actor-critic, the actor refers to the policy and the critic corresponds to the value function that characterizes the performance of the actor.
Revised: In actor-critic, the actor refers to the policy and the critic corresponds to the value function that characterizes the performance of the actor.

Original: This method directly optimizes the expected total return over the policy class by iteratively improving the actor, where the update direction is determined by the critic.
Revised: This method directly optimizes the expected total return over the policy class by iteratively improving the actor, where the update direction is determined by the critic.

Original: In particular, recently, actor-critic combined with deep neural networks (LeCun et al., 2015) achieves tremendous empirical successes in solving large-scale RL tasks, such as the game of Go (Silver et al., 2017), StarCraft (Vinyals et al., 2019), Dota (OpenAI, 2018), Rubik’s cube (Agostinelli et al., 2019; Akkaya et al., 2019), and autonomous driving (Sallab et al., 2017).
Revised: In particular, recently, actor-critic combined with deep neural networks (LeCun et al., 2015) achieves tremendous empirical successes in solving large-scale RL tasks, such as the game of Go (Silver et al., 2017), StarCraft (Vinyals et al., 2019), Dota (OpenAI, 2018), Rubik’s cube (Agostinelli et al., 2019; Akkaya et al., 2019), and autonomous driving (Sallab et al., 2017).

Original: See Li (2017) for a detailed survey of the recent developments of deep reinforcement learning.
Revised: See Li (2017) for a detailed survey of the recent developments of deep reinforcement learning.

Original: Despite these great empirical successes of actor-critic, there is still an evident chasm between theory and practice.
Revised: Despite these great empirical successes of actor-critic, there is still an evident chasm between theory and practice.

Original: Speciﬁcally, to establish convergence guarantees for actor-critic, most existing works either focus on the bi-level setting or the two-timescale setting, which are seldom adopted in practice.
Revised: Speciﬁcally, to establish convergence guarantees for actor-critic, most existing works either focus on the bi-level setting or the two-timescale setting, which are seldom adopted in practice.

Original: In particular, under the bi-level setting (Yang et al., 2019a; Wang et al., 2019; Agarwal et al., 2019;
Revised: In particular, under the bi-level setting (Yang et al., 2019a; Wang et al., 2019; Agarwal et al., 2019;

Original: Fu et al., 2019; Liu et al., 2019; Abbasi-Yadkori et al., 2019a;b; Cai et al., 2019; Hao et al., 2020;
Revised: Fu et al., 2019; Liu et al., 2019; Abbasi-Yadkori et al., 2019a;b; Cai et al., 2019; Hao et al., 2020;

Original: Mei et al., 2020; Bhandari and Russo, 2020), the actor is updated only after the critic solves the policy evaluation sub-problem completely, which is equivalent to applying the Bellman evaluation operator to the previous critic for inﬁnite times.
Revised: Mei et al., 2020; Bhandari and Russo, 2020), the actor is updated only after the critic solves the policy evaluation sub-problem completely, which is equivalent to applying the Bellman evaluation operator to the previous critic for inﬁnite times.

Original: Consequently, actor-critic under the bi-level setting is a double-loop iterative algorithm where the inner loop is allocated for solving the policy evaluation sub-problem of the critic.
Revised: Consequently, actor-critic under the bi-level setting is a double-loop iterative algorithm where the inner loop is allocated for solving the policy evaluation sub-problem of the critic.

Original: In terms of theoretical analysis, such a double-loop structure decouples the analysis for the actor and critic.
Revised: In terms of theoretical analysis, such a double-loop structure decouples the analysis for the actor and critic.

Original: For the actor, the problem is essentially reduced to analyzing the convergence of a variant of the policy gradient method (Sutton et al., 2000; Kakade, 2002) where the error of the gradient estimate depends on the policy evaluation error of the critic.
Revised: For the actor, the problem is essentially reduced to analyzing the convergence of a variant of the policy gradient method (Sutton et al., 2000; Kakade, 2002) where the error of the gradient estimate depends on the policy evaluation error of the critic.

Original: Besides, under the two-timescale setting (Borkar and Konda, 1997; Konda and Tsitsiklis, 2000; Xu et al., 2020;
Revised: Besides, under the two-timescale setting (Borkar and Konda, 1997; Konda and Tsitsiklis, 2000; Xu et al., 2020;

Original: Wu et al., 2020; Hong et al., 2020), the actor and the critic are updated simultaneously, but with disparate stepsizes.
Revised: Wu et al., 2020; Hong et al., 2020), the actor and the critic are updated simultaneously, but with disparate stepsizes.

Original: More concretely, the stepsize of the actor is set to be much smaller than that of the critic, with the ratio between these stepsizes converging to zero.
Revised: More concretely, the stepsize of the actor is set to be much smaller than that of the critic, with the ratio between these stepsizes converging to zero.

Original: In an asymptotic sense, such a separation between stepsizes ensures that the critic completely solves its policy evaluation sub-problem asymptotically.
Revised: In an asymptotic sense, such a separation between stepsizes ensures that the critic completely solves its policy evaluation sub-problem asymptotically.

Original: In other words, such a two-timescale scheme results in a separation between actor and critic in an asymptotic sense, which leads to asymptotically unbiased policy gradient estimates.
Revised: In other words, such a two-timescale scheme results in a separation between actor and critic in an asymptotic sense, which leads to asymptotically unbiased policy gradient estimates.

Original: In sum, in terms of convergence analysis, the existing theory of actor-critic hinges on decoupling the analysis for critic and actor, which is ensured via focusing on the bi-level or two-timescale settings.
Revised: In sum, in terms of convergence analysis, the existing theory of actor-critic hinges on decoupling the analysis for critic and actor, which is ensured via focusing on the bi-level or two-timescale settings.

Original: However, most practical implementations of actor-critic are under the single-timescale setting (Peters and Schaal, 2008a; Schulman et al., 2015; Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018), where the actor and critic are simultaneously updated, and particularly, the actor is updated without the critic reaching an approximate solution to the policy evaluation sub-problem.
Revised: However, most practical implementations of actor-critic are under the single-timescale setting (Peters and Schaal, 2008a; Schulman et al., 2015; Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018), where the actor and critic are simultaneously updated, and particularly, the actor is updated without the critic reaching an approximate solution to the policy evaluation sub-problem.

Original: Meanwhile, in comparison with the two-timescale setting, the actor is equipped with a much larger stepsize in the the single-timescale setting such that the asymptotic separation between the analysis of actor and critic is no longer valid.
Revised: Meanwhile, in comparison with the two-timescale setting, the actor is equipped with a much larger stepsize in the the single-timescale setting such that the asymptotic separation between the analysis of actor and critic is no longer valid.

Original: Furthermore, when it comes to function approximation, most existing works only analyze the convergence of actor-critic with either linear function approximation (Xu et al., 2020; Wu et al., 2020;
Revised: Furthermore, when it comes to function approximation, most existing works only analyze the convergence of actor-critic with either linear function approximation (Xu et al., 2020; Wu et al., 2020;

Original: Hong et al., 2020), or shallow-neural-network parameterization (Wang et al., 2019; Liu et al., 2019).
Revised: Hong et al., 2020), or shallow-neural-network parameterization (Wang et al., 2019; Liu et al., 2019).

Original: In contrast, practically used actor-critic methods such as asynchronous advantage actor-critic (Mnih et al., 2016) and soft actor-critic (Haarnoja et al., 2018) oftentimes represent both the actor and critic using deep neural networks.
Revised: In contrast, practically used actor-critic methods such as asynchronous advantage actor-critic (Mnih et al., 2016) and soft actor-critic (Haarnoja et al., 2018) oftentimes represent both the actor and critic using deep neural networks.

Original: Thus, the following question is left open:
Revised: Thus, the following question is left open:

Original: Does single-timescale actor-critic provably ﬁnd a globally optimal policy under the function approximation setting, especially when deep neural networks are employed?
Revised: Does single-timescale actor-critic provably ﬁnd a globally optimal policy under the function approximation setting, especially when deep neural networks are employed?

Original: To answer such a question, we make the ﬁrst attempt to investigate the convergence and global optimality of single-timescale actor-critic with linear and neural network function approximation.
Revised: To answer such a question, we make the ﬁrst attempt to investigate the convergence and global optimality of single-timescale actor-critic with linear and neural network function approximation.

Original: In particular, we focus on the family of energy-based policies and aim to ﬁnd the optimal policy within this class.
Revised: In particular, we focus on the family of energy-based policies and aim to ﬁnd the optimal policy within this class.

Original: Here we represent both the energy function and the critic as linear or deep neural network functions.
Revised: Here we represent both the energy function and the critic as linear or deep neural network functions.

Original: In our actor-critic algorithm, the actor update follows proximal policy optimization (PPO) (Schulman et al., 2017) and the critic update is obtained by applying the Bellman evaluation operator only once to the current critic iterate.
Revised: In our actor-critic algorithm, the actor update follows proximal policy optimization (PPO) (Schulman et al., 2017) and the critic update is obtained by applying the Bellman evaluation operator only once to the current critic iterate.

Original: As a result, the actor is updated before the critic solves the policy evaluation sub-problem.
Revised: As a result, the actor is updated before the critic solves the policy evaluation sub-problem.

Original: Such a coupled updating structure persists even when the number of iterations goes to inﬁnity, which implies that the update direction of the actor is always biased compared with the policy gradient direction.
Revised: Such a coupled updating structure persists even when the number of iterations goes to inﬁnity, which implies that the update direction of the actor is always biased compared with the policy gradient direction.

Original: This brings an additional challenge that is absent in the bi-level and the two-timescale settings, where the actor and critic are decoupled asymptotically.
Revised: This brings an additional challenge that is absent in the bi-level and the two-timescale settings, where the actor and critic are decoupled asymptotically.

Original: To tackle such a challenge, our analysis captures the joint effect of actor and critic updates on the objective function, dubbed as the “double contraction” phenomenon, which plays a pivotal role for the success of single-timescale actor-critic.
Revised: To tackle such a challenge, our analysis captures the joint effect of actor and critic updates on the objective function, dubbed as the “double contraction” phenomenon, which plays a pivotal role for the success of single-timescale actor-critic.

Original: Speciﬁcally, thanks to the discount factor of the MDP, the Bellman evaluation operator is contractive, which implies that, after each update, the critic makes noticeable progress by moving towards the value function associated with the current actor.
Revised: Speciﬁcally, thanks to the discount factor of the MDP, the Bellman evaluation operator is contractive, which implies that, after each update, the critic makes noticeable progress by moving towards the value function associated with the current actor.

Original: As a result, although we use a biased estimate of the policy gradient, thanks to the contraction brought by the discount factor, the accumulative effect of the biases is controlled.
Revised: As a result, although we use a biased estimate of the policy gradient, thanks to the contraction brought by the discount factor, the accumulative effect of the biases is controlled.

Original: Such a phenomenon enables us to characterize the progress of each iteration of joint actor and critic update, and thus yields the convergence to the globally optimal policy.
Revised: Such a phenomenon enables us to characterize the progress of each iteration of joint actor and critic update, and thus yields the convergence to the globally optimal policy.

Original: In particular, for both the linear and neural settings, we prove that, single-timescale actor-critic ﬁnds a O ( K − 1 / 2 ) -globally optimal policy after
Revised: In particular, for both the linear and neural settings, we prove that, single-timescale actor-critic ﬁnds a O ( K − 1 / 2 ) -globally optimal policy after

Original: K iterations.
Revised: K iterations.

Original: To the best of our knowledge, we seem to establish the ﬁrst theoretical guarantee of global convergence and global optimality for actor-critic with function approximation in the singletimescale setting.
Revised: To the best of our knowledge, we seem to establish the ﬁrst theoretical guarantee of global convergence and global optimality for actor-critic with function approximation in the singletimescale setting.

Original: Moreover, under the broader scope of policy optimization with nonlinear function approximation, our work seems to prove convergence and optimality guarantees for actor-critic with deep neural network for the ﬁrst time.
Revised: Moreover, under the broader scope of policy optimization with nonlinear function approximation, our work seems to prove convergence and optimality guarantees for actor-critic with deep neural network for the ﬁrst time.

Original: Contribution.
Revised: Contribution.

Original: Our contribution is two-fold.
Revised: Our contribution is two-fold.

Original: First, in the single-timescale setting with linear function approximation, we prove that, after K iterations of actor and critic updates, actor-critic returns a policy that is at most O ( K − 1 / 2 ) inferior to the globally optimal policy.
Revised: First, in the single-timescale setting with linear function approximation, we prove that, after K iterations of actor and critic updates, actor-critic returns a policy that is at most O ( K − 1 / 2 ) inferior to the globally optimal policy.

Original: Second, when both the actor and critic are represented by deep neural networks, we prove a similar O ( K − 1 / 2 ) rate of convergence to the globally optimal policy when the architecture of the neural networks are properly chosen.
Revised: Second, when both the actor and critic are represented by deep neural networks, we prove a similar O ( K − 1 / 2 ) rate of convergence to the globally optimal policy when the architecture of the neural networks are properly chosen.

Original: Related Work.
Revised: Related Work.

Original: Our work extends the line of works on the convergence of actor-critic under the function approximation setting.
Revised: Our work extends the line of works on the convergence of actor-critic under the function approximation setting.

Original: In particular, actor-critic is ﬁrst introduced in Sutton et al.
Revised: In particular, actor-critic is ﬁrst introduced in Sutton et al.

Original: Konda and Tsitsiklis (2000).
Revised: Konda and Tsitsiklis (2000).

Original: Later, Kakade (2002); Peters and Schaal (2008b) propose the natural actor-critic method which updates the policy via the natural gradient (Amari, 1998) direction.
Revised: Later, Kakade (2002); Peters and Schaal (2008b) propose the natural actor-critic method which updates the policy via the natural gradient (Amari, 1998) direction.

Original: The convergence of (natural) actor-critic with linear function approximation are studied in Bhatnagar et al.
Revised: The convergence of (natural) actor-critic with linear function approximation are studied in Bhatnagar et al.

Original: (2008; 2009); Bhatnagar (2010); Castro and Meir (2010); Maei (2018).
Revised: (2008; 2009b); Bhatnagar (2010); Castro and Meir (2010); Maei (2018).

Original: However, these works only characterize the asymptotic convergence of actor-critic and their proofs all resort to tools from stochastic approximation via ordinary differential equations (Borkar, 2008).
Revised: However, these works only characterize the asymptotic convergence of actor-critic and their proofs all resort to tools from stochastic approximation via ordinary differential equations (Borkar, 2008).

Original: As a result, these works only show that actor-critic with linear function approximation converges to the set of stable equilibria of a set of ordinary differential equations.
Revised: As a result, these works only show that actor-critic with linear function approximation converges to the set of stable equilibria of a set of ordinary differential equations.

Original: Recently, Zhang et al.
Revised: Recently, Zhang et al.

Original: (2019) propose a variant of actorcritic where Monte-Carlo sampling is used to ensure the critic and the policy gradient estimates are unbiased.
Revised: (2019) propose a variant of actor-critic where Monte-Carlo sampling is used to ensure the critic and the policy gradient estimates are unbiased.

Original: Although they incorporate nonlinear function approximation in the actor, they only establish ﬁnite-time convergence result to a stationary point of the expected total reward.
Revised: Although they incorporate nonlinear function approximation in the actor, they only establish ﬁnite-time convergence result to a stationary point of the expected total reward.

Original: Moreover, due to having an inner loop for solving the policy evaluation sub-problem, they focus on the bi-level setting.
Revised: Moreover, due to having an inner loop for solving the policy evaluation sub-problem, they focus on the bi-level setting.

Original: Moreover, under the two-timescale setting, Wu et al.
Revised: Moreover, under the two-timescale setting, Wu et al.

Original: Xu et al.
Revised: Xu et al.

Original: (2020) show that actorcritic with linear function approximation ﬁnds an ε -stationary point with (cid:101)
Revised: (2020) show that actor-critic with linear function approximation ﬁnds an ε -stationary point with (cid:101)

Original: O ( ε − 5 / 2 ) samples, where ε measures the squared norm of the policy gradient.
Revised: O ( ε − 5 / 2 ) samples, where ε measures the squared norm of the policy gradient.

Original: All of these results establish the convergence of actor-critic, without characterizing the optimality of the policy obtained by actor-critic.
Revised: All of these results establish the convergence of actor-critic, without characterizing the optimality of the policy obtained by actor-critic.

Original: In terms of the global optimality of actor-critic, Fazel et al.
Revised: In terms of the global optimality of actor-critic, Fazel et al.

Original: ; Malik et al.
Revised: ; Malik et al.

Original: Tu and
Revised: Tu and

Original: Recht (2018); Yang et al. (2019a); Bu et al.
Revised: Recht (2018); Yang et al. (2019a); Bu et al.

Original: Fu et al.
Revised: Fu et al.

Original: (2019) show that policy gradient and bi-level actor-critic methods converge to the globally optimal policies under the linear-quadratic setting, where the state transitions follow a linear dynamical system and the reward function is quadratic.
Revised: (2019) show that policy gradient and bi-level actor-critic methods converge to the globally optimal policies under the linear-quadratic setting, where the state transitions follow a linear dynamical system and the reward function is quadratic.

Original: For general MDPs, Bhandari and Russo (2019) recently prove the global optimality of vanilla policy gradient under the assumption that the families of policies and value functions are both convex.
Revised: For general MDPs, Bhandari and Russo (2019) recently prove the global optimality of vanilla policy gradient under the assumption that the families of policies and value functions are both convex.

Original: In addition, our work is also related to Liu et al.
Revised: In addition, our work is also related to Liu et al.

Original: (2019) and Wang et al.
Revised: (2019) and Wang et al.

Original: (2019), where they establish the global optimality of proximal policy optimization and (natural) actor-critic, respectively, where both the actor and critic are parameterized by two-layer neural networks.
Revised: (2019), where they establish the global optimality of proximal policy optimization and (natural) actor-critic, respectively, where both the actor and critic are parameterized by two-layer neural networks.

Original: Our work is also related to Agarwal et al.
Revised: Our work is also related to Agarwal et al.

Original: Abbasi-Yadkori et al.
Revised: Abbasi-Yadkori et al.

Original: (2019a;b);
Revised: (2019a;b);

Original: Cai et al.
Revised: Cai et al.

Original: Hao et al.
Revised: Hao et al.

Original: Mei et al.
Revised: Mei et al.

Original: (2020); Bhandari and Russo (2020), which focus on characterizing the optimality of natural policy gradient in tabular and/or linear settings.
Revised: (2020); Bhandari and Russo (2020), which focus on characterizing the optimality of natural policy gradient in tabular and/or linear settings.

Original: However, these aforementioned works all focus on bi-level actor-critic, where the actor is updated only after the critic solves the policy evaluation sub-problem to an approximate optimum.
Revised: However, these aforementioned works all focus on bi-level actor-critic, where the actor is updated only after the critic solves the policy evaluation sub-problem to an approximate optimum.

Original: Besides, these works consider linear or two-layer neural network function approximations whereas we focus on the setting with deep neural networks.
Revised: Besides, these works consider linear or two-layer neural network function approximations whereas we focus on the setting with deep neural networks.

Original: Furthermore, under the two-timescale setting, Xu et al.
Revised: Furthermore, under the two-timescale setting, Xu et al.

Original: (2020); Hong et al.
Revised: (2020); Hong et al.

Original: (2020) prove that linear actor-critic requires a sample complexity of (cid:101)
Revised: (2020) prove that linear actor-critic requires a sample complexity of (cid:101)

Original: O
Revised: O

Original: ( ε − 4 ) for obtaining an ε -globally optimal policy.
Revised: ( ε − 4 ) for obtaining an ε -globally optimal policy.

Original: In comparison, our O ( K − 1 / 2 ) convergence for single-timescale actor-critic can be translated into a similar (cid:101) O
Revised: In comparison, our O ( K − 1 / 2 ) convergence for single-timescale actor-critic can be translated into a similar (cid:101) O

Original: ( ε − 4 ) sample complexity directly.
Revised: ( ε − 4 ) sample complexity directly.

Original: Moreover, when reusing the data, our result leads to an improved (cid:101) O ( ε − 2 ) sample complexity.
Revised: Moreover, when reusing the data, our result leads to an improved (cid:101) O ( ε − 2 ) sample complexity.

Original: In addition, our work is also related to
Revised: In addition, our work is also related to

Original: Geist et al.
Revised: Geist et al.

Original: (2019), which proposes a variant of policy iteration algorithm with Bregman divergence regularization.
Revised: (2019), which proposes a variant of policy iteration algorithm with Bregman divergence regularization.

Original: Without considering an explicit form of function approximation, their algorithm is shown to converge to the globally optimal policy at a similar O ( K − 1 / 2 ) rate, where K is the number of policy updates.
Revised: Without considering an explicit form of function approximation, their algorithm is shown to converge to the globally optimal policy at a similar O ( K − 1 / 2 ) rate, where K is the number of policy updates.

Original: In contrast, our method is single-timescale actor-critic with linear or deep neural network function approximation, which enjoys both global convergence and global optimality.
Revised: In contrast, our method is single-timescale actor-critic with linear or deep neural network function approximation, which enjoys both global convergence and global optimality.

Original: Meanwhile, our proof is based on a ﬁnite-sample analysis, which involves dealing with the algorithmic errors that track the performance of actor and critic updates as well as the statistical error due to having ﬁnite data.
Revised: Meanwhile, our proof is based on a ﬁnite-sample analysis, which involves dealing with the algorithmic errors that track the performance of actor and critic updates as well as the statistical error due to having ﬁnite data.

Original: Our work is also related to the literature on deep neural networks.
Revised: Our work is also related to the literature on deep neural networks.

Original: Previous works (Daniely, 2017;
Revised: Previous works (Daniely, 2017;

Original: Jacot et al., 2018; Wu et al., 2018; Allen-Zhu et al., 2018a;b; Du et al., 2018; Zou et al., 2018; Chizat and Bach, 2018;
Revised: Jacot et al., 2018; Wu et al., 2018; Allen-Zhu et al., 2018a;b; Du et al., 2018; Zou et al., 2018; Chizat and Bach, 2018;

Original: Jacot et al., 2018; Li and Liang, 2018; Cao and Gu, 2019a;b; Arora et al., 2019; Lee et al., 2019; Gao et al., 2019) analyze the computational and statistical rates of supervised learning methods with overparameterized neural networks.
Revised: Jacot et al., 2018; Li and Liang, 2018; Cao and Gu, 2019a;b; Arora et al., 2019; Lee et al., 2019; Gao et al., 2019) analyze the computational and statistical rates of supervised learning methods with overparameterized neural networks.

Original: In contrast, our work employs overparameterized deep neural networks in actor-critic for solving RL tasks, which is signiﬁcantly more challenging than supervised learning due to the interplay between the actor and the critic.
Revised: In contrast, our work employs overparameterized deep neural networks in actor-critic for solving RL tasks, which is signiﬁcantly more challenging than supervised learning due to the interplay between the actor and the critic.

Original: Notation.
Revised: Notation.

Original: We denote by [ n ] the set { 1 , 2 , . . . , n } .
Revised: We denote by [ n ] the set { 1 , 2 , . . . , n } .

Original: For any measure ν and 1 ≤ p ≤ ∞ , we denote by (cid:107) f (cid:107) ν,p =
Revised: For any measure ν and 1 ≤ p ≤ ∞ , we denote by (cid:107) f (cid:107) ν,p =

Original: ( (cid:82) X | f ( x ) | p d ν ) 1 /p
Revised: ( (cid:82) X | f ( x ) | p d ν ) 1 /p

Original: and (cid:107) f (cid:107) p
Revised: and (cid:107) f (cid:107) p

Original: = ( (cid:82) X | f ( x ) | p d µ ) 1 /p , where µ is the Lebesgue measure.
Revised: = ( (cid:82) X | f ( x ) | p d µ ) 1 /p , where µ is the Lebesgue measure.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we introduce the background on discounted Markov decision processes (MDPs) and actor-critic methods.
Revised: In this section, we introduce the background on discounted Markov decision processes (MDPs) and actor-critic methods.

Original: 2. D ISCOUNTED MDP
Revised: 2. D ISCOUNTED MDP

Original: A discounted MDP is deﬁned by a tuple ( S , A , P, ζ, r, γ ) .
Revised: A discounted MDP is deﬁned by a tuple ( S , A , P, ζ, r, γ ) .

Original: Here S and A are the state and action spaces, respectively, P : S × S ×
Revised: Here S and A are the state and action spaces, respectively, P : S × S ×

Original: A → [0 , 1] is the Markov transition kernel, ζ : S → [0 , 1] is the initial state distribution, r : S ×
Revised: A → [0 , 1] is the Markov transition kernel, ζ : S → [0 , 1] is the initial state distribution, r : S ×

Original: A → R is the deterministic reward function, and γ ∈ [0 , 1) is the discount factor.
Revised: A → R is the deterministic reward function, and γ ∈ [0 , 1) is the discount factor.

Original: A policy π ( a | s ) measures the probability of taking the action a at the state s .
Revised: A policy π ( a | s ) measures the probability of taking the action a at the state s .

Original: We focus on a family of parameterized policies deﬁned as follows,
Revised: We focus on a family of parameterized policies deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: where P ( A ) is the probability simplex on the action space A and θ is the parameter of the policy π θ .
Revised: where P ( A ) is the probability simplex on the action space A and θ is the parameter of the policy π θ .

Original: For any state-action pair ( s, a ) ∈ S × A , we deﬁne the action-value function as follows,
Revised: For any state-action pair ( s, a ) ∈ S × A , we deﬁne the action-value function as follows,

Original: [Equation]
Revised: [Equation]

Original: where s t +1 ∼ P ( · | s t , a t ) and a t +1 ∼ π ( · | s t +1 ) for any t ≥ 0 .
Revised: where s t +1 ∼ P ( · | s t , a t ) and a t +1 ∼ π ( · | s t +1 ) for any t ≥ 0 .

Original: We use E π [ · ] to denote that the actions follow the policy π , which further affect the transition of the states.
Revised: We use E π [ · ] to denote that the actions follow the policy π , which further affect the transition of the states.

Original: We aim to ﬁnd an optimal policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) for any policy π and state-action pair ( s, a ) ∈ S × A .
Revised: We aim to ﬁnd an optimal policy π ∗ such that Q π ∗ ( s, a ) ≥ Q π ( s, a ) for any policy π and state-action pair ( s, a ) ∈ S × A .

Original: That is to say, such an optimal policy π ∗ attains a higher expected total reward than any other policy π , regardless of the initial state-action pair ( s, a ) .
Revised: That is to say, such an optimal policy π ∗ attains a higher expected total reward than any other policy π , regardless of the initial state-action pair ( s, a ) .

Original: For notational convenience, we denote by
Revised: For notational convenience, we denote by

Original: Q ∗ ( s, a ) =
Revised: Q ∗ ( s, a ) =

Original: Q π ∗ ( s, a ) for any ( s, a ) ∈ S ×
Revised: Q π ∗ ( s, a ) for any ( s, a ) ∈ S ×

Original: A hereafter.
Revised: A hereafter.

Original: Meanwhile, we denote by ν π ( s ) and ρ π ( s, a ) =
Revised: Meanwhile, we denote by ν π ( s ) and ρ π ( s, a ) =

Original: ν
Revised: ν

Original: π ( s ) ·
Revised: π ( s ) ·

Original: π ( a | s ) the stationary state distribution and stationary state-action distribution of the policy π , respectively, for any ( s, a ) ∈ S ×A .
Revised: π ( a | s ) the stationary state distribution and stationary state-action distribution of the policy π , respectively, for any ( s, a ) ∈ S ×A .

Original: Correspondingly, we denote by ν ∗ ( s ) and ρ ∗ ( s, a ) the stationary state distribution and stationary state-action distribution of the optimal policy π ∗ , respectively, for any ( s, a ) ∈ S × A .
Revised: Correspondingly, we denote by ν ∗ ( s ) and ρ ∗ ( s, a ) the stationary state distribution and stationary state-action distribution of the optimal policy π ∗ , respectively, for any ( s, a ) ∈ S × A .

Original: For ease of presentation, given any functions g 1 : S → R and g 2 : S ×
Revised: For ease of presentation, given any functions g 1 : S → R and g 2 : S ×

Original: A → R , we deﬁne two operators P and P π as follows,
Revised: A → R , we deﬁne two operators P and P π as follows,

Original: [Equation]
Revised: [Equation]

Original: where s 1 ∼ P ( · | s 0 , a 0 ) and a 1 ∼ π ( · | s 1 ) .
Revised: where s 1 ∼ P ( · | s 0 , a 0 ) and a 1 ∼ π ( · | s 1 ) .

Original: Intuitively, given the current state-action pair ( s 0 , a 0 ) , the operator P pushes the agent to its next state s 1 following the Markov transition kernel
Revised: Intuitively, given the current state-action pair ( s 0 , a 0 ) , the operator P pushes the agent to its next state s 1 following the Markov transition kernel

Original: P ( · | s 0 , a 0 ) , while the operator P π pushes the agent to its next state-action pair ( s 1 , a 1 ) following the Markov transition kernel P ( · | s 0 , a 0 ) and policy π ( · | s 1 ) .
Revised: P ( · | s 0 , a 0 ) , while the operator P π pushes the agent to its next state-action pair ( s 1 , a 1 ) following the Markov transition kernel P ( · | s 0 , a 0 ) and policy π ( · | s 1 ) .

Original: These operators also relate to the
Revised: These operators also relate to the

Original: Bellman evaluation operator T π , which is deﬁned for any function g : S ×
Revised: Bellman evaluation operator T π , which is deﬁned for any function g : S ×

Original: A → R as follows,
Revised: A → R as follows,

Original: [Equation]
Revised: [Equation]

Original: The Bellman evaluation operator T π is used to characterize the actor-critic method in the following section.
Revised: The Bellman evaluation operator T π is used to characterize the actor-critic method in the following section.

Original: By the deﬁnition in (2.2), it is straightforward to verify that the action-value function Q π is the ﬁxed point of the Bellman evaluation operator T π deﬁned in (2.4), that is, Q π
Revised: By the deﬁnition in (2.2), it is straightforward to verify that the action-value function Q π is the ﬁxed point of the Bellman evaluation operator T π deﬁned in (2.4), that is, Q π

Original: = T π Q π for any policy π .
Revised: = T π Q π for any policy π .

Original: For notational convenience, we let P (cid:96) denote the (cid:96) -fold composition PP · · · P (cid:124) (cid:123)(cid:122) (cid:125) (cid:96) .
Revised: For notational convenience, we let P (cid:96) denote the (cid:96) -fold composition PP · · · P (cid:124) (cid:123)(cid:122) (cid:125) (cid:96) .

Original: Such notation is also adopted for other linear operators such as P π and T π .
Revised: Such notation is also adopted for other linear operators such as P π and T π .

Original: 2. A CTOR -C RITIC M ETHOD
Revised: 2. A CTOR -C RITIC M ETHOD

Original: To obtain an optimal policy π ∗ , the actor-critic method (Konda and Tsitsiklis, 2000) aims to maximize the expected total reward as a function of the policy, which is equivalent to solving the following maximization problem,
Revised: To obtain an optimal policy π ∗ , the actor-critic method (Konda and Tsitsiklis, 2000) aims to maximize the expected total reward as a function of the policy, which is equivalent to solving the following maximization problem,

Original: [Equation]
Revised: [Equation]

Original: where ζ is the initial state distribution, Q π is the action-value function deﬁned in (2.2), and the family of parameterized polices Π is deﬁned in (2.1).
Revised: where ζ is the initial state distribution, Q π is the action-value function deﬁned in (2.2), and the family of parameterized polices Π is deﬁned in (2.1).

Original: The actor-critic method solves the maximization problem in (2.5) via ﬁrst-order optimization using an estimator of the policy gradient ∇ θ J ( π ) .
Revised: The actor-critic method solves the maximization problem in (2.5) via ﬁrst-order optimization using an estimator of the policy gradient ∇ θ J ( π ) .

Original: Here θ is the parameter of the policy π .
Revised: Here θ is the parameter of the policy π .

Original: In detail, by the policy gradient theorem (Sutton et al., 2000), we have
Revised: In detail, by the policy gradient theorem (Sutton et al., 2000), we have

Original: [Equation]
Revised: [Equation]

Original: Here (cid:37) π is the state-action visitation measure of the policy π , which is deﬁned as (cid:37) π ( s, a ) =
Revised: Here (cid:37) π is the state-action visitation measure of the policy π , which is deﬁned as (cid:37) π ( s, a ) =

Original: (1 − γ ) ·
Revised: (1 − γ ) ·

Original: (cid:80) ∞ t =0 γ t · Pr[ s t = s, a t = a ] .
Revised: (cid:80) ∞ t =0 γ t · Pr[ s t = s, a t = a ] .

Original: Based on the closed form of the policy gradient in (2.6), the actor-critic method consists of the following two parts: (i) the critic update, where a policy evaluation algorithm is invoked to estimate the action-value function Q π , e.g., by applying the Bellman evaluation operator T π to the current estimator of Q π , and (ii) the actor update, where a policy improvement algorithm, e.g., the policy gradient method, is invoked using the updated estimator of
Revised: Based on the closed form of the policy gradient in (2.6), the actor-critic method consists of the following two parts: (i) the critic update, where a policy evaluation algorithm is invoked to estimate the action-value function Q π , e.g., by applying the Bellman evaluation operator T π to the current estimator of Q π , and (ii) the actor update, where a policy improvement algorithm, e.g., the policy gradient method, is invoked using the updated estimator of

Original: Q π .
Revised: Q π .

Original: In this paper, we consider the following variant of the actor-critic method,
Revised: In this paper, we consider the following variant of the actor-critic method,

Original: [Equation]
Revised: [Equation]

Original: ) + γ · Q k ( s 1 , a 1 ) (cid:12) (cid:12) s 

Revised: ) + γ · Q k ( s 1 , a 1 ) (cid:12) (cid:12) s

Original: = s, a 0 = a (cid:3) , (2.7) for any ( s, a ) ∈ S × A , where s 1 ∼ P ( · | s 0 , a 0 ) , a 1 ∼ π k +1 ( · | s 1 ) , and we write E ν πk [ · ] =
Revised: = s, a 0 = a (cid:3) , (2.7) for any ( s, a ) ∈ S × A , where s 1 ∼ P ( · | s 0 , a 0 ) , a 1 ∼ π k +1 ( · | s 1 ) , and we write E ν πk [ · ] =

Original: E s ∼ ν πk [ · ] for notational convenience.
Revised: E s ∼ ν πk [ · ] for notational convenience.

Original: Here Π is deﬁned in (2.1) and KL ( π ( · | s ) (cid:107) π k ( · | s )) is the
Revised: Here Π is deﬁned in (2.1) and KL ( π ( · | s ) (cid:107) π k ( · | s )) is the

Original: Kullback-Leibler (KL) divergence between π ( · | s ) and π k ( · | s ) , which is deﬁned for any s ∈ S as follows, KL ( π ( · | s ) (cid:107) π k ( · | s ))
Revised: Kullback-Leibler (KL) divergence between π ( · | s ) and π k ( · | s ) , which is deﬁned for any s ∈ S as follows, KL ( π ( · | s ) (cid:107) π k ( · | s ))

Original: = (cid:80) a ∈A log( π ( a | s ) /π
Revised: = (cid:80) a ∈A log( π ( a | s ) /π

Original: k ( a | s ))
Revised: k ( a | s ))

Original: · π ( a | s ) .
Revised: · π ( a | s ) .

Original: In (2.7), the actor update uses the proximal policy optimization (PPO) method (Schulman et al., 2017), while the critic update applies the Bellman evaluation operator T π k +1 deﬁned in (2.4) to Q k only once, which is the current estimator of the action-value function.
Revised: In (2.7), the actor update uses the proximal policy optimization (PPO) method (Schulman et al., 2017), while the critic update applies the Bellman evaluation operator T π k +1 deﬁned in (2.4) to Q k only once, which is the current estimator of the action-value function.

Original: Furthermore, we remark that the updates in (2.7) provide a general framework in the following two aspects.
Revised: Furthermore, we remark that the updates in (2.7) provide a general framework in the following two aspects.

Original: First, the critic update can be extended to letting
Revised: First, the critic update can be extended to letting

Original: Q k +1 ← ( T π k +1 )
Revised: Q k +1 ← ( T π k +1 )

Original: τ Q k for any ﬁxed τ ≥ 1 , which corresponds to updating the value function via τstep rollouts following π k +1 .
Revised: τ Q k for any ﬁxed τ ≥ 1 , which corresponds to updating the value function via τstep rollouts following π k +1 .

Original: Here we only focus on the case with τ = 1 for simplicity.
Revised: Here we only focus on the case with τ = 1 for simplicity.

Original: Our theory can be easily modiﬁed for any ﬁxed τ .
Revised: Our theory can be easily modiﬁed for any ﬁxed τ .

Original: Moreover, the KL divergence used in the actor step can also be replaced by other Bregman divergences between probability distributions over A .
Revised: Moreover, the KL divergence used in the actor step can also be replaced by other Bregman divergences between probability distributions over A .

Original: Second, the actor and critic updates in (2.7) is a general template that admits both on- and off-policy evaluation methods and various function approximators in the actor and critic.
Revised: Second, the actor and critic updates in (2.7) is a general template that admits both on- and off-policy evaluation methods and various function approximators in the actor and critic.

Original: In the next section, we present an incarnation of (2.7) with on-policy sampling and linear and neural network function approximation.
Revised: In the next section, we present an incarnation of (2.7) with on-policy sampling and linear and neural network function approximation.

Original: Furthermore, for analyzing the actor-critic method, most existing works (Yang et al., 2019a; Wang et al., 2019; Agarwal et al., 2019; Fu et al., 2019; Liu et al., 2019) rely on (approximately) obtaining
Revised: Furthermore, for analyzing the actor-critic method, most existing works (Yang et al., 2019a; Wang et al., 2019; Agarwal et al., 2019; Fu et al., 2019; Liu et al., 2019) rely on (approximately) obtaining

Original: Q π k +1 at each iteration, which is equivalent to applying the Bellman evaluation operator T π k + inﬁnite times to Q k .
Revised: Q π k +1 at each iteration, which is equivalent to applying the Bellman evaluation operator T π k + inﬁnite times to Q k .

Original: This is usually achieved by minimizing the mean-squared Bellman error (cid:107) Q − T π k +1 Q (cid:107)ρ πk +1 , 2 using stochastic semi-gradient descent, e.g., as in the temporal-difference method (Sutton, 1988), to update the critic for sufﬁciently many iterations.
Revised: This is usually achieved by minimizing the mean-squared Bellman error (cid:107) Q − T π k +1 Q (cid:107)ρ πk +1 , 2 using stochastic semi-gradient descent, e.g., as in the temporal-difference method (Sutton, 1988), to update the critic for sufﬁciently many iterations.

Original: The unique global minimizer of the mean-squared Bellman error gives the action-value function Q π k +1 , which is used in the actor update.
Revised: The unique global minimizer of the mean-squared Bellman error gives the action-value function Q π k +1 , which is used in the actor update.

Original: Meanwhile, the two-timescale setting is also considered in existing works (Borkar and Konda, 1997; Konda and Tsitsiklis, 2000; Xu et al., 2019; 2020; Wu et al., 2020; Hong et al., 2020), which require the actor to be updated more slowly than the critic in an asymptotic sense.
Revised: Meanwhile, the two-timescale setting is also considered in existing works (Borkar and Konda, 1997; Konda and Tsitsiklis, 2000; Xu et al., 2019; 2020; Wu et al., 2020; Hong et al., 2020), which require the actor to be updated more slowly than the critic in an asymptotic sense.

Original: Such a requirement is usually satisﬁed by forcing the ratio between the stepsizes of the actor and critic updates to go to zero asymptotically.
Revised: Such a requirement is usually satisﬁed by forcing the ratio between the stepsizes of the actor and critic updates to go to zero asymptotically.

Original: In comparison with the setting with bi-level updates, we consider the single-timescale actor and critic updates in (2.7), where the critic involves only one step of update, that is, applying the Bellman evaluation operator T π to Q k only once.
Revised: In comparison with the setting with bi-level updates, we consider the single-timescale actor and critic updates in (2.7), where the critic involves only one step of update, that is, applying the Bellman evaluation operator T π to Q k only once.

Original: Meanwhile, in comparison with the two-timescale setting, where the actor and critic are updated simultaneously but with the ratio between their stepsizes asymptotically going to zero, the single-timescale setting is able to achieve a faster rate of convergence by allowing the actor to be updated with a larger stepsize, while updating the critic simultaneously.
Revised: Meanwhile, in comparison with the two-timescale setting, where the actor and critic are updated simultaneously but with the ratio between their stepsizes asymptotically going to zero, the single-timescale setting is able to achieve a faster rate of convergence by allowing the actor to be updated with a larger stepsize, while updating the critic simultaneously.

Original: In particular, such a single-timescale setting better captures a broader range of practical algorithms (Peters and Schaal, 2008a; Schulman et al., 2015; Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018), where the stepsize of the actor is not asymptotically zero.
Revised: In particular, such a single-timescale setting better captures a broader range of practical algorithms (Peters and Schaal, 2008a; Schulman et al., 2015; Mnih et al., 2016; Schulman et al., 2017; Haarnoja et al., 2018), where the stepsize of the actor is not asymptotically zero.

Original: In § 3, we discuss the implementation of the updates in (2.7) for different schemes of function approximation.
Revised: In § 3, we discuss the implementation of the updates in (2.7) for different schemes of function approximation.

Original: In § 4, we compare the rates of convergence between the two-timescale and single-timescale settings.
Revised: In § 4, we compare the rates of convergence between the two-timescale and single-timescale settings.

Original:  A LGORITHMS
Revised:  A LGORITHMS

Original: We consider two settings, where the actor and critic are parameterized using linear functions and deep neural networks (which is deferred to § A of the appendix), respectively.
Revised: We consider two settings, where the actor and critic are parameterized using linear functions and deep neural networks (which is deferred to § A of the appendix), respectively.

Original: We consider the energy-based policy π θ ( a | s ) ∝ exp( τ − 1 f θ ( s, a )) , where the energy function f θ ( s, a ) is parameterized with the parameter θ .
Revised: We consider the energy-based policy π θ ( a | s ) ∝ exp( τ − 1 f θ ( s, a )) , where the energy function f θ ( s, a ) is parameterized with the parameter θ .

Original: Also, for the (estimated) action-value function, we consider the parameterization Q ω ( s, a ) for any ( s, a ) ∈ S × A , where ω is the parameter.
Revised: Also, for the (estimated) action-value function, we consider the parameterization Q ω ( s, a ) for any ( s, a ) ∈ S × A , where ω is the parameter.

Original: For such parameterizations of the actor and critic, the updates in (2.7) have the following forms.
Revised: For such parameterizations of the actor and critic, the updates in (2.7) have the following forms.

Original: Actor Update.
Revised: Actor Update.

Original: The following proposition gives the closed form of π k +1 in (2.7).
Revised: The following proposition gives the closed form of π k +1 in (2.7).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Let π θ k
Revised: Let π θ k

Original: ( a | s ) ∝ exp( τ − 1 k f θ k
Revised: ( a | s ) ∝ exp( τ − 1 k f θ k

Original: ( s, a )) be an energy-based policy and (cid:101) π k +

Revised: ( s, a )) be an energy-based policy and (cid:101) π k +

Original: =
Revised: =

Original: argmax π E ν k (cid:2) (cid:104) Q ω k
Revised: argmax π E ν k (cid:2) (cid:104) Q ω k

Original: ( s, · ) , π ( · | s ) (cid:105) − β · KL (cid:0) π ( · | s ) (cid:107) π θ k ( · | s ) (cid:1)(cid:3) .
Revised: ( s, · ) , π ( · | s ) (cid:105) − β · KL (cid:0) π ( · | s ) (cid:107) π θ k ( · | s ) (cid:1)(cid:3) .

Original: Then (cid:101)
Revised: Then (cid:101)

Original: π k +1 has the following closed form: (cid:101) π k +1 ( a | s ) ∝ exp (cid:0) β − 1 Q ω
Revised: π k +1 has the following closed form: (cid:101) π k +1 ( a | s ) ∝ exp (cid:0) β − 1 Q ω

Original: k
Revised: k

Original: ( s, a )
Revised: ( s, a )

Original: + τ − 1 k f θ k
Revised: + τ − 1 k f θ k

Original: ( s, a ) (cid:1) , for any ( s, a ) ∈ S × A , where ν k = ν
Revised: ( s, a ) (cid:1) , for any ( s, a ) ∈ S × A , where ν k = ν

Original: π θk is the stationary state distribution of π θ k .
Revised: π θk is the stationary state distribution of π θ k .

Original: See § G.1 for a detailed proof of Proposition 3.1.
Revised: See § G.1 for a detailed proof of Proposition 3.1.

Original: Motivated by Proposition 3.1, to implement the actor update in (2.7), we update the actor parameter θ by solving the following minimization problem,
Revised: Motivated by Proposition 3.1, to implement the actor update in (2.7), we update the actor parameter θ by solving the following minimization problem,

Original: [Equation]
Revised: [Equation]

Original: where ρ k = ρ π θk is the stationary state-action distribution of π θ k .
Revised: where ρ k = ρ π θk is the stationary state-action distribution of π θ k .

Original: Critic Update.
Revised: Critic Update.

Original: To implement the critic update in (2.7), we update the critic parameter ω by solving the following minimization problem,
Revised: To implement the critic update in (2.7), we update the critic parameter ω by solving the following minimization problem,

Original: [Equation]
Revised: [Equation]

Original: where ρ k +1 =
Revised: where ρ k +1 =

Original: ρ π θk +1 is the stationary state-action distribution of π θ k +1 and the operator P π is deﬁned in (2.3).
Revised: ρ π θk +1 is the stationary state-action distribution of π θ k +1 and the operator P π is deﬁned in (2.3).

Original: 3. L INEAR F UNCTION A PPROXIMATION
Revised: 3. L INEAR F UNCTION A PPROXIMATION

Original: In this section, we consider linear function approximation.
Revised: In this section, we consider linear function approximation.

Original: More speciﬁcally, we parameterize the action-value function using Q ω ( s, a ) = ω (cid:62) ϕ
Revised: More speciﬁcally, we parameterize the action-value function using Q ω ( s, a ) = ω (cid:62) ϕ

Original: ( s, a ) and the energy function of the energy-based policy π θ using f θ ( s, a )
Revised: ( s, a ) and the energy function of the energy-based policy π θ using f θ ( s, a )

Original: = θ (cid:62) ϕ
Revised: = θ (cid:62) ϕ

Original: ( s, a ) .
Revised: ( s, a ) .

Original: Here ϕ ( s, a
Revised: Here ϕ ( s, a

Original: ) ∈ R d is the feature vector, where d > 0 is the dimension.
Revised: ) ∈ R d is the feature vector, where d > 0 is the dimension.

Original: Without loss of generality, we assume that (cid:107) ϕ
Revised: Without loss of generality, we assume that (cid:107) ϕ

Original: ( s, a ) (cid:107) 2 ≤ 1 for any ( s, a ) ∈ S × A , which can be achieved by normalization.
Revised: ( s, a ) (cid:107) 2 ≤ 1 for any ( s, a ) ∈ S × A , which can be achieved by normalization.

Original: Actor Update.
Revised: Actor Update.

Original: The minimization problem in (3.1) admits the following closed-form solution,
Revised: The minimization problem in (3.1) admits the following closed-form solution,

Original: [Equation]
Revised: [Equation]

Original: which corresponds to a step of the natural policy gradient method (Kakade, 2002).
Revised: which corresponds to a step of the natural policy gradient method (Kakade, 2002).

Original: Critic Update.
Revised: Critic Update.

Original: The minimization problem in (3.2) admits the following closed-form solution,
Revised: The minimization problem in (3.2) admits the following closed-form solution,

Original: [Equation]
Revised: [Equation]

Original: Since the closed-form solution (cid:101) ω k +1 in (3.4) involves the expectation over the stationary state-action distribution ρ k +1 of π θ k +1 , we use data to approximate such an expectation.
Revised: Since the closed-form solution (cid:101) ω k +1 in (3.4) involves the expectation over the stationary state-action distribution ρ k +1 of π θ k +1 , we use data to approximate such an expectation.

Original: More speciﬁcally, we sample { ( s (cid:96), 1 , a (cid:96), 1 ) } (cid:96) ∈
Revised: More speciﬁcally, we sample { ( s (cid:96), 1 , a (cid:96), 1 ) } (cid:96) ∈

Original: [ N ] and { ( s (cid:96), 2 , a (cid:96), 2 , r (cid:96), 2 , s (cid:48) (cid:96), 2 , a (cid:48) (cid:96), 2 ) } (cid:96)
Revised: [ N ] and { ( s (cid:96), 2 , a (cid:96), 2 , r (cid:96), 2 , s (cid:48) (cid:96), 2 , a (cid:48) (cid:96), 2 ) } (cid:96)

Original: ∈
Revised: ∈

Original: [ N ] such that ( s (cid:96), 1 , a (cid:96), 1 ) ∼ ρ k
Revised: [ N ] such that ( s (cid:96), 1 , a (cid:96), 1 ) ∼ ρ k

Original: +1 ,
Revised: +1 ,

Original: ( s (cid:96), 2 , a (cid:96), 2 ) ∼ ρ k
Revised: ( s (cid:96), 2 , a (cid:96), 2 ) ∼ ρ k

Original: +1 , r (cid:96), 2 = r
Revised: +1 , r (cid:96), 2 = r

Original: ( s (cid:96), 2 , a (cid:96), 2 ) , s (cid:48) (cid:96), 2 ∼ P ( · | s (cid:96), 2 , a (cid:96), 2 ) , and a (cid:48) (cid:96), 2 ∼ π θ k +

Revised: ( s (cid:96), 2 , a (cid:96), 2 ) , s (cid:48) (cid:96), 2 ∼ P ( · | s (cid:96), 2 , a (cid:96), 2 ) , and a (cid:48) (cid:96), 2 ∼ π θ k +

Original: ( · | s (cid:48) (cid:96), 2 ) , where N is the sample size.
Revised: ( · | s (cid:48) (cid:96), 2 ) , where N is the sample size.

Original: We approximate (cid:101) ω
Revised: We approximate (cid:101) ω

Original: k +1 using ω k +1 , which is deﬁned as follows,
Revised: k +1 using ω k +1 , which is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Here Γ R is the projection operator, which projects the parameter onto the centered ball with radius
Revised: Here Γ R is the projection operator, which projects the parameter onto the centered ball with radius

Original: R in R d .
Revised: R in R d .

Original: Such a projection operator stabilizes the algorithm (Konda and Tsitsiklis, 2000; Bhatnagar et al., 2009).
Revised: Such a projection operator stabilizes the algorithm (Konda and Tsitsiklis, 2000; Bhatnagar et al., 2009b).

Original: It is worth mentioning that one may also view the update in (3.5) as one step of the least-squares temporal difference method (Bradtke and Barto, 1996), which can be modiﬁed for the off-policy setting (Antos et al., 2007; Yu, 2010; Liu et al., 2018; Nachum et al., 2019; Xie et al., 2019; Zhang et al., 2020; Uehara and Jiang, 2019; Nachum and Dai, 2020).
Revised: It is worth mentioning that one may also view the update in (3.5) as one step of the least-squares temporal difference method (Bradtke and Barto, 1996), which can be modiﬁed for the off-policy setting (Antos et al., 2007; Yu, 2010; Liu et al., 2018; Nachum et al., 2019; Xie et al., 2019; Zhang et al., 2020; Uehara and Jiang, 2019; Nachum and Dai, 2020).

Original: Such a modiﬁcation allows the data points in (3.5) to be reused in the subsequent iterations, which further improves the sample complexity.
Revised: Such a modiﬁcation allows the data points in (3.5) to be reused in the subsequent iterations, which further improves the sample complexity.

Original: Speciﬁcally, let ρ bhv ∈ P ( S × A ) be the stationary state-action distribution induced by a behavioral policy π bhv .
Revised: Speciﬁcally, let ρ bhv ∈ P ( S × A ) be the stationary state-action distribution induced by a behavioral policy π bhv .

Original: We replace the actor and critic updates in (3.1) and (3.2) by
Revised: We replace the actor and critic updates in (3.1) and (3.2) by

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: With linear function approximation, the actor update in (3.6) is reduced to (3.3), while the critic update in (3.7) admits a closed form solution
Revised: With linear function approximation, the actor update in (3.6) is reduced to (3.3), while the critic update in (3.7) admits a closed form solution

Original: [Equation]
Revised: [Equation]

Original: , which can be well approximated using state-action pairs drawn from ρ bhv .
Revised: , which can be well approximated using state-action pairs drawn from ρ bhv .

Original: See § 4 for a detailed discussion.
Revised: See § 4 for a detailed discussion.

Original: Finally, by assembling the updates in (3.3) and (3.5), we present the linear actor-critic method in Algorithm 1, which is deferred to § B of the appendix.
Revised: Finally, by assembling the updates in (3.3) and (3.5), we present the linear actor-critic method in Algorithm 1, which is deferred to § B of the appendix.

Original:  T HEORETICAL R ESULTS
Revised:  T HEORETICAL R ESULTS

Original: In this section, we upper bound the regret of the linear actor-critic method.
Revised: In this section, we upper bound the regret of the linear actor-critic method.

Original: We defer the analysis of the deep neural actor-critic method to § C of the appendix.
Revised: We defer the analysis of the deep neural actor-critic method to § C of the appendix.

Original: Hereafter we assume that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A , where r max is a positive absolute constant.
Revised: Hereafter we assume that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A , where r max is a positive absolute constant.

Original: First, we impose the following assumptions.
Revised: First, we impose the following assumptions.

Original: Recall that ρ ∗ is the stationary state-action distribution of π ∗ , while ρ k is the stationary state-action distribution of π θ k .
Revised: Recall that ρ ∗ is the stationary state-action distribution of π ∗ , while ρ k is the stationary state-action distribution of π θ k .

Original: Moreover, let ρ ∈ P ( S × A ) be a state-action distribution with respect to which we aim to characterize the performance of the actor-critic algorithm.
Revised: Moreover, let ρ ∈ P ( S × A ) be a state-action distribution with respect to which we aim to characterize the performance of the actor-critic algorithm.

Original: Speciﬁcally, after K + 1 actor updates, we are interest in upper bounding the following regret
Revised: Speciﬁcally, after K + 1 actor updates, we are interested in upper bounding the following regret

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to { θ k } k ∈
Revised: where the expectation is taken with respect to { θ k } k ∈

Original: [ K +1] and ( s, a ) ∼ ρ .
Revised: [ K +1] and ( s, a ) ∼ ρ .

Original: Here we allow ρ to be any ﬁxed distribution for generality, which might be different from ρ ∗ .
Revised: Here we allow ρ to be any ﬁxed distribution for generality, which might be different from ρ ∗ .

Original: Assumption 4.1 (Concentrability Coefﬁcient) .
Revised: Assumption 4.1 (Concentrability Coefﬁcient) .

Original: The following statements hold.
Revised: The following statements hold.

Original: (i)
Revised: (i)

Original: There exists a positive absolute constant φ ∗ such that φ ∗ k ≤ φ ∗ for any k ≥ 1 , where φ ∗ k = (cid:107) d ρ ∗ / d ρ k (cid:107) ρ k , 2 .
Revised: There exists a positive absolute constant φ ∗ such that φ ∗ k ≤ φ ∗ for any k ≥ 1 , where φ ∗ k = (cid:107) d ρ ∗ / d ρ k (cid:107) ρ k , 2 .

Original: (ii)
Revised: (ii)

Original: We assume that for any k ≥ 1 and a sequence of policies { π i } i ≥ 1 , the k -step future-stateaction distribution ρ P π 1 · · · P π k is absolutely continuous with respect to ρ ∗ , where ρ is the same as the one in (4.1) Also, it holds for such ρ that C ρ,ρ ∗ =
Revised: We assume that for any k ≥ 1 and a sequence of policies { π i } i ≥ 1 , the k -step future-stateaction distribution ρ P π 1 · · · P π k is absolutely continuous with respect to ρ ∗ , where ρ is the same as the one in (4.1) Also, it holds for such ρ that C ρ,ρ ∗ =

Original: (1 − γ ) 2 (cid:80) ∞ k =1 k 2 γ k · c ( k ) < ∞ , where c ( k ) =
Revised: (1 − γ ) 2 (cid:80) ∞ k =1 k 2 γ k · c ( k ) < ∞ , where c ( k ) =

Original: sup { π i } i ∈
Revised: sup { π i } i ∈

Original: [ k ] (cid:107) d( ρ P π 1 · · · P π k ) /
Revised: [ k ] (cid:107) d( ρ P π 1 · · · P π k ) /

Original: d ρ ∗ (cid:107) ρ ∗ , ∞ .
Revised: d ρ ∗ (cid:107) ρ ∗ , ∞ .

Original: In Assumption 4.1, C ρ,ρ ∗ is known as the discounted-average concentrability coefﬁcient of the future-state-action distributions.
Revised: In Assumption 4.1, C ρ,ρ ∗ is known as the discounted-average concentrability coefﬁcient of the future-state-action distributions.

Revised: Such an assumption indeed measures the stochastic stability properties of the MDP, and the class of MDPs with such properties is quite large.

Original: Similar assumptions are commonly imposed in the literature (Szepesv´ari and Munos, 2005; Munos and Szepesv´ari, 2008; Antos et al., 2008a;b; Scherrer, 2013;
Revised: See Szepesv´ari and Munos (2005); Munos and Szepesv´ari (2008); Antos et al. (2008a;b); Scherrer (2013); Scherrer et al.

Original: Scherrer et al., 2015; Farahmand et al., 2016;
Revised: ; Farahmand et al.

Original:  Yang et al., 2019b; Geist et al., 2019; Chen and Jiang, 2019).
Revised: ; Yang et al. (2019b); Geist et al. ; Chen and Jiang (2019) for more examples and discussion.

Original: Assumption 4.2 (Zero Approximation Error) .
Revised: Assumption 4.2 (Zero Approximation Error) .

Original: It holds for any ω, θ ∈
Revised: It holds for any ω, θ ∈

Original: B (0 , R ) that inf [Equation]
Revised: B (0 , R ) that inf ¯ ω ∈B (0 ,R )

Original: [Equation]
Revised: E ρ πθ (cid:2)(cid:0) [ T π θ Q ω − ¯ ω (cid:62) ϕ ]( s, a ) (cid:1) 2 (cid:3)

Original: (cid:3) = 0 , where T π θ is deﬁned in (2.4).
Revised: = 0 , where T π θ is deﬁned in (2.4).

Original: Assumption 4.2 states that the Bellmanevaluation operator  maps a linear  function to a linear function.
Revised: Assumption 4.2 imposes a structural assumption of the MDP under the linear setting. Speciﬁcally speaking, it assumes that the Bellman operator of each policy maps a linear value function to a linear function.

Original: Such an assumption only aims to simplify the presentation of  our results.
Revised: Therefore, the value function associated with each policy (which is the ﬁxed point of the corresponding Bellman operator) lies in the linear function class. Since the value functions are linear here, the energy-based policy class approximately covers the optimal policy as the temperature parameter τ goes to zero. In summary, our Assumption 4.2 ensures that the energy-based policy class approximately captures the optimal policy and thus there is no approximation error. When

Original: If the approximation error is nonzero, we only need to incorporate an additional bias term into the rate of convergence.
Revised: Assumption 4.2 does not hold, we only need to add an additional bias term to the regret upper bound in our theorem without much change in the proof.

Original: Assumption 4.3 (Well-Conditioned Feature) .
Revised: Assumption 4.3 (Well-Conditioned Feature) .

Original: The minimum singular value of the matrix E ρ k
Revised: The minimum singular value of the matrix E ρ k

Original: [ ϕ ( s, a ) ϕ
Revised: [ ϕ ( s, a ) ϕ

Original: ( s, a ) (cid:62) ] is uniformly lower bounded by a positive absolute constant σ ∗ for any k ≥ 1 .
Revised: ( s, a ) (cid:62) ] is uniformly lower bounded by a positive absolute constant σ ∗ for any k ≥ 1 .

Original: Assumption 4.3 ensures that the minimization problem in (3.2) admits a unique minimizer, which is used in the critic update.
Revised: Assumption 4.3 ensures that the minimization problem in (3.2) admits a unique minimizer, which is used in the critic update.

Original: Similar assumptions are commonly imposed in the literature (Bhandari et al., 2018;  Zou et al., 2019).
Revised: Similar assumptions are commonly imposed in the literature (Bhandari et al., 2018; Xu et al., 2019; Zou et al., 2019; Wu et al., 2020).

Original: Under Assumptions 4.1, 4.2, and 4.3, we upper bound the regret of Algorithm 1 in the following theorem.
Revised: Under Assumptions 4.1, 4.2, and 4.3, we upper bound the regret of Algorithm 1 in the following theorem.

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: We assume that Assumptions 4.1, 4.2, and 4.3 hold.
Revised: We assume that Assumptions 4.1, 4.2, and 4.3 hold.

Original: Let ρ be a state-action distribution satisfying (ii) of Assumption 4.1.
Revised: Let ρ be a state-action distribution satisfying (ii) of Assumption 4.1.

Original: Also, for any sufﬁciently large K > 0 , let β
Revised: Also, for any sufﬁciently large K > 0 , let β

Original: =
Revised: =

Original: K 1 / 2 ,
Revised: K 1 / 2 ,

Original: N = Ω( KC 2 ρ,ρ ∗ · ( φ ∗ /σ ∗ ) 2 · log 2 N ) , and the sequence of policy parameters { θ k } k ∈
Revised: N = Ω( KC 2 ρ,ρ ∗ · ( φ ∗ /σ ∗ ) 2 · log 2 N ) , and the sequence of policy parameters { θ k } k ∈

Original: [ K +1] be generated by Algorithm 1.
Revised: [ K +1] be generated by Algorithm 1.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to { θ k } k ∈
Revised: where the expectation is taken with respect to { θ k } k ∈

Original: [ K +1] and ( s, a ) ∼ ρ .
Revised: [ K +1] and ( s, a ) ∼ ρ .

Original: We sketch the proof in § D. See § E.1 for a detailed proof.
Revised: We sketch the proof in § D. See § E.1 for a detailed proof.

Original: Theorem 4.4 establishes an O ( K 1 / 2 ) regret of Algorithm 1, where K is the total number of iterations.
Revised: Theorem 4.4 establishes an O ( K 1 / 2 ) regret of Algorithm 1, where K is the total number of iterations.

Original: Here O ( · ) omits terms involving (1 − γ ) − 1 and log |A| .
Revised: Here O ( · ) omits terms involving (1 − γ ) − 1 and log |A| .

Original: To better understand Theorem 4.4, we consider the ideal setting, where we have access to the action-value function Q π of any policy π .
Revised: To better understand Theorem 4.4, we consider the ideal setting, where we have access to the action-value function Q π of any policy π .

Original: In such an ideal setting, the critic update is unnecessary.
Revised: In such an ideal setting, the critic update is unnecessary.

Original: However, the natural policy gradient method, which only uses the actor update, achieves the same O ( K 1 / 2 ) regret (Liu et al., 2019; Agarwal et al., 2019;
Revised: However, the natural policy gradient method, which only uses the actor update, achieves the same O ( K 1 / 2 ) regret (Liu et al., 2019; Agarwal et al., 2019;

Original: Cai et al., 2019).
Revised: Cai et al., 2019).

Original: In other words, in terms of the iteration complexity, Theorem 4.4 shows that in the single-timescale setting, using only one step of the critic update along with one step of the actor update is as efﬁcient as the natural policy gradient method in the ideal setting.
Revised: In other words, in terms of the iteration complexity, Theorem 4.4 shows that in the single-timescale setting, using only one step of the critic update along with one step of the actor update is as efﬁcient as the natural policy gradient method in the ideal setting.

Original: Furthermore, by the regret bound in (4.2), to obtain an ε -globally optimal policy, it sufﬁces to set
Revised: Furthermore, by the regret bound in (4.2), to obtain an ε -globally optimal policy, it sufﬁces to set

Original: K (cid:16) (1 − γ ) − 6 · ε − 2 · log 2 |A| in Algorithm 1 and output a randomized policy that is drawn from { π θ k } K +1 k
Revised: K (cid:16) (1 − γ ) − 6 · ε − 2 · log 2 |A| in Algorithm 1 and output a randomized policy that is drawn from { π θ k } K +1 k

Original: =1 uniformly.
Revised: =1 uniformly.

Original: Plugging such a K into N = Ω( KC 2 ρ,ρ ∗ ( φ ∗ /σ ∗ ) 2 · log 2 N ) , we obtain that
Revised: Plugging such a K into N = Ω( KC 2 ρ,ρ ∗ ( φ ∗ /σ ∗ ) 2 · log 2 N ) , we obtain that

Original: N = (cid:101)
Revised: N = (cid:101)

Original: O ( ε − 2 ) , where (cid:101) O ( · ) omits the logarithmic terms.
Revised: O ( ε − 2 ) , where (cid:101) O ( · ) omits the logarithmic terms.

Original: Thus, to achieve an ε -globally optimal policy, the total sample complexity of Algorithm 1 is (cid:101) O
Revised: Thus, to achieve an ε -globally optimal policy, the total sample complexity of Algorithm 1 is (cid:101) O

Original: ( ε − 4 ) .
Revised: ( ε − 4 ) .

Original: This matches the sample complexity results established in Xu et al.
Revised: This matches the sample complexity results established in Xu et al.

Original: (2020); Hong et al.
Revised: (2020); Hong et al.

Original: (2020) for two-timescale actor-critic methods.
Revised: (2020) for two-timescale actor-critic methods.

Original: Meanwhile, notice that here the critic updates are on-policy and we draw N new data points in each critic update.
Revised: Meanwhile, notice that here the critic updates are on-policy and we draw N new data points in each critic update.

Original: As discussed in § 3.1, under the off-policy setting, the critic updates given in (3.7) can be implemented using a ﬁxed dataset sampled from ρ bhv , the stationary state-action distribution induced by the behavioral policy.
Revised: As discussed in § 3.1, under the off-policy setting, the critic updates given in (3.7) can be implemented using a ﬁxed dataset sampled from ρ bhv , the stationary state-action distribution induced by the behavioral policy.

Original: Under this scenario, the total number of data points used by the algorithm is equal to N .
Revised: Under this scenario, the total number of data points used by the algorithm is equal to N .

Original: Moreover, by imposing similar assumptions on ρ bhv as in (i) of Assumption 4.1 and Assumption 4.3, we can establish a similar O ( K 1 / 2 ) regret as in (4.2) for the off-policy setting.
Revised: Moreover, by imposing similar assumptions on ρ bhv as in (i) of Assumption 4.1 and Assumption 4.3, we can establish a similar O ( K 1 / 2 ) regret as in (4.2) for the off-policy setting.

Original: As a result, with data reuse, to obtain an ε -globally optimal policy, the sample complexity of Algorithm 1 is essentially (cid:101)
Revised: As a result, with data reuse, to obtain an ε -globally optimal policy, the sample complexity of Algorithm 1 is essentially (cid:101)

Original: O ( ε − 2 ) , which demonstrates the advantage of our single-timescale actor-critic method.
Revised: O ( ε − 2 ) , which demonstrates the advantage of our single-timescale actor-critic method.

Original: Besides, only focusing on the convergence to an ε -stationary point, Wu et al.
Revised: Besides, only focusing on the convergence to an ε -stationary point, Wu et al.

Original: Xu et al.
Revised: Xu et al.

Original: (2020) establish the sample complexity of (cid:101)
Revised: (2020) establish the sample complexity of (cid:101)

Original: O ( ε − 5 / 2 ) for two-timescale actor-critic, where ε measures the squared Euclidean norm of the policy gradient.
Revised: O ( ε − 5 / 2 ) for two-timescale actor-critic, where ε measures the squared Euclidean norm of the policy gradient.

Original: In contrast, by adopting the natural policy gradient (Kakade, 2002) in actor updates, we achieve convergence to the globally optimal policy.
Revised: In contrast, by adopting the natural policy gradient (Kakade, 2002) in actor updates, we achieve convergence to the globally optimal policy.

Revised: We remark that the idea of off-policy evaluation cannot be applied to typical twotimescale setting (Wu et al., 2020; Xu et al., 2020), where the critic is updated using TD learning (e.g. TD(0) and TD( λ )), since it is shown that off-policy TD method may diverge even with linear function approximation (Baird et al., 1995;

Revised: Sutton et al., 2008).

Original: To the best of our knowledge, we establish the rate of convergence and global optimality of the actor-critic method with function approximation in the single-timescale setting for the ﬁrst time.
Revised: To the best of our knowledge, we establish the rate of convergence and global optimality of the actor-critic method with function approximation in the single-timescale setting for the ﬁrst time.

Original: Furthermore, as we will show in Theorem C.5 of § B, when both the actor and the critic are represented using overparameterized deep neural networks, we establish a similar O ((1 − γ ) − 3 · log |A| ·
Revised: Furthermore, as we will show in Theorem C.5 of § B, when both the actor and the critic are represented using overparameterized deep neural networks we establish a similar O ((1 − γ ) − 3 · log |A| ·

Original: K 1 / 2 ) regret when the architecture of the actor and critic neural networks are properly chosen.
Revised: K 1 / 2 ) regret when the architecture of the actor and critic neural networks are properly chosen.

Original: To our best knowledge, this seems the ﬁrst theoretical guarantee for the actor-critic method with deep neural network function approximation in terms of the rate of convergence and global optimality.
Revised: To our best knowledge this seems the ﬁrst theoretical guarantee for the actor-critic method with deep neural network function approximation in terms of the rate of convergence and global optimality

Original: A D EEP N EURAL N ETWORK A PPROXIMATION
Revised: A D EEP N EURAL N ETWORK A PPROXIMATION

Original: In this section, we consider deep neural network approximation.
Revised: In this section, we consider deep neural network approximation.

Original: We ﬁrst formally deﬁne deep neural networks.
Revised: We ﬁrst formally deﬁne deep neural networks.

Original: Then we introduce the actor-critic method under such a parameterization.
Revised: Then we introduce the actor-critic method under such a parameterization.

Original: A deep neural network (DNN) u θ ( x ) with the input x ∈ R d , depth H , and width m is deﬁned as
Revised: A deep neural network (DNN) u θ ( x ) with the input x ∈ R d , depth H , and width m is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Here σ : R m → R m is the rectiﬁed linear unit (ReLU) activation function, which is deﬁne as σ ( y ) =
Revised: Here σ : R m → R m is the rectiﬁed linear unit (ReLU) activation function, which is deﬁne as σ ( y ) =

Original: (max { 0 , y 1 } , . . .
Revised: (max { 0 , y 1 } , . . .

Original: , max { 0 , y m } ) (cid:62) for any y =
Revised: , max { 0 , y m } ) (cid:62) for any y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y m ) (cid:62) ∈ R m .
Revised: , y m ) (cid:62) ∈ R m .

Original: Also, we have b ∈ {− 1 , 1 } m ,
Revised: Also, we have b ∈ {− 1 , 1 } m ,

Original: W 1 ∈ R d × m , and W h ∈ R m × m for 2 ≤ h ≤ H .
Revised: W 1 ∈ R d × m , and W h ∈ R m × m for 2 ≤ h ≤ H .

Original: Meanwhile, we denote the parameter of the
Revised: Meanwhile, we denote the parameter of the

Original: DNN u θ as θ =
Revised: DNN u θ as θ =

Original: ( vec ( W 1 ) (cid:62) , . .
Revised: ( vec ( W 1 ) (cid:62) , . .

Original: , vec ( W H ) (cid:62) )
Revised: , vec ( W H ) (cid:62) )

Original: (cid:62) ∈ R m all with m all =
Revised: (cid:62) ∈ R m all with m all =

Original: md
Revised: md

Original: + ( H − 1) m 2 .
Revised: + ( H − 1) m 2 .

Original: We call { W h } h ∈
Revised: We call { W h } h ∈

Original: [ H ] the weight matrices of θ .
Revised: [ H ] the weight matrices of θ .

Original: Without loss of generality, we normalize the input x such that (cid:107) x (cid:107) 2 = 1 .
Revised: Without loss of generality, we normalize the input x such that (cid:107) x (cid:107) 2 = 1 .

Original: We initialize the DNN such that each entry of W h follows the standard Gaussian distribution N (0 , 1) for any h ∈
Revised: We initialize the DNN such that each entry of W h follows the standard Gaussian distribution N (0 , 1) for any h ∈

Original: [ H ] , while each entry of b follows the uniform distribution Unif( {− 1 , 1 } ) .
Revised: [ H ] , while each entry of b follows the uniform distribution Unif( {− 1 , 1 } ) .

Original: Without loss of generality, we ﬁx b during training and only optimize { W h } h ∈
Revised: Without loss of generality, we ﬁx b during training and only optimize { W h } h ∈

Original: [ H ] .
Revised: [ H ] .

Original: We denote the initialization of the parameter θ as θ 

Revised: We denote the initialization of the parameter θ as θ

Original: =
Revised: =

Original: ( vec ( W 01 ) (cid:62)
Revised: ( vec ( W 01 ) (cid:62)

Original: [Equation]
Revised: [Equation]

Original: , vec ( W 0 H ) (cid:62) ) (cid:62) .
Revised: , vec ( W 0 H ) (cid:62) ) (cid:62) .

Original: Meanwhile, we restrict θ within the ball
Revised: Meanwhile, we restrict θ within the ball

Original: B ( θ 0 , R ) during training, which is deﬁned as follows,
Revised: B ( θ 0 , R ) during training, which is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Here { W h } h ∈ [ H ] and { W 0 h } h ∈ [ H ] are the weight matrices of θ and θ 0 , respectively.
Revised: Here { W h } h ∈ [ H ] and { W 0 h } h ∈ [ H ] are the weight matrices of θ and θ 0 , respectively.

Original: By (A.2), we have (cid:107) θ − θ 

Revised: By (A.2), we have (cid:107) θ − θ

Original: (cid:107) 2 ≤ R √ H for any θ ∈ B ( θ 0 , R ) .
Revised: (cid:107) 2 ≤ R √ H for any θ ∈ B ( θ 0 , R ) .

Original: Now, we deﬁne the family of DNNs as
Revised: Now, we deﬁne the family of DNNs as

Original: [Equation]
Revised: [Equation]

Original: where u θ is a DNN with depth H and width m .
Revised: where u θ is a DNN with depth H and width m .

Original: We parameterize the action-value function using Q ω ( s, a ) ∈ U ( m c , H c , R c ) and the energy function of the energy-based policy π θ using f θ ( s, a ) ∈ U ( m a , H a , R a ) .
Revised: We parameterize the action-value function using Q ω ( s, a ) ∈ U ( m c , H c , R c ) and the energy function of the energy-based policy π θ using f θ ( s, a ) ∈ U ( m a , H a , R a ) .

Original: Here U ( m c , H c , R c ) and
Revised: Here U ( m c , H c , R c ) and

Original: U ( m a , H a , R a ) are the families of DNNs deﬁned in (A.3).
Revised: U ( m a , H a , R a ) are the families of DNNs deﬁned in (A.3).

Original: Hereafter we assume that the energy function f θ and the action-value function Q ω share the same architecture and initialization, i.e., m a = m c , H a = H c , R a = R c , and θ 0 = ω 0 .
Revised: Hereafter we assume that the energy function f θ and the action-value function Q ω share the same architecture and initialization, i.e., m a = m c , H a = H c , R a = R c , and θ 0 = ω 0 .

Original: Such shared architecture and initialization of the
Revised: Such shared architecture and initialization of the

Original: DNNs ensure that the parameterizations of the policy and the action-value function are approximately compatible.
Revised: DNNs ensure that the parameterizations of the policy and the action-value function are approximately compatible.

Original: See Sutton et al.
Revised: See Sutton et al.

Original: ; Konda and Tsitsiklis (2000); Kakade (2002); Peters and Schaal (2008a); Wang et al.
Revised: ; Konda and Tsitsiklis (2000); Kakade (2002); Peters and Schaal (2008a); Wang et al.

Original: for a detailed discussion.
Revised: for a detailed discussion.

Original: Actor Update.
Revised: Actor Update.

Original: To solve (3.1), we use projected stochastic gradient descent, whose n -th iteration has the following form,
Revised: To solve (3.1), we use projected stochastic gradient descent, whose n -th iteration has the following form,

Original: [Equation]
Revised: [Equation]

Original: Here Γ B ( θ 0 ,R a ) is the projection operator, which projects the parameter onto the ball B ( θ 0 , R a ) deﬁned in (A.2).
Revised: Here Γ B ( θ 0 ,R a ) is the projection operator, which projects the parameter onto the ball B ( θ 0 , R a ) deﬁned in (A.2).

Original: The state-action pair ( s, a ) is sampled from the stationary state-action distribution ρ k .
Revised: The state-action pair ( s, a ) is sampled from the stationary state-action distribution ρ k .

Original: We summarize the update in Algorithm 3, which is deferred to § B of the appendix.
Revised: We summarize the update in Algorithm 3, which is deferred to § B of the appendix.

Original: Critic Update.
Revised: Critic Update.

Original: To solve (3.2), we apply projected stochastic gradient descent.
Revised: To solve (3.2), we apply projected stochastic gradient descent.

Original: More speciﬁcally, at the n -th iteration of projected stochastic gradient descent, we sample a tuple ( s, a, r, s (cid:48) , a (cid:48) ) , where ( s, a ) ∼
Revised: More speciﬁcally, at the n -th iteration of projected stochastic gradient descent, we sample a tuple ( s, a, r, s (cid:48) , a (cid:48) ) , where ( s, a ) ∼

Original: ρ k
Revised: ρ k

Original: +1 , r = r
Revised: +1 , r = r

Original: ( s, a ) , s (cid:48) ∼ P ( · | s, a ) , and a (cid:48)
Revised: ( s, a ) , s (cid:48) ∼ P ( · | s, a ) , and a (cid:48)

Original: ∼ π θ k +1 ( · | s (cid:48) ) .
Revised: ∼ π θ k +1 ( · | s (cid:48) ) .

Original: We deﬁne the residual at the n -th iteration as δ ( n ) =
Revised: We deﬁne the residual at the n -th iteration as δ ( n ) =

Original: Q ω ( n ) ( s, a ) − (1 − γ ) · r − γ · Q ω k ( s (cid:48) , a (cid:48) ) .
Revised: Q ω ( n ) ( s, a ) − (1 − γ ) · r − γ · Q ω k ( s (cid:48) , a (cid:48) ) .

Original: Then the n -th iteration of projected stochastic gradient descent has the following form,
Revised: Then the n -th iteration of projected stochastic gradient descent has the following form,

Original: [Equation]
Revised: [Equation]

Original: Here Γ B ( ω 0 ,R c ) is the projection operator, which projects the parameter onto the ball B ( ω 0 , R c ) deﬁned in (A.2).
Revised: Here Γ B ( ω 0 ,R c ) is the projection operator, which projects the parameter onto the ball B ( ω 0 , R c ) deﬁned in (A.2).

Original: We summarize the update in Algorithm 4, which is deferred to § B of the appendix.
Revised: We summarize the update in Algorithm 4, which is deferred to § B of the appendix.

Original: By assembling Algorithms 3 and 4, we present the deep neural actor-critic method in Algorithm 2, which is deferred to § B of the appendix.
Revised: By assembling Algorithms 3 and 4, we present the deep neural actor-critic method in Algorithm 2, which is deferred to § B of the appendix.

Original: Finally, we remark that the off-policy actor and critic updates given in (3.6) and (3.7) can also incorporate deep neural network approximation with a slight modiﬁcation, which enables data reuse in the algorithm.
Revised: Finally, we remark that the off-policy actor and critic updates given in (3.6) and (3.7) can also incorporate deep neural network approximation with a slight modiﬁcation, which enables data reuse in the algorithm.

Original: B D ETAILS OF A LGORITHMS
Revised: B D ETAILS OF A LGORITHMS

Original: In this section, we summarize the algorithms in § 3.
Revised: In this section, we summarize the algorithms in § 3.

Original: We ﬁrst introduce the actor-critic method with linear function approximation in Algorithm 1.
Revised: We ﬁrst introduce the actor-critic method with linear function approximation in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: Input: Number of iterations K , sample size N , temperature parameter β .
Revised: Input: Number of iterations K , sample size N , temperature parameter β .

Original: Initialization: Set τ 0 ← ∞ , and randomly initialize the actor parameter θ 

Revised: Initialization: Set τ 0 ← ∞ , and randomly initialize the actor parameter θ

Original: and the critic parameter ω 0 .
Revised: and the critic parameter ω 0 .

Original: for k = 0 , 1 , 2 , . . .
Revised: for k = 0 , 1 , 2 , . . .

Original: K do
Revised: K do

Original: Actor Update: Update θ k +1 via (3.3) with τ − 1 k
Revised: Actor Update: Update θ k +1 via (3.3) with τ − 1 k

Original: +

Revised: +

Original: =
Revised: =

Original: ( k + 1) · β − 1 .
Revised: ( k + 1) · β − 1 .

Original: Critic Update: Sample { ( s (cid:96), 1 , a (cid:96), 1 ) } (cid:96)
Revised: Critic Update: Sample { ( s (cid:96), 1 , a (cid:96), 1 ) } (cid:96)

Original: ∈
Revised: ∈

Original: [ N ] and { ( s (cid:96), 2 , a (cid:96), 2 , r (cid:96), 2 , s (cid:48) (cid:96), 2 , a (cid:48) (cid:96), 2 ) } (cid:96)
Revised: [ N ] and { ( s (cid:96), 2 , a (cid:96), 2 , r (cid:96), 2 , s (cid:48) (cid:96), 2 , a (cid:48) (cid:96), 2 ) } (cid:96)

Original: ∈
Revised: ∈

Original: [ N ] as speciﬁed in § 3.1.
Revised: [ N ] as speciﬁed in § 3.1.

Original: Update ω
Revised: Update ω

Original: k
Revised: k

Original: +1 via (3.5).
Revised: +1 via (3.5).

Original: end for
Revised: end for

Original: Output: { π θ k } k ∈ [ K +1] , where π θ k ∝ exp( τ − 1 k f θ k ) .
Revised: Output: { π θ k } k ∈ [ K +1] , where π θ k ∝ exp( τ − 1 k f θ k ) .

Original: We introduce the actor-critic method with DNN approximation in Algorithm 2, which relies on
Revised: We introduce the actor-critic method with DNN approximation in Algorithm 2, which relies on

Original: Algorithms 3 and 4 for the actor and critic updates.
Revised: Algorithms 3 and 4 for the actor and critic updates.

Original: Input: Number of iterations K, N a , N c , stepsizes α, η , and temperature parameter β .
Revised: Input: Number of iterations K, N a , N c , stepsizes α, η , and temperature parameter β .

Original: Initialization: Set τ 0 ← ∞ and initialize DNNs f θ 0 and Q ω 0 as speciﬁed in § A. for k = 0 , 1 , 2 , . .
Revised: Initialization: Set τ 0 ← ∞ and initialize DNNs f θ 0 and Q ω 0 as speciﬁed in § A. for k = 0 , 1 , 2 , . .

Original: K do
Revised: K do

Original: Actor Update: Update θ k + via Algorithm 3 with input π θ k , θ 0 , Q ω k , α , β , τ k +1 =
Revised: Actor Update: Update θ k + via Algorithm 3 with input π θ k , θ 0 , Q ω k , α , β , τ k +1 =

Original: ( k +1)
Revised: ( k +1)

Original: − 1 · β , and N a .
Revised: − 1 · β , and N a .

Original: Critic Update: Update ω k + via Algorithm 4 with input π θ k +1 , Q ω k , ω 0 , η , and N c .
Revised: Critic Update: Update ω k + via Algorithm 4 with input π θ k +1 , Q ω k , ω 0 , η , and N c .

Original: end for
Revised: end for

Original: Output: { π θ k } k ∈ [ K +1] , where π θ k ∝ exp( τ − 1 k f θ k ) .
Revised: Output: { π θ k } k ∈ [ K +1] , where π θ k ∝ exp( τ − 1 k f θ k ) .

Original: Algorithm
Revised: Algorithm

Original: Input: Policy π θ ∝ exp( τ − 1 f θ ) , initial actor parameter θ 0 , action-value function Q ω , stepsize α , temperature parameter β , temperature (cid:101) τ , and number of iterations N a .
Revised: Input: Policy π θ ∝ exp( τ − 1 f θ ) , initial actor parameter θ 0 , action-value function Q ω , stepsize α , temperature parameter β , temperature (cid:101) τ , and number of iterations N a .

Original: Initialization: Set θ (0) ← θ 0 .
Revised: Initialization: Set θ (0) ← θ 0 .

Original: for n = 0 , 1 , 2 , . . .
Revised: for n = 0 , 1 , 2 , . . .

Original: , N a − 1 do
Revised: , N a − 1 do

Original: Sample ( s, a ) as speciﬁed in § A.
Revised: Sample ( s, a ) as speciﬁed in § A.

Original: Set θ ( n + 1) ← Γ B ( θ 0 ,R a ) ( θ ( n ) − α · ( f θ ( n ) ( s, a ) − (cid:101) τ · ( β − 1 Q ω ( s, a ) + τ − 1 f θ ( s, a ))) · ∇ θ f θ ( n ) ( s, a )) .
Revised: Set θ ( n + 1) ← Γ B ( θ 0 ,R a ) ( θ ( n ) − α · ( f θ ( n ) ( s, a ) − (cid:101) τ · ( β − 1 Q ω ( s, a ) + τ − 1 f θ ( s, a ))) · ∇ θ f θ ( n ) ( s, a )) .

Original: end for
Revised: end for

Original: Output: θ = 1 /N a · (cid:80) N a n =1 θ ( n ) .
Revised: Output: θ = 1 /N a · (cid:80) N a n =1 θ ( n ) .

Original: C C ONVERGENCE R ESULTS OF A LGORITHM
Revised: C C ONVERGENCE R ESULTS OF A LGORITHM

Original: In this section, we upper bound the regret of the deep neural actor-critic method.
Revised: In this section, we upper bound the regret of the deep neural actor-critic method.

Original: Hereafter we assume that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A , where r max is a positive absolute constant.
Revised: Hereafter we assume that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A , where r max is a positive absolute constant.

Original: First, we impose the following assumptions in parallel to Assumption 4.1.
Revised: First, we impose the following assumptions in parallel to Assumption 4.1.

Original: Recall that ρ ∗ is the stationary state-action distribution of π ∗ , while ρ k is the stationary state-action distribution of π θ k .
Revised: Recall that ρ ∗ is the stationary state-action distribution of π ∗ , while ρ k is the stationary state-action distribution of π θ k .

Original: Assumption C.1 (Concentrability Coefﬁcient) .
Revised: Assumption C.1 (Concentrability Coefﬁcient) .

Original: The following statements hold.
Revised: The following statements hold.

Original: (i)
Revised: (i)

Original: There exists a positive absolute constant φ ∗ such that φ ∗ k ≤ φ ∗ for any k ≥ 1 , where φ ∗ k = (cid:107) d ρ ∗ / d ρ k (cid:107) ρ k , 2 .
Revised: There exists a positive absolute constant φ ∗ such that φ ∗ k ≤ φ ∗ for any k ≥ 1 , where φ ∗ k = (cid:107) d ρ ∗ / d ρ k (cid:107) ρ k , 2 .

Original: (ii)
Revised: (ii)

Original: For the state-action distribution ρ used to deﬁne the regret in (4.1), we assume that for any k ≥ 1 and a sequence of policies {
Revised: For the state-action distribution ρ used to deﬁne the regret in (4.1), we assume that for any k ≥ 1 and a sequence of policies {

Original: Input: Policy π θ , action-value function Q ω , initial critic parameter ω 0 , stepsize η , and number of iterations N c .
Revised: Input: Policy π θ , action-value function Q ω , initial critic parameter ω 0 , stepsize η , and number of iterations N c .

Original: Initialization: Set ω (0) ← ω 0 .
Revised: Initialization: Set ω (0) ← ω 0 .

Original: for n = 0 , 1 , 2 , . . .
Revised: for n = 0 , 1 , 2 , . . .

Original: , N c − 1 do
Revised: , N c − 1 do

Original: Sample ( s, a, r, s (cid:48) , a (cid:48) ) as speciﬁed in § A.
Revised: Sample ( s, a, r, s (cid:48) , a (cid:48) ) as speciﬁed in § A.

Original: Set δ ( n ) ← Q ω ( n ) ( s, a ) − (1 − γ ) · r − γ · Q ω ( s (cid:48) , a (cid:48) ) .
Revised: Set δ ( n ) ← Q ω ( n ) ( s, a ) − (1 − γ ) · r − γ · Q ω ( s (cid:48) , a (cid:48) ) .

Original: Set ω ( n + 1) ← Γ B ( ω 0 ,R c ) ( ω ( n ) − η · δ ( n ) · ∇ ω Q ω ( n ) ( s, a )) .
Revised: Set ω ( n + 1) ← Γ B ( ω 0 ,R c ) ( ω ( n ) − η · δ ( n ) · ∇ ω Q ω ( n ) ( s, a )) .

Original: end for
Revised: end for

Original: Output: ω = 1 /N c · (cid:80) N c n =1 ω ( n ) .
Revised: Output: ω = 1 /N c · (cid:80) N c n =1 ω ( n ) .

Original: ρ P π 1 · · · P π k is absolutely continuous with respect to ρ ∗ .
Revised: ρ P π 1 · · · P π k is absolutely continuous with respect to ρ ∗ .

Original: Also, it holds that
Revised: Also, it holds that

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: = (1 − γ )∞ (cid:88) k
Revised: = (1 − γ )∞ (cid:88) k

Original: = k 3 γ k · c ( k ) < ∞ , where c ( k )
Revised: = k 3 γ k · c ( k ) < ∞ , where c ( k )

Original: =
Revised: =

Original: sup { π i } i ∈
Revised: sup { π i } i ∈

Original: [ k ] (cid:107) d( ρ P π 1 · · · P π k ) /
Revised: [ k ] (cid:107) d( ρ P π 1 · · · P π k ) /

Original: d ρ ∗ (cid:107) ρ ∗ , ∞ .
Revised: d ρ ∗ (cid:107) ρ ∗ , ∞ .

Original: Meanwhile, we impose the following assumption in parallel to Assumption 4.2.
Revised: Meanwhile, we impose the following assumption in parallel to Assumption 4.2.

Original: Assumption C.2 (Zero Approximation Error) .
Revised: Assumption C.2 (Zero Approximation Error) .

Original: For any Q ω ∈ U ( m c , H c , R c ) and policy π , it holds that T π Q ω ∈ U ( m c , H c , R c ) , where T π is deﬁned in (2.4).
Revised: For any Q ω ∈ U ( m c , H c , R c ) and policy π , it holds that T π Q ω ∈ U ( m c , H c , R c ) , where T π is deﬁned in (2.4).

Original: Assumption C.2 states that U ( m c , H c , R c ) is closed under the Bellman evaluation operator T π , which is commonly imposed in the literature (Munos and Szepesv ´ ari, 2008; Antos et al., 2008a;
Revised: Assumption C.2 states that U ( m c , H c , R c ) is closed under the Bellman evaluation operator T π , which is commonly imposed in the literature (Munos and Szepesv ´ ari, 2008; Antos et al., 2008a;

Original: Farahmand et al., 2010; 2016; Tosatto et al., 2017; Yang et al., 2019b; Liu et al., 2019).
Revised: Farahmand et al., 2010; 2016; Tosatto et al., 2017; Yang et al., 2019b; Liu et al., 2019).

Original: We upper bound the regret of the deep neural actor-critic method in Algorithm 2 in the sequel.
Revised: We upper bound the regret of the deep neural actor-critic method in Algorithm 2 in the sequel.

Original: To establish such an upper bound, we ﬁrst establish the rates of convergence of Algorithms 3 and 4 as follows.
Revised: To establish such an upper bound, we ﬁrst establish the rates of convergence of Algorithms 3 and 4 as follows.

Original: Proposition
Revised: Proposition

Original: C.3.
Revised: C.3.

Original: For any sufﬁciently large
Revised: For any sufﬁciently large

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: , let
Revised: , let

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We denote by θ the output of Algorithm 3 with input π θ ∝ exp
Revised: We denote by θ the output of Algorithm 3 with input π θ ∝ exp

Original: ( τ − 1 f θ ) , θ 0 , Q ω , α , β , (cid:101) τ
Revised: ( τ − 1 f θ ) , θ 0 , Q ω , α , β , (cid:101) τ

Original: = ( τ − 1 + β − 1 ) − 1 , and N a .
Revised: = ( τ − 1 + β − 1 ) − 1 , and N a .

Original: Also, let (cid:101) f
Revised: Also, let (cid:101) f

Original: = (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) .
Revised: = (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) .

Original: With probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 

Revised: With probability at least 1 − exp( − Ω( R 2 / 3 a m 2 /

Original: a H a )) over the random initialization θ 0 , we have
Revised: a H a )) over the random initialization θ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Here the expectation is taken over the randomness of θ conditioning on the initialization θ 0 and ( s, a ) ∼ ρ π θ , where ρ π θ is the stationary state-action distribution of π θ .
Revised: Here the expectation is taken over the randomness of θ conditioning on the initialization θ 0 and ( s, a ) ∼ ρ π θ , where ρ π θ is the stationary state-action distribution of π θ .

Original: Proof.
Revised: Proof.

Original: See § G.2 for a detailed proof.
Revised: See § G.2 for a detailed proof.

Original: Proposition
Revised: Proposition

Original: C.4.
Revised: C.4.

Original: For any sufﬁciently large
Revised: For any sufﬁciently large

Original: N c > 0 , let m c
Revised: N c > 0 , let m c

Original: = Ω( d 3 / 2 R − 1 c
Revised: = Ω( d 3 / 2 R − 1 c

Original: H − 3 / 2 c log( m 1 / 2 c /R c ) 3 / 2 ) , H c
Revised: H − 3 / 2 c log( m 1 / 2 c /R c ) 3 / 2 ) , H c

Original: = O ( N 1 / 4 c ) , and
Revised: = O ( N 1 / 4 c ) , and

Original: [Equation]
Revised: [Equation]

Original: We denote by ω the output of Algorithm 4 with input π θ , Q ω , ω 0 , η , and N c .
Revised: We denote by ω the output of Algorithm 4 with input π θ , Q ω , ω 0 , η , and N c .

Original: Also, let (cid:101) Q
Revised: Also, let (cid:101) Q

Original: = (1 − γ ) ·
Revised: = (1 − γ ) ·

Original: r + γ · P π θ Q ω .
Revised: r + γ · P π θ Q ω .

Original: With probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) over the random initialization ω 0 , we have
Revised: With probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) over the random initialization ω 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Here the expectation is taken over the randomness of ω conditioning on the initialization ω 0 and ( s, a ) ∼ ρ π θ , where ρ π θ is the stationary state-action distribution of π θ .
Revised: Here the expectation is taken over the randomness of ω conditioning on the initialization ω 0 and ( s, a ) ∼ ρ π θ , where ρ π θ is the stationary state-action distribution of π θ .

Original: Proof.
Revised: Proof.

Original: See § G.3 for a detailed proof.
Revised: See § G.3 for a detailed proof.

Original: Propositions C.3 and C.4 characterize the errors that arise from the actor and critic updates in Algorithm 2, respectively.
Revised: Propositions C.3 and C.4 characterize the errors that arise from the actor and critic updates in Algorithm 2, respectively.

Original: In particular, if the widths m a and m c of the DNNs f θ and Q ω are sufﬁciently large, the errors characterized in Propositions C.3 and C.4 decay to zero at the rates of O ( N − 1 / 2 a ) and O ( N − 1 / 2 c ) , respectively.
Revised: In particular, if the widths m a and m c of the DNNs f θ and Q ω are sufﬁciently large, the errors characterized in Propositions C.3 and C.4 decay to zero at the rates of O ( N − 1 / 2 a ) and O ( N − 1 / 2 c ) , respectively.

Original: Propositions C.3 and C.4 act as the key ingredients to upper bounding the regret of the deep neural actor-critic method.
Revised: Propositions C.3 and C.4 act as the key ingredients to upper bounding the regret of the deep neural actor-critic method.

Original: Based on Propositions C.3 and C.4, we upper bound the regret of Algorithm 2 in the following theorem, which is in parallel to Theorem 4.4.
Revised: Based on Propositions C.3 and C.4, we upper bound the regret of Algorithm 2 in the following theorem, which is in parallel to Theorem 4.4.

Original: Theorem C.5.
Revised: Theorem C.5.

Original: We assume that Assumptions C.1 and C.2 hold.
Revised: We assume that Assumptions C.1 and C.2 hold.

Original: Let ρ be a state-action distribution satisfying (ii) of Assumption C.1.
Revised: Let ρ be a state-action distribution satisfying (ii) of Assumption C.1.

Original: Also, for any sufﬁciently large K > 0 , let N a = Ω( K 2 C 4 ρ,ρ ∗ ( φ ∗ + ψ ∗ + 1) 4 R 4a ) , N c
Revised: Also, for any sufﬁciently large K > 0 , let N a = Ω( K 2 C 4 ρ,ρ ∗ ( φ ∗ + ψ ∗ + 1) 4 R 4a ) , N c

Original: = Ω( K 2 C 4 ρ,ρ ∗ φ ∗ 4 R 4c ) , H a
Revised: = Ω( K 2 C 4 ρ,ρ ∗ φ ∗ 4 R 4c ) , H a

Original: = H c
Revised: = H c

Original: = O ( N 1 / 4 c ) , R a = R c =
Revised: = O ( N 1 / 4 c ) , R a = R c =

Original: O ( m 1 / 2 c
Revised: O ( m 1 / 2 c

Original: H − 6 c (log m c ) − 3 ) , m a = m c
Revised: H − 6 c (log m c ) − 3 ) , m a = m c

Original: = Ω( d 3 / 2 K 6 C 12 ρ,ρ ∗ ( φ ∗ + ψ ∗ +1) 12 R 16c H 42c log( m 1 / 2 c /R c ) 3 / 2 ) , β = K 1 / 2 , and the sequence { θ k } k ∈ [ K ] be generated by Algorithm 2.
Revised: = Ω( d 3 / 2 K 6 C 12 ρ,ρ ∗ ( φ ∗ + ψ ∗ +1) 12 R 16c H 42c log( m 1 / 2 c /R c ) 3 / 2 ) , β = K 1 / 2 , and the sequence { θ k } k ∈ [ K ] be generated by Algorithm 2.

Original: With probability at least 1 − 1 /K over the random initialization θ 0 and ω 0 , it holds that
Revised: With probability at least 1 − 1 /K over the random initialization θ 0 and ω 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken over the randomness of ( s, a ) ∼ ρ and { θ k +1 } k ∈ [ K ] conditioning on the initialization θ 0 and ω 0 .
Revised: where the expectation is taken over the randomness of ( s, a ) ∼ ρ and { θ k +1 } k ∈ [ K ] conditioning on the initialization θ 0 and ω 0 .

Original: Proof.
Revised: Proof.

Original: See § E.2 for a detailed proof.
Revised: See § E.2 for a detailed proof.

Original: When the architecture of the actor and critic neural networks are properly chosen, Theorem C. establishes an O ( K 1 / 2 ) regret of Algorithm 2, where K is the total number of iterations.
Revised: When the architecture of the actor and critic neural networks are properly chosen, Theorem C. establishes an O ( K 1 / 2 ) regret of Algorithm 2, where K is the total number of iterations.

Original: Speciﬁcally speaking, to establish such a regret upper bound, we need the widths m a and m c of the DNNs f θ and Q ω to be sufﬁciently large.
Revised: Speciﬁcally speaking, to establish such a regret upper bound, we need the widths m a and m c of the DNNs f θ and Q ω to be sufﬁciently large.

Original: Meanwhile, to control the errors of actor update and critic update in Algorithm 2, we also run sufﬁciently large numbers of iterations in Algorithms 3 and 4.
Revised: Meanwhile, to control the errors of actor update and critic update in Algorithm 2, we also run sufﬁciently large numbers of iterations in Algorithms 3 and 4.

Revised: In terms of the total sample complexity, to simplify our discussion, we omit constant and logarithmic terms here.

Revised: To obtain an ε -globally optimal policy, it sufﬁces to set K (cid:16) ε − 2 in Algorithm 2.

Revised: By plugging such a K into N a = Ω( K 2 C 4 ρ,ρ ∗ ( φ ∗ + ψ ∗ + 1) 4 R 4a ) and N c =

Revised: Ω( K 2 C 4 ρ,ρ ∗ φ ∗ 4 R 4c ) as required in Theorem C.5, we have N a = (cid:101)

Original: To the best of our knowledge, we establish the rate of convergence and global optimality of the actor-critic method with DNN approximation for the ﬁrst time.
Revised: O ( ε − 4 ) and N c = (cid:101) O ( ε − 4 ) . Thus, to achieve an ε -globally optimal policy, the total sample complexity of Algorithm 2 is (cid:101) O ( ε − 6 ) .

Revised: With the modiﬁcation to offpolicy setting as in § 3.1, the total sample complexity of Algorithm 2 is (cid:101) O

Revised: ( ε − 4 ) .

Revised: In comparison, Liu et al.

Revised: (2019) requires a total sample complexity of (cid:101)

Revised: O ( ε − 8 ) to achieve an ε -globally optimal policy, which is worse than our single-timescale algorithm.

Revised: Meanwhile, since Liu et al.

Revised: (2019) uses TD(0) in the critic update, which is shown to diverge under off-policy setting even with linear function approximation (Baird et al., 1995), the method of data reuse cannot be applied to Liu et al.

Revised: (2019) to eliminate the total sample complexity.

Revised: To the best of our knowledge, we establish the rate of convergence and global optimality of the actor-critic method under single-timescale setting with DNN approximation for the ﬁrst time.

Original: D P ROOF S KETCH OF M AIN T HEOREM 4. 
Revised: D P ROOF S KETCH OF M AIN T HEOREM 4.

Original: In this section, we sketch the proof of Theorem 4.4.
Revised: In this section, we sketch the proof of Theorem 4.4.

Original: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.
Revised: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.

Original: We ﬁrst upper bound (cid:80)
Revised: We ﬁrst upper bound (cid:80)

Original: Kk =

Revised: Kk =

Original: ( Q ∗ ( s, a ) − Q π θk +

Revised: ( Q ∗ ( s, a ) − Q π θk +

Original: ( s, a ))
Revised: ( s, a ))

Original: for any ( s, a ) ∈ S × A in part 1.
Revised: for any ( s, a ) ∈ S × A in part 1.

Original: Then by further taking the expectation over ρ in part 2, we conclude the proof of Theorem 4.4.
Revised: Then by further taking the expectation over ρ in part 2, we conclude the proof of Theorem 4.4.

Original: See § E.1 for a detailed proof.
Revised: See § E.1 for a detailed proof.

Original: Part 1.
Revised: Part 1.

Original: In the sequel, we upper bound (cid:80) Kk =

Revised: In the sequel, we upper bound (cid:80) Kk =

Original: ( Q ∗ ( s, a ) − Q π θk +

Revised: ( Q ∗ ( s, a ) − Q π θk +

Original: ( s, a ))
Revised: ( s, a ))

Original: for any ( s, a ) ∈ S × A .
Revised: for any ( s, a ) ∈ S × A .

Original: We ﬁrst decompose Q ∗ − Q π θ k +1 into the following three terms,
Revised: We ﬁrst decompose Q ∗ − Q π θk +1 into the following three terms,

Original: [Equation]
Revised: [Equation]

Original: the proof of which is deferred to (E.1) and (E.2) in § E.1 of the appendix.
Revised: the proof of which is deferred to (E.1) and (E.2) in § E.1 of the appendix.

Original: Here the operator P π ∗ is deﬁned in (2.3), ( I − γ P [Equation] i = [Equation]{+(cid:80) ∞ i =
+}
Revised: Here the operator P π ∗ is deﬁned in (2.3), ( I − γ P π ∗ ) − 1 = {+(cid:80) ∞ i =
+}

Original:  γ P [Equation] ∗ ) i , and A 1 ,k , A 2 ,k , and A 3 ,k are deﬁned as follows, 
Revised: ( γ P π ∗ ) i , and A 1 ,k , A 2 ,k , and A 3 ,k are deﬁned as follows, [Equation]

Original: [Equation]
Revised: [Equation]

Original: To understand the intuition behind A 1 ,k , A 2 ,k , and A 3 ,k , we interpret them as follows.
Revised: To understand the intuition behind A 1 ,k , A 2 ,k , and A 3 ,k , we interpret them as follows.

Original: Interpretation of A 1 ,k .
Revised: Interpretation of A 1 ,k .

Original: As deﬁned in (D.2), A 1 ,k arises from the actor update and measures the convergence of the policy π θ k +1 towards a globally optimal policy π ∗ , which implies the convergence of P π θ k +1 towards P π ∗ .
Revised: As deﬁned in (D.2), A 1 ,k arises from the actor update and measures the convergence of the policy π θ k +1 towards a globally optimal policy π ∗ , which implies the convergence of P π θk +1 towards P π ∗ .

Original: Interpretation of A 3 ,k .
Revised: Interpretation of A 3 ,k .

Original: Note that by (2.2) and (2.4), we have Q π θk +

Revised: Note that by (2.2) and (2.4), we have Q π θk +

Original: =
Revised: =

Original: T π θk +1 Q π θk +

Revised: T π θk +1 Q π θk +

Original: and T π θk
Revised: and T π θk

Original: + is a γ -contraction, which implies that applying the Bellman evaluation operator T π θk +1 to any Q , e.g., Q ω k , inﬁnite times yields Q π θk +1 .
Revised: + is a γ -contraction, which implies that applying the Bellman evaluation operator T π θk +1 to any Q , e.g., Q ω k , inﬁnite times yields Q π θk +1 .

Original: As deﬁned in (D.4), A 3 ,k measures the error of tracking the action-value function Q π θk +1 of π θ k +1 by applying the Bellman evaluation operator T π θk +1 to
Revised: As deﬁned in (D.4), A 3 ,k measures the error of tracking the action-value function Q π θk +1 of π θ k +1 by applying the Bellman evaluation operator T π θk +1 to

Original: Q ω k only once, which arises from the critic update.
Revised: Q ω k only once, which arises from the critic update.

Original: Also, as A 3 ,k = T π θk +1 ( Q ω k − Q π θk +1 ) , A 3 ,k measures the difference between Q π θk , which is approximated by Q ω k as discussed subsequently, and Q π θk +1 .
Revised: Also, as A 3 ,k = T π θk +1 ( Q ω k − Q π θk +1 ) , A 3 ,k measures the difference between Q π θk , which is approximated by Q ω k as discussed subsequently, and Q π θk +1 .

Original: Such a difference can also be viewed as the difference between π θ k and π θ k +1 , which arises from the actor update.
Revised: Such a difference can also be viewed as the difference between π θ k and π θ k +1 , which arises from the actor update.

Original: Therefore, the convergence of A 3 ,k to zero implies the contractions of not only the critic update but also the actor update, which illustrates the “double contraction” phenomenon.
Revised: Therefore, the convergence of A 3 ,k to zero implies the contractions of not only the critic update but also the actor update, which illustrates the “double contraction” phenomenon.

Original: We establish the convergence of A 3 ,k to zero in (D.10) subsequently.
Revised: We establish the convergence of A 3 ,k to zero in (D.10) subsequently.

Original: Interpretation of A 2 ,k .
Revised: Interpretation of A 2 ,k .

Original: Assuming that A 3 ,k − 1 converges to zero, we have T π θk Q ω k − 1 ≈ Q π θk .
Revised: Assuming that A 3 ,k − 1 converges to zero, we have T π θk Q ω k − 1 ≈ Q π θk .

Original: Moreover, assuming that the number of data points N is sufﬁciently large and ignoring the projection in (3.5), we have T π θk Q ω k − 1 = Q (cid:101) ω
Revised: Moreover, assuming that the number of data points N is sufﬁciently large and ignoring the projection in (3.5), we have T π θk Q ω k − 1 = Q (cid:101) ω

Original: k ≈ Q ω k as (cid:101) ω
Revised: k ≈ Q ω k as (cid:101) ω

Original: k deﬁned in (3.4) is an estimator of ω k .
Revised: k deﬁned in (3.4) is an estimator of ω k .

Original: Hence, we have Q π θk ≈ Q ω k .
Revised: Hence, we have Q π θk ≈ Q ω k .

Original: Such an approximation error is characterized by (cid:15)
Revised: Such an approximation error is characterized by (cid:15)

Original: c k deﬁned in (D.5) subsequently.
Revised: c k deﬁned in (D.5) subsequently.

Original: Hence, A 2 ,k measures the difference between π θ k and π θ k +1 through the difference between Q π θk ≈ Q ω k and Q π θk +1 , which relies on the convergence of A 3 ,k − 1 to zero.Q ω k Q ω k +
Revised: Hence, A 2 ,k measures the difference between π θ k and π θ k +1 through the difference between Q π θk ≈ Q ω k and Q π θk +1 , which relies on the convergence of A 3 ,k − 1 to zero.

Original: T π θ k + 1 Q ω k T π θ k Q ω k −

Original: T π ∗ Q ω k

Original: Q π θ k

Original: Q π θ k +

Original: T π ∗ Q ω k

Original: +

Original: T π ∗ Q π θ k T π ∗ Q π θ k −ε c k

Original: A 3 , k −

Original: A 2 , k −

Original: A 1 , k e k +ε c k +

Original: A 1 , k +

Original: A 3 , k

Original: A 2 , k ϑ k

Original: Critic Update:

Original: Actor Update:

Original: In the sequel, we upper bound A 1 ,k , A 2 ,k , and A 3 ,k , respectively.
Revised: In the sequel, we upper bound A 1 ,k , A 2 ,k , and A 3 ,k , respectively.

Original: To establish such upper bounds, we deﬁne the following quantities,
Revised: To establish such upper bounds, we deﬁne the following quantities,

Original: [Equation]
Revised: [Equation]

Original: To understand the intuition behind (cid:15)
Revised: To understand the intuition behind (cid:15)

Original: c
Revised: c

Original: k +1 , e k +1 , and ϑ k , we interpret them as follows.
Revised: k +1 , e k +1 , and ϑ k , we interpret them as follows.

Original: Interpretation of (cid:15) c
Revised: Interpretation of (cid:15) c

Original: k +1 .
Revised: k +1 .

Original: Recall that (cid:101)
Revised: Recall that (cid:101)

Original: ω
Revised: ω

Original: k +1 is deﬁned in (3.4), which parameterizes T π θk +1 Q ω k (ignoring the projection in (3.5)).
Revised: k +1 is deﬁned in (3.4), which parameterizes T π θ k +1 Q ω k (ignoring the projection in (3.5)).

Original: Here (cid:15) c k +1 arises from approximating (cid:101)
Revised: Here (cid:15) c k +1 arises from approximating (cid:101)

Original: ω
Revised: ω

Original: k +1 using ω k +1 as an estimator, which is constructed based on ω k and the N data points.
Revised: k +1 using ω k +1 as an estimator, which is constructed based on ω k and the N data points.

Original: In particular, (cid:15) c
Revised: In particular, (cid:15) c

Original: k +1 decreases to zero as
Revised: k +1 decreases to zero as

Original: N
Revised: N

Original: → ∞ , which is used in characterizing A 2 ,k deﬁned in (D.3).
Revised: → ∞ , which is used in characterizing A 2 ,k deﬁned in (D.3).

Original: Interpretation of e k +1 .
Revised: Interpretation of e k +1 .

Original: Assuming that A 3 ,k − 1 deﬁned in (D.4) and (cid:15)
Revised: Assuming that A 3 ,k − 1 deﬁned in (D.4) and (cid:15)

Original: c k deﬁned in (D.5) converge to zero, which implies T π θk Q ω k − 1 ≈ Q π θk and T π θk Q ω k − 1 ≈ Q ω k , respectively, we have Q ω k
Revised: c k deﬁned in (D.5) converge to zero, which implies T π θk Q ω k − 1 ≈ Q π θk and T π θk Q ω k − 1 ≈ Q ω k , respectively, we have Q ω k

Original: ≈
Revised: ≈

Original: Q π θk .
Revised: Q π θk .

Original: Therefore, as deﬁned in (D.6), e k +

Revised: Therefore, as deﬁned in (D.6), e k +

Original: =
Revised: =

Original: Q ω k − T π θk +1 Q ω
Revised: Q ω k − T π θk +1 Q ω

Original: k ≈ Q π θk − T π θk +1 Q π θk
Revised: k ≈ Q π θk − T π θk +1 Q π θk

Original: =
Revised: =

Original: ( T π θ k − T π θ k +1 )
Revised: ( T π θk − T π θk +1 )

Original: Q π θk measures the difference between π θ k and π θ k +1 , which implies the difference between T π θk and T π θk +1 .
Revised: Q π θk measures the difference between π θ k and π θ k +1 , which implies the difference between T π θk and T π θk +1 .

Original: We remark that e k +1 fully characterizes A 3 ,k deﬁned in (D.4) as shown in (D.8) subsequently.
Revised: We remark that e k +1 fully characterizes A 3 ,k deﬁned in (D.4) as shown in (D.8) subsequently.

Original: Interpretation of ϑ k .
Revised: Interpretation of ϑ k .

Original: As deﬁned in (D.7), ϑ k measures the difference between π θ k and π θ k +1 in terms of their differences with π ∗ , which are measured by the corresponding KL-divergences.
Revised: As deﬁned in (D.7), ϑ k measures the difference between π θ k and π θ k +1 in terms of their differences with π ∗ , which are measured by the corresponding KL-divergences.

Original: In particular, ϑ k is used in characterizing A 1 ,k and A 2 ,k deﬁned in (D.2) and (D.3), respectively.
Revised: In particular, ϑ k is used in characterizing A 1 ,k and A 2 ,k deﬁned in (D.2) and (D.3), respectively.

Original: We remark that (cid:15) c k +1 measures the statistical error in the critic update, while ϑ k measures the optimization error in the actor update.
Revised: We remark that (cid:15) c k +1 measures the statistical error in the critic update, while ϑ k measures the optimization error in the actor update.

Original: As discussed above, the convergence of A 3 ,k to zero implies the contraction of both the actor update and the critic update, which illustrates the “double contraction” phenomenon.
Revised: As discussed above, the convergence of A 3 ,k to zero implies the Q ω k Q ω k +

Revised: T π θ k + 1 Q ω k T π θ k Q ω k −

Revised: T π ∗ Q ω k

Revised: Q π θ k

Revised: Q π θ k +

Revised: T π ∗ Q ω k

Revised: +

Revised: T π ∗ Q π θ k T π ∗ Q π θ k −ε c k

Revised: A 3 , k −

Revised: A 2 , k −

Revised: A 1 , k e k +ε c k +

Revised: A 1 , k +

Revised: A 3 , k

Revised: A 2 , k ϑ k

Revised: Critic Update:

Revised: Actor Update: contraction of both the actor update and the critic update, which illustrates the “double contraction” phenomenon.

Original: Meanwhile, since e k +1 fully characterizes A 3 ,k as shown in (D.8) subsequently, e k + plays a key role in the “double contraction” phenomenon.
Revised: Meanwhile, since e k +1 fully characterizes A 3 ,k as shown in (D.8) subsequently, e k + plays a key role in the “double contraction” phenomenon.

Original: In particular, the convergence of e k +1 to zero is established in (D.9) subsequently.
Revised: In particular, the convergence of e k +1 to zero is established in (D.9) subsequently.

Original: See Figure 1 for an illustration of these quantities.
Revised: See Figure 1 for an illustration of these quantities.

Original: With the quantities deﬁned in (D.5), (D.6), and (D.7), we upper bound A 1 ,k , A 2 ,k , and A 3 ,k as follows,[Equation] D.
Revised: With the quantities deﬁned in (D.5), (D.6), and (D.7), we upper bound A 1 ,k , A 2 ,k , and A 3 ,k as follows,

Original: [Equation]
Revised: [Equation]

Original: the proof of which is deferred to Lemmas E.1, E.2, and E.3 in § E.1 of the appendix, respectively.
Revised: the proof of which is deferred to Lemmas E.1, E.2, and E.3 in § E.1 of the appendix, respectively.

Original: Meanwhile, by recursively expanding (D.5) and (D.6), we have
Revised: Meanwhile, by recursively expanding (D.5) and (D.6), we have

Original: [Equation]
Revised: [Equation]

Original: the proof of which is deferred to Lemma E.4 in § E.1 of the appendix.
Revised: the proof of which is deferred to Lemma E.4 in § E.1 of the appendix.

Original: By plugging (D.9) into (D.8), we have
Revised: By plugging (D.9) into (D.8), we have

Original: [Equation]
Revised: [Equation]

Original: To better understand (D.10) and how it relates to the convergence of A 3 ,k , A 2 ,k , and A 1 ,k to zero, we discuss in the following two steps.
Revised: To better understand (D.10) and how it relates to the convergence of A 3 ,k , A 2 ,k , and A 1 ,k to zero, we discuss in the following two steps.

Original: Step (i).
Revised: Step (i).

Original: We assume (cid:15) c i = 0 , which corresponds to the number of data points N → ∞ .
Revised: We assume (cid:15) c i = 0 , which corresponds to the number of data points N → ∞ .

Original: Then (D.10) yields A 3 ,k
Revised: Then (D.10) yields A 3 ,k

Original: = O ( γ k ) , which implies that A 3 ,k deﬁned in (D.4) converges to zero driven by the discount factor γ .
Revised: = O ( γ k ) , which implies that A 3 ,k deﬁned in (D.4) converges to zero driven by the discount factor γ .

Original: As discussed above, the convergence of A 3 ,k to zero also implies the contraction between π θ k and π θ k +1 of the actor update and the contraction between Q ω k and Q π θk of the critic update, which illustrates the “double contraction” phenomenon.
Revised: As discussed above, the convergence of A 3 ,k to zero also implies the contraction between π θ k and π θ k +1 of the actor update and the contraction between Q ω k and Q π θk of the critic update, which illustrates the “double contraction” phenomenon.

Original: Step (ii).
Revised: Step (ii).

Original: The convergence of A 3 ,k to zero further ensures that A 2 ,k converges to zero.
Revised: The convergence of A 3 ,k to zero further ensures that A 2 ,k converges to zero.

Original: To see this, we further assume
Revised: To see this, we further assume

Original: A 3 ,k = 0 , which together with the assumption that (cid:15) c k +1 = 0 implies
Revised: A 3 ,k = 0 , which together with the assumption that (cid:15) c k +1 = 0 implies

Original: Q π θk +1 =
Revised: Q π θk +1 =

Original: T π θk +1 Q ω
Revised: T π θk +1 Q ω

Original: k = Q ω k +1 by their deﬁnitions in (D.4) and (D.5), respectively.
Revised: k = Q ω k +1 by their deﬁnitions in (D.4) and (D.5), respectively.

Original: Then by telescoping the sum of A 2 ,k deﬁned in (D.3), which cancels out Q ω k +1 and Q π θk +1 , we obtain the convergence of A 2 ,k to zero.
Revised: Then by telescoping the sum of A 2 ,k deﬁned in (D.3), which cancels out Q ω k +1 and Q π θk +1 , we obtain the convergence of A 2 ,k to zero.

Original: Meanwhile, telescoping the sum of A 1 ,k deﬁned in (D.2) and the sum of its upper bound in (D.8) implies that A 1 ,k converges to zero.
Revised: Meanwhile, telescoping the sum of A 1 ,k deﬁned in (D.2) and the sum of its upper bound in (D.8) implies that A 1 ,k converges to zero.

Original: Now, by plugging (D.8) and (D.10) into (D.1), we establish an upper bound of (cid:80) Kk =

Revised: Now, by plugging (D.8) and (D.10) into (D.1), we establish an upper bound of (cid:80) Kk =

Original: ( Q ∗ ( s, a ) −
Revised: ( Q ∗ ( s, a ) −

Original: Q π [-θ k +
-] ( s, a )) for any ( s, a ) ∈ S × A , which is deferred to (E.12) in § E.1 of the appendix.
Revised: Q π [-θ k +
-]θk +1 ( s, a )) for any ( s, a ) ∈ S × A , which is deferred to (E.12) in § E.1 of the appendix.

Original: Hence, we conclude the proof in part 1.
Revised: Hence, we conclude the proof in part 1.

Original: See part 1 of § E.1 for details.
Revised: See part 1 of § E.1 for details.

Original: Part 2.
Revised: Part 2.

Original: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.
Revised: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.

Original: In the sequel, we take the expectation over ρ in (E.12) and upper bound each term.
Revised: In the sequel, we take the expectation over ρ in (E.12) and upper bound each term.

Original: We ﬁrst introduce the following lemma, which upper bounds (cid:15) c
Revised: We ﬁrst introduce the following lemma, which upper bounds (cid:15) c

Original: k +1 deﬁned in (D.5).
Revised: k +1 deﬁned in (D.5).

Original: Lemma D.1.
Revised: Lemma D.1.

Original: Under Assumptions 4.2 and 4.3, it holds for any k ≥ 1 that
Revised: Under Assumptions 4.2 and 4.3, it holds for any k ≥ 1 that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to randomness of ω k +1 and ( s, a ) ∼ ρ k
Revised: where the expectation is taken with respect to randomness of ω k +1 and ( s, a ) ∼ ρ k

Original: +1 .
Revised: +1 .

Original: Proof.
Revised: Proof.

Original: See § H.1 for a detailed proof.
Revised: See § H.1 for a detailed proof.

Original: On the right-hand side of (E.12) in § E.1 of the appendix, for the terms not involving (cid:15) c k +1 , i.e., M 1 ,
Revised: On the right-hand side of (E.12) in § E.1 of the appendix, for the terms not involving (cid:15) c k +1 , i.e., M 1 ,

Original: M 2 , and M 3 in (E.13), we take the expectation over ρ and establish their upper bounds in the (cid:96) ∞ - norm over ( s, a ) in Lemma E.5.
Revised: M 2 , and M 3 in (E.13), we take the expectation over ρ and establish their upper bounds in the (cid:96) ∞ - norm over ( s, a ) in Lemma E.5.

Original: On the other hand, for the terms involving (cid:15) c k +1 , i.e., M 4 and M 5 in (E.14), we take the expectation over ρ and then change the measure from ρ to ρ k +1 .
Revised: On the other hand, for the terms involving (cid:15) c k +1 , i.e., M 4 and M 5 in (E.14), we take the expectation over ρ and then change the measure from ρ to ρ k +1 .

Original: By Assumption 4.1 and Lemma D.1, which relies on ρ k +1 , we establish the upper bounds in Lemma E.6.
Revised: By Assumption 4.1 and Lemma D.1, which relies on ρ k +1 , we establish the upper bounds in Lemma E.6.

Original: See partof § E.1 for details.
Revised: See partof § E.1 for details.

Original: Combining Lemmas E.5 and E.6 yields Theorem 4.4.
Revised: Combining Lemmas E.5 and E.6 yields Theorem 4.4.

Original: See § E.1 for a detailed proof.
Revised: See § E.1 for a detailed proof.

Original: E P ROOFS OF T HEOREMS E. P ROOF OF T HEOREM 4. 
Revised: E P ROOFS OF T HEOREMS E. P ROOF OF T HEOREM 4.

Original: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.
Revised: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.

Original: We ﬁrst upper bound (cid:80)
Revised: We ﬁrst upper bound (cid:80)

Original: Kk =

Revised: Kk =

Original: ( Q ∗ ( s, a ) − Q π θk +

Revised: ( Q ∗ ( s, a ) − Q π θk +

Original: ( s, a ))
Revised: ( s, a ))

Original: for any ( s, a ) ∈ S × A in part 1.
Revised: for any ( s, a ) ∈ S × A in part 1.

Original: Then by further taking the expectation over ρ and invoking Lemma D.1 in part 2, we conclude the proof of Theorem 4.4.
Revised: Then by further taking the expectation over ρ and invoking Lemma D.1 in part 2, we conclude the proof of Theorem 4.4.

Original: Part 1.
Revised: Part 1.

Original: In the sequel, we upper bound (cid:80) Kk =

Revised: In the sequel, we upper bound (cid:80) Kk =

Original: ( Q ∗ ( s, a ) − Q π θk +

Revised: ( Q ∗ ( s, a ) − Q π θk +

Original: ( s, a ))
Revised: ( s, a ))

Original: for any ( s, a ) ∈ S × A .
Revised: for any ( s, a ) ∈ S × A .

Original: By the deﬁnition of Q ∗ in (2.2), it holds for any ( s, a ) ∈ S × A that
Revised: By the deﬁnition of Q ∗ in (2.2), it holds for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Revised: , a )

Revised: [Equation]

Revised: (E

Revised: [Equation]

Original: where P π ∗ is deﬁned in (2.3).
Revised: where P π ∗ is deﬁned in (2.3).

Original: We upper bound [(1 − γ ) · r + γ · P π ∗ Q π θk +1 − Q π θk +1 ]( s, a ) on the
Revised: We upper bound [(1 − γ ) · r + γ · P π ∗ Q π θk +1 − Q π θk +1 ]( s, a ) on the

Original: RHS of (E.1) in the sequel.
Revised: RHS of (E.1) in the sequel.

Original: By calculation, we have
Revised: By calculation, we have

Original: [Equation]
Revised: [Equation]

Original: where A 1 ,k , A 2 ,k [Equation] and A 3 ,k are deﬁned as follows,
Revised: where A 1 ,k , A 2 ,k , and A 3 ,k are deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Here T π θk +1 is deﬁned in (2.4).
Revised: Here T π θk +1 is deﬁned in (2.4).

Original: By the following three lemmas, we upper bound A 1 ,k , A 2 ,k , and
Revised: By the following three lemmas, we upper bound A 1 ,k , A 2 ,k , and

Original: A 3 ,k on the RHS of (E.2), respectively.
Revised: A 3 ,k on the RHS of (E.2), respectively.

Original: Lemma E.1.
Revised: Lemma E.1.

Original: It holds for any ( s, a ) ∈ S × A that
Revised: It holds for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Original: where ϑ k and (cid:15) a k +1 are deﬁned as follows,
Revised: where ϑ k and (cid:15) a k +1 are deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § H.2 for a detailed proof.
Revised: See § H.2 for a detailed proof.

Original: We remark that (cid:15) a k +1 = 0 for any k in the linear actor-critic method.
Revised: We remark that (cid:15) a k +1 = 0 for any k in the linear actor-critic method.

Original: Meanwhile, such a term is included in Lemma E.1 only aiming to generalize to the deep neural actor-critic method.
Revised: Meanwhile, such a term is included in Lemma E.1 only aiming to generalize to the deep neural actor-critic method.

Original: Lemma E.2.
Revised: Lemma E.2.

Original: It holds for any ( s, a ) ∈ S × A that
Revised: It holds for any ( s, a ) ∈ S × A that

Original: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: where ϑ i is deﬁned in (E.4) of Lemma E.1, (cid:15)
Revised: where ϑ i is deﬁned in (E.4) of Lemma E.1, (cid:15)

Original: a i +1 is deﬁned in (E.5) of Lemma E.1, and (cid:15) c
Revised: a i +1 is deﬁned in (E.5) of Lemma E.1, and (cid:15) c

Original: i +1 is deﬁned as follows,
Revised: i +1 is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § H.3 for a detailed proof.
Revised: See § H.3 for a detailed proof.

Original: We remark that (cid:15) a k +1 = 0 for any k in the linear actor-critic method.
Revised: We remark that (cid:15) a k +1 = 0 for any k in the linear actor-critic method.

Original: Meanwhile, such a term is included in Lemma E.2 only aiming to generalize to the deep neural actor-critic method.
Revised: Meanwhile, such a term is included in Lemma E.2 only aiming to generalize to the deep neural actor-critic method.

Original: Lemma E.3.
Revised: Lemma E.3.

Original: It holds for any ( s, a ) ∈ S × A that
Revised: It holds for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Original: where e k +1 is deﬁned as follows,
Revised: where e k +1 is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § H.4 for a detailed proof.
Revised: See § H.4 for a detailed proof.

Original: We upper bound e k +1 in (E.7) of Lemma E.3 using Lemma E.4 as follows.
Revised: We upper bound e k +1 in (E.7) of Lemma E.3 using Lemma E.4 as follows.

Original: Lemma E.4.
Revised: Lemma E.4.

Original: It holds for any ( s, a ) ∈ S × A that
Revised: It holds for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) c
Revised: where (cid:15) c

Original: i ( s, a ) is deﬁned in (E.6) of Lemma E.2 and (cid:15) b i +1 ( s ) is deﬁned as follows,
Revised: i ( s, a ) is deﬁned in (E.6) of Lemma E.2 and (cid:15) b i +1 ( s ) is deﬁned as follows,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § H.5 for a detailed proof.
Revised: See § H.5 for a detailed proof.

Original: We remark that (cid:15) b i +

Revised: We remark that (cid:15) b i +

Original: = 

Revised: =

Original: for any i in the linear actor-critic method.
Revised: for any i in the linear actor-critic method.

Original: Meanwhile, such a term is included in Lemma E.4 only aiming to generalize to the deep neural actor-critic method.
Revised: Meanwhile, such a term is included in Lemma E.4 only aiming to generalize to the deep neural actor-critic method.

Original: Combining Lemmas E.3 and E.4, we obtain the following upper bound of A 3 ,k ,
Revised: Combining Lemmas E.3 and E.4, we obtain the following upper bound of A 3 ,k ,

Original: [Equation]
Revised: [Equation]

Original: Combining (E.1), (E.2), Lemma E.1 and Lemma E.2, it holds for any ( s, a ) ∈ S × A that
Revised: Combining (E.1), (E.2), Lemma E.1 and Lemma E.2, it holds for any ( s, a ) ∈ S × A that

Original: K (cid:88) 
Revised: K (cid:88) k

Revised: =

Original: [Equation]
Revised: [Equation]

Original: where ϑ i , (cid:15) a i +1 , (cid:15) c
Revised: where ϑ i , (cid:15) a i +1 , (cid:15) c

Original: i +1 , and e k +1 are deﬁned in (E.4) of Lemma E.1, (E.5) of Lemma E.1, (E.6) of
Revised: i +1 , and e k +1 are deﬁned in (E.4) of Lemma E.1, (E.5) of Lemma E.1, (E.6) of

Original: Lemma E.2, and (E.7) of Lemma E.3, respectively.
Revised: Lemma E.2, and (E.7) of Lemma E.3, respectively.

Original: We upper bound the last term as follows,
Revised: We upper bound the last term as follows,

Original: [Equation]
Revised: [Equation]

Original: where we use the deﬁnition of ϑ k − i in (E.4) of Lemma E.1 and the non-negativity of the KL divergence in the second equality and the last inequality, respectively.
Revised: where we use the deﬁnition of ϑ k − i in (E.4) of Lemma E.1 and the non-negativity of the KL divergence in the second equality and the last inequality, respectively.

Original: By plugging (E.9) and (E.11) into (E.10)
Revised: By plugging (E.9) and (E.11) into (E.10), we have K

Original: [Equation]
Revised: [Equation]

Original: we have K
Revised: k =

Original: [Equation]
Revised: [Equation]

Original: We remark that (cid:15) a
Revised: We remark that (cid:15) a

Original: i +1 =
Revised: i +1 =

Original: (cid:15) b i +1 = 0 for any i in the linear actor-critic method.
Revised: (cid:15) b i +1 = 0 for any i in the linear actor-critic method.

Original: Meanwhile, such terms is included in (E.12) only aiming to generalize to the deep neural actor-critic method.
Revised: Meanwhile, such terms is included in (E.12) only aiming to generalize to the deep neural actor-critic method.

Original: This concludes the proof in part 1.
Revised: This concludes the proof in part 1.

Original: Part 2.
Revised: Part 2.

Original: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.
Revised: Recall that ρ is a state-action distribution satisfying (ii) of Assumption 4.1.

Original: In the sequel, we take the expectation over ρ in (E.12) and upper bound each term.
Revised: In the sequel, we take the expectation over ρ in (E.12) and upper bound each term.

Original: Recall that (cid:15) a i +1 =
Revised: Recall that (cid:15) a i +1 =

Original: (cid:15) b i +1 =
Revised: (cid:15) b i +1 =

Original: for any i in the linear actor-critic method.
Revised: for any i in the linear actor-critic method.

Original: Hence, we only need to consider terms in (E.12) that do not involve (cid:15)
Revised: Hence, we only need to consider terms in (E.12) that do not involve (cid:15)

Original: a i +1 or (cid:15) b
Revised: a i +1 or (cid:15) b

Original: i +1 .
Revised: i +1 .

Original: We ﬁrst upper bound terms on the RHS of (E.12) that do not involve (cid:15)
Revised: We ﬁrst upper bound terms on the RHS of (E.12) that do not involve (cid:15)

Original: c i +1 .
Revised: c i +1 .

Original: More speciﬁcally, for any measure ρ satisfying satisfying (ii) of Assumption 4.1, we upper bound the following three terms,
Revised: More speciﬁcally, for any measure ρ satisfying satisfying (ii) of Assumption 4.1, we upper bound the following three terms,

Original: [Equation]
Revised: [Equation]

Original: We upper bound M 1 , M 2 , and M 3 in the following lemma.
Revised: We upper bound M 1 , M 2 , and M 3 in the following lemma.

Original: Lemma E.5.
Revised: Lemma E.5.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: where M 1 , M 2 , and M 3 are deﬁned in (E.13).
Revised: where M 1 , M 2 , and M 3 are deﬁned in (E.13).

Original: Proof.
Revised: Proof.

Original: See § H.6 for a detailed proof.
Revised: See § H.6 for a detailed proof.

Original: Now, we upper bound terms on the RHS of (E.12) that involve (cid:15) c
Revised: Now, we upper bound terms on the RHS of (E.12) that involve (cid:15) c

Original: i +1 .
Revised: i +1 .

Original: More speciﬁcally, for any measure ρ satisfying (ii) of Assumption 4.1, we upper bound the following two terms,
Revised: More speciﬁcally, for any measure ρ satisfying (ii) of Assumption 4.1, we upper bound the following two terms,

Original: [Equation]
Revised: [Equation]

Original: We upper bound M 4 and M 5 in the following lemma.
Revised: We upper bound M 4 and M 5 in the following lemma.

Original: Lemma E.6.
Revised: Lemma E.6.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: where M 4 and M 5 are deﬁned in (E.14).
Revised: where M 4 and M 5 are deﬁned in (E.14).

Original: Proof.
Revised: Proof.

Original: See § H.7 for a detailed proof.
Revised: See § H.7 for a detailed proof.

Original: Now, by plugging Lemmas E.5 and E.6 into (E.12), we haveE
Revised: Now, by plugging Lemmas E.5 and E.6 into (E.12), we have

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by changing measure from ρ ∗ to ρ k +1 , it holds for any k that
Revised: Meanwhile, by changing measure from ρ ∗ to ρ k +1 , it holds for any k that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where φ ∗ k +1 is deﬁned in Assumption 4.1.
Revised: where φ ∗ k +1 is deﬁned in Assumption 4.1.

Original: Also, by Lemma D.1, it holds that
Revised: Also, by Lemma D.1, it holds that

Original: [Equation]
Revised: [Equation]

Original: Now, by plugging (E.17) into (E.16), combining the deﬁnition of ε Q = max k E ρ ∗ [ | (cid:15) c
Revised: Now, by plugging (E.17) into (E.16), combining the deﬁnition of ε Q = max k E ρ ∗ [ | (cid:15) [Equation]

Original: k +1 | ] , we have
Revised: ] , we have

Original: [Equation]
Revised: [Equation]

Original: Combining (E.15), (E.18), and the choices of parameters stated in the theorem that
Revised: Combining (E.15), (E.18) [Equation] and the choices of parameters stated in the theorem that

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Theorem4.4.
Revised: which concludes the proof of Theorem

Revised: [Equation]

Revised: [Equation]

Original: E. P ROOF OF T HEOREM C. 
Revised: E. P ROOF OF T HEOREM C.

Original: We follow the proof of Theorem 4.4 in § E.1.
Revised: We follow the proof of Theorem 4.4 in § E.1.

Original: Following similar arguments when deriving (E.12) in § E.1, we have
Revised: Following similar arguments when deriving (E.12) in § E.1, we have

Original: K

Original: [Equation]
Revised: [Equation]

Original: for any ( s, a ) ∈ S × A .
Revised: for any ( s, a ) ∈ S × A .

Original: Here (cid:15) a i +1 , (cid:15) b (cid:96) +1 , (cid:15) c
Revised: Here (cid:15) a i +1 , (cid:15) b (cid:96) +1 , (cid:15) c

Original: i +1 , and e 1 are deﬁned in (E.5), (E.8), (E.6), and (E.7), respectively.
Revised: i +1 , and e 1 are deﬁned in (E.5), (E.8), (E.6), and (E.7), respectively.

Original: Now, it remains to upper bound each term on the RHS of (E.19).
Revised: Now, it remains to upper bound each term on the RHS of (E.19).

Original: We introduce the following error propagation lemma.
Revised: We introduce the following error propagation lemma.

Original: Lemma E.7.
Revised: Lemma E.7.

Original: Suppose that
Revised: Suppose that

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) a k +1 and (cid:15)
Revised: where (cid:15) a k +1 and (cid:15)

Original: b k +1 are deﬁned in (E.5) and (E.8), respectively, φ ∗ k and ψ ∗ k are deﬁned in Assumption C.1.
Revised: b k +1 are deﬁned in (E.5) and (E.8), respectively, φ ∗ k and ψ ∗ k are deﬁned in Assumption C.1.

Original: Proof.
Revised: Proof.

Original: See § H.8 for a detailed proof.
Revised: See § H.8 for a detailed proof.

Original: Following from Lemma F.4, with probability at least 1 − O ( H c ) exp( − Ω( H − 1 c m c )) , we have | Q ω 0 | ≤ 2 .
Revised: Following from Lemma F.4, with probability at least 1 − O ( H c ) exp( − Ω( H − 1 c m c )) , we have | Q ω 0 | ≤ 2 .

Original: Also, from the fact that | r ( s, a ) | ≤ r max , we know that | Q ∗ | ≤ r max .
Revised: Also, from the fact that | r ( s, a ) | ≤ r max , we know that | Q ∗ | ≤ r max .

Original: Therefore, for any measure ρ , we have
Revised: Therefore, for any measure ρ , we have

Original: [Equation]
Revised: [Equation]

Original: Also, by changing the index of summation, we have
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: where we expand ( I − γ P π ∗ ) − 1 into an inﬁnite sum in the ﬁrst equality. Further, by changing the measure of the expectation on the RHS of (E.22)
Revised: by changing the index of summation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: (E.22) where we expand ( I − γ P π ∗ ) − 1 into an inﬁnite sum in the ﬁrst equality.

Revised: Further, by changing the measure of the expectation on the RHS of (E.22), we have

Original: [Equation]
Revised: [Equation]

Original:  where c ( t ) is deﬁned in Assumption C.1.
Revised: (E.23) where c ( t ) is deﬁned in Assumption C.1.

Original: Further, by Lemma E.7 and interchanging the summation on the RHS of (E.23), we have
Revised: Further, by Lemma E.7 and interchanging the summation on the RHS of (E.23), we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]

Original: ≤ γ

Original: K

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original: (E.24) where ε f = max i E
Revised: where ε f = max i E

Original: ρ i
Revised: ρ i

Original: [( f θ i +1 ( s, a ) − τ i +1 · ( β − 1 Q ω
Revised: [( f θ i +1 ( s, a ) − τ i +1 · ( β − 1 Q ω

Original: i ( s, a ) − τ − 1 i f θ i ( s, a ))) 2 ] 1 / 2 , and C ρ,ρ ∗ is deﬁned in Assumption C.1.
Revised: i ( s, a ) − τ − 1 i f θ i ( s, a ))) 2 ] 1 / 2 , and C ρ,ρ ∗ is deﬁned in Assumption C.1.

Original: Here in the second inequality, we use the fact that τ − 1 i +1 =
Revised: Here in the second inequality, we use the fact that τ − 1 i +1 =

Original: ( i + 1) · β − 1 , and φ ∗ i ≤ φ ∗ and ψ ∗ i ≤ ψ ∗ by Assumption C.1.
Revised: ( i + 1) · β − 1 , and φ ∗ i ≤ φ ∗ and ψ ∗ i ≤ ψ ∗ by Assumption C.1.

Original: By similar arguments in the derivation of (E.24), we have
Revised: By similar arguments in the derivation of (E.24), we have

Original: [Equation]
Revised: [Equation]

Original: where ε Q = max i E ρ ∗ [ | (cid:15) c i +1 | ] .
Revised: where ε Q = max i E ρ ∗ [ | (cid:15) c i +1 | ] .

Original: And we use the fact that β = K 1 / 2 .
Revised: And we use the fact that β = K 1 / 2 .

Original: Now, it remains to upper bound the last term on the RHS of (E.19).
Revised: Now [Equation] it remains to upper bound the last term on the RHS of (E.19).

Original: We ﬁrst consider the terms involving (cid:15) b (cid:96) +1 .
Revised: We ﬁrst consider the terms involving (cid:15) b (cid:96) +1 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: E.26) where we expand ( I − γ P π ∗ ) − 1 and ( I − γ P π θk +1 ) − 1 to inﬁnite sums in the ﬁrst equality, change the measure of the expectation in the ﬁrst inequality, and use Lemma E.7 in the last inequality.

Original: [Table] [Equation] [Table] [Equation]
Revised: Now, by changing the index of the summation, we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: (cid:96) = ∞ (cid:88)

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: j = ∞ (cid:88) i

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: = γ i + j + k − (cid:96) +1 · ( (cid:96) + 1)

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] 26) where we expand ( I − γ P π ∗  ) − 1 and ( I − γ P π θk +1 ) − 1 to inﬁnite sums in the ﬁrst equality, change the measure of the expectation in the ﬁrst inequality, and use Lemma E.7 in the last inequality. Now, by changing the index of the summation, we have
Revised: ε f · (1 + ψ ∗ (cid:96) ) ·

Original: [Equation]
Revised: c ( i + j + k − (cid:96) + 1) = γ K (cid:88) k = k (cid:88) (cid:96) = ∞ (cid:88)

Original: 27) where we use the fact that ψ ∗ (cid:96)  ≤ ψ ∗ from Assumption C.1 in the last inequality.
Revised: j = ∞ (cid:88) t = j + k − (cid:96) + γ t · ( (cid:96) + 1) ε f · (1 + ψ ∗ (cid:96) ) · c ( t ) ≤ γ

Revised: K (cid:88) k

Revised: = ∞ (cid:88)

Revised: j = ∞ (cid:88) t =

Revised: j + k (cid:88) (cid:96) =

Revised: max { 0 ,j + k − t +1 } γ t · ( (cid:96) + 1) ε f · (1 + ψ ∗ ) ·

Revised: c ( t ) , (E.27) where we use the fact that ψ ∗ (cid:96) ≤ ψ ∗ from Assumption C.1 in the last inequality.

Original: By further manipulating the order of summations of the RHS of (E.27), we haveγ
Revised: By further manipulating the order of summations of the RHS of (E.27), we have

Original: [Equation]
Revised: [Equation]

Original: where we use the deﬁnition of C ρ,ρ ∗ from Assumption C.1 in the last inequality.
Revised: where we use the deﬁnition of C ρ,ρ ∗ from Assumption C.1 in the last inequality.

Original: Now, combining (E.26), (E.27)
Revised: Now, combining (E.26), (E.27), and (E.28), we have

Original: [Equation]
Revised: [Equation]

Original: and (E.28), we have
Revised: Following from similar arguments when deriving (E.29)

Original: [Equation]
Revised: [Equation]

Original: Following from similar arguments when deriving (E.29), we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now, by plugging (E.21), (E.24), (E.25), (E.29)
Revised: Now, by plugging (E.21), (E.24), (E.25), (E.29)

Original: [Equation]
Revised: [Equation]

Original: and (E.30) into (E.19), with probability at least 1 − O
Revised: and (E.30) into (E.19) [Equation] with probability at least 1 − O

Original: ( H c ) exp( − Ω
Revised: ( H c ) exp( − Ω

Original: ( H − 1 c m c )) , we have
Revised: ( H − 1 c m c )) , we have

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, following from Propositions C.3 and C.4, it holds with probability at least 1 − 1 /K that
Revised: Meanwhile, following from Propositions C.3 and C.4, it holds with probability at least 1 − 1 /K that

Original: [Equation]
Revised: [Equation]

Original: Combining (E.31), (E.32) [Equation] and the choices of parameters stated in the theorem, it holds with probability at least 1 − 1 /K that
Revised: Combining (E.31), (E.32), and the choices of parameters stated in the theorem, it holds with probability at least 1 − 1 /K that

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Theorem C.5.
Revised: which concludes the proof of Theorem C.5.

Original: F S UPPORTING R ESULTS
Revised: F S UPPORTING R ESULTS

Original: In this section, we provide some supporting results in the proof of Theorems 4.4 and C.5.
Revised: In this section, we provide some supporting results in the proof of Theorems 4.4 and C.5.

Original: We introduce Lemma F.1, which applies to both Algorithms 1 and 2.
Revised: We introduce Lemma F.1, which applies to both Algorithms 1 and 2.

Original: To introduce Lemma F.1, for any policy π and action-value function Q , we deﬁne (cid:101) π ( a | s ) ∝ exp( β − 1 Q
Revised: To introduce Lemma F.1, for any policy π and action-value function Q , we deﬁne (cid:101) π ( a | s ) ∝ exp( β − 1 Q

Original: ( s, a )) ·
Revised: ( s, a )) ·

Original: π ( a | s ) .
Revised: π ( a | s ) .

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For any s ∈ S and π † , we have
Revised: For any s ∈ S and π † , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By calculation, it sufﬁces to show that
Revised: By calculation [Equation] it sufﬁces to show that

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of the KL divergence, it holds for any s ∈ S that
Revised: By the deﬁnition of the KL divergence [Equation] it holds for any s ∈ S that

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, for the term on the RHS of (F
Revised: Meanwhile, for the term on the RHS of (F.1), we have

Original: [Equation]
Revised: [Equation]

Original: 1), we have
Revised: Combining (F.1) and (F

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (F.1) and (F.2), we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Lemma F.1.
Revised: which concludes the proof of Lemma F.

Revised: [Equation]

Original: F. L OCAL L INEARIZATION OF DNN S
Revised: F. L OCAL L INEARIZATION OF DNN S

Original: In the proofs of Propositions C.3 and C.4 in § G.2 and § G.3, respectively, we utilize the linearization of DNNs.
Revised: In the proofs of Propositions C.3 and C.4 in § G.2 and § G. [Equation] respectively [Equation]

Revised: we utilize the linearization of DNNs.

Original: We introduce some related auxiliary results here.
Revised: We introduce some related auxiliary results here.

Original: First, we deﬁne the linearization ¯ u θ of the DNN u θ ∈ U ( w, H, R ) as follows,
Revised: First [Equation] we deﬁne the linearization ¯ u θ of the DNN u θ ∈ U ( w, H, R ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where θ 0 is the initialization of u θ .
Revised: where θ 0 is the initialization of u θ .

Original: The following lemmas characterize the linearization error.
Revised: The following lemmas characterize the linearization error.

Original: Lemma F.2.
Revised: Lemma F.2.

Original: Suppose that H
Revised: Suppose that H

Original: =
Revised: =

Original: O ( m 1 / 12 R − 1 / 6 (log m ) − 1 / 2 ) and m = Ω
Revised: O ( m 1 / 12 R − 1 / 6 (log m ) − 1 / 2 ) and m = Ω

Original: ( d 3 / 2 R − 1 H − 3 / 2 · log( m 1 / 2 /R ) 3 / 2 ) .
Revised: ( d 3 / 2 R − 1 H − 3 / 2 · log( m 1 / 2 /R ) 3 / 2 ) .

Original: Then with probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , it holds for any θ ∈ B ( θ 0 , R ) and any ( s, a ) ∈ S ×
Revised: Then with probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , it holds for any θ ∈ B ( θ 0 , R ) and any ( s, a ) ∈ S ×

Original: A that [Equation]{+(cid:107)∇ θ u θ ( s, a ) − ∇ θ u θ 
+}
Revised: A that {+(cid:107)∇ θ u θ ( s, a ) − ∇ θ u θ 
+}

Original: {+( s, a ) (cid:107) 
 = O (cid:0) R 1 / 3 m − 1 / 6 H 5 / 2 (log m ) 1 / 2 (cid:1)+} and {+(cid:107)∇ θ u θ ( s, a ) (cid:107) 
+}
Revised: {+( s, a ) (cid:107) 
 = O (cid:0) R 1 / 3 m − 1 / 6 H 5 / 2 (log m ) 1 / 2 (cid:1)+} and {+(cid:107)∇ θ u θ ( s, a ) (cid:107) 
+}

Original: [Equation]
Revised: = O ( H ) .

Original: Proof.
Revised: Proof.

Original: See the proof of Lemma A.5 in Gao et al.
Revised: See the proof of Lemma A.5 in Gao et al.

Original: for a detailed proof.
Revised: for a detailed proof.

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Suppose that H
Revised: Suppose that H

Original: =
Revised: =

Original: O ( m 1 / 12 R − 1 / 6 (log m ) − 1 / 2 ) and m = Ω
Revised: O ( m 1 / 12 R − 1 / 6 (log m ) − 1 / 2 ) and m = Ω

Original: ( d 3 / 2 R − 1 H − 3 / 2 · log( m 1 / 2 /R ) 3 / 2 ) .
Revised: ( d 3 / 2 R − 1 H − 3 / 2 · log( m 1 / 2 /R ) 3 / 2 ) .

Original: Then with probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , it holds for any θ ∈ B ( θ 0 , R ) and any ( s, a ) ∈ S
Revised: Then with probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , it holds for any θ ∈ B ( θ 0 , R ) and any ( s, a ) ∈ S

Original: ×
Revised: ×

Original: A that 
Revised: A that | u θ ( s, a ) − ¯ u

Original: [Equation]
Revised: θ ( s, a ) | = O (cid:0) R 4 / 3 m − 1 / 6 H 5 / 2 (log m ) 1 / 2 (cid:1) .

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: By mean value theorem, there exists t ∈ [0 , 1] , which depends on θ and ( s, a ) , such that
Revised: By mean value theorem, there exists t ∈ [0 , 1] , which depends on θ and ( s, a ) , such that

Original: [Equation]
Revised: [Equation]

Original: Further by Lemma F.2, we have
Revised: Further by Lemma F.2, we have

Original: [Equation]
Revised: [Equation]

Original: where we use Cauchy-Schwarz inequality in the ﬁrst inequality.
Revised: where we use Cauchy-Schwarz inequality in the ﬁrst inequality.

Original: This concludes the proof of Lemma
Revised: This concludes the proof of Lemma

Original: F.3.
Revised: F.3.

Original: We denote by x ( h ) the output of the h -th layer of the DNN u θ ∈ U ( m, H, R ) , and x ( h ) , 0 the output of the h -th layer of the DNN u θ 0 ∈ U ( m, H, R ) .
Revised: We denote by x ( h ) the output of the h -th layer of the DNN u θ ∈ U ( m, H, R ) , and x ( h ) , 0 the output of the h -th layer of the DNN u θ 0 ∈ U ( m, H, R ) .

Original: The following lemma upper bounds the distance between x ( h ) and x ( h ) , 0 .
Revised: The following lemma upper bounds the distance between x ( h ) and x ( h ) , 0 .

Original: Lemma F.4.
Revised: Lemma F.4.

Original: With probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , for any θ ∈ B ( θ 0 , R ) and any h ∈ [ H ] , we have
Revised: With probability at least 1 − exp( − Ω( R 2 / 3 m 2 / 3 H )) over the random initialization θ 0 , for any θ ∈ B ( θ 0 , R ) and any h ∈ [ H ] , we have

Original: [Equation]
Revised: [Equation]

Original: Also, with probability at least 1 − O ( H ) exp( − Ω( H − 1 m )) over the random initialization θ 0 , for any θ ∈ B ( θ 0 , R ) and any h ∈ [ H ] , it holds that
Revised: Also, with probability at least 1 − O ( H ) exp( − Ω( H − 1 m )) over the random initialization θ 0 , for any θ ∈ B ( θ 0 , R ) and any h ∈ [ H ] , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst inequality follows from Lemma A.5 in Gao et al.
Revised: The ﬁrst inequality follows from Lemma A.5 in Gao et al.

Original: (2019), and
Revised: (2019), and

Original: the second inequality follows from Lemma 7.1 in Allen-Zhu et al.
Revised: the second inequality follows from Lemma 7.1 in Allen-Zhu et al.

Original: (2018b).
Revised: (2018b).

Original: G P ROOFS OF P ROPOSITIONS G. P ROOF OF P ROPOSITION 3. 
Revised: G P ROOFS OF P ROPOSITIONS G. P ROOF OF P ROPOSITION 3.

Original: The proof follows the proof of Proposition 3.1 in Liu et al.
Revised: The proof follows the proof of Proposition 3.1 in Liu et al.

Original: First, we write the update (cid:101) π k +1 ← argmax π E ν k
Revised: First, we write the update (cid:101) π k +1 ← argmax π E ν k

Original: [ (cid:104) Q ω k ( s, · ) , π ( · | s ) (cid:105) − β · KL ( π ( · | s ) (cid:107) π θ k ( · | s ))] as a constrained optimization problem in the following way,
Revised: [ (cid:104) Q ω k ( s, · ) , π ( · | s ) (cid:105) − β · KL ( π ( · | s ) (cid:107) π θ k ( · | s ))] as a constrained optimization problem in the following way,

Original: [Equation]
Revised: [Equation]

Original: for any s ∈ S
Revised: any s ∈ S

Original: [Equation]
Revised: [Equation]

Original: We consider the Lagrangian of the above program,
Revised: We consider the Lagrangian of the above program,

Original: [Equation]
Revised: [Equation]

Original: where λ ( · ) is the dual parameter
Revised: ∈S [Equation] (cid:16)(cid:88) a

Original: [Equation]
Revised: [Equation]

Original:  which is a function on S .
Revised: a [Equation] s ) , where λ ( · ) is the dual parameter, which is a function on S .

Original: Now, by plugging in
Revised: Now, by plugging in

Original: [Equation]
Revised: [Equation]

Original: we have the following optimality condition,
Revised: we have the following optimality condition,

Original: [Equation]
Revised: [Equation]

Original: for any ( s, a ) ∈ S × A .
Revised: for any ( s, a ) ∈ S × A .

Original: Note that log( (cid:80) a (cid:48)
Revised: Note that log( (cid:80) a (cid:48)

Original: ∈A exp( τ − 1 k f θ k
Revised: ∈A exp( τ − 1 k f θ k

Original: ( s, a (cid:48) ))) is only a function of s .
Revised: ( s, a [Equation] ))) is only a function of s .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: for any ( s, a ) ∈ S × A [Equation] which concludes the proof of Proposition 3.
Revised: for any ( s, a ) ∈ S × A , which concludes the proof of Proposition 3.1.

Original: [Equation]

Original: G. P ROOF OF P ROPOSITION C. 
Revised: G. P ROOF OF P ROPOSITION C.

Original: We deﬁne the local linearization of f θ as follows,
Revised: We deﬁne the local linearization of f θ as follows,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, we denote by
Revised: Meanwhile, we denote by

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ satisﬁes that
Revised: where θ ∗ satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: By Algorithm 3, we know that
Revised: By Algorithm [Equation] we know that

Original: [Equation]
Revised: [Equation]

Original: By (G.3) and (G.4)
Revised: By (G.3) and (G.4)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that Γ B ( θ 0 ,R a ) is a contraction mapping in the ﬁrst inequality.
Revised: where we use the fact that Γ B ( θ 0 ,R a ) is a contraction mapping in the ﬁrst inequality.

Original: We upper bound term (i) and term (ii) on the RHS of (G.5) in the sequel.
Revised: We upper bound term (i) and term (ii) on the RHS of (G.5) in the sequel

Revised: [Equation]

Original: Upper Bound of Term (i).
Revised: Upper Bound of Term (i).

Original: By Cauchy–Schwarz inequality, it holds that
Revised: By Cauchy–Schwarz inequality, it holds that

Original: [Equation]
Revised: [Equation]

Original:  where we use the fact that θ ( n ) , θ ∗ ∈ B ( θ 0 , R a ) in the last inequality.
Revised: 6) where we use the fact that θ ( n ) , θ ∗ ∈ B ( θ 0 , R a ) in the last inequality.

Original: Further, by the deﬁnitions in (G.2), it holds that
Revised: Further, by the deﬁnitions in (G.2), it holds that

Original: [Equation]
Revised: [Equation]

Original: where we use (G.1) in the second equality.
Revised: where we use (G.1) in the second equality.

Original: Combining (G.6) and (G.7), we obtain the following upper bound of term (i),
Revised: Combining (G.6) and (G.7), we obtain the following upper bound of term (i),

Original: [Equation]
Revised: [Equation]

Original: Upper Bound of Term (ii).
Revised: Upper Bound of Term (ii).

Original: We now upper bound term (ii) on the RHS of (G.5).
Revised: We now upper bound term (ii) on the RHS of (G.5).

Original: It holds by
Revised: It holds by

Original: Cauchy-Schwarz inequality that
Revised: Cauchy-Schwarz inequality that

Original: [Equation]
Revised: [Equation]

Original: We upper bound term (ii).a, term (ii).b, and term (ii).c in the sequel.
Revised: (cid:123)(cid:122) (cid:125) (ii).a +4 (cid:107) g en − ¯ g en (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) (ii).b +4 (cid:107) ¯ g en − ¯ g e ∗ (cid:107)(cid:124) (cid:123)(cid:122) (cid:125) (ii).c . (G.9)

Original: Upper Bound of Term (ii).a.
Revised: Upper Bound of Term (ii).a.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by the deﬁnition of g n in (G.2), it holds that
Revised: Meanwhile, by the deﬁnition of g n in (G.2), it holds that

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst upper bound f θ as follows,
Revised: We ﬁrst upper bound f θ as follows,

Original: [Equation]
Revised: [Equation]

Original: where x ( H a ) is the output of the H a -th layer of the DNN f θ .
Revised: where x ( H a ) is the output of the H a -th layer of the DNN f θ .

Original: Further combining Lemma F.4, it holds with probability at least 1 − O
Revised: Further combining Lemma F.4, it holds with probability at least 1 − O

Original: ( H a ) exp( − Ω( H − 1 a m a ))
Revised: ( H a ) exp( − Ω( H − 1 a m a ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: (G.12)

Original: Following from similar arguments, with probability at least 1 − O ( H a ) exp( − Ω( H − 1 a m a )) , we have
Revised: Following from similar arguments, with probability at least 1 − O ( H a ) exp( − Ω( H − 1 a m a )) , we have

Original: [Equation]
Revised: [Equation]

Original:  Combining Lemma F.2, (G.10), (G.11), (G.12), and (G.13), it holds with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 3 a H a ))
Revised: (G.13) Combining Lemma F.2, (G.10), (G.11), (G.12), and (G.13), it holds with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 3 a H a ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: which establishes an upper bound of term (ii).a.
Revised: which establishes an upper bound of term (ii).a.

Original: Upper Bound of Term (ii).b.
Revised: Upper Bound of Term (ii).b.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Revised: We upper bound the three terms on the RHS of (G.15) in the sequel, respectively.

Original: For the term (cid:107) f θ ( n ) ∇ θ f θ 0 − ¯ f θ ( n ) ∇ θ f θ 0 (cid:107) 2 on the RHS of (G.15), following from Lemmas F.2 and
Revised: For the term (cid:107) f θ ( n ) ∇ θ f θ 0 − ¯ f θ ( n ) ∇ θ f θ 0 (cid:107) 2 on the RHS of (G.15), following from Lemmas F.2 and

Original: F.3, it holds with probability at least 1 − exp( − Ω( R 2 / 

Revised: F.3, it holds with probability at least 1 − exp( − Ω( R 2 /

Original: a m 2 / 3 a H a ))
Revised: a m 2 / 3 a H a ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: For the term (cid:107) f θ ( n ) ∇ θ f θ ( n ) − f θ ( n ) ∇ θ f θ 0 (cid:107) 2 on the RHS of (G.15), following from (G.13) and
Revised: For the term (cid:107) f θ ( n ) ∇ θ f θ ( n ) − f θ ( n ) ∇ θ f θ 0 (cid:107) 2 on the RHS of (G.15), following from (G.13) and

Original: Lemma F.2, with probability at least 1 − exp( − Ω
Revised: Lemma F.2, with probability at least 1 − exp( − Ω

Original: ( R 2 / 3 a m 2 / 

Revised: ( R 2 / 3 a m 2 /

Original: a H a )) , we have
Revised: a H a )) , we have

Original: [Equation]
Revised: [Equation]

Original: For the term (cid:107) (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) · ( ∇ θ f θ 0 − ∇ θ f θ ( n ) ) (cid:107) 2 on the RHS of (G.15), we ﬁrst upper bound (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) as follows,
Revised: For the term (cid:107) (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) · ( ∇ θ f θ 0 − ∇ θ f θ ( n ) ) (cid:107) 2 on the RHS of (G.15), we ﬁrst upper bound (cid:101) τ · ( β − 1 Q ω + τ − 1 f θ ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where we use (G.12), (G.13), and the fact that (cid:101) τ − 1 = β − 1 + τ − 1 .
Revised: where we use (G.12), (G.13), and the fact that (cid:101) τ − 1 = β − 1 + τ − 1 .

Original: Further combining Lemma F.2, it holds with probability at least 1 − exp( − Ω( R 2 / 

Revised: Further combining Lemma F.2, it holds with probability at least 1 − exp( − Ω( R 2 /

Original: a m 2 / 3 a H a ))
Revised: a m 2 / 3 a H a ))

Original: that
Revised: that

Revised: [Equation]

Revised: G

Original: [Equation]
Revised: [Equation]

Original: Now, combining (G.15), (G.16), (G.17), and (G.18), it holds with probability at least 1 − exp( − Ω( R
Revised: Now, combining (G.15), (G.16), (G.17), and (G.18), it holds with probability at least 1 − exp( − Ω [Equation] R

Original: [Equation]
Revised: [Equation]

Original:  that
Revised: ) that

Original: [Equation]
Revised: [Equation]

Original: which establishes an upper bound of term (ii).b.
Revised: which establishes an upper bound of term (ii).b.

Original: Upper Bound of Term (ii).c.
Revised: Upper Bound of Term (ii).c.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Further combining Lemma F.2, it holds with probability at least 1 − exp( − Ω( R
Revised: Further combining Lemma F.2, it holds with probability at least 1 − exp( − Ω( R

Original: [Equation]
Revised: [Equation]

Original: 3 a m 
Revised: m [Equation] [Equation] H

Original: [Equation]
Revised: [Equation]

Original: 3 a H a )) that
Revised: )) that

Original: [Equation]
Revised: [Equation]

Original: which establishes an upper bound of term (ii).c.
Revised: which establishes an upper bound of term (ii).c.

Original: Now, combining (G.9), (G.14), (G.19), and (G.20), we have 
Revised: Now, combining (G.9), (G.14), (G.19), and (G.20), we have E

Original: [Equation]
Revised: [Equation]

Original: ) which is an upper bound of term (ii) on the RHS of (G.5).
Revised: , (G.21) which is an upper bound of term (ii) on the RHS of (G.5).

Original: By plugging the upper bound of term (i) in (G.8) and the upper bound of term (ii) in (G.21) into (G.5), combining (G.19), with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 3 a H a )) [Equation] we have
Revised: By plugging the upper bound of term (i) in (G.8) and the upper bound of term (ii) in (G.21) into (G.5), combining (G.19), with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 3 a H a )) , we have

Original: [Equation]
Revised: [Equation]

Original: Rearranging terms in (G.22), it holds with probability at least 1 − exp( − Ω( R 2 / [-
-] a m 2 / [Equation] H 
Revised: Rearranging terms in (G.22), it holds with probability at least 1 − exp( − Ω( R 2 / [-
-]3 a m 2 / 3 a H a ))

Original: [Equation]

Original: )) that
Revised: that

Original: [Equation]
Revised: [Equation]

Original:  By telescoping the sum and using Jensen’s inequality in (G.23), we have
Revised: (G.23) By telescoping the sum and using Jensen’s inequality in (G.23), we have

Original: [Equation]
Revised: [Equation]

Original: where the last line comes from the choices that α = N − 1 / 2 a and H a
Revised: where the last line comes from the choices that α = N − 1 / 2 a and H a

Original: = O ( N 1 / 4 a ) .
Revised: = O ( N 1 / 4 a ) .

Original: Further combining
Revised: Further combining

Original: Lemma F.3 and using triangle inequality [Equation] we have
Revised: Lemma F.3 and using triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of θ ∗ in (G.3)
Revised: By the deﬁnition of θ ∗ in (G.3)

Original: [Equation]
Revised: [Equation]

Original: we know that
Revised: we know that

Revised: [Equation]

Revised: By plugging the deﬁnition of ¯ g e ∗ into (G.25), we have

Original: [Equation]
Revised: [Equation]

Original: for any
Revised: for any

Original: [Equation]
Revised: [Equation]

Original:  B
Revised: ∈ B

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: [Equation] By plugging the deﬁnition of ¯ g e ∗ into (G.25), we have [Equation] for any θ
Revised: , R a )

Original: [Equation]
Revised: [Equation]

Original: 0 , R a ) , which is equivalent to
Revised: which is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by the fact that θ 0 = ω 0 , we have
Revised: Meanwhile, by the fact that θ 0 = ω 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where the second line comes from (cid:101)
Revised: where the second line comes from (cid:101)

Original: τ − 1 = β − 1 + τ − 1 .
Revised: τ − 1 = β − 1 + τ − 1 .

Original: Note that θ ∈ B ( θ 0 , R a ) , ω ∈ B ( ω 0 , R c ) , θ 0 = ω 0 , and R a = R c , we know that (cid:101) τ · ( β − 1 ω + τ − 1 θ )
Revised: Note that θ ∈ B ( θ 0 , R a ) , ω ∈ B ( ω 0 , R c ) , θ 0 = ω 0 , and R a = R c , we know that (cid:101) τ · ( β − 1 ω + τ − 1 θ )

Original: ∈ B ( θ 0 , R a ) .
Revised: ∈ B ( θ 0 , R a ) .

Original: Therefore, with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 / 

Revised: Therefore, with probability at least 1 − exp( − Ω( R 2 / 3 a m 2 /

Original: a H a ))
Revised: a H a ))

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality comes from (G.26) [Equation] and the last inequality comes from Lemma F.3 and the fact that R c
Revised: where the ﬁrst inequality comes from (G.26), and the last inequality comes from Lemma F.3 and the fact that R c

Original: = R a , m c
Revised: = R a , m c

Original: = m a , and H c
Revised: = m a , and H c

Original: = H a .
Revised: = H a .

Original: Combining (G.24) and (G.27), by triangle inequality, we have
Revised: Combining (G.24) and (G.27), by triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: which ﬁnishes the proof of Proposition C.
Revised: which ﬁnishes the proof of Proposition C.

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF P ROPOSITION C. 
Revised: G. P ROOF OF P ROPOSITION C.

Original: The proof is similar to that of Proposition C.3 in § G.2.
Revised: The proof is similar to that of Proposition C.3 in § G.2.

Original: For the completeness of the paper, we present it here.
Revised: For the completeness of the paper, we present it here.

Original: We deﬁne the local linearization of Q ω as follows,
Revised: We deﬁne the local linearization of Q ω as follows,

Original: [Equation]
Revised: [Equation]

Original: We denote by
Revised: We denote by

Original: [Equation]
Revised: [Equation]

Original: , a 0 ) − γ · Q ω ( s 1 , a 1 ) − (1 − γ ) · r 0 (cid:1) · ∇ ω Q ω ∗ ( s 0 , a 0 ) , g e ∗ = E π θ [ g ∗ ] , ¯ g ∗ = (cid:0) ¯ Q ω ∗ ( s 0 , a 0 ) − γ · Q ω ( s 1 , a 1 ) − (1 − γ ) · r 0 (cid:1) · ∇ ω Q ω 0 ( s 0 , a 0 ) , ¯ g e ∗ = E π θ [¯ g ∗ ] , (G.29) where ω ∗ satisﬁes that
Revised: where ω ∗ satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: Here the expectation E π θ [ · ] is taken following ( s 0 , a 0 ) ∼ ρ π θ ( · ) , s 1 ∼ P ( · | s 0 , a 0 ) , a 1 ∼ π θ ( · | s 1 ) , and r 0 = r
Revised: Here the expectation E π θ [ · ] is taken following ( s 0 , a 0 ) ∼ ρ π θ ( · ) , s 1 ∼ P ( · | s 0 , a 0 ) , a 1 ∼ π θ ( · | s 1 ) , and r 0 = r

Original: ( s 0 , a 0 ) .
Revised: ( s 0 , a 0 ) .

Original: By Algorithm 4, we know that
Revised: By Algorithm 4, we know that

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We upper bound term (iii) and term (iv) on the RHS of (G.31) in the sequel.
Revised: We upper bound term (iii) and term (iv) on the RHS of (G.31) in the sequel.

Original: Upper Bound of Term (iii).
Revised: Upper Bound of Term (iii).

Original: By H ¨ older’s inequality, it holds that
Revised: By H ¨ older’s inequality, it holds that

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that ω ( n ) , ω ∗ ∈ B ( ω 0 , R c ) in the last line.
Revised: where we use the fact that ω ( n ) , ω ∗ ∈ B ( ω 0 , R c ) in the last line.

Original: Further, by the deﬁnitions in (G.29) [Equation] it holds that
Revised: Further, by the deﬁnitions in (G.29), it holds that

Original: [Equation]
Revised: [Equation]

Original: where the second equality comes from (G.28), and the last equality comes from the fact that the expectation is only taken to the state-action pair ( s 0 , a 0 ) .
Revised: where the second equality comes from (G.28), and the last equality comes from the fact that the expectation is only taken to the state-action pair ( s 0 , a 0 ) .

Original: Combining (G.32) and (G.33), we obtain the following upper bound of term (i),
Revised: Combining (G.32) and (G.33), we obtain the following upper bound of term (i),

Original: [Equation]
Revised: [Equation]

Original: Upper Bound of Term (iv).
Revised: Upper Bound of Term (iv).

Original: We now upper bound term (iv) on the RHS of (G.31).
Revised: We now upper bound term (iv) on the RHS of (G.31).

Original: It holds by
Revised: It holds by

Original: Cauchy-Schwarz inequality that
Revised: Cauchy-Schwarz inequality that

Original: [Equation]
Revised: [Equation]

Original: We upper bound term (iv).a, term (iv).b, and term (iv).c in the sequel.
Revised: We upper bound term (iv).a, term (iv).b, and term (iv).c in the sequel.

Original: Upper Bound of Term (iv).a.
Revised: Upper Bound of Term (iv).a.

Original: We now upper bound term (iv).a on the RHS of (G.35).
Revised: We now upper bound term (iv).a on the RHS of (G.35).

Original: By expanding the square, we have
Revised: By expanding the square, we have

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by the deﬁnition of g n in (G.29) [Equation] it holds that
Revised: Meanwhile, by the deﬁnition of g n in (G.29), it holds that

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst upper bound Q ω as follows,
Revised: We ﬁrst upper bound Q ω as follows,

Original: [Equation]
Revised: [Equation]

Original: where x ( H c ) is the output of the H c -th layer of the DNN Q ω .
Revised: where x ( H c ) is the output of the H c -th layer of the DNN Q ω .

Original: Further combining Lemma F.4, it holds that
Revised: Further combining Lemma F.4, it holds that

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have 
Revised: Similarly, we have | Q ω ( n ) | ≤ 2 .

Original: [Equation]
Revised: [Equation]

Original:  Combining Lemma F.2, (G.36), (G.37), (G.38)
Revised: G.39) Combining Lemma F.2, (G.36), (G.37), (G.38)

Original: [Equation]
Revised: [Equation]

Original: and (G.39), we have
Revised: and (G.39), we have

Original: [Equation]
Revised: [Equation]

Original: Upper Bound of Term (iv).b.
Revised: Upper Bound of Term (iv).b.

Original: We now upper bound term (iv).b on the RHS of (G.35).
Revised: We now upper bound term (iv).b on the RHS of (G.35).

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: We now upper bound the three terms on the RHS of (G.41) in the sequel, respectively.{+s 1 , a 1 ) + (1 − γ ) · r t (cid:1) · ( ∇ ω Q ω 
 ( s 0 , a 0 ) − ∇ ω Q ω ( n ) ( s 0 , a 0 ))+}
Revised: {+s 1 , a 1 ) + (1 − γ ) · r t (cid:1) · ( ∇ ω Q ω 
 ( s 0 , a 0 ) − ∇ ω Q ω ( n ) ( s 0 , a 0 ))+}

Original: For the term E ρ πθ [
Revised: (cid:13)(cid:13) 2 (cid:3) + E ρ πθ (cid:2) (cid:107) Q ω ( n ) ∇ ω Q ω ( n ) − ¯ Q ω ( n ) ∇ ω Q ω 0 (cid:107) 2 (cid:3) ≤ E π θ (cid:2)(cid:13)(cid:13)(cid:0) γ · Q ω s 1 , a 1 )

Original: [Equation]
Revised: + (1 − γ ) · r 0 (cid:1) · ( ∇ ω Q ω 0 ( s 0 , a 0 ) − ∇ ω Q ω ( n ) ( s 0 , a 0 )) (cid:13)(cid:13) 2 (cid:3) (G.41) + E ρ π θ (cid:2) (cid:107) ( Q ω ( n ) − ¯ Q ω ( n ) ) · ∇ ω Q ω 0 (cid:107) 2 (cid:3) + E ρ π θ (cid:2) (cid:107) Q ω ( n ) · ( ∇ ω Q ω ( n ) − ∇ ω Q ω 0 ) (cid:107) 2 (cid:3) .

Original:  ( Q ω ( n ) − ¯ Q ω ( n ) ) · ∇ ω Q ω 0 (cid:107) 2 ] on the RHS of (G.41), following from Lemmas
Revised: For the term E ρ πθ [ (cid:107) ( Q ω ( n ) − ¯ Q ω ( n ) ) · ∇ ω Q ω 0 (cid:107) 2 ] on the RHS of (G.41), following from Lemmas

Original: F.2 and F.3, it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))
Revised: F.2 and F.3, it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))

Original: that
Revised: that

Original: [Equation] For the term E
Revised: E

Original: [Equation]
Revised: [Equation]

Original:  ( n ) − ∇ ω Q ω 0 ) (cid:107) 2 ] on the RHS of (G.41), following from (G.39) and Lemma F.2, with probability at least 1 − exp( − Ω
Revised: For the term E ρ πθ [ (cid:107) Q ω ( n ) · ( ∇ ω Q ω ( n ) − ∇ ω Q ω 0 ) (cid:107) 2 ] on the RHS of (G.41), following from (G.39) and Lemma F.2, with probability at least 1 − exp( − Ω

Original: ( R 2 / 3 c m 2 / 3 c H c )) , we have
Revised: ( R 2 / 3 c m 2 / 3 c H c )) , we have

Original: [Equation]
Revised: [Equation]

Original: For the term E π θ [ [Equation] γ 
Revised: For the term E π θ [ (cid:107) ( γ · Q ω ( s 1 , a 1 ) + (1 − γ ) · r 0 ) · ( ∇ ω Q ω 0 ( s 0 , a 0 ) − ∇ ω Q ω ( n ) ( s 0 , a 0 ))

Original: [Equation] Q ω [Equation] [Equation]
Revised: (cid:107) 2 ] on the

Original:  1 ) + [Equation] [Equation] ) · ( ∇
Revised: RHS of (G.41), we ﬁrst upper bound | γ · Q ω ( s 1 , a 1 ) + (1 − γ

Original: [Equation]
Revised: [Equation]

Original: Q ω 0 ( s 0 , a 0 ) − ∇ ω Q ω ( n ) ( s 0 , a 0 )) (cid:107) 2 ] on the RHS of (G.41), we ﬁrst upper bound | γ · Q ω ( s 1 , a 1 ) + (1 − γ ) · r 0 | as follows,
Revised: · r 0 | as follows,

Original: [Equation]
Revised: [Equation]

Original: where we use (G.38) and the fact that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A .
Revised: where we use (G.38) and the fact that | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S × A .

Original: Further combining
Revised: Further combining

Original: Lemma F.2, with probability at least 1 − exp( − Ω
Revised: Lemma F.2, with probability at least 1 − exp( − Ω

Original: ( R 2 / 3 c m 2 / 3 c H c )) , we have
Revised: ( R 2 / 3 c m 2 / 3 c H c )) , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, combining (G.41), (G.42), (G.43), and (G.44), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))
Revised: Now, combining (G.41), (G.42), (G.43), and (G.44), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Upper Bound of Term (iv).c.
Revised: Upper Bound of Term (iv).c.

Original: We now upper bound term (iv).c on the RHS of (G.35).
Revised: We now upper bound term (iv).c on the RHS of (G.35).

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Further combining Lemma F.2, it holds that
Revised: Further combining Lemma F.

Original: [Equation]
Revised: [Equation]

Original: Combining (G.35), (G.40), (G.45)
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original:  and (G.46)
Revised: Combining (G.35), (G.40), (G.45), and (G.46)

Original: [Equation]
Revised: [Equation]

Original: we obtain the following upper bound for term (iv) on the RHS of (G.31),
Revised: we obtain the following upper bound for term (iv) on the RHS of (G.31),

Original: [Equation]
Revised: [Equation]

Original: We continue upper bounding (G.31).
Revised: We continue upper bounding (G.31).

Original: By plugging (G.34) and (G.47) into (G.31), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))
Revised: By plugging (G.34) and (G.47) into (G.31), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: Rearranging terms in (G.48), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))
Revised: Rearranging terms in (G.48), it holds with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c ))

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: By telescoping the sum and using Jensen’s inequality in (G.49), we have
Revised: By telescoping the sum and using Jensen’s inequality in (G.49), we have

Original: [Equation]
Revised: [Equation]

Original: where the last line comes from the choices that η = N − 1 / 2 c and H c
Revised: where the last line comes from the choices that η = N − 1 / 2 c and H c

Original: = O ( N 1 / 4 c ) .
Revised: = O ( N 1 / 4 c ) .

Original: Further combining
Revised: Further combining

Original: Lemma F.3 and using triangle inequality, we have
Revised: Lemma F.3 and using triangle inequality

Original: [Equation]
Revised: [Equation]

Original:  To establish the upper bound of E ρ π θ [( ¯ Q ω ∗ − (cid:101) Q ) 2 ] , we upper bound E ρ πθ [( ¯ Q ω ∗ − (cid:101) Q ) 2 ] in the sequel.
Revised: we have [Equation] To establish the upper bound of E ρ πθ [( ¯ Q ω ∗ − (cid:101) Q ) 2 ] , we upper bound E ρ πθ [( ¯ Q ω ∗ − (cid:101) Q ) 2 ] in the sequel.

Original: By the deﬁnition of ω ∗ in (G.30), following a similar argument to derive (G.26), we have
Revised: By the deﬁnition of ω ∗ in (G.30) [Equation] following a similar argument to derive (G.26)

Original: [Equation]
Revised: [Equation]

Original:  From the fact that (cid:101) Q ∈ U ( m c , H c , R c ) by Assumption C.2, we know that (cid:101) Q = Q (cid:101) ω for some (cid:101) ω ∈ B ( ω 0 , R c ) .
Revised: we have [Equation] From the fact that (cid:101) Q ∈ U ( m c , H c , R c ) by Assumption C.2, we know that (cid:101) Q = Q (cid:101) ω for some (cid:101) ω ∈ B ( ω 0 , R c ) .

Original: Therefore, by (G.51), with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) , we have
Revised: Therefore, by (G.51), with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) , we have

Original: [Equation]
Revised: [Equation]

Original: where we use Lemma F.3 in the last inequality.
Revised: where we use Lemma F.3 in the last inequality.

Original: Now, combining (G.50) and (G.52), by triangle inequality, with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) , we have
Revised: Now, combining (G.50) and (G.52), by triangle inequality, with probability at least 1 − exp( − Ω( R 2 / 3 c m 2 / 3 c H c )) , we have

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Proposition C.4.
Revised: which concludes the proof of Proposition C.4.

Original: H P ROOFS OF L EMMAS H. P ROOF OF L EMMA D. 
Revised: H P ROOFS OF L EMMAS H. P ROOF OF L EMMA D.

Original: W denote by (cid:101) Q =
Revised: W denote by (cid:101) Q =

Original: T π θk Q ω
Revised: T π θk Q ω

Original: k .
Revised: k .

Original: In the sequel, we upper bound E ρ k
Revised: In the sequel, we upper bound E ρ k

Original: +

Revised: +

Original: [( Q ω k +1 − Q ¯ ω k +1 ) 2 ] , where ¯ ω k +1 =
Revised: [( Q ω k +1 − Q ¯ ω k +1 ) 2 ] , where ¯ ω k +1 =

Original: Γ R ( (cid:101) ω k +1 ) and (cid:101) ω
Revised: Γ R ( (cid:101) ω k +1 ) and (cid:101) ω

Original: k +1 is deﬁned in (3.4).
Revised: k +1 is deﬁned in (3.4).

Original: Note that by the fact that (cid:107) ϕ
Revised: Note that by the fact that (cid:107) ϕ

Original: ( s, a ) (cid:107) 2 ≤ 1 uniformly, it sufﬁces to upper bound (cid:107)
Revised: ( s, a ) (cid:107) 2 ≤ 1 uniformly, it sufﬁces to upper bound (cid:107)

Original: ω
Revised: ω

Original: k +1 − (cid:101) ω k +1 (cid:107) 2 .
Revised: k +1 − (cid:101) ω k +1 (cid:107) 2 .

Original: By the deﬁnitions of ω k +1 and (cid:101) ω
Revised: By the deﬁnitions of ω k +1 and (cid:101) ω

Original: k +1 in (3.5) and (3.4), respectively, we have
Revised: k +1 in (3.5) and (3.4), respectively, we have

Original: [Equation]
Revised: [Equation]

Original: Here, we use the fact that the projection Γ R ( · ) is a contraction in the ﬁrst inequality, and triangle inequality in the second inequality.
Revised: Here, we use the fact that the projection Γ R ( · ) is a contraction in the ﬁrst inequality, and triangle inequality in the second inequality.

Original: Also, for notational convenience, we denote by (cid:98) Φ , Φ , (cid:98) v , and v in (H.1) as follows, 
Revised: Also, for notational convenience, we denote by (cid:98) Φ , Φ , (cid:98) v , and v in (H.1) as follows, [Equation] a

Original: [Equation]
Revised: [Equation]

Original: By the fact that (cid:107) ϕ
Revised: By the fact that (cid:107) ϕ

Original: ( s, a ) (cid:107) 2 ≤ 1 , | r
Revised: ( s, a ) (cid:107) 2 ≤ 1 , | r

Original: ( s, a )
Revised: ( s, a )

Original: | ≤ r max , and (cid:107) ω k (cid:107) 2 ≤ R
Revised: | ≤ r max , and (cid:107) ω k (cid:107) 2 ≤ R

Original: we have 
Revised: we have (cid:107) (cid:98) Φ (cid:107) 2 ≤ 1 , (cid:107) v (cid:107) 2 ≤ r max + R.

Original: [Equation]
Revised: (H.2)

Original: Now, following from matrix Bernstein inequality (Tropp, 2015) and Assumption [Equation] 3, we have
Revised: Now, following from matrix Bernstein inequality (Tropp, 2015) and Assumption 4.3, we have

Original: [Equation]
Revised: [Equation]

Original: where σ ∗ is deﬁned in Assumption 
Revised: where σ ∗ is deﬁned in Assumption 4.3.

Original: [Equation]

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: Now, combining (H.1), (H.2), (H.3), and (H.4), we have
Revised: Now, combining (H.1), (H.2), (H.3) [Equation] and (H.4), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, it holds that
Revised: Therefore [Equation] it holds that

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by Assumption 4.2 and the deﬁnition of ¯ ω k +1 , we have(cid:101) Q = Q ¯ ω k +1 .
Revised: Meanwhile, by Assumption 4.2 and the deﬁnition of ¯ ω k +1 , we have

Original: [Equation]
Revised: [Equation]

Original: Combining (H.5) and (H.6), we have
Revised: Combining (H.5) and (H.6), we have

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Lemma D.1.
Revised: which concludes the proof of Lemma D.1.

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: Following from the deﬁnitions of P π and P in (2.3), we have
Revised: Following from the deﬁnitions of P π and P in (2.3) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: By invoking Lemma F.1 and combining (H.7), it holds for any ( s, a ) ∈ S × A that
Revised: By invoking Lemma F.1 and combining (H.7), it holds for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Original:  where ϑ k and (cid:15) a k +1 are deﬁned in (E.4) and (E.5) of Lemma E.1, respectively.
Revised: , where ϑ k and (cid:15) a k +1 are deﬁned in (E.4) and (E.5) of Lemma E.1, respectively.

Original: We conclude the proof of Lemma E.1.
Revised: We conclude the proof of Lemma E.1.

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: By the deﬁnition that Q ∗ is the action-value function of an optimal policy π ∗ , we know that
Revised: By the deﬁnition that Q ∗ is the action-value function of an optimal policy π ∗ , we know that

Original: Q ∗ ( s, a ) ≥ Q π ( s, a ) for any policy π and state-action pair ( s, a ) ∈ S × A .
Revised: Q ∗ ( s, a ) ≥ Q π ( s, a ) for any policy π and state-action pair ( s, a ) ∈ S × A .

Original: Therefore, for any ( s, a ) ∈ S × A , we have
Revised: Therefore, for any ( s, a ) ∈ S × A , we have

Original: [Equation]
Revised: [Equation]

Original: In the sequel, we upper bound Q ∗ ( s, a ) − Q ω k
Revised: In the sequel, we upper bound Q ∗ ( s, a ) − Q ω k

Original: ( s, a ) for any ( s, a ) ∈ S × A .
Revised: ( s, a ) for any ( s, a ) ∈ S × A .

Original: We deﬁne 
Revised: We deﬁne (cid:101) Q k +1 =

Original: [Equation]
Revised: (1 − γ ) · r + γ · P π θk +1 Q ω k .

Original: By its deﬁnition, we know that (cid:101) Q k
Revised: By its deﬁnition, we know that (cid:101) Q k

Original: +

Revised: +

Original: =
Revised: =

Original: T π θk +1 Q ω
Revised: T π θk +1 Q ω

Original: k .
Revised: k .

Original: It holds for any ( s, a ) ∈ S × A that
Revised: It holds for any ( s, a ) ∈ S × A that

Revised: Q ∗ ( s, a ) − Q

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) c k+1 and A 1 ,k are deﬁned in (E.6) and (E.3)
Revised: k

Original: [Equation]
Revised: [Equation]

Original: respectively. Here
Revised: 1 ( s, a ) =

Original: [Equation]
Revised: [Equation]

Revised: where (cid:15) c

Revised: k +1 and A 1 ,k are deﬁned in (E.6) and (E.3), respectively.

Original:  we use Lemma E.1 to upper bound A 1 ,k in the last line.
Revised: Here, we use Lemma E.1 to upper bound A 1 ,k in the last line.

Original: We remark that (H.9) upper bounds Q ∗ − Q ω k +1 using Q ∗ − Q ω k .
Revised: We remark that (H.9) upper bounds Q ∗ − Q ω k +1 using Q ∗ − Q ω k .

Original: By recursively applying a similar argument as in (H.9), we have
Revised: By recursively applying a similar argument as in (H.9), we have

Original: [Equation]
Revised: [Equation]

Original: Combining (H.8) and (H.10) [Equation] it holds for any ( s, a ) ∈ S × A that[Equation]
Revised: Combining (H.8) and (H.10), it holds for any ( s, a ) ∈ S × A that

Original: (cid:3)

Original: [Equation]

Original: s

Original: [Equation]

Original: a ) ≤

Original: [Equation]

Original: [Equation]

Original: a

Original: [Equation]
Revised: [Equation]

Original: · k

Original: [Equation]

Original: i

Original: [Equation]

Original: (cid:2)

Original: [Equation]

Original: P π ∗ )

Original: [Equation]

Original: [Equation]

Original:  (cid:15) a [Equation] ) (cid:3) (
Revised: where ϑ i , (cid:15) a i +1 , and (cid:15) c

Original: [Equation]

Original: a ) + k [Equation] 1 (cid:88) i = (cid:2) ( γ P π ∗ ) k − i (cid:15) c i +1 (cid:3) ( s, a ) , where ϑ i , (cid:15) a i +1 , and (cid:15) c i +1 are deﬁned in (E.4) of Lemma E.1, (E.5) of Lemma E.1, and (E.6) of
Revised: i +1 are deﬁned in (E.4) of Lemma E.1, (E.5) of Lemma E.1, and (E.6) of

Original: Lemma E.2, respectively.
Revised: Lemma E.2, respectively.

Original: We conclude the proof of Lemma E.2.
Revised: We conclude the proof of Lemma E.2.

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: Note that for any ( s, a ) ∈ S × A , we have
Revised: Note that for any ( s, a ) ∈ S × A , we have

Original: [Equation]
Revised: [Equation]

Original: , a ) , where the term e k +1 in the last line is deﬁned in (E.7).
Revised: [Equation] where the term e k +1 in the last line is deﬁned in (E.7).

Original: We conclude the proof of Lemma E.
Revised: We conclude the proof of Lemma E.

Original: [Equation]
Revised: [Equation]

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: We invoke Lemma F.1 in § F, which gives
Revised: We invoke Lemma F.1 in § F

Original: [Equation]
Revised: [Equation]

Original: (H.
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: Combining (H.11) and the deﬁnition of P π in (2.3), we have
Revised: Combining (H.11) and the deﬁnition of P π in (2.3), we have

Original: [Equation]
Revised: [Equation]

Original: s )

Original: [Equation]

Original: (H

Original: [Equation]

Original: [Equation]

Original: By the deﬁnition of e k +1 in (E.7) [Equation] we havee
Revised: By the deﬁnition of e k +1 in (E.7), we have

Original: [Equation]
Revised: [Equation]

Original: where we use (H.12) in the ﬁrst inequality
Revised: where we use (H.12) in the ﬁrst inequality

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term on the RHS of (H.13)
Revised: For the ﬁrst term on the RHS of (H.13)

Original: [Equation]
Revised: [Equation]

Original: by (H.14)
Revised: by (H.14)

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: H [Equation] Combining (H.13) and (H.15), we have for any ( s, a ) ∈ S × A that
Revised: Combining (H.13) and (H.15), we have for any ( s, a ) ∈ S × A that

Original: [Equation]
Revised: [Equation]

Original: By telescoping (H.16)
Revised: By telescoping (H.16)

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: This ﬁnishes the proof of the lemma.
Revised: This ﬁnishes the proof of the lemma.

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: Note that (cid:107) ω 0 (cid:107) 2 ≤ R and | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S ×A , which implies that | Q ω 0 ( s, a ) | ≤
Revised: Note that (cid:107) ω 0 (cid:107) 2 ≤ R and | r ( s, a ) | ≤ r max for any ( s, a ) ∈ S ×A , which implies that | Q ω 0 ( s, a ) | ≤

Original: R and | Q ∗ ( s, a ) | ≤ r max by their deﬁnitions.
Revised: R and | Q ∗ ( s, a ) | ≤ r max by their deﬁnitions.

Original: Thus, for M 1 , we have
Revised: Thus, for M 1 , we have

Original: [Equation]
Revised: [Equation]

Original: For M 2 , by the deﬁnition of e 1 in (E.7), | ω k | ≤ R , | φ ( s, a ) | ≤ 1 , and | r ( s, a ) | ≤ r max , we have
Revised: For M 2 , by the deﬁnition of e 1 in (E.7), | ω k | ≤ R , | φ ( s, a ) | ≤ 1 , and | r ( s, a ) | ≤ r max , we have

Original: [Equation]
Revised: [Equation]

Original: for any ( s, a ) ∈ S × A .
Revised: for any ( s, a ) ∈ S × A .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, by the initialization τ 0 = ∞ in Algorithm 1, the initial policy π θ 0 ( · | s ) is a uniform distribution over A .
Revised: Meanwhile, by the initialization τ 0 = ∞ in Algorithm 1, the initial policy π θ 0 ( · | s ) is a uniform distribution over A .

Original: Therefore, it holds for any s ∈ S that
Revised: Therefore, it holds for any s ∈ S that

Original: [Equation]
Revised: [Equation]

Original: Therefore, by (H.20) [Equation] we have
Revised: Therefore, by (H.20), we have

Original: [Equation]
Revised: [Equation]

Original: .21) where we use β = K 1 / 2 .
Revised: where we use β = K 1 / 2 .

Original: We see that (H.17), (H.19) [Equation] and (H.21) upper bound M 1 , M 2 , and M 3 , respectively.
Revised: We see that (H.17), (H.19), and (H.21) upper bound M 1 , M 2 , and M 3 , respectively.

Original: We conclude the proof of Lemma E.
Revised: We conclude the proof of Lemma E.5.

Original: [Equation]

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: For M 4 , by changing the index of summation
Revised: For M 4 , by changing the index of summation, we have

Original: [Equation]
Revised: [Equation]

Original:  we have
Revised: where we expand ( I − γ P π ∗ ) − 1 into an inﬁnite sum in the ﬁrst equality. Further, by changing the measure of the expectation from ρ to ρ ∗ on the RHS of (H.22), we have

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]

Original: k

Original: [Equation]

Original: , (H.22) where we expand ( I − γ P π ∗ ) − 1 into an inﬁnite sum in the ﬁrst equality.

Original: Further, by changing the measure of the expectation from ρ to ρ ∗ on the RHS of (H.22), we have

Original: [Equation]
Revised: [Equation]

Original: (H.23) where c ( t ) is deﬁned in Assumption 4.1.
Revised: (H.23) where c ( t ) is deﬁned in Assumption 4.1.

Original: Further, by changing the index of summation on the RHS of (H.23), combining (H.22), we have
Revised: Further, by changing the index of summation on the RHS of (H.23), combining (H.22), we have

Original: [Equation]
Revised: [Equation]

Original:  where ε Q = max i E ρ ∗ [ | (cid:15) c i +1 | ] , and C ρ,ρ ∗ is deﬁned in Assumption 4.1.
Revised: (H.24) where ε Q = max i E ρ ∗ [ | (cid:15) c i +1 | ] , and C ρ,ρ ∗ is deﬁned in Assumption 4.1.

Original: Now, for M 5 , by a similar argument as in the derivation of (H.24), we have 
Revised: Now, for M 5 , by a similar argument as in the derivation of (H.24), we have M 5 ≤ ∞ (cid:88) i = K (cid:88) k = ∞ (cid:88)

Revised: j = k (cid:88) (cid:96) =

Revised: γ i + j + k − (cid:96) +1 c

Original: M
Revised: ( i + j + k − (cid:96) + 1) · ε Q = ∞ (cid:88) i = K (cid:88) k = ∞ (cid:88) j = i + j

Revised: + k (cid:88) t

Revised: = i + j + γ t c (

Original: [Equation]
Revised: [Equation]

Original: We see that (H.24) and (H.25) upper bound M 4 and M 5 , respectively.
Revised: We see that (H.24) and (H.25) upper bound M 4 and M 5 , respectively.

Original: We conclude the proof of
Revised: We conclude the proof of

Original: Lemma E.6.
Revised: Lemma E.6.

Original: H. P ROOF OF L EMMA E. 
Revised: H. P ROOF OF L EMMA E.

Original: Part 1.
Revised: Part 1.

Original: We ﬁrst show that the ﬁrst inequality holds.
Revised: We ﬁrst show that the ﬁrst inequality holds.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Here Z θ k ( s ) , Z θ k +1 ( s ) ∈ R are normalization factors, which are deﬁned as
Revised: Here Z θ k ( s ) , Z θ k +1 ( s ) ∈ R are normalization factors, which are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that
Revised: where we use the fact that

Original: [Equation]
Revised: [Equation]

Original: Thus, it remains to upper bound the right-hand side of (H.26).
Revised: Thus [Equation] it remains to upper bound the right-hand side of (H.26).

Original: We have(cid:104)
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: Taking expectation with respect to s ∼ ν ∗ on the both sides of (H.27) and using the Cauchy-Schwarz inequality, we obatin

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  where in the last inequality we use the error bound in (E.20) and the deﬁnition of φ ∗ k and ψ ∗ k in
Revised: [Equation] where in the last inequality we use the error bound in (E [Equation] 20) and the deﬁnition of φ ∗ k and ψ ∗ k in

Original: Assumption C.1.
Revised: Assumption C

Revised: [Equation]

Original: This ﬁnishes the proof of the ﬁrst inequality.
Revised: This ﬁnishes the proof of the ﬁrst inequality

Revised: [Equation]

Original: Part 2.
Revised: Part 2.

Original: The proof of the second inequality follows from a similar argument as above.
Revised: The proof of the second inequality follows from a similar argument as above.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original:  where we use the fact that
Revised: 28) where we use the fact that

Original: [Equation]
Revised: [Equation]

Original: Thus, it remains to upper bound the right-hand side of (H.28).
Revised: Thus [Equation] it remains to upper bound the right-hand side of (H.28).

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation with respect to s ∼ ν ∗ on the both sides of (H.29) and using the Cauchy-Schwarz inequality, we obatin
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]