Original: A F INITE -T IME A NALYSIS OF Q-L EARNING WITH N EURAL N ETWORK F UNCTION A PPROXIMATION
Revised: A F INITE -T IME A NALYSIS OF Q-L EARNING WITH N EURAL N ETWORK F UNCTION A PPROXIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms.
Revised: Q-learning with neural network function approximation (neural Q-learning for short) is among the most prevalent deep reinforcement learning algorithms.

Original: Despite its empirical success, the non-asymptotic convergence rate of neural Qlearning remains virtually unknown.
Revised: Despite its empirical success, the non-asymptotic convergence rate of neural Qlearning remains virtually unknown.

Original: In this paper, we present a ﬁnite-time analysis of a neural Q-learning algorithm, where the data are generated from a Markov decision process and the action-value function is approximated by a deep ReLU neural network.
Revised: In this paper, we present a ﬁnite-time analysis of a neural Q-learning algorithm, where the data are generated from a Markov decision process and the action-value function is approximated by a deep ReLU neural network.

Original: We prove that neural Q-learning ﬁnds the optimal policy with
Revised: We prove that neural Q-learning ﬁnds the optimal policy with

Original: O (1 /T ) convergence rate if the neural function approximator is sufﬁciently overparameterized, where T is the number of iterations.
Revised: O (1 / √ T ) convergence rate if the neural function approximator is sufﬁciently overparameterized, where T is the number of iterations.

Original: To our best knowledge, our result is the ﬁrst ﬁnite-time analysis of neural Q-learning under non-i.i.d. data assumption.
Revised: To our best knowledge, our result is the ﬁrst ﬁnite-time analysis of neural Q-learning under non-i.i d. data assumption.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Q-learning has been shown to be one of the most important and effective learning strategies in
Revised: Q-learning has been shown to be one of the most important and effective learning strategies in

Original: Reinforcement Learning (RL) over the past decades (Watkins & Dayan, 1992; Schmidhuber, 2015;
Revised: Reinforcement Learning (RL) over the past decades (Watkins & Dayan, 1992; Schmidhuber, 2015;

Original: Sutton & Barto, 2018), where the agent take an action based on the action-value function (a.k.a., Qvalue function) at the current state.
Revised: Sutton & Barto, 2018), where the agent takes an action based on the action-value function (a.k.a., Qvalue function) at the current state.

Original: Recent advance in deep learning has also enabled the application of Q-learning algorithms to large-scale decision problems such as mastering Go (Silver et al., 2016; 2017), robotic motion control (Levine et al., 2015;
Revised: Recent advance in deep learning has also enabled the application of Q-learning algorithms to large-scale decision problems such as mastering Go (Silver et al., 2016; 2017), robotic motion control (Levine et al., 2015;

Original: Kalashnikov et al., 2018) and autonomous driving (Shalev-Shwartz et al., 2016; Schwarting et al., 2018).
Revised: Kalashnikov et al., 2018) and autonomous driving (Shalev-Shwartz et al., 2016; Schwarting et al., 2018).

Original: In particular, the seminal work by Mnih et al.
Revised: In particular, the seminal work by Mnih et al.

Original: (2015) introduced the Deep Q-Network (DQN) to approximate the action-value function and achieved a superior performance versus a human expert in playing Atari games, which triggers a line of research on deep reinforcement learning such as Double Deep Q-Learning (Van Hasselt et al., 2016) and Dueling DQN (Wang et al., 2016).
Revised: (2015) introduced the Deep Q-Network (DQN) to approximate the action-value function and achieved a superior performance versus a human expert in playing Atari games, which triggers a line of research on deep reinforcement learning such as Double Deep Q-Learning (Van Hasselt et al., 2016) and Dueling DQN (Wang et al., 2016).

Original: Apart from its widespread empirical success in numerous applications, the convergence of Qlearning and temporal difference (TD) learning algorithms has also been extensively studied in the literature (Jaakkola et al., 1994; Baird, 1995; Tsitsiklis & Van Roy, 1997; Perkins & Pendrith, 2002;
Revised: Apart from its widespread empirical success in numerous applications, the convergence of Qlearning and temporal difference (TD) learning algorithms has also been extensively studied in the literature (Jaakkola et al., 1994; Baird, 1995; Tsitsiklis & Van Roy, 1997; Perkins & Pendrith, 2002;

Original: Melo et al., 2008; Mehta & Meyn, 2009; Liu et al., 2015; Bhandari et al., 2018; Lakshminarayanan & Szepesvari, 2018; Zou et al., 2019b).
Revised: Melo et al., 2008; Mehta & Meyn, 2009; Liu et al., 2015; Bhandari et al., 2018; Lakshminarayanan & Szepesvari, 2018; Zou et al., 2019b).

Original: However, the convergence guarantee of deep Q-learning algorithms remains a largely open problem.
Revised: However, the convergence guarantee of deep Q-learning algorithms remains a largely open problem.

Original: The only exceptions are Yang et al.
Revised: The only exceptions are Yang et al.

Original: (2019) which studied the ﬁtted Q-iteration (FQI) algorithm (Riedmiller, 2005; Munos & Szepesv´ari, 2008) with actionvalue function approximation based on a sparse ReLU network, and Cai et al. (2019) which studied the global convergence of Q-learning algorithm with an i.i.d.
Revised: (2019) which studied the ﬁtted Q-iteration (FQI) algorithm (Riedmiller, 2005; Munos & Szepesv´ari, 2008) with action-value function approximation based on a sparse ReLU network, and Cai et al. (2019a) which studied the global convergence of Q-learning algorithm with an i.i.d.

Original: observation model and action-value function approximation based on a two-layer neural network.
Revised: observation model and action-value function approximation based on a two-layer neural network.

Original: The main limitation of the aforementioned work is the unrealistic assumption that all the data used in the Q-learning algorithm are sampled i.i.d.
Revised: The main limitation of the aforementioned work is the unrealistic assumption that all the data used in the Q-learning algorithm are sampled i.i.d.

Original: from a ﬁxed stationary distribution, which fails to capture the practical setting of neural Q-learning.
Revised: from a ﬁxed stationary distribution, which fails to capture the practical setting of neural Q-learning.

Original: In this paper, in order to bridge the gap between the empirical success of neural Q-learning and the theory of conventional Q-learning (i.e., tabular Q-learning, and Q-learning with linear function approximation), we study the non-asymptotic convergence of a neural Q-learning algorithm under non-i.i.d. observations.
Revised: In this paper, in order to bridge the gap between the empirical success of neural Q-learning and the theory of conventional Q-learning (i.e., tabular Q-learning, and Q-learning with linear function approximation), we study the non-asymptotic convergence of a neural Q-learning algorithm under non-i.i.d. observations.

Original: In particular, we use a deep neural network with the ReLU activation function to approximate the action-value function.
Revised: In particular, we use a deep neural network with the ReLU activation function to approximate the action-value function.

Original: In each iteration of the neural Q-learning algorithm, it updates the network weight parameters using the temporal difference (TD) error and the gradient of the neural network function.
Revised: In each iteration of the neural Q-learning algorithm, it updates the network weight parameters using the temporal difference (TD) error and the gradient of the neural network function.

Original: Our work extends existing ﬁnite-time analyses for TD learning 
Revised: Our work extends existing ﬁnite-time analyses for TD learning (Bhandari et al., 2018) and Q-learning (Zou et al., 2019b), from linear function approximation to deep

Original: [Table]
Revised: [Table]

Original: Bhandari et al.
Revised: Bhandari et al.

Original: (2018) (cid:51) (cid:55) (cid:55)
Revised: (2018) (cid:51) (cid:55) (cid:55)

Original: O (1 /T )
Revised: [Table] [Table]

Original: Zou et al.
Revised: Zou et al.

Original: (2019b) (cid:51) (cid:55) (cid:55)
Revised: (2019b) (cid:51) (cid:55) (cid:55)

Original: O (1 /T )
Revised: O (1 /T )

Original: Cai et al.
Revised: Cai et al.

Original: (2019) (cid:55) (cid:51) (cid:55)
Revised: (2019a) (cid:55) (cid:51) (cid:55)

Original: O (1 / √ T )
Revised: O (1 / √ T )

Original: This paper (cid:51) (cid:51) (cid:51)
Revised: This paper (cid:51) (cid:51) (cid:51)

Original: O (1 /T )(Bhandari et al., 2018) and Q-learning (Zou et al., 2019b), from linear function approximation to deep neural network based function approximation.
Revised: O (1 / √ T ) neural network based function approximation.

Original: Compared with the very recent theoretical work for neural Q-learning (Yang et al., 2019; Cai et al., 2019), our analysis relaxes the non-realistic i.i.d.
Revised: Compared with the very recent theoretical work for neural Q-learning (Yang et al., 2019; Cai et al., 2019a), our analysis relaxes the non-realistic i.i.d.

Original: data assumption and applies to neural network approximation with arbitrary number of layers.
Revised: data assumption and applies to neural network approximation with arbitrary number of layers.

Original: Our main contributions are summarized as follows
Revised: Our main contributions are summarized as follows

Original: • We establish the ﬁrst ﬁnite-time analysis of Q-learning with deep neural network function approximation when the data are generated from an Markov decision process (MDP).
Revised: • We establish the ﬁrst ﬁnite-time analysis of Q-learning with deep neural network function approximation when the data are generated from a Markov decision process (MDP).

Original: We show that, when the network is sufﬁciently wide, neural Q-learning converges to the optimal action-value function up to the approximation error of the neural network function class.
Revised: We show that, when the network is sufﬁciently wide, neural Q-learning converges to the optimal action-value function up to the approximation error of the neural network function class.

Original: • We establish an O (1 /T ) convergence rate of neural Q-learning to the optimal Q-value function up to the approximation error, where T is the number of iterations.
Revised: • We establish an O (1 / √ T ) convergence rate of neural Q-learning to the optimal Q-value function up to the approximation error, where T is the number of iterations.

Original: This convergence rate matches the one for Q-learning with linear function approximation (Zou et al., 2019b).
Revised: This convergence rate matches the one for TD-learning with linear function approximation and constant stepsize (Bhandari et al., 2018).

Original: It is worth noting that although we study a more challenging setting where the data are non-i.i.d.
Revised: Although we study a more challenging setting where the data are non-i.i

Original:  and the neural network approximator has multiple layers, our convergence rate is still faster than the O (1 / √ T ) rate proved in Cai et al. (2019) with i.i.d.
Revised: d. and the neural network approximator has multiple layers, our convergence rate also matches the O (1 / √ T ) rate proved in Cai et al. (2019a) with i.i.d.

Original: data and a two- layer neural network approximator.
Revised: data and a two-layer neural network approximator.

Original: To sum up, we present a comprehensive comparison between our work and the most relevant work in terms of their respective settings and convergence rates in Table 1.
Revised: To sum up, we present a comprehensive comparison between our work and the most relevant work in terms of their respective settings and convergence rates in Table 1.

Original: Notation We denote [ n ] = { 1 , . . . , n } for n ∈ N + .
Revised: Notation We denote [ n ] = { 1 , . . . , n } for n ∈ N + .

Original: (cid:107) x (cid:107) 2 is the Euclidean norm of a vector x ∈ R d .
Revised: (cid:107) x (cid:107) 2 is the Euclidean norm of a vector x ∈ R d .

Original: For a matrix W ∈ R m × n , we denote by (cid:107) W (cid:107) 2 and (cid:107) W (cid:107) F its operator norm and Frobenius norm respectively.
Revised: For a matrix W ∈ R m × n , we denote by (cid:107) W (cid:107) 2 and (cid:107) W (cid:107) F its operator norm and Frobenius norm respectively.

Original: We denote by vec ( W ) the vectorization of W , which converts W into a column vector.
Revised: We denote by vec ( W ) the vectorization of W , which converts W into a column vector.

Original: For a semi-deﬁnite matrix Σ ∈ R d × d and a vector x ∈ R d , (cid:107) x (cid:107) Σ = √ x (cid:62) Σx denotes the
Revised: For a semi-deﬁnite matrix Σ ∈ R d × d and a vector x ∈ R d , (cid:107) x (cid:107) Σ = √ x (cid:62) Σx denotes the

Original: Mahalanobis norm.
Revised: Mahalanobis norm.

Original: We reserve the notations { C i } i =0 , 1 ,... to represent universal positive constants that are independent of problem parameters.
Revised: We reserve the notations { C i } i =0 , 1 ,... to represent universal positive constants that are independent of problem parameters.

Original: The speciﬁc value of { C i } i =1 , 2 ,... can be different line by line.
Revised: The speciﬁc value of { C i } i =1 , 2 ,... can be different line by line.

Original: We write a n = O ( b n ) if a n ≤ Cb n for some constant C > 0 and a n = (cid:101) O ( b n ) if a n = O ( b n ) up to some logarithmic terms of b n .
Revised: We write a n = O ( b n ) if a n ≤ Cb n for some constant C > 0 and a n = (cid:101) O ( b n ) if a n = O ( b n ) up to some logarithmic terms of b n .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Due to the huge volume of work in the literature for TD learning and Q-learning algorithms, we only review the most relevant work here.
Revised: Due to the huge volume of work in the literature for TD learning and Q-learning algorithms, we only review the most relevant work here.

Original: Asymptotic
Revised: Asymptotic

Original: analysis The asymptotic convergence of TD learning and Q-learning algorithms has been well established in the literature (Jaakkola et al., 1994; Tsitsiklis & Van Roy, 1997; Konda &
Revised: analysis The asymptotic convergence of TD learning and Q-learning algorithms has been well established in the literature (Jaakkola et al., 1994; Tsitsiklis & Van Roy, 1997; Konda &

Original: Tsitsiklis, 2000; Borkar & Meyn, 2000; Ormoneit & Sen, 2002; Melo et al., 2008; Devraj & Meyn, 2017).
Revised: Tsitsiklis, 2000; Borkar & Meyn, 2000; Ormoneit & Sen, 2002; Melo et al., 2008; Devraj & Meyn, 2017).

Original: In particular, Tsitsiklis & Van Roy (1997) speciﬁed the precise conditions for TD learning with linear function approximation to converge and gave counterexamples that diverge.
Revised: In particular, Tsitsiklis & Van Roy (1997) speciﬁed the precise conditions for TD learning with linear function approximation to converge and gave counterexamples that diverge.

Original: Melo et al.
Revised: Melo et al.

Original: (2008) proved the asymptotic convergence of Q-learning with linear function approximation from standard ODE analysis, and identiﬁed a critic condition on the relationship between the learning policy and the greedy policy that ensures the almost sure convergence.
Revised: (2008) proved the asymptotic convergence of Q-learning with linear function approximation from standard ODE analysis, and identiﬁed a critic condition on the relationship between the learning policy and the greedy policy that ensures the almost sure convergence.

Original: Finite-time analysis The ﬁnite-time analysis of the convergence rate for Q-learning algorithms has been largely unexplored until recently.
Revised: Finite -time analysis The ﬁnite-time analysis of the convergence rate for Q-learning algorithms has been largely unexplored until recently.

Original: In speciﬁc, Dalal et al.
Revised: In speciﬁc, Dalal et al.

Original: ; Lakshminarayanan & Szepesvari (2018) studied the convergence of TD(0) algorithm with linear function approximation under i.i.d.
Revised: ; Lakshminarayanan & Szepesvari (2018) studied the convergence of TD(0) algorithm with linear function approximation under i.i.d.

Original: data assumptions and constant step sizes.
Revised: data assumptions and constant step sizes.

Original: Concurrently, a seminal work by Bhandari et al.
Revised: Concurrently, a seminal work by Bhandari et al.

Original: (2018) provided a uniﬁed framework of analysis for TD learning under both i.i.d.
Revised: (2018) provided a uniﬁed framework of analysis for TD learning under both i.i.d.

Original: and Markovian noise assumptions with an extra projection step.
Revised: and Markovian noise assumptions with an extra projection step.

Original: The analysis has been extended by Zou et al.
Revised: The analysis has been extended by Zou et al.

Original: (2019b) to SARSA and Q-learning algorithms with linear function approximation.
Revised: (2019b) to SARSA and Q-learning algorithms with linear function approximation.

Original: More recently,
Revised: More recently,

Original: Srikant & Ying (2019) established the ﬁnite-time convergence for TD learning algorithms with linear function approximation and a constant step-size without the extra projection step under non-i.i.d.
Revised: Srikant & Ying (2019) established the ﬁnite-time convergence for TD learning algorithms with linear function approximation and a constant step-size without the extra projection step under non-i.i.d.

Original: data assumptions.
Revised: data assumptions through carefully choosing the Lyapunov function for the associated ordinary differential equation of TD update. A similar analysis was also extended to Q-learning with linear function approximation (Chen et al., 2019).

Original: Hu & Syed (2019) further provided a uniﬁed analysis for a class of TD learning algorithms using Markov jump linear system.
Revised: Hu & Syed (2019) further provided a uniﬁed analysis for a class of TD learning algorithms using Markov jump linear system.

Original: Neural function approximation Despite the empirical success of DQN, the theoretical convergence of Q-learning with deep neural network approximation is still missing in the literature.
Revised: Neural function approximation Despite the empirical success of DQN, the theoretical convergence of Q-learning with deep neural network approximation is still missing in the literature.

Original: Following the recent advances in the theory of deep learning for overparameterized networks (Du et al., 2019b;a; 
Revised: Following the recent advances in the theory of deep learning for overparameterized networks (Jacot et al., 2018; Chizat & Bach, 2018; Du et al., 2019b;a; Allen-Zhu et al.

Original: Allen-Zhu et al., 2019; Zou et al., 2019a;  Cao & Gu, 2019a; Zou & Gu, 2019), two recent work by Yang et al.
Revised: 2019b;a; Zou et al., 2019a; Arora et al., 2019; Cao & Gu, 2019a; Zou & Gu, 2019; Cai et al., 2019b), two recent work by Yang et al.

Original:  and Cai et al.
Revised: (2019) and Cai et al.

Original: (2019) proved the convergence rates of ﬁtted Q-iteration and Q-learning with a sparse multi-layer ReLU network and two-layer neural network approximation respectively, under i.i.d.
Revised: (2019a) proved the convergence rates of ﬁtted Q-iteration and Q-learning with a sparse multi-layer ReLU network and two-layer neural network approximation respectively, under i.i.d.

Original: observations.
Revised: observations.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A discrete-time Markov Decision Process (MDP) is denoted by a tuple M =
Revised: A discrete-time Markov Decision Process (MDP) is denoted by a tuple M =

Original: ( S , A , P , r, γ ) .
Revised: ( S , A , P , r, γ ) .

Original: S and
Revised: S and

Original: A are the sets of all states and actions respectively.
Revised: A are the sets of all states and actions respectively.

Original: P : S ×
Revised: P : S ×

Original: A → P ( S ) is the transition kernel such that P ( s (cid:48) | s, a ) gives the probability of transiting to state s (cid:48) after taking action a at state s .
Revised: A → P ( S ) is the transition kernel such that P ( s (cid:48) | s, a ) gives the probability of transiting to state s (cid:48) after taking action a at state s .

Original: r : S × A → [ − 1 , 1] is a deterministic reward function.
Revised: r : S × A → [ − 1 , 1] is a deterministic reward function.

Original: γ ∈ (0 , 1) is the discounted factor.
Revised: γ ∈ (0 , 1) is the discounted factor.

Original: A policy π : S → P ( A ) is a function mapping a state s ∈ S to a probability distribution π ( ·| s ) over the action space.
Revised: A policy π : S → P ( A ) is a function mapping a state s ∈ S to a probability distribution π ( ·| s ) over the action space.

Original: Let s t and a t denote the state and action at time step t .
Revised: Let s t and a t denote the state and action at time step t .

Original: Then the transition kernel P and the policy
Revised: Then the transition kernel P and the policy

Original: π determine a Markov chain { s t } t =0 , 1 ,...
Revised: π determine a Markov chain { s t } t =0 , 1 ,...

Original: For any ﬁxed policy π , its associated value function V π : S → R is deﬁned as the expected total discounted reward:
Revised: For any ﬁxed policy π , its associated value function V π : S → R is deﬁned as the expected total discounted reward:

Original: [Equation]
Revised: [Equation]

Original: The corresponding action-value function Q π : S ×
Revised: The corresponding action-value function Q π : S ×

Original: A → R is deﬁned as
Revised: A → R is deﬁned as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: , for all s ∈ S , a ∈ A .
Revised: for all s ∈ S , a ∈ A .

Original: The optimal action-value function Q ∗ is deﬁned as Q ∗ ( s, a ) =
Revised: The optimal action-value function Q ∗ is deﬁned as Q ∗ ( s, a ) =

Original: sup π Q π ( s, a ) for all ( s, a ) ∈ S × A .
Revised: sup π Q π ( s, a ) for all ( s, a ) ∈ S × A .

Original: Based on Q ∗ , the optimal policy π ∗ can be derived by following the greedy algorithm such that π ∗ ( a | s ) = 1{+
+} if Q ( s, a ) = max b ∈A Q ∗ ( s, b ) and π ∗ ( a | s ) = [-
-]
Revised: Based on Q ∗ , the optimal policy π ∗ can be derived by following the greedy algorithm such that π ∗ ( a | [Equation] = {+
+} if Q ( s, a ) = max b ∈A Q ∗ ( s, b ) and π ∗ ( a | s ) = [-
-]0 otherwise.

Original: otherwise.

Original: We deﬁne the optimal Bellman operator T as follows
Revised: We deﬁne the optimal Bellman operator T as follows

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that the optimal Bellman operator T is γ -contractive in the sup-norm and Q ∗ is the unique ﬁxed point of T (Bertsekas et al., 1995).
Revised: It is worth noting that the optimal Bellman operator T is γ -contractive in the sup-norm and Q ∗ is the unique ﬁxed point of T (Bertsekas et al., 1995).

Original:  T HE N EURAL Q-L EARNING A LGORITHM
Revised:  T HE N EURAL Q-L EARNING A LGORITHM

Original: In this section, we start with a brief review of Q-learning with linear function approximation.
Revised: In this section, we start with a brief review of Q-learning with linear function approximation.

Original: Then we will present the neural Q-learning algorithm.
Revised: Then we will present the neural Q-learning algorithm.

Original: 4. Q-L EARNING WITH L INEAR F UNCTION A PPROXIMATION
Revised: 4. Q-L EARNING WITH L INEAR F UNCTION A PPROXIMATION

Original: In many reinforcement learning algorithms, the goal is to estimate the action-value function Q ( · , · ) , which can be formulated as minimizing the mean-squared Bellman error (MSBE) [Equation] Sutton & Barto, 2018):
Revised: In many reinforcement learning algorithms, the goal is to estimate the action-value function Q ( · , · ) , which can be formulated as minimizing the mean-squared Bellman error (MSBE) (Sutton & Barto, 2018):

Original: [Equation]
Revised: [Equation]

Original: where state s is generated from the initial state distribution µ and action a is chosen based on a ﬁxed learning policy π .
Revised: where state s is generated from the initial state distribution µ and action a is chosen based on a ﬁxed learning policy π .

Original: To optimize (4.1), Q-learning iteratively updates the action-value function using the Bellman operator in (3.1), i.e., Q t +1 ( s, a ) =
Revised: To optimize (4.1), Q-learning iteratively updates the action-value function using the Bellman operator in (3.1), i.e., Q t +1 ( s, a ) =

Original: T Q t ( s, a ) for all ( s, a ) ∈ S × A .
Revised: T Q t ( s, a ) for all ( s, a ) ∈ S × A .

Original: However, due to the large state and action spaces, whose cardinalities, i.e., |S| and |A| , can be inﬁnite for continuous problems in many applications, the aforementioned update is impractical.
Revised: However, due to the large state and action spaces, whose cardinalities, i.e., |S| and |A| , can be inﬁnite for continuous problems in many applications, the aforementioned update is impractical.

Original: To address this issue, a linear function approximator is often used (Szepesvari, 2010; Sutton & Barto, 2018), where the action-value function is assumed to be parameterized by a linear function, i.e., Q ( s, a ; θ ) =
Revised: To address this issue, a linear function approximator is often used (Szepesvari, 2010; Sutton & Barto, 2018), where the action-value function is assumed to be parameterized by a linear function, i.e., Q ( s, a ; θ ) =

Original: φ ( s, a ) (cid:62) θ for any ( s, a ) ∈ S × A , where φ : S ×
Revised: φ ( s, a ) (cid:62) θ for any ( s, a ) ∈ S × A , where φ : S ×

Original: A → R d maps the state-action pair to a ddimensional vector, and θ ∈ Θ ⊆
Revised: A → R d maps the state-action pair to a ddimensional vector, and θ ∈ Θ ⊆

Original: R d is an unknown weight vector.
Revised: R d is an unknown weight vector.

Original: The minimization problem in (4.1) then turns to minimizing the MSBE over the parameter space Θ .
Revised: The minimization problem in (4.1) then turns to minimizing the MSBE over the parameter space Θ .

Original: 1: Input: learning policy π , learning rate { η t } t =0 , 1 ,... , discount factor γ , constraint set Θ , Randomly generate the entries of W (0) l from N (0 , 1 /m ) , l = 1 , . . .

Original: , m 2: Initialization: θ 0 =

Original: ( W (1) (cid:62), . . .

Original: , W ( L ) (cid:62))

Original: (cid:62) 3: for t = 0 , . . .

Original: , T − 1 do 4:

Original: Sample data ( s t , a t , r t , s t +1 ) from policy π 5: ∆ t =

Original: f

Original: ( θ t ; φ ( s t , a t )) − ( r t + γ max b ∈A f

Original: ( θ t ; φ ( s t +1 , b )))

Original: 6: g t ( θ t ) =

Original: ∇ θ f

Original: ( θ t ; φ ( s t , a t ))

Original: ∆ t 7: θ t +1 =

Original: Π Θ ( θ t − η t g t ( θ t )) 8: end for 9: Output: θ T

Original:  N EURAL Q-L EARNING
Revised: 4. N EURAL Q-L EARNING

Original: Analogous to Q-learning with linear function approximation, the action-value function can also be approximated by a deep neural network to increase the representation power of the approximator.
Revised: Analogous to Q-learning with linear function approximation, the action-value function can also be approximated by a deep neural network to increase the representation power of the approximator.

Original: Speciﬁcally, we deﬁne a L -hidden-layer neural network as follows
Revised: Speciﬁcally, we deﬁne a L -hidden-layer neural network as follows

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R
Revised: where x ∈ R

Original: d is the input data, W 1 ∈ R m × d , W L ∈ R 1 × m and W l ∈ R m × m for l = 2 , . . .
Revised: d is the input data, W 1 ∈ R m × d , W L ∈ R 1 × m and W l ∈ R m × m for l = 2 , . . .

Original: , L − 1 , θ = ( vec ( W 1 ) (cid:62) , . . .
Revised: , L − 1 , θ = ( vec ( W 1 ) (cid:62) , . . .

Original: , vec ( W L ) (cid:62) ) (cid:62) is the concatenation of the vectorization of all parameter matrices, and σ ( x ) = max { 0 , x } is the ReLU activation function.
Revised: , vec ( W L ) (cid:62) ) (cid:62) is the concatenation of the vectorization of all parameter matrices, and σ ( x ) = max { 0 , x } is the ReLU activation function.

Original: Then, we can parameterize
Revised: Then, we can parameterize

Original: Q ( s, a ) using a deep neural network as Q ( s, a ; θ ) =
Revised: Q ( s, a ) using a deep neural network as Q ( s, a ; θ ) =

Original: f ( θ ; φ ( s, a )) , where θ ∈ Θ and φ : S ×
Revised: f ( θ ; φ ( s, a )) , where θ ∈ Θ and φ : S ×

Original: A → R d is a feature mapping.
Revised: A → R d is a feature mapping.

Original: Without loss of generality, we assume that (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 in this paper.
Revised: Without loss of generality, we assume that (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 in this paper.

Original: Let π be an arbitrarily stationary policy.
Revised: Let π be an arbitrarily stationary policy.

Original: The MSBE minimization problem in (4.1) can be rewritten in the following form
Revised: The MSBE minimization problem in (4.1) can be rewritten in the following form

Original: [Equation]
Revised: [Equation]

Original: Recall that the optimal action-value function Q ∗ is the ﬁxed point of Bellman optimality operator T which is γ -contractive.
Revised: Recall that the optimal action-value function Q ∗ is the ﬁxed point of Bellman optimality operator T which is γ -contractive.

Original: Therefore Q ∗ is the unique global minimizer of (4.3).
Revised: Therefore Q ∗ is the unique global minimizer of (4.3).

Original: The nonlinear parameterization of Q ( · , · ) turns the MSBE in (4.3) to be highly nonconvex, which imposes difﬁculty in ﬁnding the global optimum θ ∗ .
Revised: The nonlinear parameterization of Q ( · , · ) turns the MSBE in (4.3) to be highly nonconvex, which imposes difﬁculty in ﬁnding the global optimum θ ∗ .

Original: To mitigate this issue, we will approximate the solution of (4.3) by project the Q-value function into some function class parameterized by θ , which leads to minimizing the mean square projected Bellman error (MSPBE):
Revised: To mitigate this issue, we will approximate the solution of (4.3) by project the Q-value function into some function class parameterized by θ , which leads to minimizing the mean square projected Bellman error (MSPBE):

Original: [Equation]
Revised: [Equation]

Original: where F = { Q ( · , · ; θ ) : θ ∈ Θ } is some function class parameterized by θ ∈ Θ , and Π F is a projection operator.
Revised: where F = { Q ( · , · ; θ ) : θ ∈ Θ } is some function class parameterized by θ ∈ Θ , and Π F is a projection operator.

Original: Then the neural Q-learning algorithm updates the weight parameter θ using the following projected descent step: θ t +1 =
Revised: Then the neural Q-learning algorithm updates the weight parameter θ using the following projected descent step: θ t +1 =

Original: Π Θ ( θ t − η t g t ( θ t )) , where the gradient term g t ( θ t ) is deﬁned as
Revised: Π Θ ( θ t − η t g t ( θ t )) , where the gradient term g t ( θ t ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and ∆ t is the temporal difference (TD) error.
Revised: and ∆ t is the temporal difference (TD) error.

Original: It should be noted that g t is not the gradient of the MSPBE nor an unbiased estimator for it.
Revised: It should be noted that g t is not the gradient of the MSPBE nor an unbiased estimator for it.

Original: The details of the neural Q-learning algorithm are displayed in Algorithm 1, where θ 0 is randomly initialized, and the constraint set is chosen to be Θ
Revised: The details of the neural Q-learning algorithm are displayed in Algorithm 1, where θ 0 is randomly initialized, and the constraint set is chosen to be Θ

Original: = B ( θ 0 , ω ) , which is deﬁned as follows
Revised: = B ( θ 0 , ω ) , which is deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original:  for some tunable parameter ω .
Revised: ) for some tunable parameter ω .

Original: It is easy to verify that (cid:107) θ − θ (cid:48)
Revised: It is easy to verify that (cid:107) θ − θ (cid:48)

Original: (cid:107) 

Revised: (cid:107)

Original: = (cid:80) Ll [Equation] [Equation] W l − W 
Revised: = (cid:80) Ll =1 (cid:107) W l − W (cid:48)

Original: [Equation]

Original: l [Equation] F [Equation]
Revised: l (cid:107) 2 F .

Original:  C ONVERGENCE A NALYSIS OF N EURAL Q-L EARNING
Revised:  C ONVERGENCE A NALYSIS OF N EURAL Q-L EARNING

Original: In this section, we provide a ﬁnite-sample analysis of neural Q-learning.
Revised: In this section, we provide a ﬁnite-sample analysis of neural Q-learning.

Original: Note that the optimization problem in (4.4) is nonconvex.
Revised: Note that the optimization problem in (4.4) is nonconvex.

Original: We focus on ﬁnding a surrogate action-value function in the neural network function class that well approximates Q ∗ .
Revised: We focus on ﬁnding a surrogate action-value function in the neural network function class that well approximates Q ∗ .

Original: 5. A PPROXIMATE S TATIONARY P OINT IN THE C ONSTRAINED S PACE
Revised: 5. A PPROXIMATE S TATIONARY P OINT IN THE C ONSTRAINED S PACE

Original: In Algorithm 1, the initial point θ0 is randomly initialized with entries independently sampled from N (0 , 1 /m ) . We remark that Cai et al.
Revised: To ease the presentation, we abbreviate f ( θ ; φ ( s, a )) as f ( θ ) when no confusion arises. We deﬁne the function class F Θ ,m as a collection of all local linearization of f ( θ ) at the initial point θ

Original: (2019) uses N (0 , 1 /d )
Revised: F Θ ,m = { f ( θ 0 )

Original: as the initialization distribution. Moreover, they have an additional 1 /m factor in the deﬁnition of the neural network function as compared to (4.2).
Revised: + (cid:104)∇ θ f

Original: Therefore, the function value of f ( θ 0 ) still remains in the order of O (1) as is in our paper and thus their deﬁnition of f ( θ ) and the initialization are directly comparable to our setting.
Revised: ( θ 0 ) , θ − θ 0 (cid:105) : θ ∈ Θ } .

Original: To ease the presentation, we abbreviate f ( θ ; φ ( s, a )) as f ( θ ) when no confusion arises. We deﬁne the function class F Θ ,m as a collection of all local linearization of f ( θ ) at the initial point θ 
Revised: 1: Input: learning policy π , learning rate { η t } t =0 , 1 ,... , discount factor γ , constraint set Θ , Randomly generate the entries of W (0) l from N (0 , 1 /m ) , l = 1 , . . . , m 2: Initialization: θ 0 =

Original: [Equation]
Revised: ( W (1) (cid:62), . . . , W ( L ) (cid:62)) (cid:62) 3: for t = 0 , . . . , T − 1 do 4:

Original: According to the implicit linearization analysis of overparameterized deep neural networks (Cao & Gu, 2019a;b; Cai et al., 2019), the function class F Θ ,m will contain the optimal action-value function Q ∗ when the number of nodes m in each layer is sufﬁciently large.
Revised: Sample data ( s t , a t , r t , s t +1 ) from policy π 5: ∆ t = f ( θ t ; φ ( s t , a t )) − ( r t + γ max b ∈A f ( θ t ; φ ( s t +1 , b )))

Original: Unlike the explicit linearization in Bhatnagar et al.
Revised: 6: g t ( θ t ) = ∇ θ f ( θ t ; φ ( s t , a t )) ∆ t 7: θ t +1 =

Original: (2009), this local linearization is only implicitly implied by the overparameterization of the DNN.
Revised: Π Θ ( θ t − η t g t ( θ t )) 8: end for Following to the local linearization analysis in Cai et al. (2019a)

Original: We now deﬁne the approximate stationary point of Algorithm 1.
Revised: , we deﬁne the approximate stationary point of Algorithm 1 as follows.

Original: Deﬁnition 5.1 (Cai et al. 
Revised: Deﬁnition 5.1 (Cai et al. (2019a)) .

Original: A point θ ∗ ∈ Θ is said to be the approximate stationary point of Algorithm 1 if for all θ ∈ Θ it holds that 
Revised: A point θ ∗ ∈ Θ is said to be the approximate stationary point of Algorithm 1 if for all θ ∈ Θ it holds that [Equation]

Original: [Equation]
Revised: [Equation]

Original: 5.2) where (cid:98) f  ∈ F Θ ,m and the temporal difference error (cid:98) ∆ is 
Revised: where (cid:98) f ( θ ; φ ( s, a )) : = (cid:98) f ( θ ) ∈ F Θ ,m and the temporal difference error (cid:98) ∆ is (cid:98) ∆( s, a, s (cid:48) ; θ ) = (cid:98)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For any (cid:98) f ∈ F Θ ,m , it holds that (cid:104)∇ θ (cid:98) f
Revised: For any (cid:98) f ∈ F Θ ,m , it holds that (cid:104)∇ θ (cid:98) f

Original: ( θ ∗ ) , θ − θ ∗ (cid:105) =
Revised: ( θ ∗ ) , θ − θ ∗ (cid:105) =

Original: (cid:104)∇ θ 
Revised: (cid:104)∇ θ f

Revised: ( θ 0 ) , θ − θ ∗ (cid:105) = (cid:98) f

Revised: ( θ ) − (cid:98) f

Original: [Equation]
Revised: ( θ ∗ ) .

Original: Deﬁnition 5.1 immediately implies
Revised: Deﬁnition 5.1 immediately implies

Original: [Equation]
Revised: [Equation]

Original: According to Proposition 4.2 in Cai et al. (2019), this further indicates (cid:98) f
Revised: According to Proposition 4.2 in Cai et al. (2019a), this further indicates (cid:98) f

Original: ( θ ∗ ) =
Revised: ( θ ∗ ) =

Original: Π F Θ ,m T (cid:98) f
Revised: Π F Θ ,m T (cid:98) f

Original: ( θ ∗ ) .
Revised: ( θ ∗ ) .

Original: In other words, (cid:98) f
Revised: In other words, (cid:98) f

Original: ( θ ∗ ) is the unique ﬁxed point of the MSPBE in (4.4).
Revised: ( θ ∗ ) is the unique ﬁxed point of the MSPBE in (4.4).

Original: Therefore, we can show the convergence of neural Q-learning to the optimal action-value function Q ∗ by ﬁrst connecting it to the minimizer (cid:98) f
Revised: Therefore, we can show the convergence of neural Q-learning to the optimal action-value function Q ∗ by ﬁrst connecting it to the minimizer (cid:98) f

Original: ( θ ∗ ) and then adding the approximation error of F Θ ,m .
Revised: ( θ ∗ ) and then adding the approximation error of F Θ ,m .

Original: 5. T HE M AIN T HEORY
Revised: 5. T HE M AIN T HEORY

Original: Before we present the convergence of Algorithm 1, let us lay down the assumptions used throughout our paper.
Revised: Before we present the convergence of Algorithm 1, let us lay down the assumptions used throughout our paper.

Original: The ﬁrst assumption controls the bias caused by the Markovian noise in the observations through assuming the uniform ergodicity of the Markov chain generated by the learning policy π .
Revised: The ﬁrst assumption controls the bias caused by the Markovian noise in the observations through assuming the uniform ergodicity of the Markov chain generated by the learning policy π .

Original: Assumption 5.2.
Revised: Assumption 5.2.

Original: The learning policy π and the transition kernel P induce a Markov chain { s t } t =0 , 1 ,... such that there exist constants λ > 0 and ρ ∈ (0 , 1) satisfying 
Revised: The learning policy π and the transition kernel P induce a Markov chain { s t } t =0 , 1 ,... such that there exist constants λ > 0 and ρ ∈ (0 , 1) satisfying sup s ∈S d TV ( P ( s

Original: [Equation]
Revised: [Equation]

Original: for all t = 
Revised: ·| s 0 = s ) , π )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: t , for all t = 0 , 1 , . . .

Original: Assumption 5.2 also appears in Bhandari et al.
Revised: Assumption 5.2 also appears in Bhandari et al.

Original: ; Zou et al.
Revised: ; Zou et al.

Original: (2019b), which is essential for the analysis of the Markov decision process.
Revised: (2019b), which is essential for the analysis of the Markov decision process.

Original: The uniform ergodicity can be established via the minorization condition for irreducible Markov chains (Meyn & Tweedie, 2012; Levin & Peres, 2017).
Revised: The uniform ergodicity can be established via the minorization condition for irreducible Markov chains (Meyn & Tweedie, 2012; Levin & Peres, 2017).

Original: For the purpose of exploration, we also need to assume that the learning policy π satisﬁes some regularity condition.
Revised: For the purpose of exploration, we also need to assume that the learning policy π satisﬁes some regularity condition.

Original: Denote b max ( θ ) = argmax b ∈A (cid:98) f
Revised: Denote b max ( θ ) = argmax b ∈A |(cid:104)∇ θ f

Original: ( θ  ; s, b max ( θ )) .
Revised: ( θ 0 ; s, b ) , θ (cid:105)| for any θ ∈ Θ .

Original: Similar to Melo et al.
Revised: Similar to Melo et al.

Original:  Zou et al.
Revised: ; Zou et al.

Original: (2019b), we deﬁne[Equation]
Revised: (2019b) ; Chen et al. (2019), we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that(cid:98) Σ π is independent of θ and only depends on the policy π and the initial point θ 0 in the deﬁnition of (cid:98) f .
Revised: Note that Σ π is independent of θ and only depends on the policy π and the initial point θ 0 in the deﬁnition of (cid:98) f .

Original: In contrast,(cid:98) Σ ∗ π ( θ ) is deﬁned based on the greedy action under the policy associated with θ .
Revised: In contrast, Σ ∗ π ( θ ) is deﬁned based on the greedy action under the policy associated with θ .

Original: The scaling parameter 1 /m is used to ensure that the operator norm of(cid:98) Σ π to be in the order of O (1) .
Revised: The scaling parameter 1 /m is used to ensure that the operator norm of Σ π to be in the order of O (1) .

Original: When f is linear, (cid:98) Σ 
Revised: It is worth noting that Σ π is different from the neural tangent kernel (NTK) or the Gram matrix in Jacot et al. Du et al. (2019a); Arora et al. (2019), which are n × n matrices deﬁned based on a ﬁnite set of data points { ( s i , a i ) } i =1 ,...,n .

Original:  π reduces to the covariance matrix of the feature vector.
Revised: When f is linear, Σ π reduces to the covariance matrix of the feature vector.

Original: Assumption 5.3.
Revised: Assumption 5.3.

Original: [-There exists a constant α > 
-]
Revised: [-There exists a constant α > 
-]

Original:  such that(cid:98) Σ π − γ 2(cid:98) Σ ∗ π [Equation] θ ) (cid:31) α I for all θ and θ 0 .
Revised: There exists a constant α > 1 such that Σ π − αγ 2 Σ ∗ π ( θ ) (cid:31) 0 for all θ and θ 0 .

Original: Here I is the identity matrix.

Original: Assumption 5.3 is also made for Q-learning with linear function approximation in Melo et al.
Revised: Assumption 5.3 is also made for Q-learning with linear function approximation in Melo et al.

Original: Zou et al.
Revised: Zou et al.

Original: (2019b).
Revised: (2019b)

Revised: ; Chen et al.

Revised: Moreover, Chen et al.

Revised: (2019) presented numerical simulations to verify the validity of Assumption 5.3.

Original: Cai et al.
Revised: Cai et al.

Original: (2019) imposed a slightly different assumption but with the same idea of quantifying how much information the learning policy π can provide about the greedy policy. 
Revised: (2019a) imposed a slightly different assumption but with the same idea that the learning policy π should be not too far away from the greedy policy. The regularity assumption on the learning policy is directly imposed on the action value function in

Revised: Cai et al.

Revised: (2019a), which can be implied by Assumption 5.3 and thus is slightly weaker.

Revised: We note that Assumption 5.3 can be relaxed to the one made in Cai et al. (2019a) without changing any of our analysis.

Revised: Nevertheless, we choose to present the current version which is more consistent with existing work on Q-learning with linear function approximation (Melo et al., 2008; Chen et al., 2019).

Original: Theorem 5.4.
Revised: Theorem 5.4.

Original: Suppose Assumptions 5.2 and 5.3 hold.
Revised: Suppose Assumptions 5.2 and 5.3 hold.

Original: The constraint set Θ is deﬁned as in (4.6).
Revised: The constraint set Θ is deﬁned as in (4.6).

Original: We set the radius as ω =
Revised: We set the radius as ω =

Original: C 0 m − 1 / 2  , the step size in Algorithm 1 as η = log T/ ( αmT ) , and the width of the neural network as m ≥ C 1 max { dL 2 log( m/δ ) , ω − 4 / 3 L − 8 / 3 log( m/ ( ωδ )) } , where δ ∈ (0 , 1) .
Revised: C 0 m − 1 / 2 L − 9 / 4 , the step size in Algorithm 1 as η = 1 / (2(1 − α − 1 / 2 ) mT ) , and the width of the neural network as m ≥ C 1 max { dL 2 log( m/δ ) , ω − 4 / 3 L − 8 / 3 log( m/ ( ωδ )) } , where δ ∈ (0 , 1) .

Original: Then with probability at least 1 − 2 δ − L 2 exp( − C 2 m 2 / 3 L ) over the randomness of the Gaussian initialization θ 0 , it holds that
Revised: Then with probability at least 1 − 2 δ − L 2 exp( − C 2 m 2 / 3 L ) over the randomness of the Gaussian initialization θ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where  τ ∗ = min { t = 0 , 1 , 2 , . ..
Revised: where β = 1 − α − 1 / 2 ∈ (0 , 1) is a constant, τ ∗ = min { t = 0 , 1 , 2 , . .

Original: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... , and { C i } i =0 ,..., 5 are universal constants independent of problem parameters.
Revised: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... , and { C i } i =0 ,..., 5 are universal constants independent of problem parameters.

Original: Remark 5.5.
Revised: Remark 5.5.

Original: Theorem 5.4 characterizes the distance between the output of Algorithm 1 to the approximate stationary point deﬁned in function class F Θ ,m .
Revised: Theorem 5.4 characterizes the distance between the output of Algorithm 1 to the approximate stationary point deﬁned in function class F Θ ,m .

Original: From (5.4), we know that (cid:98) f
Revised: From (5.4), we know that (cid:98) f

Original: ( θ ∗ ) is the minimizer of the MSPBE (4.4).
Revised: ( θ ∗ ) is the minimizer of the MSPBE (4.4).

Original: Note that τ ∗ is in the order of O (log( mT/ log T )) .
Revised: Note that τ ∗ is in the order of O (log( mT/ log T )) .

Original: Theorem 5. suggests that neural Q-learning converges to the minimizer of MSPBE with a rate in the order of O ((log( mT )) 3 /T + log m log T/m 1 / 3 ) , which reduces to (cid:101) O(1 /T )
Revised: Theorem 5.4 suggests that neural Q-learning converges to the minimizer of MSPBE with a rate in the order of O ((log( mT )) 3 / √ T + log m log T/m 1 / 6 ) , which reduces to (cid:101) O

Original:  when the width m of the neural network is sufﬁciently large.
Revised: (1 / √ T ) when the width m of the neural network is sufﬁciently large.

Original: In the following theorem, we show that neural Q-learning converges to the optimal action-value function within ﬁnite time if the neural network is overparameterized.
Revised: In the following theorem, we show that neural Q-learning converges to the optimal action-value function within ﬁnite time if the neural network is overparameterized.

Original: Theorem 5.6.
Revised: Theorem 5.6.

Original: Under the same conditions as in Theorem 5.4, with probability at least 1 − 3 δ −
Revised: Under the same conditions as in Theorem 5.4, with probability at least 1 − 3 δ −

Original: L 2 exp( − C 0 m 2 / 3 L ) over the randomness of θ 0 , it holds that
Revised: L 2 exp( − C 0 m 2 / 3 L ) over the randomness of θ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where all the expectations are taken conditional on θ 0 , Q ∗ is the optimal action-value function, δ ∈ (0 , 1) and { C i } i =0 ,..., 3 are universal constants.
Revised: where all the expectations are taken conditional on θ 0 , Q ∗ is the optimal action-value function, δ ∈ (0 , 1) and { C i } i =0 ,..., 2 are universal constants.

Original: The optimal policy π ∗ can be obtained by the greedy algorithm derived based on Q ∗ .
Revised: The optimal policy π ∗ can be obtained by the greedy algorithm derived based on Q ∗ .

Original: Remark 5.7.
Revised: Remark 5.7.

Original: The convergence rate in Theorem 5.6 can be simpliﬁes as follows
Revised: The convergence rate in Theorem 5.6 can be simpliﬁes as follows

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is the projection error of the optimal Q-value function on to the function class F Θ ,m , which decreases to zero as the representation power of F Θ ,m increases.
Revised: The ﬁrst term is the projection error of the optimal Q-value function on to the function class F Θ ,m , which decreases to zero as the representation power of F Θ ,m increases.

Original: In fact, when the width m of the DNN is sufﬁciently large, recent studies (Cao & Gu, 2019a;b) show that f ( θ ) is almost linear around the initialization and the approximate stationary point (cid:98)
Revised: In fact, when the width m of the DNN is sufﬁciently large, recent studies (Cao & Gu, 2019a;b) show that f ( θ ) is almost linear around the initialization and the approximate stationary point (cid:98)

Original: f
Revised: f

Original: ( θ ∗ ) becomes the ﬁxed solution of the MSBE (Cai et al., 2019).
Revised: ( θ ∗ ) becomes the ﬁxed solution of the MSBE (Cai et al., 2019a).

Original: Moreover, this term diminishes when the Q function is approximated by linear functions when the underlying parameter has a bounded norm (Bhandari et al., 2018;
Revised: Moreover, this term diminishes when the Q function is approximated by linear functions when the underlying parameter has a bounded norm (Bhandari et al., 2018;

Original: Zou et al., 2019b).
Revised: Zou et al., 2019b).

Original: As m goes to inﬁnity, we obtain the convergence of neural Q-learning to the optimal Q-value function with an O (1 /T ) rate.
Revised: As m goes to inﬁnity, we obtain the convergence of neural Q-learning to the optimal Q-value function with an O (1 / √ T ) rate.

Original: It is worth noting that our convergence rate is faster than the O (1 / √ T ) rate in Cai et al.

Original: (2019) proved under stronger data assumptions and a two-layer neural network.

Original:  P ROOF OF M AIN R ESULTS
Revised:  P ROOF OF M AIN R ESULTS

Original: In this section, we provide the detailed proof of the convergence of Algorithm 1.
Revised: In this section, we provide the detailed proof of the convergence of Algorithm 1.

Original: To simplify the presentation, we write f ( θ ; φ ( s, a )) as f ( θ ; s, a ) throughout the proof when no confusion arises.
Revised: To simplify the presentation, we write f ( θ ; φ ( s, a )) as f ( θ ; s, a ) throughout the proof when no confusion arises.

Original: We ﬁrst deﬁne some notations that will simplify the presentation of the proof.
Revised: We ﬁrst deﬁne some notations that will simplify the presentation of the proof.

Original: Recall the deﬁnition of g t ( · ) in (4.5).
Revised: Recall the deﬁnition of g t ( · ) in (4.5).

Original: For any θ ∈ Θ , we deﬁne 
Revised: For any θ ∈ Θ , we deﬁne the following vector-value map g that is independent of the data point.

Original: [Equation]
Revised: [Equation]

Revised: where s follows the initial state distribution µ , a is chosen based on the policy π ( ·| s ) and s (cid:48) follows the transition probability P ( ·| s, a ) .

Original: For all θ ∈ Θ , we deﬁne the following gradient terms based on the linearized function (cid:98) f ∈ F Θ ,m
Revised: Similarly, for all θ ∈ Θ , we deﬁne the following gradient terms based on the linearized function (cid:98) f ∈ F Θ ,m

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) ∆ is deﬁned in (5.3), and a population version based on the linearized function.
Revised: where (cid:98) ∆ is deﬁned in (5.3), and a population version based on the linearized function.

Original: Next, we present thefollowing technical lemmas that characterize the linearization error, stochastic bias and estimation error of Algorithm 1.
Revised: Now we present the technical lemmas that are useful in our proof of Theorem 5.4. For the gradients g t ( · ) deﬁned in (4.5) and m t ( · ) deﬁned in (6.2), we have the following lemma that characterizes the difference between the gradient of the neural network function f and the gradient of the linearized function (cid:98) f .

Original: Lemma 6.1.
Revised: Lemma 6.1.

Original: The gradient of neural network function is close to the linearized gradient.
Revised: The gradient of neural network function is close to the linearized gradient.

Original: Speciﬁcally, if θ t ∈ B ( Θ , ω ) and m and ω satisfy
Revised: Speciﬁcally, if θ t ∈ B ( Θ , ω ) and m and ω satisfy

Original: [Equation]
Revised: [Equation]

Original: then it holds that
Revised: then it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point, and (cid:107) g t
Revised: with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point, and (cid:107) g t

Original: θ t ) (cid:107) 2 ≤ (2+ γ ) C 7 (cid:112) m log( T/δ ) holds with probability at least 1 − δ − L 2 exp( − C 6 mω 2 / 3 L ) .
Revised: θ t ) (cid:107) 2 ≤ (2+ γ ) C 7 (cid:112) m log( T/δ ) holds with probability at least 1 − δ − L 2 exp( − C 6 mω 2 / 3 L ) .

Original: where { C i > 0 } i =0 ,..., 7 are universal constants.
Revised: where { C i > 0 } i =0 ,..., 7 are universal constants.

Revised: The next lemma upper bounds the bias of the non-i.i.d.

Revised: data for the linearized gradient map.

Original: Lemma 6.2.
Revised: Lemma 6.2.

Original: Suppose the step size sequence { η 0 , η 1 , . .
Revised: Suppose the step size sequence { η 0 , η 1 , . .

Original: , η T } is nonincreasing.
Revised: , η T } is nonincreasing.

Original: Then it holds that
Revised: Then it holds that

Original: [Equation]
Revised: [Equation]

Original: for any ﬁxed t ≤ T , where C 0 > 0 is an universal constant and τ ∗ = min { t = 0 , 1 , 2 , . . .
Revised: for any ﬁxed t ≤ T , where C 0 > 0 is an universal constant and τ ∗ = min { t = 0 , 1 , 2 , . . .

Original: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... .
Revised: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... .

Revised: Since (cid:98) f is a linear function approximator of the neural network function f , we can show that the gradient of (cid:98) f satisﬁes the following nice property in the constrained set Θ .

Original: Lemma 6.3.
Revised: Lemma 6.3.

Original: Under Assumption 5.3, m ( · ) deﬁned in (6.2) satisﬁes
Revised: Under Assumption 5.3, m ( · ) deﬁned in (6.2) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Now we can integrate the  results and obtain proof of Theorem 5.4.
Revised: Now we can integrate the above results and obtain proof of Theorem 5.4.

Original: Proof of Theorem 5.4.
Revised: Proof of Theorem 5.4.

Original: By Algorithm 1 and the non-expansiveness of projection Π Θ , we have
Revised: By Algorithm 1 and the non-expansiveness of projection Π Θ , we have

Original: [Equation]
Revised: [Equation]

Original: We need to ﬁnd an upper bound for the gradient norm and a lower bound for the inner product.
Revised: We need to ﬁnd an upper bound for the gradient norm and a lower bound for the inner product.

Original: According to Deﬁnition 
Revised: According to Deﬁnition 5.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the approximate stationary point θ ∗ of Algorithm 1 satisﬁes (cid:104) m ( θ ∗ ) , θ − θ ∗ (cid:105) ≥ 0 for all θ ∈ Θ .
Revised: the approximate stationary point θ ∗ of Algorithm 1 satisﬁes (cid:104) m ( θ ∗ ) , θ − θ ∗ (cid:105) ≥ 0 for all θ ∈ Θ .

Original: The inner product in (6.4) can be decomposed into
Revised: The inner product in (6.4) can be decomposed into

Original: [Equation]
Revised: [Equation]

Original: Combining results from (6.4)and (6.5)
Revised: Combining results from (6.4)and (6.5)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall constraint set deﬁned in (4.6).
Revised: Recall constraint set deﬁned in (4.6).

Original: We choose Θ = B ( θ 0 , ω )=
Revised: We have Θ = B ( θ 0 , ω )

Original:  { θ : (cid:107) W l − W (0) l (cid:107) F ≤ ω, ∀ l = 1 , . . .
Revised: = { θ : (cid:107) W l − W (0) l (cid:107) F ≤ ω, ∀ l = 1 , . . .

Original: , L } and let m and ω satisfy the condition in (6.3).
Revised: , L } and that m and ω satisfy the condition in (6.3).

Original: Term I 1 is the error of the local linearization of f ( θ ) at θ 0 .
Revised: Term I 1 is the error of the local linearization of f ( θ ) at θ 0 .

Original: By Lemma 6.1, with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point θ 0 , we have
Revised: By Lemma 6.1, with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point θ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: holds uniformly for all θ t , θ ∗ ∈ Θ .
Revised: holds uniformly for all θ t , θ ∗ ∈ Θ , where we used the fact that ω =

Revised: C 0 m − 1 / 2 L − 9 / 4 .

Original: Term I 2 is the bias of caused by the non-i.i.d.
Revised: Term I 2 is the bias of caused by the non-i.i.d.

Original: data ( s t , a t , s t +1 ) used in the update of Algorithm 1.
Revised: data ( s t , a t , s t +1 ) used in the update of Algorithm 1.

Original: Conditional on the initialization, by Lemma 6.2, we have
Revised: Conditional on the initialization, by Lemma 6.2, we have

Original: [Equation]
Revised: [Equation]

Original: where τ ∗ = min { t = 0 , 1 , 2 , . . .
Revised: where τ ∗ = min { t = 0 , 1 , 2 , . . .

Original: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... .
Revised: | λρ t ≤ η T } is the mixing time of the Markov chain { s t , a t } t =0 , 1 ,... .

Original: Term I 3 is the estimation error for the linear function approximation.
Revised: Term I 3 is the estimation error for the linear function approximation.

Original: By Lemma 6.3, we have
Revised: By Lemma 6.3, we have

Original: [Equation]
Revised: [Equation]

Revised: where β = (1 − α − 1 / 2 ) ∈

Revised: (0 , 1) is a constant.

Original: Substituting (6.7), (6.8) and (6.9) into (6.6), we can obtain
Revised: Substituting (6.7), (6.8) and (6.9) into (6.6), we have it holds that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point θ 0 , where we used the fact that (cid:107) g t (cid:107) F ≤ C 4 (2 + γ ) (cid:112) m log( T/δ ) from Lemma 6.1.
Revised: with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point θ 0 , where we used the fact that (cid:107) g t (cid:107) F ≤ C 4 (2 + γ ) (cid:112) m log( T/δ ) from Lemma 6.1.

Original: Let η 0 = η 1 = . . . =
Revised: Rearranging the above inequality yields [Equation]

Original: η T − 1 < 1 / ( αm ) , then we have
Revised: β

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that (cid:80) T − 1t =0 (1 − 
Revised: with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point θ 0 . Recall the choices of the step sizes η 0 = . . .

Original: [Equation]
Revised: =

Original: t ≤ 1 / ( αmη 0 ) . Note that (1 − αmη 0 ) T ≤ exp( − αmη 0 T ) .
Revised: η T = 1 / (2 βm √ T ) and the radius ω = C 0 m − 1 / 2 L − 9 / 4 .

Original: We further choose η 0 = log T/ ( αmT ) and ω = 1 / √ m . Then we have
Revised: Dividing the above inequality by T and telescoping it from t = 0 to T yields

Original: [Equation]
Revised: [Equation]

Original: Note that Assumption 5.3 also implies that (cid:98) Σ π (cid:31) α I since (cid:98) Σ π ( θ) (cid:31) 0  by deﬁnition, which implies E
Revised: For θ 0 , θ ∗ ∈ Θ , again by ω =

Original: [Equation] [Equation]
Revised: Cm − 1 / 2 L − 9 / 4 , we have (cid:107) θ 0 − θ ∗ (cid:107) 22 ≤ 1 /m .

Original: Since (cid:98) f ( · ) ∈ F Θ ,m , by Lemma 6.1, it holds with probability at least 1 − 2 δ − 3 L 2 exp( − C 1{+
+} m 2 / 3 L ) over the randomness of the initial point θ 0 that[Equation] αT
Revised: Since (cid:98) f ( · ) ∈ F Θ ,m , by Lemma 6.1, it holds with probability at least 1 − 2 δ − 3 L 2 exp( − C {+
+} m 2 / 3 L ) over the randomness of the initial point θ 0 that

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that γ < 1 .
Revised: where we used the fact that γ < 1 .

Revised: This completes the proof.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we provide the ﬁrst ﬁnite-time analysis of Q-learning with neural network function approximation (i.e., neural Q-learning), where the data are generated from a Markov decision process and the action-value function is approximated by a deep ReLU neural network.
Revised: In this paper, we provide the ﬁrst ﬁnite-time analysis of Q-learning with neural network function approximation (i.e., neural Q-learning), where the data are generated from a Markov decision process and the action-value function is approximated by a deep ReLU neural network.

Original: We prove that neural Q-learning converge to the optimal action-value function up to the approximation error with
Revised: We prove that neural Q-learning converge to the optimal action-value function up to the approximation error with

Original: O (1 /T ) rate, where T is the number of iterations.
Revised: O (1 / √ T ) rate, where T is the number of iterations.

Original: Our proof technique is of independent interest and can be extended to analyze other deep reinforcement learning algorithms.
Revised: Our proof technique is of independent interest and can be extended to analyze other deep reinforcement learning algorithms.

Revised: One interesting future direction would be to remove the projection step in our algorithm by applying the ODE based analysis in Srikant & Ying (2019);

Revised: Chen et al.

Original: A P ROOF OF T HEOREM 5. 
Revised: A P ROOF OF T HEOREM 5.

Original: Before we prove the global convergence of Algorithm 1, we present the following lemma that shows that near the initialization point θ 0 , the neural network function f ( θ ; x ) is almost linear in θ for all unit input vectors.
Revised: Before we prove the global convergence of Algorithm 1, we present the following lemma that shows that near the initialization point θ 0 , the neural network function f ( θ ; x ) is almost linear in θ for all unit input vectors.

Original: Lemma A.1 (Theorems 5.3 and 5.4 in Cao & Gu (2019a)) .
Revised: Lemma A.1 (Theorems 5.3 and 5.4 in Cao & Gu (2019a)) .

Original: Let θ 

Revised: Let θ

Original: =
Revised: =

Original: ( W (1) (cid:62), . . .
Revised: ( W (1) (cid:62), . . .

Original: , W ( L ) (cid:62)) (cid:62) be the initial point and θ =
Revised: , W ( L ) (cid:62)) (cid:62) be the initial point and θ =

Original: ( W (1) (cid:62) , . . .
Revised: ( W (1) (cid:62) , . . .

Original: , W ( L ) (cid:62) )
Revised: , W ( L ) (cid:62) )

Original: (cid:62) ∈ B ( θ 0 , ω ) be a point in the neighborhood of θ 0 .
Revised: (cid:62) ∈ B ( θ 0 , ω ) be a point in the neighborhood of θ 0 .

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: , and ω ≤ C 2 L − 5 (log m ) − 3 / 2 , then for all x ∈ S d − 1 , with probability at least 1 − δ it holds that
Revised: then for all x ∈ S d − 1 , with probability at least 1 − δ it holds that

Original: [Equation]
Revised: [Equation]

Original: Under the same conditions on m and ω , if θ t ∈ B ( θ 0 , ω ) for all t = 1 , . . .
Revised: Under the same conditions on m and ω , if θ t ∈ B ( θ 0 , ω ) for all t = 1 , . . .

Original: , T , then with probability at least 1 − δ , we have | f ( θ t ; φ ( s t , a t )) | ≤ C 4 (cid:112) log( T/δ ) for all t ∈
Revised: , T , then with probability at least 1 − δ , we have | f ( θ t ; φ ( s t , a t )) | ≤ C 4 (cid:112) log( T/δ ) for all t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Proof of Theorem 5.6.
Revised: Proof of Theorem 5.6.

Original: By triangle inequality, it holds that
Revised: By triangle inequality, it holds that

Original: [Equation]
Revised: [Equation]

Original: Recall that (cid:98) f
Revised: Recall that (cid:98) f

Original: ( θ ∗ ; · , · ) is the ﬁxed point of Π F T and Q ∗ ( · , · ) is the ﬁxed point of T .
Revised: ( θ ∗ ; · , · ) is the ﬁxed point of Π F T and Q ∗ ( · , · ) is the ﬁxed point of T .

Original: Then we have(cid:98)
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where  we used the fact that Π F Θ ,m T is γ -contractive.
Revised: where the ﬁrst inequality follows the triangle inequality and in the second inequality we used the fact that Π F Θ ,m T is γ -contractive.

Original: This further leads to
Revised: This further leads to

Original: [Equation]
Revised: [Equation]

Original: s
Revised: To simplify the notation

Original: [Equation]
Revised: [Equation]

Original: | . To simplify the notation, we abbreviate E [ · (cid:12)(cid:12) θ 0 ] as E [ · ] in the rest of this proof.
Revised: we abbreviate E [ · (cid:12)(cid:12) θ 0 ] as E [ · ] in the rest of this proof.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.1 and the parameter choice that ω
Revised: By Lemma A.1 and the parameter choice that ω

Original: = C 1 /  √ m , we have
Revised: = C 1 / ( √ mL 9 / 4 ) , we have

Original: E [( f ( θ T ; s, a ) − (cid:98) f
Revised: [Equation]

Original: ( θ T ; s, a )) 2 ] ≤ C 2 ( ω 4 / 3 L 4 (cid:112) m log m ) 2 ≤ C 1 C 2 m − 1 / 3 L 8 log m with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Combining the above result with Theorem 5.4, we have
Revised: Combining the above result with Theorem 5.4, we have

Original: E (cid:2) ( Q ( s, a ; θ T ) − Q ∗ ( s, a )) 2 (cid:3) ≤ 3 E

Original: [Equation]

Original: αT +

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 3 δ − L 2 exp( − C 6 m 2 / 3 L ) , which completes the proof.
Revised: with probability at least 1 − 3 δ − L 2 exp( − C 6 m 2 / 3 L ) , which completes the proof.

Original: B P ROOF OF S UPPORTING L EMMAS B. P ROOF OF L EMMA 6. 
Revised: B P ROOF OF S UPPORTING L EMMAS B. P ROOF OF L EMMA 6.

Original: Before we prove the error bound for the local linearization, we ﬁrst present some useful lemmas from recent studies of overparameterized deep neural networks.
Revised: Before we prove the error bound for the local linearization, we ﬁrst present some useful lemmas from recent studies of overparameterized deep neural networks.

Original: Note that in the following lemmas, { C i } i =1 ,... are universal constants that are independent of problem parameters such as d, θ , m, L and their values can be different in different contexts.
Revised: Note that in the following lemmas, { C i } i =1 ,... are universal constants that are independent of problem parameters such as d, θ , m, L and their values can be different in different contexts.

Original: The ﬁrst lemma states the uniform upper bound for the gradient of the deep neural network.
Revised: The ﬁrst lemma states the uniform upper bound for the gradient of the deep neural network.

Original: Note that by deﬁnition, our parameter θ is a long vector containing the concatenation of the vectorization of all the weight matrices.
Revised: Note that by deﬁnition, our parameter θ is a long vector containing the concatenation of the vectorization of all the weight matrices.

Original: Correspondingly, the gradient ∇ θ f
Revised: Correspondingly, the gradient ∇ θ f

Original: ( θ ; x ) is also a long vector.
Revised: ( θ ; x ) is also a long vector.

Original: Lemma B.1 (Lemma B.3 in Cao & Gu (2019b)) .
Revised: Lemma B.1 (Lemma B.3 in Cao & Gu (2019b)) .

Original: Let θ ∈ B ( θ 0 , ω ) with the radius satisfying
Revised: Let θ ∈ B ( θ 0 , ω ) with the radius satisfying

Original: C 1 d 3 / 2 L − 1 m − 3 / 2 ≤ ω ≤ C 2 L − 6 (log m ) − 3 / 2 .
Revised: C 1 d 3 / 2 L − 1 m − 3 / 2 ≤ ω ≤ C 2 L − 6 (log m ) − 3 / 2 .

Original: Then for all unit vectors in R d , i.e., x ∈ S
Revised: Then for all unit vectors in R d , i.e., x ∈ S

Original: d − 1 , the gradient of the neural network f deﬁned in (4.2) is bounded as (cid:107)∇ θ f ( θ ; x ) (cid:107) 2 ≤ C 3 √ m with probability at least 1 − L 2 exp( − C 4 mω 2 / 3 L ) .
Revised: d − 1 , the gradient of the neural network f deﬁned in (4.2) is bounded as (cid:107)∇ θ f ( θ ; x ) (cid:107) 2 ≤ C 3 √ m with probability at least 1 − L 2 exp( − C 4 mω 2 / 3 L ) .

Original: The second lemma provides the perturbation bound for the gradient of the neural network function.
Revised: The second lemma provides the perturbation bound for the gradient of the neural network function.

Original: Note that the original theorem holds for any ﬁxed d dimensional unit vector x .
Revised: Note that the original theorem holds for any ﬁxed d dimensional unit vector x .

Original: However, due to the choice of ω and its dependency on m and d , it is easy to modify the results to hold for all x ∈ S
Revised: However, due to the choice of ω and its dependency on m and d , it is easy to modify the results to hold for all x ∈ S

Original: d − 1 .
Revised: d − 1 .

Original: Lemma B.2 (Theorem 5 in Allen-Zhu et al. 
Revised: Lemma B.2 (Theorem 5 in Allen-Zhu et al. (2019b)) .

Original: Let θ ∈ B ( θ 0 , ω ) with the radius satisfying
Revised: Let θ ∈ B ( θ 0 , ω ) with the radius satisfying

Original: [Equation]
Revised: C 1 d 3 / 2 L − 3 / 2 m − 3 / 2 (log m ) − 3 / 2 ≤ ω ≤ C 2 L − 9 / 2 (log m ) − 3 .

Original: Then for all x ∈ S d − 1 , with probability at least 1 − exp( − C 3 mω 2 / 3 L ) over the randomness of θ 0 , it holds that
Revised: Then for all x ∈ S d − 1 , with probability at least 1 − exp( − C 3 mω 2 / 3 L ) over the randomness of θ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to bound the linearization error.
Revised: Now we are ready to bound the linearization error.

Original: Proof of Lemma 6.1.
Revised: Proof of Lemma 6.1.

Original: Recall the deﬁnition of g t ( θ t ) and m t ( θ t ) in (4.5) and (6.2) respectively.
Revised: Recall the deﬁnition of g t ( θ t ) and m t ( θ t ) in (4.5) and (6.2) respectively.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:98) f ( θ ) ∈ F Θ ,m , we have (cid:98) f
Revised: Since (cid:98) f ( θ ) ∈ F Θ ,m , we have (cid:98) f

Original: ( θ
Revised: ( θ

Original: [Equation]
Revised: [Equation]

Original: = f
Revised: = f

Original: [Equation]
Revised: [Equation]

Original: +(cid:104)∇
Revised: +

Original: [Equation]
Revised: [Equation]

Original: f ( θ
Revised: f ( θ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: , θ −θ
Revised: −

Original: [Equation]
Revised: [Equation]

Original: (cid:105) and ∇ θ (cid:98)
Revised: and ∇ θ (cid:98)

Original: [Equation]
Revised: [Equation]

Original:  θ f ( θ 0 ) .
Revised: ∇ θ f ( θ 0 ) .

Original: Then with probability at least 1 − 2 L 2 exp( − C 1 mω 2 / 3 L ) , we have
Revised: Then with probability at least 1 − 2 L 2 exp( − C 1 mω 2 / 3 L ) , we have

Original: [Equation]
Revised: [Equation]

Original: where the inequality comes from Lemmas B.1 and B
Revised: where the inequality comes from Lemmas B.1 and B.2.

Original: [Equation]

Original: By Lemma A.1, with probability at least 1 − δ , it holds that
Revised: By Lemma A.1, with probability at least 1 − δ , it holds that

Original: [Equation]
Revised: [Equation]

Original: which further implies that with probability at least 1 − δ − 2 L 2 exp( − C 1 mω 2 / 3 L ) , we have
Revised: which further implies that with probability at least 1 − δ − 2 L 2 exp( − C 1 mω 2 / 3 L ) , we have

Original: [Equation]
Revised: [Equation]

Original: For the second term in (B.1), we have
Revised: For the second term in (B.1), we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.
Revised: By Lemma A.

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ we have
Revised: with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: for all ( s t , a t ) ∈ S ×
Revised: for all ( s t , a t ) ∈ S ×

Original: A such that (cid:107) φ ( s t , a t ) (cid:107) 2 = 1 .
Revised: A such that (cid:107) φ ( s t , a t ) (cid:107) 2 = 1 .

Original: Substituting the above result into (B.2) and applying the gradient bound in Lemma B.1, we obtain with probability at least 1 − δ −
Revised: Substituting the above result into (B.2) and applying the gradient bound in Lemma B.1, we obtain with probability at least 1 − δ −

Original: L 2 exp( − C 1 mω 2 / 3 L ) that
Revised: L 2 exp( − C 1 mω 2 / 3 L ) that

Original: [Equation]
Revised: [Equation]

Original: Note that the above results require that the choice of ω should satisfy all the constraints in Lemmas
Revised: Note that the above results require that the choice of ω should satisfy all the constraints in Lemmas

Original: B.1, A.1 and B.2, of which the intersection is
Revised: B.1, A.1 and B.2, of which the intersection is

Original: [Equation]
Revised: [Equation]

Original: Therefore, the error of the local linearization of g t ( θ t ) can be upper bounded by
Revised: Therefore, the error of the local linearization of g t ( θ t ) can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: which holds with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point.
Revised: which holds with probability at least 1 − 2 δ − 3 L 2 exp( − C 1 mω 2 / 3 L ) over the randomness of the initial point.

Original: For the upper bound of the norm of g t
Revised: For the upper bound of the norm of g t

Original: [Equation]
Revised: [Equation]

Original: by Lemmas B.1 and A.
Revised: by Lemmas B.1 and A.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: holds with probability at least 1 − δ − L 2 exp( − C 1 mω 2 / 3 L ) .
Revised: holds with probability at least 1 − δ − L 2 exp( − C 1 mω 2 / 3 L ) .

Original: B. P ROOF OF L EMMA 6. 
Revised: B. P ROOF OF L EMMA 6.

Original: Let us deﬁne ζ t ( θ )
Revised: Let us deﬁne ζ t ( θ )

Original: = (cid:104) m t ( θ ) − m ( θ ) , θ − θ ∗ (cid:105) , which characterizes the bias of the data.
Revised: = (cid:104) m t ( θ ) − m ( θ ) , θ − θ ∗ (cid:105) , which characterizes the bias of the data.

Original: Different from the similar quantity ζ t in Bhandari et al.
Revised: Different from the similar quantity ζ t in Bhandari et al.

Original: (2018), our deﬁnition is based on the local linearization of f , which is essential to the analysis in our proof.
Revised: (2018), our deﬁnition is based on the local linearization of f , which is essential to the analysis in our proof.

Original: It is easy to verify that E [ m t ( θ )]
Revised: It is easy to verify that E [ m t ( θ )]

Original: = m ( θ ) for any ﬁxed and deterministic θ .
Revised: = m ( θ ) for any ﬁxed and deterministic θ .

Original: However, it should be noted that E [ m t ( θ t ) | θ t = θ ] (cid:54) = m ( θ ) because θ t depends on all historical states and actions { s t , a t , s t − 1 , a t − 1 , . . . } and m t ( · ) depends on the current observation { s t , a t , s t +1 } and thus also depends on { s t − 1 , a t − 1 , s t − 2 , a t − 2 , . . . } .
Revised: However, it should be noted that E [ m t ( θ t ) | θ t = θ ] (cid:54) = m ( θ ) because θ t depends on all historical states and actions { s t , a t , s t − 1 , a t − 1 , . . . } and m t ( · ) depends on the current observation { s t , a t , s t +1 } and thus also depends on { s t − 1 , a t − 1 , s t − 2 , a t − 2 , . . . } .

Original: Therefore, we need a careful analysis of Markov chains to decouple the dependency between θ t and m t ( · ) .
Revised: Therefore, we need a careful analysis of Markov chains to decouple the dependency between θ t and m t ( · ) .

Original: The following lemma uses data processing inequality to provide an information theoretic control of coupling.
Revised: The following lemma uses data processing inequality to provide an information theoretic control of coupling.

Original: Lemma B.3 (Control of coupling, (Bhandari et al., 2018)) .
Revised: Lemma B.3 (Control of coupling, (Bhandari et al., 2018)) .

Original: Consider two random variables X
Revised: Consider two random variables X

Original: and
Revised: and

Original: Y that form the following Markov chain:
Revised: Y that form the following Markov chain:

Original: [Equation]
Revised: [Equation]

Original: where t ∈ { 0 , 1 , 2 , . . . } and τ > 0 .
Revised: where t ∈ { 0 , 1 , 2 , . . . } and τ > 0 .

Original: Suppose Assumption 5.2 holds.
Revised: Suppose Assumption 5.2 holds.

Original: Let X (cid:48) and Y (cid:48) be independent copies drawn from the marginal distributions of X and Y respectively, i.e., P ( X (cid:48)
Revised: Let X (cid:48) and Y (cid:48) be independent copies drawn from the marginal distributions of X and Y respectively, i.e., P ( X (cid:48)

Original: = · , Y (cid:48) = · ) =
Revised: = · , Y (cid:48) = · ) =

Original: P ( X = · ) ⊗ P ( Y = · ) .
Revised: P ( X = · ) ⊗ P ( Y = · ) .

Original: Then for any bounded function h : S × S
Revised: Then for any bounded function h : S × S

Original: → R , it holds that
Revised: → R , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 6.2.
Revised: Proof of Lemma 6.2.

Original: The proof of this lemma is adapted from Bhandari et al.
Revised: The proof of this lemma is adapted from Bhandari et al.

Original: (2018), where the result was originally proved for linear function approximation of temporal difference learning.
Revised: (2018), where the result was originally proved for linear function approximation of temporal difference learning.

Original: We ﬁrst show that ζ t ( θ ) is Lipschitz.
Revised: We ﬁrst show that ζ t ( θ ) is Lipschitz.

Original: For any θ , θ (cid:48)
Revised: For any θ , θ (cid:48)

Original: ∈ B ( θ 0 , ω ) , we have
Revised: ∈ B ( θ 0 , ω ) , we have

Original: [Equation]
Revised: [Equation]

Original: which directly implies
Revised: which directly implies

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of m t
Revised: By the deﬁnition of m t

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which holds with probability at least 1 − L 2 exp( − C 4 mω 2 / 3 L ) , where we used the fact that the neural network function is Lipschitz with parameter C 3 √ m by Lemma B.1.
Revised: which holds with probability at least 1 − L 2 exp( − C 4 mω 2 / 3 L ) , where we used the fact that the neural network function is Lipschitz with parameter C 3 √ m by Lemma B.1.

Original: Similar bound can also be established for (cid:107) m t ( θ ) − m t ( θ (cid:48) )
Revised: Similar bound can also be established for (cid:107) m t ( θ ) − m t ( θ (cid:48) )

Original: (cid:107) in the same way.
Revised: (cid:107) in the same way.

Original: Note that for θ ∈ B ( θ 0 , ω ) with ω and m satisfying the conditions in Lemma 6.1, we have by the deﬁnition in (6.2) that
Revised: Note that for θ ∈ B ( θ 0 , ω ) with ω and m satisfying the conditions in Lemma 6.1, we have by the deﬁnition in (6.2) that

Original: [Equation]
Revised: [Equation]

Original: The same bound can be established for (cid:107) ¯ m t (cid:107) in a similar way.
Revised: The same bound can be established for (cid:107) ¯ m t (cid:107) in a similar way.

Original: Therefore, we have | ζ t
Revised: Therefore, we have | ζ t

Original: [Equation]
Revised: [Equation]

Original: θ )
Revised: θ )

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: | ≤ (cid:96) m,L (cid:107) θ − θ (cid:48)
Revised: | ≤ (cid:96) m,L (cid:107) θ − θ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: 2 , where (cid:96) m,L is deﬁned as
Revised: 2 , where (cid:96) m,L is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Applying the above inequality recursively, for all τ = 0 , . .
Revised: Applying the above inequality recursively, for all τ = 0 , . .

Original: , t , we have
Revised: , t , we have

Original: [Equation]
Revised: [Equation]

Original: Next, we need to bound ζ t ( θ t − τ ) .
Revised: Next, we need to bound ζ t ( θ t − τ ) .

Original: Deﬁne the observed tuple O t =
Revised: Deﬁne the observed tuple O t =

Original: ( s t , a t , s t +1 ) as the collection of the current state and action and the next state.
Revised: ( s t , a t , s t +1 ) as the collection of the current state and action and the next state.

Original: Note that θ t − τ → s t − τ → s t → O t forms a Markov chain induced by the target policy π .
Revised: Note that θ t − τ → s t − τ → s t → O t forms a Markov chain induced by the target policy π .

Original: Recall that m t ( · ) depends on the observation O t .
Revised: Recall that m t ( · ) depends on the observation O t .

Original: Let’s rewrite m ( θ , O t )
Revised: Let’s rewrite m ( θ , O t )

Original: = m
Revised: = m

Original: t ( θ ) .
Revised: t ( θ ) .

Original: Similarly, we can rewrite ζ t ( θ ) as ζ ( θ , O t ) .
Revised: Similarly, we can rewrite ζ t ( θ ) as ζ ( θ , O t ) .

Original: Let θ (cid:48) t and O (cid:48) t be independently drawn from the marginal distributions of θ t and O t respectively.
Revised: Let θ (cid:48) t and O (cid:48) t be independently drawn from the marginal distributions of θ t and O t respectively.

Original: Applying Lemma B.3 yields
Revised: Applying Lemma B.3 yields

Original: [Equation]
Revised: [Equation]

Original: where we used the uniform mixing result in Assumption 5.2.
Revised: where we used the uniform mixing result in Assumption 5.2.

Original: By deﬁnition θ (cid:48) t − τ and O (cid:48) t are independent, which implies E [ m ( θ (cid:48) t , O (cid:48) t ) | θ (cid:48) t ] = m ( θ (cid:48) t ) and
Revised: By deﬁnition θ (cid:48) t − τ and O (cid:48) t are independent, which implies E [ m ( θ (cid:48) t , O (cid:48) t ) | θ (cid:48) t ] = m ( θ (cid:48) t ) and

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any τ = 0 , . . .
Revised: Therefore, for any τ = 0 , . . .

Original: , t , we have
Revised: , t , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Deﬁne τ ∗ as the mixing time of the Markov chain that satisﬁes 
Revised: Deﬁne τ ∗ as the mixing time of the Markov chain that satisﬁes τ

Original: [Equation]
Revised: [Equation]

Revised: = min { t = 0 , 1 , 2 , . . .

Original:  When t ≤ τ ∗ , we choose τ = t in (B.4) and obtain
Revised: | [Equation] ≤ η T } . When t ≤ τ ∗ , we choose τ = t in (B.4) and obtain

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the initial point θ 0 is independent of { s t , a t , s t − 1 , a t − 1 , . . .
Revised: where we used the fact that the initial point θ 0 is independent of { s t , a t , s t − 1 , a t − 1 , . . .

Original: , s 0 , a 0 } and thus independent of ζ t ( · ) .
Revised: , s 0 , a 0 } and thus independent of ζ t ( · ) .

Original: When t > τ ∗ , we can choose τ = τ ∗ in (B.4) and obtain
Revised: When t > τ ∗ , we can choose τ = τ ∗ in (B.4) and obtain

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) C > 0 is a universal constant, which completes the proof.
Revised: where (cid:101) C > 0 is a universal constant, which completes the proof.

Original: B. P ROOF OF L EMMA 6. 
Revised: B. P ROOF OF L EMMA 6.

Original: Proof of Lemma 6.3.
Revised: Proof of Lemma 6.3.

Original: To simplify the notation, we use E π to denote E µ,π, P , namely, the expectation over s ∈ µ, a ∼ π ( ·| s ) and s (cid:48) ∼ P ( ·| s, a ) , in the rest of the proof.
Revised: To simplify the notation, we use E π to denote E µ,π, P , namely, the expectation over s ∈ µ, a ∼ π ( ·| s ) and s (cid:48) ∼ P ( ·| s, a ) , in the rest of the proof.

Original: By the deﬁnition of m in (6.2), we have
Revised: By the deﬁnition of m in (6.2), we have

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst equation we used the fact that ∇ θ (cid:98)
Revised: where in the ﬁrst equation we used the fact that ∇ θ (cid:98)

Original: f ( θ ) =
Revised: f ( θ ) =

Original: ∇ θ f
Revised: ∇ θ f

Original: ( θ 0 ) for all θ ∈ Θ and (cid:98) f ∈ F Θ ,m .
Revised: ( θ 0 ) for all θ ∈ Θ and (cid:98) f ∈ F Θ ,m .

Original: Further by the property of the local linearization of f at θ
Revised: Further by the property of the local linearization of f at θ

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: where(cid:98) Σ π is deﬁned in Assumption 5.3.
Revised: where Σ π is deﬁned in Assumption 5.3.

Original: For the other term [Equation] we deﬁne b max ( θ ) = argmax b ∈A (cid:98) f
Revised: For the other term, we deﬁne b max ( θ ) = argmax b ∈A (cid:98) f

Original: ( θ ; s (cid:48) , b ) and b max ( θ ∗ ) =
Revised: ( θ ; s (cid:48) , b ) and b max ( θ ∗ ) =

Original: argmax b ∈A (cid:98) f
Revised: argmax b ∈A (cid:98) f

Original: ( θ ∗ ; s (cid:48) , b ) .
Revised: ( θ ∗ ; s (cid:48) , b ) .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: When (cid:104)∇ θ f 
Revised: (B [Equation] For all ( s, a, s (cid:48) ) , when (cid:104)∇ θ f ( θ 0 ; s, a ) , θ − θ ∗ (cid:105) ≥ 0 , (B.6) can be upper bounded by (cid:0) (cid:98) f ( θ ; s (cid:48) , b max ) − (cid:98)

Original:  ( θ 0 ; s, a ) , θ − θ∗ (cid:105) ≥ 0 , (B.5) can be upper bounded by E
Revised: f ( θ ∗ ; s (cid:48) , b ∗ max ) (cid:1) (cid:104)∇ θ f ( θ 0 ; s, a ) , θ − θ

Original: [Equation]
Revised: [Equation]

Original: where the inequality comes from the optimality of b ∗ max and the last equality follows the fact that (cid:98) f ( θ ; · , · ) is linear. 
Revised: where the inequality comes from the optimality of b ∗ max and the last equality follows the fact that (cid:98) f ( θ ; · , · ) is linear. When (cid:104)∇ θ f

Revised: ( θ 0 ; s, a ) , θ − θ ∗ (cid:105) < 0 , using the same argument, we can upper bound (B.6) as follows

Original: Applying Cauchy-Schwarz inequality
Revised: [Equation] Combining the above result [Equation] we have for all tuples ( s, a, s (cid:48) )

Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original:  we have [Equation]
Revised: where we denote 1 + = 1 {(cid:104)∇ θ f

Original: [Equation]
Revised: ( θ 0 ; s, a ) , θ − θ ∗ (cid:105) ≥ 0 } and 1 − = 1 {(cid:104)∇ θ f

Original: where we used the fact that s and s (cid:48) have the same marginal distributions. When (cid:104)∇ θ f ( θ 0 ; s, a ) 
Revised: ( θ 0 ; s, a ) , θ − θ ∗ (cid:105) < 0 } . Taking expectation over the above inequality and applying Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: θ − θ ∗ (cid:105) <
Revised: we have E

Original: [Equation]
Revised: [Equation]

Original: using the same argument
Revised: where we used the fact that Σ ∗ π ( θ − θ ∗ ) = 1 /m E µ [ ∇ θ f

Original: [Equation]
Revised: [Equation]

Original: we can upper bound (B
Revised: θ 0 ; s,

Original: [Equation]
Revised: [Equation]

Original: 5) as follows
Revised: b max ) ∇ θ f

Revised: ( θ 0 ; s, (cid:101) b max ) (cid:62) ] and (cid:101) b max = argmax

Original: [Equation]
Revised: [Equation]

Original: Combining the above results
Revised: |

Original: [Equation]
Revised: [Equation]

Revised: θ f

Original:  we obtain
Revised: ( θ 0 ; s, b ) , θ − θ ∗ (cid:105)| according to (5.6). Substituting the above results into (B.6) [Equation] we obtain

Original: [Equation]
Revised: [Equation]

Original: which immediately implies
Revised: which immediately implies

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Assumption 5.3.
Revised: where the second inequality is due to Assumption 5.3 and the last equation is due to (B.5) and the deﬁnition of Σ π in (5.5).