Finite-Time Analysis of Adaptive Temporal Difference Learning with Deep Neural Networks ∗
Abstract
Temporal difference (TD) learning with function approximations (linear functions or neural networks) has achieved remarkable empirical success, giving impetus to the development of ﬁnite-time analysis.
As an accelerated version of TD, the adaptive TD has been proposed and proved to enjoy ﬁnite-time convergence under the linear function approximation.
Existing numerical results have demonstrated the superiority of adaptive algorithms to vanilla ones.
Nevertheless, the performance guarantee of adaptive TD with neural network approximation remains widely unknown.
This paper establishes the ﬁnite-time analysis for the adaptive TD with multi-layer ReLU networks approximation whose samples are generated from a Markov decision process.
Our established theory shows that if the width of the deep neural network is large enough, the adaptive TD using neural network approximation can ﬁnd the (optimal) value function with high probabilities under the same iteration complexity as TD in general cases.
Furthermore, we show that the adaptive TD using neural network approximation, with the same width and searching area, can achieve theoretical acceleration when the stochastic semigradients decay fast.
Introduction
Temporal difference (TD) learning is a popular and successful iterative algorithm in the area of reinforcement learning (RL) to evaluate a given policy [47, 51], often employed for critic part evaluation in various RL algorithms [28, 41, 42].
Classical TD algorithm adopts the tabular representation for the value function, which stores value estimates on a per-state basis.
In large-scale scenarios, the tabular approach becomes intractable due to a large number of states.
Thus, the function approximation techniques have been integrated with TD for better scalability and efﬁciency [4, 50, 36, 57].
The function approximation techniques have achieved remarkable empirical success and are theoretically justiﬁable when the linear function approximation is used [23].
As a special function approximation approach, deep neural networks (DNNs) have also been integrated with TD [38, 37, 33], achieving phenomenal performance in several applications.
However, from the theoretical perspective, establishing theoretical convergence guarantees for training DNNs is much more complicated than that for the linear approximation algorithms, which is still widely open.
Some convergence results of TD with
DNN approximations have been proved by the authors of [21, 7, 53], under some extra assumptions and restrictions.
The adaptive methods for DQN have been proposed in [37, 22], inspired by the adaptive stochastic algorithms to accelerate TD.
The numerical results show that the adaptive versions of TD can outperform the vanilla ones in many tasks.
Mathematically, the (projected) TD (with function approximation) can be described as
[Equation]
where V is the constrained set, and Proj V denotes the projection onto V , θ k is the iterate, and g k is the stochastic semi-gradient 2 .
Although the update scheme of TD looks similar to the stochastic gradient descent (SGD), it is much more complicated due to the Markov noise, even in the linear function approximation case.
Motivated by the adaptive SGD [20, 27], the adaptive TD with linear function approximation is proposed in [52, 46].
The main difference between the adaptive TD and the standard TD lies in the use of adaptive stepsize and momentum.
This paper considers the neural adaptive TD.
In the k th iteration of neural adaptive TD, it performs
[Equation]
where β > 0 is the momentum parameter, and η > 0 .
The points m k and v k contain past information.
The numerical results have demonstrated the advantage of adaptive TD over vanilla TD.
The adaptive
TD is proved to be convergent with linear function approximation, but the convergence remains unclear when the neural network approximation is used.
This paper proves that the adaptive TD with neural network approximation converges when the width of a ReLU network is sufﬁciently large.
Moreover, we prove that adaptive TD is faster than TD with the ReLU DNN approximation
1.1 Related Works
Analysis and the recent development of TD.
Leveraging the stochastic approximation techniques, the authors in [25] establish the ﬁrst convergence results for TD.
The limiting convergence of TD with linear function approximation is proved in [50] with the perspective of the dynamics.
Since the seminal work of [50], many works have been using the ODE-based method to study the asymptotic convergence of TD since TD update does not follow the (stochastic) gradient direction of any objective function [6, 49].
Some variants of TD have been proposed in [14] with asymptotic convergence guarantees.
The ﬁrst non-asymptotic analysis for the gradient TD, a variant of the TD, has been studied in [34].
Finite-time analysis of TD with independent and identically distributed (i.i.d.) observation assumption has been presented in [13].
The Markov sampling convergence analysis is proved in a subsequent paper [5].
In a concurrent line of research, TD has been studied from the perspective of stochastic linear systems [29].
The ﬁnite-time analysis for Markov sampling stochastic linear system has been developed by the authors of [43, 24].
The ﬁnite-time analysis of multi-agent
TD is proved in [15].
A uniﬁed analysis for a class of TD learning algorithms with Markov jump is established in [24].
Based on Nesterov’s acceleration method, a class of accelerated TD is developed and analyzed in [16].
From the algorithmic viewpoint, the adaptive TD has been recently proposed in [46, 52] to accelerate TD, inspired by the adaptive SGD.
In [44], the authors present the ﬁnite-time convergence results of decentralized TD with linear approximations.
TD with deep learning.
In contrast to the tremendous empirical success of the deep Q-networks (DQNs), the theory is still relatively weak; until recently, only a handful of papers have studied the theoretical results of TD using neural network approximation.
In [21], the authors prove the convergence rates of ﬁtting Q-iteration with a sparse multi-layer ReLU network under i.i.d.
observations.
The convergence of TD with two-layer neural network approximation is provided by [7] with i.i.d assumption on the samples.
The TD-based algorithm with multi-layer ReLU networks under Markov samples is studied in [53].
The theory of the TD with a multi-layer ReLU network relies heavily on the existing results about overparameterized deep networks [26, 12, 19, 2, 1, 3, 56].
1.2 Difference Between our Work and Existing Works, and Technical Challenges
Existing related works contain two categories: adaptive TD with linear approximations and neural TD.
However, our work is signiﬁcantly different from these related works.
In contrast to adaptive
TD with linear approximations, we consider the neural network approximation, in which case we do not have nice properties that linear approximation enjoys, and we have to consider the neural tangent kernel (NTK) regime and develop a new analysis leveraging the semi-Lipschitz continuity property. 2)
Compared to neural TD, we use the adaptive stepsize and momentum, which has never been considered in neural TD.
1.3 Our Contributions
In this paper, we consider the adaptive TD with a multi-layer ReLU network approximation under the Markov observations.
In contrast to the existing works [21, 7, 53], we study the adaptive variant of TD.
The scheme of the algorithm is much more complicated, raising tremendous challenges in theoretical analysis.
Our main theoretical contributions are summarized below:
• We extend the analysis of adaptive TD with linear function approximation [46, 52] to multi-layer neural network approximation under Markovian samplings.
The theoretical results show that adaptive TD still works for the neural network approximation, even with deep neural networks.
• We establish the ﬁnite-time analyses of adaptive TD with multi-layer ReLU network approximation under Markov observations.
In particular, we show that the adaptive algorithms guarantee convergence when the neural network is sufﬁciently wide, and adaptive TD with neural network approximation converges to a projected optimal action-value function.
The technique required to connect Adam-type algorithms and neural TD is non-trivial since they belong to two very different research areas.
To this end, we develop a new technique that uses expectation with a ﬁxed delay, which is different from the coupling technique used by the authors of [5, 53].
• We prove that the speed of the adaptive TD can be faster than the vanilla ones with multi-layer ReLU network approximation.
Speciﬁcally, we show that the adaptive ones use fewer iterations to reach the same desired error with the same network widths and searching areas.
Our theoretical results ﬁrst explained why Adam-type algorithms perform better than TD in DQN, which has been observed in practice.
Preliminaries
We introduce the notation, some basic concepts, and properties of TD in this section.
Notation: We use E [ · ] to denote the expectation with respect to the underlying probability space without stochasticity of the initial point, and we use (cid:107) · (cid:107) to denote the Frobenius norm.
σ ( · ) denotes the ReLU activation function.
We use φ : S ×
A → R d to denote the feature map.
Given a closed set
V , Proj V
( x ) represents the projection of the vector x onto V .
The initial point is deﬁned as θ init .
B ( θ , ω ) denotes the ball centred at θ with radius ω .
We use a k = ˜ O ( b k ) to hide the logarithmic factor of b k still with the same order.
We write a k = Θ( b k )
if a k = O ( b k ) and b k = O ( a k ) , and we use a k = ˜Θ( b k ) to hide the logarithmic factor.
We denote χ k as the sub-algebra that generated by θ 0 , θ 1 , . . .
, θ k , where θ k is the value in the k th iteration.
2.1 Markov Decision Process
For the sake of presentation, we consider the ﬁnite state space 3 .
Consider a Markov decision process (MDP) described as a tuple ( S , A , P , R , γ ), where S denotes the state space, A denotes the action space, P a represents the transition matrix associated with action a , and 0 < γ < 1 is the discount factor.
In this case, let P a ( s (cid:48) | s ) denote the transition probability from state s to state s (cid:48) under the action a .
The corresponding transition reward is r ( s, a ) .
We consider the stochastic policy π : S → ∆( A ) that speciﬁes a probability density of all actions given the current state s .
π ( s, a ) denotes the probability to choose action a when the current state is s , and (cid:80) a ∈A π ( s, a )
= 1 .
We consider the on policy setting, where both target and behavior policies are π .
The corresponding action-value function Q π ( s, a ) : S ×
A → R is deﬁned as
[Equation]
and the associated value function V π : S → R is deﬁned as
[Equation]
It is evident that the restriction on discount 0 < γ < 1 can guarantee the boundedness of Q π ( s, a ) .
The Markovian property of MDP yields the following celebrated Bellman equation
[Equation]
where the Bellman operator T π on a value function Q π can be represented as
[Equation]
where s ∈ S , a ∈ A .
Directly solving (3) needs O ( |S| 3 |A| 3 ) computational cost, which is very expensive since |S||A| is usually very large.
Thus, people turn to use a linear or non-linear approximation as Q π ≈ Φ( θ ) : R D
→ R |S||A| , where θ ∈ R D and D (cid:28) |S||A| .
In this way, the dimension can be signiﬁcantly reduced, and we can get an approximate solution very efﬁciently.
We collect necessary assumptions for MDP below.
Assumption 1 The transition rewards are uniformly bounded by 1, that is, | r ( s, a ) | ≤ 1 , s ∈ S , a ∈
A .
For any s ∈ A , it holds µ ( s ) =
lim t →∞ P ( s t = s | s
= s (cid:48) , a 0 = a (cid:48) ) > 0 .
There exist constants 0 ≤ ρ < 1 and ¯ κ > 0 such that
[Equation]
The boundedness of ( r ( s, s (cid:48) )) s,s (cid:48) ∈S comes from the ﬁniteness of S .
In Assumption 1, the uniform boundedness assumption can be replaced by non-uniform boundedness in the ﬁnite state case.
In this paper, it is used for simplicity.
The rest part of Assumption 1 is standard for the Markovian property.
It is well-known that irreducible and aperiodic Markov chains can always follow Assumption[30].
For Assumption 1, the time that ( s t , a t ) t ≥ 0 needed for its current state distribution to match the stationary one with (cid:15) error in total variation distance is O (log 1 (cid:15) / log 1 ρ ) .
Thus, the constant ρ represents the speed of the process accessing to the stationary distribution.
In ﬁnite-time cases, it is easy to prove that ρ is the second largest eigenvalue of P .
We can see that the smaller ρ is, the faster the process will converge to the stationary states.
2.2 Neural Temporal Difference Learning
Although we can get Q π by solving the Bellman equation induced by the given policy π , in practice
S may contain a very large number of different states and actions, and it is hard to solve the Bellman equation directly.
Thus, alternative methods have been proposed, including leveraging the linear [48] or non-linear approximations (e.g., kernel methods and neural networks [38]).
This paper is devoted to the study of the approximation using a L -hidden-layer ReLU neural network deﬁned as
[Equation]
where x ∈ R
d is the input data, W 1 ∈ R m × d , W L ∈ R 1 × m and W l ∈ R m × m for l = 2 , . . .
, L − 1 , and θ :=
[ W 1 , . . .
, W L ] denotes all the weights.
In neural TD, we use the following approximation
[Equation]
where θ ∈ R ( L − 2) m 2 + md + m is the parameter vector, and m is usually set such that ( L − 2) m {+
+} + md + m is smaller than |S||A| to reduce the difﬁculties caused by the high dimensionality.
It is worth mentioning that factor √ m is multiplied to guarantee the output to be meaningful: in [Lemma 4.4, [8]], it is proved that f ( θ ; x ) =
˜ O (1) as m is large and θ is randomly initialized.
If we remove √ m , the function value is then of the order ˜ O
(1 / √ m ) , which tends to 0 as m is large.
We stress that such a use is standard in ReLU network theory.
Assumption 2 For any state-action pair ( s, a ) ∈ S × A , we assume the feature vector is uniformly bounded such that (cid:107) φ ( s, a ) (cid:107) = 1 .
A simple normalization can make Assumption 2 hold.
This assumption is used to simplify coefﬁcients in the subsequent proofs.
Next, we present the scheme of neural temporal difference learning.
With ( s k , a k ) k ≥ 0 being a trajectory sampled from π and a hyper-parameter η > 0 , the neural TD updates with g k ← g ( θ k ; s k , a k , s k
+1 , a k +1 ) in (1), where the stochastic semi-gradient is deﬁned as [Equation] + [Equation]
a
[Equation]
+
[Equation]
[Equation]
s
[Equation]
a k ) × [ f [Equation] The projection in scheme (2) is used to ensure the boundedness of θ k and simplify the convergence analysis.
In the deep neural network approximation, the searching area V is chosen as a ball around the initialization θ init =
[ W init 1 , . . .
, W init L ] for the ease of analysis.
The detailed expression of the set
V is given as follows:
[Equation]
where 1 ≤ l ≤ L .
It is assumed that θ init is chosen randomly from the Gaussian distribution, i.e.,
W init l is drawn from N (0 , 1 /m ) with l = 1 , . . .
, m .
The collection of all local linearization of f ( θ ; φ ( s, a )) at the initial point θ init is deﬁned as
[Equation]
If the function f is linear, i.e.,
[Equation]
we can see that F V ,m = {(cid:104) θ , φ ( s, a ) (cid:105) : θ ∈ V } .
The approximate stationary point of neural TD associated with F V ,m is deﬁned as follows.
Deﬁnition 1 ([7])
A point θ ∗ ∈ V is said to be the approximate stationary point if
[Equation]
where
[Equation]
, and the temporal difference error ˆ∆ is deﬁned as
[Equation]
and ˆ f ( θ ; φ ( s, a )) ∈ F V ,m .
The convergence of neural TD is dependent on θ ∗ , and so is our result.
In [7], it has been proved that such a deﬁnition is well-deﬁned; the approximate stationary point exists, minimizing the mean squared projected Bellman error (MSPBE).
Such a fact is more straightforward if the function f is linear: the approximate stationary point of TD is identical to the unique solution to the projected
Bellman equation [50].
Require:
Parameters : η, γ , β , L , m , ω
Initialization : g 0 = 0 , m
= 0 , v 0 = 0 , W init l ∼ N (0 , 1 /m ) with l = 1 , . . .
, m , V is set as (5) k
= 1 , 2 , . . .
2.3 Neural Adaptive Temporal Difference Learning
This paper considers the neural adaptive TD.
In the k th iteration of neural adaptive TD, we sample g k = g
( θ k ; s k , a k , s k
+1 , a k +1 ) in (2) with stochastic semi-gradient (4).
The set V is chosen as (5) with θ init also being sampled from the Gaussian distribution.
2.4 Some Useful Properties
For neural adaptive TD, several bounds and properties are presented in the following lemma (c.f.
Lemma 6.1,[53]; Lemma 4.4, [8])
Lemma 1 Assume that ( θ k ) 1 ≤ k ≤ K ∈ B ( θ ∗ , ω ) , given any 0 < δ < 1 and K ∈ Z + , m , ω and L satisfying
[Equation]
then it holds
[Equation]
and
[Equation]
Denote that h k := (cid:98) ∆( [Equation] ∇
[Equation] θ k ) ( (cid:98) ∆( θ k ) is given by (8)) , it follows
[Equation]
where g k is the stochastic semi-gradient in neural adaptive TD, and
[Equation]
holds with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point, where { C i > 0 } i =1 ,..., 7 and C > 0 are universal constants.
Lemma 1 has been proved by [Lemma 6.1, [53]] and [Lemma 4.4, [8]].
The bounds (10) and (11) are only dependent on the structure of the neural networks.
Our theory is built on Lemma 1.
Without the ReLU activation, Lemma 1 cannot be guaranteed to hold, and thus we only consider deep ReLU networks in this paper.
Indeed, due to the property of deep ReLU networks, there has been a line of theoretical research on deep ReLU networks, see e.g. [54, 18, 40, 9, 55, 53].
From Lemma 1, with high probabilities, v k ≤ (2+ γ ) 2 C 27 m log( K/δ ) k when k ≤ K .
But this is the worst-case bound and may not always be achievable.
From (4), we can see that g k keeps the sparsity of ∇ θ f
( θ k ; φ ( s k , a k ) because
[Equation]
Thus, the stochastic semi-gradient g k can be very sparse, based on which the following bound is proposed.
Condition 1 In neural adaptive temporal difference learning, the following bound holds
[Equation]
upper bound of (cid:80)
Kk =1 E ( (cid:107) m k
(cid:107) 2 /v k ) lower bound of (cid:80)
Kk =1 E ( (cid:104) m k , θ ∗ − θ k (cid:105) / ( v k ) 12 )
Lemma Lemma
Lemma
We can see that α = 1 directly holds for (14) due to the bound (13); while 0 < α < 1 indicates the stochastic semi-gradients decay fast.
In applications, the sparse gradients can always obey such a condition.
We stress that such assumption is standard in the analysis of the Adam-type algorithms; see, e.g., references [32, 20, 39, 11, 35, 45].
Now, we turn to the neural approximation case.
As mentioned above, the convergence analysis uses the approximate stationary point θ ∗ .
One core part of our proofs is investigating h versus the iteration.
Due to the Markovian observation, s k , a k , s k +1 , a k +1 may miss visiting several states in a single sampling, i.e., choosing some states and actions with probability 0.
That indicates E [ h k ] may be a biased expectation of h .
To get controlled difference bound, we consider E ( h ( θ k − T ) | χ k ) and h ( θ k − T ) , where
[Equation]
Although with biased samples, the Markov property means P ( s k
= s
| s k − T = s (cid:48)
, a k − T = a [Equation] ) is sufﬁciently close to µ ( s ) when T is large.
Such a technique then yields following lemma.
Lemma 2 Assume ( θ k ) 0 ≤ k ≤ K is generated by neural adaptive TD, and condition (9) holds.
Given an integer 1 ≤ T ≤ K , as k ≥ T , we have
[Equation]
with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point.
The factor ρ T in the upper bound in Lemma 2 comes from the Markovian noise.
The lemma tells that when T is large enough, the difference between E h ( θ k − T ) and h ( θ k − T ) will be very small.
The lower bound directly holds based on the deﬁnitions above.
Lemma
For ∀ θ ∈ V , it follows that
[Equation]
Main Results
This section contains the ﬁnite-time analysis of adaptive TD using deep neural network approximations.
The descriptions of convergence results are dependent on the notion of the approximate stationary point θ ∗ .
3.1 Roadmap
We ﬁrst provide a high level explanation of the proofs and how the technical lemmas work in the proofs.
Our proofs begin with a traditional idea of algorithmic convergence analysis, i.e., bounding the gap ( (cid:107) θ ∗ − θ k (cid:107) 2 ) k ≥ 0 versus the iteration.
It is easy to have the following estimate
[Equation]
Summing this inequality from k = 1 to K and taking expectation, we are led to
[Equation]
Notice that (16) is very similar to the recursion of the well-known SGD.
Recalling the ﬁnite-time convergence analysis of SGD, we then get the big picture for the proof of our algorithms, which contains two major steps and can be presented by Figure 1.
• Step 1 .
The ﬁrst step is quite straightforward, i
e., bounding the summation (cid:80)
Kk =1 E ( (cid:107) m k
(cid:107) 2 /v k ) with high probability.
This step is proved by Lemma 4 in the supplementary materials.
• Step 2 .
The second step is to establish a lower bound of (cid:80)
Kk =1 E ( (cid:104) m k , θ ∗ − θ k (cid:105) / ( v k ) 12 ) .
However, it is much more complicated than the ﬁrst step.
To this end, we need to exploit the relation between E (cid:104) m k , θ ∗ − θ k (cid:105) / ( v k ) 12 and E (cid:104) h ( θ k ) , θ ∗ − θ k (cid:105) / ( v k − 1 ) 12 (proved by Lemma 6 in the supplementary materials).
Nevertheless, we cannot directly get Lemma 6; we have to develop a technical lemma that connects E (cid:104) θ k − θ ∗ , h ( θ k ) (cid:105) /
( v k − 1 ) 12 with E (cid:104) θ ∗ − θ k , g k (cid:105) / ( v k − 1 ) 12 (proved by Lemmain the supplementary materials).
Further with the connection between E (cid:104) h ( θ k ) , θ k − θ ∗ (cid:105) and
E (cid:0) ˆ f ( θ k ; s, a ) − ˆ f
( θ ∗ ; s, a ) (cid:1)
(Lemma 3), we then prove the lower bound.
3.2 Finite-Time Analysis of Neural Adaptive TD
Now, we are prepared to present the convergence of neural adaptive TD.
Theorem 1 Suppose ( θ k ) k ≥ 0 is generated by neural adaptive TD under the Markovian observation, and condition (9) holds, and Assumptions 1, 2, and condition (14) hold.
Given the integer T ∈ Z + , η > 0 , v 0 ≥ (cid:36) > 0 , 0 ≤ β < 1 , for K ≥ 2 2 2 − α T , we have
[Equation]
with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point, where
[Equation]
and their details are given by (29) in supplementary materials.
If we set the radius as ω = Θ( m − 1 / 2 ) 4 , and T = ln( mK ) ln 1 ρ , and η = m − 1 / 2 , with (18), the right-hand side of (17) is in the order of
[Equation]
Thus, with high probability to achieve the (cid:15) -accuracy for min 1 ≤ k ≤ K E π (cid:0) ˆ f [Equation] θ k
[Equation] s, a ) [Equation] f
[Equation] θ ∗ ; s, a ) (cid:1) 2 , we need that
[Equation]
Thus, we get the worst-case iteration complexity of K as follows
[Equation]
The worse case is α = 1 , in which case we get the complexity as (cid:101) O
(1 /(cid:15) 2 ) .
Note that even for SGD without strong convexity, the optimal complexity is O (1 /(cid:15) 2 )
[Theorem 4, [17]].
Thus, our result is nearly optimal (just with an additional logarithmic factor that barely hurting the rate).
We can see that when 0 ≤ α < 1 , the worst iteration complexity of K is smaller than (cid:101) O (cid:16) 1 (cid:15) 2 ln 2 ( 1 ρ ) (cid:17) when (cid:15) (cid:28) 1 /
[ln( 1 ρ )] , which indicates a faster speed than the neural TD .
Due to that (cid:15) is the desired error and thus can be very small, the acceleration always happens for a small α .
Note that α = 1 directly holds with any extra assumption.
In this case, the iteration complexity of the adaptive neural TD matches existing neural TD [53] in the case of DNN approximation.
It is worth mentioning that compared with the convergence rate of neural TD with ReLU deep networks, we used the same searching radius and network width as presented by the authors of [53].
In applications, the sparse stochastic semi-gradients always yield the fast decaying condition.
In other words, with sparse stochastic semi-gradients, the neural adaptive TD also accelerates the vanilla
TD even in the ReLU network approximation case.
Such a phenomenon resonates with the existing acceleration results of adaptive methods for stochastic optimization: “for sparse data, the adaptive methods are likely to perform better than non-adaptive methods” [20].
The scheme of neural TD uses a projection to the set V such that all iterates are constrained in a small neighborhood around the initialization (also called NTK regime).
Such a procedure is to use the property of deep ReLU networks [7, 53].
Thus, the searching radius is small, which is reasonable because in DNN training, the parameters usually only change in a very small range in the overparameterized regime.
Proposition 1 Assume that conditions of Theorem 1 hold, and Condition 1 holds with 0 ≤ α < 1 , ω =
Θ( m − 1 / 2 ) , η = m − 12 , m = ˜Θ(1 /(cid:15) 6 ) , (cid:15) (cid:28) 1 /
[ln( 1 ρ )] , the neural adaptive TD enjoys a faster ﬁnite-time convergence rate than the neural TD in the ReLU DNN approximation of function values with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point.
In the following, we present a proposition that characterizes the difference between our established results and the optimal action-value function.
Proposition 2 Assume conditions of Proposition 1 hold, with probability at least 1 − 2 δ − 3 L 2 exp( − C 6 mω 2 / 3 L ) over the randomness of the initial point, we have [Equation]
[-−
-]where Π F V ,m ( Q ∗ ( s, a )) is projection of Q ∗ ( s, a ) to the linear function family F V ,m , that is, Π F V ,m ( Q ∗ ( s, a )) = f ( θ init ; φ ( s, a )) + (cid:104)∇ θ f ( θ init ; φ ( s, a )) , θ † − θ init (cid:105) with θ † ∈ arg min θ ∈ V (cid:110)(cid:13) (cid:13)(cid:13)
f ( θ init ; φ ( s, a )) + (cid:104)∇ [Equation] ( θ init ; φ (
[Equation]
, a [Equation] [Equation] Proposition 2 indicates that the algorithm can ﬁnd the optimal action-value function Q ∗ ( s, a ) provided that the function family F V ,m contains Q ∗ ( s, a ) .
Conclusions
This paper studies the ﬁnite-time convergence analyses of temporal difference learning with adaptive learning rates and momentum approximated by deep ReLU neural networks using the Markovian samplings.
Our established theoretical results show that the neural adaptive temporal difference learning is convergent when the neural network is sufﬁciently wide.
Our work shows convergence results and establishes the theoretical advantages of the adaptive algorithms for neural network approximation cases, i.e., adaptive schemes achieve better rates than neural temporal difference learning when the stochastic semi-gradients decay fast.
There are numerous avenues for future works, including 1) Can we extend the multiple ReLU active functions to others, e.g., sigmoid or more general functions? 2)
Can we establish the ﬁnite-time convergence of adaptive temporal difference learning with other kinds of neural network approximation, e.g., recurrent neural networks and graph neural networks?
3) Can we get a relaxed bound for the radius of the searching area? Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See future directions presented in Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Supplementary Materials 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]