Understanding Deep Neural Function Approximation in Reinforcement Learning with ϵ -Greedy
Abstract
This paper provides a theoretical study of deep neural function approximation inreinforcement learning (RL) employed with the ϵ -greedy exploration under theonline setting. This problem setting is motivated by the successful deep Q-networks(DQN) framework that works in this regime.
We provide an initial attempt ontheoretical understanding deep RL from the perspective of function class and neuralnetworks architectures (e.g., width and depth).
In this work, we focus on valuebased algorithm with the ϵ -greedy exploration under Besov and Barron functionspaces, which aims at approximating an α -smooth Q-function in a d -dimensionalfeature space with T episodes.
We prove that scaling the width m = (cid:101) O
( T d 2 α + d )and the depth L = (cid:101) O (1) of the neural network for deep RL is sufficient for learningwith sublinear regret in Besov spaces.
Moreover, for a two layer neural networkin the Barron space, scaling the width even in sublinearly in T is sufficient.
Inparticular, for low smooth Q-function, the theory suggests that the depth shouldbe chosen as the logarithm of the width.
This matches the practical DQN setting.
Our analysis relies on transforming the estimation of temporal difference error toa generalization problem under the conditionally independent but non-identicallydistributed data setting over the averaged empirical measure.
This might have itsown interest in reinforcement learning theory for better understanding ϵ -greedyexploration in deep RL.
Introduction
Efficient reinforcement learning (RL) under the large (or even infinite) state space and action spacesetting is increasingly important and relevant challenge [1, 2, 3].
One of the first successful approachestowards this problem is the deep Q-network (DQN) [4, 5] framework, which deploys powerfulnonlinear function approximation techniques via Deep Neural Networks (DNNs) [6] to conciselyapproximate state and action spaces.
Despite its impressive practical success there is still a gap between practical uses and theoreticalunderstanding on DQN that is equipped with the ϵ -greedy policy under the online setting.
In contrast,many works directly focus on linear function approximation [7, 8], linear mixtures [9, 10] withconfidence-bound based scheme for exploration.
In fact, the ϵ -greedy policy is a popular choice indeep RL even though it has exponential sample complexity in the worst case [11].
Existing non-linear function approximation results on RL are largely based on neural tangent kernel(NTK)
[12, 13], Bellman rank [14, 15], and Eluder dimension [16, 17, 18].
Nevertheless, theseapproaches fail in truly capturing the highly non-linear properties of DQN.
Indeed, NTK (or lazy training [19]) essentially works in a linear regime [20, 21, 22], and can not efficiently learn even asingle ReLU neuron [23, 24, 25] as it requires Ω( ε − d ) samples to achieve ε approximation error,where d is the (original) or transformed feature dimension input.
Interestingly, even for two-layer neural networks, the corresponding Eluder dimension is at least inan exponential order [26, 27], which appears difficult to fully demonstrate the success of practicaldeep RL both theoretically and empirically.
Regarding to ensuring a sublinear regret, the width ofneural networks under the NTK regime [13] is required to be m = Ω( T 13 ) , where T is the numberof episodes, which does not match the used DQN in practice with small width/depth under largeepisodes [4, 28].
To bridge the large theory-practice gap, we study value iteration based on deep neural functionapproximation with the ϵ -greedy policy under the online setting, which broadly capture the keyfeatures of deep RL.
Our analysis framework is equipped with two-layer as well as deep fullyconnected neural networks in the respective Barron and Besov spaces, which can fully capture theproperties of Q-functions, e.g., smoothness by neural networks.
Besides, our results allow for reasonably finite width and depth in practice, give precise characterization of these parameters to obtain sublinear regret, and demonstrate the relationship between theproblem-dependent smoothness of Q-function and regret bounds.
These results could also motivatepractitioners to consider different architectures of implementations of deep RL.
1.1 Technical challenges and contributions
Most previous RL theory results on function approximation in the online setting work with “optimismin the face of uncertainty” principle for exploration, leading to a series of upper confidence bound(UCB)-type algorithms.
However, conceptually, optimism is sometimes too aggressive and UCB-stylealgorithms can suffer exponential sample complexity even for nonlinear bandits [27].
Empiricalmodel-based RL has also not found optimism to be very useful [29].
In the RL setting, linear/kernel function approximation [7, 13, 30] depends on a known featuremapping or the NTK matrix to design a UCB-type bonus function for optimism, which appearsinvalid for DNNs as the used confidence ellipsoid and elliptical potential lemma are not applicable fordata-dependent feature mapping.
To avoid explicitly designing a bonus function, Thompson sampling[31, 32] appears promising in a Bayesian perspective by using randomized (i.e., perturbed) versionsof the estimated model or value function [33].
Nevertheless, the bonus function is still implicitlyincluded in confidence estimate of perturbations in their UCB-style algorithm.
This work analyzes a value iteration algorithm with ϵ -greedy by neural networks.
Designing a provablyefficient exploration mechanism is beyond the scope of our work but it remains an interesting futuredirection.
The technical contribution in our analysis is how to control the temporal difference (TD) error withoutbonus function design.
In this case, the ϵ -greedy exploration is not enough to ensure the TD errornegative as conducted in UCB-style algorithms.
By a measure transform, we analyze the TD error in an L 2 -integrable space wrt historical state-actionpairs, introduce the δ -covering of the value function set to break the dependence between the episodes 1 ,and transform to generalization bounds under the independent but non-identically distributed datasetting.
To this end, we need to employ proper function spaces beyond linear/kernel Markov decisionprocesses (MDP) in our analysis: one is the Barron space [34], as the largest 2 function space fortwo-layer ReLU neural networks without curse of dimensionality [35]; the other is the Besov space[36, 37] for deep ReLU neural networks, which allows for continuous/non-continuous functions andthus is more general than Hölder spaces used in offline RL [38].
Under these proper hypothesis spaces, such generalization problem can be addressed by uniform convergence via Rademachercomplexity of these spaces under the averaged empirical measure.
Under attainable assumptions, our results show that ( i ) the problem-dependent smoothness of Qfunction affects the deep Q-learning efficiency, which can be improved by the model complexity.
Weuse α as a parameter indicating the smoothness degree of Q-function.
A larger α indicates smootherfunctions.
( ii ) for two-layer neural networks, the width m = Ω( √ T ) suffices to ensure sublinearregret.
( iii ) for deep neural networks, the width m = (cid:101) O
( T d 2 α + d ) and the depth L = (cid:101) O (1) are enoughfor sublinear regret, where (cid:101) O ( · ) omits the log terms.
Furthermore, our regret bounds can be (nearly)independent of the feature dimension d , supporting the premise of practical, high-dimensional deep RL.
1.2 Related work
Recent work on neural network function approximation beyond NTK (or the Eluder dimension)mainly restrict on the generative setting [39, 40] by assuming a simulator in which the agent canrequire any state and action, and the offline setting [38, 41].
In sequel, we review RL with functionapproximation under the online setting that DQN falls into this regime.
We also mention that,theoretical understanding of DQN can be conducted by Q learning [42], neural fitted Q-iterationalgorithm [43, 38, 44], and analyses of the target network [45].
RL with linear/kernel function approximation: RL with linear function approximation results inregret bounds (cid:101) O ( √ d 3 H 3 T ) [7] and (cid:101)
O ( dH 2 √ T ) [30], where H is the length of each episode; andlinear mixtures with regret (cid:101) O
( d √ H 3 T ) [9] and improved to (cid:101) O ( dH √ T ) [10]; kernel regime withregret (cid:101) O
( δ F H 2 √ T ) [30, 13], where δ F is the intrinsic complexity (e.g., effective dimension) ofthe function class RKHS F .
The above bounds are based on confidence ellipsoid to quantify theuncertainty in an explicit bonus function by feature mapping/kernel matrix; while Thompson sampling[32, 31] utilizes an implicit bonus function in probability estimation on uncertainty quantification,which leads to (cid:101)
O ( d 2 H 2 √ T )
[33] regret in linear function approximation.
RL with general function approximation: One prototypical scheme uses the Eluder dimension [16],which measures the degree of dependence among action rewards, resulting in (cid:101)
O ( poly ( δ F H ) √ T )regret [17, 18], where the complexity δ F depends on the Eluder dimension.
The sublinear regretunder ϵ -greedy exploration can be achieved by [46] using Eluder dimension.
Bellman rank [14] isalso another metric for general function approximation [15] by measuring the error of the functionclass under the Bellman operator.
Jin et al.
[47] introduce a new metric, Bellman Eluder dimension,by combining these two metrics, and achieve (cid:101)
O ( H √ δ F T ) , where δ F depends on this metric.
Overall, none of the above metrics can handle the nonlinear spaces that concern us.
Background and preliminaries
In this section, we provide some preliminaries with respect to online reinforcement learning based onepisodic MDPs, function spaces of two (and deep)-layer ReLU neural networks.
Notation: We denote by a ( n ) ≲ b ( n ) : there exists a positive constant c independent of n such thata ( n ) ⩽ cb ( n ) .
We use the shorthand [ n ] := { 1 , 2 , . . . , n } for some positive n .
Let X =
[0 , 1] dbe a domain of the functions, we denote the L p -integrable space by L p ( X ) endowed by the norm∥ f
∥
L p ( X ) = (cid:0) (cid:82) X | f ( x ) | p d x (cid:1) 1 /p , and the µ -integrable L p space by L p (d µ ) for a probability measureµ on X and the norm is given by ∥
f ∥ L p (d µ ) = (cid:0) (cid:82) X | f ( x ) | p d µ ( x ) (cid:1) 1 /p .
2.1 Episodic Markov decision processes
A (finite-horizon) episodic MDPs is denoted as MDP( S , A , H, P , r ) , where S is the state space withpossibly infinite states; A is the action space, often with finite actions; H is the number of steps ineach episode; P :=
{ P h } Hh =1 is the Markov transition kernel with the transition probability P h ( ·| s, a )on action a taken at state s ∈ S in the h -th step; the reward functions r :=
{ r h } Hh =1 are assumed to be deterministic.
For notational simplicity, denote X = S
× A and x = ( s, a ) , we assume X = [0 , 1] das a compact space of R d and r h : S
×
A → [0 , 1] at h -th step.
A non-stationary policy π is a collection of H functions π :=
{ π h : S → A} Hh =1 .
Given a policyπ , the (state) value function V πh : S → [0 , H ] is defined as the expected cumulative reward of the
MDP starting from step h ∈
[ H ] , i.e., V πh ( s ) =
E π (cid:2) (cid:80)
Hh ′ = h r h ′
( s h ′ , a h ′ ) (cid:12)(cid:12) s h
= s (cid:3) , ∀ s ∈ S , h ∈
[ H ]where E π [ · ] denotes the expectation with respect to the randomness of the trajectory { ( s h , a h ) } Hh =1obtained by the policy π .
Likewise, the action-value function Q πh : S ×
A → [0 , H ] is defined as
Q πh ( s, a ) =
E π (cid:2) (cid:80)
Hh ′ =
h r h ′
( s h ′ , a h ′ ) (cid:12)(cid:12) s h = s, a h = a (cid:3) .
Moreover, since the action space and episode length are both finite, there always exists an optimalpolicy π ⋆ such that V ⋆h ( s )
=
sup π V πh ( s ) for all s ∈ S and h ∈
[ H ] [48].
To simplify thenotation, denote ( P h V )( s, a ) :
=
E s ′ ∼ P h ( ·| s,a )
[ V ( s ′ )] and the Bellman operator ( T h V )( s, a ) =
r h ( s, a ) +
( P h V )( s, a ) for any measurable function
V : S → [0 , H ] .
Using this notation, the
Bellman equation associated with a policy π can be formulated as
[Equation]
Similarly, the Bellman optimality equation is given by
[Equation]
Accordingly, the optimal policy π ⋆ is the greedy policy with respect to { Q ⋆h } Hh =1 .
Due tor h ∈ [0 , 1] yielding Q h ∈ [0 , H − h + 1] , we introduce the projection operator K ( f )( s h , a h ) :
=min { f ( s h , a h ) , H − h +1 } + in our analysis to obtain the tighter bound.
Hence the Bellman optimalityoperator T ⋆h is defined as
[Equation]
By definition, the Bellman equation in Eq.
(2) is equivalent to Q ⋆h =
T ⋆h Q ⋆h +1 , ∀ h ∈
[ H ] .
In the online setting , the goal is to learn the optimal policy π ⋆ by minimizing the cumulative regretunder the interaction with the environment over a number of episodes.
For any policy π , the differencebetween V π 1 and V ⋆ 1 quantifies its sub-optimality.
Thus, after T (fixed but large) episodes, the total(expected) regret is defined as Regret( T ) =
(cid:80) Tt =1 (cid:2)
V ⋆ 1 ( s t 1 ) − V π t 1 ( s t 1 ) (cid:3) , where π t is the policyexecuted in the t -th episode and s t 1 is the initial state.
2.2 Barron spaces and Besov spaces
We begin with introduction on the Barron space associated with two-layer ReLU neural networksand then give an overview of Besov spaces for deep ReLU neural networks.
More details refer to
Appendix A. For description simplicity, we focus on the ReLU activation function in this work.
Barron spaces: A two-layer neural network with m neurons can be represented as 3 f ( x ) =1 m (cid:80) mk =1 b k σ ( w ⊤ k x ) , where σ ( x ) = max { x, 0 } is the ReLU activation function used in this work.
Ifthe parameters { ( b k , w k ) } mk =1 are associated with a signed Radon measure with finite total variation[23, 49], this function space is in fact a Barron space (cid:101) P [34], of which the Barron norm can beroughly substituted by with the ℓ 1 -path norm, i.e., ∥
[Equation]
Accordingly, it is natural to use ∥ f ∥ P to denote the Barron norm, as the discrete version.
The Barron space can be regarded as the largest function space for two layer neural networksin two folds [34, 35]: 1) direct approximation: Any function in Barron spaces can be efficientlyapproximated by two-layer neural networks with bounded ℓ 1 -path norm; 2) inverse approximation:
Any continuous function that can be efficiently approximated by two-layer neural networks withbounded ℓ 1 -path norm belongs to a Barron space.
Besov spaces: Different from two-layer neural networks in Barron spaces, the complete functionspace of deep ReLU neural networks is still open.
Previous work in approximation theory focuses on the “smoothness” of the function, e.g., Hölder spaces [50, 38] and Sobolev spaces [51].
Here weconsider the concept of α -smooth from modulus of smoothness [37], cf. , Appendix A.
Based on this, we consider a more general function space beyond Hölder spaces and Sobolev spaces,i.e., Besov spaces [36, 37], which allows for spatially inhomogeneous smoothness with spikes andjumps.
The Besov space is defined by B αp,q ( X )
= { f ∈
L p ( X ) | ∥ f ∥ B αp,q < ∞} , where the Besovnorm is ∥ f ∥
B αp,q := ∥
f ∥ L p ( X ) + | f | B αp,q .
The smoothness parameter α indicates which function at acertain smoothness degree can be represented.
For example, if α > d/p , then the related Besov spaceis continuously embedded in the set of the continuous functions; if α < d/p , then the functions inthe Besov space are no longer continuous.
The formal definition and relations to Hölder spaces and
Sobolev spaces are deferred to Appendix A.
Mixed smooth Besov spaces: Learning in Besov spaces [52] still faces the curse of dimensionalitysince the Besov space is too general.
To tackle this issue, we use the mixed Besov space (cid:101) B αp,q ( X )[53, 37], as defined by ∥ f ∥ (cid:101) B αp,q := ∥
f ∥ L p ( X ) + | f | (cid:101) B αp,q , and consider the differentiation over allcoordinates, which does not heavily depend on the feature dimension d .
Intuitively, the Besov spaceensures a function with boundedness of derivatives, of which the “sum” of the orders are smallerthan α ; while the mixed smooth Besov space allows for the “max” of the orders of derivatives overcoordinates smaller than α .
Hence, we obtain the approximation performance and the separation onthe dimension-dependent convergence rate for sublinear regret.
Value iteration via neural networks under ϵ -greedy exploration
[Figure] In this section, we lay out our algorithm 1 via value iteration by DNNs under the ϵ -greedy policy.
After initialization, at t -th episode, at h -th time step, we have observed t − 1 transition tuples,{ ( s τh , a τh , s τh +1 ) } t − 1 τ =1 and attempt to estimate { Q ⋆h } Hh =1 by the following least squares regressionproblem (cid:98) Q th = argmin f ∈F (cid:98) E th ( f ) with(cid:98) E
[Equation]
where K is the projection operator defined before for tight bound.
The function class F is given by
F = F 1 × · · · × F H , including F SNN for two-layer (Shallow) ReLU neural networks and F DNN fordeep ReLU neural networks as below.
[Equation]
where B > 0 is the ℓ 1 -path norm constraint parameter, and deep ReLU neural networks [37] as
F DNN ( L, m, S, B ) :=
[Equation]
where the weight parameters are W {+(
 )+} ∈
R m × d , W l ∈ R m × m , ∀ l ∈ { 2 , 3 , . . .
, L − 1 } , and
W L ∈ R m ; the bias parameter are b ( l ) ∈ R m , ∀ l ∈
[ L − 1] and b ( L ) ∈ R .
The depth L , the widthm , the sparsity parameter S and the norm parameter B can be further determined to achieve goodapproximation and estimation performance.
In our setting, we use the historical t − 1 episodic data { ( s τh , a τh , s τh +1 ) } t − 1 τ =1 for value functionapproximation by neural networks.
It can be found that, for each time step h , these t − 1 examplesare merely conditionally independent snf non-identically distributed.
Formally, we assume each stateaction pair is sampled from a respective (unknown) probability measure, i.e., ( s τh , a τh ) ∼ µ τh , ∀ τ ∈
[ t − 1] , where µ τh ∈ P ( S × A ) is from the collection of all probability distribution on S × A .
Takingthe averaged measure ¯ µ th := 1 t − 1 (cid:80) t − 1 τ =1 µ τh , the expected risk of Eq.
(3) is defined as
[Equation]
ϵ
1: Input: Function class F , the number of episodes T , and the parameter ϵ ∈ (0 , 1) .
2: for episode t = 1 , . . .
, T do 3:
Receive the initial state s t 1 .
Set V tH +1 as the zero function.
5: for step h = H, . . , 1 do 6: Obtain (cid:98) Q th := argmin f ∈F (cid:80) t − 1 τ =1 (cid:2) K f
( s τh , a τh ) − r h
( s τh , a τh )
− V th +1 ( s τh +1 ) (cid:3) 2 .
Obtain Q th = min { (cid:98) Q th , H − h + 1 } + and V th ( · ) = max a ∈A Q th ( · , a ) .
8: end for 9: //
ϵ -greedy for exploration 10:
Take the policy { π th } Hh =1 to be greedy policy with probability 1 − ϵ or any policy with probability ϵ .
11: for step h = 1 , . . .
, H do 12:
Take ˜ a th ∼ π th ( ·| s th ) .
Observe the reward r h ( s th , ˜ a th ) and obtain the next state s th +1 .
14: end for 15: end for
Note that, Eq.
(3) is a biased estimator of Eq.
(6) due to an extra variance term caused by the stochastictransition [54].
However, this issue can be avoided by introducing extra ε -net of the value functionset [7] in theory.
In practice, DQN [4] uses target networks to reduce the biased estimate or eveneliminate it [38], and some variants [55] can also reduce the biased estimate and performs wellwithout target networks.
Accordingly, both practical uses and theoretical guarantees provide a goodjustification to not strictly distinguish this biased estimation in our analysis for better understandingdeep RL.
Theoretical guarantees of efficient value iteration via DNNs
In this section, we present our results for value iteration achieving sublinear regret under two-layerand deep ReLU neural networks.
4.1 Efficient value iteration under two-layer ReLU neural networks
In this setting, we consider (cid:98) Q th = argmin f ∈F SNN (cid:98)
E th ( f ) in Eq.
(3), where F SNN is the function spaceof two-layer ReLU neural networks defined in Eq.
We give an assumption for two-layer ReLUneural networks, where the Bellman optimality operator maps any bounded value function to abounded Barron space ball.
This assumption is commonly used in RL theory under various functionclasses [7, 13].
Assumption 1.
Let (cid:101) R > 0 be a fixed constant.
Define P (cid:101) R = { f ∈ P : ∥ f ∥ P ≤ (cid:101) R } in the Barronspace, and assume that for any h ∈ [ H ] and Q : S × A → [0 , H ] , we have T ⋆ h Q ∈ P (cid:101) R .
Remark:
Due to Q h ∈ [0 , H − h + 1] , the radius R in fact depends on H , and thus we have (cid:101) R ≍ H .
Based on this assumption, we have the following regret bounds for two-layer ReLU neural networks,with the proof deferred to Appendix E.
Theorem 1.
Under Assumption 1, given K ∈ [1 , H ] , considering that the Q-function is learnedby two-layer ReLU neural networks with width m and bounded ℓ 1 norm B in Eq.
(4) , then underthe ϵ -greedy exploration, for any δ ∈ (0 , 1) , the total expected regret can be upper bounded withprobability at least 1 − δ Regret
[Equation]
+ [Equation] , by taking m = Ω( √ T ) and ϵ = O (cid:16) ( HK ) 2 K +2 T − 1 2( K +2) (cid:17) .
Remark: The exponential dependence on H (in the worst case for any MDP) due to the employedϵ -greedy can be avoided at an additional cost of worsening T dependence, i.e., from (cid:101) O ( T 34 ) to (cid:101)
O ( T 2 K +3 2 K +4 ) .
Note that ϵ =
O (cid:16) ( HK ) 2 K +2 T − 1 2( K +2) (cid:17) is still satisfied with ϵ ∈ (0 , 1) for any K ∈ [1 , H ] under a large episode T .
Besides, our regret bound almost gets rid of the featuredimension d , which allows for DQN for high-dimensional image data in practice. We will discussthe achieved regret later with linear function approximation under the ϵ -greedy exploration, refer to Remark in Theorem 2.
4.2 Efficient value iteration under deep ReLU neural networks
In this setting, we consider (cid:98) Q th = argmin f ∈F DNN (cid:98) E th ( f ) in Eq.
(3), where F DNN is the function spaceof deep ReLU neural networks defined in Eq.
Similar to Assumption 1, we have the followingassumption on the Besov space B , where the Bellman optimality operator maps any bounded valuefunction to a bounded Besov space ball.
Assumption 2.
Let (cid:101) R ≍ H be a fixed constant.
Define B (cid:101) R = { f ∈ B αp,q ( X ) :
∥ f ∥ B ⩽ (cid:101) R } in the
Besov space and assume that for any h ∈ [ H ] and Q : S × A → [0 , H ] , we have T ⋆h Q ∈ B (cid:101) R .
Based on this assumption, we have the following theorem on regret bound in the Besov space forefficient deep Q-learning, with the proof deferred to Appendix F.1.
Theorem 2.
Under Assumptions 2 with the smoothness parameter α > d (1 /p − 1 / 4) + in the Besovspace B αp,q ( X ) , given a constant K ∈ [1 , H ] , considering that the Q-function is learned by deep
ReLU neural networks in Eq.
(5) under the ϵ -greedy exploration, takingthe depth L ≍
[Equation]
log T , the width m ≍
[Equation]
+ d log T , then for any δ ∈ (0 , 1) , the regret bound holds with probability at least 1 − δ
Regret ( T ) ≲ (cid:16) ϵ A (cid:17) − K 2 (cid:32) H 32 T α + d 2 α + d log 3 T + H 2 √ T (cid:114) log (cid:16)δ (cid:17) log T (cid:33) + ϵHT ≲ (cid:101) O ( H H +4 H
{++2 K 2 K +
+}
A KK +2 T αK +( α + d )( K +2) (2 α + d )( K +2) ) , taking ϵ
{+= O (( HK ) 2 K +
 A KK +2 T − 2 α (2 α + d )( K +2) ) . Remark: We make the following remarks.i )+}
If the smoothness α → ∞ (i.e., the target Q function is sufficiently smooth), which holds for linearfunction approximation, we have
[Equation]
which recovers the regret bound (cid:101)
O ( T K +1 K +2 ) in [46, Theorem 3]. In the best case, taking K = 1 , ourregret bound implies (cid:101) O ( H 43 A 13 T 23 ) with H ⩾ 4 , which matches the optimal regret bound for thecontextual bandits problem in terms of dependence on T or A under the ϵ -greedy exploration [56].
Inthe worst case (for any MDP case), we can still obtain sublinear regret (cid:101) O
( T H +1 H +2 ) .ii ) It can be found that, one hand, (cid:101) O (1) depth and (cid:101) O
( T d 2 α + d ) are enough for sublinear regret in deep Q-learning.
On the other hand, the change of problem-dependent smoothness leads to less changeson the depth but more changes on the width.
To be specific, if the Q-function becomes less smoothfrom α to α 2 with α 1 > α 2 , we have different depths L 1 , L 2 and widths m 1 , m 2 under these twosettings, then we have m 2 − m 1 ⩾ T [Equation]
.iii )
According to the width-depth ratio m/L = T d 2 α + d , our theory indicates that less problemdependent smoothness of Q-function requires DNNs to be wider.
In practice, if we work in thelimited budget of parameters N in neural networks, e.g., N ≍ m 2 L , our theory implies that there is atradeoff between the depth and width on smoothness, i.e., the depth L := N 1 / 3 T − 2 d 3(2 α + d ) increasingwith α (or T ) and the width m = N 1 / 3 T d [Equation] [Equation] + d ) decreasing with α (or T ).iv )
Two-layer neural networks endowed by Barron spaces cannot be regarded as a special case ofDNNs endowed by Besov spaces.
This is because, our results in Barron spaces accommodate densetwo-layer neural networks; while our result in Besov spaces here depends on the sparsity of deepReLU neural networks [57] for statistical complexity estimation of F DNN .  
Rademacher complexity on non-i.i.d data: Lem. 6 ⇐ Lem.two layers: Thm. 1 ⇐ O ( H 2 B (cid:112) log d/n ) ⇐ Lem. 10: Rademacher complexity of Barron spaces
DNNs:   Besov spaces: Thm. 2 ⇐ O ( n − 2 α 2 α + d ) ⇐ Prop. 3 ⇐ Lem.
{+11 on LRC for Besov spaces mixed smooth Besov spaces: Prop. 1 ⇐ O ( n − 
+}
α 2 α +1 ) (or faster)
⇐
[Figure]
It can be found that, though the sub-linear regret can be achieved by Theorem 2 by choosing (cid:101)
O
(1)depth and (cid:101)
O
( T d 2 α + d ) width, the regret bound heavily depends on the feature dimension d , failing inthe curse of dimensionality , which appears ineffective on high dimensional data in deep RL.
To tacklethis issue, we consider mixed smooth Besov space (cid:101) B to alleviate the curse of dimensionality.
Assumption 3.
Let (cid:101) R ≍ H be a fixed constant.
Define (cid:101) B (cid:101)
R
=
{ f ∈ (cid:101) B αp,q : ∥ f ∥ (cid:101)
B ⩽ (cid:101) R } in the mixedsmooth Besov space and assume that ∀ h ∈
[ H ] and Q : S ×
A → [0 , H ] , then we have T ⋆h Q ∈ (cid:101) B (cid:101) R .
Based on this assumption, we have the following proposition on regret bound in the mixed-smooth
Besov space, with the proof deferred to Appendix F.2.
Proposition 1.
Under Assumptions 3 with the smoothness parameter α > 0 and u :=
(1 − 1 /q )
+defined in the mixed smooth Besov space (cid:101) B αp,q with p = 4 , given a constant K ∈ [1 , H ] , consideringthat the Q-function is learned by deep ReLU neural networks in Eq.
(5) under the ϵ -greedy exploration,takingthe depth L ≍ (cid:101) O (1) , the width m ≍ (cid:101) O
( T 1 2 α +1 ) , then for any δ ∈ (0 , 1) with probability at least 1 − δ , we have Regret ( T ) ≲   [Equation] O
[Equation]
+
[Equation]
+
[Equation]
2 K +
[Equation]
( α +1)( K +1)+1 (2 α +1)( K +2)
(cid:17) , if α ⩽ u log 2 e and taking ϵ = O (( HK ) 2 K +2 T − 2 α (2 α +1) ( K +2) ) (cid:101) O (cid:16)
H H +4 H
+2 K 2 K +2 T ˜ c ( K +1)+1 ˜
c ( K +2) (cid:17) , if α > u log 2 e and taking ϵ = O (cid:16) ( HK ) 2 K +2 T {+2(log
 e − α+} )
˜ c ( K +2) (cid:17) . where ˜ c := 1 + 2 α + (1 − 2 u ) log 2 e .
Remark:
Our regret bound gets rid of the feature dimension d and is close to the (cid:101) O ( T K +1 K +2 ) boundif the Q-function is sufficiently smooth, which holds for DQN on high dimensional image data inpractice.
Proof outline
In this section, we outline the proof of value iteration via DNNs with sublinear regret bounds presentedin Section 4.
Our proof framework includes three parts: regret decomposition, transformation between
TD error and generalization, and generalization bounds on non-iid data for TD error analysis.
Thelast two parts are the main technical contribution in this work and provide width and depth guidelinefor efficient deep RL.
The complete proof is reported in the appendix.
Regret decomposition: This part is standard and commonly studied in RL theory, e.g., [58, 7, 13].
We briefly include here for self-completeness.
Define the temporal-difference (TD) error as
[Equation]
where Γ th is a function on S × A for all h ∈
[ H ] and t ∈ [ T ] .
Accordingly, the regret can bedecomposed into two main terms related to the TD error and ϵHT due to the ϵ -greedy exploration,respectively ( c.f. Lemma 1).
Transforming TD error to generalization: To bound the TD error, previous theory work focuseson designing bonus function via confidence based algorithms to ensure TD error negative, whichmismatches with practical DQN that works with ϵ -greedy exploration.
Accordingly, we attempt to built the connection between the regret error and ∥
Γ th ∥ L 2 (d¯ µ th ) , see Lemma 3; and then transformestimation on ∥ Γ th ∥ L 2 (d¯ µ th ) to the following two terms: generalization error and approximation error,respectively ( c.f. Lemma 4)
[Equation]
The convergence rate of approximation error can be directly obtained by previous work, e.g., O (1 /m )for two-layer ReLU networks in Barron spaces [34] and O ( n − 2 α/ (2 α + d ) ) in [37] for deep ReLUnetworks; and in the next we elucidate the generalization error bounds.
Analysis of TD error based on generalization bounds for non-iid data: The key part left isto bound the generalization error under our non-i.i.d data for TD error, see the proof frameworkin Figure 1.
Following a covering argument for the function approximation class used for valuefunctions [7], we introduce two additional terms to handle conditional independence and biasedestimation, which can be upper bounded by ε -covering of F .
By doing so, we firstly verify that themaximum error in estimating the mean of any function f ∈ F can be still bounded by the Rademachercomplexity of F in Lemma 5, and then generalization bounds by Rademacher complexity still holdsby Lemma 6 via the averaged measure, which only requires the data to be independent.
To be specific, for two-layer neural networks, by computing the Rademacher complexity of F SNNin Lemma 10, we obtain the generalization error at a certain O ( H 2 B (cid:112) log d/n ) convergence rate;combining the result of approximation error in Barron spaces, we conclude the proof of Theorem 1with a sublinear regret for efficient deep RL.
For deep neural networks, by computing the local Rademacher complexity of F DNN in Lemma 11, bychoosing proper parameters in Eq.
(5), we present the generalization bounds in Besov spaces ( c.f.
Proposition 3) with O ( n − 2 α 2 α + d ) and mixed smooth Besov spaces ( c.f.
Proposition 4) with O ( n − 2 α 2 α +1 )(or faster) respectively.
Finally, combining the result of approximation error in these two spaces, thesublinear regret can be achieved in Theorem 2 that depends on d and Proposition 1 that gets rid of d ,respectively.
The above results demonstrate that, (cid:101) O (1) depth and the width in sublinear T are sufficeto efficient deep value iteration.
Conclusion and discussion
This paper provides an in-depth understanding on neural network function approximation withthe ϵ -greedy exploration under the online setting beyond linear/kernel approximation, and eluderdimension.
Our result sheds light on the theoretical guarantees of sub-linear regret bounds, with widthand depth guidelines in deep RL, i.e., 1) the width in sublinear of T and depth in a constant orderare sufficient to sublinear regret; 2) problem-dependent smoothness depending on the width/depthratio: for realistic low values of α , we observe that the depth L should be chosen approximatelyas log m .
Interestingly, we notice that this result is closely matching practical implementation of DQN.
For example, the choices of [4] m = 512 and L = 5 can be explained with our theory, indeedlog(512)
≈ 6 .
The analysis of this work is built on the ϵ greedy for exploration, which are satisfied in practical caseswhen employing DQN.
Nevertheless, designing a provably efficient exploration mechanism for deep RL could be an interesting future direction in both practice and theory.
Furthermore, our work is built on value iteration based algorithm, which is different from practical
DQN that adapts Q-learning via one-step gradient descent.
We remark that, this work focuseson neural network function approximation beyond “linear” regime, which shares the same spiritand main intention of DQN but has a large step towards theoretically understanding DQN in acomprehensive view.
For example, the experience replay and target network in DQN are not studiedin this work, as a significant role on its success in practice, and would be an interesting direction forbetter understanding DQN, e.g., [45].
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We clearly discuss the limitation
of this work in the Conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
is theoretical and generally will have no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The assump-
tions are clearly stated and well discussed.
(b) Did you include complete proofs of all theoretical results?
[Yes] All of the proofs can
be found in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]