Width and Depth Guidelines for Deep Q-Learning: A Function Approximation Perspective
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper provides a theoretical study of the successful deep Q-networks (DQN)framework from the perspective of function class and neural networks architectures(e.g.
, width and depth).
Motivated by practical deep neural networks used in DQN under an online setting with T episodes, we focus on Besov and Barronfunction spaces in approximating an α -smooth Q-function in a d -dimensionalfeature space.
Based on a probabilistic sufficient exploration (PSE) assumption,which is attainable in practice, we prove that scaling the width m = (cid:101)
O
( T d 2 α + d )and the depth L = (cid:101) O (1) of the neural network for DQN is sufficient for learningwith sublinear regret in Besov spaces.
Moreover, for a two layer neural network inthe Barron space, scaling the width even in sublinearly in T is sufficient.
Whileincreasing the model capacity improves the DQN efficiency in both cases, ourtheory suggests that the width has a more prominent effect than depth.
In particular,for low smooth Q-function, the theory suggests that the depth should be chosen asthe logarithm of the width.
This matches the practical DQN setting.
Our analysisrelies on transforming the estimation of temporal difference error to a generalizationproblem under the conditionally independent but non-identically distributed datasetting over the averaged empirical measure.
Introduction
Efficient reinforcement learning (RL) under the large (or even infinite) state space and action spacesetting is increasingly important and relevant challenge [1, 2, 3].
One of the first successful approachestowards this problem is the deep Q-network (DQN) [4, 5] framework, which deploys powerfulnonlinear function approximation techniques via Deep Neural Networks (DNNs) [6] to conciselyapproximate state and action spaces.
Despite its impressive practical success there is still a gap between practical uses and theoreticalunderstanding on DQN under the online setting e.g., the nonlinear Q-function approximation andhow much the width and depth of DNNs are enough.
In contrast, many works directly focus on linearfunction approximation [7, 8], linear mixtures [9, 10].
Existing non-linear function approximation results on RL are largely based on neural tangent kernel(NTK)
[11, 12], Bellman rank [13, 14], and Eluder dimension [15, 16, 17].
Nevertheless, theseapproaches fail in truly capturing the highly non-linear properties of DQN.
Indeed, NTK (or lazytraining [18]) essentially works in a linear regime [19, 20, 21], and can not efficiently learn even asingle ReLU neuron [22, 23, 24] as it requires Ω( ε − d ) samples to achieve ε approximation error,where d is the (original) or transformed feature dimension input.
Interestingly, even for two-layer neural networks, the corresponding Eluder dimension is at least in anexponential order [25, 26], which appears difficult to fully demonstrates the success of practical DQNboth theoretically and empirically.
Regarding to ensuring efficient deep Q-learning for sublinearregret, the width of neural networks under the NTK regime [12] is required to be m = Ω( T 13 ) , where
T is the number of episodes, which does not match the DQNs used in practice with small width/depthunder large episodes [4, 27].
To bridge the large theory-practice gap, we make a first attempt to study DQN with a slight modification but broadly capture its key features under the online setting.
Our analysis framework is equippedwith two-layer as well as deep fully-connected neural networks in the respective Barron and Besovspaces, which can fully capture the properties of Q-functions, e.g., smoothness by neural networks.
We also consider a weaker version of the independent and identically distributed (iid) assumption byexperience replay in DQN, which could be of interest beyond DQN.
Our results allow for reasonably finite width and depth in practice, give precise characterization ofthese parameters to obtain sublinear regret, and demonstrate the relationship between the problemdependent smoothness of Q-function and regret bounds.
We believe that these results can be viewedas a guide for practitioners to choose the architecture in practical implementations of DQN.
1.1 Technical challenges and contributions
Most previous RL theory results on function approximation in the online setting work with “optimismin the face of uncertainty” principle for exploration, leading to a series of upper confidence bound(UCB)-type algorithms.
However, conceptually, optimism is sometimes too aggressive and UCB-stylealgorithms can suffer exponential sample complexity even for nonlinear bandits [26].
Empiricalmodel-based RL has also not found optimism to be very useful [28].
In the RL setting, linear/kernel function approximation [7, 12, 29] depends on a known featuremapping or the NTK matrix to design a UCB-type bonus function for optimism, which appearsinvalid for DNNs as the used confidence ellipsoid and elliptical potential lemma are not applicablefor data-dependent feature mapping.
To avoid explicitly designing a bonus function, Thompson sampling [30, 31] appears promising in a
Bayesian perspective by using randomized (i.e., perturbed) versions of the estimated model or valuefunction [32].
Nevertheless, the bonus function is still implicitly included in confidence estimate ofperturbations in their UCB-style algorithm.
This work analyzes an algorithm closely related to DQN in general function classes and providesa new regret analysis framework under a realistic and attainable exploratory assumption, dubbedas probabilistic sufficient exploration (PSE), to ensure how much width and depth are enough forefficient RL.
We remark that PSE can be satisfied in practical cases of interest when employing
DQN.
Designing a provably efficient exploration mechanism is beyond the scope of our work forunderstanding DQN but it remains an interesting future direction.
The technical contribution in our analysis is how to control the temporal difference (TD) error withoutbonus function design.
By a measure transform, we analyze the TD error in an L 2 -integrable space wrthistorical state-action pairs, and then transform to generalization bounds under the independent butnon-identically distributed data setting.
To this end, we need to employ proper function spaces beyondlinear/kernel Markov decision processes (MDP) in our analysis: one is the Barron space [33], as thelargest 1 function space for two-layer ReLU neural networks without curse of dimensionality [34]; theother is the Besov space [35, 36] for deep ReLU neural networks, which allows for continuous/noncontinuous functions and thus is more general than Hölder spaces used in offline RL [37].
Underthese proper hypothesis spaces, such generalization problem benefits from the (nearly) unbiasedestimation by the target network in DQN, and thus can be addressed by uniform convergence via
Rademacher complexity of these spaces under the averaged empirical measure.
Under attainable assumptions, our results show that ( i ) the problem-dependent smoothness of Qfunction effects the deep Q-learning efficiency, which can be improved by the model complexity.
Weuse α as a parameter indicating the smoothness degree of Q-function.
A larger α indicates smootherfunctions.
( ii ) for two-layer neural networks, the width m = Ω( √ T ) suffices to ensure optimalminimax (cid:101) O ( √ T ) regret.
( iii ) for deep neural networks, the width m = (cid:101) O
( T d 2 α + d ) and the depth
L = (cid:101) O (1) are enough for sublinear regret, where (cid:101) O ( · ) omits the log terms.
Overall, we find thatthe width has a more prominent effect than depth.
Furthermore, our regret bounds can be (nearly)independent of the feature dimension d , supporting the premise of practical, high-dimensional RL.
1.2 Related work
Theoretical understanding of DQN is due to [38, 37, 39], which regards DQN as a neural fitted
Q-iteration algorithm
[40] under an offline setting.
Recent work on neural network function approximation beyond NTK (or the Eluder dimension) mainly restrict on the generative setting [41, 42] byassuming a simulator can require any state and action, and the offline setting [37, 43].
In sequel, wereview RL with function approximation under the online setting that DQN falls into this regime.
RL with linear/kernel function approximation: RL with linear function approximation results inregret bounds (cid:101) O ( √ d 3 H 3 T ) [7] and (cid:101)
O ( dH 2 √ T ) [29], where H is the length of each episode; andlinear mixtures with regret (cid:101) O
( d √ H 3 T ) [9] and improved to (cid:101) O ( dH √ T ) [10]; kernel regime withregret (cid:101) O
( δ F H 2 √ T ) [29, 12], where δ F is the intrinsic complexity (e.g., effective dimension) of thefunction class RKHS F .
The above algorithms are based on confidence ellipsoid to quantify theuncertainty in an explicit bonus function by feature mapping/kernel matrix; while Thompson sampling[31, 30] utilizes an implicit bonus function in probability estimation on uncertainty quantification,which leads to (cid:101)
O ( d 2 H 2 √ T )
[32] regret in linear function approximation.
RL with general function approximation: One prototypical scheme uses the Eluder dimension [15],which measures the degree of dependence among action rewards, resulting in (cid:101)
O ( poly ( δ F H ) √ T )regret [16, 17], where the complexity δ F depends on the Eluder dimension.
Bellman rank [13] is alsoanother metric for general function approximation [14] by measuring the error of the function classunder the Bellman operator.
Jin et al.
[44] introduce a new metric, Bellman Eluder dimension, bycombining these two metrics, and achieve (cid:101)
O ( H √ δ F T ) , where δ F depends on this metric.
Overall, none of the above metrics can handle the nonlinear spaces that concern us.
Background and preliminaries
In this section, we provide some preliminaries with respect to online reinforcement learning based onepisodic MDPs, function space of two (and deep)-layer ReLU neural networks.
Notation: We denote by a ( n ) ≲ b ( n ) : there exists a positive constant c independent of n such thata ( n ) ⩽ cb ( n ) .
We use the shorthand [ n ] := { 1 , 2 , . . . , n } for some positive n .
Let X =
[0 , 1] dbe a domain of the functions, we denote the L p -integrable space by L p ( X ) endowed by the norm∥ f
∥
L p ( X ) = (cid:0) (cid:82) X | f ( x ) | p d x (cid:1) 1 /p , and the µ -integrable L p space by L p (d µ ) for a probability measureµ on X and the norm is given by ∥
f ∥ L p (d µ ) = (cid:0) (cid:82) X | f ( x ) | p d µ ( x ) (cid:1) 1 /p .
2.1 Episodic Markov decision processes
A (finite-horizon) episodic MDPs is denoted as MDP( S , A , H, P , r ) , where S is the state space withpossibly infinite states; A is the action space, often with finite actions; H is the number of steps ineach episode; P :=
{ P h } Hh =1 is the Markov transition kernel with the transition probability P h ( ·| s, a )on action a taken at state s ∈ S in the h -th step; the reward functions r :=
{ r h } Hh =1 are assumed to bedeterministic.
For notational simplicity, denote X = S
× A and x = ( s, a ) , we assume X = [0 , 1] das a compact space of R d and r h : S
×
A → [0 , 1] at h -th step.
A non-stationary policy π is a collection of H functions π :=
{ π h : S → A} Hh =1 .
Given a policyπ , the (state) value function V πh : S → [0 , H ] is defined as the expected cumulative reward of the
MDP starting from step h ∈
[ H ] , i.e., V πh ( s )
[Equation]
E π
[Equation]
[Equation] s ∈ S , h ∈ [ H ]where E π [ · ] denotes the expectation with respect to the randomness of the trajectory { ( s h , a h ) } Hh =1obtained by the policy π .
Likewise, the action-value function Q πh : S ×
A → [0 , H ] is defined as
[Equation]
Moreover, since the action space and episode length are both finite, there always exists an optimalpolicy π ⋆ such that V ⋆h ( s )
=
sup π V πh ( s ) for all s ∈ S and h ∈
[ H ] [45].
To simplify thenotation, denote ( P h V )( s, a ) :
=
E s ′ ∼ P h ( ·| s,a )
[ V ( s ′ )] and the Bellman operator ( T h V )( s, a ) =
r h ( s, a ) +
( P h V )( s, a ) for any measurable function
V : S → [0 , H ] .
Using this notation, the
Bellman equation associated with a policy π can be formulated as
[Equation]
Similarly, the Bellman optimality equation is given by
[Equation]
[Equation]
[Equation]
Accordingly, the optimal policy π ⋆ is the greedy policy with respect to { Q ⋆h } Hh =1 .
Due tor h ∈ [0 , 1] yielding Q h ∈ [0 , H − h + 1] , we introduce the projection operator K ( f )( s h , a h ) :
=min { f ( s h , a h ) , H − h +1 } + in our analysis to obtain the tighter bound.
Hence the Bellman optimalityoperator T ⋆h is defined as
[Equation]
By definition, the Bellman equation in Eq.
(2) is equivalent to Q ⋆h =
T ⋆h Q ⋆h +1 , ∀ h ∈
[ H ] .
In the online setting , the goal is to learn the optimal policy π ⋆ by minimizing the cumulative regretunder the interaction with the environment over a number of episodes.
For any policy π , the differencebetween V π 1 and V ⋆ 1 quantifies its sub-optimality.
Thus, after T (fixed but large) episodes, the total(expected) regret is defined as Regret( T ) =
(cid:80) Tt =1 (cid:2)
V ⋆ 1 ( s t 1 ) − V π t 1 ( s t 1 ) (cid:3) , where π t is the policyexecuted in the t -th episode and s t 1 is the initial state.
2.2 Barron spaces and Besov spaces
We begin with introduction on the Barron space associated with two-layer ReLU neural networksand then give an overview of Besov spaces for deep ReLU neural networks.
More details refer to
Appendix A. For description simplicity, we focus on the ReLU activation function in this work.
Barron spaces: A two-layer neural network with m neurons can be represented as 2 f ( x ) =1 m (cid:80) mk =1 b k σ ( w ⊤ k x ) , where σ ( x ) = max { x, 0 } is the ReLU activation function used in this work.
Ifthe parameters { ( b k , w k ) } mk =1 are associated with a signed Radon measure with finite total variation[22, 46], this function space is in fact a Barron space (cid:101) P [33], of which the Barron norm can beroughly substituted by with the ℓ 1 -path norm, i.e., ∥ f ∥ (cid:101) P ⩽ ∥ f ∥
P := 1 m (cid:80) m k =1 | b k |∥ w k ∥ 1 ⩽ 2 ∥ f ∥ (cid:101) P .
Accordingly, it is natural to use ∥ f ∥ P to denote the Barron norm, as the discrete version.
The Barron space can be regarded as the largest function space for two layer neural networksin two folds [33, 34]: 1) direct approximation: Any function in Barron spaces can be efficientlyapproximated by two-layer neural networks with bounded ℓ 1 -path norm; 2) inverse approximation:
Any continuous function that can be efficiently approximated by two-layer neural networks withbounded ℓ 1 -path norm belongs to a Barron space.
Besov spaces: Different from two-layer neural networks in Barron spaces, the complete functionspace of deep ReLU neural networks is still open.
Previous work in approximation theory focuseson the “smoothness” of the function, e.g., Hölder spaces [47, 37] and Sobolev spaces [48].
Here weconsider the concept of α -smooth from modulus of smoothness [36], cf. , Appendix A.
Based on this, we consider a more general function space beyond Hölder spaces and Sobolev spaces,i.e., Besov spaces [35, 36], which allows for spatially inhomogeneous smoothness with spikes and jumps.
The Besov space is defined by B αp,q ( X )
= { f ∈
L p ( X ) | ∥ f ∥ B αp,q < ∞} , where the Besovnorm is ∥ f ∥
B αp,q := ∥
f ∥ L p ( X ) + | f | B αp,q .
The smoothness parameter α indicates which function at acertain smoothness degree can be represented.
For example, if α > d/p , then the related Besov spaceis continuously embedded in the set of the continuous functions; if α < d/p , then the functions inthe Besov space are no longer continuous.
The formal definition and relations to Hölder spaces and
Sobolev spaces are deferred to Appendix A.
Mixed smooth Besov spaces: Learning in Besov spaces [49] still faces the curse of dimensionalitysince the Besov space is too general.
To tackle this issue, we use the mixed Besov space (cid:101) B αp,q ( X )[50, 36], as defined by ∥ f ∥ (cid:101) B αp,q := ∥
f ∥ L p ( X ) + | f | (cid:101) B αp,q , and consider the differentiation over allcoordinates, which not heavily depends on the feature dimension d .
Intuitively, the Besov spaceensures a function with boundedness of derivatives, of which the “sum” of the orders are smallerthan α ; while the mixed smooth Besov space allows for the “max” of the orders of derivatives overcoordinates smaller than α .
Hence, we obtain the approximation performance and the separation onthe dimension-dependent convergence rate for sublinear regret.
Vanilla DQN algorithm via value iteration
In this section, we lay out our algorithm 1 of vanilla DQN via value iteration by DNNs.
After initialization, at t -th episode, at h -th time step, we have observed t − 1 transition tuples,{ ( s τh , a τh , s τh +1 ) } t − 1 τ =1 and attempt to estimate { Q ⋆h } Hh =1 by the following least squares regressionproblem (cid:98) Q th = argmin f
[Equation]
(cid:98) E th ( f ) with
[Equation]
where K is the projection operator defined before for tight bound.
The function class F is given by
F = F 1 × · · · × F H , including F SNN for two-layer (Shallow) ReLU neural networks and F DNN fordeep ReLU neural networks as below.
[Equation]
where B > 0 is the ℓ 1 path norm constraint parameter, and deep ReLU neural networks [36] as
[Equation] L
[Equation] [Equation]
where the weight parameters are W (
) ∈
R m × d , W l ∈ R m × m , ∀ l ∈ { 2 , 3 , . . .
, L − 1 } , and
W L ∈ R m ; the bias parameter are b ( l ) ∈ R m , ∀ l ∈
[ L − 1] and b ( L ) ∈ R .
The depth L , the widthm , the sparsity parameter S and the norm parameter B can be further determined to achieve goodapproximation and estimation performance.
In our setting, we use the historical t − 1 episodic data { ( s τh , a τh , s τh +1 ) } t − 1 τ =1 for value functionapproximation by neural networks.
It can be found that, for each time step h , these t − 1 examplesare merely conditionally independent snf non-identically distributed.
We do not employ experiencereplay in DQN for the iid assumption, which could expand the application scope of DQN.
Formally,we assume each state-action pair is sampled from a respective (unknown) probability measure, i.e.,( s τh , a τh )
∼ µ τh , ∀ τ ∈ [ t − 1] , where µ τh ∈ P ( S × A ) is from the collection of all probabilitydistribution on S × A .
Taking the averaged measure ¯ µ th := 1 t − 1 (cid:80) t − 1 τ =1 µ τh , the expected risk of Eq.
(3) is defined as
[Equation]
which is the exact Q-learning update [4].
Note that, Eq.
(3) is a biased estimator of Eq.
(6) due to anextra variance term caused by the stochastic transition [51].
However, this issue can be avoided if someschemes are employed, e.g., double sampling [52], simulator [42], and deterministic transition that isused in Atari [53] in practice.
In fact, DQN [4] uses target networks to reduce the biased estimateor even eliminate it [37], and some variants [54] can also satisfy this requirement without targetnetworks.
Accordingly, both practical uses and theoretical guarantees provide a good justification tonot strictly distinguish this biased estimation in our analysis for better understanding DQN.
Theoretical guarantees of efficient value iteration via DNNs
In this section, we present our results for value iteration achieving sublinear regret under two-layerand deep ReLU neural networks.
To begin, we need one assumption on ¯ µ th to make it applicable to online setting.
Denote G ϵ :
={ ( s, a ) : | f
( s, a ) | ⩾ ∥ f ∥ ∞ − ϵ } for any ( s, a ) ∈ S × A , where ϵ ⩽ ∥ f ∥ ∞ .
It is clear that thecardinality of G ϵ is a non-increasing function of ϵ .
For any probability measure µ on S × A , we have
[Equation]
which is also valid to our averaged measure ¯ µ th based on the historical state-action pairs.
We make afair and attainable assumption in practice on ¯ µ th ( G ϵ ) when employing DQN for some specific tasks.
Assumption 1.
[Probabilistic sufficient exploration (PSE)]
We assume that ¯ µ th ( G ϵ ) > 0 , ∀ ϵ ⩾ 1 / √ Thold with probability at least 1 − δ for some constant δ ∈ (0 , 1) .
Remark:
We discuss the rationale of the PSE assumption in theory and in practice.
( i )
In theory, our PSE assumption shares the similar spirit with [41, Theorem 4.8], i.e., at least onegood state-action pair can be visited according to the historical data, which admits the minimalrequirement on the data distribution with probability.
This is weaker than [55, Definition 3.1] thatrequires a positive visitation probability of each state-action pair.
In fact, general neural functionapproximation without extra assumptions is infeasible even with good representations [56, 57].
( ii )
In practice, exploration in DQN depends on massive training data rather than designing UCB-typebonus function, which coincides with the spirit of the PSE assumption.
This assumption can besatisfied in several cases of interest: (1) Assumption 1 can be satisfied using ϵ -greedy exploration assuggested in the original DQN algorithm [4].
(2) For deterministic dynamics in discrete states andactions spaces, we can use the first |S||A| /H episodes to visit every state-action pair once.
At thispoint we can recover a measure satisfying Assumption 1.
This may be feasible in Atari [53] wherethe frames seen by the agent are not all the possible pixel permutations but only a small subset ofthem that can be easily recognized and processed by a human.
In navigation tasks, Assumption 1holds when the initial distribution has full support over the state space.
4.1 Efficient value iteration under two-layer ReLU neural networks
In this setting, we consider (cid:98) Q th = argmin f ∈F SNN (cid:98)
E th ( f ) in Eq.
(3), where F SNN is the function spaceof two-layer ReLU neural networks defined in Eq.
We give an assumption for two-layer ReLUneural networks, where the Bellman optimality operator maps any bounded value function to abounded Barron space ball.
This assumption is commonly used in RL theory under various functionclasses [7, 12].
Assumption 2.
Let (cid:101) R > 0 be a fixed constant.
Define P (cid:101) R = { f ∈ P : ∥ f ∥ P ≤ (cid:101) R } in the Barronspace, and assume that for any h ∈ [ H ] and Q : S × A → [0 , H ]
, we have T ⋆h Q ∈ P (cid:101) R .
Remark:
Due to Q h ∈ [0 , H − h + 1] , the radius R in fact depends on H , and thus we have (cid:101) R ≍ H .
Based on this assumption, we have the following regret bounds for two-layer ReLU neural networks,with the proof deferred to Appendix E.
Theorem 1.
Under Assumptions 1, 2, considering that the Q-function is learned by two-layer ReLUneural networks with width m and bounded ℓ 1 norm B in Eq.
(4) , then for any δ ∈ (0 , 1) , the totalexpected regret can be upper bounded with probability at least 1 − δ
[Equation]
, by taking m = Ω( √ T ) .
Remark:
To ensure sub-linear regret, the width m in two-layer ReLU neural networks is requiredto m =
Ω( T β ) with β > 0 .
The result is complementary to scaling of m =
Ω( T 13 ) obtained in[12].
The results should be compared with care because the focus of the two works is different.
While, they focused on designing UCB like exploration in NTK, we focus on getting more realisticcharacterization of the width at the cost of introducing Assumption 1.
More importantly, our regret bound almost gets rid of the feature dimension d , which allows for DQNfor high-dimensional image data in practice.
4.2 Efficient value iteration under deep ReLU neural networks
In this setting, we consider (cid:98) Q th = argmin f ∈F DNN (cid:98) E th ( f ) in Eq.
(3), where F DNN is the function spaceof deep ReLU neural networks defined in Eq.
Similar to Assumption 2, we have the followingassumption on the Besov space B , where the Bellman optimality operator maps any bounded valuefunction to a bounded Besov space ball.
Assumption 3.
Let (cid:101) R ≍ H be a fixed constant.
Define B (cid:101) R = { f ∈ B αp,q ( X ) :
∥ f ∥ B ⩽ (cid:101) R } in the
Besov space and assume that for any h ∈ [ H ] and Q : S × A → [0 , H ] , we have T ⋆h Q ∈ B (cid:101) R .
Based on this assumption, we have the following theorem on regret bound in the Besov space forefficient deep Q-learning, with the proof deferred to Appendix F.1.
Theorem 2.
Under Assumptions 1, 3 with the smoothness parameter α > d
(1 /p − 1 / 4) + in the
Besov space B αp,q ( X ) , considering that the Q-function is learned by deep ReLU neural networks in
Eq.
(5) , takingthe depth L ≍
[Equation]
log T , the width m ≍
[Equation]
then for any δ ∈ (0 , 1) , the regret bound holds with probability at least 1 − δ
Regret ( T ) ≲ H
[Table]
Remark: ( i ) It is clear that the depth in (cid:101)
O (1) order and the width (cid:101)
O ( T d 2 α + d ) are enough to achievesublinear regret, which coincides with DQN in practice.
Under a special case with p = q = ∞ and α ≫ d , which requires Q-function to be infinitely differentiable, we obtain the optimal minimax(cid:101)
O ( √ T ) regret by taking the depth and width in a constant order.
( ii )
According to the width-depth ratio m/L = T d 2 α + d , our theory indicates that less problemdependent smoothness of Q-function requires DNNs to be wider.
In practice, if we work in thelimited budget of parameters N in neural networks, e.g., N ≍ m 2 L , our theory implies that there is atradeoff between the depth and width on smoothness, i.e., the depth L := N 1 / 3 T − 2 d 3(2 α + d ) increasingwith α (or T ) and the width m = N 1 / 3 T d 3(2 α + d ) decreasing with α (or T ).
( iii ) Two-layer neural networks endowed by Barron spaces cannot be regarded as a special case of
DNNs endowed by Besov spaces.
This is because, our results in Barron spaces accommodate densetwo-layer neural networks; while our result in Besov spaces here depends on the sparsity of deep
ReLU neural networks [58] for statistical complexity estimation of F DNN .
It can be found that, though the sub-linear regret can be achieved by Theorem 2 by choosing (cid:101)
O
(1)depth and (cid:101)
O
( T d 2 α + d ) width, the rate heavily depends on the feature dimension d , failing in the curseof dimensionality , which appears ineffective on high dimensional data in deep Q-learning.
To tacklethis issue, we consider mixed smooth Besov space (cid:101) B to alleviate the curse of dimensionality.
Assumption 4.
Let (cid:101) R ≍ H be a fixed constant.
Define (cid:101) B (cid:101)
R
=
{ f ∈ (cid:101) B αp,q : ∥ f ∥ (cid:101)
B ⩽ (cid:101) R } in the mixedsmooth Besov space and assume that ∀ h ∈
[ H ] and Q : S ×
A → [0 , H ] , then we have T ⋆h Q ∈ (cid:101) B (cid:101) R .
Based on this assumption, we have the following proposition on regret bound in the mixed-smooth
Besov space, with the proof deferred to Appendix F.2.
Proposition 1.
Under Assumptions 1, 4 with the smoothness parameter α > 0 and u :=
(1 − 1 /q )
+defined in the mixed smooth Besov space (cid:101) B αp,q with p = 4 , considering that the Q-function is learnedby deep ReLU neural networks in Eq.
(5) , takingthe depth L ≍ (cid:101) O (1) , the width m ≍ (cid:101) O
( T 1 2 α +1 ) , then for any δ ∈ (0 , 1) with probability at least 1 − δ , we have
[Equation]
Remark:
Our regret bound gets rid of the feature dimension d and is close to the minimaxsquare-root (cid:101) O ( √ T ) bound if the Q-function is sufficiently smooth, which holds for DQN onhigh dimensional image data in practice.
Further, if α > u log 2 e , the width can be improvedto m ≍ (cid:101) O (cid:16) T
[Equation]
(cid:17) suffice to efficient deep Q-learning with sublinear regret.
Proof outline
In this section, we outline the proof of value iteration via DNNs with sublinear regret bounds presentedin Section 4.
Our proof framework includes three parts: regret decomposition, transformation between
TD error and generalization, and generalization bounds on non-iid data for TD error analysis.
Thelast two parts are the main technical contribution in this work and provide width and depth guidelinefor efficient Q-learning.
The complete proof is reported in the appendix.
Regret decomposition: This part is standard and commonly studied in RL theory, e.g., [59, 7, 12].
We briefly include here for self-completeness.
Define the temporal-difference (TD) error asΓ t
[Equation]
[Equation]
=
[Equation]
P
[Equation]
t
[Equation]
) where Γ th is a function on S × A for all h ∈
[ H ] and t ∈ [ T ] .
Accordingly, the regret can bedecomposed into two main terms related to the TD error and the statistical error, respectively ( c.f.
Lemma 1).
The statistical error is based on the standard martingale difference sequences, and thuscan be upper bounded by Hoeffding-Azuma inequality with O ( √ H 3 T ) regret ( c.f. Lemma 2).
 
Rademacher complexity on non-i.i.d data: Lem.
6 ⇐ Lem.two layers: Thm.
1 ⇐ O ( H 2 B (cid:112) log d/n ) ⇐ Lem.
10: Rademacher complexity of Barron spaces deep layers  
[Figure]
[Figure]
Besov spaces
[Figure]
Transforming TD error to generalization: To bound the TD error, previous theory work focuses ondesigning bonus function to ensure TD error negative, which mismatches with practical DQN thatdepends on not bonus function but historical data.
Accordingly, we attempt to built the connection between the regret error and ∥
Γ th ∥ L 2 (d¯ µ th ) , see Lemma 3; and then transform estimation on ∥ Γ th ∥
L 2 (d¯ µ th )to the following two terms: generalization error and approximation error, respectively ( c.f. Lemma 4)
[Equation]
The convergence rate of approximation error can be directly obtained by previous work, e.g., O (1 /m )for two-layer ReLU networks in Barron spaces [33] and O ( n − 2 α/ (2 α + d ) ) in [36] for deep ReLUnetworks; and in the next we elucidate the generalization error bounds.
Analysis of TD error based on generalization bounds for non-iid data: The key part left isto bound the generalization error under our non-i.i.d data for TD error, see the proof frameworkin Figure 1.
Following a covering argument for the function approximation class used for valuefunctions [7], we introduce two additional terms to handle conditional independence and biasedestimation, which can be upper bounded by ϵ -covering of F .
By doing so, we firstly verify that themaximum error in estimating the mean of any function f ∈ F can be still bounded by the Rademachercomplexity of F in Lemma 5, and then generalization bounds by Rademacher complexity still holdsby Lemma 6 via the averaged measure, which only requires the data to be independent.
To be specific, for two-layer neural networks, by computing the Rademacher complexity of F SNNin Lemma 10, we obtain the generalization error at a certain O ( H 2 B (cid:112) log d/n ) convergence rate;combining the result of approximation error in Barron spaces, we conclude the proof of Theorem 1with O
( H 2 T 34 B log d ) regret for efficient Q-learning.
For deep neural networks, by computing the local Rademacher complexity of F DNN in Lemma 11, bychoosing proper parameters in Eq.
(5), we present the generalization bounds in Besov spaces ( c.f.
Proposition 3) with O ( n − 2 α 2 α + d ) and mixed smooth Besov spaces ( c.f.
Proposition 4) with O ( n − 2 α 2 α +1 )(or faster) respectively.
Finally, combining the result of approximation error in these two spaces,the regret can be upper bounded by (cid:101)
O ( H 2 T α + d 2 α + d ) in Theorem 2 and (cid:101)
O (cid:16) H 2 T
α +1 2 α +1 (cid:17) (or smaller) in
Theorem 1, respectively.
The above results demonstrate that, (cid:101) O (1) depth and the width in sublinear
T are suffice to efficient deep value iteration.
Conclusion and discussion
This paper provides an initial understanding on DQN with general function approximation by shallowand deep neural networks in the online setting.
Our results shed light on width and depth guidelinesin DQN.
In particular, the theoretical analysis provides guidelines to obtain the best regret guaranteeswith memory constraints.
For realistic low values of α , we observe that the depth L should bechosen approximately as log m .
Interestingly, we notice that this result is closely matching practicalimplementation of DQN.
For example, the choices of [4] m = 512 and L = 5 can be explained withour theory, indeed log(512)
≈ 6 .
We remark that, the analysis of this work is built on the PSE assumption for exploration, whichare satisfied in practical cases when employing DQN.
Nevertheless, designing a provably efficientexploration mechanism for DQN could be an interesting future direction in both practice and theory.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We clearly discuss the limitation
of this work in the Conclusion section.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
is theoretical and generally will have no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The assump-
tions are clearly stated and well discussed.
(b) Did you include complete proofs of all theoretical results?
[Yes] All of the proofs can
be found in the Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
N A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[Table]