O PTIMISTIC E XPLORATION WITH L EARNED F EATURES P ROVABLY S OLVES M ARKOV D ECISION P ROCESSES WITH N EURAL D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Incorporated with the recent advances in deep learning, deep reinforcement learning (DRL) has achieved tremendous success in empirical study.
However, analyzing
DRL is still challenging due to the complexity of the neural network class.
In this paper, we address such a challenge by analyzing the Markov decision process (MDP) with neural dynamics, which covers several existing models as special cases, including the kernelized nonlinear regulator (KNR) model and the linear
MDP.
We propose a novel algorithm that designs exploration incentives via learnable representations of the dynamics model by embedding the neural dynamics into a kernel space induced by the system noise.
We further establish an upper bound on the sample complexity of the algorithm, which demonstrates the sample efﬁciency of the algorithm.
We highlight that, unlike previous analyses of RL algorithms with function approximation, our bound on the sample complexity does not depend on the Eluder dimension of the neural network class, which is known to be exponentially large (Dong et al., 2021).
I NTRODUCTION
Reinforcement learning (RL) aims to accomplish sequential decision-making in an uncertain environment via iteratively interacting with the environment (see Sutton et al.
Equipped with modern function approximators such as deep neural networks, deep RL algorithms achieve tremendous empirical successes (Mnih et al., 2015; Silver et al., 2017; Hafner et al., 2019).
Despite its empirical successes the theoretical understanding of deep RL is relatively underdeveloped.
There are several recent works (Abbasi-Yadkori et al., 2019;
Wang et al., 2019; Fan et al., 2020) that analyze RL algorithms with neural network parameterization, including policy iteration (PI) (Lagoudakis & Parr, 2003), policy gradient (PG) (Williams, 1992) and deep Q-learning (Mnih et al., 2013).
However, those works depend on restrictive assumptions that either the agent has access to a simulator or the MDPs have bounded concentrability coefﬁcients, which in fact imply that the state space is already well-explored.
Another line of research (Jiang et al., 2017; Jin et al., 2020; Cai et al., 2019; Du et al., 2021) further removes such assumptions by conducting provably efﬁcient exploration in RL.
Such a direction of research typically hinges on a low-rank MDP assumption.
Thus, those works either assume that the MDP is linear in the known feature or propose computational-inefﬁcient algorithms, limiting the ability to explore the environment with neural network parameterization.
To explore the environment with neural network parameterization, a recent line of work (Wang et al., 2020; Jin et al., 2021a) analyzes the use of general function approximators in RL, covering neural network parameterization as a special case.
Such analyses typically depend on the Eluder dimension (Russo & Van Roy, 2013), which unfortunately can be exponentially large even for a simple neural network class (Dong et al., 2021) and thus makes the results statistically inefﬁcient for neural network parameterization.
Therefore, we raise the following question:
Can we design RL algorithms that can conduct provably efﬁcient exploration in structured environments with neural network parameterization?
Speciﬁcally, Our goal is to develop computational-efﬁcient algorithms whose sample efﬁciency does not depend on the Eluder dimension of neural networks for structured environments with neural network parameterization.
Our key insight is that, when the transition dynamics is captured by an energy-based model, we leverage the spectral decomposition of the kernel such that the challenge of distribution shift is characterized by the effective dimension of the kernel.
To illustrate this insight, we propose a new model called MDPs with neural dynamics , which allows neural network parameterization and captures various MDP models proposed in previous works, including the
KNR model (Kakade et al., 2020) and the linear MDP model (Jin et al., 2020).
We then propose an algorithm, namely, Exploration with Learnable Neural Features ( ELNF ), and show that ELNF is sample efﬁcient.
ELNF iteratively ﬁts the transition dynamics and reward functions with neural networks.
Upon ﬁtting the models, ELNF conducts exploration based on upper conﬁdence bounds (UCB) (Abbasi-Yadkori et al.
(2011)), which are obtained from the feature maps that correspond to the ﬁtted model.
We remark that the bonus in ELNF can be efﬁciently computed.
Contributions.
Our contribution is threefold.
First, we identify a class of models that incorporates
NN feature representation, which captures nonlinearity in the transition dynamics beyond the KNR and linear MDP model.
We also show that our proposed setting can generalize to models in previous works (Kakade et al.
(2020), Ren et al.
Second, we propose a new algorithm, namely
ELNF , which tackles our proposed MDPs with neural dynamics .
Our algorithm is computationally efﬁcient when we have an optimization oracle for the model estimation .
Third, we analyze the sample complexity of ELNF and show that ELNF is sample efﬁcient.
A key feature of ELNF is that the sample complexity of ELNF depends only on the covering number of neural network classes and does not depend on the corresponding Eluder dimension.
We highlight that our work is the ﬁrst to cover arbitrary NN classes with bounded log-covering numbers.
In contrast, previous research typically depends on the Eluder dimension (Russo & Van Roy, 2013) of the hypothesis class, which is exponentially large for simple neural network classes (Dong et al., 2021).
1. R ELATED W ORK
Our work is closely related to the line of research on provably efﬁcient exploration in the function approximation setting (Jiang et al., 2017; Jin et al., 2020;
Cai et al., 2019; Du et al., 2021; Uehara et al., 2021; Zhang et al., 2022a).
Such a line of research typically hinges on MDPs with a low-rank structure.
For instance, the study of linear MDPs (Jin et al., 2020; Cai et al., 2019) requires that the transition dynamics are linear in the known feature map.
In contrast, the feature maps are unknown in our setting and need to be estimated.
The study of low-rank MDPs (Jiang et al., 2017; Du et al., 2021; Uehara et al., 2021; Ren et al., 2022) is more closely aligned to our work in the sense that the feature map is unknown and needs to be estimated.
Jiang et al.
(2017) and Du et al.
(2021) require optimistic planning over the conﬁdence set of transition dynamics, which is computationally inefﬁcient.
Uehara et al.
(2021) and Ren et al.
(2022) propose an algorithm for low-rank MDP that is both computationally efﬁcient and sample efﬁcient.
Nevertheless, they only consider ﬁnite hypothesis classes, and require sampling from the stationary distribution of the MDP.
Our work is also related to the study of provably efﬁcient exploration with general function approximation (Wang et al., 2020; Jin et al., 2021a).
Nevertheless, previous results typically depend on the
Eluder dimension (Russo & Van Roy, 2013) of the hypothesis class, which is exponentially large for simple neural network classes (Dong et al., 2021).
Yang et al.
(2020) achieves sample-efﬁcient exploration based on the overparameterized neural networks (Simsek et al., 2021) as the function approximator.
However, their analysis hinges on the neural tangent kernel (NTK) and can not handle
NNs beyond NTK regime.
In contrast, our analysis can adapt to generic neural network classes.
Our work is also related to the analysis of model-based RL (Osband & Van Roy, 2014; Ayoub et al., 2020; Kakade et al., 2020) and representation learning (Ren et al., 2021; Nachum & Yang, 2021;
Zhang et al., 2022b).
The deﬁnition of our MDPs with neural dynamics generalizes that in Kakade et al.
and Ren et al. (2021).
In contrast to the KNR model in Kakade et al.
(2020), we can handle the inﬁnite neural network hypothesis class and do not require the nonlinear feature map to be known.
Ren et al.
(2021) require sampling from the posterior distribution of the hypothesis class, which is computational-inefﬁcient when the hypothesis class is large.
In addition, the sample complexity bound of Ren et al.
(2021) depends on the Eluder dimension of the feature map class, which is exponentially large for simple neural network classes (Dong et al., 2021).
In contrast, our sample complexity bound depends only on the neural network classes only through its capacity.
Our work is motivated by the complexity analysis of neural network classes.
Dong et al.
(2021) show that the Eluder dimension of one-layer neural network classes is exponentially large, suggesting that the previous analysis of RL algorithms based on the Eluder dimension (Russo & Van Roy, 2013) may not be applicable to neural networks.
1. N OTATION
For a vector v 2 R d , we deﬁne k v k 2 = (
P di =1 v 2 i ) 1 / 2 , where v i is the i -th element of v .
For a real-valued function f : X !
R , we deﬁne k f k 1 = max x 2 X | f ( x ) | .
For a vector-valued function f : X !
R d , we deﬁne k f k 1 , 2 = max x 2 X k f ( x ) k 2 .
For a sequence of real-valued functions r
[Equation]
R , we deﬁne k r k 1 = sup h 2 [ H ] ,x 2 X | r h ( x ) | .
We denote by N ( F , ✏ , k · k ) the ✏ -covering number of the function class
F with respect to the norm k · k , deﬁne H 1 ( F , ✏ ) = log
N ( F , ✏ , k · k 1 ) for a real-valued function class F , and deﬁne H 2 ( F , ✏ ) = log N ( F , ✏ , k · k 1 , 2 ) for a vector-valued function class F .
We further deﬁne [ n ] = { 1 , . . . , n } when n is an integer.
For a set C , we denote by  ( C ) the set of the distributions over C , and U ( C ) the uniform distribution over
C .
For g : X ! R and X n = { x 1 , . . .
, x n } ⇢ X , we deﬁne g [ X n ] =
( g ( x 1 ) , . . .
, g ( x n )) > .
P RELIMINARY
We consider an episodic MDP V ⇤ =
( S , A , H, P ⇤ , r ⇤ ) with a state space S 2 R d , an action space A , a horizon H , transition kernels
P ⇤ =
{P ⇤ h } Hh =1 , and reward functions r ⇤ =
{ r ⇤ h } Hh =1 .
We assume that the reward functions are bounded and deterministic, that is, k r ⇤ h k 1
[0 , 1] for all h 2 [ H ] .
We also assume that the action space is ﬁnite, that is, |A| < 1 .
The agent iteratively interacts with the environment as follows.
At the beginning of each episode, the agent determines a policy ⇡ = { ⇡ h } Hh =1 , where ⇡ h : S !
( A ) for any h
[ H ] .
Without loss of generality, we assume that the initial state is ﬁxed to s init 2 S across all episodes.
At the h -th step, the agent receives a state s h and takes an action a h following a h ⇠ ⇡ h ( · | s h ) .
Subsequently, the agent receives a reward
r ⇤ h
( s h , a h ) and the next state following s h +1 ⇠ P ⇤ h
+
( · | s h , a h ) .
The episode ends after the agent receives the last state s H +1 .
For a given policy ⇡ = { ⇡ h } Hh =1 , where ⇡ h : S !
( A ) for any h 2 [ H ] , we deﬁne the value function V ⇡ h and the Q -function Q ⇡ h for any h 2 [ H ] as
[Equation]
Here the expectation E ⇡ , P ⇤ [ · ] in (2.1) is taken with respect to s i +1 ⇠ P ⇤ i ( · | s i , a i ) and a i ⇠ ⇡ i ( · | s i ) for i
[Equation]
h, h + 1 , . . .
, H } .
For convenience, we deﬁne V ⇡ H +
( s ; r, P ) =
for any state s 2 S , reward function r , transition kernel
P and policy ⇡ .
For simplicity, we deﬁne the expected total reward J ( ⇡ ; r ⇤ , P ⇤ ) as J ( ⇡ ; r ⇤ , P ⇤ )
=
V ⇡ 1 ( s init ; r ⇤ , P ⇤ ) .
We have the following Bellman equations for any ( s, a, h ) 2 S ⇥
A ⇥ [ H ] and any policy ⇡ ,
[Equation]
where h · , · i A is the inner product over R A , and E P ⇤ [ · ] is taken with respect to s h +1 ⇠ P ⇤ h
( · | s h , a h ) .
The goal of RL is to ﬁnd a policy ⇡ ⇤ that maximizes the expected total reward.
Speciﬁcally, for the episodic MDP V ⇤ =
( S , A , H, P ⇤ , r ⇤ ) , We deﬁne ⇡
⇤ 2 argmax ⇡ J ( ⇡ ; r ⇤ , P ⇤ ) as an optimal policy.
Correspondingly, we deﬁne the optimal Q -function Q ⇤ h and the optimal value function V ⇤ h as
Q ⇤ h
( s, a ; r ⇤ , P ⇤ )
=
Q ⇡ ⇤ h
( s, a ; r ⇤ , P ⇤ ) and V ⇤ h ( s ; r ⇤ , P ⇤ )
=
V ⇡ ⇤ h
( s ; r ⇤ , P ⇤ ) for any ( s, a ) 2 S ⇥ A .
M ARKOV D ECISION P ROCESSES WITH N EURAL D YNAMICS
In this paper, our goal is to develop a provably efﬁcient algorithm for RL problems adapted with large feature space, such as neural networks (NNs).
To this end, we introduce the MDPs with neural dynamics, whose reward and transition dynamics are parameterized by NNs.
Motivation.
Our deﬁnition is motivated by the kernelized nonlinear regulator (KNR).
In a KNR model, the transition kernel takes the following form,
[Equation]
where  ⇤ h is a known nonlinear feature map.
Former research proposes sample-efﬁcient algorithms for such a model.
Although such a KNR setting empowers sample efﬁcient RL (Kakade et al.
, it is relatively restrictive in the following two aspects.
First, the feature map  ⇤ h and the expected reward
r ⇤ h are known a priori.
Second, the model only imposes nonlinearity on ( s h , a h ) via the known feature map, while the conditional expectation of the next state given s h , a h is a linear function of
⇤ h
( s h , a h ) .
In other words, when  ⇤ h is known, the transition dynamics can be recovered via linear system identiﬁcation methods such as ridge regression.
To generalize the KNR model, we interpret (3.1) as an energy-based model.
More speciﬁcally, we can write the transition of the MDP in (3.1) as
[Equation]
where the energy function E ( s h
+1 , s h , a h ) is deﬁned as
[Equation]
Here (3.2) omits a normalization factor, which is a function of ( s h , a h ) .
We generalize this model and impose nonlinearity on the next state s h +1 by substituting a nonlinear feature map
⇤ h
+1 ( s h +1 ) for s h +1 in (3.3).
Such a generalization allows us to incorporate the nonlinearity of the next state in the model.
In addition, we assume that the nonlinear feature maps  ⇤ h
and ⇤ h
+1 are unknown and need to be estimated from pre-speciﬁed feature classes  and , which for example, can be two classes of NNs.
We further assume that the expected reward r ⇤ is unknown and needs to be estimated from the reward function class R .
We formalize our generalization in the following deﬁnition.
Deﬁnition 3.1 (MDPs with Neural Dynamics) .
An episodic MDP ( S , A , H, P ⇤ , r ⇤ ) is an MDP with neural dynamics if its reward function r ⇤ =
{ r ⇤ h } Hh =1 belongs to a reward function class R , which is a known function class that consists of NNs, and the transition kernel of the MDP P ⇤ =
{P ⇤ h } Hh =1 takes the following form,
[Equation]
Here  ⇤
h 2  : R d ⇥ A !
R m and ⇤ h
+1: R d !
R m are two unknown feature maps, and  , are two known feature map classes that consist of NNs.
We denote by M the set of all the transition kernels that take the form of (3.4) , and let X 2 R m denote the union of the image spaces of the feature maps, namely,
[Equation]
Generality of Deﬁnition 3.1.
We remark that Deﬁnition 3.1 is a signiﬁcant generalization of stochastic nonlinear systems beyond KNR.
For instance, when ⇤ h +1 ( s h +1 )
=
s h +1 , the transition kernel in Deﬁnition 3.1 takes the following form,
[Equation]
which is the transition kernel in Ren et al.
Therefore, we recover the model in Ren et al.
(2021) when ⇤ h +1 is known to be the identity map and the reward function is known.
Moreover, the transition kernel deﬁned in (3.4) also includes a class of nonlinear dynamics satisfying
[Equation]
where S ✓ R m , ⇤
[Equation]
+1 : R m !
R m , the determinant of the Jacobian matrix of ⇤
h +1 is a constant, and ✏ h is a Gaussian noise.
Our model signiﬁcantly generalizes such a model by allowing a possibly noninvertible feature map ⇤ h
+1 .
Relationship with Kernelized Linear MDP.
Recall that K ( x, y ) =
exp( k x  y k 22 / 2) is also known as the Gaussian RBF kernel, which induces a reproducing kernel Hilbert space (RKHS) deﬁned on R m
(Rahimi et al., 2007).
(See Appendix §D for a brief introduction of RKHS.)
Intuitively, K ( x, y ) measures the proximity between x and y in the kernel space.
From this perspective, the transition kernel in (3.4) speciﬁes the next state s h +1 by measuring the proximity of the representations
⇤ h
( s h , a h ) and ⇤ h
+
( s h +1 ) .
Besides, since K ( x, y ) can be written as h k ( x ) , k ( y )
i H , where the feature map of the RKHS k is deﬁned as k ( x ) =
K ( x, · ) , and h · , · i H is the inner product of the RKHS respectively.
Thus, (3.4) can be equivalently written as
[Equation]
where Z ⇤ h
( s h , a h ) is the normalization factor in (3.4).
Thus, when Z ⇤ h is known, our model can be regarded as an RKHS extension of the linear MDP model (Jin et al., 2020).
This is the case when ⇤ is the identity maps but unknown to the learner, and Z ⇤ h
( s h , a h ) becomes a constant (Ren et al., 2021).
See Appendix §D for more details of the relationship between the model in (3.4) and RKHS.
Role of NNs in Our Model.
We would like to remark that the model speciﬁed in Deﬁnition 3.1 is not restricted to NNs.
In fact, the deﬁnition only requires proper function classes of the reward function, representations of ( s h , a h ) and s h
+1 , namely, R ,  , and .
Thus, our model can also be deﬁned for other function approximators such as polynomial spline (Unser et al., 1993), classiﬁcation and regression tree (Syrgkanis & Zampetakis, 2020).
Meanwhile, as we will see in the sequel, both our algorithm and the theoretical results do not hinge on NNs and can employ general function classes with bounded capacity.
Here we call our model neural dynamics in order to highlight that our work is the ﬁrst one that covers arbitrary NN classes with bounded log-covering numbers .
A LGORITHM
In this section, we introduce an algorithm for solving MDPs with neural dynamics in the online setting.
We ﬁrst introduce the motivation of the algorithm, and then introduce the procedure in detail.
Motivation.
To strike a balance between exploration and exploitation, our algorithm follows the principle of Optimism in the Face of Uncertainty (Lattimore & Szepesvári, 2020).
When we know the true feature maps {  ⇤ h } Hh =1 , we can apply kernel LSVI (Yang et al., 2020) to construct the exploration bonus since the energy-base transition admits a kernel structure.
(See §3 for the details.)
However, in
MDPs with neural dynamics, we do not know {  ⇤ h
Hh =1 .
A straightforward solution for handling the unknown feature maps is to learn the feature maps from the data we collect and construct the bonus based on the learned features.
However, the bonus constructed by the learned features might be invalid since the learned features have errors.
We handle the error in the learned feature by purposefully taking uniform actions when exploring the environment.
Such a sampling scheme gives us more diverse data for model estimation.
Based on this motivation, we design an iterative algorithm that outputs a policy after N iterations.
In particular, in each iteration n 2 [ N ] , our algorithm performs the following four steps: (i) sampling new data from the environment, (ii) estimating the model via maximum likelihood estimation, (iii) constructing exploration incentives using the features of the learned model, and (iv) updating the online policy for exploration via planning on the learned model.
Sampling Scheme.
As we mentioned in §3, the transition of MDPs with neural dynamics can be written as an energy-based model and admits the Gaussian RBF kernel.
To exploit the kernel structure in the transition, we explore the environment using the exploration bonus induced by the Gaussian
RBF kernel and the feature maps learned from the data, which is motivated by Yang et al.
However, since the bonus is not induced by the true underlying feature, it might fail to indicate the most uncertain state-action pairs for exploration.
To mitigate such an issue, we combine the uniform policy, which samples action from the uniform distribution over the action space, with the optimistic policy during the sampling procedure.
Intuitively, such a sampling scheme provides a wider coverage over the state-action space and better explores the environment.
[-( s h , a h ) at steps h 
 [ H ] , which leads to-]To simplify the presentation in the main text, we present the sketch of the sampling scheme as follows.
The rigorous presentation of the sampling scheme for the boundary case is deferred to Appendix §B.
In the n -th iteration of our algorithm, given the previously collected dataset D n  1 h,i for i 2 { 0 , 1 , 2 } and h 2 [ H ] , we interact with the MDP following the policy ⇡ n = { ⇡ nh } Hh =1 and obtain the new dataset D nh,i for i 2 { 0 , 1 , 2 } and h 2 [ H ] .
Speciﬁcally, for any h 2 {  1 , . . .
, H } , we start from the initial state s 1 and choose the action a ¯ h ⇠ ⇡ n ¯ h ( · | s ¯ h ) in the ¯ h -th step when ¯ h 2 { 1 , · · · , h } , and choose the action a ¯ h ⇠ U ( A ) when ¯ h 2 { h + 1 , h + 2 } .
Here U ( A ) is the uniform distribution over the action space A .
By following such a procedure, we obtain the following trajectory,
[Equation]
[Equation]
where s 1 = s init .
Then, we label the obtained trajectory as follows, s nh
+ i,i = s h
+ i , a nh + i,i = a h + i , r nh
+ i,i = r h + i , ¯ s nh + i,i =
s h
+ i +1 , (4.2) for any i 2 { 0 , 1 , 2 } . We then update the dataset D nh,i by D nh,i = D n  1 h,i [  ( s nh,i , a nh,i , r nh,i , ¯ s nh
+ i,i ) for any i 2 { 0 , 1 , 2 } and h 2 [ H ] .
We use i in (4.2) to indicate how many steps of the uniform policy we need to execute to obtain such a dataset.
Intuitively, the dataset with a bigger i has a better coverage over the state-action space S ⇥
A .
We summarize the sampling scheme in Algorithm 1.
See Figure 1 in Appendix §B for an illustration of the sampling scheme, and see Algorithm 3 in Appendix §B for the formal presentation of Algorithm 1.
Updating the datasets 6: Return: Datasets {D nh,i } h
= H
[Equation]
i
[Equation]
[Equation] [Equation]
i
[Equation] [Equation]
Model Estimation.
To estimate the model, we solve the following optimization problems,
[Equation]
Here R and M are the reward function class and the transition kernel class deﬁned in Deﬁnition 3.1.
We denote by  nh and nh +1 the feature maps that correspond to the transition kernel b
P nh estimated in (4.4).
To simplify our analysis, we assume that there exists an oracle that returns the global minimum of the optimization problems (4.3) and (4.4).
Similar assumption also arises in the previous study of RL (Fan et al., 2020; Kakade et al., 2020;
Uehara et al., 2021; Jin et al., 2021a).
When the normalization factor in (3.5) is a constant, (4.4) can be easily implemented since it is equivalent with
[Equation]
Remark 4.1 (Transition Estimation) .
We would like to remark that the method for estimating the transition is not restricted to maximum likelihood estimation (MLE).
Methods including variational autoencoder (Kingma & Welling, 2013), score matching (Hyvärinen & Dayan, 2005) can also be used for transition estimation.
Our sample complexity bound holds for any transition estimator whose total variance error has an upper bound.
Exploration Bonus.
The transition kernel in Deﬁnition 3.1 is closely related to the radial basis function (RBF) kernel.
In the sequel, we deﬁne the bonuses for exploration and update the policy based on such bonuses.
Speciﬁcally, for a ﬁxed feature map  : S !
R , we deﬁne the Gram matrix
K nh [  ] and the function k nh [  ] : S ⇥ A !
[Equation]
as follows
[Equation]
where { ( s ⌧ h, 1 , a ⌧ h, 1 , r ⌧ h, 1 , ¯ s ⌧ h, 1 ) } n ⌧ =1 2 D nh, 1 .
We then deﬁne the bonus u nh as follows,
[Equation]
where
[Equation]
Here  > 0 and  > 0 are the tuning parameters.
We remark that the form of the bonus in (4.5) aligns with the bonus in other previous works that use kernel functions for function approximation (Srinivas et al., 2009;
Yang et al., 2020).
Remark 4.2 (Dependency of Rewards on Features.) .
Here we do not require that the reward depends on the feature in the transition kernel, which is different from the literature in linear MDPs (Cai et al., 2019; Jin et al., 2020).
Common sense seems to dictate that we can not characterize the uncertainty using the feature without such a dependency.
However, the estimation error of the reward estimators b r nh in the empirical measure can be bounded from the above by the property of the least square estimator.
Therefore, the uncertainty of r ⇤ h in the distribution induced by any new policy, such as ⇡ ⇤ , can be bounded from the above by the distribution shift, which is characterized by u nh  1 .
Such an observation allows us to characterize the uncertainty of the reward estimator even when the reward does not depend on the feature in the transition kernel.
Policy Update.
We update the policy ⇡ n +1 by setting it as the optimal policy of the learned model, which can be efﬁciently computed by dynamic programming.
Due to the space limit, we defer the details of the planning algorithm to Appendix §B. We remark that we can also apply other modelbased algorithms, including Dyna (Sutton, 1991) and Gradient-Aware Model-based Policy Search (D’Oro et al., 2020), to compute the optimal policy of the learned model, and the suboptimality of the output of Algorithm 2 is bounded when the error of the planning oracle is bounded.
Remark 4.3 (Computational Efﬁciency) .
Our algorithm is oracle efﬁcient in the sense that our algorithm is computationally efﬁcient given an optimization oracle for model estimation, which also appears in the previous study (Fan et al., 2020; Kakade et al., 2020; Uehara et al., 2021).
More speciﬁcally, the bonus and the policy in each iteration can be efﬁciently computed by (4.5) and
Algorithm 4 in the appendix.
The existing literature on general function approximation requires either global optimism over the conﬁdence set (Kakade et al., 2020; Jin et al., 2021a) or posterior sampling over the hypothesis set (Ren et al., 2021), which can not be computed efﬁciently. Algorithm
1: Input: Failure probability  > 0 , tuning parameters  ,  > 0 .
{+2: Initialize: Set ⇡ 1 = { ⇡ 1 h } Hh =  1 , where ⇡ 1 h ( · | s ) = U ( A ) for any ( s, h ) 2 S ⇥ [ H ] , and set D 0 h,i = ; for all ( h, i ) 
 [ H ] ⇥ { 0 , 1 , 2 }+} .
3: for n = 1 , . . . , N do 4: Set {D nh,i } h = H,i =2 h
=1 ,i =0 by applying Algorithm 1 (Sampling Scheme) with the policy ⇡ n and the datasets {D n  1 h,i } h = H,i =2 h =1 ,i =0 as the input.
. Sampling
Set { b r nh } Hh =1 and { b
P nh } Hh =1 as in (4.3) and (4.4), respectively. . Model estimation
Set {  nh } Hh =1 and { nh +1 } Hh =1 as the feature maps corresponding to
[Equation]
=1 . Set { u nh } Hh =1 as in (4.5).
. Feature estimation and bonus construction
Set ⇡ n +1 by applying Algorithm 4 (Planning Algorithm) in the appendix with the learned model { b r nh } Hh =1 , {P nh } Hh =1 , and the bonuses { b u nh } Hh =1 as the input.
9: end for 10: Return:
[Equation]
T HEORY
In this section, we present the analysis of ELNF .
We ﬁrst present the boundedness assumption on the model.
Assumption 5.1 (Boundedness of Model) .
We assume that the state space S is a bounded set of R d , and the Lebesgue measure of S is an absolute constant.
We also assume that max { k  ( s, a ) k 2 , k ( s ) k 2 }  R for all ( s, a,  , ) 2 S ⇥ A ⇥  ⇥ .
We further assume that 0  r ( s, a )  1 for any ( s, a, r ) 2 S ⇥ A ⇥ R .
Since S is bounded, Assumption 5.1 is a reasonable regularity condition on the model.
Such a regularity assumption is standard and is also assumed in the previous works (Cai et al., 2019; Jin et al., 2020; 2021b).
Next, we introduce the following assumption, which characterizes the complexity of the NN classes.
Assumption 5.2 (Decay Rate of Covering Number) .
There exists constants C net > 0 and   0 that only depend on ( R ,  , ) , such that
[Equation]
In Assumption 5.2,  characterizes the complexity of the NN class by quantifying the growth rate of the covering number when the covering radius ✏ decays.
We remark that previous research bounds the covering number of NN classes from the above at the same scale as Assumption 5.2.
For example,
Schmidt-Hieber (2020) and Chen et al.
(2019) show that NN classes with speciﬁc structures satisfy
Assumption 5.2 with  = 0 .
See Lemmas C.3 and C.6 in Appendix §C for the details.
Theorem 5.3 (Sample Complexity of ELNF ) .
We assume that Assumption 5.2 holds with  < 1 / 2 , and we can obtain the exact solution to the optimization problems (4.3) and (4.4) .
We set the tuning parameters  and  as
[Equation]
in ELNF , where m is the dimension of the image of the feature maps, C 0 , C 00 are constants that only depend on the regularity parameters in Assumption 5.1.
Under Assumption 5.1 and 5.2, for the policy b ⇡ returned by ELNF , it holds with probability at least 1   that
[Equation]
Here C is a constant that only depends on the dimension m , the bound of the feature maps R , and
C net in Assumption 5.2, and ⇠ =
(log(48 HRN/  )) 5 / 2 is a logarithmic factor.
Proof.
See Appendix §E for a detailed proof.
In Theorem 5.3,  is the regularization parameter that trades off between bias and variance, and  is the uncertainty coefﬁcient, which scales with  and N .
We remark that our analysis is not restricted to
NN classes, and can be extended to other bounded function classes with bounded covering numbers.
We further remark that m in Theorem 5.3 is the dimension of the image of the feature maps, which can be much smaller than the dimension of the state.
Moreover, in Appendix §E, we show that the suboptimality bound in Theorem 5.3 reduces to e
O ( d eff N (2   1) / (2+2  ) ) in terms of N , where d eff is the effective dimension in Deﬁnition E.4 in the appendix.
Such a bound connects the sample efﬁciency of ELNF to the effective dimension of the
RKHS and the covering number of NN classes.
We further remark that when  = 0 in Theorem 5.3, the suboptimality bound is sublinear in N , which aligns with the previous theoretical research.
A detailed comparison with the related work can be found in Appendix §A.
In this section, we sketch the proof of Theorem 5.3 and highlight the technique that allows us to remove the dependency on the Eluder dimension of NN classes. Since the policy b ⇡ returned by
Algorithm 2 ( ELNF ) is the mixture of ⇡ 1 , . . .
, ⇡ N , we can decompose the suboptimality as
[Equation] | [Equation] [Equation]
Here Term (a) is the out-of-sample estimation error of the estimated value J , Term (b) is the error of the planning algorithm, and Term (c) is the in-sample estimation error of the estimated value J . Since the bonus u n captures the uncertainty of the estimated model with high probability, and the planning algorithm ﬁnds the optimal policy of the learned model, Term (a) and Term (b) are small with high probability, which is shown in the following lemma. Lemma 6.1 (Informal) . We deﬁne ⇣ = p
C 2 log(20 HRN/  ) N  / (1+  ) , where C 2 is a constant that depends on the regularity parameters and the parameters of the NN class.
Following the same condition of Theorem 5.3, we have Term(a)
 8 H |A| ⇣ p N holds with probability at least 1   . We also have Term(b)  0 when we use Algorithm 4 for planning.
Proof. This can be directly proved by Lemma E.5 and Lemma E.6. See Appendix §E for the details.
The remaining analysis is to connect Term (c) with the complexity measure of the model class. Former research (Wang et al., 2020; Jin et al., 2021a) that allows neural network parameterization quantiﬁes the uncertainty by the level set, and the Eluder dimension naturally appears when we telescope the in-sample error. To remove the dependency on the Eluder dimension, we want to quantify the uncertainty by the bonus deﬁned by the true feature. However, since the true feature is unknown, we can only quantify the uncertainty using the learned feature.
Inspired by Uehara et al.
(2021), we obtain the following lemma, which connects the bonus deﬁned by the learned feature with the bonus deﬁned by the true unseen feature.
Lemma 6.2 (Bonus Equivalence for the True Model, Informal) .
Following the same condition with Theorem 5.3, we have
[Equation] holds for any policy ⇡ and n  2 with high probability, where the bonus deﬁned by the true feature u ⇤ ,n = { u ⇤ ,nh } Hh =1 is deﬁned in Lemma I.1.
Proof. This is Lemma I.2. See Appendix §I.5 for the details. We remark that we cannot obtain the above lemma by directly applying the technique in Uehara et al.
(2021) since they only consider the ﬁnite-dimensional inner product. Using the lemma above, we have Term
[Equation]
Finally, we show that the sum of the bonuses deﬁned by the true feature can be bounded by the effective dimension of the RKHS induced by the noise instead of the Eluder dimension of NN classes.
We conclude the proof of Theorem 5.3 by combining the upper bounds of Term (a), Term (b), and Term (c).
Removing Dependency on Eluder Dimension.
The existing literature on RL using general function approximators relies on the Eluder dimension when bounding the regret or the suboptimality (Wang et al., 2020; Jin et al., 2021a; Ren et al., 2021), which is exponentially large for a simple neural network class (Dong et al., 2021). However, we can remove such dependency in MDPs with neural dynamics.
Our key insight is that we can regard MDPs with neural dynamics as kernel MDPs (Yang et al., 2020) whose feature is the composite map of the neural network and the feature map of the RKHS since the energy-base transition admits a kernel structure, which is shown in §3. Therefore, we can characterize the effect of the distribution shifts by the bonus deﬁned by the true feature, whose sum is bounded by the effective dimension of the RKHS instead of the Eluder dimension of the NN class, without knowing the true feature. Role of Uniform Policy in Proof.
The uniform policy in the sampling scheme enables us to bound the inﬂuence of the distribution shift and show that the bonuses deﬁned by the learned feature are valid uncertainty quantiﬁcation. In order to show that the estimated value deﬁned by the learned bonus is almost optimistic (Lemma E.5) for any policy, we need to bound the inﬂuence of the distribution shift. The data that we obtain from the uniform sampling has better coverage over S ⇥ A , and the bonuses we construct with these data quantify the uncertainty with the presence of the distribution shift. Therefore, we take uniform actions when exploring the environment to obtain a valid uncertainty quantiﬁcation without knowing the true feature, and |A| 2 in the suboptimality bound is the price paid for the uniform sampling.