A
G E N E R A L
F R A M E W O R K
F O R
S A M P L EE F F
I C
I
E N T
F U N C T
I O N
A P P R O X
I M AT
I
O N
I N
R E
I N F O R
C
E M E N T
L E A R N
I N G
Anonymous authors
Paper under double-blind review
A B S T R AC T
With the increasing need for handling large state and action spaces, general function approximation has become a key technique in reinforcement learning (RL).
In this paper, we propose a general framework that unifies model-based and model-free
RL, and an Admissible Bellman Characterization (ABC) class that subsumes nearly all Markov Decision Process (MDP) models in the literature for tractable RL.
We propose a novel estimation function with decomposable structural properties for optimization-based exploration and the functional eluder dimension as a complexity measure of the ABC class.
Under our framework, a new sample-efficient algorithm namely OPtimization-based ExploRation with Approximation (OPERA) is proposed, achieving regret bounds that match or improve over the best-known results for a variety of MDP models.
In particular, for MDPs with low Witness rank, under a slightly stronger assumption, OPERA improves the state-of-the-art sample complexity results by a factor of dH .
Our framework provides a generic interface to design and analyze new RL models and algorithms.
I N T RO D U C T I O N
Reinforcement learning (RL) is a decision-making process that seeks to maximize the expected reward when an agent interacts with the environment (Sutton & Barto, 2018).
Over the past decade,
RL has gained increasing attention due to its successes in a wide range of domains, including Atari games (Mnih et al., 2013), Go game (Silver et al., 2016), autonomous driving (Yurtsever et al., 2020), Robotics (Kober et al., 2013), etc.
Existing RL algorithms can be categorized into valuebased algorithms such as Q-learning (Watkins, 1989) and policy-based algorithms such as policy gradient (Sutton et al., 1999).
They can also be categorized as a model-free approach where one directly models the value function classes, or alternatively, a model-based approach where one needs to estimate the transition probability.
Due to the intractably large state and action spaces that are used to model the real-world complex environment, function approximation in RL has become prominent in both algorithm design and theoretical analysis.
It is a pressing challenge to design sample-efficient RL algorithms with general function approximations.
In the special case where the underlying Markov Decision Processes (MDPs) enjoy certain linear structures, several lines of works have achieved polynomial sample complexity and/or √ T regret guarantees under either model-free or model-based RL settings.
For linear MDPs where the transition probability and the reward function admit linear structure, Yang & Wang (2019) developed a variant of Q -learning when granted access to a generative model, Jin et al.
(2020) proposed an LSVI-UCB algorithm with a (cid:101) O ( √ d 3 H 3 T ) regret bound and Zanette et al.
(2020a) further extended the MDP model and improved the regret to (cid:101) O ( dH √ T ) .
Another line of work considers linear mixture MDPs Yang & Wang (2020); Modi et al.
; Jia et al.
Zhou et al.
(2021a), where the transition probability can be represented by a mixture of base models.
In Zhou et al.
(2021a), an (cid:101) O ( dH √ T ) minimax optimal regret was achieved with weighted linear regression and a Bernstein-type bonus.
Other structural MDP models include the block MDPs (Du et al., 2019) and FLAMBE (Agarwal et al., 2020b) 1 , to mention a few.
In a more general setting, however, there is still a gap between the plethora of MDP models and sample-efficient RL algorithms that can learn the MDP model with function approximation.
The question remains open as to what constitutes minimal structural assumptions that admit sampleefficient reinforcement learning.
To answer this question, there are several lines of work along this direction.
Russo & Van Roy (2013); Osband & Van Roy (2014) proposed an structural condition named eluder dimension, and Wang et al.
(2020) extended the LSVI-UCB for general linear function classes with small eluder dimension.
Another line of works proposed low-rank structural conditions, including Bellman rank (Jiang et al., 2017;
Dong et al., 2020) and Witness rank (Sun et al., 2019).
Recently, Jin et al.
(2021) proposed a complexity called Bellman eluder (BE) dimension, which unifies low Bellman rank and low eluder dimension.
Concurrently, Du et al.
(2021) proposed Bilinear
Classes, which can be applied to a variety of loss estimators beyond vanilla Bellman error.
Very recently, Foster et al.
(2021) proposed Decision-Estimation Coefficient (DEC), which is a necessary and sufficient condition for sample-efficient interactive learning.
To apply DEC to RL, they proposed a RL class named Bellman Representability, which can be viewed as a generalization of the Bilinear
Class.
Nevertheless, Sun et al.
(2019) is limited to model-based RL, and Jin et al.
(2021) is restricted to model-free RL.
The only frameworks that can unify both model-based and model-free RL are Du et al.
(2021) and Foster et al.
, but their sample complexity results when restricted to special
MDP instances do not always match the best-known results.
Viewing the above gap, we aim to answer the following question:
Is there a unified framework that includes all model-free and model-based RL classes while maintaining sharp sample efficiency?
In this paper, we tackle this challenging question and give a nearly affirmative answer to it.
We summarize our contributions as follows:
• We propose a general framework called Admissible Bellman Characterization (ABC) that covers a wide set of structural assumptions in both model-free and model-based RL, such as linear MDPs, FLAMBE, linear mixture MDPs, kernelized nonlinear regulator (Kakade et al., 2020), etc.
Furthermore, our framework encompasses comparative structural frameworks such as the low Bellman eluder dimension and low Witness rank.
• Under our ABC framework, we design a novel algorithm, OPtimization-based ExploRation with Approximation (OPERA), based on maximizing the value function while constrained in a small confidence region around the model minimizing the estimation function.
• We apply our framework to several specific examples that are known to be not sample-efficient with value-based algorithms.
For the kernelized nonlinear regulator (KNR), our framework is the first general framework to derive a √ T regret-bound result.
For the witness rank, our framework yields a sharper sample complexity with a mild additional assumption compared to prior works.
We visualize and compare prevailing sample-efficient RL frameworks and ours in Figure 1.
We can see that both the general Bilinear Class and our ABC frameworks capture most existing MDP classes, including the low Witness rank and the KNR models.
Notation.
For a state-action sequence s 1 , a 1 , . . .
, s H in our given context, we use J h :
= σ ( s 1 , a 1 , . . .
, s h ) to denote the σ -algebra generated by trajectories up to step
h ∈
[ H ] .
Let π f denote the policy of following the max- Q strategy induced by hypothesis f .
When f = f
i we write π f i as π i for notational simplicity.
We write s h ∼ π to indicate the state-action sequence are generated by step h ∈
[ H ] by following policy π ( · | s ) and transition probabilities P ( · | s, a ) of the underlying MDP model M .
We also write a h ∼ π to mean a h ∼ π ( · | s h ) for the h -th step.
Let ∥ · ∥ 2 denote the ℓ 2 -norm and ∥ · ∥ ∞ the ℓ ∞ -norm of a given vector.
Other notations will be explained at their first appearances.
P R E L I M I NA R I E S
We consider a finite-horizon, episodic Markov Decision Process (MDP) defined by the tuple M =
( S , A , P , r, H ) , where S is the space of feasible states, A is the action space.
H is the horizon in each episode defined by the number of action steps in one episode, and P :=
{ P h } h ∈ [ H ] is defined for every h ∈
[ H ] as the transition probability from the current state-action pair ( s h , a h ) ∈ S × A to the next state s h +1 ∈ S .
We use r h ( s, a ) ≥ 0 to denote the reward received at step
h ∈
[ H ] when taking action a at state s and assume throughout this paper that for any possible trajectories, (cid:80)
Hh =1 r h ( s h , a h ) ∈ [0 , 1] .
A deterministic policy π is a sequence of functions { π h : S
(cid:55)→ A} h ∈ [ H ] , where each π h specifies a strategy at step h .
Given a policy π , the action-value function is defined to be the expected cumulative rewards where the expectation is taken over the trajectory distribution generated by { ( P h ( · | s h , a h ) , π h ( · | s h )) } h ∈
[ H ] as
[Equation]
Similarly, we define the state-value function for policy π as the expected cumulative rewards as
[Equation]
We use π ∗ to denote the optimal policy that satisfies V π ∗ h ( s ) = max π V πh ( s ) for all s ∈ S (Puterman, 2014).
For simplicity, we abbreviate V π ∗ h as V ∗ h and Q π ∗ h as Q ∗ h .
Moreover, for a sequence of value functions { Q h } h ∈
[ H ] , the Bellman operator at step h is defined as:
[Equation]
We also call Q h − ( T h Q h +1 )
the Bellman error (or Bellman residual).
The goal of an RL algorithm is to find an ϵ -optimal policy such that V π 1 ( s 1 ) − V ∗ 1 ( s 1 ) ≤ ϵ .
For an RL algorithm that updates the policy π t for T iterations, the cumulative regret is defined as
[Equation]
Hypothesis Classes.
Following Du et al.
(2021), we define the hypothesis class for both model-free and model-based RL.
Generally speaking, a hypothesis class is a set of functions that are used to estimate the value functions (for model-free RL) or the transitional probability and reward (for model-based RL).
Specifically, a hypothesis class F on a finite-horizon MDP is the Cartesian product of H hypothesis classes F :=
F 1 × . . .
× F H in which each hypothesis f = { f h } h ∈
[ H ] ∈ F can be identified by a pair of value functions { Q f , V f } =
{ Q h,f , V h,f } h ∈ [ H ] .
Based on the value function pair, it is natural to introduce the greedy policy π h,f ( s ) =
arg max a ∈A Q h,f ( s, a ) at each step
h ∈
[ H ] , and the corresponding π f ( s ) as the sequence of time-dependent policies { π h,f } H − 1 h =0 .
An example of a model-free hypothesis class is defined by a sequence of action-value function { Q h,f } h ∈
[ H ] .
The corresponding state-value function is given by:
[Equation]
In another example that falls under the model-based RL setting, where for each hypothesis f ∈ F we have the knowledge of the transition matrix P f and the reward function r f .
We define the value function Q h,f corresponding to hypothesis f as the optimal value function following M f :=
( P f , r f ) :
[Equation]
We also need the following realizability assumption that requires the true model M f ∗ (model-based
RL) or the optimal value function f ∗ (model-free RL) to belong to the hypothesis class F .
Assumption 1 (Realizability) .
For an MDP model M and a hypothesis class F , we say that the hypothesis class F is realizable with respect to M if there exists a f ∗ ∈ F such that for any h ∈ [ H ] ,
Q ∗ h ( s, a ) =
Q h,f ∗ ( s, a ) .
We call such f ∗ an optimal hypothesis .
This assumption has also been made in the Bilinear Classes (Du et al., 2021) and low Bellman eluder dimension frameworks (Jin et al., 2021).
We also define the ϵ -covering number of F under a well-defined metric ρ of a hypothesis class F :
Definition 2 ( ϵ -covering Number of Hypothesis Class) .
For any ϵ > 0 and a hypothesis class F , we use N F ( ϵ ) to denote the ϵ -covering number, which is the smallest possible cardinality of (an ϵ -cover)
F ϵ such that for any f ∈ F there exists a f ′ ∈ F ϵ such that ρ ( f, f ′ ) ≤ ϵ .
Functional Eluder Dimension.
We proceed to introduce our new complexity measure, functional eluder dimension , which generalizes the concept of eluder dimension firstly proposed in bandit literature (Russo & Van Roy, 2013; 2014).
It has since become a widely used complexity measure for function approximations in RL (Wang et al., 2020; Ayoub et al., 2020; Jin et al., 2021; Foster et al., 2021).
Here we revisit its definition:
Definition 3 (Eluder Dimension) .
For a given space X and a class F of functions defined on X , the eluder dimension dim E ( F , ϵ ) is the length of the existing longest sequence x 1 , . . .
, x n ∈ X satisfying for some ϵ ′ ≥ ϵ and any 2 ≤ t ≤ n , there exist f 1 , f 2 ∈ F such that (cid:113)(cid:80)
[Equation]
≤ ϵ
[Equation]
while | f 1 ( x t ) − f 2 ( x t ) | > ϵ ′ .
The eluder dimension is usually applied to the state-action space X = S
× A and the corresponding value function class F : S ×
A → R (Jin et al., 2021; Wang et al., 2020).
We extend the concept of eluder dimension as a complexity measure of the hypothesis class, namely, the functional eluder dimension , which is formally defined as follows.
Definition 4 (Functional Eluder Dimension) .
For a given hypothesis class F and a function G defined on F × F , the functional eluder dimension (FE dimension) dim FE ( F , G, ϵ ) is the length of the existing longest sequence f 1 , . . .
, f n ∈ F satisfying for some ϵ ′ ≥ ϵ and any 2 ≤ t ≤ n , there exists g ∈ F such that (cid:113)(cid:80) t − 1 i
=1 ( G ( g, f i )) 2 ≤ ϵ ′ while | G ( g, f t ) | > ϵ ′ .
Function G is dubbed as the coupling function .
The notion of functional eluder dimension introduced in Definition 4 is generalizable in a straightforward fashion to a sequence G :=
{ G h } h ∈ [ H ] of coupling functions: we simply set dim FE ( F , G, ϵ ) =
max h ∈ [ H ] dim FE ( F , G h , ϵ ) to denote the FE dimension of { G h } h ∈ [ H ] .
The Bellman eluder (BE) dimension recently proposed by (Jin et al., 2021) is in fact a special case of FE dimension with a specific choice of coupling function sequence.
3 As will be shown later, our framework based on FE dimension with respect to the corresponding coupling function captures many specific MDP instances such as the kernelized nonlinear regulator (KNR) (Kakade et al., 2020) and the generalized linear maximal (
) Hellinger
Indeed, when the coupling function is chosen as the expected Bellman error G h ( g, f ) :
= E π h,f ( Q h,g −
T h Q g,h +1 ) where T h denotes the Bellman operator, we recover the definition of BE dimension (Jin et al., 2021), i.e. dim
FE ( F , G, ϵ ) =
dim BE ( F , G, ϵ ) .
Bellman complete model (Wang et al., 2019), which are not captured by the framework of low BE dimension.
As we will see in later sections, introducing the concept of FE dimension allows the coverage of a strictly wider range of MDP models and hypothesis classes.
A D M I S S I B L E B E L L M A N C H A R AC T E R I Z AT I O N F R A M E W O R K
In this section, we introduce the framework of admissible Bellman characterization.
3. A DMISSIBLE B ELLMAN C HARACTERIZATION
Given an MDP M , a sequence of states and actions s 1 , a 1 , . . .
, s H , two hypothesis classes F and G satisfying the realizability assumption (Assumption 1), 4 and a discriminator function class
V = { v ( s, a, s ′ ) :
S ×A×S → R } , the estimation function ℓ = { ℓ h,f ′ } h ∈ [ H ] ,f ′ ∈F is an R d s -valued function defined on the set consisting of o h :
=
( s h , a h , s h +1 ) ∈ S ×A×S , f ∈ F , g ∈ G and v ∈ V and serves as a surrogate loss function of the Bellman error.
Note that our estimation function is a vector-valued function, and is more general than the scalar-valued estimation function (or discrepancy function) used in Foster et al.
; Du et al. (2021).
The discriminator v originates from the function class the Integral Probability Metrics (IPM) (M¨uller, 1997) is taken with respect to (as a metric between two distributions), and is also used in the definition of Witness rank (Sun et al., 2019).
We use a coupling function G h,f ∗ ( f, g ) defined on F ×F to characterize the interaction between two hypotheses f, g ∈ F .
The subscript f ∗ is an indicator of the true model and is by default unchanged throughout the context.
When the two hypotheses coincide, our characterization of the coupling function reduces to the Bellman error.
Definition 5 (Admissible Bellman Characterization) .
Given an MDP M , two hypothesis classes
F , G satisfying the realizability assumption (Assumption 1) and F ⊂ G , an estimation function ℓ h,f ′ :
( S × A × S ) × F × G × V → R d s , an operation policy π op and a constant κ ∈ (0 , 1] , we say that G is an admissible Bellman characterization of ( M, F , G , ℓ ) if the following conditions hold
: (i) (Dominating Average Estimation Function)
For any f, g ∈ F
[Equation]
(ii) (Bellman Dominance)
For any ( h, f ) ∈
[ H ] × F ,
[Equation]
We further say ( M, F , G , ℓ, G ) is an ABC class if G is an admissible Bellman characterization of ( M, F , G , ℓ ) .
In Definition 5, one can choose either π op =
π g or π op =
π f .
We refer readers to §D for further explanations on π op .
The ABC class is quite general and de facto covers many existing MDP models; see §3.2 for more details.
Comparison with Existing MDP Classes.
Here we compare our ABC class with three recently proposed MDP structural classes: Bilinear Classes (Du et al., 2021), low Bellman eluder dimension (Jin et al., 2021), and Bellman Representability (Foster et al., 2021).
• Bilinear Classes.
Compared to the structural framework of Bilinear Class in Du et al.
Definition 4.3), Definition 5 of Admissible Bellman Characterization does not require a bilinear
structure and recovers the Bilinear Class when we set G h,f ∗ ( f, g )
=
⟨ W h ( g ) − W h ( f ∗ ) , X h ( f ) ⟩ .
Our ABC class is strictly broader than the Bilinear Class since the latter does not capture low eluder dimension models, and our ABC class does.
In addition, the ABC class admits an estimation function that is vector-valued , and the corresponding algorithm achieves a √ T -regret for KNR case while the BiLin-UCB algorithm for Bilinear Classes (Du et al., 2021) does not.
• Low Bellman Eluder Dimension.
Definition 5 subsumes the MDP class of low BE dimension when ℓ h,f ′
( o h , f h +1 , g h , v ) :
= Q h,g ( s h , a h ) − r h − V h
+1 ,f
( s h +1 ) .
Moreover, our definition unifies the V -type and Q -type problems under the same framework by the notion of π op .
We will provide a more detailed discussion on this in §3.2.
Our extension from the concept of the Bellman error to estimation function (i.e. the surrogate of the Bellman error) enables us to accommodate model-based RL for linear mixture MDPs, KNR model, and low Witness rank.
• Bellman Representability.
Foster et al.
(2021) proposed DEC framework which is another MDP class that unifies both the Bilinear Class and the low BE dimension.
Indeed, our ABC framework introduced in Definition 5 shares similar spirits with the Bellman Representability Definition F. in Foster et al.
Nevertheless, our framework and theirs bifurcate from the base point: our work studies an optimization-based exploration instead of the posterior sampling-based exploration in Foster et al.
Structurally different from their DEC framework, our ABC requires estimation functions to be vector-valued, introduces the discriminator function v , and imposes the weaker Bellman dominance property (i) in Definition 5 than the corresponding one as in Foster et al.
(2021, Eq.
In total, this allows broader choices of coupling function G as well as our ABC class (with low FE dimension) to include as special instances both low Witness rank and KNR models, which are not captured in Foster et al.
Decomposable Estimation Function.
Now we introduce the concept of decomposable estimation function which generalizes the Bellman error in earlier literature and plays a pivotal role in our algorithm design
and
analysis
Definition 6 (Decomposable Estimation Function) .
A decomposable estimation function ℓ : ( S ×
A × S )
× F × G × V → R d s is a function with bounded ℓ 2 -norm such that the following two conditions hold:
(i) (Decomposability)
There exists an operator that maps between two hypothesis classes T ( · ) :
F → G 5 such that for any f ∈ F , ( h, f ′ , g, v ) ∈
[ H ] × F × G × V and all possible o h ℓ h,f ′
( o h , f h +1 , g h , v ) − E s h
+
[ ℓ h,f ′
( o h , f h +1 , g h , v ) | s h , a h ] =
ℓ h,f ′
( o h , f h
+1 , T ( f ) h , v ) .
Moreover, if f = f ∗ , then T ( f ) = f ∗ holds.
(ii) (Global Discriminator
Optimality) For any f ∈ F there exists a global maximum v ∗ h ( f ) ∈ V such that for any ( h, f ′ , g, v ) ∈
[ H ] × F × G × V and all possible o h
[Equation]
Compared with the discrepancy function or estimation function used in prior work (Du et al., 2021;
Foster et al., 2021), our estimation function (EF) admits the unique properties listed as follows:
(a) Our EF enjoys a decomposable property inherited from the Bellman error — intuitively speaking, the decomposability can be seen as a property shared by all functions in the form of the difference of a J h -measurable function and a J h +1 -measurable function; (b) Our EF involves a discriminator class and assumes the global optimality of the discriminator on all ( s h , a h ) pairs; (c) Our EF is a vector-valued function which is more general than a scalar-valued estimation function (or the discrepancy function).
We remark that when f = g , E s h +
[ ℓ h,f ′
( o h , f h +1 , f h , v ) | s h , a h ] measures the discrepancy in optimality between f and f ∗ .
In particular, when f = f ∗ , E s h +1 (cid:2) ℓ h,f ′
( o h , f ∗ h
+1 , f ∗ h , v ) | s h , a h (cid:3) =
Consider a special case when ℓ h,f ′
( o h , f h +1 , g h , v ) :
= Q h,g ( s h , a h ) − r ( s h , a h ) − V h
+1 ,f
( s h +1 ) .
Then the decomposability (i) in Definition 6 reduces to
[Equation]
,g ( s h , a h ) − r ( s h , a h ) − V h
+1 ,f
( s h +1 )] − [ Q h,g ( s h , a h ) − ( T h V h +1 )
( s h , a h )]
= ( T h V h +1 )
( s h , a h ) − r ( s h , a h ) − V h
+1 ,f
( s h +1 ) .
In addition, we make the following Lipschitz continuity assumption on the estimation function.
Assumption 7 (Lipschitz Estimation Function) .
There exists a L > 0 such that for any ( h, f ′ , f, g, v ) ∈
[ H ] × F × F × G × V , ( (cid:101) f, (cid:101) g, (cid:101) v, (cid:101) f ′ )
∈ F × G × V × F and all possible o h ,
[Equation]
Note that we have omitted the subscript h of hypotheses in Assumption 7 for notational simplicity.
We further define the induced estimation function class as L = { ℓ h,f ′ ( · , f, g, v ) : ( h, f ′ , f, g, v ) ∈
[ H ] × F × F × G × V} .
We can show that under Assumption 7, the covering number of the induced estimation function class L can be upper bounded as N L ( ϵ ) ≤ N 2 F ( ϵ 4 L ) N G ( ϵ 4 L ) N V ( ϵ 4 L ) , where
N F ( ϵ ) , N G ( ϵ ) , N F ( ϵ ) are the ϵ -covering number of F , G and V , respectively.
Later in our theoretical analysis in §4, our regret upper bound will depend on the growth rate of the covering number or the metric entropy , log N F ( ϵ ) .
3. MDP I NSTANCES IN THE ABC C LASS
In this subsection, we present a number of MDP instances that belong to ABC class with low
FE dimension.
As we have mentioned before, for all special cases with ℓ h,f ′ ( o h , f h
+1 , g h , v ) :=
Q h,g ( s h , a h ) − r h − V h
+1 ,f
( s h +1 ) , both conditions in Definition 5 are satisfied automatically with G h,f ∗ ( f, g )
=
E s h ∼ π g ,a h ∼ π op [ Q h,f
( s h , a h ) − r h − V h
+1 ,f
( s h +1 )] .
The FE dimension under this setting recovers the the BE dimension.
Thus, all model-free RL models with low BE dimension (Jin et al., 2021) belong to our ABC class with low FE dimension.
In the rest of this subsection, our focus shifts to the model-based RLs that belong to the ABC class: linear mixture MDPs, low Witness rank, and kernelized nonlinear regulator.
Linear Mixture MDPs.
We start with a model-based RL with a linear structure called the linear mixture MDP (Modi et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021b).
We defer the formal definition of linear mixture MDPs to §E.1 and only provide the following proposition, which shows that linear mixture MDPs belong to the ABC class with low FE dimension.
Proposition 8 (Linear Mixture MDP ⊂ ABC with Low FE Dimension) .
The linear mixture MDP model belongs to the ABC class with estimation function
[Equation]
and
[Equation]
Moreover, it has a low FE dimension.
Low Witness Rank.
The following definition is a generalized version of the witness rank in Sun et al.
(2019), where we require the discriminator class V to be complete , meaning that the assemblage of functions by taking the value at ( s, a ) from different functions also belongs to V .
Definition 9 (Witness Rank) .
For an MDP M , a given symmetric and complete discriminator class
V =
{V h } h ∈ [ H ] , V h ⊂ S
×
A × S (cid:55)→ R and a hypothesis class F , we define the Witness rank of M as the smallest d such that for any two hypotheses f, g ∈ F , there exist two mappings X h : F → R d and W h : F → R d and a constant κ ∈ (0 , 1] , the following inequalities hold for all h ∈
[ H ] :
[Equation]
The following proposition shows that low Witness rank belongs to our ABC class with low FE dimension.
Proposition 10 (Low Witness Rank ⊂
ABC with Low FE Dimension) .
The low Witness rank model belongs to the ABC class with estimation function
[Equation]
and coupling function G h,f ∗ ( f, g ) =
⟨ W h ( g ) , X h ( f ) ⟩ .
Moreover, it has a low FE dimension.
Kernelized Nonlinear Regulator.
The kernelized nonlinear regulator (KNR) proposed recently by Mania et al.
; Kakade et al.
(2020) models a nonlinear control dynamics on an RKHS H of finite or countably infinite dimensions.
Under the KNR setting, given current s h , a h at step h ∈
[ H ] and a known feature mapping ϕ : S ×
A → H , the subsequent state obeys a Gaussian distribution with mean vector U ∗ h ϕ
( s h , a h ) and homoskedastic covariance σ 2 I , where (cid:8) U ∗ h ∈ R d s × H (cid:9)
h ∈
[ H ] are true model parameters and d s is the dimension of the state space.
Mathematically, we have for each h = 1 , . . .
, H ,
[Equation]
where ϵ h
[Equation]
Furthermore, we assume bounded reward r ∈ [0 , 1] and uniformly bounded feature map || ϕ
( s, a ) || ≤
B .
The following proposition shows that KNR belongs to the ABC class with low FE dimension.
1: Initialize : D h = ∅ for h = 1 , . . .
, H 2: for iteration t = 1 , 2 , . .
, T do 3:
Set π t := π f t where f t is taken as argmax f ∈F Q 1 ,f
( s 1 , π f ( s 1 )) subject to max v ∈V (cid:40)
t
[Equation]
1 (cid:88) i =
|| ℓ h,f i ( o ih , f h +1 , f h , v ) ||
2 − inf g h ∈G h t − 1 (cid:88)
i =
|| ℓ h,f i ( o ih , f h +1 , g h , v ) ||(cid:41) ≤
β for all h ∈
[ H ] (4.1) 4:
For any h ∈ [ H ] , collect tuple ( r h , s h , a h , s h +1 ) by rolling in s h ∼ π t and executing a h ∼ π est 5:
Augment D h
= D h ∪ { ( r h , s h , a h , s h +1 ) } 6: end for 7: Output : π out uniformly sampled from { π t } Tt =
Proposition 11 (KNR ⊂ ABC with Low FE Dimension) .
KNR belongs to the ABC class with estimation function
[Equation]
3.6) and coupling function G h,f ∗ ( f, g ) :
= (cid:112) E s h ,a h ∼ π g ||
( U h,f − U ∗ h ) ϕ
( s h , a h ) || 2 .
Moreover, it has a low FE dimension.
Although the dimension of the RKHS H can be infinite, our complexity analysis depends solely on its effective dimension d ϕ .
A L G O R I T H M A N D M A I N R E S U LT S
In this section, we present an RL algorithm for the ABC class.
Then we present the regret bound of this algorithm, along with its implications to several MDP instances in the ABC class.
4. O PERA A LGORITHM
We first present the OPtimization-based ExploRation with Approximation (OPERA) algorithm in
Algorithm 1, which finds an ϵ -optimal policy in polynomial time.
Following earlier algorithmic art in the same vein e.g., GOLF (Jin et al., 2021), the core optimization step of OPERA is optimizationbased exploration under the constraint of an identified confidence region; we additionally introduce an estimation policy π est sharing the similar spirit as in Du et al.
Due to space limit, we focus on the Q -type analysis here and defer the V -type results to §D in the appendix.
Pertinent to the constrained optimization subproblem in Eq.
(4.1) of our Algorithm 1, we adopt the confidence region based on a general DEF, extending the Bellman-error-based confidence region used in Jin et al.
As a result of such an extension, our algorithm can deal with more complex models such as low Witness rank and KNR.
Similar to existing literature on RL theory with general function approximation, our algorithm is in general computationally inefficient.
Yet OPERA is oracle efficient given the oracle for solving the optimization problem in Line 3 of Algorithm 1.
We will discuss its computational issues in detail in §E.1, §E.2 and §E.3.
4. R EGRET B OUNDS
We are ready to present the main theoretical results of our ABC class with low FE dimension:
Theorem 12 (Regret Bound of OPERA) .
For an MDP M , hypothesis classes F , G , a Decomposable
Estimation Function ℓ satisfying Assumption 7, an admissible Bellman characterization G , suppose ( M, F , G , ℓ, G ) is an ABC class with low functional eluder dimension.
For any fixed δ ∈ (0 , 1) , we choose β =
O (log( THN L (1 /T ) /δ )) in Algorithm 1.
Then for the on-policy case when π op =
π est = π t , with probability at least 1 − δ , the regret is upper bounded by
[Equation]
We defer the proof of Theorem 12, together with a corollary for sample complexity analysis, to §C in the appendix.
We observe that the regret bound of the OPERA algorithm is dependent on both the functional eluder dimension dim FE and the covering number of the induced DEF class N L ( (cid:112) 1 /T ) .
In the special case when DEF is chosen as the Bellman error, the relation dim FE ( F , G, (cid:112) 1 /T )
=
dim BE ( F , Π , (cid:112) 1 /T ) holds with Π being the function class induced by { π f , f ∈ F} , and our Theorem 12 reduces to the regret bound in Jin et al.
(Theorem 15).
We will provide a detailed comparison between our framework and other related frameworks in §A when applied to different MDP models in the appendix.
4. I MPLICATION FOR S PECIFIC MDP I NSTANCES
Here we focus on comparing our results applied to model-based RLs that are hardly analyzable in the model-free framework in §3.2.
We demonstrate how OPERA can find near-optimal policies and achieve a state-of-the-art sample complexity under our new framework.
Regret-bound analyses of linear mixture MDPs and several other MDP models can be found in §B in the appendix.
We highlight that Algorithm 1 not only provides a simple optimization-based scheme, recovers previous near-optimal algorithms in literature (Algorithms 2 and 4 in §E) when applied to specific
MDP instances, but also reduces to a novel Algorithm 3 for low witness rank MDPs with improved sample complexity.
Low Witness Rank.
We first provide a sample complexity result for the low Witness rank model structure.
Let |M| and |V| be the cardinality of the model class 7 M and discriminator class V , respectively, and W κ be the witness rank (Definition 9) of the model.
We have the following sample complexity result for low Witness rank models.
Corollary 13 (Finite Witness Rank) .
For an MDP model M with finite witness rank structureand any fixed δ ∈ (0 , 1) , we choose β
=
O (log( TH |M||V| /δ )) in Algorithm 1.
With probability at least 1 − δ , Algorithm 1 outputs an ϵ -optimal policy π out within T = (cid:101) O (cid:0)
H 2 |A| W κ β/ ( κ 2 ϵ 2 ) (cid:1) trajectories.
Proof of Corollary 13 is delayed to §E.4.
Compared with previous best-known sample complexity result of (cid:101) O (cid:0) H 3 W 2 κ |A| log( T |M||V| /δ ) / ( κ 2 ϵ 2 ) (cid:1) due to Sun et al.
(2019), our sample complexity is superior by a factor of dH up to a polylogarithmic prefactor in model parameters.
Kernel Nonlinear Regulator.
Now we turn to the implication of Theorem 12 for learning KNR models .
We have the following regret bound result for KNR.
Corollary 14 (KNR) .
For the KNR model in Eq.
(3.5) and any fixed δ ∈ (0 , 1) , we choose β =
O (cid:0) σ 2 d ϕ d s log 2 ( TH/δ ) (cid:1) in Algorithm 1.
With probability at least 1 − δ , the regret is upper bounded by (cid:101) O (cid:0) H 2 (cid:112) d
ϕ Tβ/σ (cid:1) .
We remark that neither the low BE dimension nor the Bellman Representability classes admit the
KNR model with a sharp regret bound.
Among earlier attempts, Du et al.
(2021, §6) proposed to use a generalized version of Bilinear Classes to capture models including KNR, Generalized Linear
Bellman Complete, and finite Witness rank.
Nevertheless, their characterization requires imposing monotone transformations on the statistic and yields a suboptimal O ( T 3 / 4 ) regret bound.
Our ABC class with low FE dimension is free of monotone operators, albeit that the coupling function for the
KNR model is not of a bilinear form.
C O N C L U S I O N A N D F U T U R E W O R K
In this paper, we proposed a unified framework that subsumes nearly all Markov Decision Process (MDP) models in existing literature from model-based and model-free RLs.
For the complexity analysis, we propose a new type of estimation function with the decomposable property for optimization-based exploration and use the functional eluder dimension with respect to an admissible
Bellman characterization function as the complexity measure of our model class.
In addition, we proposed a new sample-efficient algorithm, OPERA, which matches or improves the state-of-the-art sample complexity (or regret) results.
Nevertheless, we notice that some MDP instances are not covered by our framework such as the Q ∗ state-action aggregation, and the deterministic linear Q ∗ models where only Q ∗ has a linear structure.
We leave it as a future work to include these MDP models.
A P P E N D I X
The appendix is organized as follows.
§A discusses the related work, providing comparisons with previous frameworks based on both coverage and sharpness of sample complexity.
§B compares our regret bound and sample complexity on specific examples and discusses several additional examples including reactive POMDPs, FLAMBE, LQR, and the generalized linear Bellman complete model.
§C proves the main results (Theorem 12 and Corollary 27 on sample complexity of OPERA).
§D explains the V -type setting and the corresponding results.
§E discusses the OPERA algorithm when being applied to special examples (linear mixture MDPs, low Witness rank MDPs, KNRs).
§F details the delayed proofs of technical lemmas.
§G details the proofs relevant to FE dimension.
[Equation]
R E L AT E D W O R K
Tabuler RL.
Tabular RL considers MDPs with finite state space S and action space A .
This setting has been extensively studied (Auer et al., 2008; Dann & Brunskill, 2015; Brafman & Tennenholtz, 2002; Agrawal & Jia, 2017; Azar et al., 2017; Zanette & Brunskill, 2019; Zhang et al., 2020) and the minimax-optimal regret bound is proved to be (cid:101)
O ( (cid:112) H 2 |S||A| T )
(Jin et al., 2018; Domingues et al., 2021).
The minimax optimal bounds suggests that the tabular RL is information-theoretically hard for large |S| and |A| .
Therefore, in order to deal with high-dimensional state-action space arose in many real-world applications, more advanced structural assumptions that enable function approximation are in demand.
Complexity Measures for Statistical Learning.
In classic statistical learning, a variety of complexity measures have been proposed to upper bound the sample complexity required for achieving a certain accuracy, including VC Dimension (Vapnik, 1999), covering number (Pollard, 2012),
Rademacher Complexity (Bartlett & Mendelson, 2002), sequential Rademacher complexity (Rakhlin et al., 2010) and Littlestone dimension (Littlestone, 1988).
However, for reinforcement learning, it is a major challenge to find such general complexity measures that can be used to analyze the sample complexity under a general framework.
RL with Linear Function Approximation.
A line of work studied the MDPs that can be represented as a linear function of some given feature mapping.
Under certain completeness conditions, the proposed algorithms can enjoy sample complexity/regret scaling with the dimension of the feature mapping rather than |S| and |A| .
One such class of MDPs is linear MDPs (Jin et al., 2020;
Wang et al., 2019; Neu & Pike-Burke, 2020), where the transition probability function and reward function are linear in some feature mapping over state-action pairs.
Zanette et al.
(2020a;b) studied MDPs under a weaker assumption called low inherent Bellman error, where the value functions are nearly linear w.r.t.
the feature mapping.
Another class of MDPs is linear mixture MDPs (Modi et al., 2020;
Jia et al., 2020; Ayoub et al., 2020;
Zhou et al., 2021b;
Cai et al., 2020), where the transition probability kernel is a linear mixture of a number of basis kernels.
The above paper assumed that feature vectors are known in the MDPs with linear approximation while Agarwal et al.
(2020b) studied a harder setting where both the feature and parameters are unknown in the linear model.
RL with General Function Approximation.
Beyond the linear setting, a recent line of research attempted to unify existing sample-efficient approaches with general function approximation.
Osband & Van Roy (2014) proposed an structural condition named eluder dimension.
Wang et al.
(2020) further proposed an efficient algorithm LSVI-UCB for general linear function classes with small eluder dimension.
Another line of works proposed low-rank structural conditions, including Bellman rank (Jiang et al., 2017;
Dong et al., 2020) and Witness rank (Sun et al., 2019).
Yang et al.
(2020) studied the MDPs with a structure where the action-value function can be represented by a kernel function or an over-parameterized neural network.
Recently, Jin et al.
(2021) proposed a complexity called
Bellman eluder (BE) dimension.
The RL problems with low BE dimension subsume the problems with low Bellman rank and low eluder dimension.
Simultaneously Du et al.
(2021) proposed Bilinear
Classes, which can be applied to a variety of loss estimators beyond vanilla Bellman error, but with possibly worse sample complexity.
Very recently, Foster et al.
(2021) proposed Decision-Estimation
Coefficient (DEC), which is a necessary and sufficient condition for sample-efficient interactive learning.
To apply DEC to reinforcement learning, Foster et al.
(2021) further proposed a RL class named Bellman Representability, which can be viewed as a generalization of the Bilinear Class.
In Table 1, we compare our ABC framework with other structural RL frameworks in terms of the model coverage and sample complexity.
[Table]
“—” indicates that the original work of framework does not provide an explicit sample complexity result for that model (although can be computed in principle), “ ✘ ” indicates the model is not included in the framework for complexity analysis.
For models with the linear structure on a d -dimensional space, we present the sample complexity in terms of d .
For models with their own complexity measures, we use W κ to denote the witness rank, dim E the eluder dimension, d ϕ the dimension of
H in KNR and d s the dimension number of the
B A D D I T I O NA L E X A M P L E S
In this section, we compare our work with other results in the literature in terms of regret bounds/sample complexity.
First of all, as we mentioned earlier in §3 when taking DEF as ℓ h,f ′
( o h , f h +1 , g h , v ) =
Q h,g ( s h , a h ) − r h − V h
+1 ,f
( s h +1 )
the ABC function reduces to the average
Bellman error, and our ABC framework recovers the low Bellman eluder dimension framework for all cases compatible with such an estimation function.
On several model-free structures, our regret bound is equivalent to that of the GOLF algorithm (Jin et al., 2021).
For example for linear MDPs,
OPERA exhibits a (cid:101) O ( dH √ T ) regret bound that matches the state-of-the-art result on linear function approximation provided in Zanette et al.
(2020a).
For low eluder dimension models, the dependency on the eluder dimension d in our regret analysis is (cid:101) O ( √ d ) while the dependency in Wang et al.
(2020) is (cid:101) O
( d ) .
Also, for models with low Bellman rank d , our sample complexity scales linearly in d as in Jin et al.
while complexity in Jiang et al.
(2017) scales quadratically.
For model-based RL settings with linear structure that are not within the low BE dimension framework such as the linear mixture MDPs, our OPERA algorithm obtains a d FE H √ T regret bound and d 2 FE H 2 /ϵ 2 sample complexity result.
In comparison, Jia et al.
Modi et al.
(2020) proposed an
UCRL-VTR algorithm on linear mixture MDPs with a dH √ T regret bound, and Zhou et al.
(2021a) improves this result by √ H via a Bernstein-type bonus for exploration.
The Bilinear Classes (Du et al., 2021) is a general framework that covers linear mixture MDPs as a special case.
The sample complexity of the BiLin-UCB algorithm when constrained to linear mixture models is d 3 H 4 /ϵ 2 , which is dH 2 worse than that of OPERA in this work.
In terms of lower bound, when specialized to linear mixture MDPs, our result of (cid:101) O ( dH √ T ) matches the lower bound provided in Zhou et al.
(2021a)
up to a factor of H 1 / 2 .
In the rest of this section, we compare on six additional examples: the linear Q ∗ / V ∗ model (Du et al., 2021), the low occupancy complexity model (Du et al., 2021), kernel reactive POMDPs,
FLAMBE/Feature Selection, Linear Quadratic Regulator, and finally Generalized Linear Bellman
Complete.
Moreover, we added a discussion on the Q ∗ state-action aggregation model.
B. L INEAR Q ∗ / V ∗
The linear Q ∗ / V ∗ model was proposed in Du et al.
In addition to the linear structure of the optimal action-value function Q ∗ , we further assume linear structure of the optimal state-value function V ∗ .
We formally define the linear Q ∗ / V ∗ model as follows:
Definition 15 (Linear Q ∗ / V ∗ , Definition 4.5 in Du et al. 2021) .
A linear Q ∗ / V ∗ model satisfies for two Hilbert spaces H 1 , H 2 and two given feature mappings ϕ
( s, a ) :
S ×
A → H 1 , ψ ( s ′ ) :
S → H 2 , there exist w ∗ h ∈ H 1 , θ ∗ h ∈ H 2 such that
[Equation]
for any h ∈ [ H ] and ( s, a, s ′ )
∈ S ×
A × S .
Suppose that H 1 and H 2 has dimension number d 1 and d 2 , separately, Du et al.
(2021) shows that linear Q ∗ / V ∗ model belongs to the Bilinear Class with dimension d = d 1 + d 2 and BiLin-UCB algorithm achieves an (cid:101)
O ( d 3 H 4 ϵ 2 ) sample complexity.
On the other hand the sample complexity of
OPERA is of (cid:101) O
( d 2 H 2 ϵ 2 ) .
B. L OW O CCUPANCY C OMPLEXITY
The low occupancy complexity model assumes linearity on the state-action distribution and has been proposed in Du et al.
We recap its definition formally as follows:
Definition 16 (Low Occupancy Complexity, Definition 4.7 in Du et al. 2021) .
A low occupancy complexity model is an MDP M satisfying for some hypothesis class F , a Hilbert space H and feature mappings ϕ h ( · , · ) : S ×
A → H , ∀ h ∈
[ H ] that there exists a function on hypothesis classes β h : F → H such that
[Equation]
Du et al.
(2021) proved that the low occupancy complexity model belongs to the Bilinear Classes and has a sample complexity of d 3 H 4 /ϵ 2 under the BiLin-UCB algorithm.
In the meantime, the low occupancy complexity model admits an improved sample complexity of d 2 H 2 /ϵ 2 under the OPERA algorithm.
B. K ERNEL R EACTIVE POMDP S
The Reactive POMDP (Krishnamurthy et al., 2016) is a partially observable MDP (POMDP) model that can be described by the tuple ( S , A , O , T , O , r, H ) , where S and A are the state and action spaces respectively, O is the observation space, T is the transition matrix that maps each ( s, a ) ∈ S
× A to a probability measure on S and determines the dynamics of the next state as s h +1 ∼ T ( · | s h , a h ) , O is the emission measure that determines the observation o h ∼ O ( · | s h ) given current state s h .
The reactiveness of a POMDP refers to the property that the optimal value function Q ∗ depends only on the current observation and action.
In other words, for all h , there exists a f ∗ h :
O × A → [0 , 1] such that for any given trajectory τ h =
[ o 1 , a 1 , . . .
, o h ] and a h , we have
Q
[Equation]
Given the definition of a reactive POMDP, we define the kernel reactive POMDP (Jin et al., 2021) as follows:
Definition 17 (Kernel Reactive POMDP) .
A kernel reactive POMDP is a reactive POMDP that satisfies for each h ∈
[ H ] and a given seperable Hilbert space H , there exist feature mappings ϕ h : S×A → H and ψ h : S → H such that the transition matrix T h ( s ′ | s, a ) = ⟨ ϕ h ( s, a ) , ψ h ( s ′ ) ⟩ H and ψ is bounded in the sense that for any V ( · ) :
S → [0 , 1] , (cid:13)(cid:13)(cid:80) s ′ ∈S V ( s ′ )
ψ ( s ′ )
(cid:13)(cid:13)
H ≤ 1 .
In Jin et al.
, the authors showed that the kernel reactive POMDP with vanilla estimation function ℓ h
( o h , f h +1 , g h , v ) =
Q h,g ( s h , a h ) − r h − V h
+1 ,f
( s h +1 ) has V -type BE dimension bounded by the effective dimension.
According to Proposition 34, the kernel reactive POMDP model also has low FE dimension bounded by the effective dimension.
B. FLAMBE/F EATURE S ELECTION
For FLAMBE/feature selection model firstly introduced in Agarwal et al. (2020b), similarity is shared with the linear MDP setting but the main difference lies in that the feature mappings are unknown .
We formally define the feature selection model as follows:
Definition 18 (Feature Selection) .
A low rank feature selection model is an MDP M that satisfies for any h ∈
[ H ] and a given Hilbert space H , there exist unknown feature mappings µ ∗ h : S → H and ϕ ∗ : S ×
A → H such that the transition probability satisfies:
[Equation]
We consider the feature selection model with DEF ℓ h ( o h , f h +1 , g h , v ) :
= Q h,g ( s h , a h ) − r h −
V h
+1 ,f
( s h +1 ) .
In Du et al.
they have proved in Lemma A.1 that
[Equation]
where
[Equation]
We note that Eq.
(B.1) ensures condition (i) and (ii) in Definition 5 at the same time and the ABC of the feature selection setting has a bilinear structure that enables us to apply Proposition 35 to conclude low FE dimension.
B. L INEAR Q UADRATIC R EGULATOR
In a linear quadratic regulator (LQR) model (Bradtke, 1992; Anderson & Moore, 2007; Dean et al., 2020), we consider the d dimensional state space S ⊆
R
d
and K dimensinal action space A ⊆ R K .
The transition dynamics of an LQR model can be written in matrix form so that the induced value function is quadratic (Jiang et al., 2017).
We formally define the LQR model as follows:
Definition 19 (Linear Quadratic Regulator) .
A linear quadratic regulator model is an MDP M such that there exist unknown matrix A ∈ R d × d , B ∈ R d × K and Q ∈ R
d × d satisfying for ∀ h ∈
[ H ] and zero-centered random variables
ϵ h , τ h with E
[ ϵ h ϵ ⊤ h ] = Σ and E [ τ 2 h ] =
σ 2 that
[Equation]
The LQR model has been analyzed in Du et al.
and proved to belong to the Bilinear Classes.
Du et al.
(2021) used the hypothesis class defined as
[Equation]
For each hypothesis in the class f ∈ F , the corresponding policy and value function are
[Equation]
Under the above setting, we use the DEF for LQR ℓ h ( o h , f h +
[Equation]
g h ,
[Equation]
= Q h,g ( s h , a h ) − r h −
V h
+1 ,f
( s h +1 ) and Lemma A.4 in Du et al.
(2021) showed that
[Equation]
h
[Equation]
B
[Equation]
2) where
[Equation]
We note that Eq.
(B.2) ensures condition (i) and (ii) in Definition 5 simultaneously and the ABC of the LQR model setting admits a bilinear structure that enables us to apply Proposition 35 and conclude low FE dimension.
B. G ENERALIZED L INEAR B ELLMAN C OMPLETE
Next we introduce the generalized linear Bellman complete model, showing that our ABC class with low FE dimension captures this model even without the monotone operator √ x used in Du et al.
Definition 20 (Generalized Linear Bellman Complete) .
A generalized linear Bellman complete model consists of an inverse link function σ : R → R + and a hypothesis class F := (cid:8)
F h = σ ( θ ⊤ h ϕ
[Equation]
s, a )) :
θ h ∈ H , ||
θ h || ≤ R (cid:9)
h ∈
[ H ] such that for any f ∈ F and ∀ h ∈ [ H ] the Bellman completeness condition holds:
[Equation]
By the choice of the hypothesis class F , we know that there exists a mapping T h : H → H such that
[Equation]
We note that in Du et al.
they choose a discrepancy function dependent on a discriminator function v .
In this work, we choose a different estimation function that allows much simpler calculation and sharper sample complexity result.
We let
[Equation]
By Eq.
(B.3), it is easy to check that the above DEF satisfies the decomposable condition.
Assuming a ≤ σ ′ ( x ) ≤ b , Lemma 6.2 in Du et al.
(2021) has already shown the Bellman dominance property that
[Equation]
Next
[Equation]
we illustrate that the Dominating Average EF condition holds in our framework.
We have
[Equation]
where
[Equation]
Analogous to the KNR case and the proof of Lemma 30, the aforementioned model with ABC function (cid:112) ⟨ W h ( f ) , X h ( f ) ⟩ has low FE dimension.
B. Q ∗ STATE - ACTION AGGREGATION
Finally, we consider the Q ∗ state-action aggregation model (Dong et al., 2019), which cannot be covered by the bilinear classes (Du et al., 2021).
We illustrate that our ABC framework covers this model with a nonlinear coupling function.
Definition 21 ( Q ∗ state-action aggregation) .
We call an MDP M a Q ∗ state-action model if there exists a ξ ( s, a ) : S ×
A → B such that for any state-action pairs ( s, a ) , ( s ′ ,
a ′ ) ∈
S × A , if ξ ( s, a ) =
ξ ( s ′ , a ′ ) , then Q ∗ ( s, a ) =
Q ∗ ( s ′ , a ′ ) .
The dimension d of the Q ∗ state-action aggregation model is defined as the cardinality of B , i.e., d = |B| .
For each element b ∈ B , we note that for all ( s, a ) ∈ S ×
A satisfying ξ ( s, a ) =
b , they share the same value of Q ( s, a ) .
For notational simplicity, we use Q ( b ) to denote this common value for all ξ ( s, a )
= b .
Moreover, we let w ∗ h be a d -dimensional vector of all aggregated values with ( w ∗ h ) b = Q ( b ) and Q ∗ ( s, a ) can be expressed as a linear function on the aggregated values:
[Equation]
[Equation]
where ψ ( s, a ) : S ×
A → { 0 , 1 } d is a one-hot vector satisfying ( ψ ( s, a ))
b = 1 when ξ ( s, a ) =
b and ( ψ ( s, a )) b = 0 otherwise .
For this case, we use the hypothesis class defined as
[Equation]
where we take
[Equation]
We define the DEF as the Bellman residual and conclude that
[Equation]
Taking expectation over the distribution on o h given by s h
[Equation]
a h ∼ π f and s h +1 ∼ P h
[Equation]
we have
[Equation]
If we define G ( g, f ) by the right hand side of (B.5)
[Equation]
it is obvious that G h,f ∗ ( ·
[Equation]
· ) with ℓ defined in (B.4) serves as the coupling function and the estimation function of an ABC class, respectively.
In the mean time, Equation (B.5) cannot be expressed as a inner product of some W h ( f ) , X h ( g ) and thus cannot be covered by Du et al.
Nevertheless, in our ABC framework, it is unclear if the
FE dimension of F with respect to G h,f ∗ ( · , · ) can be bounded in a nontrivial way (i.e., ≪ |S| · |A| ).
C P RO O F O F M A I N R E S U LT S
In this section, we provide proofs of our main result Theorem 12 and a sample complexity corollary of the OPERA algorithm.
Originated from proof techniques widely used in confidence bound based
RL algorithms Russo & Van Roy (2013)
our proof steps generalizes that of the GOLF algorithm Jin et al.
but admits general DEF and ABCs.
We prove our main result as follows:
C. P ROOF OF T HEOREM
Proof of Theorem 12.
We recall that the objective of an RL problem is to find an ϵ -optimal policy satisfying V ∗ 1 ( s 1 ) − V π t 1 ( s 1 ) ≤ ϵ .
Moreover, the regret of an RL problem is defined as
[Equation]
, where π t is the output policy of an algorithm at time t .
Step 1: Feasibility of f ∗ .
First of all, we show that the optimal hypothesis f ∗ lies within the confidence region defined by Eq.
(4.1) with high probability:
Lemma 22 (Feasibility of f ∗ ) .
In Algorithm 1, given ρ > 0 and δ > 0 we choose β = c (log ( TH N L ( ρ ) /δ )
+ Tρ ) for some large enough constant c .
Then with probability at least 1 − δ , f ∗ satisfies for any t ∈ [ T ] :
[Equation]
Lemma 22 shows that at each round of updates the optimal hypothesis f ∗ stays in the confidence region depicted by Eq.
(4.1) with radius O ( β ) .
We delay the proof of Lemma 22 to §F.2.
Lemmatogether with the optimization procedure Line 3 of Algorithm 1 implies an upper bound of V ∗ 1 ( s 1 ) −
V π t 1 ( s 1 ) with probability at least 1 − δ as follows:
[Equation]
Step 2: Policy Loss Decomposition.
The second step is to upper bound the regret by the summation of Bellman errors.
We apply the policy loss decomposition lemma in Jiang et al.
Lemma 23 (Lemma 1 in Jiang et al. 2017) .
∀ f ∈ H ,
[Equation]
Combining Lemma 23 with Eq.
(C.1)
we have the following:
[Equation]
Step 3: Small ABC Value in the Confidence Region.
The third step is devoted to controlling the cumulative square of Admissible Bellman Characterization function.
Recalling that the ABC function is upper bounded by the average DEF, where each feasible DEF stays in the confidence region that satisfies Eq.
(4.1), we arrive at the following Lemma 24:
Lemma 24.
In Algorithm 1, given ρ > 0 and δ > 0 we choose β = c (log ( TH N L ( ρ ) /δ )
+ Tρ ) for some large enough constant c .
Then with probability at least 1 − δ , for all ( t, h ) ∈ [ T ] ×
[ H ] , we have
[Equation]
The proof of Lemma 24 makes use of Freedman’s inequality (the precise version as in Agarwal et al.
and we delay the proof to §F.1.
Step 4:
Bounding the Cumulative Bellman Error by Functional Eluder Dimension.
In the fourth step, we aim to traslate the upper bound of the cumulative squared ABC at ( f t , f i ) in Eq.
(C.3) to an upper bound of the cumulative ABC at ( f t , f t ) .
The following Lemma 25 is adapted from
Lemma 41 in Jin et al.
(2021) and Lemma 2 in Russo & Van Roy (2013).
Lemma 25 controls the sum of ABC functions by properties of the functional eluder dimension.
Lemma 25.
For a hypothesis class F and a given coupling function G ( · , · ) : F × F → R with bounded image space | G ( · , · ) | ≤ C .
For any pair of sequences { f t } t ∈ [ T ] , { g t } t ∈ [ T ] ⊆
F satisfying for all t ∈
[ T ] , (cid:80) t − 1 i =1 ( G ( f t , g i )) 2 ≤ β , the following inequality holds for all t ∈
[ T ] and ω > 0 :
[Equation]
The proof of Lemma 25 is in §F.3.
Step 5: Combining Everything.
In the final step, we combine the regret bound decomposition argument, the cumulative ABC bound, and the Bellman dominance property together to derive our final regret guarantee.
For any h ∈ [ H ] , we take G ( · , · ) =
G h,f ∗ ( · , · ) , g i = f
i , f t = f t and ω = (cid:113) 1 T in Lemma 25.
By
Eq.
(C.3) in Lemma 24, we have for any h ∈
[ H ] and t ∈ [ T ] , t (cid:88) i =
[Equation]
dim FE ( F , G h,f ∗ , (cid:112) 1 /T ) βt (cid:17) .
We recall our choice of β
=
c (log ( TH N L ( ρ ) /δ )
+ Tρ ) .
Taking ρ = 1 T , we have t (cid:88) i = | G h,f ∗ ( f i , f i ))
| ≤ O (cid:32)(cid:114) dim FE (cid:16) F , G h,f ∗ , (cid:112) 1 /T (cid:17) log ( TH N L (1 /T ) /δ )
· t (cid:33) ≤ O (cid:32)(cid:114) dim FE (cid:16) F , G, (cid:112) 1 /T (cid:17) log ( TH N L (1 /T ) /δ )
· t (cid:33) .
Combining this with property (ii) in Definition 5 and decomposition (C.2), we conclude our main result that with probability at least 1 − δ ,
[Equation]
This completes the whole proof of Theorem
[Equation]
C. S AMPLE C OMPLEXITY OF OPERA
Corollary 26 (Sample Complexity of OPERA) .
For an MDP M with hypothesis classes F
[Equation]
G that satisfies Assumption 1 and a Decomosable Estimation Function ℓ satisfying Assumption 7.
If there exists an Admissible Bellman Characterzation G with low functional eluder dimension.
For any ϵ ∈ (0 , 1]
[Equation]
we choose β = c (cid:16) log( THN L (cid:16)
[Equation]
for some large enough constant c .
For the on-policy case when π op =
π est = π t , with probability at least 1 − δ
Algorithm 1 outputs a ϵ -optimal policy π out within T trajectories where
[Equation]
Proof of Corollary 26.
By the policy loss decomposition (C.2), (C.3) in Lemma 24 and Lemma 25, we have that
[Equation]
1 ) ≤ 1 κT
T (cid:88) t =
H (cid:88) h
= (cid:12)(cid:12)
G h,f ∗ ( f t , f t ) (cid:12)(cid:12) ≤ O (cid:16) H κ (cid:115) dim FE ( F , G, ω ) (cid:18) log ( TH N L ( ρ ) /δ )
T + ρ (cid:19) + Hω κ (cid:17) .
(C.4)
Taking ω = κϵH and ρ = κ 2 ϵ 2 dim FE ( F ,G, κϵH ) H 2 , the above Eq.
(C.4) becomes
T
T (cid:88) t =
V
[Equation]
(cid:114) dim FE ( F , G, κϵH ) log ( TH N L ( ρ ) /δ )
T + ϵ (cid:17) .
Taking
T = dim FE ( F , G, κϵH ) log ( TH N L ( ρ ) /δ )
Hκ 2 ϵyields the desired result.
[Equation]
In Definition 5, we note that there are two ways to calculate the ABC of an MDP model depending on the different choices of the operating policy π op .
Specifically, if π op =
π g , we call it the Q -type
ABC.
Otherwise, if π op =
π f , we call it the V -type ABC.
For example, when taking
[Equation]
s h +1 )] the FE dimension of G h,f ∗ ( f, g ) recovers the Q -type BE dimension (Definition 8 in Jin et al.
When taking
G
[Equation] [Equation]
] the FE dimension of G h,f ∗ ( f, g ) recovers the V -type BE dimension (Definition 20 in Jin et al.
The algorithm for solving Q -type or V -type models slightly differs in the executing policy π est .
We use π est = π t for Q -type models in Algorithm 1, while π est =
U ( A ) is the uniform distribution on action set for V -type models.
The Q -type characterization and the V -type characterization have respective applicable zones.
For example, the reactive POMDP model belongs to ABC with low FE dimension with respect to Vtype ABC while inducing large FE dimension with respect to Q -type ABC.
On the contrary, the low inherent bellman error problem in Zanette et al.
(2020a) is more suitable for using a Q -type characterization rather than a V -type characterization.
For general RL models, we often prefer Q -type
ABC because the sample complexity of V -type algorithms scales with the dimension of the action space |A| .
Due to the uniform executing policy, we will only be able to derive regret bound for
Q -type characterizations, as is explained in Jin et al.
In §4 and §C, we have illustrated regret bound and sample complexity results for the Q -type cases where we let π op =
π est = π t through Algorithm 1.
In the following Corollary 27, we prove sample complexity result for V -type ABC models.
Corollary 27.
For an MDP M with hypothesis classes F , G that satisfies Assumption 1 and a
Decomposable Estimation Function ℓ satisfying Assumption 7.
If there exists an Admissible Bellman
Characterization G with low functional eluder dimension.
For any ϵ ∈ (0 , 1] , if we choose β =
O (log( THN L ( ρ ) /δ )
+ Tρ ) .
For V -type models when π op =
π est = π t , with probability at least 1 − δ Algorithm 1 outputs a ϵ -optimal policy π out within T = |A|
[Table]
[Table] [Table]
trajectories where ρ = κ 2 ϵ 2 dim FE ( F ,G, κϵ H ) H 2 .
Proof of Corollary 27.
The proof of Corollary 27 basically follows the proof of Theorem 12 and
Corollary 26.
We again have feasibility of f ∗ and policy loss decomposition.
However, due to different sampling policy, the proof of Lemma 24 differs at Eq.
(F.5).
Instead, we have
[Equation]
Thus
[Equation]
Eq.
(C.3) in Lemma 24 becomes
[Equation]
The rest of the proof follow the proof of Corollary 26 with an additional |A| factor
[Equation]
By the policy loss decomposition (C.2) and Lemma
[Equation]
we have that
[Equation]
(cid:19)
+ Hω κ (cid:17) .
(D.2)
Taking ω = κϵH and ρ =
[Equation]
2 dim FE ( F ,G, κϵH ) H 2 , the above Eq.
(D.2) becomes
[Equation]
Taking
[Equation]
yields the desired result.
[Equation]
In this section
[Equation]
we consider three specific examples: linear mixture MDPs, low Witness rank
MDPs
[Equation]
and KNRs.
We explains how our framework exhibits superior properties than other general frameworks on these three instances of MDPs.
For reader’s convenience
[Equation]
we summarize the conditions introduced in Items (i)
[Equation]
(ii) in Definition 6 and also Items (i), (ii) in Definition 5, that are essential for any RL models to fit in our framework:
• Decomposability:
[Equation]
• Global Discriminator Optimality:
[Equation]
• Dominating Average EF:
[Equation]
• Bellman Dominance:
[Equation]
E. L INEAR M IXTURE MDP S
[Equation]
→ H , ψ ( s, a ) : S×A
→ H taking values in a Hilbert space H and an unknown θ ∗ ∈ H , a linear mixture MDP assumes that for any ( s, a, s ′ ) ∈ S
× A × S and h ∈
[ H ] , the transition probability P h ( s ′ | s, a ) and the reward function r ( s, a ) are linearly parameterized as
[Equation]
In this case, we choose F h
=
G h
= { θ h ∈ H} .
Thus, the hypothesis classes F and G consist of the set of parameters θ 1 , . .
, θ H ∈ H .
Moreover, for each hypothesis class f = ( θ 1 ,f , . .
, θ H,f ) ∈ F , the value function with respect to f satiafies for any h ∈
[ H ] that
[Equation]
where ϕ V h +1 ,f ( s, a ) :
= (cid:80) s ′ ∈S ϕ
( s, a, s ′ )
V h
+1 ,f ( s ′ ) .
It is natural to define the DEF by
[Equation]
If we use Φ t − 1 h to denote the matrix (cid:16) ( ψ + ϕ V
h
+1 ,f 1 )( s 1 h , a 1 h ) , . . .
, ( ψ
[Equation]
ϕ V h
+1 ,
[Equation]
1 )( s t
[Equation]
1 h , a t − 1 h ) (cid:17) and y t − 1 h to denote the vector (cid:0) r h − V h
+1 ,f 1 ( s i h +1 ) , . . .
, r h − V h
+1 ,f t − 1 ( s t − 1 h +1 ) (cid:1) , Eq.
(4.1) in Algorithm 1 under linear mixture setting can be written in a matrix form as:
[Equation]
Taking (cid:98) θ h,t = arg min θ || θ
⊤ Φ t − 1 h − y t
[Equation]
||
[Equation]
= (cid:16) Φ t − 1 h (cid:0) Φ t − 1 h (cid:1) ⊤ (cid:17) − 1 Φ t − 1 h (cid:0) y t − 1 h (cid:1) ⊤ and Σ t − 1 h :=
Φ t − 1 h (cid:0) Φ t − 1 h (cid:1) ⊤ , simple algebra yields ||
θ ⊤
[Equation]
So Algorithm 1 reduces to Algorithm 2.
In particular, the confidence region defined by Eq.
(E.2) in Algorithm 2 is the same as the confidence region used in the upper confidence RL with value-targeted regression (UCRL-VTR)
algorithm (Jia et al., 2020; Ayoub et al., 2020).
While in UCRL-VTR, they perform step-by-step local optimization within the confidence region, resulting in a confidence bonus added upon the Q value function,
our
Algorithm 2 follows a global optimization scheme, where the objective is the total expected return
[Table] [Table]
[Table]
by following the optimal policy under the current hypothesis.
The design principle of the global optimization is the same as the ELEANOR algorithm
(Zanette et al., 2020a).
In fact, the difference between UCRL-VTR with Algorithm 2 is analogous to the difference between LSVI-UCB (Jin et al., 2020) with ELEANOR (Zanette et al., 2020a).
Algorithm 2 exhibits a dH √ T regret bound and d 2 H 2 /ϵ 2 sample complexity result, as will be shown later in this subsection.
Compared with the d 3 H 4 /ϵ 2 sample complexity in Du et al.
(2021), our algorithm improves over the best-known results on general frameworks that subsumes linear mixture
MDPs.
We provide more comparisons on the linear mixture model in §B.
In terms of computation, assume that there exists a planning oracle for the optimization problem in
Line 3 of Algorithm 2 that requires B time complexity to solve.
Then for each t ∈ [ T ] , h ∈
[ H ] , the computational complexity of the rest of the algorithm is dominated by the computation of (cid:0) Σ t − 1 h (cid:1) − 1 , and the total computational complexity would be O ( BT + d 2 HT ) .
Next, we proceed to prove that a linear mixture MDP belongs to ABC class with low FE dimension.
Proof of Proposition 8.
In the linear mixture model
[Equation]
we choose hypothesis class F h
=
G h
= { θ h ∈
H} , and DEF function
[Equation]
(a) Decomposability.
Taking expectation over s h
+
and we obtain that
[Equation]
Thus, we have
[Equation]
(b)
Global
Discriminator
Optimality
holds automatically
since ℓ is independent of v .
(c) Dominating Average EF.
We have the following inequality for linear mixture models:
[Equation]
(d) Bellman Dominance.
On the other hand, we know that
[Equation]
(e) Low FE Dimension.
Observe from Eqs.
(E.4) and (E.5) that we can choose ABC function of an linear mixture MDP as
[Equation]
The next Lemma 28 proves that the FE dimension of F with respect to the coupling function
G h,f ∗ ( f, g ) is less than the effective dimension d of the parameter space H .
Lemma 28.
The linear mixture MDP model has FE dimension ≤ (cid:101) O
( d ) with respect to the ABC defined in (E.6).
We prove Lemma 28 in §G.
Thus, we conclude our proof of Proposition 8.
From the above Proof of Proposition 8, we see that linear mixture MDPs perfectly fit our framework.
We apply Theorem 12 and Corollary 26 to linear mixture MDPs and conclude directly that Algorithmhas a regret upper bound of dH √ T together with a sample complexity upper bound of d 2 H 2 /ϵ 2 , matching the best-known results that uses a Hoeffding-type bonus for exploration.
E. L OW W ITNESS R ANK MDP S
In this subsection, we provide a novel method for solving low Witness rank MDPs as a direct application of the OPERA algorithm.
The witness rank is an important model-based assumption that covers several structural models including the factored MDPs (Kearns, 1998).
Also, all models with low Bellman rank structure belong to the class of low Witness rank models while the opposite does not hold (Sun et al., 2019).
Although the witness rank models can be solved in a model-free manner, model-free algorithms cannot find near-optimal solutions of general witness rank models in polynomial time.
Meanwhile, existing frameworks (Sun et al., 2019; Du et al., 2021) with an efficient algorithm does not exhibit sharp sample complexity results.
We recall that in low Witness rank settings, hypotheses on model-based parameters (transition kernel and reward function) are made.
Based on this, there are two recent lines of related approaches.
Sun et al.
(2019) first proposed an algorithm that eliminates candidate models with high estimated witness model misfits.
On the other hand, Du et al.
(2021) proposed a general algorithmic framework that would imply an optimization-based algorithm on low Witness rank models.
We prove an improved sample complexity result over existing literature and illustrate the differences in design scheme of our algorithm.
We present the pseudocode in Algorithm 3.
Note that in
Eq.
(E.7)
, we replace the DEF in Eq.
(4.1) by (3.4).
Next, we elaborate the design scheme of our algorithm in comparison with Sun et al.
and Du et al.
Note that the DEF E (cid:101) s ∼ g h v ( s h , a h , (cid:101) s ) − v ( s h , a h , s h +1 ) is similar with the discrepancy function used in Du et al.
(2021) except for an importance sampling factor.
Moreover, after taking sup over discriminator functions, the expected DEF equals the witnessed model misfit in Sun et al.
Although Du et al.
(2021) did not explicitly give an algorithm for witness rank, we observe some general differences between
OPERA and BiLin-UCB (Du et al., 2021).
The confidence region used in Algorithm 3 (simplified version for comparison) is (cid:80) i
[( ℓ if ) 2 − inf g ( ℓ ig ) 2 ] ≤ β centered at the optimal hypothesis, while the confidence region used in BiLin-UCB is (cid:80) i (cid:16) 1 m (cid:80) j ≤ m ℓ i, ( j ) f (cid:17) 2 ≤ β ′ that bound an estimate of ℓ centered at 0 .
Similarly as in BiLin-UCB, Sun et al.
(2019) also attempts to bound a batched estimate of ℓ .
Their algorithm constantly eliminates out of range models, enforcing small witness model misfit on prior distributions.
The analysis in Sun et al.
and Du et al.
(2021), however, does not enforce the additional assumption on the discriminator class; we obtain a sharper sample complexity as in Corollary 13.
[Table]
[Table] [Table]
[Table] [Table]
If we assume that there exists a planning oracle for solving the optimization problem in Line 3 of
Algorithm 3 with B time complexity.
The computation in the rest of the algorithm is dependent on the structure of discriminator class V and the hypothesis class G .
We omit the discussion here as the planning oracle with a total computational complexity of O ( BT ) is usually the dominating term.
In the forthcoming, we prove that low Witness rank MDPs belongs to ABC class with low FE dimension.
Proof of Proposition 10.
In the low Witness rank model, we choose hypothesis class F h =
G h = M , and DEF function
[Equation]
Without loss of generality, we assume that the discriminator class V is rich enough in the sense that if ∀ s, a ∈ S × A , v s,a ( · , · , · ) ∈ V , then v ( s, a, s ′ ) := v s,a ( s, a, s ′ )
∈ V (if not, we can use a rich enough V ′ induced by V ), an assumption generally satisfied by common discriminator classes.
For example, Total variation, Exponential family, MMD, Factored MDP in Sun et al.
(2019) all use a rich enough discriminator class.
Also, if V = { v : ∥ v ∥ ∞ ≤ c } for some absolute constant c , the function class is rich enough.
(a) Decomposability.
Taking expectation over s h +1 of Eq.
(E.8) and we obtain that
[Equation]
Thus, we have
[Equation]
(b)
Global Discriminator Optimality.
Eq
(E.
) implies that
[Equation]
We define v ∗ h ( f )( s, a, s ′ )
= v s,a ( s, a, s ′ )
where
[Equation]
It is easy to verify that v ∗ h ( f ) satisfies for all h ∈
[ H ] and ( s h , a h ) ∈ S × A ,
[Equation]
Finally, the symmetry of V concludes the global discriminator optimality.
(c) Dominating Average EF.
We have the following inequality for low Witness rank model:
[Equation]
where the last inequality (i) follows Definition 9 of witness rank.
(d) Bellman Dominance.
On the other hand, by Definition 9 we know that
[Equation]
(e) Low FE Dimension.
We see from Eq.
(E.10) and (E.11) that we can choose ABC function with low Witness rank RL model as
[Equation]
The next Lemma 29 proves that the FE dimension of F with respect to the coupling function
G h,f ∗ ( f, g ) is less than the dimension W κ of the witness model.
Lemma 29.
The low Witness rank MDP model has FE dimension ≤ (cid:101)
O
( W κ ) with respect to the
ABC defined in (E.12).
We prove Lemma 29 in §G.
Thus, we conclude our proof of Proposition 10.
By Proposition 10 we can straightforwardly derive the sample complexity by applying Corollary 27.
For better understanding of the context, we present a complete proof of the sample complexity result of witness rank model in §E.4.
E. K ERNELIZED N ONLINEAR R EGULATOR
In the KNR setting introduced in §3.2, the norm of s h
+
might be arbitrarily large if the random vector ϵ h +1 is large in magnitude.
On the contrary, our framework requires the boundedness of the
DEF.
To resolve this issue, we note the tail bound of one-dimensional Gaussian distribution indicates that for any given positive x :
[Equation]
Thus, for TH i.i.d.
R d s -valued random vectors ϵ th ∼ N (0 , σ 2 I ) and a fixed δ ∈ (0 , 1)
, there exists an event B with P ( B ) ≥ 1 − δ such that ∥ ϵ th ∥ ∞ ≤ O (cid:16) σ (cid:112) log( THd s /δ ) (cid:17) holds on event B .
We first provide the application of OPERA on the KNR model, the algorithm is written in Algorithm 4.
Note that by similar algebra as in Eq.
(E.2), the confidence set (E.13) is equivalent to
[Equation]
where Σ t − 1 h :
=
Φ t − 1 h (Φ t − 1 h ) ⊤ and (cid:98)
U h,f is the optimal solution to the least square problem arg min U (cid:80) t − 1 i =
|| Uϕ ( s ih , a ih ) − s ih +
|| 2 .
The OPERA algorithm reduces to the LC 3 algorithm in Kakade et al.
(2020) except that LC 3 is under a homogeneous setting.
The only difference between
Algorithm 4 and LC 3 is that in Eq.
(E.13), LC 3 sums over t and H and we can only sum over t because of the inhomogeneous setting.
Bringing in the choice of β in Corollary 14 yields a regret bound of (cid:101) O (cid:16)(cid:113) d 2 ϕ d s H 4 T (cid:17) .
In comparison,
LC 3 in Kakade et al.
(2020) has a regret bound of (cid:101) O (cid:16)(cid:112)
d ϕ ( d s + d ϕ ) H 3 T (cid:17) .
The improved factor of √ H is due to the reduction from the time-inhomogeneous setting to the time-homogeneous setting.
Thus, our regret bound matches the state-of-the-art result on KNR instances (Kakade et al., 2020) regarding the dependencies on d ϕ , d s , H .
However, d 2 ϕ d s in our result is slightly looser than do 3:
Set subject to t − 1 (cid:88)
i
=
||
U h,f ϕ ( s ih , a ih )
− s ih +
|| 2 − inf g h ∈G h t − 1 (cid:88)
i =
||
U h,g ϕ ( s ih , a ih )
− s ih +
|| 2 ≤ β for all h ∈
[ H ] (E.13) 4:
For any h ∈ [ H ] , collect tuple ( r h , s h , a h , s h +1 ) by executing s h , a h ∼ π t 5:
Augment D h
D h ∪
} 6: end for 7: d ϕ ( d s + d ϕ ) in Kakade et al.
and can be possibly improved by instance-specific analysis of
KNR.
Similar to the linear mixture MDP case, if we assume that there exists a planning oracle for the optimization problem in Line 3 of Algorithm 4 that requires B time complexity to solve, the rest of the algorithm can be solved efficiently in O ( d ϕ ( d ϕ + d s ) HT ) time complexity.
So the total computational complexity of Algorithm 4 is O ( BT + d ϕ ( d ϕ + d s ) HT ) .
In addition, we would like to remark that we can adapt algorithms from the optimal control literature such as MPPI (Williams et al., 2015) and DMDMPC (Wagener et al., 2019) to solve the optimization problem in Line 3 of Algorithm 4.
This approach has been used in Kakade et al.
(2020), where they designed the LC 3 algorithm for solving KNRs.
In particular, they leveraged the MPPI algorithm for the planning oracle and provided rich empirical results.
Proof of Proposition 11.
In the KNR model, we choose hypothesis class F h =
G h
= { U ∈ H → R d s : || U || ≤ R } , and DEF function
[Equation]
(a) Decomposability.
Taking expectation over s h
+
and we obtain that
[Equation]
Thus, we have
[Equation]
(b) Global Discriminator Optimality holds automatically since ℓ is independent of v .
(c) Dominating Average EF.
We have the following inequality for the KNR model:
[Equation]
d)
Bellman Dominance.
On the other hand, we know that
[Equation]
e
) Low FE Dimension.
We see from Eqs.
(E.14) and (E.15) that we can choose ABC function of an linear mixture MDP as
[Equation]
and KNR has an ABC with κ = σ 2 H .
The next Lemma 30 proves that the FE dimension of F with respect to the coupling function G h,f ∗ ( f, g ) can be controlled by d ϕ :
Lemma 30.
The KNR model has FE dimension ≤ (cid:101) O
( d ϕ ) with respect to the ABC defined in (E.16).
We prove Lemma 30 in §G.
Thus, we conclude our proof of Proposition 11.
E. P ROOF OF C OROLLARY
In this subsection, we provide sample complexity guarantee for models with low Witness rank.
In the main text in §4.3 we presented our Corollary 13 for M and V with finite cardinality for convenience of comparison with previous works.
Here, we prove general result for model class M and discriminator class V with finite ρ -covering.
Proof of Corollary 13.
We start the proof by showing that V ∗ ( s 1 ) − V π t 1 ( s 1 ) can be upper bounded by a sum of Bellman errors, which is a simple deduction from the policy loss decomposition lemma in Jiang et al.
and is the same as the equality in Eq.
(C.2) in the proof of Theorem 12 in §C.
Next, we verify that f ∗ satisfies constraint (E.7) so that taking f t = arg max V 1 ,f
( s 1 ) in the confidence region yields V ∗ 1 ( s 1 ) ≤ V 1 ,f t ( s 1 ) .
Lemma 31 (Feasibility of f ∗ ) .
In Algorithm 3, given ρ > 0 and δ > 0 , we choose β = c
(log ( TH |M
ρ ||V ρ | /δ )
+ Tρ ) for some large enough constant c , then with probability at least 1 − δ , f ∗ satisfies for any t ∈
[ T ] :
[Equation]
We prove Lemma 31 in §F.5.
The next Lemma 32 is devoted to controlling the average squared DEF.
Lemma 32.
In Algorithm 3, given ρ > 0 and δ > 0 , we choose β = c
(log ( TH |M ρ ||V ρ | /δ )
+ Tρ ) for some large enough constant c , then with probability at least 1 − δ , for all ( t, h ) ∈
[ T ] ×
[ H ] , we have
[Equation]
Proof is delayed to §F.4.
By Lemma 32 and properties of the witness rank in Definition 9, we have t − 1 (cid:88)
[Equation]
Applying Lemma 25 with G h,f ∗ ( f, g ) :
= ⟨ W h ( f ) , X h ( g ) ⟩ and g i
= f i , f t = f t , we have
[Equation]
Policy loss decomposition (C.2) yields
[Equation]
Taking ω = κϵH and
ρ =
ϵ 2 dim FE ( F ,G, ϵH ) H 2 , the above Eq.
(C.4) becomes
[Equation]
Taking
[Equation]
yields the desired result.
E. P ROOF OF C OROLLARY
We can directly apply Theorem 12 to the KNR model based on Proposition 11 to obtain the regret bound result.
For better understanding of our framework, we illustrate the main features in the proof of Corollary 14 that are different from the proof of Theorem 12.
Proof of Corollary 14.
To resolve the unboundedness issue
[Equation]
we unfold the analysis of KNR case and conclude a high-probability event B analogous to the argument in §E.3.
However
[Equation]
doing so would impose an additional √ d s factor induced by estimating the ℓ 2 -norm of multivariate Gaussians.
In lieu to this
[Equation]
we present a sharper convergence analysis that incorporates KNR instance-specific structures.
We recall the DEF of the KNR model:
[Equation]
We first define an auxilliary random variable
[Equation]
By the boundedness of operator U h,
[Equation]
, U ∗ h and uniform boundedness of ϕ ( s, a ) , we obtain that ||
( U h,f − U ∗ h ) ϕ
( s th , a th ) || 2 ≤ 4 B 2 U B 2 .
The conditional distribution of (cid:10) ( U h,f − U ∗ h ) ϕ
( s th , a th ) , ϵ th +1 (cid:11) is a zero-mean Gaussian with variance
σ 2 || ( U h,f − U ∗ h ) ϕ
( s th , a th ) || 2 ≤ 4 B 2 U B 2 σ 2 .
By the tail bound of Gaussian distributions along with standard union bound, we know that with probability at least 1 − δ ,
[Equation]
holds uniformly for all t ∈
[ T ] and h ∈
[ H ] .
Thus, we bound the absolute value of the auxillary variable X t by | X t | ≤ Rσ where R is positive and of order O (cid:16)(cid:112) log( TH/δ ) (cid:17) .
Taking expectation with respect to s h
+1 , we have
[Equation]
On the other hand,
[Equation]
By taking Z t =
X t ( h, f, v ) − E s h +
[ X t ( h, f, v ) | s h , a h ] with | Z t | ≤ 2 Rσ in Freedman’s inequality (F.1) in Lemma 33, we have for any η satisfying 0 < η < 1 2 R 2 σ 2 almost surely, with probability at least 1 − δ : t (cid:88)
i
[Equation]
Optimizing over η , we have t (cid:88) i =
[Equation]
Following the same Freedman’s inequality (Lemma 33) and ρ -covering argument as as in the proof of Theorem 12 with derivations detailed in §F.1, we have with probability ≥ 1 − δ and β
[Equation]
TH
[Equation] [Equation]
Feasibility of f ∗ can be derived by taking the same auxilliary random variable and analyze on − (cid:80) ti =1 X
i ( h, f, v ) as in the proof of Lemma 31.
As explained in §E.3 , we can apply Lemma 25 with ω = (cid:113) 1 T , ρ =
[Equation] [Equation]
and have t
[Equation]
The rest of the proof follows by applying Bellman dominance, policy loss decomposition and calculating the FE dimension based on G h,f ∗ ( f, g ) , which is shown in Lemma 30.
We therefore obtain that
[Equation]
F P RO O F O F T E C H N I C A L L E M M A S
We start with introducing the Freedman’s inequality that are crucial in proving concentration properties in our main results.
Lemma 33 (Freedman-Style Inequality, Agarwal et al. 2014) .
Consider an adapted sequence { Z t , J t } t =1 , 2 ,...,T that satisfies E [ Z t | J t − 1 ] = 0 and Z t ≤ R for any t = 1 , 2 , . . .
T .
Then for any δ > 0 and η ∈ [0 , 1 R ] , it holds with probability at least 1 − δ that
[Equation]
Before proving our technical lemmas, we note that for notational simplicity we use the expectation E s h +1 [ · | s h , a h ] to denote the conditional expectation with respect to the transition probability of the true model at h .
The value of s h , a h is data dependent (might be s ih , a ih or s th , a th depending on the function inside the expectation).
F. P ROOF OF L EMMA
Proof of Lemma 24.
We recall that ℓ has a bounded ℓ 2 -norm in Definition 6 and assume that || ℓ h,f ′ ( · , f h +1 , g h , v ) || ≤ R for ∀ h ∈
[ H ] , f ′ , f ∈ F , g ∈ G , v ∈ V throughout the paper.
For a sequence of data D h = { r th , s th , a th , s th +1 } t =1 , 2 ,...,T , we first build an auxiliary random variable defined for every ( t, h, f, v ) ∈
[ T ] ×
[ H ] × F × V and consider
[Equation]
i
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
where the randomness is due to uniformly sampling the data sequence D h .
We know that | X t ( h, f ) | ≤
R 2 .
Take conditional expectation of X i with respect to s h , a h , we have by definition that
[Equation]
Using the fact that ∥ a ∥ 2 − ∥ b ∥ 2 = ⟨ a − b, a + b ⟩ for arbitrary vectors a, b and property (i) in
Definition 6 we have
[Equation]
On the other hand
[Equation]
By taking Z t =
[Equation]
t ( h, f, v ) | s h , a h ] with | Z t | ≤ 2 R 2 in Freedman’s inequality (F.1) in Lemma 33, we have for any η satisfying 0 < η < 1 2 R 2 , with probability at least 1 − δ : t
[Equation]
i
[Equation] [Equation]
Taking η = √ log( δ − 1 ) 2 R √
[Equation]
ti =1 E
[ X i ( h,f,v ) | s h ,a h ] ∨ 12 R 2 , we have t (cid:88) i =
[Equation]
Similarly by applying Freedman’s inequality to (cid:80) ti =1 − Z t and combining with Eq.
(F.2), we have that for any three-tuple ( t, h, f ) , the following holds with probability at least 1 − 2 δ :
[Equation]
t (cid:88) i =
Z i (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) ≤ O   2 R (cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88) i =
E s h
+
[Equation]
[ X i ( h, f, v ) | s h , a h ] log( δ − 1 ) + 2 R 2 log( δ − 1 ) 
 .
We note that in §3 we have that L admits a ρ -covering of F , G , V , meaning that for any ℓ h,f ′ ( · , f, g, v ) and a ρ > 0 there exists a (cid:101) ρ and a four-tuple ( (cid:101) f ′ , (cid:101) f, (cid:101) g, (cid:101) v ) ∈ F (cid:101) ρ × F (cid:101) ρ × G (cid:101) ρ × V (cid:101) ρ such that (cid:13)(cid:13)(cid:13) ℓ h, (cid:101) f ′ ( · , (cid:101) f, (cid:101) g, (cid:101) v ) − ℓ h,f ′ ( · , f, g, v ) (cid:13)(cid:13)(cid:13) ∞ ≤ ρ, where F (cid:101) ρ , G (cid:101) ρ , V (cid:101)
ρ are (cid:101) ρ -covers of F , G , V respectively.
This is denoted by ( (cid:101) f ′ , (cid:101) f, (cid:101) g, (cid:101) v ) ∈
L ρ .
In definition of X t , (cid:101) g is always taken as (cid:101) f or a function of
T ( (cid:101) f ) .
Then if T is Lipschitz, as it is mostly the expectation operator, we omit the (cid:101) g in the tuple and use ( (cid:101) f ′ , (cid:101) f, (cid:101) v ) ∈
L ρ to denote an element in the ρ -covering.
By taking a union bound over L ρ , we have with probability at least 1 − 2 δ that the following holds for any ( (cid:101)
f i , (cid:101) f, (cid:101) v ) ∈
L ρ ,
[Equation]
F
[Equation]
where (cid:101)
X i ( h, (cid:101) f, (cid:101) v ) :
= || ℓ h, (cid:101) f i ( o ih , (cid:101)
f h +1 , (cid:101)
f h , (cid:101) v ) ||
2 − || ℓ h, (cid:101) f
i ( o ih , (cid:101)
[Equation] [Equation]
(cid:101)
[Equation] [Equation]
, (cid:101)
[Equation]
) ||
[Equation]
and ι = log (cid:16) HT N L ( ρ ) δ (cid:17) .
Further for any X i ( h, f t , v ) , we choose the three-tuple ( (cid:101)
f i , (cid:101) f t , (cid:101) v ) :
= arg min ( (cid:101)
[Equation]
, (cid:102)
[Equation]
(cid:101)
[Equation] [Equation]
(cid:12)(cid:12)(cid:12) X
[Equation]
[Equation] [Equation]
[Equation]
) − (cid:101) X
[Equation] [Equation]
, (cid:101)
[Equation]
, (cid:101) v ) (cid:12)(cid:12)(cid:12) ≤ ρ and by the ρ -covering argument, we arrive at t
[Equation]
i =
[Equation] [Equation]
=
[Equation]
(cid:105) ≤
[Equation]
i =
[Equation]
(cid:2)
[Equation]
where ( i ) comes from the constraint (4.1) of Algorithm
[Equation]
Combining (F.3) with (F.4)
[Equation]
we derive the following
[Equation]
Applying the ρ -covering argument as in before
[Equation]
we conclude
[Equation]
Global optimality of the discriminator in (ii) of Definition 6 implies that v ∗ h is the optimal discriminator under any distribution or summation of s h , a h (and thus max is interchangeable with summation):
[Equation]
Thus
[Equation]
we have
[Equation]
and also t
[Equation]
We apply property (i) in Definition 5 and conclude that
[Equation]
which finishes the proof of Lemma
[Equation]
F. P ROOF OF L EMMA
Proof of Lemma 22.
For a data set D h
= { r th , s th , a th , s th +1 } t =1 , 2 ,...
T , we first build an auxillary random variable defined for every ( t, h, f, v ) ∈
[ T ] ×
[ H ] × F × V
[Equation]
By similar derivations as in the proof of Lemma 24, we have
[Equation]
Take Z t =
X t ( h, f, v ) − E s h +
[ X t ( h, f, v ) | s h
[Equation]
a h ] with | Z t | ≤ 2 R 2 in Freedman’s inequality (F.1) in Lemma 33.
Then via the same procedure as in the proof of Lemma 24 we have that for any four-tuple ( t, h, f, v ) , the following holds with probability at least 1 − 2 δ :
[Equation]
Thus, we have −
[Equation]
By the same ρ -covering argument as in the proof of Lemma 24, there exists a ρ -covering of L such that we can take a union bound over L ρ and have − (cid:80) t − 1 i =1 (cid:101)
X i ( h, (cid:101) f, (cid:101) v ) ≤ O (cid:0) R
ι
+ Rtρ (cid:1) where ι = log (cid:16) HT N L ( ρ ) δ (cid:17) .
Then for f ∗ , any f ∈ F and any v ∈ V , we can use the nearest three-tuple ( (cid:101)
f i , (cid:101) f, (cid:101) v ) in the ρ -covering and conclude that
[Equation]
This in sum finishes our proof of Lemma 22 with β
= O (cid:0) R 2 ι + Rρt (cid:1) .
F. P ROOF OF L EMMA
Proof of Lemma 25.
The proof basically follows Appendix §C of Russo & Van Roy (2013) and
Appendix §D of Jin et al. (2021).
We first prove that for all t ∈ [ T ] , t (cid:88)
[Equation]
Let m :=
(cid:80) tk =1 1 ( | G ( f k , g k ) | > ϵ ) , then there exists { s 1 , . . .
, s m } which is a subsequence of [ t ] such that G ( f s 1 , g s 1 ) , . .
, G ( f s m , g s m ) > ϵ .
We first show that for the sequence { f s 1 , . . .
, f s m } ⊆ F , there exists j ∈ [ m ] such that f s j is ϵ -independent on at least L = ⌈ ( m − 1) / dim FE ( F , G, ϵ ) ⌉ disjoint sequences in { f s 1 , . . .
, f s j − 1 } (Russo & Van Roy, 2013).
We will prove this by following procedure.
Starting with singleton sequences B 1 = { f s 1 } , . . .
, B L = { f s L } and j = L + 1 .
For each j , if f s j is ϵ -dependent on
B 1 , . . .
, B L we already achieved our goal and the process stops.
Otherwise, there exist i ∈
[ L ] such that f s j is ϵ -dependent of B i and update B i =
B i ∪ { f s j } .
Then we add increment j byand continue the process.
By the definition of FE dimension, the cardinally of each set B 1 , . . .
, B L cannot larger than dim FE ( F , G, ϵ ) at any point in this process.
Therefore, by pigeonhole principle the process stops by step j
=
L dim FE ( F , G, ϵ ) + 1 ≤ m .
Therefore, we have proved that there exists j such that | G ( f s j , g s j ) | > ϵ and f s j is ϵ -independent with at least L = ⌈ ( m − 1) / dim FE ( F , G, ϵ ) ⌉ disjoint sequences in { f s 1 , . . .
, f s j − 1 } .
For each of the sequences { (cid:98) f 1 , . . .
, (cid:98) f l } , by definition of the FE dimension in Definition 3 we have that
[Equation]
Summing all of bounds (F.7) for L disjoint sequences together we have that
[Equation]
The left hand side of (F.8) can be upper bounded by β 2 due to the condition of lemma.
Therefore, we have proved that β 2 ≥ ⌈ ( m − 1) / dim FE ( F , G, ϵ )
⌉ · ϵ 2 which completes the proof of (F.6).
Now let d = dim FE ( F , G, ω ) and sort | G ( f 1 , g 1 ) | , . .
, | G ( f t , g t ) | in a nonincreasing order, denoted by e 1 , . . .
, e t .
Then we have that
[Equation]
For k ∈ [ t ] , we want to give an upper bound for those e k 1 ( e k > ω ) .
Assume e k > ω , then for any α such that e k > α ≥ ω , by (F.6), we have that
[Equation]
which implies that α ≤ (cid:112) dβ/ ( k − d ) .
Taking the limit α → e − k , we have that e k ≤ min { (cid:112) dβ/ ( k − d ) , C } .
Finally, we have that t
[Equation]
Plugging (F.10) into (F.9) completes the proof.
F. P ROOF OF L EMMA
Proof of Lemma 32.
We assume that ∥ v ∥ ∞ ≤ B and treat B as an absolute constant ( B = 2 in Sun et al.
in the following derivations.
For a dataset D h
= { r th , s th , a th , s th +1 } t =1 , 2 ,...
T , we first build an auxillary random variable defined for every ( t, h, f, v ) ∈
[ T ] ×
[ H ] × F × V
X t
[Equation]
where the randomness lies in the sampling of the dataset D h .
We know that | X t ( h, f ) | ≤ 4 B 2 almost surely.
Take conditional expectation of X i with respect to s h , a h , we have by definition that
[Equation]
Using the fact that a 2 − b 2 = ( a − b )( a + b ) and E (cid:101) s ∼ f v ( s ih , a ih , (cid:101) s ) − E (cid:101) s ∼ f ∗ v ( s ih , a ih , (cid:101) s ) is nonrandom given s h , a h , we have
[Equation]
On the other hand,
[Equation]
By taking Z t =
X t ( h, f, v ) − E s h +
[ X t ( h, f, v ) | s h
[Equation]
a h ] with | Z t | ≤ 8 B 2 a.s. in Freedman’s inequality (F.1) in Lemma
[Equation]
by the same procedure as in the proof of Lemma
[Equation]
we have that for any four-tuple ( t, h, f, v )
[Equation]
the following holds with probability at least 1 − 2 δ :
[Equation]
Let M ρ be a ρ -cover of M and V ρ a ρ -cover of V .
By taking a union bound over all ( t, h, f ′ , v ‘) ∈ [ T ] ×
[ H ] × M ρ × V ρ , we have with probability at least 1 − 2 δ that the following holds for any f ′ ∈ M ρ , v ′ ∈ V ρ ,
[Equation]
(F.12) where ι = log( HT |M ρ ||V ρ | δ ) .
Further for any f t calculated at t ∈ [ T ] and any v ∈ V , we choose f ′ = arg min (cid:101)
f ∈M ρ dist ( (cid:101) f, f t ) where dist is the distance measure on M , v ′ =
min v ′ ∈V ρ ( v ′ , v ) and conclude t − 1 (cid:88) i =
[Equation]
=
t
[Equation]
i = (cid:104)(cid:0)
[Equation] [Equation]
where ( i ) is due to the constraint of Algorithm 3.
Combining (F.12) with (F.13)
[Equation]
we derive the following
[Equation]
Note that f ′ is chosen as the nearest model to f t in the ρ -covering of M and for any v there exists a nearest v ′ in the ρ -covering of V , we conclude
[Equation]
Note we also have proved property (ii) in Definition 6 in §E.2, and we apply the global optimality of the discriminator as in the proof of Lemma 24 and obtains
[Equation]
s h , a h (cid:3) ≤ O ( β + Btρ + B 2 ι ) .
Multiplying (cid:2) E
[Equation]
by
[Equation]
, taking expectation on s ih ∼ π i , a ih ∼ π f and again using the global discriminator optimality, we arrive at t
[Equation]
[Equation]
which concludes the proof.
F. P ROOF OF L EMMA
Proof of Lemma 31.
For a dataset D h
= { r th , s th , a th , s th +1 } t =1 , 2 ,...
T , we first build an auxillary random variable defined for every ( t, h, f, v ) ∈
[ T ] ×
[ H ] × F × V
X t ( h, f, v ) :
= (cid:104)(cid:0) E
[Equation]
s
[Equation]
( s th , a th , (cid:101) s ) −
[Equation]
( s th
[Equation]
a th
[Equation]
s th +1 ) (cid:1)
[Equation]
− (cid:0) E (cid:101) s ∼ f
[Equation]
v ( s th , a th , (cid:101) s ) − v ( s th , a th , s th +1 ) (cid:1) 2 (cid:105) .
By Eq.
(F.11), with probability at least 1 − 2 δ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) t (cid:88)
i
[Equation]
Let M ρ be a ρ -cover of M and V ρ a ρ -cover of V .
By taking a union bound over all ( t, h, f ′ , v ‘) ∈ [ T ] ×
[ H ] × M ρ × V ρ , we have with probability at least 1 − 2 δ that the following holds for any f ′ ∈
Z ρ , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) t (cid:88) i =
[Equation]
i =
[Equation]

 , where ι = log (cid:16) HT
|M ρ ||V ρ | δ (cid:17) .
Thus, we have − t (cid:88) i =
X i ( h, f ′ , v ′ ) ≤
O (cid:0) B 2 ι (cid:1) .
Further for any f ∈ F and any v ∈ V , we choose f ′ = arg min (cid:101)
f ∈M ρ dist ( (cid:101) f, f ) where dist is the distance measure on M , v ′ =
min v ′ ∈V ρ ( v ′ , v ) and have −
[Equation]
t
[Equation]
(cid:0)
[Equation]
f
[Equation]
( s th
[Equation]
a th
[Equation]
(cid:101)
[Equation]
s th
[Equation]
a th
[Equation]
≤ O
[Equation]
B
[Equation]
Bρt
[Equation]
Thus, max v ∈V
[Equation]
t −
[Equation]
i = (cid:0)
[Equation]
(cid:101) s ∼ g
[Equation] [Equation]
a
[Equation]
which concludes the proof.
[Equation]
In the following proposition, we prove that the Bellman eluder (BE) dimension (Jin et al., 2021) is a special case of the FE dimension when G h ( g, f ) :
=
E π h,f ( g h −
T
h
g h +1 ) .
Proposition 34.
For any hypothesis class F , taking coupling function G to be the union of { G h :
F h × F h → R } h =1 ,...,H with each G h ( g, f ) :
= E π h,f ( g h − T
h g h +1 ) .
[Equation]
Proof of Proposition 34.
By definition of the functional eluder dimension,
[Equation]
where dim FE ( F , G h , ϵ ) is the length n of the longest sequence satisfying for every t ∈ [ n ] , (cid:113)(cid:80) t − 1 i =
( G h ( g t , f i )) 2 ≤ ϵ ′ and | G h ( g t , f t ) | > ϵ ′ .
Bringing in G h ( g, f ) :
= E π h,f ( g h − T
h g h +1 ) , we have f 1 , . . .
, f n is also the longest sequence that satisfies for some g 1 , . . .
, g n that
[Equation]
i =
[Equation]
and
[Equation]
Thus, dim DE (( I − T h ) F , Π h , ϵ ) ≥ n .
Taking maximum over h ∈ [ H ] , we have dim
[Equation]
which concludes our proof.
Combining Proposition 34 with Proposition 29 in Jin et al.
(2021), it is straightforward to conclude that FE dimension is smaller than the effective dimension.
In particular, Proposition 33 says dim FE is controlled by dim BE , Proposition 29 in Jin et al.
(2021) says dim BE is controlled by the effective dimension dim eff , therefore low effective dimension would imply ABC with low FE dimension.
In the following paragraphs and Proposition 35 we prove this conclusion from sketch to grant a better understanding of the FE dimension.
The effective dimension (Jin et al., 2021) (or equivalently, critical information gain (Du et al., 2021))
d eff ( X , ϵ ) of a set X is defined as the smallest interger n > 0 such that
[Equation]
Remark 5.2 in Du et al.
(2021) showed that for finite dimensional setting with X ⊆ R d and || x || ≤ B , d eff ( X , ϵ )
= (cid:101) O
( d ) .
Moreover, the effective dimension can be small even for infinite dimensional
RKHS case.
In the next proposition, we prove that when the coupling function exhibits a bilinear structure
G ( f, g ) =
⟨ W ( f ) , X ( g ) ⟩ H with feature space X :=
{ X ( g ) ∈ H : g ∈ F} and ∥ X ( g )
∥
H ≤ √ B , the functional eluder dimension in Definition 4 is always less than the effective dimesion of X .
Proposition 35.
For any hypothesis class F and coupling function G ( · , · ) : F × F → R that can be expressed in bilinear form ⟨ W ( f ) , X ( g ) ⟩ H , we have
[Equation]
Proof of Proposition 35.
The proof basically follows the proof of Proposition 29 in Jin et al.
(2021) with modifications specified for the functional eluder dimension.
Given a hypothesis class F and a coupling function G ( · , · ) : F × F → R .
Suppose there exists an ϵ ’-independent sequence f 1 , . . .
, f n ∈ F such that there exist g 1 , . .
. , g n ∈ F ,
[Equation]
When G ( f, g ) := ⟨ W ( f ) , X ( g ) ⟩ H , the above becomes
[Equation]
Defining Σ t = (cid:80) t − 1 i =1 X
( f i ) X
( f i ) ⊤ + ϵ
′ 2 B
· I , we have by Eq.
(G.2) that ∥ W ( g t )
∥ Σ t ≤ √ 2 ϵ ′ .
Furthermore,
[Equation]
Thus, we have ∥ X ( f t )
∥
2Σ − 1 t ≥ 12 for any t ∈ [ n ] .
By applying the log-determinant argument, we have n (cid:88)
[Equation]
The above equality implies
[Equation]
Taking n = d eff ( X , ϵ/ √ B ) yields
[Equation]
which contradicts with the inequality (G.3) and concludes our proof.
We now provide the detailed proofs of Lemmas 28, 29 and 30.
G. P ROOF OF L EMMA
Proof of Lemma 28.
Taking
[Equation]
where W h ( f ) :
= θ h,f − θ ∗ h , X h ( g ) :
=
E s h ,a h ∼ π g [ ψ ( s h , a h ) + (cid:80) s ′
ϕ
( s h , a h , s ′ )
V h
+1 ,g ( s ′ )] in
Proposition 35.
Properties of the effective dimension yield that the FE dimension of the linear mixture
MDP model is ≤ (cid:101) O
( d ) .
G.
P ROOF OF L EMMA
Proof of Lemma 29.
Taking G h,f ∗ ( f, g ) :
= ⟨ W h ( f ) , X h ( g ) ⟩ in Proposition 35, and properties of the effective dimension yields the conclusion that the FE dimension of low Witness rank MDP model is ≤ (cid:101)
O
( W κ ) .
G. P ROOF OF L EMMA
We first introduce two auxillary lemmas:
Lemma 36.
Let random variable x i ∈ R d and E ∥ x i ∥ 22 ≤ B 2 .
Then we have that
[Equation]
Proof.
We first have
[Equation]
Therefore, using the Determinant-Trace inequality, we get the first result,
[Equation]
Dividing n from the both side of the inequality completes the proof.
The following lemma is a variant of the well-known Elliptical Potential Lemma (Dani et al., 2008;
Srinivas et al., 2009; Abbasi-Yadkori et al., 2011; Agarwal et al., 2020a).
Lemma 37 (Randomized elliptical potential) .
Consider a sequence of random vectors { x 0 , . .
, x T − 1 } .
Let λ > 0 and Σ
=
λI and Σ t = Σ 0 + (cid:80) t − 1 i =
E [ x i x ⊤ i ] , we have that min t
[Equation]
[ T ]
[Equation]
Proof.
By definition of Σ t we have that
[Equation]
Denote Λ t = Σ − 1 / 2 t
E [ x t x ⊤ t ](Σ t ) − 1 / 2 with eigenvalue λ 1 , . .
. , λ d ≥ 0 , we have that det( I + Λ t
[Equation]
=
[Equation]
1 ( λ i + 1) ≥ 1 +
[Equation]
i =
[Equation]
λ
[Equation]
(G.5)
Plugging (G.5) into (G.4) gives that
[Equation]
Taking telescope sum from t = 0 to t
=
T − 1 completes the proof.
Proof of Lemma 30.
Given a hypothesis class F and a coupling function G ( · , · ) : F × F → R . Let n to be defined as follows,
[Equation]
Then we have that n = (cid:101) O
( d ϕ ) .
We will prove dim FE ( F , G, ϵ ) ≤ n by contradiction.
Suppose that dim FE ( F , G, ϵ ) > n , there exists an ϵ ′ -independent (where ϵ ′ ≥ ϵ ) sequence f 1 , . . .
, f n ∈ F such that there exist g 1 , . .
. , g n ∈ F ,
[Equation]
Recall that the ABC function of KNR model is defiend as,
[Equation]
Therefore
[Equation]
condition (G.6) can be reduced to
[Equation]
Denote U h,g t ,j , j ∈
[ d s ] and U ∗ h,j , j ∈ [ d s ] to be the rows of U h,g t and U ∗ h .
Taking square over both side of the inequalities in (G.7) gives that
[Equation]
Define Σ t = (cid:80) t − 1 i =1 E s h ,a h ∼ π fi [ ϕ ( s h , a h ) ϕ
( s h , a h ) ⊤ ] + ( ϵ ′ 2 / 4 d s R 2 ) · I .
Then by (G.8), we have that d
[Equation]
where the first equality is by the Cauchy-Schwartz inequality and the last inequality is by ∥ U h,g t ,j ∥ 2 ≤
∥
U h,g t ∥ 2 ≤ R , ∥ U ∗ h,j ∥ 2 ≤ ∥ U ∗ h ∥ 2 ≤ R .
Furthermore we have that
[Equation]
where the last inequality is by the Cauchy-Schwarz inequality for random variables.
Thus, we have that E s h
[Equation]
a
[Equation]
∥ Σ − 1 / 2 t ϕ
[Equation]
s h , a h ) ∥
22 ≥ 1 / 2 for all t ∈ [ n ] .
By applying Lemma 37, we have that
[Equation]
The above equation further implies that
[Equation]
On the other hand, Lemma 36 implies that
[Equation]
n
[Equation]
This leads to a contradiction because ϵ ′ ≥ ϵ and log(3 / 2) >
e − 1 .
We complete the proof of dim FE ( F , G, ϵ ) =
(cid:101) O
( d ϕ ) .
[Figure]
H E X P E R I M E N T
In this section, we carry out experiments to evaluate the empirical performance of our algorithm
OPERA for linear mixture MDPs (Algorithm 2).
In this experiment, we construct an MDP M with dimension d = 3 and episode length H = 5 .
The state space S consists of H + 2 different states x 1 , . . .
, x H +2 and the action space
A
= {− 1 , 1 } d − 1 consists of 2 d − 1 different actions.
For each step h ∈
[ H ] and episode k ∈
[ K ] , we assume that the reward function r ( s, a ) is known (so no need to introduce ψ as in Section E.1).
In particular, for all 1 ≤ h ≤ H , the reward function r h ( s, a ) = 1 if
and only if s = x H +2 and r h ( s, a )
{+= 
+}
otherwise.
For each step h ∈
{+[ H ] and corresponding transition probability function P h , x H +
+}
and x H +2 are absorbing states.
For other states x h (1 ≤ h ≤ H ) , the transition probability satisfies that
[Equation]
where each 1 d − 1 is a ( d − 1)
-dimensional vector of all ones, a ∈ A is also a ( d − 1) -dimensional vector.
Then we have that P h ( x h ′ | x h , a ) =
⟨ θ ∗ h ,
ϕ ( x h
a
x
h ) ⟩ where θ ∗ h =
[0 . 01 · 1 d − 1 , 1] , and ϕ ( x h ′ , a , x h ) are as follows,
• ϕ ( x h ′ , a , x h ) =
[ − a , 0 .
95] if 1 ≤ h ≤ H and h ′ =
h + 1 , • ϕ ( x h ′ , a , x h ) =
[ a , 0 .
if 1 ≤ h ≤ H and h ′ =
H + 2 , • ϕ ( x h ′ , a , x h ) =
[ 0 d − 1 , 1] if h ′ =
h = H + 1 , • ϕ ( x h ′ , a , x h ) =
[ 0 d − 1 , 1] if h ′ =
{+h = H + 2 , • ϕ ( x h ′ , a , x h ) = 
+}
d , otherwise.
Here 0 d − 1 is a ( d − 1) -dimensional
vector of all zeros.
We compare our algorithm OPERA with the following two baselines: Optimal (optimal policy) and Random (uniformly random policy which chooses actions uniformly from A ).
For numerical stability, we add λI to Σ ( t ) h with λ = 1 and use CVX (Diamond & Boyd, 2016; Agrawal et al., 2018) to approximately solve (E.3) when we implement Algorithm 2.
The cumulative rewards of different algorithms averaged over 10 runs for the first 10000 episodes are plotted in Figure 2.
We can see that OPERA performs much better than the random policy and can converge to the optimal policy after 10000 episodes.