On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RL
Abstract
We study reward-free reinforcement learning (RL) under general non-linear function approximation, and establish sample efficiency and hardness results under various standard structural assumptions.
On the positive side, we propose the RFOLIVE (Reward-Free O LIVE ) algorithm for sample-efficient reward-free exploration under minimal structural assumptions, which covers the previously studied settings of linear MDPs (Jin et al., 2020b), linear completeness (Zanette et al., 2020b) and low-rank MDPs with unknown representation (Modi et al., 2021).
Our analyses indicate that the explorability or reachability assumptions, previously made for the latter two settings, are not necessary statistically for reward-free exploration.
On the negative side, we provide a statistical hardness result for both reward-free and reward-aware exploration under linear completeness assumptions when the underlying features are unknown, showing an exponential separation between low-rank and linear completeness settings.
Introduction
Designing a reward function which faithfully captures the task of interest remains a central practical hurdle in reinforcement learning (RL) applications.
To address this, a series of recent works (Jin et al., 2020a; Zhang et al., 2020b; Wang et al., 2020a; Zanette et al., 2020b; Qiu et al., 2021) investigate the problem of reward-free exploration, where the agent initially interacts with its environment to collect experience (“online phase”), that enables it to perform offline learning of near optimal policies for any reward function from a potentially pre-specified class (“offline phase”).
Reward-free exploration also provides a basic form of multitask RL, enabling zero-shot generalization, across diverse rewards, and provides a useful primitive in tasks such as representation learning (Agarwal et al., 2020; Modi et al., 2021).
So far, most of the study of reward-free RL has focused on tabular and linear function approximation settings, in sharp contrast with the literature on reward-aware RL,
Setting
Reference
Linear MDP
Wang et al.
(2020a)
Zanette et al.
(2020b)
Theorem Theorem
Modi et al.
Theorem 5 (intractable)
The detailed comparisons of existing sample complexity rates and our corollaries can be found in
Appendix A. where abstract structural conditions identify when general function approximation can be used in a provably sample-efficient manner (Jiang et al., 2017; Jin et al., 2021; Du et al., 2021).
In this paper, we seek to bridge this gap and undertake a systematic study of reward-free RL in a model-free setting with general function approximation.
We devise an algorithm, RFO LIVE , which is non-trivially adapted from its reward-aware counterpart (Jiang et al., 2017), and provide polynomial sample complexity guarantees under general conditions that significantly relax the assumptions needed by prior reward-free RL works.
Our results produce both algorithmic contributions and important insights about the tractability of reward-free RL, as we summarize below (see also Table 1).
Algorithmic contribution: beyond linearity A unique challenge in reward-free RL is that the agent must exhaustively explore the environment during the online phase, since it does not know which states will be rewarding in the offline phase.
A natural idea to tackle this challenge is to deploy a reward-aware RL “base algorithm” with the 0 reward function, since this algorithm must explore to certify that there is indeed no reward.
Prior works adopt this idea with optimistic value-iteration (VI) approaches, which use proxy reward functions to drive the agent to new states.
However these optimistic methods rely heavily on linearity assumptions to construct the proxy reward, and it is difficult to extend them to general function approximation.
Instead of using optimistic VI, our basic building block is the O LIVE 2 algorithm of Jiang et al.
(2017), a constraint-gathering and elimination algorithm that is a central workhorse for reward-aware RL with general function approximation.
In the online phase of RFO LIVE , we run this algorithm with the 0 reward function, and we save the set of constraints gathered (in the form of separate datasets) for use in the offline phase.
Algorithmic contribution: novel offline module Prior works for reward-free RL typically use regression approaches (Ernst et al., 2005; Chen and Jiang, 2019; Jin et al., 2020b) in the offline phase, e.g., FQI (Modi et al., 2021; Zanette et al., 2020b), or its optimistic variants (Zhang et al., 2020b;
Wang et al., 2020a).
In the offline phase of RFO LIVE , rather than relying on regression, we enforce the constraints gathered in the online phase, which amounts to eliminating functions that have large average Bellman errors on state-distributions visited in the online phase.
This generic elimination scheme does not rely on tabular or linear structures and allows us to move beyond these assumptions to obtain reward-free guarantees in much more general settings.
Implications: positive results The major assumptions that enable our sample complexity guarantees are Bellman-completeness (Assumption 2) and low Bellman Eluder dimension (Definition 5 and
Definition 7); see Rows 3 and 4 in Table 1.
These conditions significantly relax prior assumptions in the more restricted settings.
Furthermore, prior works in the linear completeness and low-rank
MDP settings require explorability/reachability assumptions (Zanette et al., 2020b; Modi et al., 2021), which, roughly speaking, assert that every direction in the state-action feature space can be visited with sufficient probability.
These assumptions are often not needed in reward-aware RL but suspected to be necessary for model-free reward-free settings.
Our results do not depend on such assumptions, showing that they are not necessary for sample-efficient reward-free exploration either.
Implications: negative results We develop lower bounds, showing that some of the structural assumptions made here are not easily relaxed further.
While the settings of linear completeness with known features (Row 3), and low-rank MDPs with unknown features (Row 4) are both independently tractable, we show a hardness result against learning under linear completeness when the features are unknown, even under a few additional assumptions (Row 6).
Taken together, our results take a significant step in bridging the sizeable gap in our understanding of reward-aware and reward-free settings and bring the two closer to an equal footing.
Related work In recent years, we have seen a wide range of results for reward-aware RL under general function approximation (Jiang et al., 2017;
Dann et al., 2018; Sun et al., 2019; Wang et al., 2020c; Jin et al., 2021; Du et al., 2021).
These works develop statistically efficient algorithms using structural assumptions on the function class.
Despite their generality, a trivial extension to the reward-free setting incurs an undesirable linear dependence on the size of the reward class.
There also exists a line of research on reward-free RL in various settings: tabular MDPs (Jin et al., 2020a; Zhang et al., 2020b;
Kaufmann et al., 2021; Ménard et al., 2021; Yin and Wang, 2021; Wu et al., 2022), MDPs with the linear structure (Wang et al., 2020a;
Zhang et al., 2021; Zanette et al., 2020b; Huang et al., 2021; Wagenmaker et al., 2022), kernel MDPs (Qiu et al., 2021), block/low-rank
MDPs (Misra et al., 2020; Agarwal et al., 2020; Modi et al., 2021), and multi-agent settings (Bai and
Jin, 2020; Liu et al., 2021).
Many of these settings can be subsumed by our more general setup.
Our offline module uses average Bellman error constraints, which is related to a line of work in offline
RL (Xie and Jiang, 2020; Jiang and Huang, 2020; Chen and Jiang, 2022; Zanette and Wainwright, 2022).
However, there is only one dataset in the standard offline RL setting, and these works form multiple average Bellman error constraints using an additional helper class for reweighting, and need to impose additional realizability- or even completeness-type assumptions on such a class.
In contrast, we naturally collect multiple datasets in the online phase, so we do not require a parametric class for reweighting during offline learning.
Preliminaries
Markov Decision Processes (MDPs)
We consider a finite-horizon episodic Markov decision process (MDP) defined as M =
( X , A , P, H ) , where X is the state space,
A is the action space,
P =
( P 0 , . .
, P H − 1 ) with P h :
X ×
A → ∆( X ) is the transition dynamics, and H is the number of timesteps in each episode.
If the number of actions is finite, we denote the cardinality |A| by
K .
In each episode, an agent generates a trajectory τ = ( x 0 , a 0 , x 1 , . . .
, x H − 1 , a H − 1 , x H ) by taking a sequence of actions a 0 , . .
. , a H − 1 , where x 0 is a fixed starting state and x h +1 ∼ P h ( · | x h , a h ) .
For simplicity, we will use a i : j to denote a i , . . .
, a j and use the notation [ H ] to refer to { 0 , 1 , . . .
, H − 1 } .
We use the notation π to denote a collection of H (deterministic) policy functions π
= ( π 0 , . .
, π H − 1 ) , where π h : X → A .
For any h ∈ [ H ] with h ′ >
h , we use the notation π h : h ′ to denote the policies ( π h ,
π h +1 . . .
, π h ′ ) .
For any policy π and reward function
R =
( R 0 , . . .
, R H − 1 ) with R h :
X × A → [0 , 1] , we define the policy-specific action-value (or
Q-) function as Q πR,h ( x, a ) =
E π [ (cid:80) H − 1 h ′ =
h R ( x h ′ , a h ′ )
| x h = x, a h = a ] and state-value function as V πR,h ( x ) =
E π [ Q πR,h ( x, a h ) | x h = x, a h ∼ π ] .
We also use v πR =
V πR, 0 ( x 0 ) to denote the expected return of policy π .
For any fixed reward function R , there exists a policy π ∗ R such that v ∗ R =
V π ∗ R R,h ( x ) =
sup π V πR,h ( x ) for all x ∈ X and h ∈
[ H ] , where v ∗ R denotes the optimal expected return under R .
We use T Rh to denote the reward-dependent Bellman operator: ∀ f h +1 ∈ R X×A , ( T R h f h +1 )( x, a ) :
= R h ( x, a ) + E [max a ′ ∈A f
h
+
( x ′ , a ′ )
| x ′ ∼ P h ( · | x, a )] and similarly define
T 0 h for the operator with zero reward.
The optimal action-value function (under reward R ) Q ∗ R satisfies the Bellman optimality equation Q ∗ R,h = T Rh Q ∗ R,h +1 , ∀ h ∈
[ H ] .
Reward
-free RL with function approximation
We study reward-free RL with value function approximation, wherein, the agent is given a function class F = F 0 × . . .
× F H − 1 where F h :
X ×
A → [ − ( H − h − 1) , H − h − 1] , ∀ h ∈
[ H ] .
4 Without loss of generality, we assume 0 ∈ F h , ∀ h ∈
[ H ] and f H ≡ 0 , ∀ f ∈ F .
For any f ∈ F , we use V f,h to denote its induced state-value function, i.e.,
V f,h ( x ) = max a f h ( x, a ) and
π f ( x ) as its greedy policy, i.e., π f,h ( x ) = argmax a f h ( x, a ) .
When these functions take x h as input and there is no confusion, we may drop the subscript h and use
V f ( x h ) and π f ( x h ) .
In reward-free RL, the agent is given access to a reward class R , but the specific reward function is only selected after the agent finishes interacting with the environment.
Specifically, the agent operates in two phases: an online phase where it explores the given MDP M to collect a dataset of trajectories D without the reward information, and an offline phase, where it uses the collected dataset
D to optimize for any revealed reward function R ∈ R .
Our goal is to investigate the statistical efficiency of reward-free RL with general non-linear function approximation: how many trajectories does the agent need to collect in the online phase such that in the offline phase, with probability at least 1 − δ , for any R ∈ R , it can compute a near-optimal policy π R satisfying v π R R ≥ v ∗ R − ε ?
We measure the statistical efficiency in terms of the structural complexity of function class F , reward class R , horizon H , accuracy ε and failure probability δ .
As for expressivity assumptions, we assume the function class F is realizable and complete.
Realizability requires that the optimal function Q ∗ R belongs to the reward-appended class F + R , which is natural in the reward-free setting where the agent uses F to capture reward-independent information.
Completeness requires that the Bellman backups of and F h +
+ R h
+1 belong to F h , and additionally that the Bellman backup of F h +1 − F h +1 belongs to F h − F h .
Assumption 1 (Realizability of the function class) .
We assume ∀ R ∈ R , h ∈
[ H ] , Q ∗ R,h ∈ F h + R h , where F h + R h
= { f h + R h : f h ∈ F h } .
Assumption 2 (Completeness) .
We assume ∀ h ∈
[ H ] , T
h F h +1 , T 0 h
( F h +
+ R h +1 ) ⊆
F h and
T 0 h ( F h
+1 − F h +1 ) ⊆
F h − F h , where F h − F h = { f h − f ′ h : f h , f ′ h ∈ F h } .
Next we define the covering number, which measures the statistical capacities of function classes.
Definition 1 ( ε -covering number, e.g., Wainwright (2019)) .
We use N F ( ε ) to denote the ε -covering number of a set F = F 0 × . . .
×F H − 1 under metric σ ( f, f ′ )
=
max h ∈ [ H ] ∥
f h − f ′ h ∥ ∞ for f, f ′ ∈ F .
We define it as N F ( ε ) =
min |F cover | such that F cover ⊆ F and for any f ∈ F , there exists f ′ ∈ F cover that satisfies σ ( f, f ′ ) ≤ ε .
For the reward class R , N R ( ε ) is defined in the same way.
Finally, as our guarantees depend on Bellman Eluder (BE) dimensions—which are structural properties of the MDP that enable sample-efficient exploration—we will need the following definitions (see
Russo and Van Roy, 2013; Jin et al., 2021) which the later definitions of BE dimensions will build on.
Definition 2 ( ε -independence between distributions) .
Let F ′ be a function class defined on some space X ′ , and ν, µ 1 , . . .
, µ n be probability measures over X ′ .
We say ν is ε -independent of { µ i } ni = w.r.t.
F ′ if ∃ f ′ ∈ F ′ such that (cid:112)(cid:80) ni
=
( E µ i [ f ′ ]) 2 ≤ ε , but | E ν [ f ′ ] | > ε .
Definition 3 (Distributional Eluder (DE) dimension) .
Let F ′ be a function class defined on some space X ′ , and Γ ′ be a family of probability measures over X ′ .
The DE dimension d de ( F ′ , Γ ′ , ε ) is the length of the longest sequence { ρ i } ni
=1 ⊆ Γ ′ s.t.
∃ ε ′ ≥ ε where ρ i is ε
′ -independent of { ρ j } i − 1 j =1 , ∀ i = 1 , . . . , n .
We also introduce the notation D F :=
{D F ,h } h ∈ [ H ] , where D F ,h denotes the collection of all possible roll-in distributions at the h -th step generated by π f for some f ∈ F .
Formally, D F ,h :=
{ d π f h } f ∈F where d π f h ( x, a ) =
P π f [ x h = x, a h
= a ] is the state-action occupancy measure.
RFO LIVE algorithm and results
In this section, we describe our main algorithm RFO LIVE , a reward-free variant of O LIVE (Jiang et al., 2017; Jin et al., 2021).
The algorithmic template for RFO LIVE is shown in the pseudocode
(Algorithm 1) and it can be instantiated with both Q-type and V-type versions of O LIVE from Jin et al.
In the pseudocode, we use □ as a placeholder for the respective Q/V-type definitions.
For clarity, we will describe the Q-type RFO LIVE algorithm and its results in Section 3.1 and then state the differences for the V-type version and corresponding results in Section 3.2.
Before introducing our algorithm, we define the following average Bellman error:
Definition 4 (Average Bellman error) .
We denote E R as the average Bellman error under reward R :
[Equation]
As shorthand, we use E R Q ( f, π, h ) =
E R ( f, π, π, h ) to represent the Q-type average Bellman error and E R V ( f, π, h ) =
E R ( f, π, π f , h ) to represent the V-type average Bellman error (Jin et al., 2021).
We use E 0 to represent the average Bellman errors under 0 reward.
actv trajectories { ( x ( i ) 0 ,
( i ) 0 , . . .
, x ( i ) H − 1 , a ( i ) H − 1 ) } n actv
i
=
following π t for all h ∈
[ H ] and form estimates ˜
E 0 ( f t , π t , π t , h ) for each h ∈
[ H ] via Eq.
6: if (cid:80) H − 1 h =
˜ E 0 ( f t , π t , π t , h ) ≤
Hε actv then 7:
Set T = t and exit the loop.
8: end if 9:
Pick any h t ∈
[ H ] for which ˜ E 0 ( f t , π t , π t , h t ) > ε actv .
Set π est =
π t (Q-type) or π est = Unif ( A ) , i.e., draw actions uniformly at random (V-type).
Collect n elim samples D t = { ( x ( i ) h t , a ( i ) h t , x
( i ) h t +1 ) } n elim i =1 where a 0: h t − 1 ∼ π t and a h t ∼ π est .
For all f ∈ F t , compute estimate ˆ E
□
( f, π t , h t ) via Eq.
(2) (Q-type) or Eq.
(4) (V-type).
Update F t +1 = { f ∈ F t : | ˆ E 0 □ ( f, π t , h t ) | ≤ ε elim } .
14: end for 15: Save the collected tuples { ( h t , π t , D t ) } T − 1 t =0 for the offline phase.
Offline phase, the reward function R =
( R 0 , . . .
, R H − 1 ) is revealed.
Construct F off ( R ) =
F + R , set Π t est = { π t } (Q-type) or Π t est =
Π on := { π f : f ∈ Z on } (i.e., the greedy policies induced by Z on ) (V-type).
17: For each t ∈ [ T ] , g ∈ F off ( R ) , and π ∈ Π t est , compute estimate ˆ E R ( g, π t , π, h t ) via Eq.
(3) (Q-type) or
3.1 Q-type RFO LIVE
Our algorithm, reward-free O LIVE (RFO LIVE ) described in Algorithm 1, takes the function class F , the accuracy parameter ε , and the failure probability δ as input.
As we are in the reward-free setting, it operates in two phases: an online exploration phase where it collects a dataset without an explicit reward signal, and an offline phase where it computes a near-optimal policy after the reward function
R is revealed.
Below, we describe the two phases and the intuition behind the algorithm design.
Online exploration phase During the online phase, we first set elimination thresholds ε actv , ε elim and sample sizes n actv , n elim and construct the following function class F on used in the online phase:
[Equation]
The detailed specification of these parameters are deferred to Theorem 1 and Theorem 3.
Subsequently, we simulate Q-type O LIVE with the function class F on using the zero reward function R = 0 and the specified parameters.
Similar to O LIVE , we initialize F 0 =
F on and maintain a version space
F t ⊆
F t − 1 ⊆ F on of surviving functions after each iteration.
In each iteration, we first find the optimistic function f t ∈ F t (line 4) and set π t = π f t .
In line 5, we collect n actv trajectories to estimate the Q-type average Bellman error ˜
E 0 Q ( f t , π t , h ) =
˜ E
( f t , π t , π t , h ) under zero reward:
[Equation]
If the low average Bellman error condition in line 6 is satisfied, then we terminate the online phase and otherwise, we pick a step h t where the estimate ˜ E
( f t , π t , π t , h t ) > ε actv (line 9).
Then we collect n elim trajectories using a 0: h t ∼ π t and set D t as the transition tuples at step h t .
Using D t , we construct the Q-type average Bellman error estimates ˆ E 0 Q ( f, π t , h t ) for all f ∈ F t in line 12:
[Equation]
Finally, in line 13, we eliminate all the f ∈ F t whose average Bellman error estimate ˆ E
[Equation]
Q ( f,
[Equation]
, h
[Equation]
> ε elim .
The online phase returns tuples { ( h t , π t , D t ) } T − 1 t =0 where T is the total number of iterations and each dataset D t consists of n elim transition tuples.
Offline elimination phase In the offline phase, the reward function R is revealed, and we first construct the reward-appended function class F off ( R )
=
F + R := { ( f 0 + R 0 , . . .
, f H − 1 + R H − 1 ) : f h ∈
F h , ∀ h ∈
[ H ] } .
Using the class Π t est = { π t } from line 16 and the collected tuples { ( h t , π t , D t ) } T − 1 t =0 , we estimate the reward-dependent average Bellman error
(Definition 4) for all iterations t ∈
[ T ] of the online phase:
[Equation]
RFO LIVE eliminates all g ∈ F off ( R ) whose average Bellman error estimates are large (line 18) and returns the optimistic function ˆ g from the surviving set (line 19).
Remark Similar to its counterparts in reward-aware general function approximation setting (Jiang et al., 2017; Dann et al., 2018; Jin et al., 2021; Du et al., 2021), RFO LIVE is in general not computationally efficient.
We leave addressing computational tractability as a future direction.
3.1.1 Main results for Q-type RFO LIVE
In this part, we present the theoretical guarantee of Q-type RFO LIVE .
We start with introducing the
Q-type Bellman Eluder (BE) dimension (Jin et al., 2021).
Definition 5 (Q-type BE dimension) .
Let ( I − T Rh ) F :=
{ f h − T Rh f h +1 : f ∈ F} be the set of
Bellman differences of F at step h , and Γ = { Γ h } H − 1 h =0 where Γ h is a set of distributions over X × A .
The ε -BE dimension of F w.r.t.
Γ is defined as dim R qbe ( F , Γ , ε ) :
= max h ∈
[ H ] d de (cid:0) ( I −T Rh ) F , Γ h , ε (cid:1) .
We can now state our sample complexity result for Q-type RFO LIVE .
To simplify presentation, we state the result here assuming parametric growth of the covering numbers, that is log( N F ( ε )) ≤ d F log(1 /ε ) and log( N R ( ε ))
≤ d R log(1 /ε ) .
Theorem 1 (Q-type RFO LIVE , parametric case) .
Fix δ ∈ (0 , 1) .
Given a reward class R and a function class F that satisfies Assumption 1 and Assumption 2, with probability at least 1 − δ , for any
R ∈ R , Q-type RFO LIVE (Algorithm 1) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
where d qbe = dim 0 qbe ( F − F , D F−F , ε/ (4 H )) .
The
The more general statement along with the specific values of ε actv , ε elim , n actv , n elim are deferred to
Appendix C.2, where we also present the proof.
We remark that we only need the covering number of R to set these parameters and do not use any other information about the reward class.
We pause to compare Theorem 1 to the reward-aware case.
First, our BE dimension involves the “difference” function class F − F under zero reward as opposed to the original class with the given reward, and our completeness assumption is also related to such a “difference” function class.
As we will see, these differences are inconsequential for our examples of interest.
Second, our sample complexity has an additional H 4 dependence because (a) we consider a different reward normalization from Jiang et al.
; Jin et al.
(2021) and (b) we use a smaller threshold in the online phase to ensure sufficient exploration.
Similar gaps in H factors between reward-free and reward-aware learning also appear in Wagenmaker et al.
We also pay for the complexity of R in a lower order term, which is standard in reward-free RL (Zhang et al., 2020a; Modi et al., 2021).
We believe that a similar adaptation of G OLF (Jin et al., 2021) for the reward-free setting may provide a sharper result with improved dependence on H and d qbe , analogously to the reward-aware setting.
3.1.2 Q-type RFO LIVE for known representation linear completeness setting
Here, we instantiate the general guarantee of Q-type RFO LIVE to the linear completeness setting.
Definition 6 (Linear completeness setting (Zanette et al., 2020b)) .
We call feature ϕ
lc =
( ϕ lc0 , . .
ϕ lc H − 1 ) with ϕ lc
h : X ×
A → R d lc , ∥ ϕ lc h ( · ) ∥ 2 ≤ 1 , ∀ h ∈
[ H ] a linearly complete feature, if for any B > 0 , h ∈
[ H − 1] and ∀ f h +1 ∈ Q h +1 ( { ϕ lc } , B ) we have: min
[Equation]
∥
[Equation]
−
T
h f h +1 ∥ ∞
= 0 , where Q h ( { ϕ lc } , B ) = {⟨ ϕ lc
h , θ h ⟩ : ∥ θ h ∥ 2 ≤ B √ d lc } .
When the linearly complete features (Definition 6) ϕ
lc are known, we can construct the function class
F ( { ϕ lc } )
=
F 0 ( { ϕ lc } , H − 1) × . . .
× F H − 1 ( { ϕ lc } , 0) , where F h ( { ϕ lc } , B h ) = (cid:8) f h ( x h , a h ) =
(cid:10) ϕ
lc h
( x h , a h ) , θ h (cid:11) :
∥ θ h ∥ 2 ≤ B h √ d lc , ⟨ ϕ lc h ( · ) , θ h ⟩ ∈ [ − B h , B h ] (cid:9) consists of appropriately bounded linear functions of ϕ lc .
Here superscript and subscript
lc imply that the notations are related to the linear completeness setting.
It is easy to verify that F ( { ϕ lc } ) satisfies the assumptions in Theorem 1.
This gives us the following corollary (see the full statement and the proof in Appendix C.4):
Corollary 2 ( Informal ) .
Fix δ ∈ (0 , 1) .
Consider an MDP M that satisfies linear completeness (Definition 6) with known feature ϕ lc , and the linear reward class R = R 1 × . . .
× R h , where
R h = (cid:8) ⟨ ϕ lc h , η h ⟩ : ∥
η h ∥ 2 ≤ √ d lc , ⟨ ϕ lc h ( · ) , η h ⟩ ∈ [0 , 1] (cid:9) .
With probability at least 1 − δ , for any
R ∈ R , Q-type RFO LIVE (Algorithm 1) with F = F ( { ϕ lc } ) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of samples is ˜ O (cid:0) H
d 3lc log(1 /δ )
/ε 2 (cid:1) .
The reward normalization above, called explicit regularity in Zanette et al. (2020b), is standard.
Compared to that work, our result implies that explorability is not necessary, which significantly relaxes the existing assumptions for this setting.
Our result can also be easily extended to handle approximately linearly complete features (i.e., low inherent Bellman error).
On the other hand, our algorithm is not computationally efficient owing to our general function approximation setting.
Although our sample complexity bound appears to be worse in H factors compared with their upper bound of ˜ O (cid:0) d 3lr H 5 log(1 /δ )
/ε 2 (cid:1) , it is indeed incomparable because their bound only holds when ε ≤ ˜ O
( ν min / √ d lc ) ( ν min is their explorability factor).
Thus, there is an implicit dependence on 1 /ν min in their result, which could make the bound arbitrarily worse than ours.
More discussions are deferred to Appendix A and Appendix C.4.
3.2 V-type RFO LIVE
In this section, we describe the instantiation of RFO LIVE with V-type definitions.
For V-type
RFO LIVE , we also assume that the action space is finite with size K .
Online exploration phase Instead of using F on , we use its ( ε elim / 64) -cover
Z on and maintain a version space F t across iterations.
8 Since the on-policy version of Q-type and V-type Bellman errors are the same, the termination check in line 5 and line 6 are unchanged.
If the algorithm does not terminate in line 6, we again identify a deviation step h t such that ˜ E 0 V ( f t , π t , h t ) =
˜ E
( f t , π t , π t , h t ) > ε actv .
Instead of using π t to collect trajectories, we use a 0: h t − 1 ∼ π t and choose a h t uniformly at random to collect the dataset of n elim transition tuples at step h t .
Compared to
Q-type RFO LIVE , we estimate ˆ E 0 V for all f ∈ F t in line 12 using importance sampling (IS):
[Equation]
Finally, in line 13, we eliminate all f ∈ F t whose V-type average Bellman error estimates are large.
Offline elimination phase In the offline phase, we consider the same reward-appended function class F off ( R ) when reward R ∈ R is revealed.
For V-type RFO LIVE , in line 16, we define the policy class Π t est =
Π on which consists of greedy policies with respect to all f ∈ Z on .
Using dataset D t , we estimate E R ( g, π t , π ′ , h t ) for all g ∈ F off ( R ) , π ′ ∈ Π on ,
t ∈
[ T ] from its empirical version: ˆ E R ( g, π t , π ′ , h t ) = n elim
[Equation]
and eliminate invalid functions in line 18.
Finally, we return the optimistic policy ˆ π from the surviving set.
Apart from estimating different average Bellman errors, the noticeable difference between Q-type and V-type RFO LIVE is that the latter uses IS to correct the uniformly drawn action to some policy π ′ ∈ Π on to witness the average Bellman error (Jiang et al., 2017).
3.2.1 Main results for V-type RFO LIVE
Here we present the theoretical guarantee of V-type RFO LIVE .
Firstly, we introduce the V-type
Bellman Eluder (BE) dimension (Jin et al., 2021).
Definition 7 (V-type BE dimension) .
Let ( I − T Rh ) V F ⊆
( X → R ) be the state-wise Bellman difference class of F at step h defined as ( I − T Rh ) V F := (cid:8) x
(cid:55)→ ( f h − T Rh f h +1 )
( x, π f h ( x )) : f ∈
F (cid:9) .
Let Γ =
{ Γ h } H − 1 h =0 where Γ h is a set of distributions over X .
The V-type ε -BE dimension of F with respect to Γ is defined as dim R vbe ( F , Γ , ε ) :
= max h ∈
[ H ] d de (cid:0) ( I − T Rh ) V F , Γ h , ε (cid:1) .
We now state the guarantee for V-type RFO LIVE , assuming polynomial covering number growth.
Theorem 3 (V-type RFO LIVE , parametric case) .
Fix δ ∈ (0 , 1) .
Given a reward class R , a function class F that satisfies Assumption 1, Assumption 2, with probability at least 1 − δ , for any R ∈ R ,
V-type RFO LIVE outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
log(1 /δ )
/ε 2 (cid:1) , where d vbe = dim 0 vbe ( F − F , D F−F , ε/ (8 H )) .
The detailed proof and the specific values of ε actv , ε elim , n actv , n elim are deferred to Appendix D.2.
Our rate is again loose in H factors when compared with the reward-aware version.
Compared with the
Q-type version
, here we also incur a dependence on K = |A| , analogous to the reward-aware case.
3.2.2 V-type RFO LIVE for unknown representation low-rank MDPs
As a special case, we instantiate our V-type RFO LIVE result to low-rank MDPs (Modi et al., 2021):
Definition 8 (Low-rank factorization) .
A transition operator P h :
X ×
A → ∆( X ) admits a lowrank decomposition of dimension
d lr if there exists ϕ lr h : X × A → R d lr and µ lr h : X → R d lr s.t.
∀ x, x ′ ∈ X , a ∈ A : P h ( x ′ | x, a ) =
(cid:10) ϕ
lr h ( x, a ) , µ lr h ( x ′ ) (cid:11) , and additionally ∥
ϕ lr h ( · ) ∥ 2 ≤ 1 and ∀ f ′ :
X → [ − 1 , 1] , we have (cid:13) (cid:13) (cid:82) f ′ ( x ) µ lr h ( x ) dx (cid:13) (cid:13) 2 ≤ √ d lr .
We say M is low-rank with embedding dimension d lr , if for each h ∈ [ H ] , the transition operator P h admits a rank- d lr decomposition.
Here superscript and subscript
lr imply that the notations are related to low-rank MDPs.
As in Modi et al.
(2021), we consider low-rank MDPs in a representation learning setting, where we are given realizable feature class Φ lr rather than the feature ϕ
lr
=
( ϕ lr0 , . .
ϕ lr H − 1 ) directly:
Assumption 3 (Realizability of low-rank feature class) .
We assume that a finite feature class Φ lr = Φ lr0 × . . .
×
Φ lr H − 1 satisfies ϕ lr h ∈
Φ lr h , ∀ h ∈
[ H ] .
In addition, ∀ h ∈ [ H ] , ϕ h ∈
Φ lr h , ∥ ϕ h ( · )
∥ 2 ≤ 1 .
Similar to the linear completeness setting (Section 3.1.2), we construct F (Φ lr )
=
F 0 (Φ lr , H − 1) × . . .
×F H − 1 (Φ lr , 0) , where F h (Φ lr , B h ) =
{ f h ( x h , a h ) = ⟨ ϕ h ( x h , a h ) , θ h ⟩ : ϕ h ∈
Φ lr h , ∥ θ h ∥ 2 ≤
B h √ d lr , ⟨ ϕ h ( · ) , θ h ⟩ ∈ [ − B h , B h ] } .
In Proposition 4, we show that the V-type Bellman Eluder dimension of F (Φ lr ) − F (Φ lr ) in this case is ˜ O
( d lr ) which leads to the following corollary:
Corollary 4 ( Informal, parametric case ) .
Fix δ ∈ (0 , 1) .
Consider a low-rank MDP M of embedding dimension d lr with a realizable feature class Φ lr (Assumption 3) and a reward class R .
With probability at least 1 − δ , for any R ∈ R , V-type RFO LIVE (Algorithm 1) with F (Φ lr ) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
We defer the full statement and detailed proof of the corollary to Appendix D.3.
In the low-rank
MDP setting, Modi et al.
(2021) propose a more computationally viable algorithm, but additionally require a reachability assumption.
Our result shows that reachability is not necessary for statistically efficiency, which opens an interesting avenue for designing an algorithm that is both computationally and statistically efficient without reachability.
Moreover, our result significantly improves upon their sample complexity bound.
The detailed comparisons are deferred to Appendix A and Appendix D.3.
Notice that here K shows up in our bound.
As another corollary, in the linear MDP (Definitionplus ϕ lr is known), Q-type RFO LIVE yields a K independent bound.
The details can be found in
Appendix C.5.
3.3 Intuition and proof sketch for RFO LIVE
We first provide the intuition.
Since the online phase of RFO LIVE is equivalent to running O LIVE with 0 reward function, any policy π f attains zero value (i.e., V π f 0 , 0 ( x 0 ) = 0 , ∀ f ∈ F on ).
By the policy loss decomposition lemma (Jiang et al., 2017), the value error for the greedy policy, V f ( x 0 ) − V π f 0 , 0 ( x 0 ) , is small when the algorithm stops (line 6).
Therefore, all f ∈ F on which predict large values V f ( x 0 ) must have been eliminated before O LIVE terminates.
This implies that, in the online phase, we gather a diverse set of constraints (roll-in distributions π t ) that can witness the average Bellman error of functions in F on .
In this sense, our algorithm focuses on function space elimination and does not try to reach all latent states or directions (Modi et al., 2021; Zanette et al., 2020b), which is the key conceptual difference that enables us to avoid reachability and explorability assumptions.
On the technical side, note that the way we use O LIVE in the offline setting is novel to our knowledge and is crucial to getting a good sample complexity under our assumptions, as opposed to more standard FQI style approaches.
Because we have to coordinate between the online and offline phases, the analysis bears significant novelty beyond the original analysis of O LIVE (and its reward-aware follow-up works), and this is one of our key contributions.
The most crucial part is to show that any bad g ∈ F off ( R ) whose average Bellman error is large under the true reward R will be eliminated in the offline phase.
To prove this, we construct ˜ f ∈ F on that has the same average Bellman error as g and predicts a large positive value V ˜ f
( x 0 ) , which implies that it will be eliminated during the online phase.
Finally, by our construction, the constraint used to eliminate ˜ f
directly witnesses the average Bellman error of g , thus ruling out g in the offline phase.
We discuss it in more detail in
Appendix C.3.
Hardness result for unknown representation linear completeness setting
In Section 3.1.2, we showed that Q-type RFO LIVE requires polynomial sample for reward-free RL in the known feature linear completeness setting.
For low-rank MDPs, when given a realizable feature class, we showed V-type RFO LIVE is statistically efficient in Section 3.2.2.
A natural next step is to relax the low-rank assumption on the MDP and show a sample efficiency result for the more general linear completeness and unknown feature case.
However, below we state a hardness result which shows that a polynomial dependence on the feature class ( (cid:12)(cid:12) Φ lc (cid:12)(cid:12) ) or an exponential dependence on H is unavoidable .
We first introduce the realizability of a linearly complete feature class.
Assumption 4 (Realizability of the linearly complete feature class) .
We assume that there exists a finite candidate feature class Φ lc =
Φ lc0 × · · · × Φ lc H − 1 , such that ∀ h ∈
[ H ] , we have ϕ
lc h ∈
Φ lc h .
In addition, ∀ h ∈ [ H ] , ϕ h ∈ Φ lc h , ∀ ( x, a ) ∈ X × A , we have ∥ ϕ h ( · ) ∥ 2 ≤ 1 .
Now we state of hardness result for learning in the linear completeness setting with a realizable feature class (Assumption 4).
A complete proof and more discussions are provided in Appendix E.
Theorem 5.
There exists a family of MDPs M , a reward class R and a feature set Φ lc , such that ∀ M ∈ M , the ( M, Φ lc ) pair satisfies Assumption 4, yet it is information-theoretically impossible for an algorithm to obtain a poly (cid:0) d lc , H, log( | Φ lc | ) , log( |R| ) , 1 /ε, log(1 /δ ) (cid:1) sample complexity for reward-free exploration with the given reward class R .
The hardness result in Theorem 5 is also applicable to easier settings: (i) learning with a generative model (or using a local access protocol, Hao et al.
, (ii) reward-free learning with explorability (Zanette et al., 2020b) and reachability (Modi et al., 2021) assumptions and (iii) reward-aware learning as R is a known singleton class.
Thus, the result highlights an exponential separation between the low-rank MDP and linear completeness assumptions by showing that linearly complete true feature ϕ lc ∈ Φ lc is not sufficient for polynomial sample efficiency and additional assumptions are required to account for the unknown representation.
Conclusion and discussion
In this paper, we investigated the statistical efficiency of reward-free RL under general function approximation.
The proposed algorithm, RFO LIVE , is the first algorithm to address reward-free exploration under general function approximation.
Contrary to prior works which either try to reach all states or all directions in the feature space, RFO LIVE follows a value function elimination template and ensures that the collected exploration data can be used to identify and eliminate nonoptimal value functions for downstream planning.
This significantly sets us apart from the existing reward-free exploration works.
Our positive results significantly relax the existing assumptions in the reward-free exploration framework.
Our negative result shows the first sharp separation between low-rank MDP and the linear completeness settings with unknown representations.
In addition, we provide an algorithm-specific counterexample in Appendix F that shows RFO LIVE can fail when the completeness assumption is violated.
As realizability alone is sufficient for reward-aware RL (Jiang et al., 2017; Jin et al., 2021; Du et al., 2021), our results also elicit the further question:
Are realizability-type assumptions sufficient for statistically efficient reward-free RL?
We conjecture that the answer is no, and we believe that the hardness between reward-aware and reward-free RL has a deep connection to the sharp separation between realizability and completeness (Chen and Jiang, 2019;
Wang et al., 2020b, 2021; Xie and Jiang, 2021; Weisz et al., 2021a,b, 2022;
Foster et al., 2021).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Table 1, Theorem 1, Theorem 3, Theorem 5, Corollary 2, Corollary 4, Corollary 6.
(b) Did you describe the limitations of your work?
[Yes] Our results hold for assumed structural properties and the RFO LIVE algorithm is not computationally efficient as our focus is statistical efficiency.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?[Yes]
See Theorem 1, Theorem 3, Theorem 5, Corollary 2, Corollary 4, Corollary 6.
(b) Did you include complete proofs of all theoretical results?[Yes]
We include all the proofs in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Comparisons of sample complexity rates
In this section, we provide comparisons of sample complexity rates.
Some more specific and detailed discussions can be found in Appendix C.4, Appendix C.5, and Appendix D.3 respectively.
In Table 2, we transfer all bounds in related works into our notations and compare them with ours.
[Table]
Red assumptions are what prior works need that are avoided by us.
For simplicity, we only show the orders and hide polylog terms (i.e., using ˜ O ( · ) notation)
η min is the reachability factor in Modi et al.
In linear MDPs, our bound (Corollary 6) is d lr H 3 worse compared with the most recent work (Wagenmaker et al., 2022), but our result is also independent of K .
It should be noted that both these bounds are sub-optimal in H dependence when compared to the lower bound of Ω (cid:0)
d 2 H 2 /ε 2 (cid:1) shown in Wagenmaker et al.
In the reward-aware setting, G OLF has a sharper rate than the subroutine
O LIVE under the completeness type assumption (Jin et al., 2021).
Since in RFO LIVE we only collect data when running a single (zero) reward O LIVE during the online phase and completeness (Assumption 2) is satisfied in our paper, we believe that there also exists a reward-free version of
G OLF (by running G OLF with zero reward function in the online phase and performing function elimination in the offline phase) that can potentially improve an Hd lr factor.
As for the linear completeness setting, our rate (Corollary 2) appears to be H 3 worse than Zanette et al.
(2020b).
However, we want to remark that they need to assume ε to be “asymptotically small” (more specifically, ε ≤ ˜ O
( ν min / √ d lc ) , where ν min is their explorability factor).
Thus there is an implicit dependence on 1 /ν min in their sample complexity bound.
Since such a factor can be arbitrarily large while H is always bounded in a finite horizon problem, our bound could be much better than theirs.
Again, there could be an Hd lc tighter bound for the reward-free version of G OLF , which implies that the optimal d lc dependence in the linear completeness setting could also be improved.
In low-rank MDPs, it is easy to see that our result (Corollary 4) significantly improves upon the rate of Modi et al.
(2021) in d lr and K factors, while slightly worse in the H factor.
In addition, they require the reachability assumption ( η min is their reachability factor), which means that their bound can be arbitrary worse than ours.
Similar dependence on reachability factor 1 /η min also exists in the sample complexity bounds of the more restricted block MDPs (Du et al., 2019; Misra et al., 2020) as they assume the reachability assumption.
Finally, regarding lower bounds, we do not necessarily need a direct one in our general function approximation setting (or even the more restricted linear completeness setting/low-rank MDPs) to compare with.
The lower bound for reward-free exploration in linear MDPs (Wagenmaker et al., 2022) is applicable to each of these and shows the necessary dependence on the respective complexity measures.
Coming up with a method which incorporates general function approximation while incurring better sample complexity rates on these special instances is a challenging and interesting avenue for future work.
B Discussions on Q-type and V-type
In this paper we study both Q-type and V-type, and they are not specific to the reward-free exploration.
Different versions (Q-type and V-type) already exist in the reward-aware general function approximation RL (e.g., Jiang et al.
; Jin et al.
; Du et al.
They capture different scenarios of interest and so far, it seems difficult to unify them even in the reward-aware setting.
Therefore, to give a comprehensive treatment of general function approximation, we consider both together.
The algorithms and analyses for the two types are not very different, with only moderate differences.
Since we consider the BE dimension and it subsumes Bellman rank (Jin et al., 2021), we first provide a detailed comparison between Q-type and V-type Bellman rank.
As discussed in Agarwal and Zhang (2022), V-type permits representation learning and other non-linear scenarios that are not easily captured in Q-type.
For instance, any contextual bandit problem is admissible under the V-type assumption (the V-type Bellman rank is 1), while Q-type does not capture all finite action, non-linear contextual bandit problems with a realizable reward.
We refer the reader to the detailed lower bound on the Q-type Bellman rank in the contextual bandit setting in Appendix B of Agarwal and Zhang (2022).
In contrast, Q-type has a more linear like structure, but it also includes problems whose
V-type Bellman rank is large (e.g., linear completeness setting in Zanette et al.
(2020a)).
Further,
V-type RFO LIVE (or V-type O LIVE ) requires one uniform action in exploration and therefore has an additional K factor (the cardinality of action space) in the sample complexity bound.
Then we discuss the BE dimension.
It can be shown that the Q-type BE dimension could also be exponentially larger than the V-type BE dimension.
In Agarwal and Zhang (2022), the authors show that the Q-type Bellman rank for a contextual bandit instance can be made arbitrarily higher whereas the V-type Bellman rank is always 1 for a CB setting.
Here, we show that the same instance can be shown to have high Bellman Eluder dimension as well.
The construction considers a context distribution which is uniform on 1 , . . .
, N , where we have N unique contexts.
We have two actions { a 1 , a 2 } .
We also have |F| =
N + 1 with the following structure:
[Equation]
For i < N + 1 , we have f i ( x, a ) =
f ∗ ( x, a ) when x ̸ = i , and f i ( x, a 1 ) = 1 , f i ( x, a 2 ) =
.implying that the function f
i makes incorrect prediction on context i for action a 1 .
Now, the bound on V-type BE dimension can be obtained by using the Bellman rank to BE dimension conversion result in Proposition 21 from Jin et al.
(2021) or Proposition 3 in our paper.
Since, the V-type
Bellman rank is 1 , the BE dimension is bounded as dim R vbe ( F , D F , 1 / (2 N ))
≤ ˜ O
(log( N )) .
We now show that the Q-type BE dimension is Ω( N ) .
Consider the sequence of policies π 1 , . .
. , π N .
For any i ∈ { 1 , . . . , N } and sequence π 1 , . . .
π i − 1 , (Q-type) Bellman residuals incurred by the function f i is: (cid:114)(cid:80) i − 1 k =1 (cid:16) 1 N (cid:80) Nj =1 E a ∼
π
k
[ f i ( j, a ) − f ∗ ( j, a )] (cid:17) 2 = 0 .
The same residual on the distribution induced by π i can be written as (cid:12)(cid:12)(cid:12)(cid:80) Nj =1 1 N E a ∼ π i
[ f i ( j, a ) − f ∗ ( j, a )] (cid:12)(cid:12)(cid:12) = 1 /N .
Hence, π i is 1 / (2 N ) -independent of { π 1 , . . .
π i − 1 } (recall Definition 2).
Thus, for ε = 1 / (2 N ) , the sequence π 1 , . . .
, π N can be used to show that the DE dimension (Definition 3) and the Q-type BE dimension for this instance is ˜ O ( N ) .
Hence, the Q-type BE dimension is exponentially larger than the V-type
BE dimension for this instance.
C Q-type RFO LIVE results
In this section, we present the results related to Q-type RFO LIVE .
In Appendix C.1, we introduce the theoretical guarantee of Q-type O LIVE (Jiang et al., 2017; Jin et al., 2021) for completeness.
In
Appendix C.2, we show the detailed proof of the sample complexity bound of Q-type RFO LIVE (Theorem 1).
In Appendix C.4, we discuss the instantiation of Q-type RFO LIVE to the known representation linear completeness setting.
In Appendix C.5, we provide another instantiation of
Q-type RFO LIVE to the linear MDP with known feature.
C.1 Q-type O LIVE
We first introduce the following assumption that will be useful for the O LIVE results (Propositionand Proposition 2).
Notice that this realizability assumption is for the single reward-aware O LIVE , where the function class captures reward-appended optimal value functions.
Thus it is different from our reward-free realizability assumption (Assumption 1).
Assumption 5 ( ε -approximate realizability of the single-reward function class) .
For the reward function R , optimal Q-function Q ∗ R , and the value function class F , there exists Q c R ∈ F so that
[Equation]
Then we state the sample complexity result of Q-type O LIVE (Algorithm 2 in Jin et al.
In this paper, we consider the uniformly bounded reward setting ( 0 ≤ r h ≤ 1 , ∀ h ∈
[ H ] ) instead of bounded total reward setting ( ∀ h ∈
[ H ] , r h ≥
and
(cid:80) H − 1 h =
r h ≤ 1 ) in Jiang and Agarwal (2018);
Jiang et al.
; Jin et al.
Therefore we need to pay an additional H 2 dependency in n actv and n elim because the range of value function is H times larger than the original ones, which induces an additional H 2 factor in the concentration inequalities.
Proposition 1 (Sample complexity of Q-type O LIVE , modification of Theorem 18 in Jin et al.
Under Assumption 5 with exact realizability (zero approximation error), if we set
[Equation]
where d qbe = dim R qbe ( F , D F , ε/ (4 H )) and ι = c 1 log( Hd qbe /δε ) , then with probability at least 1 − δ , Q-type O LIVE (Algorithm 2 in Jin et al.
with F will output an ε -optimal policy (under a single reward function R ) using at most O ( Hd qbe ( n actv + n elim )) episodes.
Here c 1 is a large enough constant.
This sample complexity result directly follows from Jin et al.
(2021) with minor adaptation of the parameters.
We refer the reader to Jin et al.
(2021) for the detailed proof.
C.2 Proof of Q-type RFO LIVE under general function approximation
In this part, we first provide the general statement of Theorem 1 and then show the detailed proof.
We also provide a detailed discussion on the different and novel part in our proof compared with Jiang et al.
; Jin et al.
Theorem (Full version of Theorem 1) .
Fix δ ∈ (0 , 1) .
Given a reward class R and a function class
F that satisfies Assumption 1 and Assumption 2, with probability at least 1 − δ , for any R ∈ R ,
Q-type RFO LIVE (Algorithm 1) with F outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
In RFO LIVE , we set
[Equation]
and n
[Equation]
=
[Equation]
ε, where d qbe = dim 0 qbe ( F − F , D F−F , ε/ (4 H )) , ι = c 2 log( Hd qbe /δε ) , and c 2 is a large enough constant.
Proof.
From the online phase of Q-type RFO LIVE (Algorithm 1), we can see that this phase is equivalent to running Q-type O LIVE (Algorithm 2 in Jin et al.
with the input function class F − F , the specified parameters ε actv , ε elim , n elim , n actv and under the reward function R = 0 .
In Proposition 1, we know that realizability (Assumption 5) holds because 0 ∈ F − F = F on .
Then the sample complexity is immediately from our specified values of ε actv , ε elim , n actv , n elim and
Proposition 1 as we only collect samples in the online phase.
Notice that the log-covering number log ( N F on ( · ))
=
log ( N F−F ( · )) ≤ 2 log ( N F ( · )) and such a constant 2 is absorbed by large enough
c 2 .
Therefore, it remains to show that the algorithm can indeed output an ε -optimal policy with probability 1 − δ in the offline phase.
We will show the following three claims hold with probability at least 1 − δ .
Claim 1 For any g ∈ F off ( R ) , if ∃ h ∈ [ H ] , s.t.
|E
R Q ( g, π g , h ) | ≥ ε/H , then it will be eliminated in the offline phase.
Claim 2 Q ∗ R ∈ F off ( R ) and Q ∗ R will not be eliminated in the offline phase.
Claim 3 At the end of the offline phase, picking the optimistic function from the survived value functions gives us ε -optimal policy.
Before showing these three claims, we first state properties from the online phase of Q-type RFO LIVE and the concentration results in the offline phase.
Properties from the online phase of Q-type RFO LIVE
From the equivalence between the online phase of Q-type RFO LIVE (Algorithm 1) and Q-type O LIVE (Algorithm 2 in Jin et al.
with reward 0 , we know that with probability at least 1 − δ/ 4 , the online phase terminates within d qbe H + iterations.
In addition, with probability at least 1 − δ/ 4 , the following properties (Eq.
(6) and Eq.
(7)) hold for the first d qbe H + 1 iterations:
(i) When the online phase exits at iteration T in line 7 (i
e
, the elimination procedure
is
not activated in RFO
LIVE )
, for any f ∈ F T , it predicts no more than ε/H value:
[Equation]
The first equality is due to any policy evaluation has value 0 under the reward function 0 .
The second equality is due to the policy loss decomposition in Jiang et al.
The second inequality is adapted from the “concentration in the activation procedure” part of the proof for Theorem 18 in Jin et al.
(ii) For T ≤ d qbe H + 1 , the concentration argument holds for any f ∈ F on and t ∈
[ T ] :
[Equation]
This is from the “concentration in the elimination procedure” step of the proof for Theorem 18 in Jin et al.
and we adapt it with our parameters.
Concentration results in the offline phase In RFO LIVE we use ˆ E R ( g, π t , π, h t ) and π ∈ Π est in line 18.
Since we are in the Q-type version, we have Π est = { π t } .
In addition, from Definition 4, we know that E R ( g, π t , π t , h t ) =
E R Q ( g, π t , h t ) .
Therefore, in line 18, it is equivalent to eliminating according to ˆ E R Q ( g, π t , h t ) .
Throughout this proof, we will use E R Q ( g, π t , h t ) and ˆ E R Q ( g, π t , h t ) notations for simplicity.
Now we show the concentration results in the offline phase.
Let R be an ( ε elim / 64) -cover of R .
For every R ∈ R , let R c = argmin R ′ ∈R max h ∈
[ H ] ∥
R h −
R ′ h ∥ ∞ .
Firstly, consider any fixed R ∈ R and let Z ( R ) be an ( ε elim / 64 )-cover of F off ( R ) with cardinality N F off ( R ) ( ε elim / 64) =
N F ( ε elim / 64) .
For every g ∈ F off ( R ) , let g c
=
argmin g ′ ∈Z ( R ) max h ∈
[ H ] ∥
g h − g ′ h
∥ ∞ .
Applying Hoeffding’s inequality to all
( t, g ′ ) ∈
[ T ] × Z ( R ) and taking a union bound, we have that with probability at least 1 − δ/ (2 N R ( ε elim / 64)) , the following holds for all
( t, g ′ ) ∈
[ T ] × Z ( R )
[Equation]
The second inequality is due to ε elim = ε/ (cid:0) 8 H 2 (cid:112)
d qbe (cid:1) , ι = c 2 log( Hd qbe /δε ) , and n elim =
( H 6 log( N F ( ε elim / 64))
+ H 4 log( N R ( ε elim /
[Equation]
) d
[Equation]
with c 2 in ι being chosen large enough.
Therefore for any g ∈ F off ( R ) , we get
[Equation]
Union bounding over R ∈ R , with probability at least 1 − δ/ 2 , for all t ∈
[ T ] , R ∈ R , g ∈ F off ( R ) , we have
[Equation]
Therefore
[Equation]
with probability at least 1 − δ/
[Equation]
for all t ∈ [ T ]
[Equation]
R ∈ R
[Equation]
g ∈ F off ( R )
[Equation]
we have
[Equation] [Equation]
All statements in our subsequent proof are under the event that all the different high-probability events (the online phase terminates within d qbe H + 1 iterations, and Eq.
(6), Eq.
(7), Eq.
(8) hold for the first d qbe H + 1 iterations) discussed above hold with a total failure probability of δ .
Proof of Claim 1 Consider any g ∈ F off ( R ) such that ∃ h ∈
[ H ] , |E R Q ( g, π g , h ) | ≥ ε/H .
Recall the definition of F off ( R ) , we know that g can be written as g
=
( g 0 , . .
, g H − 1 ) =
( f 0 + R 0 , . . .
, f H − 1 +
R H − 1 ) , f h ∈ F h .
We will discuss the positive average Bellman error and the negative average
Bellman error cases separately.
Case (i) of Claim
[Equation]
Since g h = f h + R h , we know that
[Equation]
Here we construct a function ˜ f
that has the same value as f h − T
h g h +1 at level h , uses zero reward
Bellman backup for any level before h , and assigns zero value after level h .
More formally, it is defined as
[Equation]
From the definition of Q-type average Bellman error and the construction, we know that for any policy π we can translate the Q-type reward-dependent average Bellman error for a function g ∈ F off ( R ) to the zero reward Q-type average Bellman error of a function ˜ f ∈ F on as the following
[Equation]
where in the third equality we notice that ˜ f
h +1 =
We can verify that ˜ f
= ( ˜ f 0 , . . . ,
˜
f H − 1 ) ∈ F on .
First let us consider level h ′
=
h .
From completeness (Assumption 2), we know that T
h
g h
+1 =
T 0 h ( f h +
+ R h +1 )
∈ F h .
Therefore, we have ˜ f h = f h −
T
h g h +1 ∈ F h − F h .
Then we consider level 0 ≤ h ′ ≤ h − 1 .
By the definition, we use zero reward
Bellman backup.
From completeness and ˜ f h ∈ F h − F h , we have ˜ f h − 1 =
T
h ˜ f h ∈
F h − 1 − F h − 1 .
By performing this inductive process backward, we have ˜ f h ′ ∈ F h ′ − F h ′ for any 0 ≤ h ′ ≤ h − 1 .
For level h + 1 ≤ h ′ ≤ H − 1 , we immediately get ˜ f h ′ = 0 ∈ F h ′ − F h ′ .
Therefore, we can see ˜ f
= ( ˜ f 0 , . . . ,
˜
f H − 1 ) ∈ F on from the definition of F on .
From the construction of ˜ f (zero reward Bellman backup for level 0 ≤ h ′ ≤ h − 1 ), we have
[Equation]
where π ˜ f is the greedy policy of ˜ f
and in fact it is the optimal policy when treating ˜ f h as the reward at level h
and there are no intermediate rewards.
From the first property of the online phase (Eq.
, we know that all the survived value functions at the end of the online phase predict no more than ε/H .
Therefore ˜ f will be eliminated.
We assume it is eliminated at iteration t by policy π t in level h t .
From the Bellman backup construction of ˜ f , we know that ˜ f can only be eliminated at level h .
This can be seen from the following argument: By the construction of ˜ f , we have E 0 Q ( ˜ f, π, h ′ )
=
for any π and h ′ ∈ [ H ] , h ′ ̸
= h .
Applying the second property of the online phase (Eq. (7)), we have
[Equation]
[Equation]
(cid:12)(cid:12) ≤ 3 ε elim / 4 , which gives us (cid:12)(cid:12)(cid:12)
ˆ E 0 Q ( ˜ f, π t , h t ) (cid:12) (cid:12)(cid:12) ≤ 3 ε elim / 4 if h t ̸ = h .
Since the elimination threshold is set to ε elim , ˜ f will not be eliminated at level h t ̸ = h .
This implies that at some iteration t in the online phase, we will collect some π t that eliminates ˜ f
at level h , i.e., it satisfies (cid:12)(cid:12)(cid:12) ˆ E 0 Q ( ˜ f, π t , h t ) (cid:12)(cid:12)(cid:12) > ε elim and h t = h .
Applying the second property of the online phase (Eq. (7)), we have (cid:12)(cid:12)(cid:12) ˆ E 0 Q ( ˜ f, π t , h t ) − E 0 Q ( ˜ f, π t , h t ) (cid:12)(cid:12)(cid:12) ≤ ε elim / 8 .
This tells us (cid:12)(cid:12)(cid:12) E 0 Q ( ˜ f, π t , h t ) (cid:12)(cid:12)(cid:12) > 7 ε elim / 8 .
Then from Eq.
(9) we have
[Equation]
>
[Equation]
Finally, the concentration argument of the offline phase (Eq.
(8)) implies that (cid:12)(cid:12)(cid:12) ˆ E R Q ( g, π t , h t ) − E R
[Equation]
( g, π
[Equation]
, h
[Equation]
) (cid:12)(cid:12)(cid:12) < ε elim / 4 .
Hence, we get (cid:12)(cid:12)(cid:12) ˆ E R Q ( g, π t , h t ) (cid:12)(cid:12)(cid:12) > ε elim / 2 .
This means that we will eliminate such g by π t in the offline phase.
Case (ii) of Claim 1 E R Q
[Equation]
g
[Equation]
E
[Equation]
− ε/
[Equation]
Same as before, we have E R Q ( g, π g , h ) =
E [ f h ( x h , a h ) − ( T 0 h
g h +1 )
( x h , a h ) | a 0: h ∼ π g ] ≤ − ε/H .
Now we let ˜ f
h be the negated version of the one in case (i), and define ˜ f as
[Equation]
Following the same steps as in case (i) we can verify that ˜ f ∈ F on , and that V ˜ f
( x 0 ) ≥ ε/H .
From here the argument is identical to case (i).
Proof of Claim 2 (i) From the assumption, we know that realizability condition Q ∗ R =
( Q ∗ R, 0 , . . .
Q ∗ R,H − 1 ) ∈
F off ( R ) holds.
(ii)
For the second argument, we note that E R Q ( Q ∗ R , π, h )
=
for any π and h ∈ [ H ] by the definition of the average Bellman error.
From the concentration argument in the offline phase (Eq.
(8)), we have (cid:12)(cid:12)(cid:12) ˆ E R Q
[Equation]
Q
[Equation]
t ) (cid:12)(cid:12) + ε elim / 4 = ε elim / 4 .
As a result, Q ∗ R will not be eliminated.
Proof of Claim 3 From Claim 1, we know that in the offline phase for any g ∈ F off ( R ) , if ∃ h ∈ [ H ] , s.t.
|E R Q ( g, π g , h ) | ≥ ε/H , then it will be eliminated.
Therefore from the policy loss decomposition in Jiang et al.
(2017), for all survived g ∈ F sur ( R ) in the offline phase, we have
[Equation]
Since Q ∗ R is not eliminated, similar as Jiang et al.
; Jin et al.
(2021), we have
[Equation]
Notice that Claim 3 directly implies that RFO LIVE returns an ε -near optimal policy.
This completes the proof.
C.3 Technical novelty over reward-aware O LIVE
The key step of the analyses of reward-aware O LIVE (Jiang et al., 2017; Jin et al., 2021) is to show that any bad function whose average Bellman error is large under the given reward function is eliminated (recall that they only have the online phase and the reward is always revealed).
This is ensured by the online exploration process.
However, the difficulty in our reward-free RL setting is that such a reward function is only revealed in the offline phase, where we no longer actively explore.
To overcome this difficulty, we use completely new and novel proof techniques here: For each bad function g ∈ F off ( R ) with a large average Bellman error under the true reward R , we construct a surrogate function ˜ f in the online phase.
Our construction guarantees that ˜ f has the same large average Bellman error as g , but the error is instead under the zero reward which we use during exploration (Eq.
Then we show that all these constructed ˜ f belong to the “difference” function class F on and ˜ f will be eliminated in the online phase since we use F on and zero reward there.
The collected data tuples (gathered constraints) that eliminate ˜ f will be used in the offline phase and they guarantee eliminating its corresponding bad function g .
Notice that in the design/definition of ˜ f , we need to guarantee that it has a large average Bellman error at the same timestep as g does so that it can correctly witness the average Bellman error of g , which we ensure via a Bellman backup construction.
In summary, both the construction of the surrogate function ˜ f and the translation of average Bellman error from bad function g ∈ F off ( R ) to ˜ f ∈ F on are novel to the best of our knowledge.
They reflect crucial difference between reward-aware and reward-free RL.
And at the same time, no reward-aware
RL works have used such mechanisms before.
We also provide a counterexample in Appendix G that shows other variant of O LIVE could fail even under realizability, completeness, and low Bellman Eluder dimension, where we know RFO LIVE has polynomial sample complexities.
C.4 Q-type RFO LIVE for known representation linear completeness setting
We first discuss why stating a specific B is equivalent to stating any B > 0 in Definition 6.
Assuming the statement hold for B , we will show that it holds for any B ′ > 0 .
The reason is the following.
Consider any Q ′
h
+1 =
⟨ ϕ
lc
h
+1 , θ ′ h
+1 ⟩ ∈ Q
h +1 ( { ϕ lc } , B ′ ) , where ∥ θ ′ h +
∥ 2 ≤ B ′ √ d lc .
For Q h
+1 =
(cid:68) ϕ
lc h
+1 , θ ′ h +1 B √ d lc ∥
θ ′ h
+
∥ 2 (cid:69) ∈ Q
h +1 ( { ϕ lc } , B ) , there exists θ h that satisfies ⟨ ϕ
lc h , θ h ⟩ = T
h Q h +1 and ∥ θ h ∥ 2 ≤ B √ d lc .
Then we know that ∥ θ ′ h +1 ∥ 2 B √ d lc ⟨ ϕ lc h , θ h ⟩ = T 0 h Q ′ h
+1 .
Now we can choose θ ′ h
=
∥ θ ′ h
+1 ∥ 2 B √ d lc θ h , and therefore we have T
0 h Q ′ h
+1 =
⟨ ϕ
lc h , θ ′ h ⟩ and ∥ θ ′ h
∥ 2 ≤ ∥ θ ′ h
+
∥ 2 ≤ B ′ √ d
lc satisfies the norm constraint, i.e., ⟨ ϕ lc h , θ ′ h ⟩ ∈ Q h ( { ϕ lc } , B ′ ) .
Next, we show the formal corollary statement and the detailed proof of the theoretical result of Q-type
RFO LIVE when instantiated to linear completeness setting.
Corollary (Full version of Corollary 2) .
Fix δ ∈ (0 , 1) .
Consider an MDP M that satisfies linear completeness (Definition 6) with the known feature ϕ lc , and the linear reward class R =
R 1 × . . .
× R h , where R h = (cid:8) ⟨ ϕ lc h , η h ⟩ : ∥ η h ∥ 2 ≤ √ d lc , ⟨ ϕ lc h ( · ) , η h ⟩ ∈ [0 , 1] (cid:9) .
With probability at least 1 − δ , for any R ∈ R , Q-type RFO LIVE (Algorithm 1) with F = F ( { ϕ lc } ) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
In RFO LIVE , we set
[Equation]
where ι = c 3 log( Hd lc /δε ) and c 3 is a large enough constant.
We remark that although the ˜ O (cid:16) H 5 d 3lc log(1 /
[Equation]
) ε(cid:17) sample complexity rate of Zanette et al.
(2020b) looks better than us, they need to assume ε ≤ ˜ O
( ν min / √ d lc ) , where ν min is their explorability factor.
Thus there is an implicit dependence on 1 /ν min in their sample complexity bound and their results are incomparable to us.
More related discussions can be found in Appendix A.
Proof.
We first verify that F ( { ϕ lc } ) satisfies the assumptions in Theorem 1.
Here we have that
F
[Equation] [Equation]
[Equation]
{ ϕ lc } , H − 1) × . . .
× F H − 1 ( { ϕ lc } , 0) , where
[Equation]
We first verify the realizability assumption (Assumption 1).
For the last level, we have
[Equation]
In addition, Q ∗ R,H − 1 =
R H − 1 = ⟨ ϕ lc H − 1 , η H − 1 ⟩ , where ∥ η H − 1 ∥ 2 ≤ √ d lc and ⟨ ϕ lc H − 1 ( · ) , η H − 1 ⟩ ∈ [0 , 1] .
Then for level H − 2 , we have
[Equation]
where ∥ θ ′ H − 2 ∥ 2 ≤ √ d lc and ⟨ ϕ lc H − 2 ( · ) ,
θ ′ H − 2 ⟩ ∈ [0 , 1] .
Here we apply the property of linear completeness (Definition 6).
Therefore, we can set θ ∗ H − 2 =
θ ′ H − 2 and get Q ∗ R,H − 2 =
R H − 2 + ⟨ ϕ lc H − 2 , θ ∗ H − 2 ⟩ ∈ F H − 2 ( { ϕ lc } , 1) + R H − 2 .
Continuing this induction process backward, we get
Q ∗ R,h ∈ F h + R h , ∀ h ∈
[ H ] , thus Q ∗ R ∈ F + R .
For completeness assumption (Assumption 2), again from the property of linear completeness, for any h ∈ [ H ] , f h +1 ∈
F h
+1 ,
R h +1 ∈ R
h
+1 , we have that
[Equation]
where || θ f + R,h || 2 ≤ ( H − h − 1) √ d lc and (cid:10)
ϕ
lc h ( · ) , θ f + R,h (cid:11) ∈
[ − ( H − h − 1) , H − h − 1] .
Thus (cid:10) ϕ
lc h , θ f
+ R,h (cid:11) ∈ F h , which implies that for any f h +1 ∈ F h +1 , R h +1 ∈ R
h +
we have T 0 h ( f h +
+
R h +1 ) ∈ F h .
Similarly, we can show T
h f h +1 ∈ F h .
Moreover, for any f h +1 , f ′ h
+1 ∈
F
h
+1 , we can assume that f h +1 =
⟨ ϕ
lc h
+1 , θ h +1 ⟩ and f ′
h
+1 =
⟨ ϕ
lc
h
+1 , θ ′ h +1 ⟩ , where ∥ θ h +1 ∥ 2 , ∥ θ ′ h +1 ∥ 2 ≤ ( H − h − 2) √ d lc and ⟨ ϕ
lc
h
+1 ( · ) , θ h +1 ⟩ , ⟨ ϕ lc
h +1 ( · ) , θ ′ h +1 ⟩ ∈ [ − ( H − h − 2) , H − h − 2] .
Therefore, we have f h +1 ( · ) − f ′
h
+1 ( · ) ∈ [ − 2( H − h − 2) , 2( H − h − 2)] .
From linear completeness (Definition 6), we know that there exists θ ′′ h that satisfy ⟨ ϕ lc h , θ ′′ h ⟩ = T 0 h (cid:0) ⟨ ϕ
lc
h
+1 , θ h +1 − θ ′ h +1 ⟩ (cid:1) and ∥ θ ′′ h ∥ 2 ≤ 2( H − h − 2) √ d lc with (cid:10) ϕ
lc h ( · ) , θ ′′ h (cid:11) ∈ [ − 2 ( H − h − 2) , 2 ( H − h − 2)] .
Now, choosing θ h = θ ′′ h / 2 and θ ′ h
= − θ ′′ h / 2 , we know that ⟨ ϕ
lc
h , θ h ⟩ − ⟨ ϕ lc h , θ ′ h ⟩ = T 0 h
( f h +1 − f ′ h +1 ) and ⟨ ϕ lc h , θ h ⟩ , ⟨ ϕ
lc
h , θ ′ h ⟩ ∈ F h .
Hence we have
T 0 h ( f h +1 − f ′
h
+1 ) ∈
F h − F h .
Therefore, from the above discussions, we get that completeness holds.
Invoking Theorem 1, the covering number argument (Lemma 8), and the bound on Bellman Eluder dimension (Proposition 6), we know that the output policy is ε -optimal
and the sample complexity is
[Equation]
As a final remark, Zanette et al.
(2020b) assume R is unknown but linear in ϕ lc .
In this case, we can instead construct F off ( R ) =
F 0 ( { ϕ lc } , H ) × . . .
× F H − 1 ( { ϕ lc } , 1) , where the norm bound and the value range bound in F h ( { ϕ lc } , H − h ) are larger than that in F h ( { ϕ lc } , H − h − 1) , thus capturing the reward-appended functions.
One can easily follow the proof of Theorem 1 and get the sample complexity result when using this new F off ( R ) in the offline phase of RFO LIVE .
This variant has the same sample complexity as Corollary 2.
C.5 Q-type RFO LIVE for known representation linear MDPs
In this part, we instantiate the general theoretical guarantee of Q-type RFO LIVE (Theorem 1) to the linear MDP setting, where the transition dynamics satisfy the low-rank decomposition (Definition 8) and ϕ lr is known.
We construct the function class F ( { ϕ lr } ) as F ( { ϕ lr } ) =
F 0 ( { ϕ lr } , H − 1) × . . .
× F H − 1 ( { ϕ lr } , 0) , where
[Equation]
In the following, we state the sample complexity result.
Corollary 6 (Q-type RFO LIVE for linear MDPs) .
Fix δ ∈ (0 , 1) .
Consider an MDP M that admits a low-rank factorization in Definition 8 and the feature ϕ lr is known, and we are given a reward function class R .
With probability at least 1 − δ , for any reward function R ∈ R , running Q-type version of RFO LIVE (Algorithm 1) with F = F ( { ϕ lr } ) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
In RFO LIVE , we set
[Equation]
and
[Equation]
where ι = c 4 log( Hd lr /δε ) and c 4 is a large enough constant.
Remark If we consider the entire linear reward class R = R 0 × . . .
× R H − 1 , where R h
= {⟨ ϕ lr , η h ⟩ : ∥
η h ∥ 2 ≤ √ d lr , ⟨ ϕ lr h , η h ⟩ ∈ ( X × A → [0 , 1]) } , then invoking Corollary 6 and applying the similar covering argument of Lemma 8 on the entire linear reward function class R yields the sample complexity
[Equation]
Recently, Wagenmaker et al.
(2022) improved the sharpest rate in the reward-free linear MDPs to ˜ O (cid:18)
[Equation]
Although the focus of our work is not to obtain the optimal rate, the sample complexity bound of RFO LIVE is also independent of K and not much worse than the current state of the art.
In the reward-aware setting, G OLF has a sharper rate than the subroutine
O LIVE under the completeness type assumption (Jin et al., 2021).
Since in RFO LIVE we only collect data when running a single (zero) reward O LIVE during the online phase and completeness (Assumption 2) is satisfied in our paper, we believe that there also exists a reward-free version of G OLF (by running G OLF with zero reward function in the online phase and performing function elimination in the offline phase) that can potentially improve an Hd qbe factor compared with RFO LIVE , thus matching the optimal d lr dependence in linear MDPs.
Proof.
Similar as the proof of Corollary 4, we can verify that Assumption 1 and Assumption 2 hold.
Invoking Theorem 1 and noticing that the covering number argument (Lemma 8) and the bound on
Q-type Bellman Eluder dimension (Proposition 5) completes the proof.
D V-type RFO LIVE results
In this section, we present the results related to V-type RFO LIVE .
In Appendix D.1, we provide the theoretical guarantee of V-type O LIVE (Jiang et al., 2017; Jin et al., 2021) for completeness.
In
Appendix D.2, we show the detailed proof of the sample complexity bound of V-type RFO LIVE (Theorem 3).
In Appendix D.3, we discuss the instantiation of V-type RFO LIVE to low-rank MDPs.
D.1 V-type O LIVE
First, we state the sample complexity of V-type O LIVE .
Similar as Q-type O LIVE , since we consider the uniformly bounded reward setting ( 0 ≤ r h ≤ 1 ) instead of bounded total reward setting ( ∀ h ∈
[ H ] , r h ≥
and
(cid:80) H − 1 h =
r h ≤ 1 ), we need to pay an additional H 2 dependency in n actv and n elim .
Proposition 2 (Sample complexity of V-type O LIVE , modification of Theorem 23 in Jin et al.
Assume ( ε 128 H √ d vbe = ε elim / 8 ) single-reward approximate realizability holds for F in Assumptionand F is finite.
If we set
[Equation]
and n
[Equation]
= H 4 d vbe K log( |F| )
ι εwhere d vbe = dim R vbe (cid:0) F , D F , ε/ 8 H (cid:1) and ι = c 5 log( Hd vbe K/δε ) , then with probability at least 1 − δ , V-type O LIVE (Algorithm 4 in Jin et al.
with F will output an ε -optimal policy (under a single reward R ) using at most O ( d vbe H ( n actv + n elim )) episodes.
Here c 5 is a large enough constant.
D.2 Proof of V-type RFO LIVE under general function approximation
In this part, we first provide the general statement of Theorem 3 and then show the detailed proof.
Theorem (Full version of Theorem 3) .
Fix δ ∈ (0 , 1) .
Given a reward class R and a function class
F that satisfies Assumption 1 and Assumption 2, with probability at least 1 − δ , for any R ∈ R ,
V-type RFO LIVE (Algorithm 1) with F outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
In RFO LIVE , we set
[Equation]
and
[Equation]
where d vbe = dim 0 vbe ( F − F , D F−F , ε/ (8 H )) , ι = c 6 log( Hd vbe K/δε ) and c 6 is a large enough constant.
Proof.
This proof follows the similar structure as the proof of Theorem 1.
The major difference is now we consider a discretized function class Z on in the online phase and consider a class of policy Π on in the offline elimination.
When we construct Z on (an ( ε elim / 64) -cover of F on ), w.l.o.g, we can assume
∈ Z on , therefore the approximate realizability (Assumption 5) holds.
From the online phase of V-type RFO LIVE (Algorithm 1), we can see that this phase is equivalent to running V-type RFO LIVE (Algorithm 4 in
Jin et al.
with the input function class Z on , the specified parameters ε actv , ε elim , n elim , n actv , and under the reward function R = 0 .
Then the sample complexity is immediately from our specified values of ε actv , ε elim , n actv , n elim and Proposition 2 as we only collect samples in the online phase.
Notice that we have the bound log ( |Z on | ) ≤ 2 log ( N F ( ε elim / 64)) and such a constant 2 is absorbed by large enough c 6 .
Therefore, it remains to show that the algorithm can indeed output an ε -optimal policy with probability 1 − δ in the offline phase.
We will show the following three claims hold with probability at least 1 − δ .
Claim 1 For any g ∈ F off ( R ) , if ∃ h ∈ [ H ] , s.t. |E
R V ( g, π g , h ) | ≥ ε/H , then it will be eliminated in the offline phase.
Claim 2 Q ∗ R ∈ F off ( R ) and Q ∗ R will not be eliminated in the offline phase.
Claim 3 At the end of the offline phase, picking the optimistic function from the survived value functions gives us ε -optimal policy.
Before showing these three claims, we first state show properties from the online phase of V-type
RFO LIVE and the concentration results in the offline phase.
Properties from the online phase of V-type RFO LIVE
From the equivalence between the online phase of V-type RFO LIVE (Algorithm 1) and V-type O LIVE (Algorithm 4 in Jin et al.
with reward 0 , we know that with probability at least 1 − δ/ 4 , the online phase terminates within d vbe H + 1 iterations.
In addition, with probability at least 1 − δ/ 4 , the following properties (Eq.
(10) and Eq.
(11)) hold for the first d vbe H + 1 iterations: (i) When the online phase exists at iteration T in line 7 (i.e., the elimination procedure is not activated in RFO LIVE ), for any f ∈ F T , it predicts no more than ε/ (2 H ) value:
[Equation]
The first equality is due to any policy evaluation has value 0 under under the reward function 0 .
The second equality is due to the policy loss decomposition in Jiang et al.
The second inequality is adapted from the “concentration in the activation procedure” part of the proof for Theorem 23 in
Jin et al.
(ii) For T ≤ d vbe H + 1 , the concentration argument holds for any f ∈ Z on and t ∈
[ T ] :
[Equation]
This is from the “concentration in the elimination procedure” step of the proof for Theorem 23 in Jin et al.
and we adapt it with our parameters.
Concentration results in the offline phase Let R be an ( ε elim / 64) -cover of R .
For every R ∈ R , let R c = argmin R ′ ∈R max h ∈
[ H ] ∥ R h − R ′ h
∥ ∞ .
First consider any fixed π ′ ∈ Π on and R ∈ R .
Let
Z ( R ) be an ( ε elim / 64 )-cover of F off ( R ) with cardinality N F off ( R ) ( ε elim / 64) =
N F ( ε elim / 64) .
For every g ∈ F off ( R ) , let g c
=
argmin g ′ ∈Z ( R ) max h ∈
[ H ] ∥
g h − g ′ h
∥ ∞ .
Then we consider any fixed ( t, g ′ ) ∈
[ T ] × Z ( R ) and calculate the upper bound of the second moment for g
[Equation]
Let y ( x h t , a h t , r h t , x
[Equation]
) = g
[Equation]
t
[Equation]
x
[Equation]
t , a
[Equation]
i )
[Equation]
t
[Equation]
x
[Equation] [Equation]
1 (cid:17) ⊆
[ − 2 H, 2 H ] , then we have
[Equation]
Applying Bernstein’s inequality and noticing the variance of the random variable is upper bounded by the second moment, with probability at least 1 −
[Equation]
|
[Equation]
| we have
[Equation]
The second inequality follows from ε elim = ε/ (cid:0) 32 H 2 √ d vbe (cid:1) , ι = c 6 log( Hd vbe K/δε ) , and n elim =
[Equation]
elim
[Equation]
[Equation] [Equation]
ε
[Equation]
with c 6 in ι being chosen large enough.
Here we also notice that | Π on | =
|Z on | and log ( |Z on | ) =
log ( N F on ( ε elim / 64)) ≤ 2 log ( N F ( ε elim / 64)) .
Union bounding over ( t, g ′ ) ∈
[ T ] × Z ( R ) , with probability at least 1 − δ 2 N R ( ε elim / 64) | Π on | , we have that for any fixed π ′ ∈ Π on , R ∈ R and all
g ′ ∈ Z ( R ) , t ∈
[ T ]
[Equation]
Union bounding over π ′ ∈ Π on , R ∈ R , we have that with probability at least 1 − δ/ 2 , for all
R ∈ R , π ′ ∈ Π on , g ∈ F off ( R ) , t ∈ [ T ] ,
[Equation]
Therefore, with probability at least 1 − δ/ 2 , for all R ∈ R , π ′ ∈ Π on , g ∈ F off ( R ) , t ∈ [ T ] , we have
[Equation] [Equation]
All statements in our subsequent proof are under the event that all the different high-probability events (the online phase terminates within d vbe H + 1 iterations, and Eq.
(10), Eq.
(11), Eq.
(12) hold for the first d vbe H + 1 iterations) discussed above hold with a total failure probability of δ .
Proof of Claim 1 We consider any g ∈ F off ( R ) that satisfies ∃ h ∈
[ H ] , such that |E R V ( g, π g , h ) | ≥ ε/H .
Recall the definition of F off ( R ) , we know that g can be written as g
=
( g 0 , . .
, g H − 1 ) =
( f 0 + R 0 , . . .
, f H − 1 + R H − 1 ) =
( f 0 + R 0 , . . .
, f H − 1 + R H − 1 ) .We will discuss the positive average
Bellman error and the negative average Bellman error case separately.
Case (i) of Claim
[Equation]
Since E R V ( g, π g , h ) =
E R Q ( g, π g , h ) , similar as the proof of Theorem 1, we know that
[Equation]
Same as in the proof of Theorem 1, here we construct a function ˜ f as
[Equation]
From the definition of V-type average Bellman error and the construction, we know that for any policy π we can translate the zero reward V-type average Bellman error of a function ˜ f ∈ F on with roll-in policy π to the average Bellman error under R for a function g ∈ F off ( R ) with roll-in policy π
: h − 1 ◦ π ˜ f,h
(Definition 4) as the following
[Equation]
where the second equality is due to ˜
f h +1 = 0 .
As the construction of ˜ f and the assumptions of F are the same as that in Q-type RFO LIVE and we use the same F on = F − F in both places, following the same proof of Theorem 1 directly gives us that ˜ f
= ( ˜ f 0 , . . . ,
˜
f H − 1 ) ∈ F on and V ˜ f
( x 0 ) ≥ ε/H .
Since in the online phase we use Z on , which is an ( ε elim / 64 )-cover of F on , we know that there exists
˜ f c ∈
Z on such that max h ′ ∈
[ H ] ∥ ˜ f h ′ − ˜ f
c h ′ ∥ ∞ ≤ ε elim / 64 ≤ ε/ (2 H ) .
Notice that
since ∀ h +1 ≤ h ′ ≤
H − 1 we have 0 ∈ Z h ′ and f h ′ = 0 , thus w.l.o.g., we can assume that ˜ f
c h ′ = 0 ,
∀ h +1 ≤ h ′ ≤ H − 1 .
From the definition of ˜ f c and ˜ f 0 ( x 0 , π ˜ f
( x 0 ))
=
V ˜ f
( x 0 ) ≥ ε/H , we have
[Equation]
From the first property of the online phase (Eq.
, we know that all the survived value functions at the end of the online phase predict no more than ε/ (2 H ) .
Therefore ˜
f
c will be eliminated.
We assume it is eliminated at iteration t by policy π t in level h t .
For any policy π and h ′ ∈
[ H ] , h ′ ̸
= h , we have
[Equation]
= − ε elim / 32 .
The first inequality is from the definition of ˜ f c .
The second inequality is due to π ˜ f is the greedy policy of ˜ f .
The last equality is due to the construction of ˜ f .
Similarly, on the other end, we also have
[Equation]
Therefore, for any policy π and h ′ ∈
[ H ] , h ′ ̸
= h
we get
[Equation]
From the Bellman backup construction of ˜ f
c , we know that ˜ f
c can only be eliminated at level h .
This can be seen from the following argument: Applying the concentration result of the online phase (Eq.
, we have | ˆ E 0 V
( ˜ f c , π t , h t ) − E 0 V
( ˜ f c , π t , h t ) | ≤ 3 ε elim / 4 .
Further notice that Eq.
(14), we have | ˆ E 0 V
( ˜ f c , π t , h t ) | ≤ 3 ε elim / 4 + ε elim / 32 if h t ̸ = h .
Since the elimination threshold is set to ε elim , ˜ f
c will not be eliminated at level h t ̸ = h .
This implies that at some iteration t in the online phase, we will collect some π t that eliminates
˜ f c at level h , i.e., it satisfies | ˆ E 0 V ( ˜ f c , π t , h t ) | > ε elim and h t
= h .
Applying concentration argument for the online phase (Eq.
, we have | ˆ E 0 V
( ˜ f c , π t , h t ) − E 0 V
( ˜ f c , π t , h t ) | ≤ 3 ε elim / 16 .
Therefore,
[Equation] [Equation]
From the definition of the average Bellman error and ˜ f, ˜ f c , we have the following equations
[Equation]
and
[Equation]
where π ′ ∈ Π t est =
Π on .
Because π ˜ f
c is the greedy policy of ˜ f c and ˜ f
c ∈ Z on , we know that in the offline phase π ′ =
π ˜ f c ∈ Π on will be chosen for elimination.
Then we get
[Equation]
where the first equality is due to ˜ f
c h +1 = 0 .
Similarly we have
[Equation]
Further, using the concentration argument for the offline phase (Eq.
, we get
[Equation]
Hence, if E 0 V ( ˜ f c , π t , h t ) ≥ 0 , we get E 0 V
( ˜ f c , π t , h t ) ≥ 13 ε elim / 16 from Eq.
(15), which yields ˆ E
[Equation] [Equation]
[Equation] [Equation]
Otherwise, we are in the case E 0 V
( ˜ f c , π t , h t ) <
and we have E 0 V
( ˜ f c , π t , h t ) < − 13 ε elim / 16 from
Eq.
This yields ˆ E R (cid:16)
[Equation]
≤
[Equation]
≤ − 13 ε elim / 16 + ε elim / 32 + ε elim / 4 < − ε elim / 2 .
Thus we always have (cid:12)(cid:12)(cid:12) ˆ E R (cid:16) g, π t , π ˜ f c , h t (cid:17)(cid:12)(cid:12)(cid:12) > ε elim / 2 , which implies that we eliminate such g by π t 0: h t − 1 ◦ π ˜ f c ,h t in the offline phase.
Case (ii) of Claim 1 E R
[Equation] [Equation]
− ε/H .
Same as before, we have E R V ( g, π g , h ) =
E
[Equation]
f
[Equation]
−
[Equation]
≤ − ε/H .
Now we let ˜ f
h be the negated version of the one in case (i), and define ˜ f as ˜
[Equation] [Equation]
′ , a h ′
[Equation]
Following the same steps as in case (i) we can verify ˜ f ∈ F on , construct ˜ f c ∈
Z on with V ˜ f c ( x 0 ) ≥ ε/ (2 H ) , and show that g is eliminated by π t 0: h t − 1 ◦ π ˜ f c ,h t in the offline phase for some t, h t .
Proof of Claim 2 (i) From the assumption, we know that realizability condition Q ∗ R =
( Q ∗ R, 0 , . . .
Q ∗ R,H − 1 ) ∈
F off ( R ) holds.
(ii)
For the second argument, we note that E ( Q ∗ R , π, π ′ , h ) = 0 for any π
, h ∈ [ H ] ,
π ′ ∈
Π t est by the definition of the average Bellman error.
From the concentration argument in the offline phase (Eq.
, we have | ˆ E R ( Q ∗ R , π t , π ′ , h t )
| ≤ |E R ( Q ∗ R , π t , π ′ , h t ) | + ε elim / 4 = ε elim / 4 .
As a result, Q ∗ R will not be eliminated.
Proof of Claim 3 From Claim 1, we know that in the offline phase, for any g ∈ F off ( R ) , if ∃ h ∈ [ H ] , s.t.
|E R V ( g, π g , h ) | ≥ ε/H , then it will be eliminated.
Therefore from the policy loss decomposition in Jiang et al.
(2017), for all survived g ∈ F sur ( R ) in the offline phase, we have
[Equation]
Since Q ∗ R is not eliminated, similar as Jiang et al.
(2017), we have
[Equation]
In sum, we can see the three claims hold with probability at least 1 − δ .
Since Claim 3 directly implies that RFO LIVE returns an ε -near optimal policy, we complete the proof.
D.3 V-type RFO LIVE for unknown representation low-rank MDPs
Here we provide the details of instantiating V-type RFO LIVE to low-rank MDPs (Agarwal et al., 2020; Modi et al., 2021; Uehara et al., 2021).
Firstly we remark that they assume the normalization in Definition 8 holds for f ′ :
X → [0 , 1] instead of f ′ :
X → [ − 1 , 1] .
We use the different version for ease of presentation and our results also hold under their normalization.
In addition, both versions are implied by the definition in Jin et al.
(2020b).
Now we show the complete corollary statement.
Corollary (Full version of Corollary 4) .
Fix δ ∈ (0 , 1) .
Consider a low-rank MDP M of embedding dimension d lr with a realizable feature class Φ lr (Assumption 3) and a reward function class R .
With probability at least 1 − δ , for any R ∈ R , V-type RFO LIVE (Algorithm 1) with F (Φ lr ) outputs a policy ˆ π that satisfies v ˆ πR ≥ v ∗ R − ε .
The required number of episodes is
[Equation]
In RFO LIVE , we
[Equation]
and
[Equation]
, where ι = c 7 log( HdK/δε ) and c 7 is large enough constant.
Before the formal proof, we provide some discussions and comparisons.
Firstly, when R is finite, the bound becomes ˜ O (cid:18)
[Equation]
(cid:19) .
Compared with Modi et al.
(2021), our result significantly improves upon their ˜ O (cid:16) H
[Equation]
[Equation] [Equation]
5min
[Equation]
and does not require the reachability assumption ( η min is their reachability factor).
On the other hand, their algorithm is more computationally viable and achieves the optimal deployment complexity (Huang et al., 2021).
With the additional access to and the realizability assumption of the right feature candidate class Υ lr in low-rank MDPs, another related work Agarwal et al.
(2020) provide a computationally efficient reward-free exploration guarantee but their rate H 22 d 7lr K 9 log( | Φ lr
||
Υ lr | /δ )
εis also much worse than ours.
In the sequel, we present the detailed proof for the corollary.
Proof.
We first verify that F satisfies the assumptions in Theorem 3.
Here we have that F =
F (Φ lr )
=
F 0 (Φ lr , H − 1) × . . .
× F H − 1 (Φ lr , 0) , where
[Equation]
Applying the property of linear MDPs (Lemma 9) gives us that
[Equation]
where ∥ θ ∗ h ∥ 2 ≤ ( H − ( h + 1)) √ d lr and ⟨ ϕ lr h ( · ) , θ ∗ h ⟩ ∈ [0 , H − ( h + 1)] .
Therefore, for any h ∈ [ H ] , we have Q ∗ R,h ∈ F h (Φ lr , H − h − 1)
+ R h
= F h + R h .
This implies that for any R ∈ R , we get
Q ∗ R ∈ F + R , thus, realizability (Assumption 1) holds.
Again, applying Lemma 9, for any h ∈
[ H ] , f h +1 ∈
F h
+1 ,
R h +1 ∈ R
h
+1 , we have that
[Equation]
where ∥ θ ∗ f + R,h ∥ 2 ≤ ( H − h − 1) √ d lr and ⟨ ϕ lr h ( · ) , θ ∗ f + R,h ⟩ ∈ [ − ( H − h − 1) , H − h − 1] .
Thus ⟨ ϕ lr h , θ ∗ f + R,h ⟩ ∈ F h .
This implies that for any f h +1 ∈ F h +1 , R h +1 ∈ R
h
+1 , we have T 0 h ( f h +
+
R h +1 ) ∈ F h .
Similarly, we can show T
h f h +1 ∈ F h .
Moreover, for any f h +1 , f ′ h
+1 ∈
F
h
+1 , we have that ∥ f h
+1 − f ′ h +1 ∥ ∞ ≤ 2( H − h − 2) .
Therefore, there exists θ ∗ f − f ′ ,h such that T 0 h
( f h +1 − f ′ h
+1 )
= ⟨ ϕ lr h , θ ∗ f − f ′ ,h ⟩ ⊆
( X × A → [ − 2( H − h − 2) , 2( H − h − 2)]) and ∥ θ ∗ f − f ′ ,h ∥ 2 ≤ 2( H − h − 2) √ d lr .
Then choosing θ h
= θ ∗ f − f ′ ,h / 2 , θ ′ h
= − θ ∗ f − f ′ ,h / 2 and f h = ⟨ ϕ lr h , θ h ⟩ , f ′ h =
⟨ ϕ lr h , θ ′ h ⟩ gives us both f h ( · ) , f ′ h ( · ) ∈ [ − ( H − h − 2) , ( H − h − 2)] ⊆
[ − ( H − h − 1) , H − h − 1] and ∥ θ h ∥ 2 , ∥ θ ′ h
∥ 2 ≤ ( H − h − 1) √ d lr .
Therefore, we have that f h − f ′ h ∈
F h − F h .
The above discussions imply that completeness (Assumption 2) holds.
Invoking Theorem 3 and further noticing the covering number argument (Lemma 8) and the bound on V-type Bellman Eluder dimension (Proposition 4), we know that the output policy is ε -optimal
and the sample complexity is
[Equation]
E Hardness result for unknown representation linear completeness setting
In this section, we provide the detailed construction and proof for the hardness result and more discussions.
Theorem (Restatement of Theorem 5) .
There exists a family of MDPs M , a reward class R and a feature set Φ lc , such that ∀ M ∈ M , the ( M, Φ lc ) pair satisfies Assumption 4 (i.e., Φ lc is realizable linear complete feature class for any M ∈ M ), yet it is information-theoretically impossible for an algorithm to obtain a poly (cid:0) d lc , H, log( | Φ lc | ) , log( |R| ) , 1 /ε, log(1 /δ )
(cid:1) sample complexity for reward-free exploration with the given reward class R .
Proof.
We present an exponential tree MDP as a hard instance, similar to the lower bound instances in Modi et al.
(2020), and design “one-hot” realizable feature inspired by the construction in Zanette et al.
(2020a) which they used to show that a low-IBE (Inherent Bellman Error) setting does not imply a low-rank/linear MDP.
Family of hard instances We consider a class of deterministic finite state MDPs M with a singleton reward class.
In our construction, for simplicity, the MDPs have a layered structure where the set of states an agent can encounter at any two timesteps h and h ′ ( h ̸ = h ′ ) are disjoint.
Hence, we denote the respective state spaces for each timestep h as X h , and we always have x h ∈ X h .
In this layered
MDP, for each timestep h ∈
[ H ] , we only define the corresponding feature ϕ , rewards at each X h , and transition functions from X h to X
h
+1 .
To convert it to the non-layered MDPs, at each timestep h ∈
[ H ] , we only need to let the features ϕ and reward functions be 0 at the states outside X h and let transitions have 0 probability when transiting from states in X h to states outside X h +1 and define the transition functions from some states outside X h arbitrarily.
Consider a complete binary tree of depth H − 2 (we count the first layer x 0 as depth 0).
The vertices at each level h from the state space X h and the two outgoing edges represent the available actions at each state.
The reward class is a singleton class { R } , where all states get zero reward other than R H − 1 ( x + , NULL)
= 1 .
The starting state of the MDP is the root node x 0 and the dynamics are deterministic at all levels: each action { left , right } transits to the corresponding child node.
Of all the 2 H − 2 nodes at level H − 2 , on one node x ∗ H − 2 , one action a ∗ H − 2 transits to x + with probability 1 whereas the other action and all actions for other nodes transit to x − (i.e., only
P H − 1 ( x + | x ∗ H − 2 , a ∗ H − 2 ) = 1 ).
As we have 2 H − 1 many choices for ( x ∗ H − 2 , a ∗ H − 2 ) , we have |M| = 2 H − 1 .
We provide an illustration for H = 4 in Figure 1.
[Figure]
The complete binary tree portion ranges from timestep 0 to 2 , and x + , x − belong to timestep 3.
On timestep h = 2 , only ( x ∗ 2 , a ∗ 2 ) =
( x 12 , left) transits to the good state x + , and all other state-action pairs transit to bad state x − .
Rewards for all state-action pairs are 0 other than R 3 ( x + , NULL)
= 1 .
Constructing the feature class We now construct a feature class Φ lc such that for any MDP
M ∈ M , Φ lc satisfies Assumption 4 (i.e., the linearly complete feature under M belongs to Φ lc ).
We define the feature class in the following way: for each timestep h ∈
[ H − 1] , we define Φ lc h
= (cid:8) ϕ
ih : i ∈ [2 h +1 ] , ϕ ih
[ j, a ] =
[ i = 2 · j + a ] , ∀ j ∈ [2 h ] , a ∈ { 0 , 1 } (cid:9) , where ϕ ih [ j, a ] denotes the value of feature ϕ ih on the j -th state ( x jh ) and action a at level h .
Finally, the two nodes at timestep H − 1 have a feature value of ϕ H − 1 ( x + , NULL)
= 1 for the rewarding node and ϕ H − 1 ( x − , NULL)
=
for the non-rewarding node.
Since we define a feature class of size (cid:12) (cid:12) Φ lc h (cid:12) (cid:12) = 2 h +1 for each level
h ∈
[ H − 1] , the total size of the product class is (cid:12)(cid:12)
Φ lc (cid:12)(cid:12) =
Π H − 2 h =0 (cid:12)(cid:12)
Φ lc h (cid:12)(cid:12) =
2 ( H − 1) H/ 2 .
Verifying Assumption 4 Notice that from our construction of M , there is a one-on-one correspondence between one of 2 H − 1 state-action pair ( x ∗ H − 2 , a ∗ H − 2 ) and one of 2 H − 1 MDP M ∈ M .
Therefore, for i -th such state-action pair ( i ∈ [2 H − 1 ] ) at level H − 2 , we use M i to denote its corresponding MDP.
Now consider any MDP M i ∈ M .
For any level h ∈ [ H − 1] , let i h denote the state-action pair (among 2 h +1 -many state-action pairs at depth h ) which lies along the path from the root to the rewarding node x + .
To verify the realizability condition, we show that the feature ϕ
lc ,i = (cid:16) ϕ
i 0 0 , . . .
, ϕ i h h , . . .
, ϕ
i H − 2 H − 2 , ϕ
H − 1 (cid:17) ∈
Φ lc satisfies the linear completeness structure (Definition 6).
Firstly, note that by definition (cid:12) (cid:12) (cid:12) (cid:12) ϕ
i h h ( x h , a h ) (cid:12) (cid:12) (cid:12) (cid:12) 2 ≤ 1 for all h ∈
[ H ] , x h , a h .
Now, we verify that the requirements in Definition 6 are satisfied.
For any h ∈ [ H − 1] and pair ( x h , a h ) with ϕ lc ,ih ( x h , a h ) = 0 , all subsequent states x h +1 reachable from x h and any action a also have ϕ lc ,ih +1 ( x h +1 , a )
= 0 : zero-feature intermediate state-action pairs only transit to zero feature value states at the next timestep.
Therefore, the backup condition is satisfied by default:
[Equation]
On the other hand, for any h ∈ [ H − 1] and pair ( x h , a h ) with ϕ lc ,ih ( x h , a h ) = 1 , we have ϕ lc ,ih +
( x h +1 , a ) = 1 for one action along the path to x + and ϕ lc ,ih +
( x h +1 , a ′ )
=
for the other.
For any θ h +1 ∈ R (notice that d lc = 1 in our construction), we have
[Equation]
Thus, for both cases, we can set θ h
= θ h +1 or 0 to satisfy the linear completeness condition ⟨ ϕ lc ,ih ( x h , a h ) , θ h ⟩ − (cid:16) T 0 h ⟨ ϕ lc ,ih +1 , θ h +1 ⟩ (cid:17) ( x h , a h ) = 0 .
Hence, the chosen feature mapping ϕ lc ,i satisfies the linear completeness structure in Definition 6.
Lower bound for exploration Learning in this family of MDPs M is provably hard as the feature and reward classes do not reveal any information about the pair ( x ∗ H − 2 , a ∗ H − 2 ) and the agent has to try each of the 2 H − 1 paths (Krishnamurthy et al., 2016).
Hence, any learning agent has to sample Ω(2 H ) trajectories to find the optimal policy in any given instance M ∈ M .
The stated lower bound statement follows from the fact that d lc = 1 , A = 2 , 1 /ε is constant and the sample complexity is Ω(2 H ) which scales with (cid:12)(cid:12)
Φ lc (cid:12)(cid:12) =
2 ( H − 1) H/ 2 and |X| =
2 H − 1 + 1 .
Discussions The family M of hard instances highlights a fundamental distinction between the low-rank and linear completeness settings when underlying true representations are unknown.
Our result further highlights that assuming reachability (Modi et al., 2021) and/or explorability (Zanette et al., 2020b) does not alleviate the fundamental hardness.
Reachability is satisfied as for each MDP in M , each node at every level can be reached with probability 1 by taking the correct actions which lie along the path from the root node.
Similarly, for explorability, we need to verify that for any θ ∈ R , the constant max π min | θ | =1 | E π [ ⟨ ϕ h ( x h , a h ) , θ ⟩ ] | is large for all h ∈
[ H ] (notice that θ is one dimension so ∥ θ ∥ 2 = | θ | ).
Again, it is easy to see that for both values of θ ∈ {− 1 , 1 } , the policy corresponding to the path from root node x 0 to x + maximizes this constant for all steps with a value of 1 .
Moreover, our constructed family of hard instances is quite general as it is applicable to the settings of online reward-specific exploration and learning with a generative model.
In order to verify this for the former setting, note that our reward class is a singleton reward { R } and exposing this reward (reward class) to the agent still does not disclose any information about the pair ( x ∗ H − 2 , a ∗ H − 2 ) to the agent.
Hence, the required number of trajectories to identify this pair is again Ω(2 H ) .
Similarly, for a generative model, the problem of identifying the pair ( x ∗ H − 2 , a ∗ H − 2 ) is inherently a best-arm identification problem among the 2 H − 1 possibilities.
Thus, the existing lower bounds for best-arm identification (Krishnamurthy et al., 2016) directly lead to a sample complexity bound of Ω(2 H ) .
In addition, we can see from the construction that our hardness result also shows that a polynomial in |X| dependence is unavoidable in this case.
We also remark that the stated hardness result can be easily tweaked to show a 1 /ε 2 dependence for identifying an ε -optimal policy by moving from deterministic transitions at timestep H − 1 to stochastic transition probabilities:
P H − 1 ( x + | x ∗ H − 2 , a ∗ H − 2 ) = 12 + ε ,
P H − 1 ( x − | x ∗ H − 2 , a ∗ H − 2 ) = 12 − ε , and P H − 1 ( x + | x H − 2 , a H − 2 ) =
P H − 1 ( x − | x H − 2 , a H − 2 )
=
if x H − 2 ̸ = x ∗ H − 2 or a H − 2 ̸ = a ∗ H − 2 .
The realizable feature ϕ lc ,i will be a two-dimensional representation after this modification, where we change previous one dimension values 0 and 1 to two dimension (0 , 0) and (1 , − 1) respectively.
The hardness result highlights the insufficiency of realizability of a linearly complete feature (Assumption 4) in the representation learning setting and indicates that realizability of stronger completeness style features may be necessary for provably efficient reward-free RL.
F Algorithm-specific counterexample of RFO LIVE
In this section, we show an algorithm-specific counterexample of RFO LIVE (Algorithm 1) that satisfies realizability (Assumption 1) and has a low Bellman Eluder dimension, while only violates completeness (Assumption 2).
Together with the positive results (Theorem 1 and Theorem 3), we conjecture that realizability-type assumptions are not sufficient for statistically efficient reward-free
RL.
As we know that O LIVE (Jiang et al., 2017; Jin et al., 2021) only requires realizability and low Bellman Eluder dimension for reward-aware RL and RFO LIVE is its natural extension to the reward-free setting, we believe that the hardness between reward-aware and reward-free RL has a deep connection to the sharp separation between realizability and completeness (Chen and Jiang, 2019;
Wang et al., 2020b, 2021; Xie and Jiang, 2021; Weisz et al., 2021a,b, 2022; Foster et al., 2021).
Theorem 7.
There exists an MDP M , a function class F , a reward class R , where Assumptionholds and the function class F − F has a low Bellman Eluder dimension ( d qbe defined in Theorem 1).
However, with probability 0.25, (Q-type) RFO LIVE with infinite amount of data cannot output a 0.1-optimal policy for some R ∈ R .
Proof.
We first discuss the counterexample shown in Figure 2 and Table 3 at a high level.
In our construction, with probability 0.25, the agent will only explore and collect data at some specific place because it is sufficient to eliminate all candidate functions predicting large positive values at x 0 .
Then in the offline phase, the agent cannot eliminate some bad function because of lack of support in the collected data.
Then performing function elimination in the offline phase fails.
We provide more details in the sequel.
[Figure]
right
( x 0 , left) ( x 0 , right) ( x A , NULL) ( x B , NULL)
R
R  0.
Q ∗ R
Q ∗ R0.
f R    f R0.
f bad 0.
f R 1 + R 1 = Q ∗ R    f R 2 + R 2 = Q ∗ R0.
f R 1 + R  0.
f bad + R0.
Construction In Figure 2, taking action left and action right in state x 0 transits to x A and x B respectively.
We denote the null action at x A , x B and the null state at level H = 2 as NULL and x NULL respectively.
In this example, the length of horizon is H = 2 .
We construct F = F 0 × F 1 , where F 0 = { 0 , f R 1 , 0 , f R 2 , 0 , f bad , 0 } and F 1 = { 0 , f bad , 1 } .
In addition, we construct R = R 0 ×R 1 , where R 0 = { 0 } and R 1 =
{ R 1 , 1 , R 2 , 1 } .
Recall that the second subscript of f ∈ F 0 , F 1 and
R ∈ R 1 is the index for the timestep.
The details are shown in Table 3.
Notice that here we use the layered MDP for simplicity.
To convert it to a non-layered MDP, we only need to set corresponding values in the transition function, reward function, and f ∈ F to be 0.
Verifying realizability and low Bellman Eluder dimension One can immediately see that realizability (Assumption 1) is satisfied.
For example, we have Q ∗ R 1 ,
= f R 1 , 1 + R 1 , 1 = 0 + R 1 , 1 , which implies that Q ∗ R 1 , 1 ∈ F 1 + R 1 .
Similarly, we can verify that Q ∗ R 2 , 1 ∈ F 1 + R 1 , Q ∗ R 1 , 0 ∈
F 0 + R 0 , Q ∗ R 2 , 0 ∈ F 0 + R 0 .
In addition, F − F has a low Bellman Eluder dimension.
It is because the Bellman Eluder dimension can be upper bounded by the Bellman rank (Proposition 3) and the Bellman rank can be upper bounded by the number of states (Jiang et al., 2017).
Therefore, the Bellman Eluder dimension is just a small bounded finite number.
Later we will show that with even infinite amount data RFO LIVE fails, which implies that we cannot get a polynomial sample complexity bound in this case.
Violation of completeness We can easily see that for f bad , 1 ∈ F 1 and R 2 , 1 ∈ R 1 , its Bellman backup T 0 0 ( f bad , 1 + R 2 , 1 ) / ∈ F 0 .
This means that completeness (Assumption 2) does not hold.
RFO LIVE fails in the counterexample We first consider running (Q-type) RFO LIVE on this counterexample during the online phase.
In the following, we will assume the more favorable case where the agent can collect infinitely many number of samples in line 5 and line 11 (i.e., no statistical/estimation error for the average Bellman error in the empirical version).
The agent will pick the most optimistic function for exploration.
In the first iteration, such an optimistic function at level 0 will be equal to f R 1 , 0 − 0 .
Therefore, starting from x 0 , the agent will choose the action left .
With at least probability 0.5, it will pick level 0 to eliminate and collect data (i.e., collecting data at ( x 0 , left ) in line 11).
The reason is that for line 9, the large average
Bellman error always exists at ( x 0 , left ) while the Bellman error could be large at ( x left , NULL ).
By adversarial tie-breaking, there is at least 0.5 probability that ( x 0 , left ) is chosen.
Now consider the case that the agent pick ( x 0 , left ) to collect data in line 11.
We can see that only function 0 , f bad − f R 2 , f R 2 − f bad ∈ F −F will survive while all other functions violate the collected constraint.
Here we notice that for any f ∈ F − F we have V f ( x A )
= 0 or V f ( x A ) = ± 0 .
So the survived function f ∈ F − F belongs to one of the following cases: (i) f ( x 0 , left) =
and V f ( x A ) = 0 , (ii) f ( x 0 , left)
= 0 .
01 and V f ( x A ) =
01 , or (iii) f
( x 0 , left) = − 0 .
01 and
V f ( x A ) = − 0 .
At the second iteration, RFO LIVE will choose f bad − f R 2 (i.e., case (ii)) and action right .
Due to adversarial tie-breaking, we have that with probability 0.5, the agent collects data at ( x B , NULL ) and eliminates f bad − f R 2 .
In this case, for the third iteration, the agent chooses function 0 and then terminates in line 6 since the average Bellman error is 0.
For the offline phase, let us consider the reward function R 2 and the elimination on F off ( R 2 )
=
F +
R 2 .
Recall that in the online phase, we only collect data on ( x 0 , left , x A ) and ( x B , NULL , x NULL ).
It is easy to see that we will eliminate f R 1 + R 2 from this constraint.
However, we cannot eliminate either f R 2 + R 2 or f bad + R 2 because they all have zero average Bellman error under these two constraints.
Then by optimistic selection criteria, the agent will pick f bad + R 2 .
This induces a sub-optimal policy ( right ) with accuracy ε = 0 .
Therefore, with probability at least 0.25, (Q-type) RFO LIVE fails to output a 0.1 optimal policy for some reward R ∈ R in this counterexample.
G Discussions on other variants of O LIVE
In this section, we briefly discuss that some other variant of O LIVE in the reward-free setting could easily fail under Assumption 1, Assumption 2, and low Bellman Eluder dimension (where we know that RFO LIVE works).
One adaptation of O LIVE to the reward free case is to perform exploration on the joint function class space and we call it J OINT O LIVE .
More specifically, we maintain a version space F t + R t ⊆
F + R during the online phase.
In each online iteration, we pick the most optimistic function f t on = f t + R t = argmax f + R ∈F t + R t V f + R ( x 0 ) and explore according to π t = π f t on .
Here for f t on , we decompose it as the sum of f t ∈ F t and R t ∈ R t .
Then we roll out policy π t and estimate the average Bellman error.
For the termination condition, like O LIVE , we check whether f t on has a small average Bellman error under reward R t .
If the algorithm is not terminated, we pick a level for elimination and collect the constraint.
At the end of each online iteration, we shrink the version space of F t and R t according to the average Bellman error.
For the offline phase, we use collected constraints to perform elimination like RFO LIVE and then output the greedy policy of the optimistic survived function.
We will show that this variant could get stuck in the following counterexample even in the case that we are allowed to collect infinite amount of samples to build estimates (i.e., no statistical/estimation error).
Construction We consider the MDP in Figure 2 and reward function class R = R 0 × R 1 , where R 0 = { 0 } and R 1 = { R 1 , 1 , R 2 , 1 } .
The function class F = F 0 × F 1 is constructed as
F 0 = { 0 , f R 1 , 0 , f R 2 , 0 , f bad , 0 } and F 1 = { 0 } .
Compared with the counterexample in Theorem 7, the difference is that f bad is changed and now we only have a single function 0 in F 1 .
The details as shown in Table 4.
As discussed in the proof of Theorem 7, we can convert the layered MDP here to a non-layered one.
[Table]
Verifying realizability, completeness, and low Bellman Eluder dimension Realizability and low
Bellman Eluder dimension can be verified in the same way as the counterexample for RFO LIVE in Theorem 7.
For completeness (Assumption 2), one can easily verify that by noticing we have
F 1 = { 0 } now.
J OINT
O LIVE fails in the counterexample In J OINT
O LIVE , the agent will pick the optimistic function in the joint function space to explore during the online phase.
At the first iteration, there are two candidates f R 1 + R 1 and f R 1 + R 2 .
By adversarial tie-breaking, the agent will choose f R
+ Rwith probability 0.5 and choose action left .
Then the agent will terminate immediately because the average Bellman error is 0 for f R 1 + R 1 everywhere under reward R 1 .
For the offline phase, we similarly consider reward R 2 .
It is easy to see that f R 1 + R 2 will be eliminated while both f R 2 + R 2 and f bad + R 2 will survive.
Then by optimistic selection, the agent will choose the greedy policy of f bad + R 2 .
This induces a sub-optimal policy ( right ) with accuracy ε = 0 .
Therefore, with probability 0.5, J OINT O LIVE fails in this counterexample.
H Auxiliary results
In this section, we provide auxiliary results for the paper.
We show covering number arguments in
Appendix H.1 and some bounds on Bellman Eluder dimensions in Appendix H.2.
H.1 Covering number
In this part, we present the covering number argument for the linear function class used in the paper.
Lemma 8 (Size of ε -cover for linear function class) .
We have three claims here
1. Consider F ( { ϕ lc } )
=
F 0 ( { ϕ lc } , H − 1) × . . .
× F H − 1 ( { ϕ lc } , 0) , where F h ( { ϕ lc } , B h ) = (cid:8) f h ( x h , a h ) =
(cid:10) ϕ
lc h
( x h , a h ) , θ h (cid:11) :
∥ θ h ∥ 2 ≤ B h √ d lc , ⟨ ϕ lc h ( · ) , θ h ⟩ ∈ [ − B h , B h ] (cid:9) .
Then we have N F ( { ϕ lc } ) ( ε ) ≤ (cid:16) 2 H 2 √ d lc ε (cid:17)
d lc . 2.
Consider F ( { ϕ lr } )
=
F 0 ( { ϕ lr } , H − 1) × . . .
× F H − 1 ( { ϕ lr } , 0) , where F h ( { ϕ lr } , B h ) = (cid:8) f
[Equation]
, a
[Equation] [Equation]
, a
[Equation]
) , θ h (cid:11) : ∥
[Equation]
lr ,
[Equation]
ϕ lr h ( · ) , θ h
[Equation]
∈ [ − B h , B h ] (cid:9) .
Then we have N F ( { ϕ lr } ) ( ε ) ≤ (cid:16) 2 H 2 √
d lr ε (cid:17) d lr . 3.
Consider F (Φ lr )
=
F 0 (Φ lr , H − 1) × . . .
× F H − 1 (Φ lr , 0) , where F h (Φ lr , B h ) =
(cid:8) f h ( x h , a h
[Equation]
= ⟨
[Equation]
( x
[Equation] [Equation]
, θ
[Equation]
: ϕ
[Equation]
[Equation] [Equation]
] (cid:9) .
Then we have N F (Φ lr ) ( ε ) ≤ | Φ lr | (cid:16) 2 H 2 √
d lr ε
[Equation]
Proof.
This is a standard result.
For the first one, we can construct a cover over { θ h : ∥ θ h ∥ 2 ≤ ( H − h − 1) √ d lc } in the 2-norm at scale ε/H for each level h ∈
[ H ] .
Then this cover immediately implies a cover over the function F ( { ϕ lc } ) .
The covering number directly follows the covering number of the 2-norm ball.
The second follows the same steps.
For the third result, we additionally union over ϕ ∈ Φ lr .
H.2 Bounds on the Bellman Eluder dimension
In this part, we show that Q-type and V-type Bellman Eluder dimensions for the instantiated linear
MDP, low-rank MDP, and linear completeness with known feature settings are indeed small.
We will use the following relation between Bellman rank and Bellman Eluder dimension from Jin et al.
Proposition 3 (Bellman rank ⊆
Bellman Eluder dimension, Proposition 11 and 21 in Jin et al.
If an MDP with function class F has Q-type (or V-type)
Bellman rank d br with normalization parameter ζ , then the respective Bellman Eluder dimension dim R qbe ( F , D F , ε ) (or dim R vbe ( F , D F , ε ) ) is bounded by ˜ O (cid:16) 1 + d R br log (cid:16) 1 + ζε (cid:17)(cid:17) .
Linear/low-rank MDPs Before stating the result for low-rank MDPs, we recall the following well-known property for the class:
Lemma 9 (Jin et al.
(2020b);
Modi et al.
Consider a low-rank MDP M (Definition 8) with embedding dimension d lr .
For any function f : X → [ − c, c ] , we have:
[Equation]
where θ ∗ f ∈ R d lr
and we have ∥ θ ∗ f ∥ 2 ≤ c √ d lr .
A similar linear representation is true for
E a ∼ π h +
[ f ( x h +1 , a ) | x h , a h ] where f : X ×
A → [ − c, c ] and a policy π h +1 :
X → A .
Proof.
For state-value function f , we have
[Equation]
where θ ∗ f := (cid:82) f ( x h +1 )
µ lr h ( x h +1 )
d ( x h +1 ) is a function of f .
Additionally, we obtain ∥ θ ∗ f ∥ 2 ≤ c √ d lr from Definition 8.
For Q-value function f , we similarly have
[Equation]
where θ ∗ f := (cid:82)(cid:82) f
( x h +1 , a h +1 )
π ( a h +1 | x h +1 )
µ lr h ( x h +1 )
d ( x h +1 )
d ( a h +1 ) and ∥ θ ∗ f ∥ 2 ≤ c √ d lr .
Now, we can state the following bound on the V-type Bellman Eluder dimension for low-rank MDPs:
Proposition 4 (Low-rank MDP) .
Consider a low-rank MDP M of embedding dimension d lr with a realizable feature class Φ lr (Assumption 3).
Define the corresponding linear function class
F (Φ lr )
=
F 0 (Φ lr , H − 1) × . . .
× F H − 1 (Φ lr , 0) using
[Equation]
Then, for the difference class F on = F (Φ lr ) − F (Φ lr ) we have
[Equation]
Proof.
We start by showing that the V-type Bellman rank for function class F on in the low-rank case is small.
To that end, consider the Bellman error defined for any roll-in policy π and function f ∈ F on :
[Equation]
where we used Lemma 9 for low-rank MDPs to write E [ f h +1 ( x h +1 , a h +1 )
| x h , a h , a h +1 ∼ π f ] as ⟨ ϕ lr h ( x h , a h ) , θ ∗ f,h ⟩ .
Here, f h +1 ∈ [ − 2( H − h − 2) , 2( H − h − 2)] and f h ∈ [ − 2( H − h − 1) , 2( H − h − 1)] implying that f h − (cid:68) ϕ
lr h ( x h , a h ) , θ ∗ f,h (cid:69) ∈ [ − 4( H − h − 1) , 4( H − h − 1)] .
Therefore, using
Lemma 9 again, we have:
[Equation]
where ∥ ˜
θ ( f ) ∥ 2 ≤ 4( H − h − 1) √ d lr and ( ν ( π ))
( x h − 1 , a h − 1 )
=
E [ ϕ lr h − 1 ( x
[Equation]
h − 1 ∼ π ] .
Hence, the V-type Bellman rank for this function class is bounded by d lr with normalization parameter 4( H − h − 1) √ d lr .
Finally using Proposition 3, we get the desired bound on the Bellman Eluder dimension.
For linear MDPs, where the feature ϕ lr is the known feature case, we show that its Q-type Bellman
Eluder dimension is also small:
Proposition 5 (Linear MDP) .
Consider a low-rank MDP M (Definition 8) with embedding dimension d lr and ϕ lr is known.
Define the corresponding linear class F ( { ϕ lr } )
=
F 0 ( { ϕ lr } , H − 1) × . . .
×
F H − 1 ( { ϕ lr } , 0) using
F h
[Equation]
{ ϕ lr } , B h )=
[Equation]
x h , a h )
[Equation]
x h , a h )
[Equation]
θ
[Equation]
: ∥
[Equation]
lr , ⟨ ϕ lr
[Equation]
· ) , θ
[Equation] [Equation]
[ − B h , B h ] (cid:111) .
Then, for the difference class F on = F ( { ϕ lr } ) − F ( { ϕ lr } ) we have
[Equation]
(cid:19)(cid:19) .
and dim 0 vbe ( F on , D F on , ε ) ≤ O (cid:18) 1 + d
[Equation]
log (cid:18)
[Equation]
H √ d
[Equation]
ε (cid:19)(cid:19) .
Proof.
The V-type Bellman Eluder dimension bound (Proposition 4) implies the same upper bound for dim 0 vbe ( F on , D F on , ε ) , where F on is defined using the singleton feature class (cid:8)
ϕ lr (cid:9) .
For Q-type
Bellman Eluder dimension, we again start with the Q-type Bellman rank.
For any f ∈ F on , we have:
[Equation] [Equation]
[Equation] [Equation]
[Equation]
Using the same magnitude calculations for θ ∗ f,h , we have ∥ θ h − θ ′ h − θ ∗ f,h ∥ 2 ≤ 4( H − h − 1) √ d lr .
Therefore, we again have the Q-type Bellman rank bounded by d lr with normalization parameter 4( H − h − 1) √ d lr .
Using Proposition 3, we get the same bound on the Q-type Bellman Eluder dimension.
Linear completeness setting For the linear completeness setting in the known feature case, we show that its Q-type Bellman Eluder dimension is small.
Proposition 6 (Linear completeness setting) .
Consider an MDP M that satisfies linear completeness (Definition 6) with feature ϕ lc .
Define the corresponding linear class F ( { ϕ lc } )
=
F 0 ( { ϕ lc } , H − 1) × . . .
× F H − 1 ( { ϕ lc } , 0) using
[Equation]
Then, for the difference class F on = F ( { ϕ lc } ) − F ( { ϕ lc } ) we have:
[Equation]
Proof.
Consider the Q-type Bellman rank, for any f ∈ F on , we have:
[Equation]
where the penultimate step follows from Definition 6 with the ∥ θ h − θ ′ h ∥ 2 , ∥ θ ∗ f,h ∥ 2 , ∥ θ h +1 − θ ′
h +1 ∥ 2 ≤ 2( H − h − 1) √ d lc .
Thus, we have ∥ θ h − θ ∗ f,h ∥ 2 ≤ 4( H − h − 1)
√ d
lc implying a Q-type Bellman rank bound of d lc with normalization parameter 4( H − h − 1) √ d lc .
Using Proposition 3, we get the stated bound on the Q-type Bellman Eluder dimension for linear completeness setting.