A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximation
Abstract
The current paper studies sample-efficient Reinforcement Learning (RL) in settings where only the optimal value function is assumed to be linearly-realizable.
It has recently been understood that, even under this seemingly strong assumption and access to a generative model, worst-case sample complexities can be prohibitively (i.e., exponentially) large.
We investigate the setting where the learner additionally has access to interactive demonstrations from an expert policy, and we present a statistically and computationally efficient algorithm (D ELPHI ) for blending exploration with expert queries.
In particular, D ELPHI requires ˜ O
( d ) expert queries and a poly ( d, H, |A| , 1 /ε ) amount of exploratory samples to provably recover an ε -suboptimal policy.
Compared to pure RL approaches, this corresponds to an exponential improvement in sample complexity with surprisingly-little expert input.
Compared to prior imitation learning (IL) approaches, our required number of expert demonstrations is independent of H and logarithmic in 1 /ε , whereas all prior work required at least linear factors of both in addition to the same dependence on d .
Towards establishing the minimal amount of expert queries needed, we show that, in the same setting, any learner whose exploration budget is polynomially-bounded (in terms of d, H, and |A| ) will require at least ˜Ω( √ d ) oracle calls to recover a policy competing with the expert’s value function.
Under the weaker assumption that the expert’s policy is linear (rather than their value function), we show that the lower bound increases to Ω( d ) .
Introduction
Many potential applications of reinforcement learning (RL) have intractably-large state spaces.
Thus, we seek provably-correct methods which have statistical and computational requirements that are independent of the size of the state space.
This requires some modelling assumptions.
One dominating approach has been to introduce function approximation and to posit that the MDP or its value functions are well-represented by the function approximation scheme which is employed.
A basic starting point which still lacks comprehensive understanding is the case of linear value function approximation, which models value functions as lying in the span of a known d -dimensional feature mapping and asks for methods which have sample complexities that are polynomial only in d, H, and possibly |A| ( H and A are the horizon and action sets of the MDP, respectively).
This desideratum was recently understood to be impossible for the “minimal” case where only the optimal value function (or optimal action-value function) is assumed to be linear – i.e. there exist MDPs satisfying this assumption where the statistical complexity of any algorithm will be exponentially large, either in d or in H [WAS21; WSG21; WWK21; FKQR21].
Furthermore, this also holds in the case where the learner is equipped with a generative model (or simulator), allowing them to sample transitions from any state-action pair of the MDP.
In recent years much has been said about linear value approximation under stronger assumptions, for example under determinism
[WR13], linear/low-rank MDPs [JYWJ20; AJSWY20], Bellman-closedness [LSW20; ZLKB20], or the existence of a “core set” [SS20; ZLKB19].
These stronger assumptions can recover polynomial statistical complexities (thought not always computational ones), but are restrictive and oftentimes unrealistic.
In this work, we consider an alternative possibility for recovering polynomial complexities which does not further restrict the class of MDPs under consideration.
We do this by assisting the learner with some additional side information about the problem.
Specifically, we assume that there is a deterministic expert policy (which need not be the optimal policy) that the learner can query at any state, whereupon they will be informed of the expert’s action at that state.
Indeed, such information can often be made readily available if we have some form of prior knowledge (or human input) about the problem.
Leveraging such expert demonstrations has been studied in interactive imitation learning (IL), with common applications in simulated domains [RGB11; Ros13; RB14; SVGBB17].
As we will see, however, the amount of expert queries required by a pure IL approach is significantly higher than what we need.
Since interaction with a (human) expert might be costly, we wish to minimize the burden of the expert by having the learner explore mostly on their own, and only query the expert in a judicious manner.
The question asked by this work, then, is:
Under linearity of the optimal value function, what is the minimal amount of expert data required for sample/computationally-efficient learning?
Our main result is the D ELPHI algorithm for exploring with an interactive expert.
D ELPHI assumes that the expert’s value function is linear, and that the agent has access to a RESET function which lets them return to the state most recently seen .
Under these conditions, our method uses surprisingly-few expert queries combined with some modest (polynomial) amount of exploration to recover the expert policy.
Formally, D ELPHI recovers a policy that is ε − optimal (with respect to the expert policy) with O ( d log( B/ε )) oracle calls and ˜ O ( d 2 H 5 |A| B 4 ε) exploratory samples, where B is a bound on the ℓ 2 -norm of the unknown linear parameter.
Thus, our results show that merely ˜ O
( d ) expert queries enable an exponential improvement in sample complexity when compared to RL without expert advice.
Furthermore, the number of oracle calls is completely independent of the horizon of the problem and is logarithmic in 1 /ε , whereas prior work in IL leveraging expert advice requires (at best) linear factors of both in addition to scaling with d .
We also show that D ELPHI is computationally efficient, that it is robust to optimization and misspecification errors, and that it can be extended to the case where the action -value function of the expert is linear when the MDP dynamics are deterministic.
Towards establishing the optimality of our algorithm, we study the capabilities of expert-augmented learners which have fixed exploration budgets.
More specifically, we ask: what is the minimal number of expert queries required by any algorithm which is constrained to a polynomially-bounded exploration budget?
We show that any polynomially-bounded learner (in terms of d, H, and |A| ) will require at least ˜Ω( √ d ) oracle calls to recover a policy competing with the expert’s value function.
In the weaker setting where only the expert’s policy is linear, we show that this lower bound increases to Ω( d ) , matching our upper bound up to logarithmic factors.
The rest of the paper is structured as follows: in Section 2 we review background and present the problem setting.
Section 3 describes our algorithm, its guarantee, a sketch of the proof, and discusses some extensions.
Section 4 gives the lower bound on the number of expert queries needed.
We conclude with an overview of related work and some discussion in Sections 5 and 6.
Background & Problem Setting
Notation We write Dists( X ) for the set of probability measures on some set X .
We write [ N ] := { 1 , . . .
, N } .
The direct product ⊕ corresponds to “concatenating” two vectors, i.e. for any two vectors u ∈ R n and v ∈ R m , we have u ⊕ v =
( u 1 , . . .
, u n , v 1 , . .
. , v m )
⊤ ∈ R n + m .
We write ⊗ for the tensor (or outer) product of two vectors, defined by u ⊗ v = uv ⊤ ∈ R n × m , and ♭ ( u ⊗ v ) ∈ R n · m for the flattening (or vectorization) of said tensor product.
MDPs, Policies, and Value Functions
The typical environment in RL is modelled as an MDP [Put14; Sze10; SB18].
We consider here finite-horizon MDPs, which are specified by a tuple M = ( S , A , R , P , H, µ 0 ) , where S is a state space,
A
=
[ A ] is a finite action set, R : S×A → Dists([0 , 1]) is a (bounded) reward distribution function with expectation r ( s, a ) , P : S ×
A → Dists( S ) is the transition distribution function with probability vectors P ( s, a ) =
[ P ( s ′ | s, a )] s ′ ∈S ∈ R |S| , H ∈ N is the horizon, and µ 0 ∈ Dists( S ) is the starting distribution.
Note that we have assumed that the action space is finite, although the state space may be infinite.
Without loss of generality we assume that S is a disjoint union of per-horizon state spaces, i.e. S = ∪ h ∈
[ H ] S h .
A (non-stationary) policy π = ( π 1 , · · · , π H ) prescribes a sequence of actions π h : S h
→ Dists( A ) , and its value function is v π ( s ) =
E [ (cid:80) Hh ′ =
h r ( s h ′ , a h ′ ) | s h
= s, a h ′ ∼ π h ′
( s h ′ )] , where s ∈ S h .
The action-value function q π ( s, a ) is defined similarly, save that the first action taken is a and the proceeding actions follow π .
Value functions satisfy the recursive relationship:
[Equation]
(1) where we have the shorthands r ( s, π ) =
E a ∼ π ( s )
[ r ( s, a )] , P ( s, π ) =
E a ∼ π ( s ) [ P ( s, a )] , and the
Bellman operator T π ( · ) :
= r ( s, π ) + ⟨ P ( s, π ) , ( · ) ⟩ .
The Bellman operator has v π as its unique fixed point.
The optimal policy is written as π ⋆ , and its value function is denoted as v ⋆ := v π ⋆ .
The objective is to find a π maximizing v π ( µ 0 ) :
=
E s 0 ∼ µ
[ v π ( s 0 )] .
Function Approximation With an Interactive Expert In the RL setting, the MDP is unknown and must be explored.
As stated in the introduction, we seek sample complexities which are independent of the number of states.
This is evidently not possible without further assumptions.
To assist the learner, we assume that the agent has further access to an oracle , which, upon being queried, returns the action of an expert policy π ◦ for the state.
The expert policy need not be the optimal policy.
We will assume that π ◦ is deterministic, which is satisfied for example when π ◦ =
π ⋆ or when π ◦ is the greedy policy of some value function.
Assumption 2.1 (Interactive expert) .
There is an oracle which can be queried at the current state s , which returns the action π ◦ ( s ) .
Syntactically, the oracle is queried via the ORACLE ( s ) function.
We note that the oracle can only be queried at the state the learner is currently visiting.
The objective of the learner is to recover a policy which competes with the expert policy, namely a ˆ π such that:
[Equation]
where v ◦ :
= v π ◦ is the value function of the expert.
In the sequel we refer to a policy satisfying
Equation 2 as ε -optimal .
Our next assumption concerns the interplay of the expert policy with linear value approximation.
Namely, we assume that the expert policy has a value function which is linear in a set of known features.
Assumption 2.2 ( v ◦ -linearity, with bounded features) .
The value function v ◦ of the expert is linear with known features φ ∈ R d , i.e.
[Equation]
for some unknown θ ◦ ∈ R d .
We further assume that ∥ φ ( s ) ∥ 2 ≤ 1 ∀ s and that ∥ θ ◦ ∥ 2 ≤ B for some known B ∈ R d .
This assumption can be extended to the case of (slight) misspecification, as will be shown in
Subsection 3.2.
Our last assumption is that the agent has the ability to “reset” to the state most recently seen.
Assumption 2.3 (Resets) .
After experiencing a transition ( s, a, r, s ′ ) in the MDP, the agent can return to the state s .
Syntactically, this is done via the RESET () function.
Our RESET assumption is weaker than the full generative model access [KMN02] or the “local” simulator setting [WAJAYJS21; LCCGW21; HLYAYS22] which has appeared in prior works.
As noted in the introduction, existing exponential lower bounds entail that Assumptions 2.2 and 2.
together are not enough to enable sample-efficient learning.
Thus, any efficient learner which is provided with the above assumptions (2.1, 2.2, and 2.3) must necessarily make use of the expert.
The D ELPHI algorithm
We are ready to describe our approach and give the main result.
We begin by supposing that the starting distribution is deterministic (we will see later that this comes at no loss of generality).
Theorem 3.1.
Suppose Assumptions 2.1, 2.3, and 2.2 hold.
Then the D ELPHI algorithm will recover a policy ˆ π such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − ε with probability ≥ 1 − δ , using O ( d ln( B/ε )) oracle calls and ˜ O
( d 2 H 5 AB 4 ε) interactions with the MDP.
Furthermore this algorithm is computationally efficient.
The pseudo-code for D ELPHI is given in Algorithm 1, which uses Algorithm 2 ( measureTD ) as a sub-routine to measure expectations.
D ELPHI will solve for a parameter ˆ θ such that an induced policy π ˆ θ (defined via Eq.
(4)) will compete with the expert’s value function.
We note that deploying the policy π ˆ θ also requires the RESET functionality, since expectations must be estimated from a small number of samples at each state encountered.
This is a consequence of our assumption that v ◦ is linear (rather than, e.g., q ◦ or the MDP itself), since selecting actions based on state-value functions will always require one-step look-aheads.
Intuition for D ELPHI
Recall that the expert policy π ◦ satisfies v ◦ =
T π ◦ v ◦ , and that this fixed point is unique.
We say that a candidate value function is consistent at a state s if v ( s ) =
T π ◦ v ( s ) .
Note that we need consistency to hold globally (i.e. at all states) in order to ensure that v = v π ◦ .
Our methodology is based on ensuring that consistency on a small number of well-chosen states will guarantee global consistency.
D ELPHI is inspired by a recent algorithm of [WAJAYJS21] called TensorPlan.
As in TensorPlan,
D ELPHI proceeds via a “guess and check” procedure: at every iteration, we pick the optimistic linear parameter which is consistent on the past expert data that we have seen.
We then check whether this choice of parameter is globally consistent, by playing n rollout rollouts of length H with a policy derived from the parameter.
More specifically, for any θ the policy
π θ takes the form
[Equation]
where v θ ( · ) := ⟨ φ ( · ) , θ ⟩ , and ˆ r
( s, a ) and ˆ E s ′ ∼ P
( s,a ) are estimated via the RESET function.
After a certain number of rollouts, one of two things happen: either this policy encounters a state where there is no consistent action (i.e. the above minimum has a strictly positive value), or we only encounter states that are consistent.
In the first case, we query the oracle for its expert action and use the transition for that action to update the parameter set.
In the second case, we derive (cf.
Lemma 3.7) that if no inconsistencies are observed for several rollouts, then our “virtual value” v θ is close to the true value under π θ (i.e., v π θ ).
Using that θ was optimistic (i.e. v θ ( s 0 ) ≥ v θ ◦ ( s 0 ) ), this implies that we are optimal.
The only thing left to argue is that the number of iterations (i.e. the number of times that we can continue finding new parameters which are not globally consistent) is small.
Using linearity of v ◦ , it turns out roughly d inconsistencies are sufficient for this.
To see this, note that we can re-write the
Bellman equation for any v θ ( · )
= ⟨ φ ( · ) , θ ⟩ as:
[Equation]
where we have introduced the notation ∆ s,a :=
r ( s, a ) ⊕ ( E [ φ ( s ′ )] − φ ( s )) and used linearity of expectation, linearity of inner products, the definition of v θ , and the definition of the direct product.
We call the vector ∆ s,a the temporal difference (TD) vector for ( s, a ) .
Equation (5) is precisely an orthogonality constraint in d + 1 dimensions.
Since the parameter θ t which is chosen at time t was consistent on past data, it is orthogonal to the previous t − 1 TD vectors which have been generated from interactions with the oracle.
If we happen to find a state which has no consistent action, then the TD vector corresponding to the expert action at that state must not be in the span of the previous expert TD vectors (otherwise it would be consistent).
It follows that the iteration complexity is at most d + 1 , since there are at most d + 1 linearly independent vectors in R d +1 .
We use the Eluder dimension [RVR13] to generalize this argument to the case where the expectations are estimated.
The next section solidifies the above intuition and sketches the proof more formally.
2: Θ 0 ← do 5:
Pick θ
∈ θ ∈
Θ t − 1 (cid:0) v θ ( s 0 ) :
= θ ⊤ φ ( s 0 ) (cid:1)
▷ Optimistic choice over Θ t −6:
consistent ← true 7: for m = 1 to n rollout do ▷ n rollout number of rollouts with θ t -induced policy 8:
S t,m,h = s▷
Initialize rollout 9: for h to H do 10: for a ∈ [ A ] do ▷ each action 11: ˆ∆ S t,m,h ,a ← measureTD ( S t,m,h , a, n eval )
▷ Measure TD vector at ( s, a ) 12: end for 13: if min a (cid:12)(cid:12)(cid:12) ⟨ ˆ∆ S t,m,h ,a , 1 ⊕ θ t ⟩ (cid:12)(cid:12)(cid:12) > ε tol then ▷
No consistent action 14: consistent ← false 15: a ◦ t ← ORACLE ( S t,m,h )
▷ Query oracle for π ◦ ( S t,m,h ) 16:
˜∆ S t,m,h ,a ◦ t ← measureTD ( S t,m,h , a ◦ t , 4 E d n eval )
▷ Refined data 17: Θ t ← Θ t − 1 ∩ { θ | |⟨ ˜∆ S t,m,h ,a ◦ t , 1 ⊕ θ ⟩| ≤ ε tol } ▷ New admissible parameters 18:
Exit current iteration, t
Line
19: end if 20:
(cid:12)(cid:12)(cid:12) ⟨ ˆ∆ S t,m,h ,a , 1 ⊕ θ t ⟩ (cid:12)(cid:12)(cid:12) ▷ Else
consistent, keep playing 21:
Play A t,m,h , get R t,m,h , S t,m,h +1 ∼ MDP ▷ Roll forward 22: end for 23: end for 24: if consistent =
= true then 25: return θ t
▷ No inconsistency for m rollouts = ⇒ success 26: end if 27: end for 28: return θ E d +
Algorithm
1: Inputs: s, a, φ ( · ) , n, RESET () 2: for i = 1 to n do 3:
Play action a at s , receive sample R l and S
′ l from MDP 4: ∆ i ← R l ⊕ ( φ ( S ′ l ) − φ ( s )) 5:
RESET () 6: end for 7: return ˆ∆ s,a := 1 n (cid:80) i ∈ [ n ] ∆ i
3.1 Proof sketch
The full proof comes in 4 parts.
The proofs for all Lemmas and the precise values of all hyperparameters are given in Appendix A.
Lemmas 3.2 and 3.3 gives concentration bounds which establish that, with high probability, the measurements ˆ∆ (Line 11) and ˜∆ (Line 16) concentrate to the average ∆ s,a = E R ( s,a ) r ⊕ (cid:0)
E P ( s,a ) φ ( s ′ ) − φ ( s ) (cid:1) .
Lemma 3.4 establishes that, with high probability, the true optimal parameter θ ◦ is not eliminated from Θ t for any parameter set that is encountered.
It follows (Lemma 3.5) by optimism that v θ t ( s 0 ) ≥ v ◦ ( s 0 ) with high probability, where θ t is the parameter chosen at time t .
Lemma 3.6 establishes an iteration bound: the algorithm will terminate after at most t =
E d iterations of the outermost loop (and thus after at most E
Lastly, Lemma 3.7 establishes that if n rollout number of rollouts occur without observing a consistency break, then the virtual value ( v θ ( s 0 ) ) must be roughly equal to the true value under the executed policy ( v π θ ( s 0 ) ).
Theorem 3.1 combines all the ingredients to conclude the proof.
Part 1: Concentration bounds Recall that ∆ s,a :=
r ( s, a ) ⊕ ( E [ φ ( s ′ )] − φ ( s )) is the true TD vector, ˆ∆ is the estimated TD vectors obtained with n eval samples (in Line 11), and ˜∆ is the “refined data” obtained with 4 E d n eval samples (in Line 16).
The following lemmas establish concentration of ˆ∆ and ˜∆ to the true TD vector.
Lemma 3.2 (Concentration of ˆ∆ s,a (Line 11)) .
For any s, a ∈ S ×
A that is observed throughout the execution D ELPHI , with n eval samples in Line 11, we have that with probability ≥ 1 − δ , (cid:13)(cid:13)(cid:13) ˆ∆
s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤
ε eval and thus that ⟨ 1 ⊕ θ, ˆ∆ s,a − ∆ s,a ⟩ ≤ ¯ ε eval .
Lemma 3.3 ( ˜∆ s,a concentrates even more (Line 16)) .
Similarly, for all s, a where we call the oracle, with probability 1 − δ , we have (cid:13)(cid:13)(cid:13) ˜∆ s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤ ε eval / (2 √ E d ) , and thus, ∀ θ ∈ Ball ℓ 2 ( B ) , |⟨ 1 ⊕ θ, ˜∆ s,a − ∆ s,a ⟩| ≤ ¯ ε eval / (2 √ E d ) .
Part 2: Optimism This part shows that (with high probability) the true optimal parameter is not eliminated from the version space, and thus by optimism that the predicted value v t upper bounds v ◦ .
Lemma 3.4 ( θ ◦ not eliminated) .
With probability ≥ 1 − δ , θ ◦ ∈ Θ t for all iterations t ∈ [ E d + 1] .
Lemma 3.5 (Optimism) .
Under the event of Lemma 3.4, we have v t ( s 0 ) ≥ v ◦ ( s 0 ) , ∀ t ∈ [ E d ] .
Part 3: Iteration bound To bound the iteration complexity of our algorithm, we use the notion of
Eluder dimension [RVR13].
Loosely, the Eluder dimension with respect to some target function is the longest sequence of points ( x i ) such that there exists functions differing from the target function on x i but which correctly fit it on x 1 , . . .
, x i − 1 .
A formal definition is provided in Appendix A. We will use the result that the Eluder dimension of linear functions is O ( d ln( B/ε ))
[RVR13; LKFS21].
Lemma 3.6 (Iteration Complexity) .
With probability ≥ 1 − 2 δ , the iteration complexity of the algorithm is at most the Eluder dimension at scale ¯ ε eval , i.e. E d = O
( d ln( B/ ¯ ε eval )) .
Part 4: Consistency, and putting everything together The last thing to show is that if we are consistent at all states for several rollouts then the virtual value v θ will be close to the value of the v π θ .
Lemma 3.7 (Consistency = ⇒ accurate prediction) .
If m rollouts have occured without any inconsistencies (i.e., the if statement of Line 13 never gets triggered), then v π θ ( s 0 ) > v θ ( s 0 )
− 5 H ¯ ε eval − ε roll with probability ≥ 1 − 3 δ .
We are now ready to put everything together.
Proof (of Theorem 3.1).
Assume all events introduced so far (i.e. the events in Lemma 3.2, Lemma 3.3, and Lemma 3.7).
Together these happen with probability ≥ 1 − 3 δ , so we can re-define δ (cid:55)→ δ/such that the events happen together with probability ≥ 1 − δ (this only increases factors inside logarithms by 3 ).
By Lemma 3.7, we have:
[Equation]
where the second step follows from plugging in the definitions of n eval (Eq.
(12)), n rollout (Eq.
(10)), ¯ ε eval , (Eq.
and ε roll (Eq.
and the final step follows by optimism (Lemma 3.5).
The total sample complexity of our algorithm is: E d = ˜
O
( d ) oracle calls, and Nn eval =
( E d + 1)
Hn rollout An eval =
˜ O
( d 2 H 5 AB 4 ε 4 target ) exploration cost.
The last claim to verify is that of computational efficiency.
We note that the only computationally intensive step from iteration is
Line 5, i.e. the optimization problem corresponding to the optimistic choice over the parameter set max
[Equation]
θ ⊤ φ ( s 0 ) .
This is readily seen to be a convex program, since the objective is a linear function and the constraint set is a convex set in R d (it is initialized at Θ 0 =
Ball ℓ 2 ( B ) and every update intersects it with half-spaces (Line 17)).
Besides the ℓ 2 -norm constraint, all other constraints can be represented with a linear program, since each absolute value constraint can be split into two inequalities.
By Lemma 3.6 the number of constraints will be at most 2 E d =
˜ O
( d ) (and these can easily be converted to standard form).
Thus, there are a plethora of polynomial-time methods for solving this convex program [BBV04; Bub+15].
3.2 Extensions
In this section, we show that D ELPHI can be extended to work with optimization errors, under stochastic starting distributions, with misspecification, and with linear q ◦ whenever dynamics are deterministic.
Optimization errors and stochastic start state If the optimistic program can only be solved upto accuracy ε opt ≤ cε target (for c ∈ [0 , 1) ) at each iteration, then Lemma 3.5 becomes v t ( s 0 ) ≥ v ◦ ( s 0 ) − ε opt
and this only appears in Eq.
We then need 5 H ¯ ε eval + ε roll ≤ (1 − c ) ε target , which is achieved by increasing n rollout and n eval by a factor of 1 / (1 − c ) 2 .
This increases the final exploration cost by 1 / (1 − c ) 4 .
For stochastic starting states, we simply work with v θ ( µ 0 ) =
E s 0 ∼ µ
[ v θ ( s 0 )]
(resp.
v ◦ ( µ 0 ) )
wherever v θ ( s 0 ) (resp.
v ◦ ( s 0 ) ) previously appeared.
The “starting feature” E s 0 ∼ µ
[ φ ( s 0 )] must be estimated from samples, which is then used for the optimistic program in Line 5 with φ ( s 0 ) replaced by this expectation.
The error is easily bounded as before by Hoeffding’s inequality, and will simply propagate additively through the proof.
Misspecified value functions and innacurate simulators
D ELPHI inherits some robustness properties from TensorPlan.
Namely, with a constant increase in exploration cost, D ELPHI continues to work under errors in the modelling assumptions.
The first case is where the expert value function is not linear but rather is approximately linear up to some uniform error.
Formally, we say that
the
MDP is η -misspecified for the expert policy π ◦ and the feature map φ if there exists θ ◦ such that sup s | v ◦ ( s ) − ⟨ φ ( s ) , θ ◦ ⟩| ≤ η .
The second case is where the simulator itself is flawed.
Formally, we say that the simulator is λ -innacurate if a transition ( r, s ′ ) from any state-action pair ( s, a ) of the MDP is instead observed as (Π( r + λ s,a ) , s ′ ) , where Π is the projection onto [0 , 1] and λ s,a is a constant uniformly bounded by λ .
The following result (proved in Appendix A.5) states that D ELPHI can tolerate misspecification or simulator inaccuracies of order roughly ¯ ε eval √ E d
= O ( 1 H √ d ) .
Theorem 3.8 (D ELPHI with misspecification) .
Redefine n ′ eval = 4 n eval (Eq. 12) and all subsequent hyperparameters which depend on n ′ eval .
Then we have that, for all MDPs that are at most ¯ ε eval 8 √ E dmisspecified or for all simulators that are at most ¯ ε eval 4 √ E d -innaccurate, the conclusions of Theorem 3.
continue to hold when running D ELPHI with the new hyperparameters.
q ◦ -linearity, in deterministic dynamics Rather than working with the Bellman equation v θ ( s ) =
T π ◦ v θ ( s ) we work with q θ ( s, a ) =
T π ◦ q θ
( s, a ) , which can be linearized similarly to Eq.
Namely:
[Equation]
This derivation holds generally, although to be able to speak of consistency at ( s, a ) with respect to a next action a ′ , we now assume deterministic dynamics, so that the above becomes
[Equation]
where s ′ is the unique successor of ( s, a ) and a ′ is the action that we are checking consistency for.
The algorithm proceeds as before, except rather than checking all actions at a given state (Line 13), we check all proceeding actions a ′ against the current ( s, a ) , and play the one with the smallest TD error (analogously to Line 20 of Algorithm 1).
How many oracle calls are necessary?
Is D ELPHI optimal in terms of its number of oracle calls?
To answer this question, we must argue that no algorithm can find an ε − optimal solution with less than Ω( d ) expert queries.
As stated, we are competing with agents which can (for example) exhaustively search the state-space and do not refer to the expert at all.
Thus, an exploration budget must be imposed (formally, a maximal amount of allowed interaction with the MDP, excluding oracle calls).
To make matters more interesting, we set this cap to be any polynomial amount, resulting in the following question:
Under Assumptions 2.1, 2.2, and 2.3, what is the minimal amount of expert queries needed for any algorithm with a poly ( d, H, A, 1 ε ) exploration budget to find an ε − optimal solution?
We firsly note that, measured in the worst-case, this minimal amount of expert queries is strictly positive since there exists MDPs satisfying v ⋆ -linearity for which no algorithm can return a sound solution with poly ( d, H, A, 1 ε ) queries [WAS21; WSG21].
Secondly, while it was possible to restrict ourselves simply to learners which have the same exploration requirements as D ELPHI , we opted to study algorithms with arbitrary polynomial exploration budgets, since it is a more fundamental question about the limits of exploration and the benefits of expert advice.
Note that any solution to this question must, a priori, have an exponential sample complexity for pure RL (otherwise the agent does not need to resort to the expert).
Interestingly, most constructions which exhibit exponential lower bounds for linearly-realizable RL can be solved with a single query from the oracle (e.g., [WAS21; WWK21]).
These constructions rely on having an exponentially large action set with a single correct action that effectively solves the MDP.
Our main lower bound comes from extending the recent lower bound of [WSG21], which is also the only known construction for an exponential lower bound which has a polynomial action set (rather than exponential).
Our result is that at least ˜Ω( √ d ) oracle calls are necessary:
Theorem 4.1.
There exists a family of MDPs satisfying 2.1, 2.2, and 2.3, such that any algorithm with poly ( d, H, A ) exploration budget will need at least ˜Ω( √ d ) oracle calls to recover a policy such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − 0 .
Our second lower bound considers an alternative assumption, which instead posits that the expert policy is linear.
Formally, this assumption says:
Assumption 4.2 ( π ◦ -linearity, with bounded features) .
The policy π ◦ of the expert is linear with known features φ :
[Equation]
d , i.e.
[Equation]
for some unknown θ ◦ ∈ R d \ { 0 } .
5 We further assume that ∥ φ ( s ) ∥ 2 ≤ 1 ∀ s and that ∥ θ ◦ ∥ 2 ≤ B for some known B ∈ R d .
When π ◦ = π ⋆ , it is easy to see that this assumption is implied by the assumption that q ⋆ is linearlyrealizable.
In general however, q ◦ -linearity does not imply that π ◦ is linear (see Appendix C for an example), although it does imply that the greedy policy derived from q ◦ is linear.
We give a lower bound for this case which matches the upper bound of D ELPHI up to logarithmic factors.
Theorem 4.3.
There exists a family of MDPs satisfying assumptions 2.1, 4.2, and 2.3 such that any algorithm with poly ( d, H, A ) exploration budget will need at least Ω( d ) oracle calls to recover a policy such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − 0 .
Intuition for the lower bound Theorems 4.1 and 4.3 use the same MDP construction but with different features.
We give some intuition for the MDP construction which is used, but due to its intricacy a full description (and the information-theoretic proof) are deferred to Appendix B. Loosely, the learner has to find a hidden hypercube vector s ⋆ ∈ {± 1 } p .
The action space is A =
[ p ] , and each action corresponds to flipping one of the bits of the current vector.
The MDP has K “phases” which each correspond to p bit flips (thus H ≈ Kp ).
A linear reward is given only if a sufficiently small neighborhood of s ⋆ is reached, and the reward (thus the value) will decay geometrically in each subsequent phase that the neighborhood is not reached.
Intuitively, the oracle needs to be used
≈ p times, since each oracle calls only reveals one action, and thus one bit of the optimal vector s ⋆ .
The reason that this results in a ˜Ω( √ d ) lower bound (rather than ˜Ω( d ) ) is that the value function will experience a scale transition when going from states where s ⋆ is reachable given the remaining steps in the current phase to states where s ⋆ is no longer reachable.
As just described, the value (and thus the features) will be one order of magnitude smaller in this latter portion of the state space.
As this would betray the location of the secret parameter, the value function is instead augmented to be quadratic in p (roughly, the product of the distance achieved at the end of this phase and that of the next phase), which requires that p ≈ √ d in order to observe linearity.
On the other hand, the lower bound for π ◦ -linearity (Theorem 4.3) can remain linear in d , since the definition (Eq.
8) is scale-insensitive .
Closing the gap between our upper bound of ˜ O
( d ) and our lower bound of ˜Ω( √ d ) remains a challenging but interesting question.
In finite horizons, with access to a generative model or a RESET method, we suspect that the only mechanism for creating a hard MDP is by geometrically decaying the maximum possible value for each stage, such that at the final stage of the MDP the (random) reward becomes exponentially small in H (that is the approach taken here and in [WAS21; WSG21]).
If the geometric decaying happens in “phases” then this implies that Ω(1) of the value is located in the first “phase”.
The tension, then, is to have such a construction while hiding this large value, and forcing the learner to rely on several oracle calls to find it.
Prior constructions [WAS21; WWK21] have hid the large initial value by choosing an exponentially large action set, although as discussed above these examples are solved with a single query of the expert oracle.
Extending the “needle in a haystack” to occur over multiple decisions (cf.
the “phases” used above) leads to increased oracle requirements, although due to the scale transition phenomenon observed above it is far from clear how to do this with phases of length ≈ d .
The question thus boils down to: is geometric value reduction necessary for exponential lower bounds in this setting, and if so can we avoid the scale transition problem?
The situation is likely to be different in the online RL setting.
For example, the construction of [WWK21] extends [WAS21] to the online setting, and does not need to decay rewards but instead adds an Ω(1) probability of death at every transition.
This mechanism evidently does not work when the agent has resets, and since it is not known whether D ELPHI can be extended to the online setting we opted to keep the settings consistent between our upper bound and our lower bound.
Related works
The closest body of work to our setting is the field of interactive IL.
As in our setting, interactive IL considers the case where the learner has access to an expert oracle that can be queried adaptively.
It differs from our setting, however, since traditionally in IL the learner does not observe reward information.
We further differ from the IL setting since we consider value function approximation rather than general policy classes, and since we assume access to a RESET function.
Despite that many demonstrations of interactive IL occur in simulated domains [RGB11; Ros13; RB14], the benefits of this feature have not previously been studied.
Our assumption of v ◦ linearity entails that many IL methods are not directly applicable.
Indeed, the policy π ◦ itself does not need to be linear (despite that v ◦ is), so it is unclear which policy class to use for those algorithms.
Assuming for the sake of comparison that linear policies can be used, IL methods would still obtain worse oracle rates.
Indeed, using results from [AJKS19], Behaviour Cloning (for the passive case) or
AggreVaTe [RB14; SVGBB17] (for the interactive case) have worst-case errors O (
[Equation]
) for discounted MDPs, which roughly translates to an oracle complexity of N = O ( dH 4 /ε 2 ) when using the standard reduction H (cid:55)→ (1 − γ ) − 1 .
This is in sharp contrast to our ˜ O ( d ) oracle calls, which in independent of H and logarithmic in 1 /ε , and demonstrates the improvement due to exploration with the help of value-function approximation.
Beyond these approaches, another intuitive method would be to perform regression by doing a Monte Carlo estimation for the value of v ◦ ( s ) for each s along a certain “good” set of features.
This would require collecting rollouts from those states, which will again introduce a factor of H in the number of oracle queries.
Our algorithm instead finds a set of state-action pairs where the Temporal difference (TD) errors (which we represent as vectors) span orthogonal directions.
These can be estimated with a single transition, and this "local fitting" approach is novel to the IL literature and avoids the factors of H and 1 /ε from previous works.
In terms of other linear structure in IL, the works of [AN04; SS07] assume a known transition function and unknown linear rewards, and derive expert complexities of O (cid:16) dH 2 log( dH/ε ) ε(cid:17) and O ( H 2 log( d ) ε) respectively, but the algorithms involve (tabular) planning in MDPs and thus are not computationally efficient.
Most relevant is the recent work of [RHYLJR21], which, in the reward-free case, assumes that the expert policy is linear (i.e., our Assumption 4.2).
A sample complexity of ˜ O ( dH/ε ) is shown for Behaviour Cloning in this case, though no lower bound is given.
On the technical side, the D ELPHI algorithm is inspired by a recent algorithm called TensorPlan
[WAJAYJS21].
TensorPlan works for pure RL under Assumptions 2.2 and 2.3 but has a sample complexity scaling as poly ( (cid:0) dHε (cid:1) A , B ) and is computationally intractable.
Our extension of TensorPlan naturally incorporates the expert demonstrations, while simultaneously (1) having low oracle requirements, (2) addressing the exponential sample complexity of TensorPlan, and (3) rendering the algorithm computationally efficient.
Our approach is based only on finding value functions which satisfy the Bellman equation.
Bellman error minimization approaches have appeared in other works (e.g. [JLM21; ZLKB20; CJ19]), but have always required a restrictive “Bellman closedness” assumption.
As discussed, our lower bound construction is an extension of the recent remarkable lower bound of [WSG21], although several aspects of the construction have been modified to obtain better rates.
In particular, we modified the reward/value functions, the feature mappings, and introduced an expert policy which differs from the optimal policy.
For the proof, our setting is more complex as the learner has adaptive access to a second information source (the oracle), and a more sophisticated information-theoretic argument was needed to show that the oracle does not leak too much information.
Conclusion
We presented the D ELPHI algorithm for RL with an interactive expert.
We saw that, with ˜ O ( d ) oracle calls, exponential improvements in sample complexity are possible for RL with linearly-realizable optimal value functions.
Compared to prior works on learning with an interactive expert, we also saw that D ELPHI ’s oracle requirements were smaller, and in fact are independent of the horizon of the
MDP.
It would be interesting and fruitful to resolve the gap between the oracle complexity required by D ELPHI and the one obtained from our lower bound (either answer would be surprising to the authors).
It would also be fruitful to study the case of linearly-realizable action-value functions in stochastic MDPs, which would potentially enable our method to be extended to the online setting.
Acknowledgments and Disclosure of Funding
PA gratefully acknowledges funding from the Natural Sciences and Engineering Research Council of
Canada (NSERC).
Work done in part while PA was an intern at Amazon.
[-+
-]
[-The start state is s 
-]
[-V t = 0 under the set E w ⋆ n (the planner is never close to w ⋆ ), we have Y t = 
-]