A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximation
Abstract
The current paper studies sample-efficient Reinforcement Learning (RL) in settings where only the optimal value function is assumed to be linearly-realizable.
It has recently been understood that, even under this seemingly strong assumption and access to a generative model, worst-case sample complexities can be prohibitively (i.e., exponentially) large.
We investigate the setting where the learner additionally has access to interactive demonstrations from an expert policy, and we present a statistically and computationally efficient algorithm (D ELPHI ) for blending exploration with expert queries.
In particular, D ELPHI requires ˜ O
( d ) expert queries and a poly ( d, H, |A| , 1 /ε ) amount of exploratory samples to provably recover an ε -suboptimal policy.
Compared to pure RL approaches, this corresponds to an exponential improvement in sample complexity with surprisingly-little expert input.
Compared to prior imitation learning (IL) approaches, our required number of expert demonstrations is independent of H and logarithmic in 1 /ε , whereas all prior work required at least linear factors of both in addition to the same dependence on d .
Towards establishing the minimal amount of expert queries needed, we show that, in the same setting, any learner whose exploration budget is polynomially-bounded (in terms of d, H, and |A| ) will require at least ˜Ω( √ d ) oracle calls to recover a policy competing with the expert’s value function.
Under the weaker assumption that the expert’s policy is linear (rather than their value function), we show that the lower bound increases to Ω( d ) .
Introduction
Many potential applications of reinforcement learning (RL) have intractably-large state spaces.
Thus, we seek provably-correct methods which have statistical and computational requirements that are independent of the size of the state space.
This requires some modelling assumptions.
One dominating approach has been to introduce function approximation and to posit that the MDP or its value functions are well-represented by the function approximation scheme which is employed.
A basic starting point which still lacks comprehensive understanding is the case of linear value function approximation, which models value functions as lying in the span of a known d -dimensional feature mapping and asks for methods which have sample complexities that are polynomial only in d, H, and possibly |A| ( H and A are the horizon and action sets of the MDP, respectively).
This desideratum was recently understood to be impossible for the “minimal” case where only the optimal value function (or optimal action-value function) is assumed to be linear – i.e. there exist MDPs satisfying this assumption where the statistical complexity of any algorithm will be exponentially large, either in d or in H [WAS21; WSG21; WWK21; FKQR21].
Furthermore, this also holds in the case where the learner is equipped with a generative model (or simulator), allowing them to sample transitions from any state-action pair of the MDP.
In recent years much has been said about linear value approximation under stronger assumptions, for example under determinism
[WR13], linear/low-rank MDPs [JYWJ20; AJSWY20], Bellman-closedness [LSW20; ZLKB20], or the existence of a “core set” [SS20; ZLKB19].
These stronger assumptions can recover polynomial statistical complexities (thought not always computational ones), but are restrictive and oftentimes unrealistic.
In this work, we consider an alternative possibility for recovering polynomial complexities which does not further restrict the class of MDPs under consideration.
We do this by assisting the learner with some additional side information about the problem.
Specifically, we assume that there is a deterministic expert policy (which need not be the optimal policy) that the learner can query at any state, whereupon they will be informed of the expert’s action at that state.
Indeed, such information can often be made readily available if we have some form of prior knowledge (or human input) about the problem.
Leveraging such expert demonstrations has been studied in interactive imitation learning (IL), with common applications in simulated domains [RGB11; Ros13; RB14; SVGBB17].
As we will see, however, the amount of expert queries required by a pure IL approach is significantly higher than what we need.
Since interaction with a (human) expert might be costly, we wish to minimize the burden of the expert by having the learner explore mostly on their own, and only query the expert in a judicious manner.
The question asked by this work, then, is:
Under linearity of the optimal value function, what is the minimal amount of expert data required for sample/computationally-efficient learning?
Our main result is the D ELPHI algorithm for exploring with an interactive expert.
D ELPHI assumes that the expert’s value function is linear, and that the agent has access to a RESET function which lets them return to the state most recently seen .
Under these conditions, our method uses surprisingly-few expert queries combined with some modest (polynomial) amount of exploration to recover the expert policy.
Formally, D ELPHI recovers a policy that is ε − optimal (with respect to the expert policy) with O ( d log( B/ε )) oracle calls and ˜ O ( d 2 H 5 |A| B 4 ε) exploratory samples, where B is a bound on the ℓ 2 -norm of the unknown linear parameter.
Thus, our results show that merely ˜ O
( d ) expert queries enable an exponential improvement in sample complexity when compared to RL without expert advice.
Furthermore, the number of oracle calls is completely independent of the horizon of the problem and is logarithmic in 1 /ε , whereas prior work in IL leveraging expert advice requires (at best) linear factors of both in addition to scaling with d .
We also show that D ELPHI is computationally efficient, that it is robust to optimization and misspecification errors, and that it can be extended to the case where the action -value function of the expert is linear when the MDP dynamics are deterministic.
Towards establishing the optimality of our algorithm, we study the capabilities of expert-augmented learners which have fixed exploration budgets.
More specifically, we ask: what is the minimal number of expert queries required by any algorithm which is constrained to a polynomially-bounded exploration budget?
We show that any polynomially-bounded learner (in terms of d, H, and |A| ) will require at least ˜Ω( √ d ) oracle calls to recover a policy competing with the expert’s value function.
In the weaker setting where only the expert’s policy is linear, we show that this lower bound increases to Ω( d ) , matching our upper bound up to logarithmic factors.
The rest of the paper is structured as follows: in Section 2 we review background and present the problem setting.
Section 3 describes our algorithm, its guarantee, a sketch of the proof, and discusses some extensions.
Section 4 gives the lower bound on the number of expert queries needed.
We conclude with an overview of related work and some discussion in Sections 5 and 6.
Background & Problem Setting
Notation We write Dists( X ) for the set of probability measures on some set X .
We write [ N ] := { 1 , . . .
, N } .
The direct product ⊕ corresponds to “concatenating” two vectors, i.e. for any two vectors u ∈ R n and v ∈ R m , we have u ⊕ v =
( u 1 , . . .
, u n , v 1 , . .
. , v m )
⊤ ∈ R n + m .
We write ⊗ for the tensor (or outer) product of two vectors, defined by u ⊗ v = uv ⊤ ∈ R n × m , and ♭ ( u ⊗ v ) ∈ R n · m for the flattening (or vectorization) of said tensor product.
MDPs, Policies, and Value Functions
The typical environment in RL is modelled as an MDP [Put14; Sze10; SB18].
We consider here finite-horizon MDPs, which are specified by a tuple M = ( S , A , R , P , H, µ 0 ) , where S is a state space,
A
=
[ A ] is a finite action set, R : S×A → Dists([0 , 1]) is a (bounded) reward distribution function with expectation r ( s, a ) , P : S ×
A → Dists( S ) is the transition distribution function with probability vectors P ( s, a ) =
[ P ( s ′ | s, a )] s ′ ∈S ∈ R |S| , H ∈ N is the horizon, and µ 0 ∈ Dists( S ) is the starting distribution.
Note that we have assumed that the action space is finite, although the state space may be infinite.
Without loss of generality we assume that S is a disjoint union of per-horizon state spaces, i.e. S = ∪ h ∈
[ H ] S h .
A (non-stationary) policy π = ( π 1 , · · · , π H ) prescribes a sequence of actions π h : S h
→ Dists( A ) , and its value function is v π ( s ) =
E [ (cid:80) Hh ′ =
h r ( s h ′ , a h ′ ) | s h
= s, a h ′ ∼ π h ′
( s h ′ )] , where s ∈ S h .
The action-value function q π ( s, a ) is defined similarly, save that the first action taken is a and the proceeding actions follow π .
Value functions satisfy the recursive relationship:
[Equation]
(1) where we have the shorthands r ( s, π ) =
E a ∼ π ( s )
[ r ( s, a )] , P ( s, π ) =
E a ∼ π ( s ) [ P ( s, a )] , and the
Bellman operator T π ( · ) :
= r ( s, π ) + ⟨ P ( s, π ) , ( · ) ⟩ .
The Bellman operator has v π as its unique fixed point.
The optimal policy is written as π ⋆ , and its value function is denoted as v ⋆ := v π ⋆ .
The objective is to find a π maximizing v π ( µ 0 ) :
=
E s 0 ∼ µ
[ v π ( s 0 )] .
Function Approximation With an Interactive Expert In the RL setting, the MDP is unknown and must be explored.
As stated in the introduction, we seek sample complexities which are independent of the number of states.
This is evidently not possible without further assumptions.
To assist the learner, we assume that the agent has further access to an oracle , which, upon being queried, returns the action of an expert policy π ◦ for the state.
The expert policy need not be the optimal policy.
We will assume that π ◦ is deterministic, which is satisfied for example when π ◦ =
π ⋆ or when π ◦ is the greedy policy of some value function.
Assumption 2.1 (Interactive expert) .
There is an oracle which can be queried at the current state s , which returns the action π ◦ ( s ) .
Syntactically, the oracle is queried via the ORACLE ( s ) function.
We note that the oracle can only be queried at the state the learner is currently visiting.
The objective of the learner is to recover a policy which competes with the expert policy, namely a ˆ π such that:
[Equation]
where v ◦ :
= v π ◦ is the value function of the expert.
In the sequel we refer to a policy satisfying
Equation 2 as ε -optimal .
Our next assumption concerns the interplay of the expert policy with linear value approximation.
Namely, we assume that the expert policy has a value function which is linear in a set of known features.
Assumption 2.2 ( v ◦ -linearity, with bounded features) .
The value function v ◦ of the expert is linear with known features φ ∈ R d , i.e.
[Equation]
for some unknown θ ◦ ∈ R d .
We further assume that ∥ φ ( s ) ∥ 2 ≤ 1 ∀ s and that ∥ θ ◦ ∥ 2 ≤ B for some known B ∈ R d .
This assumption can be extended to the case of (slight) misspecification, as will be shown in
Subsection 3.2.
Our last assumption is that the agent has the ability to “reset” to the state most recently seen.
Assumption 2.3 (Resets) .
After experiencing a transition ( s, a, r, s ′ ) in the MDP, the agent can return to the state s .
Syntactically, this is done via the RESET () function.
Our RESET assumption is weaker than the full generative model access [KMN02] or the “local” simulator setting [WAJAYJS21; LCCGW21; HLYAYS22] which has appeared in prior works.
As noted in the introduction, existing exponential lower bounds entail that Assumptions 2.2 and 2.
together are not enough to enable sample-efficient learning.
Thus, any efficient learner which is provided with the above assumptions (2.1, 2.2, and 2.3) must necessarily make use of the expert.
The D ELPHI algorithm
We are ready to describe our approach and give the main result.
We begin by supposing that the starting distribution is deterministic (we will see later that this comes at no loss of generality).
Theorem 3.1.
Suppose Assumptions 2.1, 2.3, and 2.2 hold.
Then the D ELPHI algorithm will recover a policy ˆ π such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − ε with probability ≥ 1 − δ , using O ( d ln( B/ε )) oracle calls and ˜ O
( d 2 H 5 AB 4 ε) interactions with the MDP.
Furthermore this algorithm is computationally efficient.
The pseudo-code for D ELPHI is given in Algorithm 1, which uses Algorithm 2 ( measureTD ) as a sub-routine to measure expectations.
D ELPHI will solve for a parameter ˆ θ such that an induced policy π ˆ θ (defined via Eq.
(4)) will compete with the expert’s value function.
We note that deploying the policy π ˆ θ also requires the RESET functionality, since expectations must be estimated from a small number of samples at each state encountered.
This is a consequence of our assumption that v ◦ is linear (rather than, e.g., q ◦ or the MDP itself), since selecting actions based on state-value functions will always require one-step look-aheads.
Intuition for D ELPHI
Recall that the expert policy π ◦ satisfies v ◦ =
T π ◦ v ◦ , and that this fixed point is unique.
We say that a candidate value function is consistent at a state s if v ( s ) =
T π ◦ v ( s ) .
Note that we need consistency to hold globally (i.e. at all states) in order to ensure that v = v π ◦ .
Our methodology is based on ensuring that consistency on a small number of well-chosen states will guarantee global consistency.
D ELPHI is inspired by a recent algorithm of [WAJAYJS21] called TensorPlan.
As in TensorPlan,
D ELPHI proceeds via a “guess and check” procedure: at every iteration, we pick the optimistic linear parameter which is consistent on the past expert data that we have seen.
We then check whether this choice of parameter is globally consistent, by playing n rollout rollouts of length H with a policy derived from the parameter.
More specifically, for any θ the policy
π θ takes the form
[Equation]
where v θ ( · ) := ⟨ φ ( · ) , θ ⟩ , and ˆ r
( s, a ) and ˆ E s ′ ∼ P
( s,a ) are estimated via the RESET function.
After a certain number of rollouts, one of two things happen: either this policy encounters a state where there is no consistent action (i.e. the above minimum has a strictly positive value), or we only encounter states that are consistent.
In the first case, we query the oracle for its expert action and use the transition for that action to update the parameter set.
In the second case, we derive (cf.
Lemma 3.7) that if no inconsistencies are observed for several rollouts, then our “virtual value” v θ is close to the true value under π θ (i.e., v π θ ).
Using that θ was optimistic (i.e. v θ ( s 0 ) ≥ v θ ◦ ( s 0 ) ), this implies that we are optimal.
The only thing left to argue is that the number of iterations (i.e. the number of times that we can continue finding new parameters which are not globally consistent) is small.
Using linearity of v ◦ , it turns out roughly d inconsistencies are sufficient for this.
To see this, note that we can re-write the
Bellman equation for any v θ ( · )
= ⟨ φ ( · ) , θ ⟩ as:
[Equation]
where we have introduced the notation ∆ s,a :=
r ( s, a ) ⊕ ( E [ φ ( s ′ )] − φ ( s )) and used linearity of expectation, linearity of inner products, the definition of v θ , and the definition of the direct product.
We call the vector ∆ s,a the temporal difference (TD) vector for ( s, a ) .
Equation (5) is precisely an orthogonality constraint in d + 1 dimensions.
Since the parameter θ t which is chosen at time t was consistent on past data, it is orthogonal to the previous t − 1 TD vectors which have been generated from interactions with the oracle.
If we happen to find a state which has no consistent action, then the TD vector corresponding to the expert action at that state must not be in the span of the previous expert TD vectors (otherwise it would be consistent).
It follows that the iteration complexity is at most d + 1 , since there are at most d + 1 linearly independent vectors in R d +1 .
We use the Eluder dimension [RVR13] to generalize this argument to the case where the expectations are estimated.
The next section solidifies the above intuition and sketches the proof more formally.
2: Θ 0 ← do 5:
Pick θ
∈ θ ∈
Θ t − 1 (cid:0) v θ ( s 0 ) :
= θ ⊤ φ ( s 0 ) (cid:1)
▷ Optimistic choice over Θ t −6:
consistent ← true 7: for m = 1 to n rollout do ▷ n rollout number of rollouts with θ t -induced policy 8:
S t,m,h = s▷
Initialize rollout 9: for h to H do 10: for a ∈ [ A ] do ▷ each action 11: ˆ∆ S t,m,h ,a ← measureTD ( S t,m,h , a, n eval )
▷ Measure TD vector at ( s, a ) 12: end for 13: if min a (cid:12)(cid:12)(cid:12) ⟨ ˆ∆ S t,m,h ,a , 1 ⊕ θ t ⟩ (cid:12)(cid:12)(cid:12) > ε tol then ▷
No consistent action 14: consistent ← false 15: a ◦ t ← ORACLE ( S t,m,h )
▷ Query oracle for π ◦ ( S t,m,h ) 16:
˜∆ S t,m,h ,a ◦ t ← measureTD ( S t,m,h , a ◦ t , 4 E d n eval )
▷ Refined data 17: Θ t ← Θ t − 1 ∩ { θ | |⟨ ˜∆ S t,m,h ,a ◦ t , 1 ⊕ θ ⟩| ≤ ε tol } ▷ New admissible parameters 18:
Exit current iteration, t
Line
19: end if 20:
(cid:12)(cid:12)(cid:12) ⟨ ˆ∆ S t,m,h ,a , 1 ⊕ θ t ⟩ (cid:12)(cid:12)(cid:12) ▷ Else
consistent, keep playing 21:
Play A t,m,h , get R t,m,h , S t,m,h +1 ∼ MDP ▷ Roll forward 22: end for 23: end for 24: if consistent =
= true then 25: return θ t
▷ No inconsistency for m rollouts = ⇒ success 26: end if 27: end for 28: return θ E d +
Algorithm
1: Inputs: s, a, φ ( · ) , n, RESET () 2: for i = 1 to n do 3:
Play action a at s , receive sample R l and S
′ l from MDP 4: ∆ i ← R l ⊕ ( φ ( S ′ l ) − φ ( s )) 5:
RESET () 6: end for 7: return ˆ∆ s,a := 1 n (cid:80) i ∈ [ n ] ∆ i
3.1 Proof sketch
The full proof comes in 4 parts.
The proofs for all Lemmas and the precise values of all hyperparameters are given in Appendix A.
Lemmas 3.2 and 3.3 gives concentration bounds which establish that, with high probability, the measurements ˆ∆ (Line 11) and ˜∆ (Line 16) concentrate to the average ∆ s,a = E R ( s,a ) r ⊕ (cid:0)
E P ( s,a ) φ ( s ′ ) − φ ( s ) (cid:1) .
Lemma 3.4 establishes that, with high probability, the true optimal parameter θ ◦ is not eliminated from Θ t for any parameter set that is encountered.
It follows (Lemma 3.5) by optimism that v θ t ( s 0 ) ≥ v ◦ ( s 0 ) with high probability, where θ t is the parameter chosen at time t .
Lemma 3.6 establishes an iteration bound: the algorithm will terminate after at most t =
E d iterations of the outermost loop (and thus after at most E
Lastly, Lemma 3.7 establishes that if n rollout number of rollouts occur without observing a consistency break, then the virtual value ( v θ ( s 0 ) ) must be roughly equal to the true value under the executed policy ( v π θ ( s 0 ) ).
Theorem 3.1 combines all the ingredients to conclude the proof.
Part 1: Concentration bounds Recall that ∆ s,a :=
r ( s, a ) ⊕ ( E [ φ ( s ′ )] − φ ( s )) is the true TD vector, ˆ∆ is the estimated TD vectors obtained with n eval samples (in Line 11), and ˜∆ is the “refined data” obtained with 4 E d n eval samples (in Line 16).
The following lemmas establish concentration of ˆ∆ and ˜∆ to the true TD vector.
Lemma 3.2 (Concentration of ˆ∆ s,a (Line 11)) .
For any s, a ∈ S ×
A that is observed throughout the execution D ELPHI , with n eval samples in Line 11, we have that with probability ≥ 1 − δ , (cid:13)(cid:13)(cid:13) ˆ∆
s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤
ε eval and thus that ⟨ 1 ⊕ θ, ˆ∆ s,a − ∆ s,a ⟩ ≤ ¯ ε eval .
Lemma 3.3 ( ˜∆ s,a concentrates even more (Line 16)) .
Similarly, for all s, a where we call the oracle, with probability 1 − δ , we have (cid:13)(cid:13)(cid:13) ˜∆ s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤ ε eval / (2 √ E d ) , and thus, ∀ θ ∈ Ball ℓ 2 ( B ) , |⟨ 1 ⊕ θ, ˜∆ s,a − ∆ s,a ⟩| ≤ ¯ ε eval / (2 √ E d ) .
Part 2: Optimism This part shows that (with high probability) the true optimal parameter is not eliminated from the version space, and thus by optimism that the predicted value v t upper bounds v ◦ .
Lemma 3.4 ( θ ◦ not eliminated) .
With probability ≥ 1 − δ , θ ◦ ∈ Θ t for all iterations t ∈ [ E d + 1] .
Lemma 3.5 (Optimism) .
Under the event of Lemma 3.4, we have v t ( s 0 ) ≥ v ◦ ( s 0 ) , ∀ t ∈ [ E d ] .
Part 3: Iteration bound To bound the iteration complexity of our algorithm, we use the notion of
Eluder dimension [RVR13].
Loosely, the Eluder dimension with respect to some target function is the longest sequence of points ( x i ) such that there exists functions differing from the target function on x i but which correctly fit it on x 1 , . . .
, x i − 1 .
A formal definition is provided in Appendix A. We will use the result that the Eluder dimension of linear functions is O ( d ln( B/ε ))
[RVR13; LKFS21].
Lemma 3.6 (Iteration Complexity) .
With probability ≥ 1 − 2 δ , the iteration complexity of the algorithm is at most the Eluder dimension at scale ¯ ε eval , i.e. E d = O
( d ln( B/ ¯ ε eval )) .
Part 4: Consistency, and putting everything together The last thing to show is that if we are consistent at all states for several rollouts then the virtual value v θ will be close to the value of the v π θ .
Lemma 3.7 (Consistency = ⇒ accurate prediction) .
If m rollouts have occured without any inconsistencies (i.e., the if statement of Line 13 never gets triggered), then v π θ ( s 0 ) > v θ ( s 0 )
− 5 H ¯ ε eval − ε roll with probability ≥ 1 − 3 δ .
We are now ready to put everything together.
Proof (of Theorem 3.1).
Assume all events introduced so far (i.e. the events in Lemma 3.2, Lemma 3.3, and Lemma 3.7).
Together these happen with probability ≥ 1 − 3 δ , so we can re-define δ (cid:55)→ δ/such that the events happen together with probability ≥ 1 − δ (this only increases factors inside logarithms by 3 ).
By Lemma 3.7, we have:
[Equation]
where the second step follows from plugging in the definitions of n eval (Eq.
(12)), n rollout (Eq.
(10)), ¯ ε eval , (Eq.
and ε roll (Eq.
and the final step follows by optimism (Lemma 3.5).
The total sample complexity of our algorithm is: E d = ˜
O
( d ) oracle calls, and Nn eval =
( E d + 1)
Hn rollout An eval =
˜ O
( d 2 H 5 AB 4 ε 4 target ) exploration cost.
The last claim to verify is that of computational efficiency.
We note that the only computationally intensive step from iteration is
Line 5, i.e. the optimization problem corresponding to the optimistic choice over the parameter set max
[Equation]
θ ⊤ φ ( s 0 ) .
This is readily seen to be a convex program, since the objective is a linear function and the constraint set is a convex set in R d (it is initialized at Θ 0 =
Ball ℓ 2 ( B ) and every update intersects it with half-spaces (Line 17)).
Besides the ℓ 2 -norm constraint, all other constraints can be represented with a linear program, since each absolute value constraint can be split into two inequalities.
By Lemma 3.6 the number of constraints will be at most 2 E d =
˜ O
( d ) (and these can easily be converted to standard form).
Thus, there are a plethora of polynomial-time methods for solving this convex program [BBV04; Bub+15].
3.2 Extensions
In this section, we show that D ELPHI can be extended to work with optimization errors, under stochastic starting distributions, with misspecification, and with linear q ◦ whenever dynamics are deterministic.
Optimization errors and stochastic start state If the optimistic program can only be solved upto accuracy ε opt ≤ cε target (for c ∈ [0 , 1) ) at each iteration, then Lemma 3.5 becomes v t ( s 0 ) ≥ v ◦ ( s 0 ) − ε opt
and this only appears in Eq.
We then need 5 H ¯ ε eval + ε roll ≤ (1 − c ) ε target , which is achieved by increasing n rollout and n eval by a factor of 1 / (1 − c ) 2 . This increases the final exploration cost by 1 / (1 − c ) 4 .
For stochastic starting states, we simply work with v θ ( µ 0 ) =
E s 0 ∼ µ
[ v θ ( s 0 )]
(resp.
v ◦ ( µ 0 ) )
wherever v θ ( s 0 ) (resp.
v ◦ ( s 0 ) ) previously appeared.
The “starting feature” E s 0 ∼ µ
[ φ ( s 0 )] must be estimated from samples, which is then used for the optimistic program in Line 5 with φ ( s 0 ) replaced by this expectation.
The error is easily bounded as before by Hoeffding’s inequality, and will simply propagate additively through the proof.
Misspecified value functions and innacurate simulators
D ELPHI inherits some robustness properties from TensorPlan.
Namely, with a constant increase in exploration cost, D ELPHI continues to work under errors in the modelling assumptions.
The first case is where the expert value function is not linear but rather is approximately linear up to some uniform error.
Formally, we say that
the
MDP is η -misspecified for the expert policy π ◦ and the feature map φ if there exists θ ◦ such that sup s | v ◦ ( s ) − ⟨ φ ( s ) , θ ◦ ⟩| ≤ η .
The second case is where the simulator itself is flawed.
Formally, we say that the simulator is λ -innacurate if a transition ( r, s ′ ) from any state-action pair ( s, a ) of the MDP is instead observed as (Π( r + λ s,a ) , s ′ ) , where Π is the projection onto [0 , 1] and λ s,a is a constant uniformly bounded by λ .
The following result (proved in Appendix A.5) states that D ELPHI can tolerate misspecification or simulator inaccuracies of order roughly ¯ ε eval √ E d
= O ( 1 H √ d ) .
Theorem 3.8 (D ELPHI with misspecification) .
Redefine n ′ eval = 4 n eval (Eq. 12) and all subsequent hyperparameters which depend on n ′ eval .
Then we have that, for all MDPs that are at most ¯ ε eval 8 √ E dmisspecified or for all simulators that are at most ¯ ε eval 4 √ E d -innaccurate, the conclusions of Theorem 3.
continue to hold when running D ELPHI with the new hyperparameters.
q ◦ -linearity, in deterministic dynamics Rather than working with the Bellman equation v θ ( s ) =
T π ◦ v θ ( s ) we work with q θ ( s, a ) =
T π ◦ q θ
( s, a ) , which can be linearized similarly to Eq.
Namely:
[Equation]
This derivation holds generally, although to be able to speak of consistency at ( s, a ) with respect to a next action a ′ , we now assume deterministic dynamics, so that the above becomes
[Equation]
where s ′ is the unique successor of ( s, a ) and a ′ is the action that we are checking consistency for.
The algorithm proceeds as before, except rather than checking all actions at a given state (Line 13), we check all proceeding actions a ′ against the current ( s, a ) , and play the one with the smallest TD error (analogously to Line 20 of Algorithm 1).
How many oracle calls are necessary?
Is D ELPHI optimal in terms of its number of oracle calls?
To answer this question, we must argue that no algorithm can find an ε − optimal solution with less than Ω( d ) expert queries.
As stated, we are competing with agents which can (for example) exhaustively search the state-space and do not refer to the expert at all.
Thus, an exploration budget must be imposed (formally, a maximal amount of allowed interaction with the MDP, excluding oracle calls).
To make matters more interesting, we set this cap to be any polynomial amount, resulting in the following question:
Under Assumptions 2.1, 2.2, and 2.3, what is the minimal amount of expert queries needed for any algorithm with a poly ( d, H, A, 1 ε ) exploration budget to find an ε − optimal solution?
We firsly note that, measured in the worst-case, this minimal amount of expert queries is strictly positive since there exists MDPs satisfying v ⋆ -linearity for which no algorithm can return a sound solution with poly ( d, H, A, 1 ε ) queries [WAS21; WSG21].
Secondly, while it was possible to restrict ourselves simply to learners which have the same exploration requirements as D ELPHI , we opted to study algorithms with arbitrary polynomial exploration budgets, since it is a more fundamental question about the limits of exploration and the benefits of expert advice.
Note that any solution to this question must, a priori, have an exponential sample complexity for pure RL (otherwise the agent does not need to resort to the expert).
Interestingly, most constructions which exhibit exponential lower bounds for linearly-realizable RL can be solved with a single query from the oracle (e.g., [WAS21; WWK21]).
These constructions rely on having an exponentially large action set with a single correct action that effectively solves the MDP.
Our main lower bound comes from extending the recent lower bound of [WSG21], which is also the only known construction for an exponential lower bound which has a polynomial action set (rather than exponential).
Our result is that at least ˜Ω( √ d ) oracle calls are necessary:
Theorem 4.1.
There exists a family of MDPs satisfying 2.1, 2.2, and 2.3, such that any algorithm with poly ( d, H, A ) exploration budget will need at least ˜Ω( √ d ) oracle calls to recover a policy such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − 0 .
Our second lower bound considers an alternative assumption, which instead posits that the expert policy is linear.
Formally, this assumption says:
Assumption 4.2 ( π ◦ -linearity, with bounded features) .
The policy π ◦ of the expert is linear with known features φ :
[Equation]
d , i.e.
[Equation]
for some unknown θ ◦ ∈ R d \ { 0 } .
5 We further assume that ∥ φ ( s ) ∥ 2 ≤ 1 ∀ s and that ∥ θ ◦ ∥ 2 ≤ B for some known B ∈ R d .
When π ◦ = π ⋆ , it is easy to see that this assumption is implied by the assumption that q ⋆ is linearlyrealizable.
In general however, q ◦ -linearity does not imply that π ◦ is linear (see Appendix C for an example), although it does imply that the greedy policy derived from q ◦ is linear.
We give a lower bound for this case which matches the upper bound of D ELPHI up to logarithmic factors.
Theorem 4.3.
There exists a family of MDPs satisfying assumptions 2.1, 4.2, and 2.3 such that any algorithm with poly ( d, H, A ) exploration budget will need at least Ω( d ) oracle calls to recover a policy such that v ˆ π ( s 0 ) ≥ v ◦ ( s 0 ) − 0 .
Intuition for the lower bound Theorems 4.1 and 4.3 use the same MDP construction but with different features.
We give some intuition for the MDP construction which is used, but due to its intricacy a full description (and the information-theoretic proof) are deferred to Appendix B. Loosely, the learner has to find a hidden hypercube vector s ⋆ ∈ {± 1 } p .
The action space is A =
[ p ] , and each action corresponds to flipping one of the bits of the current vector.
The MDP has K “phases” which each correspond to p bit flips (thus H ≈ Kp ).
A linear reward is given only if a sufficiently small neighborhood of s ⋆ is reached, and the reward (thus the value) will decay geometrically in each subsequent phase that the neighborhood is not reached.
Intuitively, the oracle needs to be used
≈ p times, since each oracle calls only reveals one action, and thus one bit of the optimal vector s ⋆ .
The reason that this results in a ˜Ω( √ d ) lower bound (rather than ˜Ω( d ) ) is that the value function will experience a scale transition when going from states where s ⋆ is reachable given the remaining steps in the current phase to states where s ⋆ is no longer reachable.
As just described, the value (and thus the features) will be one order of magnitude smaller in this latter portion of the state space.
As this would betray the location of the secret parameter, the value function is instead augmented to be quadratic in p (roughly, the product of the distance achieved at the end of this phase and that of the next phase), which requires that p ≈ √ d in order to observe linearity.
On the other hand, the lower bound for π ◦ -linearity (Theorem 4.3) can remain linear in d , since the definition (Eq.
8) is scale-insensitive .
Closing the gap between our upper bound of ˜ O
( d ) and our lower bound of ˜Ω( √ d ) remains a challenging but interesting question.
In finite horizons, with access to a generative model or a RESET method, we suspect that the only mechanism for creating a hard MDP is by geometrically decaying the maximum possible value for each stage, such that at the final stage of the MDP the (random) reward becomes exponentially small in H (that is the approach taken here and in [WAS21; WSG21]).
If the geometric decaying happens in “phases” then this implies that Ω(1) of the value is located in the first “phase”.
The tension, then, is to have such a construction while hiding this large value, and forcing the learner to rely on several oracle calls to find it.
Prior constructions [WAS21; WWK21] have hid the large initial value by choosing an exponentially large action set, although as discussed above these examples are solved with a single query of the expert oracle.
Extending the “needle in a haystack” to occur over multiple decisions (cf.
the “phases” used above) leads to increased oracle requirements, although due to the scale transition phenomenon observed above it is far from clear how to do this with phases of length ≈ d .
The question thus boils down to: is geometric value reduction necessary for exponential lower bounds in this setting, and if so can we avoid the scale transition problem?
The situation is likely to be different in the online RL setting.
For example, the construction of [WWK21] extends [WAS21] to the online setting, and does not need to decay rewards but instead adds an Ω(1) probability of death at every transition.
This mechanism evidently does not work when the agent has resets, and since it is not known whether D ELPHI can be extended to the online setting we opted to keep the settings consistent between our upper bound and our lower bound.
Related works
The closest body of work to our setting is the field of interactive IL.
As in our setting, interactive IL considers the case where the learner has access to an expert oracle that can be queried adaptively.
It differs from our setting, however, since traditionally in IL the learner does not observe reward information.
We further differ from the IL setting since we consider value function approximation rather than general policy classes, and since we assume access to a RESET function.
Despite that many demonstrations of interactive IL occur in simulated domains [RGB11; Ros13; RB14], the benefits of this feature have not previously been studied.
Our assumption of v ◦ linearity entails that many IL methods are not directly applicable.
Indeed, the policy π ◦ itself does not need to be linear (despite that v ◦ is), so it is unclear which policy class to use for those algorithms.
Assuming for the sake of comparison that linear policies can be used, IL methods would still obtain worse oracle rates.
Indeed, using results from [AJKS19], Behaviour Cloning (for the passive case) or
AggreVaTe [RB14; SVGBB17] (for the interactive case) have worst-case errors O (
[Equation]
) for discounted MDPs, which roughly translates to an oracle complexity of N = O ( dH 4 /ε 2 ) when using the standard reduction H (cid:55)→ (1 − γ ) − 1 .
This is in sharp contrast to our ˜ O ( d ) oracle calls, which in independent of H and logarithmic in 1 /ε , and demonstrates the improvement due to exploration with the help of value-function approximation.
Beyond these approaches, another intuitive method would be to perform regression by doing a Monte Carlo estimation for the value of v ◦ ( s ) for each s along a certain “good” set of features.
This would require collecting rollouts from those states, which will again introduce a factor of H in the number of oracle queries.
Our algorithm instead finds a set of state-action pairs where the Temporal difference (TD) errors (which we represent as vectors) span orthogonal directions.
These can be estimated with a single transition, and this "local fitting" approach is novel to the IL literature and avoids the factors of H and 1 /ε from previous works.
In terms of other linear structure in IL, the works of [AN04; SS07] assume a known transition function and unknown linear rewards, and derive expert complexities of O (cid:16) dH 2 log( dH/ε ) ε(cid:17) and O ( H 2 log( d ) ε) respectively, but the algorithms involve (tabular) planning in MDPs and thus are not computationally efficient.
Most relevant is the recent work of [RHYLJR21], which, in the reward-free case, assumes that the expert policy is linear (i.e., our Assumption 4.2).
A sample complexity of ˜ O ( dH/ε ) is shown for Behaviour Cloning in this case, though no lower bound is given.
On the technical side, the D ELPHI algorithm is inspired by a recent algorithm called TensorPlan
[WAJAYJS21].
TensorPlan works for pure RL under Assumptions 2.2 and 2.3 but has a sample complexity scaling as poly ( (cid:0) dHε (cid:1) A , B ) and is computationally intractable.
Our extension of TensorPlan naturally incorporates the expert demonstrations, while simultaneously (1) having low oracle requirements, (2) addressing the exponential sample complexity of TensorPlan, and (3) rendering the algorithm computationally efficient.
Our approach is based only on finding value functions which satisfy the Bellman equation.
Bellman error minimization approaches have appeared in other works (e.g. [JLM21; ZLKB20; CJ19]), but have always required a restrictive “Bellman closedness” assumption.
As discussed, our lower bound construction is an extension of the recent remarkable lower bound of [WSG21], although several aspects of the construction have been modified to obtain better rates.
In particular, we modified the reward/value functions, the feature mappings, and introduced an expert policy which differs from the optimal policy.
For the proof, our setting is more complex as the learner has adaptive access to a second information source (the oracle), and a more sophisticated information-theoretic argument was needed to show that the oracle does not leak too much information.
Conclusion
We presented the D ELPHI algorithm for RL with an interactive expert.
We saw that, with ˜ O ( d ) oracle calls, exponential improvements in sample complexity are possible for RL with linearly-realizable optimal value functions.
Compared to prior works on learning with an interactive expert, we also saw that D ELPHI ’s oracle requirements were smaller, and in fact are independent of the horizon of the
MDP.
It would be interesting and fruitful to resolve the gap between the oracle complexity required by D ELPHI and the one obtained from our lower bound (either answer would be surprising to the authors).
It would also be fruitful to study the case of linearly-realizable action-value functions in stochastic MDPs, which would potentially enable our method to be extended to the online setting.
Acknowledgments and Disclosure of Funding
PA gratefully acknowledges funding from the Natural Sciences and Engineering Research Council of
Canada (NSERC).
Work done in part while PA was an intern at Amazon.
A Proof of Theorem 3.
Hyperparameters for the algorithm We define
[Equation]
As we will see, E d is the upper bound on the number of parameters observed before global consistency holds, n eval is the number of samples taken to estimate each TD vector, n rollout is the number of rollouts executed for each parameter, N is the maximum number of state-action pairs which will be seen by D ELPHI , ε eval is the error between the estimated TD vector and the true TD vector, and ε roll is the error of replacing the expected total rewards with the average over several rollouts.
A.1 Part 1: Concentration Inequalities
Recall that we write ∆
[Equation] [Equation]
for the true TD vector for state-action ( s, a ) .
We write ˆ∆ s,a for any estimated TD vector resulting from Line 11, and ˜∆ s,a for any doubly-estimated TD vector resulting from Line 16.
Lemma 3.2 (Concentration of ˆ∆ s,a (Line 11)) .
For any s, a ∈ S ×
A that is observed throughout the execution D ELPHI , with n eval samples in Line 11, we have that with probability ≥ 1 − δ , (cid:13)(cid:13)(cid:13) ˆ∆
s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤
ε eval and thus that ⟨ 1 ⊕ θ, ˆ∆ s,a − ∆ s,a ⟩ ≤ ¯ ε eval .
Proof.
Pick a single s, a , and omit the dependence on s, a for cleanliness.
Starting with the reward concentration: at every step we collect n eval iid samples from R ( s i , a i ) .
By Hoeffding, since rewards are bounded in [0 , 1] , the empirical average satisfies
[Equation]
Onto the transition probabilities.
Since the features are such that ∥ φ ∥ 2 ≤ 1 , we also have that ∥ φ ∥ ∞ ≤ 1 .
Thus, each coordinate of the feature map is bounded by 1 .
By Hoeffding, for each coordinate i of the vector, we have that
[Equation]
Thus, with a union bound over these events for each coordinate i and the event for the reward concentration, we have that w.p. ≥ 1 − δ/N
[Equation]
Finally, by a union bound over all N state-action pairs observed by D ELPHI , we have that for every s, a encountered:
[Equation]
We further find that: (cid:13)(cid:13)(cid:13) ˆ∆ − ∆ (cid:13)(cid:13)(cid:13) 2 ≤ √ d + 1 ε eval (since ∆ and ˆ∆ are ( d + 1) -dimensional).
By
Cauchy-Schwartz this gives, ∀ θ ∈ Ball ℓ 2 ( B ) :
[Equation]
Lemma 3.3 ( ˜∆ s,a concentrates even more (Line 16)) .
Similarly, for all s, a where we call the oracle, with probability 1 − δ , we have (cid:13)(cid:13)(cid:13) ˜∆ s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤ ε eval / (2 √ E d ) , and thus, ∀ θ ∈ Ball ℓ 2 ( B ) , |⟨ 1 ⊕ θ, ˜∆ s,a − ∆ s,a ⟩| ≤ ¯ ε eval / (2 √ E d ) .
Proof.
Follows from the same proof as Lemma 3.2, just replace n eval by 4 E d n eval .
A.2 Part 2: Optimism
Lemma 3.4 ( θ ◦ not eliminated) .
With probability ≥ 1 − δ , θ ◦ ∈ Θ t for all iterations t ∈ [ E d + 1] .
Proof.
Let ˜∆ t be the TD vector which is added to Θ t at time t ∈
[ E d ] .
Note that, by Lemma 3.3, for any given t , we have
[Equation]
for all t , w.p. ≥ 1 − δ .
The parameter set is defined as
[Equation]
Note that, for θ ◦ , this is equivalent to requiring
[Equation]
since ⟨ 1 ⊕ θ ◦ , ∆ t ⟩ = 0 for all ∆ t (recall Equation 5).
This holds by Eq.
Lemma 3.5 (Optimism) .
Under the event of Lemma 3.4, we have v t ( s 0 ) ≥ v ◦ ( s 0 ) , ∀ t ∈ [ E d ] .
Proof (of Lemma 3.5).
Under the event of Lemma 3.4, θ ◦ is never eliminated from Θ t .
Thus, by the optimistic update rule, we have θ t
+
=
argmax θ ∈ Θ t θ ⊤ φ ( s 0 ) , and since θ ◦ ∈ Θ t , then
[Equation]
A.3 Part 3: Iteration Complexity
To bound the iteration complexity of our algorithm, we will need to introduce the notion of Eluder dimension.
We use a simplified form introduced by [LKFS21], although the first definition is due to [RVR13].
Definition A.1 (Eluder dimension [LKFS21]) .
Let F be a real-valued function class on domain X .
Fix a reference function f ⋆ ∈ F , and a scale ε .
The Eluder dimension of F at a scale ε , w.r.t.
f ⋆ , is the length τ ∈ N of the longest sequence of points (( x 1 , f 1 ) , . . .
( x τ , f τ )) such that
[Equation]
An Eluder sequence of length τ (with respect to f ⋆ ) is any sequence ( x i , f i ) τi =1 which satisfies Eq (18) for each i .
In other words, the Eluder dimension is the length of the longest sequence of points such that, for each i , we can find a new function f i which is large with respect to f ⋆ on x i but correctly fits f ⋆ on historical data x 1 , . . .
, x j , for j < i .
We will use the folllowing bound for the Eluder dimension of linear functions with d -dimensional parameters.
Lemma A.2 ([RVR13]) .
For any f ⋆ , the Eluder dimension of the function class F = { f θ ( x ) =
θ
⊤ x } , assuming that ∥ θ ∥ 2 ≤ B and ∥ x ∥ 2 ≤ γ , is
[Equation]
This gives us enough to prove our iteration bound – we will show that the sequence of linear parameters chosen by our algorithm together with each new TD vector obtained from the oracle forms an Eluder sequence with respect to θ ⋆
Lemma 3.6 (Iteration Complexity) .
With probability ≥ 1 − 2 δ , the iteration complexity of the algorithm is at most the Eluder dimension at scale ¯ ε eval , i.e. E d = O
( d ln( B/ ¯ ε eval )) .
Proof (of Lemma 3.6).
Assume the events of Lemma 3.2 and Lemma 3.3, which happen together with probability ≥ 1 − 2 δ .
Our function class is F = { f θ ( x ) = ⟨ 1 ⊕ θ, x ⟩} .
This is a subset of all linear functions on d + 1 dimensions, so by Lemma A.2, 7 it has Eluder dimension at least
O (( d + 1) ln( B/ε ))
=
O ( d ln( B/ε )) .
We pick f ⋆ = f θ ◦ .
We will show that ∀
i , the sequence ( ˜∆ i , f θ i ) forms an Eluder sequence with respect to f ⋆ , from which it will follow that its length is bounded by E d .
We do this by induction.
The base case is obvious.
By the constraint set definition (Line 17), we have | f θ i ( ˜∆ j ) | = |⟨ 1 ⊕ θ i , ˜∆ j ⟩| ≤
[Equation]
ε eval
[Equation]
√ E d ∀ j < i .
Assuming the event of Lemma 3.4, we also have | f θ ◦ ( ˜∆ j ) | ≤ ¯ ε eval 2 √ E d , since θ ◦ ∈ Θ i − 1 .
Thus
[Equation]
where in the last inequality we have used i − 1 ≤ E d by the induction hypothesis.
Thus, the second condition of the Eluder dimension is satisfied.
For the first condition, we want to show that | f
θ i ( ˜∆ i ) − f θ ◦ ( ˜∆ i ) | > ¯ ε eval .
Note that, by Lemma 3.4, we have | f θ ◦ ( ˜∆ i ) | ≤ ¯ ε eval / (2 √ E d ) ≤ ¯ ε eval .
Recall from Line 13 that (cid:12)(cid:12)(cid:12) ⟨ ˆ∆ i , 1 ⊕ θ i ⟩ (cid:12)(cid:12)(cid:12) =
|
f
θ i ( ˆ∆ i ) | > ε tol .
Using concentration and linearity, we have | f
θ i ( ˆ∆ i ) | − | f
θ i ( ˜∆ i ) | ≤ 2¯ ε eval .
Thus, | f θ i ( ˜∆ i )
| ≥ ε tol − 2¯ ε eval =
2¯ ε eval since ε tol = 4¯ ε eval .
Putting this together gives
[Equation]
And we are done.
A.4 Part 4: Consistency, and Putting Everything
Together
Lemma 3.7 (Consistency = ⇒ accurate prediction) .
If m rollouts have occured without any inconsistencies (i.e., the if statement of Line 13 never gets triggered), then v π θ ( s 0 ) > v θ ( s 0 )
− 5 H ¯ ε eval − ε roll with probability ≥ 1 − 3 δ .
Proof (of Lemma 3.7).
Assume the event of Lemmas 3.2 and Lemmas 3.3, which together happen with probability ≥ 1 − 2 δ .
The third event which we assume will be introduced shortly.
For cleanliness, let us write θ for the final parameter which observes m rollouts without consistency break.
Observe the following calculation:
[Equation]
[Equation]
Now observe that after n rollout number of rollouts, we have n rollout unbiased estimates of the expected trajectories.
Thus we can approximate
[Equation]
where S ij and A ij are the states and actions in horizon j ∈
[ H ] of rollout i ∈
[ n rollout ] .
More precisely, using Hoeffding’s and that ⟨
[Equation]
(cid:17) ] , 1 ⊕ θ ⟩ ≤ 1 + 2 B , we can get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n rollout (cid:88)
[Equation]
w.p. ≥ 1 − δ E d +1 , where we wrote ∆ ij = E [ r S ij ,A ij ] ⊕ (cid:16) P S ij ,A ij φ ( · ) − φ ( S j ) (cid:17) .
By a union bound this happens for all t ∈ [ E d + 1] with probability ≥ 1 − δ .
Picking up where we left off: v π θ ( s 0 ) ≥ ⟨ φ ( s 0 ) , θ ⟩ +
[Equation]
(evaluation error)
[Equation]
(consistency holds) =
[Equation]
and we are done.
The second inequality follows from Lemma 3.2, which holds for all steps in all m trajectories in all E d + 1 iterations.
The third inequality holds since, for every S ij , there exists a consistent action, i.e. an action such that |⟨ ˆ∆ ij , 1 ⊕ θ ⟩| ≤ ε tol = 4¯ ε eval .
A.5 Simulator inaccuracy and misspecification
We start with the case of inaccurate simulators.
Recall that we say that a simulator is λ -innacurate if the samples obtained are of the form (Π( r + λ s,a ) , s ′ ) , for any ( s, a ) and for some constant λ s,a that satisfies | λ s,a | ≤ λ .
Theorem 3.8 (D ELPHI with misspecification) .
Redefine n ′ eval = 4 n eval (Eq. 12) and all subsequent hyperparameters which depend on n ′ eval .
Then we have that, for all MDPs that are at most ¯ ε eval 8 √ E dmisspecified or for all simulators that are at most ¯ ε eval 4 √ E d -innaccurate, the conclusions of Theorem 3.
continue to hold when running D ELPHI with the new hyperparameters.
We can repeat the proof of D ELPHI , and in fact the only difference will be in Part 1 of the proof (Lemmas 3.2 and 3.3) will continue to hold, which also implies that the rest of the proof will continue to hold.
Lemma A.3 (Concentration of ˆ∆ s,a (Line 11)) .
For any s, a ∈ S ×
A that is observed throughout the execution D ELPHI , with n eval samples in Line 11, we have that with probability ≥ 1 − δ , (cid:13)(cid:13)(cid:13) ˆ∆
s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤
ε eval and thus that ⟨ 1 ⊕ θ, ˆ∆ s,a − ∆ s,a ⟩ ≤ ¯ ε eval .
Lemma A.4 ( ˜∆ s,a concentrates even more (Line 16)) .
Similarly, for all s, a where we call the oracle, with probability 1 − δ , we have (cid:13)(cid:13)(cid:13) ˜∆ s,a − ∆ s,a (cid:13)(cid:13)(cid:13) ∞ ≤ ε eval / (2 √ E d ) , and thus, ∀ θ ∈ Ball ℓ 2 ( B ) , |⟨ 1 ⊕ θ, ˜∆ s,a − ∆ s,a ⟩| ≤ ¯ ε eval / (2 √ E d ) .
Proof (of Lemmas A.3 and A.4.
We repeat the proof of Lemmas 3.2 and 3.3.
Note that ˆ∆ s,a is an average of i.i.d.
random variables with mean ∆ ′ s,a = ( r ( s, a ) + λ s,a ) ⊕ P s,a φ − φ ( s ) .
As before, by
Hoeffdings, we have |⟨ 1 ⊕ θ, ˆ∆ − ∆ ′
⟩| ≤ ¯ ε eval / 2 , where the factor of 1 / 2 is due to the definition of n ′ eval .
This gives that
[Equation]
The proof of the second part follows similarly, since we have
[Equation]
Next we handle the value misspecification case.
Recall that the MDP is η -misspecified for the expert policy π ◦ and the features φ if there exists θ ◦ such that sup s | v ◦ ( s ) − ⟨ φ ( s ) , θ ◦ ⟩| ≤ η .
Here, we use a reduction argument to show that an η -misspecified MDP can be reduced to a 2 η -innacurate simulator of a realizable MDP.
Namely, using the same reduction as Appendix D of [WAJAYJS21], we can construct an alternative MDP M ′ such that M ′ is realizable but is a 2 η -inaccurate simulator of M .
The result then follows from the first part of the proof.
B Proof of SectionB.1 The MDP construction
Theorems 4.1 and 4.3 use the same MDP construction, which is inspired by the recent and remarkable lower bound of [WSG21] (itself an extension of the lower bound of [WAS21]).
We give an overview of the MDP and its features, and describe the changes from the original construction of [WSG21].
The state space consists of a hypercube in p dimensions, S = {± 1 } p , for some p ∈ N , and assume for simplicity that p is divisible by 4 .
More specifically, the state space at stage h also contains the history of vectors encountered so far ( s 1 , . . .
, s h ) , which is uniquely defined as the transitions are deterministic.
The dimension p will end up being p ≈ √ d for the value-based lower bound and p ≈ d for the policy-based lower bound.
We write ρ ( · , · ) for the Hamming distance on S , recalling that is a bilinear function of its arguments, i.e. ρ ( s 1 , s 2 ) =
( p − ⟨ s 1 , s 2 ⟩ ) .
The action set is A =
[ p ] , and each action a ∈ [ p ] will correspond to flipping the a th bit of the current state.
Each trajectory of horizon H has K phases ( K ∈ N ), and each phase consists of p steps (thus, H =
Kp ).
We write s 0 , s 1 , . . .
, s k , . . .
s K for the states reached at the end of each phase, and when we need to we will use s k,i for a state at step i of phase k .
The start state is s
=
⃗ 1 , the all-ones vector.
There is a special “goal state” s ⋆ , and reward is given only if a) at the end of any phase, ρ ( s k , s ⋆ ) ≤ p/ 4 , or b) the learner reaches horizon H , i.e. the end of phase K .
The reward function decays geometrically, and is defined according to the sequence of states s 0 , . . .
s
k obtained at the end of each phase.
Letting g ( s 1 , s 2 ) :
= 1 − ρ ( s 1 , s 2 )
/p
denote one minus the proportion of bits where two states differ, the reward function for reaching a p/ 4 -neighbourhood of s ⋆ at
stage k
(condition a) just described) is deterministic and has value
[Equation]
The reward function at stage H (condition b) above) is always given and has the same expectation as
Eq.
(19), but will be a Bernoulli random variable.
Modulo some exceptions (to be described shortly), the transition function is deterministic and is defined via P ( τ ( s, a ) | s, a ) = 1 , where τ ( s, a )
=
( s 1 , . . .
, − s a , . . .
s p )
corresponds to the new vector obtained from flipping the bit at index a ∈
[ p ] .
The exceptions to these transition dynamics are if 1) a state within a p/ 4 -neighbourhood of s ⋆ is reached, or 2) a move is repeated.
In the first case, the MDP transitions to a game-over state ⊥ (after which nothing else is possible and no reward is given).
In the second case, if the move is repeated within the first p/ 4 steps of a phase then the MDP similarly transitions to ⊥ , and if the move is repeated in the second 3 p/ 4 turns then the current state becomes frozen until the end of the current phase (i.e. no further bit flips are allowed).
This implies that each bit can only be flipped once in each phase, and further that g ( s i − 1 , s i ) ≤ 3 / 4 for each “legal” trajectory in the MDP (thus, the reward decays geometrically, cf.
Equation (19)).
So far, the main modification in our construction from that of [WSG21] is the reward function.
Their g function is chosen to be 2 nd order in ρ ( s 1 , s 2 ) , while ours is linea in ρ .
We now introduce the definition of the expert policy: we simply choose it to be the one which will flip the earliest index such that s k,i differs from s ⋆ and such that its index has not yet been played in the round.
If no such index exists, or if s k,i = s
⋆ , the expert policy will simply freeze the current round by repeating the earliest index that has already been played.
(Note that there is always a repeated index in this case, and that repeating an index will lead to freezing instead of termination, since at the start of any phase k
the trajectory must satisfy ρ ( s k , s ⋆ ) >
p/ 4 , otherwise the trajectory would have terminated).
It will turn out that the expert’s trajectory from the start state will be identical to that of the optimal policy (and, thus, will be equally difficult to compete with, in the sense of Eq.
While our expert policy might seem optimal, this need not be the case for arbitrary states since it might sometimes be more advantageous for a state in phase k which can no longer reach the p/ 4 -neighbourhood of s ⋆ to simply aim at minimizing the inevitable factor of g ( s k − 1 , s k ) that it will incur.
The following lemma (proved in Appendix B.2) gives an expression for the value function corresponding to our expert policy, and shows that the value function satisfies Assumption 2.2.
Lemma B.1.
Let s k,i be a state in round i of phase k .
Note that, from s k,i , the state s k +1 that π ◦ will reach at the end of the current phase is deterministic.
The value function of π ◦ is
[Equation]
and furthermore this is linear with features φ v of dimension d = Θ( p 2 ) .
The next lemma gives an expression showing that this same expert policy is linear in a different set of features φ π , and thus that the expert policy satisfies Assumption 4.2.
Lemma B.2.
There exists a feature map φ π and parameter θ π of dimension d = Θ( p ) such that
[Equation]
This lemma is also proved in Appendix B.2.
The essence of our lower bound is that each oracle call will reveal one bit of the secret state s ⋆ , and thus without ˜Ω( p ) calls and learner will need exponentially-many exploratory samples to solve this
MDP.
Thus, for the value-based lower bound, an agent with a d -dimensional feature map must be given an MDP which has a p =
Θ( √ d ) -dimensional state space.
However, for the policy-based lower bound, we can give the agent a p =
Θ( d ) -dimensional MDP.
Information-theoretic lower bound Information-theoretically, a learning algorithm can solve this
MDP only if they recover the secret state s ⋆ .
We let MDP ( p, K ) denote an instance of the above
MDP with dimension p and parameters K .
Following the approach of [WSG21], we prove the sample complexity hardness by reducing each MDP ( p, K ) to an instance of an abstract game called
C UBE G AME ( p, K ) .
Details on C UBE G AME , and the proof of the following theorem, are deferred to
Appendix B.3.
The main result is the following:
Theorem B.3.
Any learner which solves MDP ( p, K ) can be used to solve C UBE G AME ( p, K ) .
Unless the number of oracle calls is Ω( p log p ) , any learner which is 0 .
01 -optimal on C UBE G AME ( p, K ) , with a sample complexity of N , will need
N ≥ 2 Ω( p ∧ K ) .
Combined with the fact that a learner provided with d -dimensional features is given an MDP with parameter p ≈ √ d , this gives the result of Theorem 4.1.
B.2 Proofs of Lemma B.1, B.
We need some more notation in order to linearize the value function.
Let s k,i ̸ = ⊥ be a state in step i of phase k .
We define the variable ct flip k,i =
ρ ( s k, 0 , s k,i ) , which simply measures the number of components flipped so far in round k of step i , and fix k,i ∈ { 0 , 1 } p which is a vector with 1 at a component j if said component is currently frozen (i.e. if it has been played or if the entire state has been frozen), and 0 otherwise.
Similarly, there are two scalars e fix k,i and e ¬ fix k,i which simply count the number of components which disagree with s ⋆ that are currently frozen (for e fix k,i ) or not frozen (for e ¬ fix k,i ).
Note that only e fix k,i and e ¬ fix k,i depend on s ⋆ , and in fact we have: e fix k,
[Equation]
[Equation]
(20) e ¬ fix k,
[Equation]
=
[Equation]
( ⟨ ⃗
[Equation]
, ¬ fix k,
[Equation]
−
[Equation]
fix k,
[Equation]
k,
[Equation]
⋆
[Equation]
) , (21) where fix k,i · s k,i denotes component-wise multiplication, i.e. ( fix k,i · s k,i )
j
= ( fix k,i ) j
· ( s k,i ) j .
Now, we have that:
Lemma B.4 (Value of v ◦ ) .
Let s k,i be a state in round i of phase k .
Let s k +1 denote the state that π ◦ will reach at the end of the current phase when starting from s k,i (and note that this choice is deterministic given s k,i , and that it may not be in the p/ 4 -neighbourhood of s ⋆ ).
Then we have:
[Equation]
or, overloading notation and letting g ( x ) = 1 − x/p , we have
[Equation]
Proof.
Identical to [WSG21, Lemma 4.9]
Lemma B.5.
The value function v ◦ is linear in features φ v with dimension d = Θ( p 2 ) .
Proof.
Starting from Equation 22, we observe that only e fix k,i and e ¬ fix k,i depend on s ⋆ .
The first term in parentheses in simply a scalar which multiplies the features.
We thus calculate linear expressions for x = ct flip k,i + e ¬ fix k,i and y =
e fix k,i .
Starting with y =
e fix k,i , we have:
[Equation]
where a = 12 ⟨ ⃗ 1 , fix k,i ⟩ and b = − 12 fix k,i · s k,i .
Similarly:
[Equation]
where c = ct flip k,i + 12 ( ⟨ ⃗ 1 , ¬ fix k,i ⟩ ) and d = − 12 ¬ fix k,i · s k,i .
Thus we have: g ( y )
= 1 − y/p = 1 − ( a + ⟨ d, s ⋆ ⟩ ) /p
= (1 − a/p ) − ⟨ ¯ b, ¯ s ⋆ ⟩ = a ′ + ⟨ ¯ b, ¯ s ⋆ ⟩ , where a ′ =
(1 − a/p ) , ¯ b = − b/ √ p and ¯ s ⋆ = s ⋆ / √ p .
Similarly, g ( x ) = 1 − x/p = 1 − ( c + ⟨ d, s ⋆ ⟩ ) /p
=
c ′ − ⟨ ¯ d, ¯ s ⋆ ⟩ , where c ′ =
1 − c/p and ¯ d = d/ √ p .
Putting this together we have that
[Equation]
where we have use a property of the tensor product that ⟨ a 1 , b 1 ⟩⟨ a 2 , b 2 ⟩ = ⟨ a 1 ⊗ a 2 , b 1 ⊗ b 2 ⟩ , where a 1 ⊗ a 2 , b 1 ⊗ b 2 ∈ R p × p and their inner product is interpreted as the inner product between the vectorized matrices.
Thus, if we take
[Equation]
and φ
[Equation]
then we have v ◦ ( s k,i ) =
⟨ φ ( s k,i ) , θ v ⟩ as desired.
Thus v ◦ is linear with features in dimension 1 + p + p 2 .
Note that the norm of the features and the parameter θ v is also bounded by constants.
This completes the proof for v ◦ -linearity.
We next tackle the analogous statement for π ◦ -linearity.
Lemma B.6.
There exists a feature map φ π and parameter θ π of dimension d ≈ p such that
[Equation]
Proof.
Let s k,i be a state of interest.
Recall that π ◦ will either 1) flip the earliest index which has not been fixed such that the value of s k,i at that index disagrees with s ⋆ , or 2) if no such index exists, freeze the current round by playing a frozen index.
First consider a state s k,i such that π ◦ ( s k,i ) will flip an index.
Since the index flipped was previously incorrect and will thereafter agree with s ⋆ on that bit, this corresponds to minimizing the distance between s ′ and s ⋆ amongst all possible s ′ which can be reached in one step from s k,i (i.e. amongst all possible { τ ( s k,i , a ) } a , recalling that τ ( s k,i , a ) is our notation for the transition function of the MDP).
Thus, π ◦ ( s k,i ) ∈ argmin a { ρ ( τ ( s k,i , a ) , s ⋆ ) } .
This can be written linearly as argmin a { 12 ( p − ⟨ τ ( s k,i , a ) , s ⋆ ⟩ ) }
= argmax a { 12 ( ⟨ τ ( s k,i , a ) , s ⋆ ⟩ − p ) }
= argmax a { 12 ( ⟨ p ⊕ ⟨ τ ( s k,i , a ) , s ⋆ ⟩ , 1 ⊕ s ⋆ ⟩ ) } .
The second case is that π ◦ will freeze the round at the state s k,i .
This means that there are no indices which are incorrect that have not been frozen in this round.
Again, this corresponds to minimizing the distance between τ ( s k,i , a ) and s ⋆ : all other choices will either result in a game over (which has 0 value) or will flip an incorrect bit (which increases the distance).
Thus, again we have π ◦ ( s k,i ) ∈ argmin
a
ρ
( τ ( s k,i , a ) , s ⋆ ) } = argmax a { 12 ( ⟨ p ⊕ ⟨ τ ( s k,i , a ) , s ⋆ ⟩ , 1 ⊕ s ⋆ ⟩ ) } .
Thus, in either case, we have that π ◦ is linear with features φ π ( s, a ) =
p ⊕ τ (
Since the definition of the argmax is scale-insensitive, we can further normalize to obtain that the features are bounded in magnitude by a constant.
B. C UBE G AME with expert advice, and Proof of Theorem B.
Following the approach of [WSG21], we give our lower bound by providing a reduction to an abstract game called C UBE G AME .
Any learning algorithm which can solve the MDPs in our construction can also be used to solve C UBE G AME , and thus it follows that the learner will be subject to the same lower bound.
For our setting, we modify the reward function C UBE G AME and augment the learner with the ability to query an expert, which will behave identically to the expert policy which we have defined in our MDPs (that is, it will flip the first bit which is incorrect or give a special actions to indicate if all of the bits are correct).
For the rest of this section, when referring to C UBE G AME we are referring to our modified version.
In [WSG21] it is shown that any algorithm which outputs a 0 .
01 − optimal answer for the (expert-less)
C UBE G AME will need a query complexity of N ≥ 2 Ω( p ∧ K ) .
In what follows, we will provide the analogous proof of this for our modified game.
Thus, the main result of this section is the following, which states that if the learner is not given a budget of Ω( p/ log p ) expert queries, the sample complexity remains exponential.
A learning algorithm for C UBE G AME will be called a planner , and a planner which returns a 0 .
01 -optimal answer at the end of C UBE G AME will be called sound .
Rules of C UBE G AME
C UBE G AME is a bandit-like game which is defined by two parameters: a length K ∈ N + and a dimension p ∈ N + .
The “action space” is W = {± 1 } p .
Recall that ρ ( x, y ) =
12 ( p − x ⊤ y ) is the Hamming distance between two vectors in W .
The secret parameter which solves the game is housed in the set W ⋆ = { w ∈ W | p 4 ≤ ρ ( ⃗ 1 , w ) ≤ 3 p 4 } .
The planner can only input sequences of vectors where each vector is sufficiently far from the previous one.
Formally, for any k ∈ [ K ] , we let W ◦ k
= { ( w i ) i ∈
[ k ] ∈ W k | ∀ i ∈
[ k ] : ρ ( w i − 1 , w i ) ≥ p/ 4 } , with w 0 :
=
⃗ 1 .
The action space is: A = ∪ k ∈ [ K ] W ◦ k , thus the planner can input any sequence of length k ≤ K satisfying that ( w i )
∈
W ◦ k .
The reward function is defined by
[Equation]
where: g ( x ) = 1 − x p , with base case f
(()) =
g ( ρ ( w 0 , w ⋆ )) .
While the planner plays, it chooses sequence lengths L t ∈
[ K ] and input sequences S t = ( w ti ) i ∈
[ L t ] ∈
W ◦ L t .
If it chooses to stop playing, it chooses an output S t = ( w ti ) i ∈ [8] ∈ W ◦ 8 (we distinguish this case by letting L t = 0 denote that the planner has chosen to terminate).
Thus the number of actions taken is N = min { t ∈ N + | L t = 0 } .
After any action S t , the reward is given if either ρ ( w tL t , w ⋆ ) <
p/ 4 or if L t = K .
In both cases the reward is Ber ( f w ⋆ ( S t )) , a Bernoulli random variable with mean f w ⋆ ( · ) .
Similarly, if the planner is done (i.e. the input is S N ∈ W ◦ 8 ), then the reward given is R = f w ⋆ (( w Ni ) i ∈
[ k ⋆ ] ) , where k ⋆ = min { 8 , min { k | ρ ( w Nk , w ⋆ ) <
p/ 4 }} .
The last thing to specify is the oracle.
Here, we allow the planner to query the oracle part-way through a sequence.
Namely, if the planner chooses to input a sequence of length L t < K , then the oracle can be queried at the end of the sequence, and a second sequence of length L 2 t ≤ K − L t can be inputted.
This can be repeated as many times as desired, given that the total sequence length remains ≤ K .
The oracle will simulate the expert policy from before: upon being called at a vector w k,i , it will either return the index of the first bit which does not agree with w ⋆ , or it will return a special action indicating that all bits are correct.
We are now ready for the main theorem.
Theorem B.7.
Any sound planner for C UBE G AME which has less than Ω( p/ log p ) oracle queries, will have a sample complexity
[Equation]
The proof comes in 5 lemmas, two of which (Lemmas B.8 and B.10) are analogous to properties from the expert-less C UBE G AME .
The other 3 lemmas are information-theoretic and are specific to the oracle setting.
First, some properties about the reward function of C UBE G AME .
Lemma B.8 (Properties of f w ⋆ ) .
For any w ⋆ ∈ W ⋆ , k ∈ N , s = ( w k ′ )
k ′ ∈ [ k ] ∈ W ◦ k , we have
[Equation]
Proof.
The proof is analogous to [WSG21, Lemma 4.2], substituting our first-order g function.
The following parameters will control our sample complexity:
[Equation]
where
[Equation]
We will see that N = Ω( n ) for any sound planner.
In the original game of [WSG21], the interaction protocol is captured by ( X t , Y t ) t ∈ [ N ] , where
• N = min { t ∈ N + | L t = 0 } is the interaction length • L t ∈
[ K ] is the input length chosen, • S t is the sequence ( w i ) i ∈
[ L t ] inputted, satisfying ρ ( w i − 1 , w i ) ≥ p• X t =
( L t , S t ) , • U t = 1 { ρ ( w L t − 1 , w ⋆ ) <
p/ 4 } , • V t = 1 { ρ ( w L t , w ⋆ ) <
p/ 4 } , and • Z t = 0 unless V t = 1 or L t = K , in which case Z t =
Ber ( f w ⋆ ( S t )) , • Y t =
( U t , V t , Z t ) .
In our case, we need the sequence S t to include every bit flip, thus S t = ( w k,i )
k ∈
[ L t ] ,i ∈ [ p ] We also have two new variables, namely O t =
( o k,i ) which is an indicator that the oracle was called at step i of phrase k and E t
= ( e k,i ) which is the answer returned by the oracle.
Thus our new interaction protocol is defined by X t
=
( L t , S t , O t , E t ) and Y t =
( U t , V t , Z t ) , where Y t remains unchanged.
The planner A , with n interactions, in the environment defined by w ⋆ , defines a distribution over the environment:
[Equation]
Note that p ( x i | x 1: i − 1 , o 1: i − 1 , e 1: i
[Equation]
1 , y 1: i
[Equation]
1 ) and p ( o i | x 1: i , o
[Equation]
, e
[Equation]
i
[Equation]
1 , y 1:
[Equation]
1 ) are decisions made by the planner and p ( e i | x i , o i ) p ( y i | x i ) and p ( y i | x i ) are obtained by querying the environment.
We define the “abstract game (0 , w ⋆ ) ” to always yield reward 0, and which has the same oracle as environment w ⋆ .
It’s distribution will be written as P A (0 ,w ⋆ ) .
Let E w ⋆ n be the event that in the first n steps the planner does not hit on any vector that is close to w ⋆ :
[Equation] [Equation]
t > N or ( t = N and min i
[Equation]
] ρ ( w Ni , w ⋆ ) ≥ p 4 ) or ( t < N and ρ ( w L t − 1 , w ⋆ ) ≥ p 4 and ρ ( w L t , w ⋆ ) ≥ p 4 ) (cid:27)
Lemma B.9 (A first change of measure) .
For any planner A and any w ⋆ ∈ W , we have
[Equation]
Proof.
It will be shown that
[Equation]
Since 1 − ε ≥ 1 − 1 1+9 .
9 n by our definition of n we have that (1 − ε ) n ≥ (1 − 1 1+9 . 9 n ) n ≥ lim n →∞ (1 − 1 1+9 . 9 .n ) n > 9 / 10 , and thus Equation (27) implies our result.
Let H be the set of all possible histories ( x t , o t , e t , y t ) of length n , and E h =
E w ⋆ n ∩ { H = h } .
Note that E w ⋆ n is the disjoint union of E h , so it is enough to show that for each h we have:
[Equation]
for each h such that P A (0 ,w ⋆ ) ( E h ) > 0 .
So, let h =
( x t , o t , e t , y t ) be such that P A (0 ,w ⋆ ) ( E h ) > 0 .
This implies in particular that y t
= (0 , 0 , 0) ∀ t .
Now, both P Aw ⋆ and P A (0 ,w ⋆ ) factorize according to Eq.
(26), giving:
[Equation]
: i , o 1: i − 1 , e 1: i − 1 , y 1: i − 1 )
p w ⋆ ( e i | x i , o i ) p w ⋆ ( y i | x i )
p A ( x i | x 1: i − 1 , o 1: i − 1 , e 1: i − 1 , y 1: i − 1 )
p A ( o i | x 1: i , o 1: i − 1 , e 1: i − 1 , y 1: i − 1 ) p (0 ,w ⋆ ) ( e i | x i , o i ) p (0 ,w ⋆ )
( y i | x i ) .
Since we are conditioning on the same fixed history, the terms involving decisions made by the planner will cancel, and similarly the variable E i also behaves the same in both environments (since the oracle for w ⋆ is the same).
We are left with:
[Equation]
The denominator always has probability 1 in the environment (0 , w ⋆ ) , and since U t =
V t = 0 under the set E w ⋆ n (the planner is never close to w ⋆ ), we have Y t =
⇐⇒ Z t = 0 , so it remains to control
[Equation]
Again since the planner is never close to w ⋆ , Z t = 1 only if l t = K , in which case we have
P
Aw ⋆ ( Z t = 1 | x t ) ≥ (3 / 4) K +1 =
ε by definition of the reward f w ⋆ obtained from reaching level
K .
The next lemma simply bounds the number of vectors in W which are close to any fixed vector.
Lemma B.10 (Hypercube counting) .
For ˜ w ∈ W , let W close ( ˜ w )
= { w ∈ W | ρ ( w, ˜ w ) <
p/ 4 } .
Then | W close ( ˜ w ) | ≤ 2 p exp( − p/ 8)
Proof.
Identical to [WSG21, Lemma 4.4].
Recall that E w ⋆ n is the “bad event” for the planner.
We study its complement, ( E w ⋆ n ) c , which satisfies ( E w ⋆ n ) c ⊂ { w ⋆ ∈ Z } , where
[Equation]
i.e. the event that for some t we have ρ ( w tL t − 1 , w ⋆ ) <
p/ 4 or ρ ( w tL t , w ⋆ ) <
p/ 4 or that for some i ∈ [8] we have ρ ( w Ni , w ⋆ ) <
p/ 4 .
We define the “abstract game” (0 , 0) , where the planner has access to an oracle but, when queried, rather than giving information about the “true” w ⋆ , the oracle will simply return a uniformly random bit in [ p ] .
Lemma B.11 (A second change of measure) .
For any planner A with an oracle budget of x , we have that:
[Equation]
Proof.
As before, consider the set of histories H , and let Z h
= Z ∩ { H = h } .
Writing out the importance ratio gives:
[Equation]
Again, as before, all the terms involving the planner will cancel, since they are conditioned on the same history and thus the planner will make the same decisions.
Similarly, in both games the reward is deterministically 0 thus the p (0 ,w ⋆ )
( y i | x i ) =
p
[Equation] [Equation]
( y i | x i ) .
We are left with
[Equation]
Note that the top probability is deterministic (since the true expert is) and is only equal to 1 at most x times (recalling that x is the total number of oracle calls allowed).
We are simply left with the (inverse of the) probability that the random oracle returns any given answer, which is 1 /p .
Thus we end up with ρ = p x .
Lemma B.12 (Finding a bad w ⋆ for planner A ) .
For any abstract planner A there exists w ⋆ ∈ W ⋆ such that
[Equation]
Proof.
Note that, by Lemma B.9, it is sufficient to show that P A 0 ,w ⋆ (( E w ⋆ n ) c ) ≤ 110 .
Recall that for any ˆ w ∈ W ⋆ we have ( E ˆ w n ) c ⊆ { ˆ w ∈ Z } , where
[Equation]
By a union bound and Lemma B.10 we have that | Z | ≤ (2 n + 8)2 p exp( − p/ 8) .
Since W ⋆ =
W \ W close (1) \ W close ( − 1) , this also gives that | W ⋆ | ≥ 2 p (1 − 2 exp( − p/ 8)) .
We pick w ⋆ according to:
[Equation]
Putting things together and using Lemma B.11 gives:
[Equation]
Rearranging gives that
[Equation]
where the last line followed from our bound on n (Eq.
We are now ready to prove Theorem B.7.
In fact, there is not much left to do.
Proof (of Theorem B.7).
Let the planner A be sound and have an expected query cost ¯ N , and w ⋆ the vector from the previous lemma.
Then by Markov’s inequality we have:
[Equation]
Letting E ′ =
E w ⋆ n ∩ { N − 1 < n } we have
[Equation]
Under event E ′ , the output of the planner satisfies ρ ( w N i , w ⋆ ) ≥ p/ 4 for all i ∈ [8] , so the reward at the end of the game is R < (3 / 4) 9 .
Combined with soundness this gives
[Equation]
which requires ¯ N > 0 .
02 n , namely
[Equation]
Lastly, note that when x ≤ p 16log p we have
[Equation]
thus n = Ω(exp( p/ 16)) and in particular N = Ω { 2 p ∧ K } .
To prove Theorem B.3, the last thing to show is that a learner which solves the MDP can be used to solve C UBE G AME .
This reduction follows exactly as in Section 4.8 of [WSG21]
C On π ◦ linearity
This section shows that, when π ◦ ̸ = π ⋆ , q ◦ can be linear with d − dimensional features yet these features do not realize π ◦ -linearity.
The MDP is as follows: the states are arranged in a binary tree of length H .
The action space is { ℓ, r } , corresponding to the ℓ eft and r ight actions.
Transitions are deterministic.
The reward for every ℓ eft action is − 1 , the reward for every r ight action is +1 .
See Figure 1.
[Figure]
Note that we can identify every state with the action sequence that led to it (with the starting state corresponding to the empty sequence).
The policy π ◦ is defined such that, if s = ( a 0 , . .
. , ℓ ) then π ◦ ( s ) =
r and otherwise if s = ( a 0 , . .
, r ) then π ◦ ( s ) = ℓ .
Thus the policy will alternate the action taken at every step.
This defines the q ◦ function: q ◦ ( s, a ) =
  0 , if h = 0 mod− 1 , if h = 1 mod 2 and a = ℓ +1 , if h = 1 mod 2 and a = r
Note that we can linearize this in one dimension via the features φ ◦ ( s, a ) =
q ◦ ( s, a ) and θ = 1 .
However, these features do not linearly-realize π ◦ : since θ > 1 then the argmax at every odd horizon will always be the right action (since φ ◦ ( s, r ) = 1 and
φ ◦ ( s, ℓ ) = − 1 ).
Note that π ′ =
argmax a { q ◦ ( s, a ) } , the greedy policy derived from q ◦ is by definition linear with those features.
For the special case where π ◦ =
π
⋆ then the greedy policy π ′ lines up with the policy π ◦ , so we get linearity for free in that case.