Conﬁdent Approximate Policy Iteration for Efﬁcient Local Planning in @ c -realizable MDPs
Abstract
We consider approximate dynamic programming in W -discounted Markov decision processes and apply it to approximate planning with linear value-function approximation.
Our ﬁrst contribution is a new variant of A PPROXIMATE P OLICY I TERATION (API), called C ONFIDENT A PPROXIMATE P OLICY I TERATION (CAPI), which computes a deterministic stationary policy with an optimal error bound scaling linearly with the product of the effective horizon  and the worstcase approximation error Y of the action-value functions of stationary policies.
This improvement over API (whose error scales with  2 ) comes at the price of an  -fold increase in memory cost.
Unlike Scherrer and Lesner [2012], who recommended computing a non-stationary policy to achieve a similar improvement (with the same memory overhead), we are able to stick to stationary policies.
This allows for our second contribution, the application of CAPI to planning with local access to a simulator and 3 -dimensional linear function approximation.
As such, we design a planning algorithm that applies CAPI to obtain a sequence of policies with successively reﬁned accuracies on a dynamically evolving set of states.
The algorithm outputs an ˜ O( p 3Y ) -optimal policy after issuing ˜ O( 3 4 / Y 2 ) queries to the simulator, simultaneously achieving the optimal accuracy bound and the best known query complexity bound, while earlier algorithms in the literature achieve only one of them. This query complexity is shown to be tight in all parameters except  .
These improvements come at the expense of a mild (polynomial) increase in memory and computational costs of both the algorithm and its output policy.
Introduction
A key question in reinforcement learning is how to use value-function approximation to arrive at scaleable algorithms that can ﬁnd near-optimal policies in Markov decision processes (MDPs).
A ﬂurry of recent results aims at solving this problem efﬁciently with varying models of interaction with the MDP.
In this paper we focus on the problem of planning with a simulator when using linear function approximation.
A simulator is a “device” that, given a state-action pair as a query, returns a next state and reward generated from the transition kernel of the MDP that is simulated.
Depending on the application, such a simulator is often readily available (e.g., in chess, go, Atari).
Planning with simulator access comes with great beneﬁts: for example, in a recent work, Wang et al.
[2021] showed that under some conditions it is exponentially more efﬁcient to ﬁnd a near-optimal policy if a simulator of the MDP (that can reset to a state) is available compared to the online case where a learner interacts with its environment by following trajectories but without the help of a simulator.
Our setting of ofﬂine, local planning considers the problem of ﬁnding a policy with near-optimal value at a given initial state B 0 in the MDP.
The planner can issue queries to the simulator, and has to ﬁnd and output a near-optimal policy with high probability.
The efﬁciency of a planner is measured in four ways: the suboptimality of the policy found, that is, how far its value is from that of the optimal policy; the query cost , that is, the number of queries issued to the simulator; the computational cost , which is the number of operations used; and the memory cost , which is the amount of memory used (we adopt the real computation model for these costs).
There are several interaction models between the planner and the simulator
[Yin et al., 2022].
The most permissive one is called the generative model , or random access .
Here, the planner receives the set of all states and is allowed to issue queries for any state and action.
Coding a simulator that supports this model can be challenging, as oftentimes the set of states is computationally difﬁcult to describe.
Instead of random access , we consider the more practical and more challenging local access setting, where the planner only sees the initial state and the set of states received as a result to a query to the simulator.
Consequently, the queries issued have to be for a state that has already been encountered this way (and any available action), while the simulator needs to support the ability to reset the MDP state only to previously seen states.
A simple approach in practice to support this model is saving and reloading checkpoints during the operation of the simulator.
To handle large, possibly inﬁnite state spaces, we use linear function approximation to approximate the action-value functions @ c of stationary, deterministic policies c (for background on MDPs, see the next section). A feature-map is a good ﬁt to an MDP if the worst-case error of using the feature-map to approximate value functions of policies of the MDP is small:
Deﬁnition 1.1 ( @ c -realizability: uniform policy value-function approximation error) . Given an
MDP, the uniform policy value-function approximation error induced by a feature map i , which maps state-action pairs ( B , 0 ) to the Euclidean ball of radius ! centered at zero in R 3 , over a set of parameters belonging to the 3 -dimensional centered Euclidean ball of radius ⌫ is
[Equation]
where the outermost supremum is over all possible stationary deterministic memoryless policies (i.e., maps from states to actions) of the MDP.
Our goal is to design algorithms that scale gracefully with the uniform approximation error Y at the expense of controlled computational cost.
To achieve nontrivial guarantees, the uniform approximation error needs to be “small”.
This (implicit) assumption is stronger than the @ ¢ -realizability assumption (where the approximation error is only considered for optimal policies), which Weisz et al.
[2021] showed an exponential query complexity lower bound for.
At the same time, it is (strictly) weaker than the linear MDP assumption [Zanette et al., 2020], for which there are efﬁcient algorithms to ﬁnd a near-optimal policy in the online setting (without a simulator) [Jin et al., 2020], even in the more challenging reward-free setting where the rewards are only revealed after exploration [Wagenmaker et al., 2022].
In the local access setting, the planner learns the features i ( B , 0 ) of a state-action pair only for those states B that have already been encountered.
In contrast, in the random access setting, the whole feature map i (· , ·) , of (possibly inﬁnite) size 3 |S||A| (where S and A are the state and action sets, resp.), is given to the planner as input.
In the latter setting, when only the query cost is counted, Du et al.
[2019] and Lattimore et al.
[2020] proposed algorithms (the latter working in the misspeciﬁed, Y > 0 regime) that issue a number of queries that is polynomial in the relevant parameters, but require a barycentric spanner or near-optimal design of the input features.
In the worst case, computing any of these sets scales polynomially in |S| and |A| , which can be prohibitive.
In the case of local access , considered in this paper, the best known bound on the suboptimality of the computed policy is achieved by C ONFIDENT MC-P OLITEX [Yin et al., 2022].
In the more permissive random access setting, the best known query cost is achieved by Lattimore et al.
Our algorithm, CAPI-Q PI -P LAN (given in Algorithm 3), achieves the best of both while only assuming local access . This is shown in the next theorem; in the theorem Y is as deﬁned in Deﬁnition 1.1, W is the discount factor, and E ¢ and E c are the state value functions associated with the optimal policy and policy c , respectively (precise deﬁnitions of these quantities are given in the next section).
A comparison to other algorithms in the literature is given in Table 1; there the accuracy parameter l of the algorithms is set to Y , but a larger l can be used to trade off suboptimality guarantees for an improved query cost.
Theorem 1.2.
For any conﬁdence parameter X 2 ( 0 , 1 ] , accuracy parameter l > 0 , and initial state B 0 2 S , with probability at least 1  X , CAPI-Q PI -P LAN (Algorithm 3) ﬁnds a policy c with
[Equation]
while executing at most ˜ O  3 ( 1  W )  4 l  queries in the local access setting.
CAPI-Q PI -P LAN is based on C ONFIDENT MC-LSPI, another algorithm of Yin et al.
[2022], which relies on policy iteration from a core set of informative state-action pairs, but achieves inferior performance both in terms of suboptimality and query complexity.
However, CAPI-Q PI -P LAN ’s improvements come at the expense of increased memory and computational costs, as shown in the next theorem: compared to C ONFIDENT MC-LSPI, the memory and computational costs of our algorithm increase by a factor of the effective horizon  =
˜ O
( 1 /( 1  W )) , and the policy computed by CAPI-Q PI -P LAN uses a 3 factor more memory for storage and a 3 2  factor more computation to evaluate.
Theorem 1.3 (Memory and computational cost) .
The memory and computational cost of running CAPI-Q PI -P LAN (Algorithm 3) are ˜ O  3 2 /( 1  W )  and ˜ O  3 4 |A|( 1  W )  5 l  , respectively, while the memory and computational costs of storing and evaluating the ﬁnal policy outputted by CAPIQ PI -P LAN , respectively, are ˜ O  3 2 /( 1  W )  and ˜ O  3 3 |A|/( 1  W )  .
Next we present a lower bound corresponding to Theorem 1.2 that holds even in the more permissive random access setting, and shows that CAPI-Q PI -P LAN trades of the query cost and the suboptimality of the returned policy asymptotically optimally up to its dependence on 1 /( 1  W ) :
Theorem 1.4 (Query cost lower bound) . Let U 2 ( 0 , [Equation] , and Y  0 .
Then there is a class M of MDPs with uniform policy value-function approximation error at most Y such that any planner that guarantees to ﬁnd an U -optimal policy c (i.e., E ¢ ( B 0 )  E c ( B 0 )  U ) with probability at least X for all " 2 M when used with a simulator for " with random access , the worst-case (over M ) expected number of queries issued by the planner is at least [Equation]
If l is set to Y for CAPI-Q PI -P LAN , the ﬁrst term of Eq.
(2) implies that any planner with an asymptotically smaller (apart from logarithmic factors) suboptimality guarantee than Eq.
(1) executes exponentially many queries in expectation.
The second term of Eq.
(2), which is shown to be a lower bound in Theorem H.3 even in the more general setting of linear MDPs with zero misspeciﬁcation ( Y = 0 ), matches the query complexity of Theorem 1.2 up to an ˜ O
(( 1  W ) 2 ) factor.
Thus, the lower bound implies that the suboptimality and query cost bounds of Theorem 1.2 are tight up to logarithmic factors in all parameters except the 1 /( 1  W ) -dependence of the query cost bound.
At the heart of our method is a new algorithm, which we call C
ONFIDENT
A PPROXIMATE P OLICY I TERATION (CAPI). This algorithm, which belongs to the family of approximate dynamic programming algorithms [Bertsekas, 2012, Munos, 2003, 2005], is a novel variant of A PPROXIMATE P OLICY
I TERATION (API) [Bertsekas and Tsitsiklis, 1996]: in the policy improvement step, CAPI only updates the policy in states where it is conﬁdent that the update will improve the performance. This simple modiﬁcation allows CAPI to avoid the problem of “classical” approximate dynamic programming algorithms (approximate policy and value iteration) of inﬂating the value function evaluation error by a factor of  2 where  = ˜
O ( 1 /( 1  W )) (for discussions of this problem, see also the papers by Scherrer and Lesner, 2012 and Russo, 2020), and reduce this inﬂation factor to  . A similar result has already been achieved by Scherrer and Lesner [2012], who proposed to construct a non-stationary policy that strings together all policies obtained while running either approximate value or policy iteration.
However, applying this result to our planning problem is problematic, since the policies to be evaluated are non-stationary, and hence including them in the policy set we aim to approximate may drastically increase the error Y as compared to Deﬁnition 1.1, which only considers stationary memoryless policies. While the improvements provided by CAPI allows CAPI-Q PI -P LAN to match the performance of C ONFIDENT MC-P OLITEX in terms of suboptimality, it is unlikely that a simple modiﬁcation of C ONFIDENT MC-P OLITEX would lead to an algorithm which matches CAPI-Q PI -P LAN ’s [Table] [This work]
[Table] performance in terms of query cost (see Table 1): Both methods evaluate a sequence of policies at an ˜ O( Y ) accuracy each (requiring ˜ O ( 1 / Y 2 ) queries, omitting the dependence on other parameters). However, while CAPI-Q PI -P LAN (and C ONFIDENT MC-LSPI) evaluates
O( log ( 1 / Y ))
(again in terms of Y only) policies to ﬁnd one which is ˜ O( Y ) -optimal, C ONFIDENT MC-P OLITEX needs to compute ˜ O ( 1 / Y 2 ) policies to achieve the same. As a consequence, C ONFIDENT MC-P OLITEX only achieves ˜ O
( 1 / Y 4 ) query complexity [Table]
and to match CAPI-Q PI -P LAN ’s ˜ O ( 1 / Y 2 ) complexity, one would need to come up with either signiﬁcantly better policy evaluation methods (potentially using the similarity in the subsequent policies) or a much faster (exponential vs. square-root) convergence rate in the suboptimality of the policy sequence produced by C ONFIDENT MC-P OLITEX .
The rest of the paper is organized as follows: The model and notation are introduced in Section 2.
CAPI is introduced and analyzed in Section 3.
Planning with @ c
-realizability is introduced in Section 4, with CAPI-Q PI -P LAN being built-up and analyzed in Sections 4.1 and 4.2.
In particular, the proof of Theorem 1.2 is given in Section 4.2.
Several proofs are relegated to appendices, in particular, Theorem 1.3 is proved and implementation details of CAPI-Q PI -P LAN are discussed in Appendix G, while Theorem 1.4 is proved in Appendix H.
Notation and preliminaries
Let N = { 0 , 1 , . . .
} denote the set of natural numbers, N + = { 1 , 2 , . . .
the positive integers.
For some integer 8 , let [ 8 ] = { 0 , . .
, 8  1 } .
For G 2 R , let d G e denote the smallest integer i such that 8  G .
For a positive deﬁnite + 2 R 3 ⇥ 3 and G 2 R 3 , let k G k [-
-]2 + =
G > +G .
For matrices  and ⌫ , we say that  ⌫ ⌫ if   ⌫ is positive semideﬁnite.
Let I be the 3 -dimensional identity matrix.
For compatible vectors
G , H , let h G , H i be their inner product: h G , H i =
G > H .
Let M 1 ( - ) denote the space of probability distributions supported on the set - (throughout, we assume that the f -algebra is implicit).
We write 0 ⇡ Y 1 for 0 , 1 , Y 2 R if | 0  1 |  Y .
We denote by ˜ O(·) and ˜ ⇥ (·) the variants of the big-O notation that hide polylogarithmic factors.
A Markov Decision Process (MDP) is a tuple " =
(S , A , Q) , where S is a measurable state space,
A is a ﬁnite action space, and Q : S ⇥ A !
M 1 (S ⇥ [ 0 , 1 ]) is the transition-reward kernel.
We deﬁne the transition and reward distributions % :
S ⇥ A ! M 1 (S) and R : S ⇥ A !
M 1 ([ 0 , 1 ]) as the marginals of
Q .
By a slight abuse of notation, for any B 2 S and 0 2 A , let % (·| B , 0 ) and
R(·| B , 0 ) denote the distributions
% ( B , 0 ) and R( B , 0 ) , respectively.
We further denote by A ( B , 0 )
=
Ø
G d R( G | B , 0 )
the expected reward for an action 0 2 A taken in a state B 2 S .
Without loss of generality, we assume that there is a designated initial state B 0 2 S .
Starting from any state B 2 S , a stationary memoryless policy c : S !
M 1 (A) interacts with the MDP in a sequential manner for time-steps C 2 N , deﬁning a probability distribution P c , B over the episode trajectory { ( 8 ,  8 , ' 8 } 8 2 N as follows: ( 0 = B deterministically,  8 ⇠ c ( ( 8 ) , and ( ( 8 + 1 , ' 8 ) ⇠ Q( ( 8 ,  8 ) .
By a slight variation, let P c , B , 0 denote (for some 0 2 A ) the distribution of the trajectory when  0 = 0 deterministically, while the distribution of the rest of the trajectory is deﬁned analogously.
This allows us to conveniently deﬁne the expected state-value and action-value functions in the discounted setting we consider, for some discount factor 0 < W < 1 , respectively, as
E c ( B ) = E c , B " ’ C 2 N
W C ' C # and @ c ( B , 0 ) =
E c , B , 0 " ’ C 2 N
W C ' C # for all ( B , 0 ) 2 S ⇥ A , (3) where throughout the paper we use the convention that E • is the expectation operator corresponding to a distribution P • (e.g., E c , B is the expectation with respect to P c , B ).
It is well known (see, e.g., Puterman, 1994) that there exists an optimal stationary deterministic memoryless policy c ¢ such that sup c E c ( B ) =
E c ¢ ( B ) and sup c @ c ( B , 0 ) = @
c ¢ ( B , 0 ) for all ( B , 0 ) 2 S ⇥ A .
Let E ¢ =
E c ¢ and @ ¢
= @ c ¢ .
For any policy c , E c and @ c are known to satisfy the Bellman equations [Puterman, 1994]:
[Equation]
Finally, we call a policy c deterministic if for all states, c ( B ) is a distribution that assigns unit weight to one action and zero weight to the others.
With a slight abuse of notation, for a deterministic policy c , we denote by c ( B ) the action c chooses (deterministically) in state B 2 S .[-We assume that arg max 0 
 A 147 breaks ties in a consistent manner by ordering the actions (using the notation A =-]
Conﬁdent Approximate Policy Iteration
In this section we introduce C ONFIDENT A PPROXIMATE P OLICY I TERATION (CAPI), our new approximate dynamic programming algorithm.
In approximate dynamic programming, the methods are designed around oracles that return either an approximation to the application of the Bellman optimality operator to a value function (“approximate value iteration”), or an approximation to the value function of some policy (“approximate policy iteration”).
Our setting is the second.
The novelty is that we assume access to the accuracy of the approximation and use this knowledge to modify the policy update, which leads to improved guarantees on the suboptimality of the computed policy.
We present the pseudocodes of API [Bertsekas and Tsitsiklis, 1996] and CAPI jointly in Algorithm 1: starting from an arbitrary (deterministic) policy c 0 , the algorithm iterates a policy estimation (Line 2) and a policy update step (Line 3)  times.
The policy update for API is greedy with respect to the action-value estimates ˆ @ and is deﬁned as c ˆ @ ( B )
= arg max 0
A ˆ @ ( B , 0 ) .
We assume that arg max 0 2 A breaks ties in a consistent manner by ordering the actions (using the notation
A =
(A 1 , . . . , A |A| ) ) and always choosing action A 8 with the lowest index 8 that achieves the maximum. For CAPI, the policy update further relies on a global estimation-accuracy parameter l , and a set of ﬁxed-states
S ﬁx ✓ S . For the purposes of this section, it is enough to keep S ﬁx = {} .
CAPI updates the policy to one that acts greedily with respect to ˆ @ only on states that are not in
S ﬁx and where it is conﬁdent that this leads to an improvement over the previous policy (Case 5a); otherwise, the new policy will return the same action as the previous one (Case 5b).
To decide, ˆ @ ( B , c ( B ))
+ l is treated as the upper bound on the previous policy’s value, and max 0
A ˆ @ ( B , 0 )
l as the lower bound of the action-value of the greedy action (Eq. 5):
c ˆ @ , c , S
[Equation]
=
[Equation] if [Equation]
S [Equation] ˆ @
[Equation]
B
[Equation]
(5a) c (
[Equation]
otherwise.
(5b)
Note that c ˆ @ , c , S ﬁx also depends on l , however, this dependence is omitted from the notation (as l is kept ﬁxed throughout).
CAPI can also be seen as a reﬁnement of C ONSERVATIVE P OLICY I TERATION (CPI) of Kakade and Langford [2002] with some important differences: While CPI introduces a global parameter to ensure the update stays close to the previous policy, CAPI has no such parameter, and it dynamically decides when to stay close to (more precisely, use) the previous policy, individually for every state, based on whether there is evidence for a guaranteed improvement.
Let c be any stationary deterministic memoryless policy, ˆ @ c : S ⇥ A !
R be any function, l 2 R + , and
S ﬁx ✓ S .
First, we show that as long as ˆ @ c is an l -accurate estimate of @ c , the CAPI policy update only improves the policy’s values:
Lemma 3.1 (No deterioration) .
Let c
=
c ˆ @ c , c , S ﬁx .
Assume that for all B 2 S \ S ﬁx and 0 2 A , ˆ @ c ( B , 0 ) ⇡
l @ c ( B , 0 ) .
Then, for any B 2 S , E c 0 ( B )  E c ( B ) .
Proof.
Fix any B 2 S .
If B 2 S ﬁx or ˆ @ c ( B , c ( B ))
+ l  max 0
A ˆ @ c ( B , 0 )  l , then c 0 ( B ) =
c ( B ) and therefore @
c ( B , c 0 ( B )) =
E c ( B ) .
Otherwise, B 8 S ﬁx and ˆ @ c ( B , c ( B ))
+ l  max 0
A ˆ @ c ( B , 0 )  l , hence c 0 ( B ) = {+arg max 0 
+}
A ˆ @ c ( B , 0 [-
-]) , and it follows by our assumptions that @ c ( B , c 0 ( B ))  1: for 8 = 1 to  do 2: ˆ @ E STIMATE ( c 8  1 ) 3: c⇢ c ˆ @
API c ˆ @ , c 8  1 , S ﬁx CAPI 4: return c  ˆ @ c ( B , c 0 ( B ))
l = max 0
A ˆ @ c ( B , 0 )  l > ˆ @ c ( B , c ( B )) + l  @ c ( B , c ( B ))
= E c ( B ) .
Therefore, in any case, @ c ( B , c 0 ( B ))
E c ( B ) .
Since this holds for any B 2 S , the Policy Improvement Theorem [Sutton and Barto, 2018, Section 4.2] implies that for any B
[Equation]
Next we introduce two approximate optimality criterion for a policy on a set of states:
Deﬁnition 3.2 (Policy optimality on a set of states) .
A policy c is  -optimal (for some   0 ) on a set of states S 0 ✓ S , if for all B 2 S 0 , E ¢ ( B )  E c ( B )   .
Deﬁnition 3.3 (Next-state optimality on a set of states) .
A policy c is next-state  -optimal (for some   0 ) on a set of states S 0 ✓ S , if for all B 2 S 0 and all actions 0 2 A , Ø
[Equation]
Note that in the special case of S
= S the ﬁrst property implies the second, that is, if c is  -optimal on S , then it is also next-state  -optimal on S .
Next, we show that the suboptimality of a policy updated by CAPI evolves as follows (the proof is relegated to Appendix A):
Lemma 3.4 (Iteration progress) .
Let c
=
c ˆ @ c , c , S ﬁx .
Assume that for all B 2 S \ S ﬁx and 0 2 A , ˆ @ c ( B , 0 ) ⇡
l @ c ( B , 0 ) , and that c is next-state  -optimal on S \ S ﬁx .
Then c 0 is ( 4 l + W  ) -optimal on
S \ S ﬁx .
3.1 CAPI guarantee with accurate estimation everywhere
To obtain a ﬁnal suboptimality guarantee for CAPI, ﬁrst consider the ideal scenario in which we assume that we have a mechanism to estimate @
c ( B , 0 ) up to some l accuracy for all B 2 S and2 A , and for any policy c :
Assumption 3.5.
There is an oracle called E STIMATE that accepts a policy c and returns ˆ @ c : S ⇥ A !
R such that for all B 2 S and 0 2 A , ˆ @ c ( B , 0 ) ⇡
l @ c ( B , 0 ) .
Theorem 3.6 (CAPI performance) .
Assume CAPI (Algorithm 1) is run with S ﬁx = {} , iteration count to  = d log l / log W e , and suppose that the estimation used in Line 2 satisﬁes Assumption 3.5.
Then the policy c  returned by the algorithm is 5 l /( 1  W ) -optimal on S .
Proof.
We prove by induction that policy c 8 is  8 -optimal on S for  8 = 4 l Õ 9 2 [ 8 ] W 9 + W 8 1  W .
This holds immediately for the base case of 8 = 0 , as rewards are bounded in [ 0 , 1 ] and thus E ¢ ( B ) /( 1  W ) for any B .
Assuming now that the inductive hypothesis holds for 8  1 we observe that c 8  1 is next-state  -optimal on
S =
S \ S ﬁx .
Together with Assumption 3.5, this implies that the conditions of Lemma 3.4 are satisﬁed for c = c 1 , which yields E ¢ ( B )
E c 8 ( B )  4 l + W  8  1 =  8 , ﬁnishing the induction. Finally, by the deﬁnition of  , c  is
[Table]
optimal with    4 l 1  W + W  1  W  5 l 1  W .
⇤
Local access planning with @ c -realizability
Our planner, CAPI-Q PI -P LAN , is based on the C ONFIDENT MC-LSPI algorithm of Yin et al.
This latter algorithm gradually builds a core set of state-action pairs whose corresponding features are informative.
The @ -values of the state-action pairs in the core set are estimated using rollouts.
The procedure is restarted with an extended core set whenever the algorithm encounters a new informative feature.
If such a new feature is not encountered, the estimation error can be controlled, and the estimation is extended to all state-action pairs using the least-squares estimator.
Finally, the extended estimation is used in Line 2 of API.
CAPI-Q PI -P LAN improves upon C ONFIDENT MC-LSPI in two ways.
First, using CAPI instead of API improves the ﬁnal suboptimality bound by a factor of the effective horizon. Second, we Algorithm
1: Input: state B , action 0 , deterministic policy c , set of states S 0 ✓ S , accuracy l > 0 , failure probability Z 2 ( 0 , 1 ] 2: Initialize:  d log (( l / 4 )( 1  W ))/
log W e , =
⌃ ( l / 4 )  2 ( 1  W )  2 log ( 2 / Z )/
⌥ 3: for 8 = 1 to = do 4: ( ( , ' 8 , 0 ) S IMULATOR ( B , 0 ) 5: for ⌘ = 1 to   1 do 6: if ( 8 S 0 then return ( discover , ( ) 7:  c ( ( ) 8: ( ( , ' 8 , ⌘ ) S IMULATOR ( ( ,  ) ù Call to the simulator oracle 9: return ( success , 1 = Õ =8 = Õ   1 ⌘ = 0 W ⌘ ' 8 , ⌘ ) apply a novel analysis on a more modular variant of the C ONFIDENT R OLLOUT subroutine used in
C ONFIDENT MC-LSPI, which delivers @ -estimation accuracy guarantees with respect to a large class of policies simultaneously.
This allows for a dynamically evolving version of policy iteration, that does not have to restart whenever a new informative feature is encountered.
Intuitively, this prevents duplication of work.
4.1 Estimation oracle
To obtain an algorithm for planning with local access whose performance degrades gracefully with the uniform approximation error, we must weaken Assumption 3.5.
This is because under local access, we cannot guarantee to cover all states or hope to obtain accurate @ -value estimates for all states.
Instead, we are interested in an accuracy guarantee that holds for @ -values only on some subset S 0 ✓ S of states, but holds simultaneously for any policy that agrees with c on S
but may take arbitrary values elsewhere.
For this, we deﬁne the extended set of policies:
Deﬁnition 4.1.
Let ⇧ det be the set of all stationary deterministic memoryless policies, c 2 ⇧ det , and
S 0 ✓ S .
For ( c , S 0 ) , we deﬁne ⇧ c , S 0 to be the set of policies that agree with c on
[Equation]
We aim to ﬁrst accurately estimate @
c ( B , 0 ) for some speciﬁc ( B , 0 ) pairs, based on which we extend the estimates to other state-action pairs using least-squares.
To this end, we ﬁrst devise a subroutine called M EASURE (Algorithm 2).
M EASURE is a modularized variant of the C ONFIDENT R OLLOUT subroutine of Yin et al.
The modularity of our variant is due to the parameter S 0 that corresponds to the set of states on which the planner is “conﬁdent” for C ONFIDENT R OLLOUT .
M EASURE unrolls the policy c starting from ( B , 0 ) for a number of episodes, each lasting  steps, and returns with the average measured reward.
Throughout, we let  = d log (( l / 4 )( 1  W ))/ log W e be the effective horizon.
At the end of this process, M EASURE returns status success along with the empirical average @
-value, where compared to Eq.
(3), the discounted summation of rewards is truncated to  .
If, however, the algorithm encounters a state not in its input S 0 , it returns with status discover , along with that state.
This is because in such cases, the algorithm could no longer guarantee an accurate estimation with respect to any member of the extended set of policies.
The next lemma, proved in Appendix B, shows that M EASURE provides accurate estimates of the action-value functions for members of the extended policy set.
Lemma 4.2.
For any input parameters B 2 S , 0 2 A , c 2 ⇧ det , S 0 ⇢ S , l > 0 , Z 2 ( 0 , 1 ) , M EASURE either returns with ( discover[-, B 0 ) for some B 0 8 S 
-]
[Equation]
B 0 ) for some B 0 8 S 0 (Line 6), or it returns with ( success , ˜ @ ) such that with probability at least
[Equation]
Suppose we have a list of state-action pairs ⇠ =
( B 8 , 0 8 ) 8 2 [| ⇠ |] and corresponding @ -estimates ¯ @
= ( ¯ @ 8 ) 8 2 | ⇠ | .
We use the regularized least-squares estimator LSE (Eq. 8) to extend the estimates for all state-action pairs, with regularization parameter _ = l 2 / ⌫ 2 (recall that ⌫ is deﬁned in
Deﬁnition 1.1):
[Equation]
For ⇠ = ¯ @
= () (the empty sequence), we deﬁne LSE ⇠ , ¯ @ (· , ·) = 0 .
This estimator satisﬁes the guarantee below.
Lemma 4.3.
Let c be a stationary deterministic memoryless policy.
Let ⇠ = ( B 8 , 0 8 ) 8 2 [ = ] be sequences of state-action pairs of some length = 2 N and ¯ @
= ( ¯ @ 8 ) 8 2 [ = ] a sequence of corresponding reals such that for all2 [ [Equation] ] , @ c [Equation]
B 8 , 0 8 )
⇡ l ¯ @ 8 . Then, for all B , 0 2 S ⇥ A ,
[Equation]
where Y is the uniform approximation error from Deﬁnition 1.1.
The proof is given in Appendix C.
The order of the estimation accuracy bound (Eq. 9) is optimal, as shown by the lower bounds of Du et al.
[2019] and Lattimore et al.
We intend to use the LSE estimator given in Eq.
(8) and the bound in Lemma 4.3 only for state-action pairs where k i ( B , 0 ) k
+ ( ⇠ )
 1 (and = =
˜ O( 3 ) ).
We call these state-action pairs covered by ⇠ , and we call a state B covered by ⇠ if for all their corresponding actions 0 , the pair ( B , 0 ) is covered by ⇠ :
[Equation]
We will use the parameter S ﬁx of CAPI (see CAPI’s update rule in Eq. 5) to ensure policies are only updated on covered states, where the approximation error is well-controlled by Eq.
4.2 Main algorithm
Finally, we are ready to introduce CAPI-Q PI -P LAN , presented in Algorithm 3, our algorithm for planning with local access under approximate @ c -realizability.
For this, we deﬁne levels ; = 0 , 1 , . . .
,  , and corresponding suboptimality requirements: For any ; 2 [  + 1 ] , let
[Equation]
for some ˜ 3 = ˜ ⇥ ( 3 ) deﬁned in Eq.
For each level ; , the algorithm maintains a policy c ; and a set of covered states on which it can guarantee that c ; is a  ; -optimal policy.
More speciﬁcally, this set is Cover ( ⇠ ; ) , where ⇠ ; is a list of state-action pairs with elements ⇠ ; , 8 = ( B 8; , 0 8; ) for 8 2 [| ⇠ ; |] .
The algorithm maintains the following suboptimality guarantee below, which we prove in Appendix E after showing some further key properties of the algorithm.
Lemma 4.4.
Assuming that Eq.
(6) holds whenever M EASURE returns success , c ; is  ; -optimal on Cover ( ⇠ ; )
(Deﬁnition 3.2) for all ; 2 [  + 1 ] at the end of every iteration of the main loop of CAPI-Q PI -P LAN .
CAPI-Q PI -P LAN aims to improve the policies, while propagating the members of ⇠ ; to ⇠ ; + 1 , and so on, all the way to ⇠  .
During this, whenever the algorithm discovers a state-action pair with a sufﬁciently “new” feature direction, this pair is appended to the sequence ⇠ 0 corresponding to level 0 , as there are no suboptimality guarantees yet available for such a state.
However, such a discovery can only happen ˜ O( 3 ) times.
When, eventually, all discovered state-action pairs end up in ⇠  , the ﬁnal suboptimality guarantee is reached, and the algorithm returns with the ﬁnal policy.
Note that in the local access setting we consider, the algorithm cannot enumerate the set Cover ( ⇠ ; ) , but can answer membership queries, that is, for any B 2 S it encounters, it is able to decide if B 2 Cover ( ⇠ ; ) .
The algorithm maintains sequences ¯ @ ; , corresponding to ⇠ ; , for each level ; .
Whenever a new ( B , 0 ) pair is appended to the sequence ⇠ ; , a corresponding ?
symbol is appended to the sequence ¯ @ ; , to signal that an estimate of @ c ; ( B , 0 ) is not yet known.
After initializing ⇠ 0 to cover the initial state B 0 (Lines 4 to 6), the algorithm measures @
c ✓ ( B , 0 ) for the smallest level ✓ for which there still exists a ? in the corresponding ¯ @ ✓ . After a successful measurement, if there are no more ? ’s left at this level (i.e., in ¯ @ ✓ ), the algorithm executes a policy update on c ✓ (Line 17) using the least-squares estimate obtained from the measurements at this level, but only for states in Cover ( ⇠ ✓ ) (using S ﬁx = S \ Cover ( ⇠ ✓ ) ). Next, Line 18 merges this new policy c 0 with the existing policy c ✓ + 1 of the next level, setting c ✓ + 1 to be the policy c 00 deﬁned as [Equation]
1: Input: initial state B 0 2 S , dimensionality 3 , parameter bound ⌫ , accuracy l , failure probability X >2: Initialize:  d log (( l / 4 )( 1  W ))/ log W e , for ; 2 [  + 1 ] , ⇠ ; () , ¯ @ ; () , c ; policy that always returns action A 1 , _ l 2 / ⌫3: while True do ù main loop 4: if 9 0 2 A , ( B 0 , 0 ) 8 ActionCover ( ⇠ 0 ) then 5: append ( B 0 , 0 ) to ⇠ 0 , append ? to ¯ @ 0 6: break 7: let ✓ be the smallest integer such that ¯ @ ✓ , < = ?
for some < ; set ✓ =
if no such ; exists 8: if ✓ =
then return c  9: ( status , result )
M EASURE ( B <✓ , 0 <✓ , c ✓ , Cover ( ⇠ ✓ ) , l , X /( ˜ 3 ))
ù recall ⇠ ✓ , < =
( B <✓ , 0 <✓ ) 10: if status = discover then 11: append ( result , 0 ) to ⇠ 0 for some 0 such that ( result , 0 ) 8 ActionCover ( ⇠ 0 ) 12: append ?
to ¯ @ 0 13: break 14: ¯ @ ✓ , < result 15: if ö < 0 such that ¯ @
✓ , <
= ?
then 16: ˆ @
LSE ⇠ ✓ , ¯ @ ✓ 17: cc ˆ @ , c ✓ , S\ Cover ( ⇠ ✓ ) 18: c ✓ +( B 7!
c ✓ + 1 ( B ) if B 2 Cover ( ⇠ ✓ + 1 ) else c 0 ( B )) 19: for ( B , 0 ) 2 ⇠ ✓ such that ( B , 0 ) 8 ⇠ ✓ + 1 do 20: append ( B , 0 ) to ⇠ ✓ + 1 , ?
to ¯ @ ✓ +
This ensures that the existing policy c ✓ + 1 remains unchanged by c 00 (its replacement) on states that are already covered by ⇠ ✓ + 1 , and therefore c 00 2 ⇧ c ✓ + 1 , Cover ( ⇠ ✓ + 1 ) =
⇧ c 00 , Cover ( ⇠ ✓ + 1 ) .
We also observe that ⇠ ; can only grow for any ; (elements are never removed from these sequences), thus for any update where ⇠ ; is assigned a new value ⇠ 0 ; (Lines 5, 11, and 20), + ( ⇠ 0 ; ) ⌫ + ( ⇠ ; ) , and therefore Cover ( ⇠ 0 ; ) ◆ Cover ( ⇠ ; ) and ⇧ c ; , Cover ( ⇠ 0 ; ) ✓ ⇧ c ; , Cover ( ⇠ ; ) .
Combining these properties yields the following result:
Lemma 4.5.
If for any ; 2 [  ] , c ; and ⇠ ; take some values c old ; and ⇠ old ; at any point in the execution of the algorithm, then at any later point during the execution, c ; 2 ⇧ c ; , Cover ( ⇠ ; ) ✓ ⇧ c old ; , Cover ( ⇠ old ; ) .
Any value in ¯ @ ; that is set to anything other than ? will never change again.
Since as long as the sample paths generated by M EASURE in Line 9 of CAPI-Q PI -P LAN remain in Cover ( ⇠ ; ) , their distribution is the same under any policy from ⇧ c ; , Cover ( ⇠ ; ) , the ¯ @ ; estimates are valid for these policies, as well.
Combined with Lemma 4.5, we get that the accuracy guarantees of Lemma 4. continue to hold throughout:
Lemma 4.6.
Assuming that Eq.
(6) holds whenever M EASURE returns success , for any level ; and index < such that ¯ @ ; , < < ? , @ c
( B <; , 0 <; )
⇡ l ¯ @ ; , < for all c 0 2 ⇧ c ; , Cover ( ⇠ ; ) throughout the execution of CAPI-Q PI -P LAN .
Once c ✓ + 1 is updated in Line 18, in Line 20 we append to the sequence ⇠ ✓ + 1 all members of ⇠ ✓ that are not yet in ⇠ ✓ + 1 , while adding a corresponding ?
to ¯ @ ✓ + 1 indicating that these @
-values are not yet measured for policy c ✓ + 1 .
Thus, whenever all ?
values disappear from some level ; 2 [  + 1 ] , by the end of that iteration ⇠ ; + 1 = ⇠ ; , and hence ActionCover ( ⇠ ; ) =
ActionCover ( ⇠ ; + 1 ) .
Together with the fact that for any ; 2 [  + 1 ] , whenever a new state-action pair is appended to ⇠ ; , an ? symbol is appended to ¯ @ ; , we have by induction the following result:
Lemma 4.7.
Throughout the execution of CAPI-Q PI -P LAN ,after Line 7 when ✓ is set,
[Equation]
As a result, whenever the M EASURE call of Line 9 outputs ( discover , B ) for some state B , by Lemma 4.2, there is an action 0 2 A such that ( B , 0 ) 8 ActionCover ( ⇠ ✓ ) =
ActionCover ( ⇠ 0 ) .
This explains why adding such an ( B , 0 ) pair to ⇠ 0 is always possible in Line 11.
Consider the 8 th time Line 11 is executed, and denote B by B 8 and 0 by 0 8 , and +
= _
I + Õ 8  1 C
= 1 i ( B C , 0 C ) i ( B C , 0 C ) > .
Observe that as + 8 = + ( ⇠ ) , ( B 8 , 0 8 ) 8 ActionCover ( ⇠ 0 ) implies k i ( B 8 , 0 8 )
k +  1> 1 .
Therefore,
Õ 8C = 1 min { 1 , k i ( B C , 0 C ) k
+  1 C } = 8 , and thus by the elliptical potential lemma [Lattimore and Szepesvári, 2020, Lemma 19.4], 8  2 3 log ⇣ [Equation] + [Equation]
3_ ⌘ . This inequality is satisﬁed by the largest value of 8 , that is, the total number of times M EASURE returns with discover .
Since any element of ⇠ ; is also an element of ⇠ 0 for any ; 2 [  + 1 ] , we have that at any time during the execution of CAPI-Q PI -P LAN ,
[Equation]
When CAPI-Q PI -P LAN returns at Line 8 with the policy c  , it is   -optimal on Cover ( ⇠  ) by Lemma 4.4 when the estimates of M EASURE are correct.
Furthermore, B 0 2 Cover ( ⇠ 0 ) is guaranteed by Lines 4 to 6, and hence B2 Cover ( ⇠  ) by Lemma 4.7 when the algorithm ﬁnishes.
Hence, bounding   using the deﬁnition of  immediately gives the following result:
Lemma 4.8.
Assuming that Eq.
(6) holds whenever M EASURE returns success , the policy c returned by CAPI-Q PI -P LAN is  -optimal on { B 0 } for
[Equation]
To ﬁnish the proof of Theorem 1.2, we only need to analyze the query complexity and the failure probability (i.e., the probability of Eq.
(6) not being satisﬁed for some M EASURE call that returns success ) of CAPI-Q PI -P LAN :
Proof of Theorem 1.2.
Both the total failure probability and query complexity of CAPI-Q PI -P LAN depend on the number of times M EASURE is executed, as this is the only source of randomness and of interaction with the simulator.
M EASURE can return discover at most | ⇠ 0 | times, which is bounded by ˜ 3 by Eq.
For every ; 2 [  ] , M EASURE is executed exactly once with returning success for each element of ⇠ ; .
Hence, by Eq.
(13) again, M EASURE returns success at most ˜ 3 times, each satisfying Eq.
(6) with probability at least 1  Z = 1  X /( ˜ 3 ) by Lemma 4.2.
By the union bound, M EASURE returns success in all occasions with probability at least 1  X .
Hence Eq.
(6) holds with probability at least 1  X , which, combined with Lemma 4.8, proves Eq.
Each successful run of M EASURE executes at most =
queries ( = is set in Line 2 of Algorithm 2).
Since  < ( 1  W )  1 log ( 4 l  1 ( 1  W )  1 ) =
˜ O
(( 1  W )  1 ) , in total CAPI-Q PI -P LAN executes at most ˜ O  3 ( 1  W )  4 l  queries.
As this happens at most ˜ 3 times, we obtain the desired bound on the query complexity.
⇤
Conclusions and future work
In this paper we presented C ONFIDENT A PPROXIMATE P OLICY I TERATION , a conﬁdent version of
API, which can obtain a stationary policy with a suboptimality guarantee that scales linearly with the effective horizon  = ˜ O
( 1 /( 1  W )) .
This scaling is optimal as shown by Scherrer and Lesner [2012].
CAPI can be applied to local planning with approximate @
c -realizability (yielding the CAPI-Q PIP LAN algorithm) to obtain a sequence of policies with successively reﬁned accuracies on a dynamically evolving set of states, resulting in a ﬁnal, recursively deﬁned policy achieving simultaneously the optimal suboptimality guarantee and best query cost available in the literature.
More precisely,
CAPI-Q PI -P LAN achieves ˜ O
( Y p 3 ) suboptimality, where Y is the uniform policy value-function approximation error.
We showed that this bound is the best (up to polylogarithmic factors) that is achievable by any planner with polynomial query cost.
We also proved that the ˜ O  3 4 Y  query cost of CAPI-Q PI -P LAN is optimal up to polylogarithmic factors in all parameters except for  ; whether the dependence on  is optimal remains an open question.
Finally, our method comes at a memory and computational cost overhead, both for the ﬁnal policy and the planner.
It is an interesting question if this overhead necessarily comes with the API-style method we use (as it is also present in the works of Scherrer and Lesner, 2012, Scherrer, 2014), or if it is possible to reduce it by, for example, compressing the ﬁnal policy into one that is greedy with respect to some action-value function realized with the features.
Acknowledgements
The authors would like to thank Tor Lattimore and Qinghua Liu for helpful discussions.