Confident Approximate Policy Iteration for Efficient Local Planning in 𝑞 𝜋 -realizable MDPs
Abstract
We consider approximate dynamic programming in 𝛾 -discounted Markov decision processes and apply it to approximate planning with linear value-function approximation.
Our first contribution is a new variant of A PPROXIMATE P OLICY I TERATION (API), called C ONFIDENT A PPROXIMATE P OLICY I TERATION (CAPI), which computes a deterministic stationary policy with an optimal error bound scaling linearly with the product of the effective horizon 𝐻 and the worstcase approximation error 𝜀 of the action-value functions of stationary policies.
This improvement over API (whose error scales with 𝐻 2 ) comes at the price of an 𝐻 -fold increase in memory cost.
Unlike Scherrer and Lesner [2012], who recommended computing a non-stationary policy to achieve a similar improvement (with the same memory overhead), we are able to stick to stationary policies.
This allows for our second contribution, the application of CAPI to planning with local access to a simulator and 𝑑 -dimensional linear function approximation.
As such, we design a planning algorithm that applies CAPI to obtain a sequence of policies with successively refined accuracies on a dynamically evolving set of states.
The algorithm outputs an ˜ O(√ 𝑑𝐻𝜀 ) -optimal policy after issuing ˜ O( 𝑑𝐻 4 / 𝜀 2 ) queries to the simulator, simultaneously achieving the optimal accuracy bound and the best known query complexity bound, while earlier algorithms in the literature achieve only one of them.
This query complexity is shown to be tight in all parameters except 𝐻 .
These improvements come at the expense of a mild (polynomial) increase in memory and computational costs of both the algorithm and its output policy.
Introduction
A key question in reinforcement learning is how to use value-function approximation to arrive at scaleable algorithms that can find near-optimal policies in Markov decision processes (MDPs).
A flurry of recent results aims at solving this problem efficiently with varying models of interaction with the MDP.
In this paper we focus on the problem of planning with a simulator when using linear function approximation.
A simulator is a “device” that, given a state-action pair as a query, returns a next state and reward generated from the transition kernel of the MDP that is simulated.
Depending on the application, such a simulator is often readily available (e.g., in chess, go, Atari).
Planning with simulator access comes with great benefits: for example, in a recent work, Wang et al.
[2021] showed that under some conditions it is exponentially more efficient to find a near-optimal policy if a simulator of the MDP (that can reset to a state) is available compared to the online case where a learner interacts with its environment by following trajectories but without the help of a simulator.
Our setting of offline, local planning considers the problem of finding a policy with near-optimal value at a given initial state 𝑠 0 in the MDP.
The planner can issue queries to the simulator, and has to find and output a near-optimal policy with high probability.
The efficiency of a planner is measured in four ways: the suboptimality of the policy found, that is, how far its value is from that of the optimal policy; the query cost , that is, the number of queries issued to the simulator; the computational cost , which is the number of operations used; and the memory cost , which is the amount of memory used (we adopt the real computation model for these costs).
There are several interaction models between the planner and the simulator
[Yin et al., 2022].
The most permissive one is called the generative model , or random access .
Here, the planner receives the set of all states and is allowed to issue queries for any state and action.
Coding a simulator that supports this model can be challenging, as oftentimes the set of states is computationally difficult to describe.
Instead of random access , we consider the more practical and more challenging local access setting, where the planner only sees the initial state and the set of states received as a result to a query to the simulator.
Consequently, the queries issued have to be for a state that has already been encountered this way (and any available action), while the simulator needs to support the ability to reset the MDP state only to previously seen states.
A simple approach in practice to support this model is saving and reloading checkpoints during the operation of the simulator.
To handle large, possibly infinite state spaces, we use linear function approximation to approximate the action-value functions 𝑞 𝜋 of stationary, deterministic policies 𝜋 (for background on MDPs, see the next section).
A feature-map is a good fit to an MDP if the worst-case error of using the feature-map to approximate value functions of policies of the MDP is small:
Definition 1.1 ( 𝑞 𝜋 -realizability: uniform policy value-function approximation error) .
Given an
MDP, the uniform policy value-function approximation error induced by a feature map 𝜑 , which maps state-action pairs ( 𝑠, 𝑎 ) to the Euclidean ball of radius 𝐿 centered at zero in R 𝑑 , over a set of parameters belonging to the 𝑑 -dimensional centered Euclidean ball of radius 𝐵 is
[Equation]
where the outermost supremum is over all possible stationary deterministic memoryless policies (i.e., maps from states to actions) of the MDP.
Our goal is to design algorithms that scale gracefully with the uniform approximation error 𝜀 at the expense of controlled computational cost.
To achieve nontrivial guarantees, the uniform approximation error needs to be “small”.
This (implicit) assumption is stronger than the 𝑞 ★ -realizability assumption (where the approximation error is only considered for optimal policies), which Weisz et al.
[2021] showed an exponential query complexity lower bound for.
At the same time, it is (strictly) weaker than the linear MDP assumption [Zanette et al., 2020], for which there are efficient algorithms to find a near-optimal policy in the online setting (without a simulator)
[Jin et al., 2020], even in the more challenging reward-free setting where the rewards are only revealed after exploration [Wagenmaker et al., 2022].
In the local access setting, the planner learns the features 𝜑 ( 𝑠, 𝑎 ) of a state-action pair only for those states 𝑠 that have already been encountered.
In contrast, in the random access setting, the whole feature map 𝜑 (· , ·) , of (possibly infinite) size 𝑑 |S||A| (where S and A are the state and action sets, resp.), is given to the planner as input.
In the latter setting, when only the query cost is counted, Du et al.
[2019] and Lattimore et al.
[2020] proposed algorithms (the latter working in the misspecified, 𝜀 > 0 regime) that issue a number of queries that is polynomial in the relevant parameters, but require a barycentric spanner or near-optimal design of the input features.
In the worst case, computing any of these sets scales polynomially in |S| and |A| , which can be prohibitive.
In the case of local access , considered in this paper, the best known bound on the suboptimality of the computed policy is achieved by C ONFIDENT MC-P OLITEX [Yin et al., 2022].
In the more permissive random access setting, the best known query cost is achieved by Lattimore et al.
Our algorithm, CAPI-Q PI -P LAN (given in Algorithm 3), achieves the best of both while only assuming local access .
This is shown in the next theorem; in the theorem 𝜀 is as defined in Definition 1.1, 𝛾 is the discount factor, and 𝑣 ★ and 𝑣 𝜋 are the state value functions associated with the optimal policy and policy 𝜋 , respectively (precise definitions of these quantities are given in the next section).
A comparison to other algorithms in the literature is given in Table 1; there the accuracy parameter 𝜔 of the algorithms is set to 𝜀 , but a larger 𝜔 can be used to trade off suboptimality guarantees for an improved query cost.
Theorem 1.2.
For any confidence parameter 𝛿 ∈ ( 0 , 1 ] , accuracy parameter 𝜔 > 0 , and initial state 𝑠 0 ∈ S , with probability at least 1 − 𝛿 , CAPI-Q PI -P LAN (Algorithm 3) finds a policy 𝜋 with
[Equation]
while executing at most ˜ O (cid:0) 𝑑 ( 1 − 𝛾 ) − 4 𝜔 − 2 (cid:1) queries in the local access setting.
CAPI-Q PI -P LAN is based on C ONFIDENT MC-LSPI, another algorithm of Yin et al.
[2022], which relies on policy iteration from a core set of informative state-action pairs, but achieves inferior performance both in terms of suboptimality and query complexity.
However, CAPI-Q PI -P LAN ’s improvements come at the expense of increased memory and computational costs, as shown in the next theorem: compared to C ONFIDENT MC-LSPI, the memory and computational costs of our algorithm increase by a factor of the effective horizon 𝐻 =
˜ O
( 1 /( 1 − 𝛾 )) , and the policy computed by CAPI-Q PI -P LAN uses a 𝑑𝐻 factor more memory for storage and a 𝑑 2 𝐻 factor more computation to evaluate.
Theorem 1.3 (Memory and computational cost) .
The memory and computational cost of running
CAPI-Q PI -P LAN (Algorithm 3) are ˜ O (cid:0)
𝑑 2 /( 1 − 𝛾 ) (cid:1) and ˜ O (cid:0) 𝑑 4 |A|( 1 − 𝛾 ) − 5 𝜔 − 2 (cid:1) , respectively, while the memory and computational costs of storing and evaluating the final policy outputted by CAPIQ PI -P LAN , respectively, are ˜ O (cid:0) 𝑑 2 /( 1 − 𝛾 ) (cid:1) and ˜ O (cid:0) 𝑑 3 |A|/( 1 − 𝛾 ) (cid:1) .
Next we present a lower bound corresponding to Theorem 1.2 that holds even in the more permissive random access setting, and shows that CAPI-Q PI -P LAN trades of the query cost and the suboptimality of the returned policy asymptotically optimally up to its dependence on 1 /( 1 − 𝛾 ) :
Theorem 1.4 (Query cost lower bound) .
Let 𝛼 ∈ ( 0 ,
[Equation] [Equation]
and 𝜀 ≥ 0 .
Then there is a class M of MDPs with uniform policy value-function approximation error at most 𝜀 such that any planner that guarantees to find an 𝛼 -optimal policy 𝜋 (i.e., 𝑣 ★ ( 𝑠 0 )
− 𝑣 𝜋 ( 𝑠 0 ) ≤ 𝛼 ) with probability at least 1 − 𝛿 for all 𝑀 ∈ M when used with a simulator for 𝑀 with random access , the worst-case (over M ) expected number of queries issued by the planner is at least
[Equation]
If 𝜔 is set to 𝜀 for CAPI-Q PI -P LAN , the first term of Eq.
(2) implies that any planner with an asymptotically smaller (apart from logarithmic factors) suboptimality guarantee than Eq.
(1) executes exponentially many queries in expectation.
The second term of Eq.
(2), which is shown to be a lower bound in Theorem H.3 even in the more general setting of linear MDPs with zero misspecification ( 𝜀 = 0 ), matches the query complexity of Theorem 1.2 up to an ˜ O
(( 1 − 𝛾 ) 2 ) factor.
Thus, the lower bound implies that the suboptimality and query cost bounds of Theorem 1.2 are tight up to logarithmic factors in all parameters except the 1 /( 1 − 𝛾 ) -dependence of the query cost bound.
At the heart of our method is a new algorithm, which we call C
ONFIDENT
A PPROXIMATE P OLICY
I TERATION (CAPI).
This algorithm, which belongs to the family of approximate dynamic programming algorithms [Bertsekas, 2012, Munos, 2003, 2005], is a novel variant of A PPROXIMATE
P OLICY
I TERATION (API)
[Bertsekas and Tsitsiklis, 1996]: in the policy improvement step, CAPI only updates the policy in states where it is confident that the update will improve the performance.
This simple modification allows CAPI to avoid the problem of “classical” approximate dynamic programming algorithms (approximate policy and value iteration) of inflating the value function evaluation error by a factor of 𝐻 2 where 𝐻 =
˜
O
( 1 /( 1 − 𝛾 )) (for discussions of this problem, see also the papers by Scherrer and Lesner, 2012 and Russo, 2020), and reduce this inflation factor to 𝐻 .
A similar result has already been achieved by Scherrer and Lesner [2012], who proposed to construct a non-stationary policy that strings together all policies obtained while running either approximate value or policy iteration.
However, applying this result to our planning problem is problematic, since the policies to be evaluated are non-stationary, and hence including them in the policy set we aim to approximate may drastically increase the error 𝜀 as compared to Definition 1.1, which only considers stationary memoryless policies.
While the improvements provided by CAPI allows CAPI-Q PI -P LAN to match the performance of C ONFIDENT MC-P OLITEX in terms of suboptimality, it is unlikely that a simple modification of C ONFIDENT MC-P OLITEX would lead to an algorithm which matches CAPI-Q PI -P LAN ’s
[Table]
performance in terms of query cost (see Table 1): Both methods evaluate a sequence of policies at an ˜ O( 𝜀 ) accuracy each (requiring ˜ O
( 1 / 𝜀 2 ) queries, omitting the dependence on other parameters).
However, while CAPI-Q PI -P LAN (and C ONFIDENT MC-LSPI) evaluates O( log ( 1 / 𝜀 )) (again in terms of 𝜀 only) policies to find one which is ˜ O
( 𝜀 ) -optimal
[Table]
C ONFIDENT MC-P OLITEX needs to compute ˜ O
( 1 / 𝜀 2 ) policies to achieve the same.
As a consequence, C ONFIDENT MC-P OLITEX only achieves ˜ O
( 1 / 𝜀 4 ) query complexity, and to match CAPI-Q PI -P LAN ’s
˜ O
( 1 / 𝜀 2 ) complexity, one would need to come up with either significantly better policy evaluation methods (potentially using the similarity in the subsequent policies) or a much faster (exponential vs. square-root) convergence rate in the suboptimality of the policy sequence produced by C ONFIDENT MC-P OLITEX .
The rest of the paper is organized as follows: The model and notation are introduced in Section 2.
CAPI is introduced and analyzed in Section 3.
Planning with 𝑞 𝜋 -realizability is introduced in
Section 4, with CAPI-Q PI -P LAN being built-up and analyzed in Sections 4.1 and 4.2.
In particular, the proof of Theorem 1.2 is given in Section 4.2.
Several proofs are relegated to appendices, in particular, Theorem 1.3 is proved and implementation details of CAPI-Q PI -P LAN are discussed in
Appendix G, while Theorem 1.4 is proved in Appendix H.
Notation and preliminaries
Let N = { 0 , 1 , . . .
} denote the set of natural numbers, N + = { 1 , 2 , . . .
the positive integers.
For some integer 𝑖 , let [ 𝑖 ] = { 0 , . .
, 𝑖 − 1 } .
For 𝑥 ∈ R , let ⌈ 𝑥 ⌉ denote the smallest integer i such that 𝑖 ≥ 𝑥 .
For a positive definite 𝑉 ∈ R 𝑑 × 𝑑 and 𝑥 ∈ R 𝑑 , let ∥ 𝑥 ∥ 2 𝑉 = 𝑥 ⊤ 𝑉𝑥 .
For matrices 𝐴 and 𝐵 , we say that 𝐴 ⪰ 𝐵 if 𝐴 − 𝐵 is positive semidefinite.
Let I be the 𝑑 -dimensional identity matrix.
For compatible vectors 𝑥, 𝑦 , let ⟨ 𝑥, 𝑦 ⟩ be their inner product: ⟨ 𝑥, 𝑦 ⟩ = 𝑥 ⊤ 𝑦 .
Let M 1 ( 𝑋 ) denote the space of probability distributions supported on the set 𝑋 (throughout, we assume that the 𝜎 -algebra is implicit).
We write 𝑎 ≈ 𝜀 𝑏 for 𝑎, 𝑏, 𝜀 ∈ R if | 𝑎 − 𝑏 | ≤ 𝜀 .
We denote by ˜ O(·) and ˜ Θ (·) the variants of the big-O notation that hide polylogarithmic factors.
A Markov Decision Process (MDP) is a tuple 𝑀 =
(S , A , Q) , where S is a measurable state space,
A is a finite action space, and Q : S ×
A → M 1 (S × [ 0 , 1 ]) is the transition-reward kernel.
We define the transition and reward distributions 𝑃 : S ×
A → M 1 (S) and R : S ×
A → M 1 ([ 0 , 1 ]) as the marginals of Q .
By a slight abuse of notation, for any 𝑠 ∈ S and 𝑎 ∈ A , let 𝑃 (·| 𝑠, 𝑎 ) and
R(·| 𝑠, 𝑎 ) denote the distributions
𝑃 ( 𝑠, 𝑎 ) and R( 𝑠, 𝑎 ) , respectively.
We further denote by 𝑟 ( 𝑠, 𝑎 ) = ∫
1 0 𝑥 d R
( 𝑥 | 𝑠, 𝑎 ) the expected reward for an action 𝑎 ∈ A taken in a state 𝑠 ∈ S .
Without loss of generality, we assume that there is a designated initial state 𝑠 0 ∈ S .
Starting from any state 𝑠 ∈ S , a stationary memoryless policy 𝜋 : S → M 1 (A) interacts with the MDP in a sequential manner for time-steps 𝑡 ∈ N , defining a probability distribution P 𝜋,𝑠 over the episode trajectory { 𝑆 𝑖 , 𝐴 𝑖 , 𝑅 𝑖 } 𝑖 ∈ N as follows:
𝑆 0 =
𝑠 deterministically, 𝐴 𝑖 ∼ 𝜋 ( 𝑆 𝑖 ) , and ( 𝑆 𝑖 + 1 , 𝑅 𝑖 ) ∼ Q( 𝑆 𝑖 , 𝐴 𝑖 ) .
By a slight variation, let P 𝜋,𝑠,𝑎 denote (for some 𝑎 ∈ A ) the distribution of the trajectory when 𝐴
=
𝑎 deterministically, while the distribution of the rest of the trajectory is defined analogously.
This allows us to conveniently define the expected state-value and action-value functions in the discounted setting we consider, for some discount factor 0 <
𝛾 < 1 , respectively, as 𝑣 𝜋 ( 𝑠 ) =
E 𝜋,𝑠 (cid:34)∑︁ 𝑡 ∈ N 𝛾 𝑡 𝑅 𝑡 (cid:35) and 𝑞 𝜋 ( 𝑠, 𝑎 ) =
E 𝜋,𝑠,𝑎 (cid:34)∑︁ 𝑡 ∈ N 𝛾 𝑡 𝑅 𝑡 (cid:35) for all ( 𝑠, 𝑎 ) ∈ S × A , (3) where throughout the paper we use the convention that E • is the expectation operator corresponding to a distribution P • (e.g., E 𝜋,𝑠 is the expectation with respect to P 𝜋,𝑠 ).
It is well known (see, e.g.,
Puterman, 1994) that there exists an optimal stationary deterministic memoryless policy 𝜋 ★ such that sup 𝜋 𝑣 𝜋 ( 𝑠 ) =
𝑣 𝜋 ★ ( 𝑠 ) and sup 𝜋 𝑞 𝜋 ( 𝑠, 𝑎 ) =
𝑞 𝜋 ★ ( 𝑠, 𝑎 ) for all ( 𝑠, 𝑎 ) ∈ S × A .
Let 𝑣 ★ = 𝑣 𝜋 ★ and 𝑞 ★ = 𝑞 𝜋 ★ .
For any policy 𝜋 , 𝑣 𝜋 and 𝑞 𝜋 are known to satisfy the Bellman equations [Puterman, 1994]: 𝑣
[Equation]
) =
∑︁ 𝑎 ∈A 𝜋 ( 𝑎 | 𝑠 ) 𝑞 𝜋 ( 𝑠, 𝑎 ) and 𝑞 𝜋 ( 𝑠, 𝑎 ) = 𝑟 ( 𝑠, 𝑎 ) +
𝛾 ∫ 𝑠 ′ ∈S
𝑣 𝜋 ( 𝑠 ′ )
d 𝑃 ( 𝑠 ′ | 𝑠, 𝑎 ) for all ( 𝑠, 𝑎 ) ∈S × A .
Finally, we call a policy 𝜋 deterministic if for all states, 𝜋 ( 𝑠 ) is a distribution that assigns unit weight to one action and zero weight to the others.
With a slight abuse of notation, for a deterministic policy 𝜋 , we denote by 𝜋 ( 𝑠 ) the action 𝜋 chooses (deterministically) in state 𝑠 ∈ S .
Confident Approximate Policy Iteration
In this section we introduce C ONFIDENT A PPROXIMATE P OLICY
I TERATION (CAPI), our new approximate dynamic programming algorithm.
In approximate dynamic programming, the methods are designed around oracles that return either an approximation to the application of the Bellman optimality operator to a value function (“approximate value iteration”), or an approximation to the value function of some policy (“approximate policy iteration”).
Our setting is the second.
The novelty is that we assume access to the accuracy of the approximation and use this knowledge to modify the policy update, which leads to improved guarantees on the suboptimality of the computed policy.
We present the pseudocodes of API [Bertsekas and Tsitsiklis, 1996] and CAPI jointly in Algorithm 1: starting from an arbitrary (deterministic) policy 𝜋 0 , the algorithm iterates a policy estimation (Line 2) and a policy update step (Line 3) 𝐼 times.
The policy update for API is greedy with respect to the action-value estimates ˆ 𝑞 and is defined as 𝜋 ˆ 𝑞 ( 𝑠 )
=
arg max 𝑎 ∈A ˆ 𝑞 ( 𝑠, 𝑎 ) .
We assume that arg max 𝑎 ∈A breaks ties in a consistent manner by ordering the actions (using the notation
A =
(A 1 , . . .
, A |A| ) ) and always choosing action A 𝑖 with the lowest index 𝑖 that achieves the maximum.
For CAPI, the policy update further relies on a global estimation-accuracy parameter 𝜔 , and a set of fixed-states
S fix ⊆ S .
For the purposes of this section, it is enough to keep S fix = {} .
CAPI updates the policy to one that acts greedily with respect to ˆ 𝑞 only on states that are not in S fix and where it is confident that this leads to an improvement over the previous policy (Case 5a); otherwise, the new policy will return the same action as the previous one (Case 5b).
To decide, ˆ 𝑞 ( 𝑠, 𝜋 ( 𝑠 ))
+ 𝜔 is treated as the upper bound on the previous policy’s value, and max 𝑎 ∈A ˆ 𝑞 ( 𝑠, 𝑎 ) − 𝜔 as the lower bound of the action-value of the greedy action (Eq. 5):
[Equation]
Note that 𝜋 ˆ 𝑞,𝜋, S fix also depends on 𝜔 , however, this dependence is omitted from the notation (as 𝜔 is kept fixed throughout).
CAPI can also be seen as a refinement of C ONSERVATIVE P OLICY I TERATION (CPI) of Kakade and Langford [2002] with some important differences: While CPI introduces a global parameter to ensure the update stays close to the previous policy, CAPI has no such parameter, and it dynamically decides when to stay close to (more precisely, use) the previous policy, individually for every state, based on whether there is evidence for a guaranteed improvement.
Let 𝜋 be any stationary deterministic memoryless policy, ˆ 𝑞 𝜋 : S ×
A → R be any function, 𝜔 ∈ R + , and S fix ⊆ S .
First, we show that as long as ˆ 𝑞 𝜋 is an 𝜔 -accurate estimate of 𝑞 𝜋 , the CAPI policy update only improves the policy’s values:
Lemma 3.1 (No deterioration) .
Let 𝜋 ′ =
𝜋 ˆ 𝑞 𝜋 ,𝜋, S fix .
Assume that for all 𝑠 ∈ S \ S fix and 𝑎 ∈ A , ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) ≈ 𝜔 𝑞 𝜋 ( 𝑠, 𝑎 ) .
Then, for any 𝑠 ∈ S , 𝑣 𝜋 ′ ( 𝑠 ) ≥ 𝑣 𝜋 ( 𝑠 ) .
Proof.
Fix any 𝑠 ∈ S .
If 𝑠 ∈ S fix or ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ( 𝑠 )) + 𝜔 ≥ max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) − 𝜔 , then 𝜋 ′ ( 𝑠 ) =
𝜋 ( 𝑠 ) and therefore 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 ))
= 𝑣 𝜋 ( 𝑠 ) .
Otherwise, 𝑠 ∉ S fix and ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ( 𝑠 ))
+ 𝜔 ≤ max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) − 𝜔 , hence 𝜋 ′ ( 𝑠 ) =
arg max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) , and it follows by our assumptions that 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 )) ≥
I
1: for 𝑖 = 1 to 𝐼 do 2: ˆ 𝑞 ← E STIMATE ( 𝜋 𝑖 − 1 ) 3: 𝜋 𝑖 ← (cid:26) 𝜋 ˆ 𝑞
API 𝜋 ˆ 𝑞,𝜋 𝑖 − 1 , S fix
CAPI 4: return 𝜋 𝐼 ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 ))
− 𝜔 = max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) − 𝜔 > ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ( 𝑠 ))
+ 𝜔 ≥ 𝑞 𝜋 ( 𝑠, 𝜋 ( 𝑠 ))
= 𝑣 𝜋 ( 𝑠 ) .
Therefore, in any case, 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 )) ≥ 𝑣 𝜋 ( 𝑠 ) .
Since this holds for any 𝑠 ∈ S , the Policy Improvement Theorem [Sutton and Barto, 2018, Section 4.2] implies that for any 𝑠 ∈ S , 𝑣 𝜋 ′ ( 𝑠 ) ≥ 𝑣 𝜋 ( 𝑠 ) .
□
Next we introduce two approximate optimality criterion for a policy on a set of states:
Definition 3.2 (Policy optimality on a set of states) .
A policy 𝜋 is Δ -optimal (for some Δ ≥ 0 ) on a set of
states S ′ ⊆ S , if for all 𝑠 ∈ S ′ , 𝑣 ★ ( 𝑠 ) − 𝑣 𝜋 ( 𝑠 ) ≤ Δ .
Definition 3.3 (Next-state optimality on a set of states) .
A policy 𝜋 is next-state Δ -optimal (for some Δ ≥ 0 ) on a set of
states S ′ ⊆ S , if for all 𝑠 ∈ S ′ and all actions 𝑎 ∈ A , ∫ 𝑠 ′ ∈S
( 𝑣 ★ ( 𝑠 ′ )
− 𝑣 𝜋 ( 𝑠 ′ ))
d 𝑃 ( 𝑠 ′ | 𝑠, 𝑎 ) ≤ Δ .
Note that in the special case of S ′ =
S the first property implies the second, that is, if 𝜋 is Δ -optimal on S , then it is also next-state Δ -optimal on S .
Next, we show that the suboptimality of a policy updated by CAPI evolves as follows (the proof is relegated to Appendix A):
Lemma 3.4 (Iteration progress) .
Let 𝜋 ′ =
𝜋 ˆ
𝑞 𝜋 ,𝜋, S fix .
Assume that for all 𝑠 ∈ S \ S fix and 𝑎 ∈ A , ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) ≈ 𝜔 𝑞 𝜋 ( 𝑠, 𝑎 ) , and that 𝜋 is next-state Δ -optimal on S \ S fix .
Then 𝜋 ′ is ( 4 𝜔 + 𝛾 Δ ) -optimal on S \ S fix .
3.1 CAPI guarantee with accurate estimation everywhere
To obtain a final suboptimality guarantee for CAPI, first consider the ideal scenario in which we assume that we have a mechanism to estimate 𝑞 𝜋 ( 𝑠, 𝑎 ) up to some 𝜔 accuracy for all 𝑠 ∈ S and 𝑎 ∈ A , and for any policy 𝜋 :
Assumption 3.5.
There is an oracle called E STIMATE that accepts a policy 𝜋 and returns ˆ 𝑞 𝜋 :
S ×
A → R such that for all 𝑠 ∈ S and 𝑎 ∈ A , ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) ≈ 𝜔 𝑞 𝜋 ( 𝑠, 𝑎 ) .
Theorem 3.6 (CAPI performance) .
Assume CAPI (Algorithm 1) is run with S fix = {} , iteration count to 𝐼 = ⌈ log 𝜔 / log 𝛾 ⌉ , and suppose that the estimation used in Line 2 satisfies Assumption 3.5.
Then the policy 𝜋 𝐼 returned by the algorithm is 5 𝜔 /( 1 − 𝛾 ) -optimal on S .
Proof.
We prove by induction that policy 𝜋 𝑖 is Δ 𝑖 -optimal on S for Δ 𝑖
= 4 𝜔 (cid:205)
𝑗 ∈[ 𝑖 ] 𝛾 𝑗 + 𝛾 𝑖 1 − 𝛾 .
This holds immediately for the base case of 𝑖 = 0 , as rewards are bounded in [ 0 , 1 ] and thus 𝑣 ★ ( 𝑠 ) ≤ 1 /( 1 − 𝛾 ) for any 𝑠 .
Assuming now that the inductive hypothesis holds for 𝑖 − 1 we observe that 𝜋 𝑖 − 1 is next-state Δ -optimal on S
=
S \ S fix .
Together with Assumption 3.5, this implies that the conditions of Lemma 3.4 are satisfied for 𝜋
= 𝜋 𝑖 − 1 , which yields 𝑣 ★ ( 𝑠 ) − 𝑣 𝜋 𝑖 ( 𝑠 ) ≤ 4 𝜔 + 𝛾 Δ 𝑖 − 1 =
Δ 𝑖 , finishing the induction.
Finally, by the definition of 𝐼 , 𝜋 𝐼 is Δ 𝐼 -optimal with Δ 𝐼 ≤ 4 𝜔 1 − 𝛾
+ 𝛾 𝐼 1 − 𝛾 ≤ 5 𝜔 1 − 𝛾 .
□
Local access planning with 𝑞 𝜋 -realizability
Our planner, CAPI-Q PI -P LAN , is based on the C ONFIDENT MC-LSPI algorithm of Yin et al.
This latter algorithm gradually builds a core set of state-action pairs whose corresponding features are informative.
The 𝑞 -values of the state-action pairs in the core set are estimated using rollouts.
The procedure is restarted with an extended core set whenever the algorithm encounters a new informative feature.
If such a new feature is not encountered, the estimation error can be controlled, and the estimation is extended to all state-action pairs using the least-squares estimator.
Finally, the extended estimation is used in Line 2 of API.
CAPI-Q PI -P LAN improves upon C ONFIDENT MC-LSPI in two ways.
First, using CAPI instead of API improves the final suboptimality bound by a factor of the effective horizon.
Second, we
Algorithm
1: Input: state 𝑠 , action 𝑎 , deterministic policy 𝜋 , set of states S ′ ⊆ S , accuracy 𝜔 > 0 , failure probability 𝜁 ∈ ( 0 , 1 ] 2: Initialize: 𝐻 ← ⌈ log (( 𝜔 / 4 )( 1 − 𝛾 ))/ log 𝛾 ⌉ , 𝑛 ← (cid:6) ( 𝜔 / 4 ) − 2 ( 1 − 𝛾 ) − 2 log ( 2 / 𝜁 )/ 2 (cid:7) 3: for 𝑖 = 1 to 𝑛 do 4: ( 𝑆, 𝑅 𝑖, 0 ) ← S IMULATOR ( 𝑠, 𝑎 ) 5: for ℎ = 1 to 𝐻 − 1 do 6: if 𝑆 ∉ S ′ then return ( discover , 𝑆 ) 7: 𝐴 ← 𝜋 ( 𝑆 ) 8: ( 𝑆, 𝑅 𝑖,ℎ ) ← S IMULATOR ( 𝑆, 𝐴 )
⊲ Call to the simulator oracle 9: return ( success , 1 𝑛 (cid:205)
𝑛𝑖 = 1 (cid:205)
𝐻 − 1 ℎ =
𝛾 ℎ 𝑅 𝑖,ℎ ) apply a novel analysis on a more modular variant of the C ONFIDENT R OLLOUT subroutine used in
C ONFIDENT MC-LSPI, which delivers 𝑞 -estimation accuracy guarantees with respect to a large class of policies simultaneously.
This allows for a dynamically evolving version of policy iteration, that does not have to restart whenever a new informative feature is encountered.
Intuitively, this prevents duplication of work.
4.1 Estimation oracle
To obtain an algorithm for planning with local access whose performance degrades gracefully with the uniform approximation error, we must weaken Assumption 3.5.
This is because under local access, we cannot guarantee to cover all states or hope to obtain accurate 𝑞 -value estimates for all states.
Instead, we are interested in an accuracy guarantee that holds for 𝑞 -values only on some subset
S ′ ⊆ S of states, but holds simultaneously for any policy that agrees with 𝜋 on S ′ but may take arbitrary values elsewhere.
For this, we define the extended set of policies:
Definition 4.1.
Let Π det be the set of all stationary deterministic memoryless policies, 𝜋 ∈ Π det , and
S ′ ⊆ S .
For ( 𝜋, S ′ ) , we define Π 𝜋, S ′ to be the set of policies that agree with 𝜋 on 𝑠 ∈ S ′ :
[Equation]
We aim to first accurately estimate 𝑞 𝜋
( 𝑠, 𝑎 ) for some specific ( 𝑠, 𝑎 ) pairs, based on which we extend the estimates to other state-action pairs using least-squares.
To this end, we first devise a subroutine called M EASURE (Algorithm 2).
M EASURE is a modularized variant of the C ONFIDENT R OLLOUT subroutine of Yin et al.
The modularity of our variant is due to the parameter S ′
that corresponds to the set of states on which the planner is “confident” for C ONFIDENT R OLLOUT .
M EASURE unrolls the policy 𝜋 starting from ( 𝑠, 𝑎 ) for a number of episodes, each lasting 𝐻 steps, and returns with the average measured reward.
Throughout, we let 𝐻 = ⌈ log (( 𝜔 / 4 )( 1 − 𝛾 ))/ log 𝛾 ⌉ be the effective horizon.
At the end of this process, M EASURE returns status success along with the empirical average 𝑞 -value, where compared to Eq.
(3), the discounted summation of rewards is truncated to 𝐻 .
If, however, the algorithm encounters a state not in its input S ′ , it returns with status discover , along with that state.
This is because in such cases, the algorithm could no longer guarantee an accurate estimation with respect to any member of the extended set of policies.
The next lemma, proved in Appendix B, shows that M EASURE provides accurate estimates of the action-value functions for members of the extended policy set.
Lemma 4.2.
For any input parameters 𝑠 ∈ S , 𝑎 ∈ A , 𝜋 ∈ Π det , S ′ ⊂ S , 𝜔 > 0 , 𝜁 ∈ ( 0 , 1 ) ,
M EASURE either returns with ( discover , 𝑠 ′ ) for some 𝑠 ′ ∉ S ′
(Line 6), or it returns with ( success , ˜ 𝑞 ) such that with probability at least 1 − 𝜁 ,
[Equation]
Suppose we have a list of state-action pairs 𝐶 =
( 𝑠 𝑖 , 𝑎 𝑖 ) 𝑖 ∈[| 𝐶 |] and corresponding 𝑞 -estimates ¯ 𝑞
= ( ¯ 𝑞 𝑖 ) 𝑖
∈| 𝐶 | .
We use the regularized least-squares estimator LSE (Eq. 8) to extend the estimates for all state-action pairs, with regularization parameter 𝜆 = 𝜔 2 / 𝐵 2 (recall that 𝐵 is defined in
Definition 1.1):
[Equation]
For 𝐶 = ¯ 𝑞 = () (the empty sequence), we define LSE 𝐶, ¯ 𝑞 (· , ·) = 0 .
This estimator satisfies the guarantee below.
Lemma 4.3.
Let 𝜋 be a stationary deterministic memoryless policy.
Let 𝐶 =
( 𝑠 𝑖 , 𝑎 𝑖 )
𝑖 ∈[ 𝑛 ] be sequences of state-action pairs of some length 𝑛 ∈ N and ¯ 𝑞
= ( ¯ 𝑞 𝑖 ) 𝑖 ∈[ 𝑛 ] a sequence of corresponding reals such that for all 𝑖 ∈ [ 𝑛 ] , 𝑞 𝜋 ( 𝑠 𝑖 , 𝑎 𝑖 ) ≈
𝜔 ¯ 𝑞 𝑖 .
Then, for all 𝑠, 𝑎 ∈ S × A ,
[Equation]
LSE 𝐶
[Equation]
¯
[Equation]
𝑠
[Equation]
𝑎
[Equation]
where 𝜀 is the uniform approximation error from Definition 1.1.
The proof is given in Appendix C.
The order of the estimation accuracy bound (Eq. 9) is optimal, as shown by the lower bounds of Du et al.
[2019] and Lattimore et al.
We intend to use the LSE estimator given in Eq.
(8) and the bound in Lemma 4.3 only for state-action pairs where ∥ 𝜑 ( 𝑠, 𝑎 )
∥
𝑉 ( 𝐶 ) − 1 ≤ 1 (and 𝑛 = ˜ O
( 𝑑 ) ).
We call these state-action pairs covered by 𝐶 , and we call a state 𝑠 covered by 𝐶 if for all their corresponding actions 𝑎 , the pair ( 𝑠, 𝑎 ) is covered by 𝐶 :
[Equation]
We will use the parameter S fix of CAPI (see CAPI’s update rule in Eq. 5) to ensure policies are only updated on covered states, where the approximation error is well-controlled by Eq.
4.2 Main algorithm
Finally, we are ready to introduce CAPI-Q PI -P LAN , presented in Algorithm 3, our algorithm for planning with local access under approximate 𝑞 𝜋 -realizability.
For this, we define levels 𝑙 = 0 , 1 , . . .
, 𝐻 , and corresponding suboptimality requirements:
For any 𝑙 ∈ [ 𝐻 + 1 ] , let
[Equation]
for some ˜ 𝑑 = ˜ Θ ( 𝑑 ) defined in Eq.
For each level 𝑙 , the algorithm maintains a policy 𝜋 𝑙 and a set of covered states on which it can guarantee that 𝜋 𝑙 is a Δ 𝑙 -optimal policy.
More specifically, this set is Cover ( 𝐶 𝑙 ) , where 𝐶 𝑙 is a list of state-action pairs with elements 𝐶 𝑙,𝑖 =
( 𝑠 𝑖𝑙 , 𝑎 𝑖𝑙 ) for 𝑖 ∈ [| 𝐶 𝑙 |] .
The algorithm maintains the following suboptimality guarantee below, which we prove in Appendix E after showing some further key properties of the algorithm.
Lemma 4.4.
Assuming that Eq.
(6) holds whenever M EASURE returns success , 𝜋 𝑙 is Δ 𝑙 -optimal on Cover ( 𝐶 𝑙 )
(Definition 3.2) for all 𝑙 ∈
[ 𝐻 + 1 ] at the end of every iteration of the main loop of
CAPI-Q PI -P LAN .
CAPI-Q PI -P LAN aims to improve the policies, while propagating the members of 𝐶 𝑙 to 𝐶 𝑙 + 1 , and so on, all the way to 𝐶 𝐻 .
During this, whenever the algorithm discovers a state-action pair with a sufficiently “new” feature direction, this pair is appended to the sequence 𝐶 0 corresponding to level 0 , as there are no suboptimality guarantees yet available for such a state.
However, such a discovery can only happen ˜ O
( 𝑑 ) times.
When, eventually, all discovered state-action pairs end up in 𝐶 𝐻 , the final suboptimality guarantee is reached, and the algorithm returns with the final policy.
Note that in the local access setting we consider, the algorithm cannot enumerate the set Cover ( 𝐶 𝑙 ) , but can answer membership queries, that is, for any 𝑠 ∈ S it encounters, it is able to decide if 𝑠 ∈ Cover ( 𝐶 𝑙 ) .
The algorithm maintains sequences ¯ 𝑞 𝑙 , corresponding to 𝐶 𝑙 , for each level 𝑙 .
Whenever a new ( 𝑠, 𝑎 ) pair is appended to the sequence 𝐶 𝑙 , a corresponding ⊥ symbol is appended to the sequence ¯ 𝑞 𝑙 , to signal that an estimate of 𝑞 𝜋 𝑙 ( 𝑠, 𝑎 ) is not yet known.
After initializing 𝐶 0 to cover the initial state 𝑠
(Lines 4 to 6), the algorithm measures 𝑞 𝜋 ℓ ( 𝑠, 𝑎 ) for the smallest level ℓ for which there still exists a ⊥ in the corresponding ¯ 𝑞 ℓ .
After a successful measurement, if there are no more ⊥ ’s left at this level (i.e., in ¯ 𝑞 ℓ ), the algorithm executes a policy update on 𝜋 ℓ (Line 17) using the least-squares estimate obtained from the measurements at this level, but only for states in Cover ( 𝐶 ℓ ) (using S fix =
S \ Cover ( 𝐶 ℓ ) ).
Next, Line 18 merges this new policy 𝜋 ′ with the existing policy 𝜋 ℓ + 1 of the next level, setting 𝜋 ℓ + 1 to be the policy 𝜋 ′′ defined as
[Equation]
1: Input: initial state 𝑠 0 ∈ S , dimensionality 𝑑 , parameter bound 𝐵 , accuracy 𝜔 , failure probability 𝛿 > 2: Initialize: 𝐻 ← ⌈ log (( 𝜔 / 4 )( 1 − 𝛾 ))/ log 𝛾 ⌉ , for 𝑙 ∈
[ 𝐻 + 1 ] , 𝐶 𝑙 ← () , ¯ 𝑞 𝑙 ← () , 𝜋 𝑙 ← policy that always returns action
A 1 , 𝜆 ← 𝜔 2 / 𝐵3: while True do ⊲ main loop 4: if ∃ 𝑎 ∈ A , ( 𝑠 0 , 𝑎 )
∉ ActionCover ( 𝐶 0 ) then 5: append ( 𝑠 0 , 𝑎 ) to 𝐶 0 , append ⊥ to ¯ 𝑞6: break 7: let ℓ be the smallest integer such that ¯ 𝑞 ℓ,𝑚 = ⊥ for some 𝑚 ; set ℓ =
𝐻 if no such 𝑙 exists 8: if ℓ = 𝐻 then return 𝜋 𝐻 9: ( status , result ) ← M EASURE ( 𝑠 𝑚ℓ , 𝑎 𝑚ℓ , 𝜋 ℓ , Cover ( 𝐶 ℓ ) , 𝜔, 𝛿 /( ˜ 𝑑𝐻 ))
⊲ recall 𝐶 ℓ,𝑚 =
( 𝑠 𝑚ℓ , 𝑎 𝑚ℓ ) 10: if status = discover then 11: append ( result , 𝑎 ) to 𝐶 0 for some 𝑎 such that ( result , 𝑎 )
∉ ActionCover ( 𝐶 0 ) 12: append ⊥ to ¯ 𝑞13: break 14: ¯ 𝑞 ℓ,𝑚 ← result 15: if (cid:154) 𝑚 ′ such that ¯ 𝑞 ℓ,𝑚 ′ =
⊥ then 16: ˆ 𝑞 ← LSE 𝐶 ℓ , ¯ 𝑞 ℓ 17: 𝜋 ′ ← 𝜋 ˆ 𝑞,𝜋 ℓ , S\ Cover ( 𝐶 ℓ ) 18: 𝜋 ℓ + 1 ← ( 𝑠 ↦→ 𝜋 ℓ +
( 𝑠 ) if 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) else
𝜋 ′ ( 𝑠 )) 19: for ( 𝑠, 𝑎 ) ∈
𝐶 ℓ such that ( 𝑠, 𝑎 ) ∉ 𝐶 ℓ
+ 1 do 20: append ( 𝑠, 𝑎 ) to 𝐶 ℓ + 1 , ⊥ to ¯ 𝑞 ℓ +
This ensures that the existing policy 𝜋 ℓ + 1 remains unchanged by 𝜋 ′′ (its replacement) on states that are already covered by 𝐶 ℓ + 1 , and therefore 𝜋 ′′ ∈ Π 𝜋 ℓ + 1 , Cover ( 𝐶 ℓ + 1 ) =
Π 𝜋 ′′ , Cover ( 𝐶 ℓ + 1 ) .
We also observe that 𝐶 𝑙 can only grow for any 𝑙 (elements are never removed from these sequences), thus for any update where 𝐶 𝑙 is assigned a new value 𝐶 ′ 𝑙
(Lines 5, 11, and 20), 𝑉 ( 𝐶 ′ 𝑙 )
⪰ 𝑉 ( 𝐶 𝑙 ) , and therefore
Cover ( 𝐶 ′ 𝑙 ) ⊇
Cover ( 𝐶 𝑙 ) and Π 𝜋 𝑙 , Cover ( 𝐶 ′ 𝑙 ) ⊆
Π 𝜋 𝑙 , Cover ( 𝐶 𝑙 ) .
Combining these properties yields the following result:
Lemma 4.5.
If for any 𝑙 ∈ [ 𝐻 ] , 𝜋 𝑙 and 𝐶 𝑙 take some values 𝜋 old 𝑙 and 𝐶 old 𝑙 at any point in the execution of the algorithm, then at any later point during the execution, 𝜋 𝑙 ∈ Π 𝜋 𝑙 , Cover ( 𝐶 𝑙 )
⊆
Π 𝜋 old 𝑙 , Cover ( 𝐶 old 𝑙 ) .
Any value in ¯ 𝑞 𝑙 that is set to anything other than ⊥ will never change again.
Since as long as the sample paths generated by M EASURE in Line 9 of CAPI-Q PI -P LAN remain in Cover ( 𝐶 𝑙 ) , their distribution is the same under any policy from Π 𝜋 𝑙 , Cover ( 𝐶 𝑙 ) , the ¯ 𝑞 𝑙 estimates are valid for these policies, as well.
Combined with Lemma 4.5, we get that the accuracy guarantees of Lemma 4.
continue to hold throughout:
Lemma 4.6.
Assuming that Eq.
(6) holds whenever M EASURE returns success , for any level 𝑙 and index 𝑚 such that ¯ 𝑞 𝑙,𝑚 ≠ ⊥ , 𝑞 𝜋 ′
( 𝑠 𝑚𝑙 , 𝑎 𝑚𝑙 ) ≈
𝜔 ¯ 𝑞 𝑙,𝑚 for all 𝜋 ′ ∈ Π 𝜋 𝑙 , Cover ( 𝐶 𝑙 ) throughout the execution of CAPI-Q PI -P LAN .
Once 𝜋 ℓ + 1 is updated in Line 18, in Line 20 we append to the sequence 𝐶 ℓ + 1 all members of 𝐶 ℓ that are not yet in 𝐶 ℓ + 1 , while adding a corresponding ⊥ to ¯ 𝑞 ℓ + 1 indicating that these 𝑞 -values are not yet measured for policy 𝜋 ℓ + 1 .
Thus, whenever all ⊥ values disappear from some level 𝑙 ∈ [ 𝐻 + 1 ] , by the end of that iteration 𝐶 𝑙 + 1 = 𝐶 𝑙 , and hence ActionCover ( 𝐶 𝑙 ) =
ActionCover ( 𝐶 𝑙 + 1 ) .
Together with the fact that for any 𝑙 ∈ [ 𝐻 + 1 ] , whenever a new state-action pair is appended to 𝐶 𝑙 , an ⊥ symbol is appended to ¯ 𝑞 𝑙 , we have by induction the following result:
Lemma 4.7.
Throughout the execution of CAPI-Q PI -P LAN ,after Line 7 when ℓ is set,
ActionCover
[Equation]
As a result, whenever the M EASURE call of Line 9 outputs ( discover , 𝑠 ) for some state 𝑠 , by
Lemma 4.2, there is an action 𝑎 ∈
A such that ( 𝑠, 𝑎 )
∉
ActionCover ( 𝐶 ℓ ) =
ActionCover ( 𝐶 0 ) .
This explains why adding such an ( 𝑠, 𝑎 ) pair to 𝐶 0 is always possible in Line 11.
Consider the 𝑖 th time Line 11 is executed, and denote 𝑠 by 𝑠 𝑖 and 𝑎 by 𝑎 𝑖 , and 𝑉 𝑖 = 𝜆 I + (cid:205)
𝑖 − 1 𝑡 = 1 𝜑 ( 𝑠 𝑡 , 𝑎 𝑡 ) 𝜑 ( 𝑠 𝑡 , 𝑎 𝑡 ) ⊤ .
Observe that as 𝑉 𝑖 = 𝑉 ( 𝐶 ) , ( 𝑠 𝑖 , 𝑎 𝑖 )
∉
ActionCover ( 𝐶 0 ) implies ∥ 𝜑 ( 𝑠 𝑖 , 𝑎 𝑖 )
∥
𝑉 − 1 𝑖 > 1 .
Therefore,
(cid:205) 𝑖𝑡
= 1 min { 1 , ∥ 𝜑 ( 𝑠 𝑡 , 𝑎 𝑡 )∥
𝑉 − 1 𝑡 } = 𝑖 , and thus by the elliptical potential lemma [Lattimore and Szepesvári, 2020, Lemma 19.4], 𝑖 ≤ 2 𝑑 log (cid:16) 𝑑𝜆 + 𝑖𝐿 2 𝑑𝜆 (cid:17) .
This inequality is satisfied by the largest value of 𝑖 , that is, the total number of times M EASURE returns with discover .
Since any element of 𝐶 𝑙 is also an element of 𝐶 0 for any 𝑙 ∈
[ 𝐻 + 1 ] , we have that at any time during the execution of CAPI-Q PI -P LAN ,
[Equation]
When CAPI-Q PI -P LAN returns at Line 8 with the policy 𝜋 𝐻 , it is Δ 𝐻 -optimal on Cover ( 𝐶 𝐻 ) by
Lemma 4.4 when the estimates of M EASURE are correct.
Furthermore, 𝑠 0 ∈ Cover ( 𝐶 0 ) is guaranteed by Lines 4 to 6, and hence 𝑠 0 ∈ Cover ( 𝐶 𝐻 ) by Lemma 4.7 when the algorithm finishes.
Hence, bounding Δ 𝐻 using the definition of 𝐻 immediately gives the following result:
Lemma 4.8.
Assuming that Eq.
(6) holds whenever M EASURE returns success , the policy 𝜋 returned by CAPI-Q PI -P LAN is Δ -optimal on { 𝑠 0 } for
[Equation]
To finish the proof of Theorem 1.2, we only need to analyze the query complexity and the failure probability (i.e., the probability of Eq.
(6) not being satisfied for some M EASURE call that returns success ) of CAPI-Q PI -P LAN :
Proof of Theorem 1.2.
Both the total failure probability and query complexity of CAPI-Q PI -P LAN depend on the number of times M EASURE is executed, as this is the only source of randomness and of interaction with the simulator.
M EASURE can return discover at most | 𝐶 0 | times, which is bounded by ˜ 𝑑 by Eq.
For every 𝑙 ∈ [ 𝐻 ] , M EASURE is executed exactly once with returning success for each element of 𝐶 𝑙 .
Hence, by Eq.
(13) again, M EASURE returns success at most ˜ 𝑑𝐻 times, each satisfying Eq.
(6) with probability at least 1 − 𝜁 = 1 − 𝛿 /( ˜ 𝑑𝐻 ) by Lemma 4.2.
By the union bound,
M EASURE returns success in all occasions with probability at least 1 − 𝛿 .
Hence Eq.
(6) holds with probability at least 1 − 𝛿 , which, combined with Lemma 4.8, proves Eq.
Each successful run of M EASURE executes at most 𝑛𝐻 queries ( 𝑛 is set in Line 2 of Algorithm 2).
Since 𝐻 < ( 1 − 𝛾 ) − 1 log ( 4 𝜔 − 1 ( 1 − 𝛾 ) − 1 )
=
˜ O
(( 1 − 𝛾 ) − 1 ) , in total CAPI-Q PI -P LAN executes at most ˜ O (cid:0) 𝑑 ( 1 − 𝛾 ) − 4 𝜔 − 2 (cid:1) queries.
As this happens at most ˜ 𝑑𝐻 times, we obtain the desired bound on the query complexity.
□
Conclusions and future work
In this paper we presented C ONFIDENT A PPROXIMATE P OLICY I TERATION , a confident version of
API, which can obtain a stationary policy with a suboptimality guarantee that scales linearly with the effective horizon 𝐻 = ˜
O
( 1 /( 1 − 𝛾 )) .
This scaling is optimal as shown by Scherrer and Lesner [2012].
CAPI can be applied to local planning with approximate 𝑞 𝜋 -realizability (yielding the CAPI-Q PIP LAN algorithm) to obtain a sequence of policies with successively refined accuracies on a dynamically evolving set of states, resulting in a final, recursively defined policy achieving simultaneously the optimal suboptimality guarantee and best query cost available in the literature.
More precisely,
CAPI-Q PI -P LAN achieves ˜ O
( 𝜀 √ 𝑑𝐻 ) suboptimality, where 𝜀 is the uniform policy value-function approximation error.
We showed that this bound is the best (up to polylogarithmic factors) that is achievable by any planner with polynomial query cost.
We also proved that the ˜ O (cid:0) 𝑑𝐻 4 𝜀 − 2 (cid:1) query cost of CAPI-Q PI -P LAN is optimal up to polylogarithmic factors in all parameters except for 𝐻 ; whether the dependence on 𝐻 is optimal remains an open question.
Finally, our method comes at a memory and computational cost overhead, both for the final policy and the planner.
It is an interesting question if this overhead necessarily comes with the API-style method we use (as it is also present in the works of Scherrer and Lesner, 2012, Scherrer, 2014), or if it is possible to reduce it by, for example, compressing the final policy into one that is greedy with respect to some action-value function realized with the features.
Acknowledgements
The authors would like to thank Tor Lattimore and Qinghua Liu for helpful discussions.
A Proof of Lemma 3.
Take any 𝑠 ∈ S \ S fix .
[Equation]
where the first equality holds because 𝜋 ′ is deterministic, and the inequality is true because
[Equation]
by Lemma 3.1.
Next observe that
[Equation]
since, as 𝑠 ∉ S fix , either 𝜋 ′ ( 𝑠 ) is defined by Case 5a as 𝜋 ′ ( 𝑠 ) =
arg max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) and so ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 )) = max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) , or it is defined by Case 5b in which case ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 ))
=
ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ( 𝑠 )) ≥ max 𝑎 ∈A ˆ 𝑞 𝜋 ( 𝑠, 𝑎 ) − 2 𝜔 .
Combining Eqs.
(14) and (15), we obtain
[Equation]
where in the first line we added and subtracted ˆ 𝑞 𝜋 ( 𝑠, 𝜋 ′ ( 𝑠 )) , and the second inequality holds as ˆ 𝑞 𝜋 ( 𝑠, 𝑎 )
≈ 𝜔 𝑞 𝜋 ( 𝑠, 𝑎 ) for 𝑠 ∉ S fix and 𝑎 ∈ A by the assumptions of the lemma.
We continue by adding and subtracting max 𝑎 ∈A 𝑞
𝜋 ( 𝑠, 𝑎 ) :
[Equation]
where in the fifth line we used that 𝜋 is next-state Δ -optimal by assumption.
□
B Proof of Lemma 4.
For an episode trajectory { 𝑆 ℎ , 𝐴 ℎ , 𝑅 ℎ } ℎ ∈ N , let 𝐾 be the smallest positive integer such that 𝑆 𝐾 ∉ S ′ .
For any 𝑖 ∈ { 1 , . . . , 𝑛 } , let 𝐼 𝑖 denote the indicator of the event that at the 𝑖 th iteration of the outer loop of
Algorithm 2, the algorithm encounters 𝑆 ∉ S ′ in Line 6.
Note that E 𝜋,𝑠,𝑎 [ 𝐼 𝑖 ]
= P 𝜋,𝑠,𝑎 [ 1 ≤ 𝐾 < 𝐻 ] .
Then, by Hoeffding’s inequality (see, e.g., Lattimore and Szepesvári [2020]), with probability at least 1 − 𝜁 / 2 ,
[Equation] [Equation]
M EASURE only returns success if all indicators are zero; therefore, the above inequality implies that if M EASURE returns success then, with probability at least 1 − 𝜁 / 2 , we have
[Equation]
[Equation]
Recall that if M EASURE returns ( success , ˜ 𝑞 ) , then ¯ 𝑞 = 1 𝑛 (cid:205)
𝑛𝑖 = 1 (cid:205)
𝐻 − 1 ℎ =
𝛾 ℎ 𝑅 𝑖,ℎ .
Since
[Equation]
another application of Hoeffding’s inequality yields that 𝑞 𝜋 ( 𝑠, 𝑎 ) and ¯ 𝑞 are close with high probability: with probability at least 1 − 𝜁 / 2 ,
[Equation]
where we also used that the range of the sum of the rewards above for every 𝑖 is [ 0 , 1 /( 1 − 𝛾 )] .
Pick any 𝜋 ′ ∈ Π 𝜋, S ′ .
Observe that for any 𝑠 ∈ S and 𝑎 ∈ A , the distribution of the trajectory 𝑆 0 , 𝐴 0 , 𝑅 0 , 𝑆 1 , 𝐴 1 , 𝑅 1 , . . .
, 𝐴 𝐾 − 1 , 𝑅 𝐾 − 1 , 𝑆 𝐾 is the same under P 𝜋 ′ ,𝑠,𝑎 and P 𝜋,𝑠,𝑎 , as 𝜋 and 𝜋 ′ select the same actions for states in S ′ .
By Eqs.
(3) to (4), we can write (cid:12)
[Equation]
Combining Eqs.
(16) to (18)
[Equation]
it follows by the union bound that if M EASURE returns with ( success , ˜ 𝑞 ) , then with probability at least 1 − 𝜁 ,
[Equation]
□
C Proof of Lemma 4.
We start the proof by showing that there exists a 𝜃 ∈ R 𝑑 such that ∥ 𝜃 ∥ 2 ≤ 𝐵 and for all 𝑠 ∈ S and 𝑎 ∈ A , 𝑞 𝜋 ( 𝑠, 𝑎 ) ≈ 𝜀 ⟨ 𝜃, 𝜑 ( 𝑠, 𝑎 )⟩ .
For any finite set 𝑊 ⊆ S × A , max ( 𝑠,𝑎 )∈
𝑊 | 𝑞 𝜋 ( 𝑠, 𝑎 ) − ⟨ 𝜑 ( 𝑠, 𝑎 ) , 𝜃 ′ ⟩ | is a continuous function of 𝜃 ′ , hence it attains its infimum on the compact set { 𝜃 ′ ∈ R 𝑑 : ∥ 𝜃 ′ ∥ 2 ≤ 𝐵 } .
By Definition 1.1, this infimum is at most 𝜀 .
Therefore, the compact sets Θ 𝑠,𝑎 = { 𝜃 ′ ∈ R 𝑑 : ∥ 𝜃 ′ ∥ 2 ≤ 𝐵 and | 𝑞 𝜋 ( 𝑠, 𝑎 ) − ⟨ 𝜑 ( 𝑠, 𝑎 ) , 𝜃 ′ ⟩ | ≤ 𝜀 } are non-empty for all ( 𝑠, 𝑎 ) ∈ S × A , and any intersection of a finite collection of these sets is also non-empty.
Therefore, (cid:209) ( 𝑠,𝑎 )∈S×A Θ 𝑠,𝑎 is non-empty by [Rudin et al., 1976,
Theorem 2.36], and any element 𝜃 of this set satisfies Eq.
For the remainder of this proof, fix such a 𝜃 .
For any 𝑖 ∈ [ 𝑛 ] , with a slight abuse of notation, we introduce the shorthand 𝜑 𝑖
= 𝜑 ( 𝑠 𝑖 , 𝑎 𝑖 ) , and let ˆ 𝑞 𝑖 = ⟨ 𝜃, 𝜑 𝑖 ⟩ and 𝜉 𝑖 = ¯
𝑞 𝑖 − ˆ 𝑞 𝑖 .
Note that by the triangle inequality, | 𝜉 𝑖 | ≤ | ¯ 𝑞 𝑖 − 𝑞 𝜋 ( 𝑠 𝑖 , 𝑎 𝑖 )| + | 𝑞 𝜋 ( 𝑠 𝑖 , 𝑎 𝑖 ) − ˆ 𝑞 𝑖 | ≤ 𝜔 + 𝜀 .
Let ¯ 𝜃 = 𝑉 ( 𝐶 ) − 1 (cid:205)
𝑖 ∈[ 𝑛 ] 𝜑 𝑖 ¯ 𝑞 𝑖 and ˆ 𝜃 = 𝑉 ( 𝐶 ) − 1 (cid:205)
𝑖 ∈[ 𝑛 ] 𝜑 𝑖 ˆ 𝑞 𝑖 .
For any 𝑣 ∈ R 𝑑 by the Cauchy-Schwarz inequality,
[Equation]
(cid:11)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:10) ¯ 𝜃
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:42)
[Equation]
(cid:43)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
To bound the first term on the right-hand side above, observe that
[Equation]
where in the last line we used that 𝑉 ( 𝐶 ) ⪰ 𝜆 I .
The second term can be bounded as
[Equation]
where the first inequality holds by the triangle inequality, the second by our bound on | 𝜉 𝑖 | , the third by the Cauchy-Schwartz inequality, and the fourth by the positivity of 𝜆 .
Putting it all together, for any 𝑠 ∈ S and 𝑎 ∈ A , using the previous bounds with 𝑣 = 𝜑 ( 𝑠, 𝑎 ) ,
[Equation]
completing the proof.
□
D Deriving next-state optimality of 𝜋 ℓ for Lemma 4.
Lemma D.1.
Assume that Eq.
(6) holds whenever M EASURE returns success .
At any point of
CAPI-Q PI -P LAN after Line 16 is executed, for any 𝜋 ′′ ∈ Π 𝜋 ℓ , Cover ( 𝐶 ℓ ) , 𝑠 ∈ Cover ( 𝐶 ℓ ) , and 𝑎 ∈ A ,
[Equation]
Proof.
By Lemma 4.6 and Eq.
(6), ¯ 𝑞 𝑙,𝑚 ≈ 𝜔 𝑞 𝜋 ′′ ( 𝐶 𝑙,𝑚 ) for all 𝑚 ∈ [| 𝐶 ℓ |] (recall that 𝐶 𝑙,𝑚 is the 𝑚 th state-action pair in 𝐶 𝑙 ).
Therefore, applying Lemma 4.3 with 𝑞 𝜋 ′′ , 𝐶 ℓ and ¯ 𝑞 ℓ , as ˆ 𝑞 = LSE 𝐶 ℓ , ¯ 𝑞 ℓ , we get that for any 𝑠 ∈ Cover ( 𝐶 ℓ ) and all 𝑎 ∈ A ,
[Equation]
where the second inequality holds because ∥ 𝜑 ( 𝑠, 𝑎 )
∥ 𝑉 ( 𝐶 ℓ ) − 1 ≤ 1 since 𝑠 ∈ Cover ( 𝐶 ℓ ) , | 𝐶 ℓ | ≤ ˜ 𝑑 by
Eq.
(13), and the definition of 𝜆 .
□
Lemma D.2.
Assume that Eq.
(6) holds whenever M EASURE returns success .
Consider a time when
Lines 17 to 20 of CAPI-Q PI -P LAN are run and assume that at this time, for all 𝑙 ∈ [ 𝐻 + 1 ] , 𝜋 𝑙 is Δ 𝑙 -optimal on Cover ( 𝐶 𝑙 ) .
Then, 𝜋 ℓ is next-state ( Δ ℓ + 4 ( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 )/ 𝛾 ) -optimal on Cover ( 𝐶 ℓ ) .
Proof.
Let 𝜋 + ℓ be defined as in Eq.
As 𝜋 + ℓ ∈ Π 𝜋 ℓ , Cover ( 𝐶 ℓ ) , by Lemma D.1, for any 𝑠 ∈ Cover ( 𝐶 ℓ ) and all 𝑎 ∈ A ,
[Equation]
Similarly, applying Lemma D.1 with 𝜋 ℓ (which trivially belongs to Π 𝜋 ℓ , Cover ( 𝐶 ℓ ) ), we also have | ˆ 𝑞 ( 𝑠, 𝑎 ) − 𝑞 𝜋 ℓ ( 𝑠, 𝑎 )| ≤
[Equation]
𝜔
[Equation]
𝜀
[Equation]
√︁ ˜ 𝑑
[Equation]
Therefore,
[Equation] [Equation]
Since 𝜋 ℓ is Δ ℓ -optimal on Cover ( 𝐶 ℓ ) by assumption, this makes 𝜋 + ℓ Δ -optimal on Cover ( 𝐶 ℓ ) for
[Equation]
[Equation]
For a trajectory in the MDP, let the random variable 𝜏 be the first time the state is in Cover ( 𝐶 ℓ ) :
[Equation]
Since 𝜋 + ℓ agrees with 𝜋 ★ on states not in Cover ( 𝐶 ℓ ) , the distribution of the trajectory up to and including 𝑆 𝜏 is the same under both policies, starting from any state 𝑠 ∈ S .
Therefore, for any 𝑠 ∈ S ,
[Equation]
as 𝛾 𝜏 ≤ 1 and 𝜋 + ℓ is Δ -optimal on Cover ( 𝐶 ℓ ) .
That is, 𝜋 + ℓ is also Δ -optimal on S (with Δ defined in
Eq. 21).
Using this, for any 𝑠 ∈ Cover ( 𝐶 ℓ )
[Equation]
and 𝑎 ∈ A , we have ∫
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the third inequality holds by Eq.
Therefore 𝜋 ℓ is next-state ( Δ ℓ + 4 ( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 )/ 𝛾 )) - optimal on Cover ( 𝐶 ℓ ) .
□
E Poof of Lemma 4.
Proof of Lemma 4.4.
We prove by induction on the iterations of the main loop of CAPI-Q PI -P LAN the inductive hypothesis: at the start of iteration 𝑖 , for all 𝑙 ∈
[ 𝐻 + 1 ] , 𝜋 𝑙 is Δ 𝑙 -optimal on Cover ( 𝐶 𝑙 ) .
We first observe that after initialization, 𝐶 𝑙 is the empty sequence for every 𝑙 , so we can apply
Lemma 4.3 with 𝑞 ★ and empty sequences ( 𝑛 = 0 ) to get that for any 𝑠 ∈ Cover (()) and 𝑎 ∈ A , 𝑞 ★ ( 𝑠, 𝑎 ) ≤ 𝜀 + √ 𝜆𝐵 = 𝜀 + 𝜔 .
Then, 𝑣 ★ ( 𝑠 ) ≤ 𝜀 + 𝜔 ≤ Δ 𝑙 .
Therefore, at initialization, any policy is Δ 𝑙 -optimal on Cover ( 𝐶 𝑙 ) for any 𝑙 ∈
[ 𝐻 + 1 ] .
Assuming that the inductive hypothesis holds at the start of some iteration, it is left to prove that it continues to hold at the end of the iteration (assuming Eq.
(6) holds whenever M EASURE returns success ); this implies that the hypothesis also holds at the start of the next iteration and hence also proves the lemma.
For any ( 𝑠, 𝑎 ) appended to 𝐶 0 , the inductive hypothesis trivially continues to hold as Δ 0 = 1 /( 1 − 𝛾 ) ≥
𝑣 ★ ( 𝑠 ) for any 𝑠 ∈ S because the rewards are bounded in [ 0 , 1 ] .
The only other case in which 𝐶 𝑙 or 𝜋 𝑙 changes for any 𝑙 is in Lines 18 and 20, where the changes happen only for 𝑙 = ℓ + 1 .
We will use Lemma 3.4 to analyze the effect of these updates, thus next we show that the conditions of the lemma are satisfied:
(a) In Lemma D.
we show that 𝜋 ℓ is next-state ( Δ ℓ + 4 ( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 )/ 𝛾 ) -optimal on Cover ( 𝐶 ℓ ) .
In the proof of the lemma, we introduce a policy in Eq.
(22) that acts as 𝜋 ℓ on states in Cover ( 𝐶 ℓ ) , and as an optimal stationary deterministic memoryless policy 𝜋 ★ otherwise:
[Equation]
Intuitively, this policy corrects 𝜋 ℓ on the low-confidence states.
The proof of Lemma D.2 then uses the fact that this policy is also 𝑞 𝜋 -realizable
(Definition 1.1) and satisfies 𝜋 + ℓ ∈ Π 𝜋 ℓ , Cover ( 𝐶 ℓ ) to show (i) that the 𝑞 -values of 𝜋 ℓ and 𝜋 + ℓ are close on the measured state-action pairs (via Lemma 4.
and Lemma D.1); (ii) an optimality guarantee on 𝜋 + ℓ for all 𝑠 ∈ S ; and, as a consequence, (iii) the next-state optimality of 𝜋 ℓ .
(b) Next, to analyze the effect of Line 18, we introduce hypothetical 𝑞 -approximators ˜ 𝑞 𝑙 for 𝑙 ∈
[ 𝐻 + 1 ] , defined as follows:
At initialization, ˜ 𝑞 𝑙 ( 𝑠, 𝑎 ) = 0 for all 𝑙 ∈ [ 𝐻 + 1 ] , 𝑠 ∈ S , and 𝑎 ∈ A .
It is updated every time after Line 16 of the algorithm is executed as
[Equation]
In other words, ˜ 𝑞 ℓ is only updated to the newly computed ˆ 𝑞 for states that are not in Cover ( 𝐶 ℓ + 1 ) , and stays unchanged for other states.
We show in Lemma F.2 that the new policy that 𝜋 ℓ + 1 is updated to, which is constructed in two steps (Lines 17–18), can be expressed as the result of a single CAPI policy update that uses ˜ 𝑞 :
[Equation]
We show in Lemma F.1 that ˜ 𝑞
ℓ ≈ 𝜔 ′ 𝑞 𝜋 ℓ with 𝜔 ′ =
( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 ) on Cover ( 𝐶 ℓ ) .
By the above, we can apply Lemma 3.4 with policy 𝜋 ℓ , 𝑞 -approximation ˜ 𝑞 ℓ (with approximation error guarantee 𝜔 ′ on Cover ( 𝐶 ℓ ) , and S fix =
S \ Cover ( 𝐶 ℓ ) to get that the new value of 𝜋 ℓ + 1 is a Δ ℓ + 1 = ( 8 ( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 ) + 𝛾 Δ ℓ ) -optimal policy on Cover ( 𝐶 ℓ ) .
By the end of the loop in Line 20,
Cover ( 𝐶 ℓ + 1 ) =
Cover ( 𝐶 ℓ ) , so 𝜋 ℓ + 1 is Δ ℓ + 1 -optimal on Cover ( 𝐶 ℓ + 1 ) .
This finishes the proof that the inductive hypothesis continues to hold at the end of the iteration, finishing the proof of the lemma.
□
F Auxiliary results for Lemma 4.4 about ˜ 𝑞 𝑙
Throughout the execution of CAPI-Q PI -P LAN , for 𝑙 ∈ [ 𝐻 + 1 ] , let ˜ 𝑞 − 𝑙 , 𝜋 − 𝑙 , 𝐶 − 𝑙 denote the values of variables ˜ 𝑞 ℓ , 𝜋 ℓ , 𝐶 ℓ , respectively, at the time when Lines 16–20 were most recently executed with ℓ = 𝑙 in a previous iteration of the main loop of CAPI-Q PI -P LAN .
If such a time does not exist, let their values be the initialization values.
Thus, 𝐶 − 𝑙 may (only) change at the start of some iteration 𝑖 if Lines 16–20 were executed with ℓ = 𝑙 in the previous iteration 𝑖 − 1 .
Observe that whenever this happens, Lines 16–20 may also change 𝐶 ℓ + 1 in iteration 𝑖 − 1 , and this is the only time 𝐶 𝑙 + 1 can be changed for any 𝑙 ∈ [ 𝐻 ] .
After this, at the beginning of iteration 𝑖 , 𝐶 𝑙 + 1 always has the same elements as 𝐶 − 𝑙 .
Therefore, since it also holds at the initialization of the algorithm, we conclude that at the start of each iteration,
[Equation]
Lemma F.1.
Assume that Eq.
(6) holds whenever M EASURE returns success .
Then, whenever
Line 18 of CAPI-Q PI -P LAN is executed, for all 𝑠 ∈ Cover ( 𝐶 ℓ ) and 𝑎 ∈ A , (cid:12) (cid:12)(cid:12) ˜ 𝑞 ℓ ( 𝑠, 𝑎 ) − 𝑞 𝜋 ′′
( 𝑠, 𝑎 ) (cid:12) (cid:12)(cid:12) ≤ ( 𝜔 + 𝜀 )( √︁ ˜ 𝑑 + 1 ) for all 𝜋 ′′ ∈ Π 𝜋 ℓ ,
[Equation]
Proof.
We prove this by induction for every time Line 18 is executed with any value of ℓ .
We first observe that after initialization, 𝐶 𝑙 is the empty sequence for every 𝑙 , so we can apply Lemma 4.
with 𝑞 ★ and empty sequences ( 𝑛 = 0 ) to get that for any 𝑠 ∈ Cover (()) and 𝑎 ∈ A , 𝑞 𝜋 ′′ ( 𝑠, 𝑎 ) ≤ 𝑞 ★ ( 𝑠, 𝑎 ) ≤ 𝜀 + √ 𝜆𝐵 = 𝜀 + 𝜔 .
Also, ˜ 𝑞 𝑙 (· , ·) = 0 at initialization, so Eq.
(25) holds for any value of ℓ .
Consider a time when Line 18 is executed and assume the inductive hypothesis holds for the previous time Line 18 was executed with the same value of ℓ (or at the initialization if this is the first time), that is,
[Equation]
To prove that the statement now holds for any 𝑠 ∈ Cover ( 𝐶 ℓ ) , first consider any 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) =
Cover ( 𝐶 − ℓ ) .
For such an 𝑠 , by Lemma 4.5 we have that Π 𝜋 ℓ , Cover ( 𝐶 ℓ ) ⊆
Π 𝜋 − ℓ , Cover ( 𝐶 − ℓ ) .
Also, by definition, ˜ 𝑞 ℓ ( 𝑠, ·) =
˜ 𝑞 − ℓ ( 𝑠, ·) for 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) .
Combining with the inductive hypothesis, it follows that Eq.
(25) holds for 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) .
It remains to show that Eq.
(25) also holds for 𝑠 ∈ Cover ( 𝐶 ℓ ) \ Cover ( 𝐶 ℓ + 1 ) .
For such an 𝑠 , ˜ 𝑞 ℓ ( 𝑠, ·) = ˆ 𝑞 ( 𝑠, ·) by definition, and hence Lemma D.1 implies that Eq.
(25) holds in this case.
Combining the two cases, it follows that the inductive hypothesis continues to hold when Line 18 is executed.
□
Lemma F.2.
Throughout the execution of CAPI-Q PI -P LAN , at the start of any iteration, for all 𝑙 ∈ [ 𝐻 ] ,
[Equation]
Proof.
We prove this by induction for the start of any iteration.
Eq.
(26) holds at the start of the algorithm due to its initialization (because at initialiaztion, ˜
𝑞 − 𝑙 ( 𝑠, 𝑎 ) =
for all 𝑠, 𝑎 , and hence by our tie-breaking rule, the policy on the right-hand side of Eq.
(26) always chooses action A 1 , which is the initial policy for 𝜋 𝑙 ).
In what follows, we use the fact that for any 𝑞 : S × A → R , policy 𝜋 , and S fix ⊆ S , the CAPI policy update 𝜋 𝑞,𝜋, S fix is a policy whose value at any 𝑠 ∈ S only depends on 𝑞 ( 𝑠, ·) , 𝜋 ( 𝑠 ) , and whether or not 𝑠 ∈ S fix , by definition (Eq. 5).
Therefore, for an alternative 𝑞 ′ , 𝜋 ′ , S ′ fix , for any 𝑠 ∈ S , 𝜋 𝑞,𝜋, S fix ( 𝑠 ) =
𝜋 𝑞 ′ ,𝜋 ′ , S ′ fix ( 𝑠 ) whenever the following three conditions hold: (C1)
𝑞 ( 𝑠, 𝑎 ) =
𝑞 ′ ( 𝑠, 𝑎 ) for all 𝑎 ∈ A ; (C2) 𝜋 ( 𝑠 ) =
𝜋 ′ ( 𝑠 ) ; and (C3) either both or none of S fix and S ′ fix include 𝑠 .
Assume the inductive hypothesis holds at the beginning of some iteration.
Let 𝜋 ′′ be the policy
Line 18 updates 𝜋 ℓ + 1 to, noting that this is the only place where policies are updated.
All we need to prove is that 𝜋 ′′ is equal to ˜ 𝜋
= 𝜋 ˜ 𝑞 ℓ ,𝜋 ℓ , S\ Cover ( 𝐶 ℓ ) .
First, for any 𝑠 ∉ Cover ( 𝐶 ℓ + 1 ) , 𝜋 ′′ ( 𝑠 ) =
𝜋 ′ ( 𝑠 )
=
𝜋 ˆ 𝑞,𝜋 ℓ , S\ Cover ( 𝐶 ℓ ) ( 𝑠 ) and ˆ 𝑞 ( 𝑠, ·) =
˜ 𝑞 ℓ ( 𝑠, ·) by definition.
Hence, 𝜋 ′′ ( 𝑠 ) =
𝜋 ˆ 𝑞,𝜋 ℓ , S\ Cover ( 𝐶 ℓ ) ( 𝑠 ) = 𝜋 ˜ 𝑞 ℓ ,𝜋 ℓ , S\ Cover ( 𝐶 ℓ ) ( 𝑠 ) =
˜ 𝜋
( 𝑠 ) , as all of conditions (C1)-(C3) are satisfied for 𝑠 (C2 and C3 hold trivially).
Next, take any 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) =
Cover ( 𝐶 − ℓ ) .
Then, by Line 18, 𝜋 ′′ ( 𝑠 ) = 𝜋 ℓ + 1 ( 𝑠 ) .
By the inductive hypothesis, the current value of 𝜋 ℓ + 1 can be written as 𝜋 ˜ 𝑞 − ℓ ,𝜋 − ℓ , S\ Cover ( 𝐶 − ℓ ) .
We prove that this policy takes the same value as ˜ 𝜋 at 𝑠 , by showing conditions (C1)-(C3).
First, by Lemma 4.5, 𝜋 ℓ ∈ Π 𝜋 − ℓ , Cover ( 𝐶 − ℓ ) .
Thus, as 𝑠 ∈ Cover ( 𝐶 − ℓ ) , 𝜋 ℓ ( 𝑠 ) =
𝜋 − ℓ ( 𝑠 ) , showing condition (C2).
Furthermore, as 𝑠 ∈ Cover ( 𝐶 ℓ + 1 ) , by definition, ˜ 𝑞 ℓ ( 𝑠, ·) =
˜ 𝑞 − ℓ ( 𝑠, ·) , showing condition (C1).
Finally, as 𝑠 ∈
Cover ( 𝐶 ℓ + 1 ) =
Cover ( 𝐶 − ℓ ) ⊆
Cover ( 𝐶 ℓ ) , 𝑠 ∉ S \ Cover ( 𝐶 − ℓ ) and 𝑠 ∉ S \ Cover ( 𝐶 ℓ ) , showing condition (C3).
Combining the two cases, 𝜋 ′′ ( 𝑠 )
=
˜ 𝜋
( 𝑠 ) for any 𝑠 ∈ S , finishing the induction.
□
G Efficient implementation and proof of Theorem 1.
In this section we consider the efficient implementation of CAPI-Q PI -P LAN in terms of memory and computational costs of both the algorithm itself and the final policy it outputs.
Focusing on the memory cost, first we can observe that throughout the execution of the algorithm, 𝐶 𝑙 for all 𝑙 ∈ [ 𝐻 + 1 ] only stores up to ˜ 𝑑 unique state-action pairs altogether (cf.
Eq.
(13)), as they use the same pairs; let 𝑊 =
( 𝑠 𝑖 , 𝑎 𝑖 )
𝑖 ∈
ˆ 𝑑 denote these for some ˆ 𝑑 ≤ ˜ 𝑑 .
Furthermore, throughout the execution of the algorithm, for any level 𝑙 , the only features that 𝜋 𝑙 depends on are the features associated with members of 𝑊 .
Storing all these features takes 𝑑 ˆ 𝑑 memory.
Denote all the policies that CAPI-Q PI -P LAN constructs in Line 18, in order, as 𝜋 ( 0 ) , 𝜋 ( 1 ) , . . .
, 𝜋 ( 𝑛 − 1 ) , where 𝑛 is the number of times Line 18 is executed.
Recall from the proof of Theorem 1.2 that the number of times
M EASURE returns success , which is an upper bounds on 𝑛 , is itself bounded by ˜ 𝑑𝐻 , hence 𝑛 ≤ ˜ 𝑑𝐻 .
Together, Lines 17-18 construct a policy that, for an 𝑠 ∈ S , decides whether the action should be arg max 𝑎 ∈A ⟨ 𝜑 ( 𝑠, 𝑎 ) , 𝜃 ⟩ for some 𝜃 given by LSE (Eq.
(8)), or the value of the policy should be determined by a recursive call to a previously constructed policy, either 𝜋 ℓ + 1 or 𝜋 ℓ (through 𝜋 ′ ).
Now there exist some 𝑎, 𝑏 ∈ [ 𝑛 ] such that 𝜋 ( 𝑎 ) =
𝜋 ℓ and 𝜋 ( 𝑏 ) =
𝜋 ℓ + 1 before the new policy is constructed in Line 18.
To implement the new 𝜋 ℓ + 1 constructed policy, it is enough therefore to store, in addition to the existing policies, 𝜃 (from ˆ 𝑞 ), the decision rules, and the indices 𝑎 and 𝑏 .
The decision rules are fully defined by 𝜃 , 𝐶 ℓ , and 𝐶 ℓ + 1 .
It is therefore enough to further store 𝐶 ℓ , 𝐶 ℓ + 1 ⊆
𝑊 , which can be encoded as ˆ 𝑑 -dimensional vectors each, storing the bitmask of which state-action pairs are included.
We also store the current value of ℓ (the level) for the newly constructed policy.
Together, a policy thus consumes 3 + 𝑑 + 2 ˆ 𝑑 memory.
We store all policies constructed, along with the features of 𝑊 , and the final value of 𝑉 ( 𝐶 𝐻 ) − 1 , at a memory cost of 𝑑 ˆ 𝑑 + ˜ 𝑑𝐻 ( 3 + 𝑑 + ˆ 𝑑 ) + 𝑑 2 =
˜ O
( 𝑑 2 /( 1 − 𝛾 )) .
This is the memory cost of the final policy outputted by CAPI-Q PI -P LAN .
The memory cost of running CAPI-Q PI -P LAN itself is of the same order, as additionally storing 𝐶 𝑙 , ¯ 𝑞 𝑙 , and 𝑉 ( 𝐶 𝑙 ) − 1 for 𝑙 ∈
[ 𝐻 + 1 ] takes ˜ O
( 𝑑 2 /( 1 − 𝛾 )) memory.
To efficiently implement the final policy found by CAPI-Q PI -P LAN with the stored information described above, we start from evaluating the last policy constructed, 𝜋 ( 𝑖 ) for 𝑖 =
𝑛 − 1 .
We introduce auxiliary variables ˜ 𝑉 ( 𝐶 𝑙 ) − 1 and ˜ 𝐶 𝑙 for 𝑙 ∈
[ 𝐻 + 1 ] to efficiently track the required values of 𝑉 ( 𝐶 𝑙 )
−and 𝐶 𝑙 .
We keep updating these variables so that for 𝑙 ∈ { ℓ, ℓ + 1 } , they match the values of 𝑉 ( 𝐶 𝑙 )
−and 𝐶 𝑙 , respectively, at the time of construction of the current policy 𝜋 ( 𝑖 ) under consideration, where ℓ is the (saved) level of 𝜋 ( 𝑖 ) .
For 𝑖 = 𝑛 − 1 , observe that when it was constructed, 𝐶
=
𝐶 1 = · · · = 𝐶 𝐻 by Lemma 4.7.
We therefore start by initializing variables ˜
𝑉 ( 𝐶 0 ) − 1 , . .
˜ 𝑉
( 𝐶 𝐻 ) − 1 to the saved final value of 𝑉 ( 𝐶 𝐻 ) − 1 , and variables ˜ 𝐶 0 , . . . ,
˜ 𝐶 𝐻 to 𝑊 .
Implementing the decisions of a policy takes an order of |A| 𝑑 2 computation ( |A| vector and matrix multiplications), after which we recover either the policy output or a previously constructed policy to recurse into.
For the latter case, we have to consider the evaluation of this policy, denoted by 𝜋 ( 𝑖 ′ ) .
Let the (saved) level of 𝜋 ( 𝑖 ′ ) be ℓ ′ .
Before we set 𝑖 to 𝑖 ′ and start evaluating it, we need to update the values of ˜ 𝑉 ( 𝐶 𝑙 ) and 𝐶 𝑙 for 𝑙 ∈ { ℓ ′ , ℓ ′ + 1 } .
The updates are needed for these two levels only, as the decision rule of policy 𝑖 ′ only depends on these levels, as shown before.
Let us describe the update procedure for some 𝑙 ∈ { ℓ ′ , ℓ ′ + 1 } : Since 𝜋 ( 𝑖 ′ ) was constructed earlier than 𝜋 ( 𝑖 ) (i.e., 𝑖 ′ < 𝑖 ), and 𝐶 𝑙 ′ can only grow during the algorithm for any 𝑙 ′ ∈
[ 𝐻 + 1 ] , we only need to remove members of the variable ˜ 𝐶 𝑙 to match the value of 𝐶 𝑙 at the time of construction of 𝜋 ( 𝑖 ′ ) .
The members to be removed are given by the difference of the members of ˜ 𝐶 𝑙 and the bitmasks stored for 𝜋 ( 𝑖 ′ ) for level 𝑙 .
For each state-action pair ( 𝑠, 𝑎 ) removed, we also need to update ˜ 𝑉 ( 𝐶 𝑙 ) − 1 to (cid:0)
˜ 𝑉
( 𝐶 𝑙 ) − 𝜑 ( 𝑠, 𝑎 ) 𝜑 ( 𝑠, 𝑎 ) ⊤ (cid:1) − 1 , which can be done in order 𝑑 2 computation using the Sherman–Morrison–Woodbury formula [Max, 1950].
The total number of such removal operations for any level 𝑙 is bounded by the sum of the number of state-action pairs in the initialization of ˜ 𝐶 𝑙 ′ (for 𝑙 ′ ∈
[ 𝐻 + 1 ] ), that is, by ( 𝐻 + 1 ) ˆ 𝑑 .
As a result, the computational cost of the final policy of CAPI-Q PI -P LAN is ˜ O
(( 𝐻 + 1 ) ˆ 𝑑𝑑 2 )
+ 𝑛 ˜
O(|A| 𝑑 2 )
=
˜ O
( 𝑑 3 |A|/( 1 − 𝛾 )) .
Finally, we consider the computational cost of running CAPI-Q PI -P LAN .
The number of iterations of the outer loop is bounded by ˜ O( 𝑑𝐻 ) =
˜ O
( 𝑑 /( 1 − 𝛾 )) , as each iteration involves either a M EASURE call that returns success , or a new member added to some 𝐶 𝑙 .
For each iteration, Line 4 takes ˜ O
( 𝑑 2 |A|) ,
Line 7 takes ˜ O
( 𝑑 /( 1 − 𝛾 )) , Line 11 takes ˜ O
( 𝑑 2 |A|) computation; for Line 16, calculating 𝜃 , the second component of the inner product of the least-squares predictor in Eq.
(8) takes ˜ O
( 𝑑 2 ) computation, and if 𝐶 𝑙 ever changes for some 𝑙 , updating 𝑉 ( 𝐶 𝑙 ) − 1 by the Sherman–Morrison–Woodbury takes ˜ O
( 𝑑 2 ) computation.
Overall, all the operations except those associated to the M EASURE call of
Line 9 take ˜ O
( 𝑑 3 |A|/( 1 − 𝛾 )) computation in total.
We conclude our calculations by considering the computational cost of the M EASURE calls, which will dominate the overall computational cost.
Line 6 of Algorithm 2 has a computational cost of order 𝑑 2 |A| , while the majority of the computational cost comes from evaluating the policy at Line 7.
By our previous calculations, this takes ˜ O
( 𝑑 3 |A|/( 1 − 𝛾 )) computation and happens (at most) once for each simulator call.
Using the query cost bound of Theorem 1.2, we conclude that the computational cost of CAPI-Q PI -P LAN is ˜
O( 𝑑 4 |A|( 1 − 𝛾 ) − 5 𝜔 − 2 ) .
□
H Query cost lower bounds with random access
In this section we prove lower bounds on the worst-case expected query cost of planning algorithms with a simulator supporting random access .
Recall from Section 1 that in this setting a planner can issue queries for any state-action pair, not just the ones already visited.
As this is a more powerful access to the simulator than local access , statements that hold for all planners using random access (as such, all lower bounds presented in this section) trivially hold for planners using local access .
We prove two bounds, Theorem H.2 and Theorem H.3, whose combination trivially implies Theorem 1.4.
Formally, the planner interacts with a random access simulator that simulates some MDP 𝑀 as follows:
at step 𝑡 starting from 1 , given the whole interaction history 𝐻 𝑡
=
( 𝑆 1 , 𝐴 1 , 𝑅 1 , 𝑆 ′ 1 , . . .
, 𝑆 𝑡 − 1 , 𝐴 𝑡 − 1 , 𝑅 𝑡 − 1 , 𝑆 ′ 𝑡 − 1 ) (where 𝐻 1 is the empty sequence by definition), the planner either selects a state-action pair ( 𝑆 𝑡 , 𝐴 𝑡 ) , or halts and outputs a stationary memoryless policy.
The planner is allowed to randomize.
Let 𝜏 denote the number of queries the planner sends to the simulator before it halts, and 𝜋 𝜏 the policy it outputs.
If the planner does not stop, the simulator responds to the query ( 𝑆 𝑡 , 𝐴 𝑡 ) by returning ( 𝑆 ′ 𝑡 , 𝑅 𝑡 ) sampled independently from the transition-reward kernel Q( 𝑆 𝑡 , 𝐴 𝑡 ) of 𝑀 .
Let P 𝑀 denote the probability measure associated with this procedure, and let E 𝑀 denote the expectation operator corresponding to P 𝑀 .
Both P 𝑀 and E 𝑀 implicitly depend on the planner, which is omitted in the notation for brevity but will always be clear from the context.
Using this notation, clearly E 𝑀 ( 𝜏 ) is the expected query cost of the planner on 𝑀 .
As usual, we only consider the query complexity of planners which are reasonable in the sense that they can find a near-optimal policies for a class of MDPs:
Definition H.1 (Soundness and query complexity) .
A planner is said to be ( 𝛼, 𝛿 ) -sound for an MDP 𝑀 if, when used with a simulator of 𝑀 , it halts almost surely (i.e., P 𝑀 ( 𝜏 < ∞)
= 1 ) and outputs a policy 𝜋 𝜏 that is 𝛼 -optimal for 𝑀 with probability at least 1 − 𝛿 , that is,
[Equation]
where 𝑣 ★ and 𝑣 𝜋 𝜏 are the value-functions of the optimal policy and 𝜋 𝜏 in the MDP 𝑀 and 𝑠 0 is the initial state of 𝑀 .
A planner is ( 𝛼, 𝛿 ) -sound for a class of MDPs M if it is ( 𝛼, 𝛿 ) -sound for every MDP in the class.
The query complexity of a planner over M is defined as the maximum of its expected query cost over the members of the class.
In the rest of the section, for 𝑑 ≥ 1 and 𝐿 > 0 , we use B 𝑑 ( 𝐿 ) = { 𝑥 ∈ R 𝑑 :
∥ 𝑥 ∥ ≤ 𝐿 } to denote the 𝑑 -dimensional Euclidean ball of radius 𝐿 centered at the origin.
H.1 Exponential lower bound for planners with small suboptimality
We first show an exponential query complexity lower bound for sound planners that guarantee a small suboptimality bound.
The result is a simple application of the techniques in Lattimore et al.
[2020], and establishes the barrier for the suboptimality attainable by query-efficient planners:
Theorem H.2.
Let 𝛿 ≤ 0 .
9 , 𝛼 ≤ 0 .
49 /( 1 − 𝛾 ) , and 𝜀 ≥ 0 , 𝑑 ≥ 3 .
There is a class of MDPs M with uniform policy value-function approximation error 𝜀 for some 𝑑 -dimensional feature map such that the query complexity of any ( 𝛼, 𝛿 ) -sound planner over M is at least exp (cid:16) Ω
[Equation]
Proof.
Our proof is based on a similar complexity lower bound of Lattimore et al.
[2020] for the multi-armed bandit setting, which is a special case of our problem.
As such, we start by rewriting the class of bandit problems they used in their proof in our MDP framework, introducing a set of MDPs ˜
M each of which gets into a terminal state with no rewards after the first step.
Let 𝛼 ′
= 2 .
01 𝛼 ( 1 − 𝛾 ) ≤ 1 and 𝑘 = (cid:106) exp (cid:16) 𝑑 − 28 (cid:0) 𝜀𝛼 ′ (cid:1) 2 (cid:17)(cid:107) .
˜
M = { ˜ 𝑀 1 , . . . ,
˜ 𝑀 𝑘 } is defined to be a set of 𝑘 MDPs as follows:
Each MDP in ˜ M has 𝑘 actions (i.e., A
=
[ 𝑘 ] ) and two states: S =
( 𝑠 0 , 𝑠 1 ) with 𝑠 0 being the initial state, and deterministic transitions 𝑃 ( 𝑠 1 | 𝑠, 𝑎 ) = 1 and 𝑃 ( 𝑠 0 | 𝑠, 𝑎 ) =
for all ( 𝑠, 𝑎 ) ∈ S × A .
For any 𝑖 ∈ [ 𝑘 ] , the reward distribution R 𝑖 for MDP ˜
𝑀 𝑖 is defined as follows: rewards for state 𝑠 1 are deterministically zero, that is, R 𝑖
( 0 | 𝑠 1 , 𝑎 ) = 1 for all 𝑎 ∈ A , making 𝑠 1 an absorbing state with zero reward, while rewards for state 𝑠 0 are deterministically 𝛼 ′ for action 𝑖 and zero otherwise, that is, R 𝑖
( 𝛼 ′ | 𝑠 0 , 𝑖 )
= 1 and R 𝑖 ( 0 | 𝑠 0 , 𝑗 )
= 1 for 𝑗 ∈
[A] with 𝑗 ≠ 𝑖 .
Since this class of MDPs is equivalent to the class of muti-armed bandit problems defined by Lattimore et al.
[2020], their proof of Corollary 3.3 implies that
• there exists a feature map ˜ 𝜑 :
S ×A → B 𝑑 − 1 ( 1 ) such that 𝜀 is the maximum uniform policy value-function approximation error (Definition 1.1) over ˜ M equipped with features ˜ 𝜑 ; and • any planner that almost surely outputs an 𝛼 ′ -optimal deterministic policy for all ˜ 𝑀 ∈
˜ M (when run with a random access simulator for ˜ 𝑀 ) executes at least
[Equation]
queries in expectation.
We construct a new set M = { 𝑀 1 , . . .
, 𝑀 𝑘 } of 𝑘 MDPs where for each 𝑖 ∈ [ 𝑘 ] , 𝑀 𝑖 is a slight modification of ˜
𝑀 𝑖 , always returning to the initial state 𝑠 0 instead of stopping after the first step: as such, the only modification is that the transition probabilities for all 𝑀 ∈ M are 𝑃 ( 𝑠
| 𝑠, 𝑎 ) = 1 and 𝑃 ( 𝑠 1 | 𝑠, 𝑎 ) =
for all ( 𝑠, 𝑎 ) ∈ S × A .
Let 𝜑 : S ×
A → B 𝑑 ( 2 ) be the features for all MDPs in M , where for all ( 𝑠, 𝑎 ) ∈ S × A , 𝜑 ( 𝑠, 𝑎 ) is a concatenation of the ( 𝑑 − 1 )
-dimensional
˜ 𝜑 ( 𝑠, 𝑎 ) and the scalar 1 , so that the 𝑑 th coordinate of 𝜑 ( 𝑠, 𝑎 ) is 𝜑 ( 𝑠, 𝑎 ) 𝑑 = 1 .
Fix any 𝑖 ∈
[ 𝑘 ] and any stationary deterministic memoryless policy 𝜋 , and let ˜ 𝜃 be the parameter realizing the low approximation error for ˜
𝑀 𝑖 and ˜ 𝜑 , that is, satisfying Eq.
(19) (see Appendix C for a proof that such a ˜ 𝜑 exists).
In what follows, we denote 𝑞 - and 𝑣 -functions (with arbitrary superscripts) of an MDP 𝑀 by adding 𝑀 as a superscript to the corresponding function.
Let 𝜃 be a concatenation of ˜ 𝜃 and the scalar 𝛾𝑣 𝜋𝑀 𝑖 ( 𝑠 0 ) .
For any ( 𝑠, 𝑎 ) ∈ S × A ,
[Equation]
The uniform policy value-function approximation error therefore remains at most 𝜀 for 𝑀 𝑖 with feature map 𝜑 , and this is true for any 𝑖 ∈
[ 𝑘 ] .
We can therefore take any ( 𝛼, 𝛿 ) -sound planner with query complexity 𝑇 (for some 𝑇 ≥ 0 ) over M , and provide it with a simulator of 𝑀 𝑖 for any 𝑖 ∈ [ 𝑘 ] (which we can trivially build with access to a simulator of ˜ 𝑀 𝑖 ), to get a policy 𝜋 that is 𝛼 -optimal for 𝑀 𝑖 with P 𝑀 𝑖 -probability at least 1 − 𝛿 .
Recall that the rewards of 𝑀 𝑖 arefor every action apart from a single optimal action, 𝑖 , where the reward is 𝛼 ′ .
Thus, 𝑣 ★𝑀 𝑖 ( 𝑠 0 )
=
𝛼 ′ /( 1 − 𝛾 ) and 𝑣 𝜋𝑀 𝑖 ( 𝑠 0 ) =
𝛼 ′ 𝜋 ( 𝑖 | 𝑠 0 )/( 1 − 𝛾 )
=
𝜋 ( 𝑖 | 𝑠 0 )
𝑣 ★𝑀 𝑖 ( 𝑠 0 ) .
Thus, with probability at least 1 − 𝛿 , 𝑣 ★𝑀 𝑖 ( 𝑠 0 ) − 𝑣 𝜋𝑀 𝑖 ( 𝑠 0 ) ≤ 𝛼 < 0 .
5 𝛼 ′ /( 1 − 𝛾 )
= 0 .
5 𝑣 ★𝑀 𝑖
( 𝑠 0 ) .
Therefore, 𝜋 ( 𝑖 | 𝑠 0 ) > 0 . 5 .
As we know that the optimal action achieves a deterministic reward of 𝛼 ′ , we can test with a single query whether the action that 𝜋 assigns the highest probability to is optimal.
If not, we can run the planner again and repeat the check.
Since each run of the planner is successful with probability at least 1 − 𝛿 , independently of each other, almost surely one of the checks eventually passes and we output the deterministic policy that chooses the optimal action.
Now the number of times the planner needs to be run is a stopping time (with respect to the sequence of the runs) with expectation at most 1 /( 1 − 𝛿 ) , hence the expected query cost of the whole procedure is at most ( 𝑇 + 1 )/( 1 − 𝛿 ) by Wald’s equation.
Note that the same policy is 𝛼 ′ -optimal for ˜ 𝑀 𝑖 .
Therefore, the planner defined above almost surely outputs an 𝛼 ′ -optimal deterministic policy for any MDP in ˜ M , and hence by Eq.
(27) we have
[Equation]
Therefore 𝑇 = exp (cid:16) Ω (cid:0) 𝑑 (cid:0)
𝜀 𝛼 ( 1 − 𝛾 ) (cid:1) 2 (cid:1)(cid:17) , finishing the proof.
□
H
.2 Lower bound for linear MDPs
We close this section by proving a lower bounds on the query complexity of random access planners for linear MDPs (c.f. Theorem H.3).
We start by recalling the definition of linear MDPs [Zanette et al., 2020]: An MDP with countable state space is said to be linear if there exists a feature map 𝜑 : S ×
A → B 𝑑 ( 𝐿 ) , a state-transition feature map 𝜓 : S → R 𝑑 , and a reward parameter 𝜃 𝑟 ∈ B 𝑑 ( 𝐵 ) such that 𝑟 ( 𝑠, 𝑎 ) = ⟨ 𝜑 ( 𝑠, 𝑎 ) , 𝜃 𝑟 ⟩ and 𝑃 ( 𝑠 ′ | 𝑠, 𝑎 ) =
⟨ 𝜑 ( 𝑠, 𝑎 ) , 𝜓 ( 𝑠 ′ )⟩ for any ( 𝑠, 𝑎, 𝑠 ′ )
∈ S ×
A × S , and (cid:205) 𝑠 ∈S
∥ 𝜓 ( 𝑠 )∥ 2 ≤ 𝐵 .
Clearly, any linear MDP satisfies Definition 1.1 with 𝜀 = 0 .
As such, the lower bounds presented below trivially transfer to the 𝜀 uniform policy value-function approximation error case for any 𝜀 ≥ 0 .
Theorem H.3.
Let 𝛿 ∈ ( 0 , 0 . 08 ] , 𝛾 ∈ [ 712 , 1 ] , 𝐻 = 1 /( 1 − 𝛾 ) , 𝛼 ∈ ( 0 , 0 .
𝛾𝐻 /( 1 + 𝛾 ) 2 ] , and 𝑑 ≥ 3 .
Then there is a class of linear MDPs M such that the query complexity of any ( 𝛼, 𝛿 ) -sound planner over M is at least Ω (cid:0) 𝑑 2 𝐻 3 / 𝛼 2 (cid:1) .
In the remainder of the section we prove the above bound.
Throughout we assume that the conditions in Theorem H.3 are satisfied.
We start with the construction of the class M of MDPs, then prove several auxiliary results, before finally presenting the proof of the theorem.
The construction of M is based on a combination of hard tabular MDPs [Xiao et al., 2022] and hard linear bandit problems [Lattimore and Szepesvári, 2020, Section 24.1].
Each MDP in M has two states: S = { 𝑠 0 , 𝑠 1 } with 𝑠 0 being the initial state.
The action space is the intersection of a unit sphere and a ( 𝑑 − 2 )
-dimensional
hypercube:
A = {± 1 /√ 𝑑 − 2 } 𝑑 − 2 .
We construct MDPs 𝑀 𝛽 for all 𝛽 ∈ A , and let M = { 𝑀 𝛽 | 𝛽 ∈ A} .
The feature map 𝜑 is defined, for any 𝑎 ∈ A , as 𝜑 ( 𝑠 0 , 𝑎 ) = ( 1 , 0 , 𝑎 ⊤ ) ⊤ and 𝜑
[Equation]
[Equation] [Equation]
[Equation]
) ⊤ . We define the linear MDPs 𝑀 𝛽 to have deterministic rewards for any 𝛽 ∈ A .
Thus, 𝑀 𝛽 is fully defined by its reward parameter 𝜃 𝑟 and state-transition feature map 𝜓 , according to the definition of linear MDPs.
Let 𝜃 𝑟
= ( 1 , 0 , . .
. , 0 ) ⊤ , making state 𝑠
the only rewarding state, as then for all 𝑎 ∈ A , [Equation] ⟩ = 0 .
Let Δ = 4 ( 1 + 𝛾 ) 2 𝛼 /( 𝛾𝐻 2 ) ; since 𝛼 ≤ 0 .
𝛾𝐻 /( 1 + 𝛾 ) 2 , Δ ≤ 0 .
2 / 𝐻 = 0 .
2 ( 1 − 𝛾 ) .
Let 𝜓 ( 𝑠 0 ) =
( 𝛾, 0 , Δ 𝛽 ⊤ ) ⊤ and 𝜓 ( 𝑠 1 ) =
( 1 − 𝛾, 1 , − Δ 𝛽 ⊤ ) ⊤ .
This implies that
[Equation]
Our assumptions guarantee that 𝑃 𝛽 defines a valid transition kernel with probabilities in [ 0 , 1 ] .
The
MDP starts in 𝑠 0 and rewards are collected until the state transitions to 𝑠 1 , which is a terminal state with zero reward.
For the proof, we also need the following notation and supporting lemmas.
Notation.
The probability measure P 𝑀 𝛽 induced by the interconnection of a planner and a simulator for 𝑀 𝛽 is written for simplicity as P 𝛽 .
Similarly, E 𝑀 𝛽 is written as E 𝛽 .
𝑣 𝛽 (with arbitrary superscripts) denotes value functions (corresponding to the superscripts) of 𝑀 𝛽 .
For any integer 𝑖 ∈ { 1 , . . .
, 𝑑 − 2 } , err 𝑖 ( 𝜋, 𝛽 ) =
(cid:205)
𝑎 ∈A 𝜋 ( 𝑎 | 𝑠 0 ) 𝐼 sgn ( 𝑎 𝑖 )
≠ sgn ( 𝛽 𝑖 ) denotes the average error of a policy 𝜋 at the 𝑖 th coordinate, where 𝑎 𝑖 and 𝛽 𝑖 are the 𝑖 th components of 𝑎 and 𝛽 , respectively, and 𝐼 𝐸 is the indicator function of event 𝐸 .
With a slight abuse of notation, for a stationary memoryless policy 𝜋 , we let 𝜋 ⊤ 𝛽 denote (cid:205)
𝑎 ∈A 𝜋 ( 𝑎 | 𝑠 0 ) 𝑎 ⊤ 𝛽 .
Lemma H.4.
For any 𝑀 𝛽 ∈ M , the value function of a stationary memoryless policy 𝜋 is given by
[Equation]
and
[Equation]
Proof.
It clearly holds that 𝑣 𝜋𝛽 ( 𝑠 1 ) = 0 .
From the Bellman equation, 𝑣 𝜋𝛽 ( 𝑠 0 )
=
1 + 𝛾 ( 𝛾 + Δ 𝜋 ⊤ 𝛽 ) 𝑣
𝜋𝛽 ( 𝑠 0 ) , and the claim follows from solving this equation for 𝑣 𝜋𝛽 ( 𝑠 0 ) .
□
It is easy to see that the optimal policy in 𝑀 𝛽 is defined by 𝜋 ★𝛽 ( 𝛽 | 𝑠 0 )
=
1 (the actions in 𝑠 1 do not matter).
Hence, by the above lemma,
[Equation]
Because 1 − 𝜋 ⊤ 𝛽 = 2 (cid:205) 𝑑 − 2 𝑖 = 1 err 𝑖 ( 𝜋, 𝛽 )/
( 𝑑 − 2 ) , 𝑣 ★𝛽
[Equation]
Accordingly, to prove a lower bound on the suboptimality of 𝜋 , we need a lower bound for the sum of errors, (cid:205)
𝑑 − 2 𝑖
= 1 err 𝑖 ( 𝜋, 𝛽 ) .
To this end, Lemma H.5 below plays a key role.
Lemma H.5 (Error Probability Lower Bound) .
For any planner there exists a 𝛽 ∈ A such that
[Equation]
To prove Lemma H.5, we need some technical lemmas.
First, let F 𝑡 for any 𝑡 ∈ N + denote the 𝜎 -algebra generated by random variables in 𝐻 𝑡 , with F 1 being the trivial 𝜎 -algebra.
F = (F 𝑡 ) ∞
𝑡 = 1 is chosen to be the filtration.
The following lemma is adopted from Exercise 15.7 of Lattimore and
Szepesvári [2020] with a slight modification.
Lemma H.6 (KL-divergence decomposition) .
Let 𝑀 and 𝑀 ′ be two MDPs differing only in their transition probability kernels, denoted by 𝑃 and 𝑃 ′ , respectively.
Then, for any any F -adapted stopping time 𝜏 satisfying P 𝑀 ( 𝜏 < ∞)
= 1 , and an F 𝜏 -measurable 1 random variable 𝑍 ,
[Equation]
where P 𝑍𝑀 and P 𝑍𝑀 ′ are the laws of 𝑍 under P 𝑀 and P 𝑀 ′ , respectively, N 𝑡 ( 𝑠, 𝑎 ) denotes the number of queries with ( 𝑠, 𝑎 ) ∈ S
× A up to time step 𝑡 , and KL (· , ·) denotes the Kullback-Leibler (KL-) divergence of two distributions.
The next lemma provides an upper bound on the KL-divergence of certain next-state distributions.
A similar result appears in the proof of Lemma 6.8 of Zhou et al.
[2020], but it requires that 𝛾 ≥ 2 / 3 ; ours only requires the weaker assumption that 𝛾 ≥ 7 / 12 .
Lemma H.7.
Take any 𝛽, 𝛽 ′ ∈
A that only differ at a single coordinate.
Then for any action 𝑎 ∈ A ,
[Equation]
Proof.
Our proof relies on Proposition 2 of Xiao et al.
[2022]: for two Bernoulli distributions Ber ( 𝑝 ) and Ber ( 𝑝 ′ ) with parameters 𝑝, 𝑝 ′ ∈ ( 0 , 1 ) , it holds that
[Equation]
Since 𝑃 𝛽 ( 𝑠 1 | 𝑠 0 , 𝑎 ) = 1 − 𝛾 − Δ 𝛽 ⊤ 𝑎 and 𝑃 𝛽 ′ ( 𝑠 1 | 𝑠 0 , 𝑎 ) = 1 − 𝛾 − Δ ( 𝛽 ′ )
⊤ 𝑎 ,
[Equation]
𝛽
[Equation]
[Equation]
[Equation]
for any action 𝑎 ∈ A .
Note that
[Equation]
where ( 𝑎 ) is due to the fact that 𝑥 ( 1 − 𝑥 ) is monotone decreasing for 𝑥 ≥ 0 .
5 and 𝛾 + Δ 𝛽 ⊤ 𝑏 ≥ 𝛾 − Δ ≥ 0 .
5 since 𝛾 ≥ 7 / 12 and Δ ≤ 0 .
2 ( 1 − 𝛾 ) , ( 𝑏 ) follows since 0 .
5 ≤ 𝛾 + Δ , and ( 𝑐 ) holds because Δ ≤ 0 .
2 ( 1 − 𝛾 ) .
Combining this result with Eq. (31) concludes the proof of the lemma.
□
Now we are ready to prove Lemma H.5.
Proof of Lemma H.5.
Let 𝛽 ( 𝑖 ) be a vector obtained by flipping the sign of 𝛽 ’s 𝑖 th coordinate.
Then,
[Equation]
where P err
[Equation] [Equation]
∈ M 1 ([ 0 , 1 ]) are the laws of the random variable err 𝑖 ( 𝜋 𝜏 , 𝛽 ) in 𝑀 𝛽 and 𝑀 𝛽 ( 𝑖 ) , respectively , and the last line follows from an improved Bretagnolle-Huber inequality (inequality (14.11) of Lattimore and Szepesvári [2020]).
Applying Lemmas H.6 and H.7 to the
KL-divergence in the exponent in the right hand side of the above inequality together with the fact that (cid:205) ( 𝑠,𝑎 )∈S×A E 𝛽 [N 𝜏 ( 𝑠, 𝑎 )] ≤ E 𝛽 [ 𝜏 ] , we can further lower-bound the last line by
[Equation]
Therefore,
[Equation]
where the first equality holds because for any 𝛽 , there is exactly one 𝛽 ( 𝑖 ) in A .
As max 𝛽 ∈A 𝑓 ( 𝛽 ) ≥ (cid:205) 𝛽 ∈A 𝑓 ( 𝛽 )/|A| for any 𝑓 : A → R , arg max 𝛽 ∈A (cid:205)
𝑑 − 2 𝑖 = 1 P 𝛽 (cid:0) err 𝑖 ( 𝜋 𝜏 , 𝛽 ( 𝑖 ) ) ≥ 1 / 2 (cid:1) satisfies the claim of the lemma.
□
Now we are ready to prove Theorem H.3.
Proof of Theorem H.3.
Take any ( 𝛼, 𝛿 ) -sound planner on M .
Let err ( 𝜋, 𝛽 ) : = (cid:205)
𝑑 − 2 𝑖 = 1 err 𝑖 ( 𝜋, 𝛽 ) for brevity.
From Eq.
[Equation]
where the first inequality holds because 𝜋 ⊤ 𝛽 ≥ − 1 for any stationary memoryless policy 𝜋 , the second inequality is due to the Markov inequality, while the last inequality holds by Lemma H.5.
From Eq.
(29) and 𝜋 ⊤ 𝛽 ≤ 1 we also have that
[Equation]
where the second inequality holds because
[Equation]
Combining this result with Eq.
[Equation]
Note that err ( 𝜋 𝜏 , 𝛽 ) >
( 𝑑 − 2 )/ 8 implies that 𝑣 ★𝛽 ( 𝑠
[Equation]
) − 𝑣 𝜋 𝜏 𝛽 ( 𝑠 0 ) >
𝛼 since similarly to Eq.
(32) (i.e., without the expectation)
[Equation]
where the last equality follows because Δ = 4 ( 1 + 𝛾 ) 2 𝛼 /( 𝛾𝐻 2 ) = 4 ( 1 − 𝛾 2 ) 2 𝛼 / 𝛾 .
Therefore,
P 𝛽
[Equation]
where ( 𝑎 ) follows since Δ ≤ 0 .
2 ( 1 − 𝛾 ) , and ( 𝑏 ) follows since 0 ≤ 0 .
4 𝑥 /( 1 + 𝑥 ) ≤ 0 .
2 for 𝑥 ∈ [ 0 , 1 ] .
This implies that unless E 𝛽 [ 𝜏 ] ≥ Ω (cid:0) 𝑑 2 𝐻 3 / 𝛼 2 (cid:1) , the algorithm is not ( 𝛼, 𝛿 ) -sound.
Indeed if
E
[Equation]
it holds that P 𝛽 (cid:16) 𝑣 ★𝛽 ( 𝑠 0 ) − 𝑣 𝜋 𝜏 𝛽 ( 𝑠 0 ) >
𝛼 (cid:17) > 𝛿 , contradicting the assumption that the planner is ( 𝛼, 𝛿 ) - sound on M .
□