Near Instance-Optimal PAC Reinforcement Learning for Deterministic MDPs
Abstract
In probably approximately correct (PAC) reinforcement learning (RL), an agentis required to identify an ε -optimal policy with probability 1 − δ .
While minimaxoptimal algorithms exist for this problem, its instance-dependent complexity remains elusive in episodic Markov decision processes (MDPs).
In this paper, wepropose the first nearly matching (up to a horizon squared factor and logarithmicterms) upper and lower bounds on the sample complexity of PAC RL in deterministic episodic MDPs with finite state and action spaces.
In particular, our boundsfeature a new notion of sub-optimality gap for state-action pairs that we call thedeterministic return gap.
While our instance-dependent lower bound is writtenas a linear program, our algorithms are very simple and do not require solvingsuch an optimization problem during learning.
Their design and analyses employnovel ideas, including graph-theoretical concepts (minimum flows) and a newmaximum-coverage exploration strategy.
Introduction
In reinforcement learning [RL, 38] an agent interacts with an environment modeled as a Markovdecision process (MDP) by sequentially selecting actions and receiving feedback in the form ofreward signals.
Depending on the application, the agent may seek to maximize the cumulativerewards received during learning (which is typically phrased as a regret minimization problem)or to minimize the number of learning interactions (i e., the sample complexity ) for identifying anear-optimal policy.
The latter pure exploration problem was introduced in [20] under the name of
Probably Approximately Correct (PAC) RL:
given two parameters ε, δ > 0 , the agent must returna policy that is ε -optimal with probability at least 1 − δ .
Our work focuses on this problem in thecontext of episodic (a.k.a.
finite-horizon) tabular MDPs.
The PAC RL problem has been mostly studied under the lens of minimax (or worst-case) optimality.
In the episodic setting, the algorithm proposed in [12] has sample complexity bounded by
O ( SAH 2 log(1 /δ ) /ε 2 ) for an MDP with S states, A actions, horizon H , and time-homogeneoustransitions and rewards (i.e., not depending on the stage).
This is minimax optimal for such a context[11].
Similarly, in [33] the authors designed a strategy with O ( SAH 3 log(1 /δ )
/ε 2 ) complexity intime-inhomogeneous MDPs, which was later shown to be minimax optimal [17].
While the minimax framework provides a strong notion of statistical optimality, it does not accountfor one of the most desirable properties for an RL algorithm: the ability to adapt to the difficulty ofthe MDP instance.
For this reason, researchers recently started to investigate the instance-dependentcomplexity of PAC RL.
Earlier attempts were made in the simplified setting where the agent hasaccess to a generative model (i.e., it can query observations from any state-action pair using asimulator) in γ -discounted infinite-horizon MDPs [48, 5].
The online setting, where the agent can only sample trajectories from the environment, has been studied by [4] for discounted MDPs and by[44] for episodic time-inhomogeneous MDPs.
All these works derive sample complexity bounds thatscale with certain gaps between optimal value functions.
For instance, in the episodic setting, thevalue gap ∆ h ( s, a ) :
= V ⋆h ( s ) − Q ⋆h ( s, a ) 1 intuitively characterizes the degree of sub-optimality ofaction a for state s at stage h .
Unfortunately, these bounds are known to be sub-optimal and how toachieve instance optimality remains one of the main open questions.
In fact, recent works on regretminimization [40, 13] showed that value gaps are often overly conservative, and the same holds for
PAC RL.
We refer the reader to Appendix A for a deeper discussion on problem-dependent results in
RL and the review of other related PAC learning frameworks.
The main challenge towards instance optimality is that existing lower bounds for exploration problemsin MDPs [5, 40, 4, 13] are written in terms of non-convex optimization problems.
Their “implicit”form makes it hard to understand the actual complexity of the setting and, thus, to design optimalalgorithms.
Existing solutions either derive explicit sufficient complexity measures that inspirealgorithmic design [44], or solve (a relaxation of) the optimization problem from the lower boundusing the empirical MDP as a proxy for the unknown MDP [4].
The latter extends the Track-and-Stopidea originally proposed in [21] for bandits ( H = 1 ), and requires in particular a large amount offorced exploration.
Both solutions have limitations.
On the one hand, it is not clear if and how suchsufficient complexity measures or relaxations are related to an actual lower bound.
On the otherhand, strategies solving a black-box optimization problem to find an optimal exploration strategyare typically very inefficient and often come with either only asymptotic ( δ → 0 ) guarantees or withpoor (far from minimax optimal) sample complexity in the regime of moderate δ .
Contributions
This paper presents a complete study of PAC RL in tabular deterministic episodic
MDPs with time-inhomogeneous transitions, a sub-class of stochastic MDPs where state transitionsare deterministic and the agent observes stochastic rewards from unknown distributions.
Our firstcontribution is an instance-dependent lower bound on the sample complexity of any PAC algorithm.
We show that the number of visits n τh ( s, a ) to any state-action-stage triplet ( s, a, h ) at the stoppingtime τ satisfies
[Equation]
where ∆ h ( s, a ) :
= V ⋆ 1 − max π ∈ Π s,a,h V π 1 , with V π 1 the expected return of policy π , V ⋆ 1 the optimalexpected return, and Π s,a,h the set of all deterministic policies that visit ( s, a ) at stage h .
We callthese quantities the deterministic return gaps due to their closeness with the return gaps introducedin [13] for general MDPs.
In deterministic MDPs, the deterministic return gaps are actually H timeslarger than the return gaps and they are never smaller than value gaps.
Our lower bound on thesample complexity τ is then the value of a minimum flow with local lower bounds (1), i.e., roughlythe minimum number of policies that must be played to ensure (1) for all ( s, a, h ) .
To our knowledge,this is the first instance-dependent lower bound for the PAC setting in episodic MDPs.
On the algorithmic side, we design EPRL, a generic elimination-based method for PAC RL, andcouple it with a novel adaptive sampling rule called maximum-coverage sampling .
The latter is asimple strategy which does not require solving the optimization problem from the lower bound atlearning time in a Track-and-Stop fashion.
Instead, it greedily selects the policy that maximizes thenumber of visited under-sampled triplets ( s, a, h ) , i.e. those having received the least amount ofvisits so far.
We prove that EPRL is ( ε, δ ) -correct under any sampling rule.
Moreover, we show thatthe sample complexity of EPRL with max-coverage sampling matches our instance-dependent lowerbound up to logarithmic factors and a multiplicative O ( H 2 ) term, while also being minimax optimal.
Finally, we perform numerical simulations on random deterministic MDPs which reveal that EPRLcan indeed improve over existing minimax-optimal algorithms tailored for the deterministic case.
Preliminaries
Let M :=
( S , A , { f h , ν h } h ∈
[ H ] , s 1 , H ) be a deterministic time-inhomogeneous finite-horizon MDP,where S is a finite set of S states, A is a finite set of A actions, f h : S ×
A → S and ν h : S × A →1 V ⋆ and Q ⋆ respectively denote the optimal value and action-value functions, that are defined in Section 2.
P ( R ) are respectively the transition function and the reward distribution at stage
h ∈
[ H ] , s 1 ∈ Sis the unique initial state, and H is the horizon.
Without loss of generality, we assume that, at eachstage h ∈ [ H ] and state s ∈ S , only a subset
A h ( s ) ⊆
A of actions is available.
We denote byr h ( s, a ) :
=
E x ∼ ν h ( s,a ) [ x ] the expected reward after taking action a in state s at stage h .
A deterministic policy π = { π h } h ∈
[ H ] is a sequence of mappings π h : S → A .
We let Π :=
{ π | ∀ h ∈
[ H ] , s ∈ S : π h ( s ) ∈
A h ( s ) } be the set of all valid deterministic policies.
Executing apolicy π ∈ Π on MDP M yields a deterministic sequence of states and actions ( s πh , a πh )
h ∈
[ H ] , wheres π 1 =
s 1 , a πh = π h
( s πh ) for all h ∈
[ H ] , and s πh =
f h − 1 ( s πh − 1 , a πh − 1 ) for all h ∈ { 2 , . . .
, H } .
We let
S h := { s ∈ S | ∃ π ∈ Π : s πh = s } be the subset of states that are reachable at stage h ∈
[ H ] .
Finally,we define N := (cid:80)
Hh =1 (cid:80) s ∈S h |A
h ( s ) | as the total number of reachable state-action-stage triplets.
For each ( s, a, h ) , the action-value function Q πh ( s, a ) of a policy π ∈ Π quantifies the expected returnwhen starting from s at stage h , playing a and following π thereafter.
In deterministic MDPs, it has thesimple expression Q πh ( s, a ) =
r h ( s, a )+ V πh +
f h ( s, a )) , where V πh ( s ) :
= Q πh ( s, π h ( s )) is the corresponding value function (with V πH +1 ( s ) = 0 ).
The expected return of π is simply its value at the initialstate, i.e., V π 1 ( s 1 ) =
(cid:80)
Hh =1 r h ( s πh , a πh ) .
We let Π ⋆ :=
{ π ⋆ ∈ Π : V π ⋆ 1 ( s 1 ) =
max π ∈ Π
V
π
1 ( s 1 ) } bethe set of optimal policies , i.e., those with maximal return.
Finally, we denote by V ⋆h ( s ) and Q ⋆h ( s, a )the optimal value and action-value function, respectively.
These are related by the Bellman optimalityequations as Q ⋆h ( s, a ) =
r h ( s, a ) + V {+⋆h +
+}
[-+
-] ( f h ( s, a )) and V ⋆h ( s ) = max a ∈A h ( s ) Q ⋆h ( s, a ) .
Learning problem The agent interacts with an MDP M in episodes indexed by t ∈ N .
At thebeginning of the t -th episode, the agent selects a policy π t ∈ Π based on past history through its sampling rule , executes it on M , and observes the corresponding deterministic trajectory ( s π t h ,
a π t h ) h ∈
[ H ]together with random rewards ( y th ) h ∈
[ H ] , where y th ∼ ν h ( s π t h , a π t h ) .
At the end of each episode, theagent may decide to terminate the process through its stopping rule and return a policy (cid:98) π prescribedby its recommendation rule .
We denote by τ its random stopping time.
An algorithm for PACidentification is thus made of a triplet ( { π t } t ∈ N , τ, (cid:98) π ) .
The goal of the agent is two-fold.
First, forgiven parameters ε, δ > 0 , it must return an ε -optimal policy with probability at least 1 − δ .
Definition 1.
An algorithm is ( ε, δ ) -PAC on a set of MDPs M if, for all M ∈ M , it stops a.s. with P M (cid:16) V
[Equation]
Second, it should stop as early as possible, i.e., by minimizing the sample complexity τ .
Henceforth,we assume that the transition function f is known but not the reward distribution ν .
Note that if thetransitions are unknown, the agent can still estimate them (since it knows that M is deterministic)with at most N ≤ SAH episodes.
Minimum flows We review some basic concepts from graph theory which will be at the coreof our algorithms and analyses later.
Full details can be found in Appendix B. First note that adeterministic MDP (without reward) can be represented as a directed acyclic graph (DAG) with onearc for each available state-action-stage triplet.
Let E := { ( s, a, h ) : h ∈
[ H ] , s ∈ S h , a ∈
A h ( s ) }be the set of arcs in the DAG.
The minimum flow problem, originally introduced in [43] andlater studied in, e.g., [1, 2, 10], consists of findining a flow (i.e., an allocation of visits) ofminimal value which satisfies certain demand constraints in each arc of the graph.
In our specificsetting, we define a flow as any non-negative function η : E → [0 , ∞ ) that belongs to the following setΩ := [Equation] E
[Equation]
This implies that a flow, seen as an allocation of visits to the arcs, satisfies the navigation constraints(i.e.
, incoming and outcoming flows are equal at each state).
The minimum flow for a non-negativelower-bound function c : E → [0 , ∞ ) is the solution to the following linear program (LP):
[Equation]
Intuitively, the goal is to minimize the amount of flow leaving the initial state while satisfying thenavigation and demand constraints.
We note that more efficient algorithms exist for this problemthan the LP formulation, e.g., the variant of the Ford-Fulkerson method proposed in [10] which isguaranteed to find an integer solution when the lower bound function is integer-valued.
The Complexity of PAC RL in Deterministic MDPs
Before stating our lower bound, we formally introduce the new notion of sub-optimality gap itfeatures and compare it with other notions that appeared in the literature.
On sub-optimality gaps The most popular notion of sub-optimality gap is the so-called value gap .
It was introduced first in the discounted infinite-horizon setting [e.g., 48] and later for episodic MDPs[e.g.
Formally, in the latter context, the value gap of any action
a ∈
A h ( s ) in state s ∈ S hat stage h ∈ [ H ] is ∆ h ( s, a ) :
= V ⋆h ( s ) − Q ⋆h ( s, a ) .
Such a notion of gap appears in the complexitymeasure for PAC RL proposed in [44].
In the deterministic setting, such a complexity measure canbe written as C ( M , ε ) =
(cid:80) ( s,a,h ) 1 max( (cid:101)
∆ h ( s,a ) ,ε ) 2 , where (cid:101) ∆ h ( s, a ) = min a ′ :
∆ h ( s,a ′ ) > 0 ∆ h ( s, a ′ )
ifa is the unique optimal action at ( s, h ) , and (cid:101) ∆ h
( s, a ) =
∆ h ( s, a ) otherwise.
Intuitively, the (inverse)value gap is proportional to the difficulty of learning whether an action a is sub-optimal for state s atstage h .
Then, C ( M , ε ) is proportional to the difficulty of learning a near optimal action at all statesand stages.
Recent works [40, 13] showed that this is actually not necessary: if one only cares aboutcomputing a policy maximizing the return at the initial state, it is not necessary to learn an optimalaction at states which are not visited by such an optimal policy, in particular when the return of allpolicies visiting the state is small.
The return gap [13] was introduced to cope with this limitation.
Indeterministic MDPs, it can be expressed as gap h ( s, a ) :
= 1 H min π ∈ Π s,a,h (cid:80) hℓ =1 ∆ ℓ ( s πℓ , a πℓ ) , wherewe denote by Π s,a,h := { π ∈ Π : s πh = s, a πh = a } the subset of deterministic policies that visit( s, a ) at stage h .
In words, the return gap of ( s, a, h ) is proportional to the sum of value gaps alongthe best trajectory (i.e., one with maximal return) that visits ( s, a ) at stage h .
Intuitively, this meansthat, if ∆ h ( s, a ) is extremely small but all policies visiting ( s, a ) at stage h need to play a highlysub-optimal action before, then ∆ h ( s, a )
≪
gap h ( s, a ) .
In the deterministic case, our lower boundreveals that the normalization by H is not necessary, and we define the deterministic return gap to be
[Equation]
Using the well-known relationship V ⋆ 1 ( s 1 ) − V π 1 ( s 1 )
= (cid:80)
Hh =1 ∆ h ( s πh , a πh )
[e.g., 40, Proposition 5],it is easy to see that ∆ h ( s, a ) ≤ ∆ h ( s, a ) =
H × gap h ( s, a ) .
Lower Bound We now present our instance-dependent lower bound based on deterministic returngaps, which will guide us in the design and analysis of sample efficient algorithms.
This result is thefirst instance-dependent lower bound for PAC RL in the episodic setting.
Lower bounds for ε -bestarm identification in a bandit model (which corresponds to H = S = 1 ) were derived in [32, 14, 22],while problem-dependent regret lower bounds for finite-horizon MDPs are provided in [13, 40].
We consider the class M 1 of deterministic MDPs with Gaussian rewards of unit variance 2 , in whichν h
( s, a ) =
N ( r h ( s, a ) , 1) .
Let Π ε :=
{ π ∈ Π : V π 1 ( s 1 ) ≥ V ⋆ 1 ( s 1 ) − ε } be the set of all ε -optimalpolicies and denote by Z εh := { s ∈ S h , a ∈
A h ( s ) : Π s,a,h ∩ Π ε ̸ = ∅} the set of state-action pairsthat are reachable at stage h by some ε -optimal policy.
Note that ∆ h ( s, a ) ≤ ε for all ( s,
a ) ∈
Z εh .
Theorem 1.
Fix any MDP M ∈ M 1 .
Then, any algorithm which is ( ε, δ ) -PAC on the class M 1 mustsatisfy, for any h ∈ [ H ] , s ∈ S h , and a ∈ A h ( s ) ,
[Equation]
where ∆ h min := min ( s ′ ,a ′ ):∆ h ( s ′ ,a ′ ) > 0 ∆ h ( s ′ , a ′ ) if |Z εh | = 1 and ∆ h min := 0 otherwise.
Moreover,for c : E → [0 , ∞ ) the lower bound function defined above,
[Equation]
The first lower bound (3) is on the number of visits required for any state-action-stage triplet.
It 169 intuitively shows that an ( ε, δ ) -PAC algorithm must visit each triplet proportionally to its inversedeterministic return gap.
The second one (4) shows that the actual sample complexity of the algorithm 171 must be at least the value of a minimum flow computed with the local lower bounds (3), i.e. that the algorithm must play the minimum number of episodes (i.e., policies) that guarantees (3) for each 173 ( s, a, h ) .
Intuitively, due to the navigation constraints of the MDP, there might be no algorithm whichtightly matches (3) for each ( s, a, h ) , and (4) is exactly enforcing these constraints.
While φ ⋆ ( c ) has 175 no explicit form, Lemma 6 in Appendix B gives an idea of how it scales with the gaps:
[Equation]
Observe that the quantity on the right-hand side resembles the complexity measure C ( M , ε )
[44],except that value gaps are replaced by return gaps.
This implies that, in general, our lower bound canbe much smaller than this complexity.
For instance, in an MDP with extremely small value gaps instates which are not visited by an optimal policy, φ ⋆ ( c ) does not scale with such gaps at all.
In Appendix C.2 we further provide a minimax lower bound for PAC RL in deterministic MDPsscaling as Ω (cid:0) SAH 2 log (1 /δ ) /ε
2 (cid:1) , with a reduced H 2 dependency compared to the H 3 that appearin the stochastic case [17].
We note that faster rates for deterministic MDPs have already beenobtained in other RL settings [e.g., 47].
The BPI-UCRL algorithm [28] particularized to deterministic MDPs is matching this lower bound and is thus minixal optimal.
We now present the first algorithmwhich is simultaneously minimax optimal for deterministic MDPs and nearly matching (up to O ( H 2 )and logarithmic factors) the lower bound of Theorem 1.
EPRL and Max-Coverage Sampling
We propose a general Elimination-based scheme for PAC RL, called EPRL (Algorithm 1).
At eachepisode t ∈ N , the algorithm plays a policy π t selected by some sampling rule.
Then, based on thecollected samples, the algorithm updates its statistics and eliminates all actions which are detected assub-optimal with enough confidence.
This procedure is repeated until a stopping rule triggers.
Formally, EPRL maintains an estimate ˆ r th ( s, a ) :
= 1 n th ( s,a ) (cid:80) tl =1 y lh 1 (cid:0) s lh
= s, a lh = a (cid:1) , with ˆ r 0 h = 0 ,of the unknown mean reward r h ( s, a ) for each ( s, a, h ) .
Here n th ( s, a ) := (cid:80) tl
=1 1 (cid:0) s lh
= s, a lh = a (cid:1)is the number of times ( s, a ) is visited at stage h up to episode t .
We define the following upper andlower confidence intervals to the value functions of a policy π ∈ Π :
[Equation]
where b th ( s, a ) is a bonus function , i.e., the width of the confidence interval at ( s, a, h ) .
As commonin the literature, we assume that rewards are bounded in [0 , 1] , which allows us to choose
[Equation]
Elimination rule Algorithm 1 keeps a set of active (or candidate) actions
A th ( s ) for each stageh ∈
[ H ] , state s ∈ S h , and episode t ∈ N .
Let Π t :=
{ π ∈ Π | ∀ s, h : π h ( s )
∈
A th ( s ) ∨A th ( s ) = ∅}be
the subset of active policies that only play active actions at episode t .
Note that an active policy canplay an arbitrary action in states where all actions have been eliminated.
As can be seen in Line 7 of
Algorithm 1, action a is eliminated from A th ( s ) if max [Equation] where we recall that Π s,a,h denotes the set of all deterministic policies that visit s, a at stage h , thatis when we are confident that none of the policies visiting ( s, a ) at stage h is optimal.
The maximumrestricted to Π s,a,h can be easily computed by standard dynamic programming (e.g., it is enough toset the reward to −∞ for all state-action pairs different than ( s, a ) at stage h ).
If Π s,a,h ∩ Π t − 1 = ∅ ,we set the maximum to −∞ so that the elimination rule triggers.
Remark 1. While defining Π t simplifies the presentation, EPRL neither stores nor enumerates theset of active policies.
In particular, EPRL does not eliminate policies but rather ( s, a, h ) triplets.
Thesets A th ( s ) can be updated in polynomial time by dynamic programming without ever computing Π t .
[Table]
1: Input: deterministic MDP (without reward)
M :=
( S , A , { f h } h ∈ [ H ] , s 1 , H ) , ε , δ 2: Initialize A 0 h ( s ) ← A h ( s ) for all h ∈
[ H ] , s ∈ S h 3: Set n 0 h
( s, a ) ← 0 for all h ∈
[ H ] , s ∈ S h , a ∈
A h ( s ) 4: for t = 1 , . . .
do 5:
Play π t ← S
AMPLING R ULE () 6:
Update statistics n th ( s, a ) , ˆ r th ( s, a ) 7:
A th ( s ) ← A t − 1 h ( s ) ∩ (cid:110) a ∈ A : max π ∈ Π s,a,h ∩ Π t − 1 V t,π 1 ( s 1 ) ≥ max π ∈ Π V t,π 1 ( s 1 ) (cid:111) 8: where Π t − 1 ← (cid:8) π ∈ Π | ∀ s, h : π h ( s ) ∈ A t − 1 h ( s ) ∨ A t − 1 h ( s ) = ∅ (cid:9) (need not be stored/computed) 9: if max π ∈ Π t (cid:16) V π,t 1 ( s 1 ) − V π,t 1 ( s 1 ) (cid:17) ≤ ε or ∀ h ∈ [ H ] , s ∈ S h : |A
th ( s ) | ≤ 1 then 10:
Stop and recommend (cid:98) π ∈ arg max π ∈ Π t V π,t 1 ( s 1 ) 11: end if 12: end for 13: function M AX C OVERAGE () 14:
Let k t ← min h ∈
[ H ] ,s ∈S h ,a ∈A t − 1 h ( s ) n t − 1 h ( s, a )
+ 1 and ¯ t k t ← inf l ∈ N { l : k l = k t } 15: if t mod 2 = 1 then 16: return π t ← arg max π ∈ Π (cid:80)
[Equation]
17: else 18: return π t ←
[Equation]
M
[Equation]
19: end if
Stopping rule EPRL uses two different stopping rules (Line 8). The first one checks whether,for all active policies π ∈ Π t , the {+confidence interval on the return, V π,t 1 ( s 1 ) − V π,t 1 ( s 1 ) =2 (cid:80) Hh =
+}
b th ( s πh , a πh ) , which we refer to as diameter , is below ε .
The second one checks whethereach set
A th ( s ) contains either 1 action or 0 actions (which happens when the state is unreachable byan optimal policy).
In both cases, we recommend the optimistic (active) policy (Line 9).
Sampling rule While EPRL may be used with different sampling rules, we recommend the maxcoverage sampling rule described in Algorithm 1.
This sampling rule aims at ensuring that no( s, a, h ) triplet remains under-visited for too long.
This is achieved by selecting the policy whichgreedily maximizes the number of visited under-sampled triplets, denoted by U t .
The quantityk t = min ( s,a,h ): a ∈A t − 1 h ( s ) n t − 1 h ( s, a ) + 1 can be interpreted as the target minimum number of visitsfrom active triplets that we want to achieve in round t and permit to defineπ t = arg max π ∈ Π H (cid:88) h = 1 (( s πh , a πh , h ) ∈ U t ) with U t = (cid:110) ( s, a, h ) : a ∈
A t kt − 1 h
[Equation]
where t k = inf { t : k t = k } is the first round in which the target is set to k .
The argmax over Π canbe computed using dynamic programming.
We emphasize that this argmax is not restricted to the setof active policies, meaning that we may play eliminated actions in order to augment the coverage (thatis, the minimal number of visits) faster.
Every even round, max-coverage instead chooses an activepolicy maximizing the diameter featured in the stopping rule (max-diameter sampling).
As we shallsee in our analysis, this dichotomous behavior is needeed in order to maintain minimax-optimality.
Theoretical guarantees
Our first result, proved in Appendix D.2.1, shows that EPRL is ( ε, δ )
-PAC under any sampling rule.
Itfollows from the fact that 1) the choice of bonus function (5) ensures that all the confidence intervalsare valid and 2) state-action pairs from optimal trajectories are never eliminated when this holds.
Theorem 2.
Algorithm 1 is ( ε, δ ) -PAC provided that the sampling rule makes it stop almost surely.
We now analyze the sample complexity of EPRL combined with max-coverage sampling.
Theorem 3.
(Informal version of Theorem 8 in Appendix D.3)
With probability at least 1 − δ , 235 the sample complexity of EPRL combined with the maximum-coverage sampling rule satisfiesτ = (cid:101) O
( φ ⋆ ( g )) , where g : E → [0 , ∞ ) is the lower bound function defined by
[Equation]
Moreover, with the same probability, τ = (cid:101) O
( SAH 2 εlog(1 /δ )) , where (cid:101)
O hides logarithmic terms.
First note that EPRL combined with such a sampling rule is minimax optimal , since it matches theworst-case lower bound derived in Appendix C.2.
In addition, the leading term in the instancedependent complexity is the value of a minimum flow with a lower bound function g that, in thecase multiple disjoint optimal trajectories exist 4 , matches the gap-dependence in (1).
If we supposethat there exist at least two disjoint optimal trajectories, in which case ∆ min =
∆ h min = 0 , then,thanks to Lemma 7 in Appendix B, one can easily see that φ ⋆ ( g ) ≤ 48 H 2 φ ⋆ ( c ) + φ ⋆ ( g ′ ) , whereg ′
h ( s, a ) :
= (cid:101)
O ( H 2 / max (cid:0) ∆ h
( s, a ) , ε (cid:1) 2 ) does not depend on δ and c is the “optimal” lower boundfunction from (1).
Hence, in the asymptotic regime ( δ → 0 ), φ ⋆ ( g ) matches our lower bound up toa
O ( H 2 ) multiplicative factor 5 .
Finally, our sample complexity bound has an extra multiplicativelogarithmic term which roughly scales as O (log( H ) log
( H log(1 /δ ) /ε )) .
While this term makes thedependence on δ sub-optimal by a log log(1 /δ ) factor, we show in Appendix E that it can be removedin the specific case of tree-based MDPs [13]. Remark 2.
We believe that the sub-optimality on H could be reduced to a single H factor by boostingthe lower bound.
In Appendix E, we show that this is indeed possible in tree-based MDPs.
As for theupper bound, reducing H 2 to H is likely to require tighter concentration bounds on values.
Remark 3.
In Appendix D.4, we prove that, when using the max-diameter sampling rule (Line 17in Algorithm 1) at each step, the sample complexity is (cid:101) O
( (cid:80) [Equation] max
[Equation]
While this scales with the same gaps as Theorem 3, it is only a naive upper bound to the minimumflow value (see Section 3).
The intuition is that max-diameter sampling alone does not ensure that alltriplets are visited sufficiently often, which prevents us from tightly controlling their elimination times.
Proof sketch The complete proof is given in Appendix D.3.
It first relies on the following crucialresult which relates the deterministic return gaps to the sum of confidence bonuses.
Lemma 1 (Diameter vs gaps) .
With probability at least 1 − δ , for any t ∈ N , h ∈
[ H ] , s ∈ S h , a ∈
A h ( s ) , if a ∈ A th ( s ) and the algorithm did not stop at the end of episode t ,
[Equation]
where ∆ min :=
min h ∈ [ H ] min s ∈S h min a :∆ h ( s,a ) > 0 ∆ h ( s, a ) if there exists a unique optimal trajectory ( s ⋆h , a ⋆h ) h ∈
[ H ] , and ∆ min := 0 in the opposite case.
In our analysis, we refer to the set of consecutive time steps { t ∈ N : k t = k } as the k -th period .
Using the fact that in period k + 1 each active triplet has been visited at least k times, one can use
Lemma 1 to obtain an upper bound κ s,a,h ≃
H 2 log(1 /δ ) max ( ∆ h ( s,a ) , ∆ min ,ε ) 2 on the last period in which ( s, a, h )is active (Lemma 18 in Appendix D.3).
A crucial step of the proof is then to upper bound the durationof the k -th period, d k := (cid:80) τt =1 1 ( k t = k ) .
Lemma 2.
d k ≤ 2(log( H ) + 1) φ ⋆ ( c k ) where c kh ( s, a ) = 1 ( a ∈ A t k − 1 h ( s ) , n t k − 1 h ( s, a ) < k ) .
The intuition behind this result is as follows.
Recall that the goal of the max-coverage sampling rulein period k is to visit at least once each ( s, a, h ) that is active
(i.e., a ∈
A t k − 1 h ( s ) ) and undersampled(i.e., n t k − 1 h ( s, a ) < k ).
By definition, the minimum flow φ ⋆ ( c k ) is the minimum number of policiesthat need to be played to achieve this goal.
Interestingly, Lemma 2 shows that the number of policiesplayed by max-coverage to visit all active undersampled triplets is very close to its theoreticalminimum, despite the fact that the algorithm never computes an actual minimum flow.
We prove thisby interpreting max-coverage sampling as a greedy maximization of some coverage function (relatedto a minimum flow problem) and leveraging the theory of sub-modular maximization [e.g., 30].
Thanks to Lemma 2, we have that τ ≤ 2(log( H ) + 1) (cid:80) k
τ k =
φ ⋆ ( c k ) , where k τ is the index of theperiod at which the algorithm stops.
To bound this quantity we carefully apply the theory of minimumflows and their dual problem of maximum cuts .
Let us define a cut C as any subset of states containingthe initial state and let E ( C ) be the set of arcs that connect states in C with states not in C .
Thewell-known min-flow-max-cut theorem (Theorem 4 stated in Appendix B) states that, for any lowerbound function c , φ ⋆ ( c ) = max C∈ [Equation]
( s,a,h ) c h ( s, a ) , where C denotes the set of all valid cuts.
Then,
[Equation]
⋆ ( g ) , where g : E → [0 , ∞ ) is defined by g h ( s, a )
= κ s,a,h + 1 .
The proof is concluded by using thisresult in the decomposition into periods, while noting that (cid:80)
k
τ k =1 1 k ≤ max ( s,a,h ) log( κ s,a,h )
+ 1 .
Experiments
We compare numerically EPRL to the minimax optimal BPI-UCRL algorithm [28], adapted to thedeterministic setting, on synthetic MDP instances.
For EPRL, we experiment with the two samplingrule: max-coverage (maxCov) and max-diameter (maxD, see Line 17 of Algorithm 1).
We defer to
Appendix F some implementation details, including a precise description of the BPI-UCRL baseline.
We generate random “easy” deteministic MDP instances with Gaussian rewards of variance 1 usingthe following protocol.
For fixed S, A, H the mean rewards r h
( s, a ) are drawn i.i.d.
from a uniformdistribution over [0 , 1] and for each state-action pair, the next state is chosen uniformly at random in{ 1 , . . .
, S } .
Finally, we only keep MDP instances whose minimum value gap, denoted by ∆ min , islarger than 0 .
Our first observation is that depending on the MDP, the identity of the best performingalgorithm can be different.
In Figure 1 we show the distribution of the sample complexity (estimatedover 10 Monte Carlo simulations) for three different MDPs obtained from our sampling procedurewith S, A = 2 and H = 3 and for algorithms that are run with parameters δ = 0 .
1 and ε = 1 .
5∆ min .
[Figure]
To get a better understanding of this phenomenon, we then generated 10 MDP instances of size( S, A, H ) =
(2 , 2 , 3) and for each MDP we ran EPRL and BPI-UCRL for 25 values of ε in a grid[0 .
05∆ min , 10∆ min ] and δ = 0 .
We ran 10 Monte-Carlo simulations for each value of the triplet( MDP , algorithm A , ε ) , in order to estimate the expected sample complexity E A [ τ δ ] .
In Figure 2 weplot the relative performance (ratio of sample complexities) of different algorithms as a function of the value of ε/ ∆ min : each point corresponds to a different MDP and a different value of ε .
Weobserve that for large values of ε/ ∆ min , BPI-UCRL has a smaller sample complexity than bothversions of EPRL, with a ratio never exceeding 2 (resp. 3 ) for max-diameter (resp.
max-coverage).
However, in the more interesting small ε/ ∆ min regime EPRL is better by several orders of magnitude.
This is expected since, for small ε , EPRL is able, through its elimination rule, to identify the optimalpolicy long before the diameter goes below ε .
We observe that the threshold of ε/ ∆ min at which
EPRL algorithms become a better choice than BPI-UCRL seems to vary with the MDP.
[Figure]
Our experiments also reveal an intriguing phenomenon: the use of max-diameter sampling within
EPRL often outperforms max-coverage sampling, even if there exists MDPs (2 out of 10 in ourexperiments) in which max-coverage is indeed empirically better.
We leave as future work to obtain abetter characterizations of MDPs for which EPRL with max-coverage sampling performs best.
Discussion
We derived an instance-dependent and a worst-case lower bound characterizing the complexity of
PAC RL in deterministic MDPs, and proposed a general elimination algorithm together with a novelmaximum-coverage sampling rule that nearly matches them (up to O
( H 2 ) and logarithmic factors).
We conclude with some discussion about our results and future directions.
Max-coverage vs max-diameter While minimax optimality can be easily achieved with verysimple strategies (like max-diameter or BPI-UCRL), instance optimality requires careful algorithmicdesign.
Our coverage-based strategy is built around the idea of “uniformly” exploring the whole MDP,while using an elimination strategy to ensure that no ( s, a, h ) is sampled much more than what thelower bound prescribes.
Notably, this sampling rule is very simple, while exiting PAC RL algorithmswith instance-dependent complexity are all quite involved [44, 4].
Moreover, max-coverage samplingnaturally extends to stochastic MDPs, e.g., by doing optimistic planning on an MDP with a rewardfunction equal to 1 for under-sampled triplets and 0 for the others.
Finally, in our experiments onrandom instances, we observed that max-diameter is often comparable or better than max-coverage.
We leave as future work to investigate whether the latter is also provably near instance-optimal.
Computational aspects Our sampling rule requires solving one dynamic program per episode,which takes O ( N ) time.
The bottleneck is the elimination rule, which requires O ( N 2 ) per-episodetime complexity to solve one dynamic program for each active triplet.
However, we note thateliminations could be checked periodically (e.g., even at exponentially-separated times) withoutsignificantly compromising the sample complexity guarantees.
Instance optimality in stochastic MDPs The main open question is how to achieve (near) instanceoptimality for PAC RL in stochastic MDPs.
We believe that many of the results presented in thispaper could help in this direction.
First, our lower bound could be extended to the stochastic case bymodifying return gaps to include visitation probabilities and minimum flows to account for stochasticnavigation constraints.
Second, the maximum-coverage sampling rule easily extends to stochastic MDPs, while our elimination rule could be adapted by solving a constrained MDP problem [e.g.,18] to find the optimistic return of policies visiting a certain ( s, a, h ) with a least some probability.
Studying how these components behave in stochastic MDPs is an exciting direction for future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our
results are theoretical and the paper is not oriented towards a specific application, so
a
wider broader impact discussion is not applicable.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] These arestated in the preamble of every Theorem.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See theappendix.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See the appendix.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the appendix.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]