IMED-RL: Regret optimal learning of ergodic Markov decision processes
Abstract
We consider reinforcement learning in a discrete, undiscounted, infinite-horizon
Markov Decision Problem (MDP) under the average reward criterion, and focus on the minimization of the regret with respect to an optimal policy, when the learner does not know the rewards nor the transitions of the MDP.
In light of their success at regret minimization in multi-armed bandits, popular bandit strategies, such as the optimistic UCB , KL-UCB or the Bayesian Thompson sampling strategy, have been extended to the MDP setup.
Despite some key successes, existing strategies for solving this problem either fail to be provably asymptotically optimal, or suffer from prohibitive burn-in phase and computational complexity when implemented in practice.
In this work, we shed a novel light on regret minimization strategies, by extending to reinforcement learning the computationally appealing Indexed
Minimum Empirical Divergence ( IMED ) bandit algorithm.
Traditional asymptotic problem-dependent lower bounds on the regret are known under the assumption that the MDP is ergodic .
Under this assumption, we introduce IMED-RL and prove that its regret upper bound asymptotically matches the regret lower bound.
We discuss both the case when the supports of transitions are unknown, and the more informative but a priori harder-to-exploit-optimally case when they are known.
Rewards are assumed light-tailed, semi-bounded from above.
Last, we provide numerical illustrations on classical tabular MDPs, ergodic and communicating only, showing the competitiveness of IMED-RL in finite-time against state-of-the-art algorithms.
IMED-RL also benefits from a light complexity.
Introduction
We study Reinforcement Learning (RL) with an unknown finite Markov Decision Problem (MDP) under the average-reward criterion in which a learning algorithm interacts sequentially with the dynamical system, without any reset, in a single and infinite sequence of observations, actions, and rewards while trying to maximize its total accumulated rewards over time.
Formally, we consider a finite MDP M =
( S , A , p , r ) where S is the finite set of states, A
= ( A s ) s ∈S specifies the set of actions available in each state
and we introduce the set of pairs X M =
( s, a ) : s ∈ S , a ∈ A s } for convenience.
Further 1 , p : X M → P ( S ) is the transition distribution function and r : X M → P ( R ) the reward distribution function, with corresponding mean reward function denoted by m : X M → R .
An agent interacts with the MDP at discrete time steps t ∈ N ∗ and yields a random sequence ( s t , a t , r t ) t of states, actions, and rewards in the following way.
At each time step t , the agent observes the current state s t and decides the action a t to take based on s t and possibly past information, i.e. previous elements of the sequence.
After playing a t , it observes a reward r t ∼ r
( s t , a t ) , the current state of the MDP changes to s t +1 ∼ p ( ·| s t , a t ) and the agent proceeds sequentially.
In the average-
Given reward setting , one is interested in maximizing the limit, 1 T (cid:80)
Tt =1 r t , when T → ∞ , providing it exists.
This setting is a popular framework for studying sequential decision making problems; it can be traced back to seminal papers such as those of Graves and Lai [1997] and Burnetas and Katehakis [1997]
This theoretical framework allows to study the exploration-exploitation trade-off that arises from the sequential optimization problem a learner is trying to solve while being uncertain about the very problem it is optimizing.
In this paper, one is interested in developing a sampling strategy that is optimal amongst strategies that aim at maximizing the average-reward, i.e. balancing exploration and exploitation in an optimal way.
To assert optimality, we define the notion of regret and state a regret lower bound with the purpose of defining a theoretically sound notion of optimality that is problem-dependent .
While regret defines the discrepancy to optimality of a learning strategy, a problem-dependent regret lower bound will formally assess the minimal regret that any learning algorithm must incur on a given
MDP problem by computing a minimal rate of exploration.
Because this minimal rate of exploration depends on the problem, it is said to be problem-dependent, as opposed to worst case regret study that can exist in the MDP literature ( e.g. Jaksch et al.
Regret lower bounds currently exist in the literature when the MDP M is assumed to be ergodic 2 .
Hence we hereafter make this assumption, in order to be able to compare the regret of our algorithm to an optimal bound.
Similarly, to ensure fast enough convergence of the empirical estimate of the reward to the true mean, an assumption controlling the rate of convergence to the mean is necessary.
Assumption 1 (Light-tail rewards) .
For all x ∈ X M , the moment generating function of the reward exists in a neighborhood of 0 : ∃ λ x > 0 , ∀ λ ∈ R such that | λ | <
λ x , E R ∼ r ( x ) [exp( λR )] < ∞ .
Policy Regret and ergodicity are defined using properties of the set of stationary deterministic policies
Π( M ) on M .
On M , each stationary deterministic policy π : S → A s defines a Markov reward process, i.e. a Markov chain on S with kernel p π : s ∈ S
(cid:55)→ p ( ·| s, π ( s ))
∈ P ( S ) together with rewards r π : s ∈ S (cid:55)→ r
( s, π ( s ))
∈ P ( R ) and associated mean rewards m π : s ∈ S (cid:55)→ m ( s, π ( s )) ∈ R .
The t -steps transition kernel of π on M is denoted p tπ .
We denote p π = lim T →∞ 1 T
T (cid:80) t =
[Equation]
p
[Equation]
π : S → P ( S ) the Cesaro-average of p π .
A learning agent is executing a sequence of policies π t ∈ Π( M ) , t ⩾ 1 , where π t depends on past information ( s t ′ , a t ′ , r t ′ )
t ′ <t .
With a slight abuse of notation, a sequence of identical decision rules, π t = π for all t , is also denoted π .
Gain The cumulative reward (value) at time T , starting from an initial state s 1 of policy π = ( π t ) t is formally given by
[Equation]
For π ∈ Π( M ) , the average-reward 1 T V s 1 ( M , π, T ) tends to ( p π m ) ( s 1 ) as T → ∞ .
The gain of policy π ∈ Π( M ) , when starting from state s 1 is defined by g π ( s 1 ) =
( p π m )( s 1 ) and the optimal gain is defined as g ⋆ ( s 1 ) =
max π ∈ Π( M ) g π ( s 1 ) .
O s ( M )
= { π ∈ Π : g π ( s ) =
g ⋆ ( s ) } is the set of policies achieving maximal gain on M starting from state s .
Definition 1 (Regret) .
The regret at time T of a learning policy π = ( π t ) t starting at state s on an
MDP M is defined with respect to any π ⋆ ∈
O s ( M ) , as
[Equation]
In this paper, we aim to find a learning algorithm with asymptotic minimal regret.
The Lemmawill prove that for all optimal policies, π ⋆ , regrets are the same up to a bounded term that therefore does not count in asymptotic analysis.
Some authors such as Bourel et al.
[2020] define the regret as T g M ( s ) − V s ( M , π, T ) which is equal to the one we defined up to a bounded term (again by
Lemma 1).
No stationary policy can be optimal at all time and the important fact is that all those notions of regret induce the same asymptotic lower bound.
In the considered setting, the learning agent interacts with the MDP without any reset.
The minimal assumption would be to allow the agent to come back with positive probability from any initial mistake in finite time, so that the agent is not stuck in a sub-optimal area of the system.
This is assuming that the MDP is communicating , that is ∀ s, s ′ , ∃ π, t ∈ N : p
tπ ( s ′ | s ) > 0 .
However, in the literature, lower bounds on the regret are stated for MDPs satisfying a stronger assumption, ergodicity .
Since one is interested in crafting an algorithm matching a lower bound, we consider this stronger assumption.
Assumption 2 (Ergodic MDP) .
The MDP M is ergodic, that is ∀ s, s ′ , ∀ π,
[Equation]
N : p
[Equation]
( s
[Equation]
) > 0 .
Intuitively, this means that for all policies and all couples of states, there exists a finite trajectory of positive probability between the states.
Interestingly, the ergodic property can be assumed on the
MDP or on the set of policies in which we seek an optimal one.
For instance, in any communicating
MDP all ε -soft policies 3 are ergodic; more in the Experiment section 5 and
Appendix E.
Related work Had the MDP only one state, it would be a bandit problem.
Lower bound on the bandit regret and algorithms matching this lower bound, sometimes up to a constant factor, are well studied in the bandit literature.
Therefore, bandit sampling strategies with known theoretical guarantees have inspired RL algorithms.
The KL-UCB algorithm (Burnetas and Katehakis [1996],
Maillard et al.
[2011]), has inspired the strategy of the seminal paper of Burnetas and Katehakis [1997],
as well the more recent KL-UCRL strategy (Filippi et al.
Talebi and Maillard [2018]).
Inspired by the UCB algorithm (Agrawal [1995], Auer et al.
[2002]), a number of strategies implementing the optimism principle have emerged such as UCRL (Auer and Ortner [2006]), UCRL2 (Jaksch et al.
[2010]) and UCRL3 (Bourel et al.
[2020] (and beyond, Azar et al.
[2017], Dann et al.
[2017] for the related episodic setup).
The strategy PSRL (Osband et al.
[2013]) is inspired by Thompson sampling (Thompson [1933]).
Outline and contribution In this work, we build on the IMED strategy (Honda and Takemura [2015]), a bandit algorithm that benefits from practical and optimal guarantees but has never been used by the RL community.
We fill this gap by proposing the IMED-RL algorithm which we prove to be asymptotically optimal for the average-reward criterion.
We revisit the notion of skeleton (Equation 12) introduced in the seminal work of Burnetas and Katehakis [1997], with a subtle but key modification that prevents a prohibitive burn-in phase (see Appendix G for further details).
Further, this novel notion of skeleton enables IMED-RL to remove any tracking or hyperparameter and mimic a stochastic-policy-iteration-like algorithm.
Further, this skeleton scales naturally with the studied
MDP as it does not explicitly refer to absolute quantities such as the time.
We prove that our proposed
IMED-RL is asymptotically optimal and show its numerical competitivity.
Building on IMED , we make an additional assumption on the reward that is less restrictive than the common bounded reward hypothesis made in the RL community.
Assumption 3 (Semi-bounded rewards) .
For all x ∈ X , r ( x ) belongs to a subset F x ⊂ P ( R ) known to the learner.
There exists a known quantity m max ( x
) ∈ R such that for all x ∈ X , the support
Supp ( r ( x )) of the reward distribution is semi-bounded from above, Supp ( r ( x ))
⊂ ]
−∞ , m max ( x )] , and its mean satisfies m ( x ) < m max ( x ) .
Ergodic assumption While many recent works focused on worst-case regret bounds only (e.g.
Domingues et al.
[2021], Zanette and Brunskill [2019], Jin et al.
[2018] and citations therein), studying problem-dependent optimal regret bounds has been somewhat overlooked.
Being more general is always more appealing but the restriction from communicating MDPs to ergodic MDPs allows us to target exact asymptotic optimality ; not just bound, not just worst-case bound.
Ergodic MDPs is the only case in which explicit problem-dependent lower bounds are known and hence can be directly used to build a strategy.
Indeed, the main challenge towards problem-dependent optimality is that existing lower bounds for exploration problems in MDPs are usually written in terms of non-convex optimization problems.
This implicit form makes it hard to understand the actual complexity of the setting and, thus, to design optimal algorithms.
Existing proof strategies for state-of-the-art algorithms ( UCRL , PSRL , etc ) ensure a regret for communicating MDPs but fail to provide optimality guarantees even in the ergodic case.
We believe that deriving a sharp result in the ergodic case might prove to be insightful to pave the way towards the communicating case.
From a theoretical standpoint, related to UCRL type strategy, modern analysis of KL-UCRL by Talebi and Maillard [2018] also makes the ergodic assumption.
This hypothesis has also been used in the theoretical work of
Tewari and Bartlett [2007] and the work of Ok et al.
[2018] that concerns structured MDPs.
Related to this assumption are works that are interested in identification and sample complexity.
Wang [2017] introduced a primal-dual method to compute an ε -optimal policy and bound the number of sample transitions to reach this goal.
Jin and Sidford [2020] relaxed the ergodic hypothesis by using a mixing hypothesis that implies the uniqueness of recurrent class for each policy.
In this setting, the authors also derive a bound on the number of samples to compute an ε -optimal policy.
Regret lower bound
In this section, we recall the regret lower bound for ergodic MDPs and provide a few insights about it.
Characterizing optimal policies Relying on classical results that can be found in the books of
Puterman [1994] and Hernández-Lerma and Lasserre [1996], we give a useful characterization of optimal policies that is used to derive a regret lower bound.
Under the ergodic Assumption 2 of MDP
M , for all policy π ∈ Π( M ) , the gain is independent from the initial state, i.e. g π ( s ) =
g π ( s ′ ) for all states s and s ′ , and we denote it g π .
Similarly, the set of optimal policies O ( M ) is state-independent since O s ( M ) =
O s ′ ( M ) .
Any policy π satisfy the following fixed point property
[Equation]
Poisson
[Equation]
where b π : S → R is called the bias function and is defined up to an additive constant by b π ( s )
= (cid:16) ∞
[Equation]
− p π ) m π (cid:17) ( s ) .
We highlight that bias plays a role similar to the value function in the discounted reward setting in which the gain is always zero and Equation 3 reduces to the Bellman equation, giving a direction in which extend our results to this other RL setting.
Interestingly, for any communicating and a fortiori ergodic MDP, the span S ( b π ) = max s ∈S b π ( s ) − min s ∈S b π ( s ) of the bias function of any policy is bounded, which allows to decompose the regret in the useful following way.
Lemma 1 (Regret decomposition) .
Under the ergodic assumption 2, for all optimal policy ⋆ ∈ O ( M ) , the regret of any policy π = ( π t ) t can be decomposed as
[Equation]
where N s,a ( T ) =
(cid:80) Tt =1 1 { s t =
s, a t = a } counts the number of time the state-action pair ( s, a ) has been sampled and ∆ s,a ( M ) is the sub-optimality gap of the state-action pair ( s, a ) in M , ∆ s,a ( M )
= m ( s, a ) + p a b ⋆ ( s ) − m ⋆ ( s ) − p ⋆ b ⋆ ( s ) = m ( s, a ) + p a b ⋆ ( s )
− g ⋆ − b
⋆
( s ) (5) with p a = p ( ·| s, a ) by a slight abuse of notation.
Action a ∈
A s is optimal if and only if ∆ s,a ( M ) = 0 , otherwise, it is said sub-optimal.
This result can be found in Puterman [1994] and is rederived in Appendix C.
Under the ergodic Assumption 2 of MDP M , all optimal policies satisfy a Poisson equation while some are also being characterized by the optimal Poisson equation (see Hernández-Lerma and Lasserre [1996]), used to compute the optimal gain and a bias function associated to an optimal policy,
[Equation]
Lower bound To assess the minimal sampling complexity of a sub-optimal state action pair, one must compute how far a sub-optimal state-action pair is from being optimal from an information point-of-view.
A sub-optimal state-action pair ( s, a ) ∈
X M is said to be critical if it can be made optimal by changing reward r ( s, a ) and transition p ( ·| s, a ) while respecting the assumptions on the rewards and transitions.
Formally, let φ M : P ( R × S )
→ R ,
[Equation]
denotes the potential function of ν ⊗ q in M , where ν ⊗ q is the product measure of ν and q .
A pair ( s, a ) ∈
X M is critical if it is sub-optimal and there exists ν ∈ F s,a and q ∈ P ( S ) such that
[Equation]
where γ s ( M ) def = g M + b M ( s ) .
[Equation]
Note that γ s ( M ) = max a ∈A s φ M ( r ( s, a ) ⊗ p ( s, a )) by the optimal Poisson equation (6).
Definition 2 (Sub-optimality cost) .
The sub-optimality cost of a sub-optimal state-action pair ( s, a ) ∈
X M is defined as K s,a ( M ) def = K s,a ( M , γ s ( M ))
where
[Equation]
and KL denotes the Kullback-Leibler divergence between distributions.
A lower bound on the regret may now be stated for a certain class of learner, the set of uniformly consistent learning algorithm, i.e. those policies π = ( π t ) t such that E π, M ( N s,a ( T ))
=
o ( T α ) for all sub-optimal state-action pair ( s, a ) and 0 < α < 1 (see Agrawal et al.
Theorem 1 (Regret lower bound Burnetas and Katehakis [1997]) .
Let M =
( S , A , p , r ) be an MDP satisfying Assumptions 1, 2, 3.
For all uniformly consistent learning algorithm π ,
[Equation]
with the convention that 1 / ∞ = 0 .
The regret lower bound is
[Equation]
where C ( M ) = (cid:8) ( s, a ) : 0 < K s,a ( M ) < ∞ (cid:9) is called the set of critical state-action pairs.
Those are the state-action pairs ( s, a ) that could be confused for an optimal one if we were to change their associated rewards and transitions distributions at the displacement cost of K s,a ( M ) .
The IMED-RL Algorithm
In this section we introduce and detail the IMED-RL algorithm, whose regret matches this fundamental lower bound and extends the IMED strategy from Honda and Takemura [2015] to ergodic MDPs.
Indeed, for a single-state MDP, that is a multi-armed bandit, IMED-RL simply reduces to IMED .
Empirical quantities IMED-RL is a model-based algorithm that keeps empirical estimates of the transitions p and rewards r as opposed to model-free algorithm such as Q-learning.
We denote by ˆ r t ( s, a )
=
ˆ r
( s, a ; N s,a ( t )) and ˆ p t ( s, a )
=
ˆ p ( s, a ; N s,a ( t ))
the empirical reward distributions and transition vectors after t time steps, i.e. using N s,a ( t ) samples from the distribution r ( s, a ) .
Initially, ˆ p ( s, a ; 0) is the uniform probability over the state space and ˆ p ( s, a ; k ) =
(1 − 1 /k )
ˆ p ( s, a ; k − 1) + (1 /k ) s k , where s k is a vector of zeros except for a one at index s k , the k th samples drawn from p ( ·| s, a ) .
This defines at each time step t an empirical MDP (cid:99) M t =
( S , A , ˆ p t , ˆ r t ) .
On this empirical
MDP, for each state, some actions have been sampled more than others and their empirical quantities are therefore better estimated.
We call skeleton at time t the subset of state-action pairs that can be considered sampled enough at time t ; it is defined by restricting A s to A s ( t ) for all state s ∈ S , with
[Equation]
Since x> log 2 x , A s ( t ) ̸ = ∅ , hence A ( t ) =
( A s ( t ))
s contains at least one deterministic policy.
We note that the MDP M ( A ( t )) def =
( S , A ( t ) , p , r ) defined by restricting the set of actions to A ( t ) ⊆
A is an ergodic MDP.
The restricted empirical MDP (cid:99) M t ( A ( t )) def =
( S , A ( t ) , ˆ p t , ˆ r t ) also is ergodic thanks to the ergodic initialization of the estimate ˆ p .
Inspired by IMED , we define the IMED-RL index.
Definition 3 ( IMED-RL index) .
For all state-action pairs ( s, a ) ∈ X M , let us define K s,a ( t ) def =
K s,a (cid:16)(cid:99)
M t ( A ( t )) , ˆ γ s ( t ) (cid:17) with empirical threshold ˆ γ s ( t )
[Equation]
max
[Equation]
[Equation] [Equation]
( s,
[Equation] [Equation]
ˆ p ( s, a ))
Then, the IMED-RL index of ( s, a ) at time t , H s,a ( t ) , is defined as
[Equation]
Note that ˆ γ s ( t ) ̸
= γ s ( ˆ M t ( A ( t )))
as the maximum is taken over all a ∈A s an not just a ∈A s ( t ) .
Known support of transitions Were the support of transition known, the infimum in sub-optimality cost K s,a defined by Equation 9 would be redefined as one over the set { q ∈P ( S ) :
Supp ( q ) =
Supp ( p ( ·| s, a )) } , modifying both the lower bound and IMED-RL index.
IMED-RL algorithm
The IMED-RL algorithm consists in playing at each time step t , an action a t of minimal IMED-RL index at the current state s t .
The intuition behind the IMED-RL index is similar to the one of the IMED index for bandits and stems from an information theoretic point-of-view of the lower bound.
At a given time t , the frequency of play N s,a ( t ) N s ( t ) of action a ∈
A s in state s ∈ S , should be larger than or equal to its posterior probability of being the optimal action in that state, exp ( − N s,a ( t ) K s,a ( t )) , that is to say N s,a ( t ) N s ( t ) ⩾ exp ( − N s,a ( t ) K s,a ( t )) .
Taking the logarithm and rearranging the terms, this condition rewrites H s,a ( t ) ⩾ log N s ( t ) at each time step t .
The action that is the closest to violate this condition or that violates this condition the most is the one of minimal
IMED-RL index, arg min a H s,a ( t ) , the one IMED-RL decides to play.
Algorithm
Require: State-Action space X M of MDP M , Assumptions 1, 2,
Require:
Initial state sfor t ⩾ 1 do
Sample a t ∈ arg min a ∈A st H s,a ( t ) end for
Intuitions of the IMED-RL algorithm root to the control theory of MDPs and optimal bandit theory;
IMED-RL intertwines the two and the regret proof exactly follows from the following intuitions.
Control
In control theory, we assume that both the expected rewards and transitions probabilities of an MDP M are known.
Policy iteration (see Puterman [1994], Bertsekas and Shreve [1978]) is an algorithm that computes a sequence ( π n ) n of deterministic policies that are increasingly strictly better until an optimal policy is reached.
In the average-reward setting and under the ergodic assumption, a policy π is strictly better than another policy π ′ if g π ( M ) > g π ′ ( M ) .
The policy iteration algorithm computes the sequence of policies recursively in the following way.
Initially, an arbitrary deterministic policy π 0 is chosen.
At step n + 1 ∈ N ∗ , it computes m π n and b π n then swipes through the states s ∈ S in an arbitrary order until it reaches one state s such that there exists a ∈ A ( s ) with m ( s, a ) + p ( ·| s, a ) b π n > m π n ( s ) + p π ( s ) b π n .
If such an s does not exist, then it returns π n as an optimal policy.
Otherwise, π n +1 is defined as π n +1 ( s ′ )
=
π n ( s ′ ) for all s ̸ = s ′ and π n +1 ( s ) ∈ arg max { m ( s, a ) + p ( ·| s, a )
b π n } .
Such a step is called a policy improvement step.
Policy iteration is guaranteed to finish in a finite number as the cardinal of Π( M ) is finite.
At each step n ∈ N ∗ , φ M ( π n ) is a local function that takes into account the whole dynamic of the MDP and allows to compute, via an argmax , an optimal choice of improvement (or optimal action) based on local information; φ M ( π n ) ( r ( s, a ) ⊗ p ( ·| s, a ))
= m ( s, a )
+ p ( s, a ) b π n .
IMED-RL uses φ (cid:99) M ( A ( t )) and improves the skeleton similarly to policy iteration as it can be seen in the analysis 4.
Bandit control A degenerate case of MDP would be one where there is only one state s with φ M ( φ ) ( r ( s, a ))
= m ( s, a ) by choosing the bias function to be zero 6 .
Playing optimally consists in playing an action with largest expected reward at each time step t , a t ∈ arg max a ∈A s m ( s, a ) .
Bandit Learning occurs when rewards are unknown; this is the bandit problem.
In that case, a lower bound on the regret similar to 1 exists.
Under some assumptions on the reward distributions, optimal algorithms whose regret upper bounds asymptotically match the lower bound can derived.
IMED Honda and Takemura [2015], KL-UCB Maillard et al.
[2011], Cappé et al.
[2013] are two such examples that use indexes, i.e. computes a number I s,a ( t ) at each time step and play a t ∈ arg min
I s,a ( t ) .
Such indexes are crafted to correctly handle the exploration-exploitation trade-off.
RL in Ergodic MDPs The delayed rewards caused by the dynamic of the system is the main source of difficulty arising from having more than one state.
IMED-RL combines control and bandit theory in the following way.
At each time step t , a restricted MDP (cid:99)
M t ( A ( t )) is built from the empirical one (cid:99) M t .
If the condition to belong to the skeleton is selective enough, then the potentials on the restricted empirical MDP (cid:99) M t ( A ( t )) may become close to those of the restricted true MDP M ( A ( t )) , that is ∥ φ (cid:99) M t
( A ( t )) −
φ M ( A ( t ))
∥ ∞ is small.
We want to make policy improvements by finding, at each state s an action a ′ ∈ arg max φ M ( A ( t ))
( r ( s, a ) ⊗ p ( ·| s, a )) , play it enough that it belongs to the skeleton which will modify φ and repeat until φ M ( A ( t )) =
φ M .
Using φ , the global dynamic is reduced to a local function so that at each state, the agent is presented a bandit problem.
This bandit problem is well estimated if ∥ φ (cid:99) M t ( A ( t )) −
φ M ( A ( t ))
∥ ∞ is small.
As opposed to the control setting, the learning agent cannot choose the state in which to make the policy improvement step and it may be possible that no policy improvement step is possible at state s t .
However, thanks to the ergodic assumptionthe agent is guaranteed to visit such a state in finite time, if it exists.
There is a trade-off between the adptativity of the skeleton, i.e. how quickly one can add an improving action to define a new φ , and concentration of statistical quantities defined on the restricted MDP.
Related work Our notion of skeleton is built on the work of Burnetas and Katehakis [1997].
We improve on their original notion of skeleton by correcting some troubles happening in the small sample regime.
In particular, this forces the authors to introduce some forcing mechanism.
The issues of the original definition and improvement induced by ours are listed in Appendix G. One key point of our definition is that the skeleton is defined using only empirical quantities, the number of samples, and does not depends on some arbitrary reference, such as the absolute time.
Regret of IMED-RL
In this section we state the main theoretical result of this paper, which consists in the IMED-RL regret upper bound.
We then sketch a few key ingredients of the proof.
Theorem 2 (Regret upper bound for Ergodic MDPs) .
Let M =
( S , A , p , r ) be an MDP satisfying assumptions 1, 2, 3.
Let 0 < ε ⩽ 13 min π ∈ Π( M ) min
( s,a )
∈X M {| ∆ s,a ( M ( π ))
| : | ∆ s,a ( M ( π )) | > 0 } .
The regret of IMED-RL is upper bounded,
[Equation]
where Γ s ( M ) is constant that depends on the MDP M and state s ; it is made explicit in the proof detailed in Appendix D. A Taylor expansion allows to write the regret upper bound as
[Equation]
Were the semi-bounded reward assumption changed to a bounded reward one with known upper and lower bound, the O (cid:16) (log T ) 10 / 11 (cid:17) could be made a O (1) as explained in Appendix E.
Theorem 3 (Asymptotic Optimality) .
IMED-RL is asymptotically optimal, that is,
[Equation]
The proof of Theorem 3 is immediate from Theorem 2 by first dividing Equation 14 by log T , then by taking the limit T → ∞ , and finally taking the limit ε → 0 .
Remark While the regret lower bound , Theorem 1, is asymptotic by nature, our main Theoremstates a finite time upper bound on the regret of IMED-RL .
Indeed, both Equations 14 and 15 are valid for all time T .
The term O (1) appearing in Equation 14 does not depend on time T and is a constant that depends on both the MDP M and ε .
This dependency is hard to be made explicit as this term is computed as limits of convergent series that are derived in the proof, see Appendix D. In Equation 14, the constant (cid:80) ( s,a ) ∈C ( M )
[Equation]
M ) in front of log T does not exactly match the asymptotic upper bound (cid:80) ( s,a ) ∈C ( M ) ∆ s,a ( M ) K s,a ( M ) because of the ε -term in the denominators.
Equation 15 states that using a bounded reward hypothesis, instead of semi-bounded, allows the constant in front of the leading log T term to exactly match the asymptotic one, even in the finite time regret upper bound.
In both cases, Theorem 3 states that asymptotic optimality is achieved.
This Theorem proves the optimality of IMED-RL since the upper bound on the regret matches the lower bound of Theorem 1.
Such a bound was asymptotically matched by the algorithm proposed by
Burnetas and Katehakis [1997] and we recall that this algorithm and its problems are discussed in
Appendix G.
On the other hand, the current state-of-the-art algorithms UCRL3 and PSRL , while having some theoretical guarantees, have not been proved to match the regret lower bound.
On the practical side, Q-learning is often used without much theoretical guarantee because of its usually strong practical performances.
In the experiments, we will compare IMED-RL to those three algorithms.
Related work Theorems 2 and 3 prove that IMED-RL is achieving the optimal rate of exploration (in the exploitation-exploration tradeoff sense) for ergodic MDPs.
Its theoretical guarantees are problem-dependent rather than worst-case/min-max.
Comparing to the log T bound derived for
UCRL in Theorem 4 of Jaksch et al.
[2010], less known than the √ T bound, shows the benefit of our analysis for each instance, as we improve the constant factors in the leading terms: their dependency is 34 D 2 S 2 A/ ∆ , where ∆ is a sub-optimality gap and D the diameter of the MDP.
Sketch of proof Though a full proof is given in Appendix D, we sketch here the main proof ideas that follow directly from the intuitions behind the IMED-RL conception.
The regret is decomposed into two terms, the bandit term when the local bandit problems defined by φ (cid:99) M t ( A ( t )) is well estimated, and the skeleton improvement term that controls the probability that the local bandit problem is not well estimated.
This second term is managed by controlling the number of policy improvement steps and using concentration properties of empirical quantities defined on the skeleton.
The main Theorem 2 follows from the following proposition that is proved in Appendix D. Recall from Lemma 1 that for all state-action pair x ∈ X M , N x ( T )
=
(cid:80) Tt =1 1 { ( s t , a t ) =
x } counts the number of time the state-action pair x has been sampled.
Proposition 1.
For all state-action pair x ∈ X M , for all ε > 0 ,
[Equation]
where we introduced the bandit term, B x ( T ) , and the skeleton improvement term, S ( T ) ,
[Equation]
Furthermore, E ( S ( T )) =
O (1) , E ( B x ( T ))
=
O (1) for a non-critical state-action pair, while for a critical state-action pair x ,
[Equation]
Numerical experiments
In this section, we discuss the practical implementation and numerical aspects of IMED-RL and extend the discussion in Appendix F. Source code is available on github 7 .
Computing IMED-RL index At each time step, we run the value iteration algorithm on (cid:99) M t ( A ( t )) to compute the optimal bias and the associated potential function φ (cid:99) M t
( A ( t )) .
This task is standard.
Once done, one must compute the value of the optimization problem K s,a ( t ) which belongs to the category of convex optimization problem with linear constraint.
Such problems have been studied under the name of partially-finite convex optimization , e.g. in Borwein and Lewis [1991].
It is possible to compute K s,a ( t ) by considering the Legendre-Fenchel dual and one does not need to compute the optimal distribution to know the value of the optimization problem.
Proposition 2 (Index computation, Honda and Takemura [2015] Theorem 2) .
Let ( s, a ) be in X M ,
M = m max ( s, a ) + max s ′ ∈S b M ( s ) , and γ > φ M ( r ( s, a ) ⊗ p ( ·| s, a )) , then K s,a ( M , γ ) =
[Equation]
If γ ⩽ φ M ( r ( s, a ) ⊗ p ( ·| s, a )) , then K s,a ( M , γ ) = 0 .
In particular, this Proposition 2 sometimes allows to write K s,a ( t ) almost in close form, e.g. when
F s,a defined in Asumptions 3 is a set of multinomials with unknown support (and only the upper bound m max is known).
In Appendix F, we discuss this numerical computation further.
Computational complexity
In terms of state and actions spaces sizes, the complexity of IMED-RL at each time step scales as O ( S 2 A ) , the complexity of value iteration.
Indeed, at each time step,
IMED-RL runs value iteration using actions available in the skeleton, then computes the indexes of the available actions at the current state, and finally pick an argmin .
The complexity of value iteration is O ( S 2 A ) , the complexity of computing the A necessary indexes is O ( SA ) , and the complexity of picking an argmin amongst those A indexes is O
( A ) .
Therefore, the per-time-step complexity of
IMED-RL scales as O ( S 2 A ) .
However, this scaling is mainly an upper-bound as value iteration is run with actions that are within the skeleton.
By design of the skeleton, we experimentally observe that, after some time, the skeleton contains one action per state (the optimal one).
We provide more details in Appendix F, Lazy update paragraph.
Practical comparison In practice, most of the complexity of IMED-RL is in the analysis rather than in the algorithm: compared to PSRL and UCRL3 , IMED-RL does not take a confidence parameter nor any hyperparameter.
Also, IMED-RL uses value iteration as a routine, which is faster than the extended value iteration used in UCRL3 .
Q-learning technically takes an exploration parameter ( ε -greedy exploration) or exploration scheme when it is slowly decreased with time.
Environments In different environments, we illustrate in Figure 2 and Figure 3 the performance of IMED-RL against the strategies UCRL3 Bourel et al.
[2020], PSRL Osband et al.
[2013] and Qlearning (run with discount γ = 0 .
99 and optimistic initialization).
As stated during the introduction, any finite communicating MDP can be turned into an ergodic one, since on such MDPs, any stochastic policy π : S → P ( A s ) with full support Supp ( π ( s ))
=
A s is ergodic.
Hence by mixing its transition p with that obtained from playing a uniform policy, formally p ε ( ·| s, a ) =
(1 − ε ) p ( ·| s, a ) + ε (cid:80) a ′ ∈A s p ( ·| s, a ′ ) / |A
s | , for an arbitrarily small ε > 0 one obtain an ergodic MDP.
In the experiments, we consider an ergodic version of the classical n -state river-swim environment, 2 -room and 4 -room with ε = 10 − 3 , and classical communicating versions ( ε = 0 ).
[Figure]
n-states RiverSwim environment As illustrated by Figure 2, the performances of IMED-RL are particularly good and the regret of IMED-RL is below the regrets of all its competitors,
even when the
MDP is communicating only.
This numerical performance grounds numerically the previous theoretical analysis.
While using IMED-RL in communicating MDPs is not endorsed by our theoretically analysis, it is interesting to see how much this hypothesis amounts in the numerical performances of
IMED-RL .
We therefore ran an experiment on another classical environment, 2-rooms.
n-rooms environment As illustrated by Figure 3, the performances of IMED-RL are particularly good, even surprisingly good, in this communicating only environment.
Those experiments are a clue that the IMED-RL strategy may still be reasonable, although not necessarily optimal in some communicating MDPs.
All experiments take less than an hour to run on a standard CPU.
Future work Although not intended for non-ergodic MDPs, IMED-RL exhibits state-of-the-art numerical performances in communicating only MDPs (see Appendix F.2 for additional experiments).
Hence, IMED-RL might prove to be insightful to pave the way towards the communicating case as it seems possible to get a controlled regret also in the case of communicating MDPs.
Both the problem-dependent and worst-case regret bounds are interesting in this regard.
Another direction we intend to explore is the adaptation of IMED-RL main ideas to function approximation frameworks, such as neural networks and kernel methods.
Conclusion
In this paper, we introduced IMED-RL , a numerically efficient algorithm to solve the average-reward criterion problem under the ergodic assumption for which we derive an upper bound on the regret matching the known regret lower bound.
Further, its surprisingly good numerical performances in communicating only MDPs open the path to future work in MDPs that are communicating only.
Acknowledgments and Disclosure of Funding
This work has been supported by the French Ministry of Higher Education and Research, Inria, Scool, the Hauts-de-France region, the MEL and the I-Site ULNE regarding project R-PILOTE-19-004APPRENF
The PhD of Fabien Pesquerel grant
École Normale Supérieure.
A Table of contents
In the following appendices, we recall and introduce notations (Appendix B) useful for understanding
IMED-RL and the proof of the regret upper-bound given in Appendix D. Before that, we introduce some technical results in Appendix C that will help for the main proof and increase its readability.
In Appendix E, we discuss the ergodic Assumption 2 made in the paper, followed in Appendix
F in which we experiments more with IMED-RL and especially, beyond the ergodic assumption, towards the communication one.
Finally, we detail in a small note why our modification of the skeleton’s notion introduced by Burnetas and Katehakis [1997] was key to the empirical success of our proposed IMED-RL .
1 Introduction
2 Regret lower bound
The IMED-RL Algorithm 4 Regret of IMED-RL 5 Numerical experiments
A
Table of
contents
B Notations
C Technical results
D Proof of Theorem (REGRET)
D.1 Bandit term . .
D.1.
The threshold is not underestimated . . . . . . . . . . . . . . . . . . . . .
D.1.
The threshold is underestimated . . . . . . . . . . . . . . . . . . . . . . .
D.1.3 Bandit term: upper bound . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Skeleton improvement term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2.
Skeleton improvement: upper
bound . .
D.
Regret upper bound
E Assumptions
F.
Solving the optimization problem K s,a . . . . . . . . . . . . . . . . . . . . . . .
F.
Additional experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Notations
Notations of exact quantities
M = ( S , A , p , r } , a MDP
S , state space,
A =
( A s ) s ∈S , action space
X M = { ( s, a ) : s ∈ S , a ∈ A s } , state-action space p : X M → P ( S ) , transition distribution r : X M → P ( R ) , reward distribution m ( s, a ) =
E r ∼ r
( s,a )
[ r ] , mean reward function Π ( M )
= { π : s ∈ S
(cid:55)→ π ( s ) ∈
A s } , stationary deterministic policy space ∀ π ∈ Π( M ) , p π : s ∈ S (cid:55)→ p ( ·| s, π ( s )) =
p ( s, π ( s ))
∈ P ( S ) ,
r π : s ∈ S (cid:55)→ r
( s, π ( s ))
∈ P ( R ) , m π : s ∈ S
(cid:55)→ m ( s, π ( s )) ∈ R , MDP related quantities defined by a policy π
M ( D ) =
( S , D , p , r ) , MDP M with action space restricted to D b π : s ∈ S (cid:55)→ b π ( s ) ∈ R , bias function as defined in Poisson Equationg ⋆ = g M , optimal gain on MDP M b M , optimal bias function as defined in the optimal Poisson Equation∆ s,a ( M )
= m ( s, a ) + p ( s, a ) b M − g M − b M ( s ) , sub-optimality gap γ s ( M )
=
g M + b M ( s ) , optimality threshold φ M ( ν ⊗ q ) =
E R ∼
ν
[ R ] + q b M , potential function
O s ( M ) = { a ∈ A s : φ M ( r ( s, a ) ⊗ p ( s, a ) =
γ s ( M )) } , optimal action in state s
K s,a ( M , γ ) = inf ν ∈F s,a q ∈P ( S ) { KL ( r ( s, a ) ⊗ p ( ·| s, a ) ,
ν ⊗ q ) :
φ M ( ν ⊗ q ) > γ } , sub-optimality cost for threshold γ
K s,a ( M ) =
K s,a ( M , γ s ( M )) , sub-optimality cost
C ( M ) =
(cid:8) ( s, a ) : 0 < K s,a ( M ) < ∞ (cid:9) , set of critical state-action pairs
C sa, M ( F sa ⊗ P ( S ))
= { ν ⊗ q ∈ F sa ⊗ P ( S ) :
φ M ( ν ⊗ q ) > γ s ( M ) } , set of critical distributions
C sa,
[Equation]
F
[Equation] [Equation]
ν
[Equation]
> γ
[Equation]
M ) − ε }
Notations of empirical quantities
N s,a ( T ) =
(cid:80) Tt =1 1 { s t =
s, a t = a } , counting random variable for state-action pair ( s, a )
N s ( T ) =
(cid:80) Tt =1 1 { s t = s } , counting random variable for state s
A s ( t ) = (cid:8) a ∈
A s : N s,a ( t ) ⩾ log 2 (max a ′ ∈A s N s,a ′ ( t ))
(cid:9) , skeleton at state s at time t (this is a random variable)
ˆ γ s ( t ) def = max a ∈A s φ ˆ M t
( A ( t ))
(ˆ r ( s, a ) ⊗ ˆ p ( s, a )) , empirical optimality threshold
K s,a ( t ) def = K s,a (cid:16) (cid:99) M t ( A ( t )) ,
ˆ γ s ( t ) (cid:17) , empirical sub-optimality cost
H s,a ( t ) =
N s,a ( t ) K s,a ( t ) + log N s,a ( t ) , IMED-RL index
R s 1 ( M , π, T ) =
(cid:80) x ∈X M E π,s 1 [ N x ( T )]
∆ x ( M ) + (cid:16)(cid:104)(cid:81) Tt =1 p π t − p t⋆ (cid:105) b M (cid:17) ( s 1 ) , regret of a policy π (with ⋆ ∈ O ( M ) , expression proved in Lemma 1)
Useful notations in proofs Π =
| Π( M ) | , cardinal of policy space σ Π+1 t = t + 1 , σ νt =
σ ν +1 t − ⌊ t Π+1 ⌋ for all ν ∈ { 1 , · · · , Π } and σ 0 t = 1 , boarders of a
M -adapted sub-division (see Definition 5)
[Equation]
sub-
interval ν
N
[Equation]
k
[Equation]
1 { s t = s } , number of visits state at s during sub-interval I νt 0 < κ < 1Π+ 0 < β < β M , where β M is defined in Proposition
V
[Equation]
s
[Equation]
{ N νs ( t ) ⩾ κβt } , an event controlling the number of visit in sub-interval ν
[Equation]
ν =
[Equation]
V νt , an event controlling the number of visits in each sub-interval
S
[Equation]
∥ ˆ p
[Equation]
( x ) − p ( x ) ∥ ∞
⩽ δ, | ˆ m k
( x ) − m ( x ) | ⩽ δ (cid:9) , an event controlling the precision on the empirically restricted MDP during interval ν
S t ( δ ) =
(cid:84) Π ν =1 S νt , an event controlling the precision in each sub-interval but the first
I ( s, M , γ ) =
{ a ∈ A s : φ M ( r ( s, a ) ⊗ p ( s, a )) >
γ } x ∗ sa, M = arg
[Equation]
, where B = m max ( s, a ) + max s ′ ∈S b M ( s ) λ sa, M ,ε = sup (cid:40)
[Equation]
⩾
[Equation]
(cid:41) , where B = m max ( s, a ) + max s ′ ∈S b M ( s ) and µ s,a = φ M ( r ( s, a ) ⊗ p ( s, a ))
Λ ∗ sa, M ( x ) =
sup λ (cid:40) λ (
[Equation]
s,a
[Equation]
R
[Equation]
(cid:2)
[Equation]
(cid:0) λ (
[Equation]
+ b
[Equation]
( S )) (cid:1)(cid:3)(cid:41) , where µ s,a = φ M ( r ( s, a ) ⊗ p ( s, a ))
˜Λ ∗ sa, M ( x ) = sup λ (cid:40) λx − log
[Equation]
R ∼
[Equation]
) (cid:104)(cid:0)
[Equation] [Equation]
γ
[Equation]
M
[Equation]
(cid:1) x
[Equation]
(cid:1) λ (cid:105)(cid:41)
K εs,a ( t ) def = inf ν ∈F s,a q ∈P ( S ) { KL (ˆ r t ( s, a ) ⊗ ˆ p t ( s, a ) ,
ν ⊗ q ) :
φ M ( ν ⊗ q ) > γ s ( M ) − ε }
K ε,ns,a def = inf ν ∈F s,a q ∈P ( S ) { KL (ˆ r ( s, a, n ) ⊗ ˆ p t ( s, a, n ) , ν ⊗ q ) :
φ M ( ν ⊗ q ) > γ s ( M ) − ε }
C Technical results
In this section we state a few lemmas and define a few objects that are useful for the regret analysis of IMED-RL .
More precisely,
Lemma 1 is about expressing the regret with respect to the number of pulls N x ( t ) of sub-optimal state-action pairs x ∈ X M and show that ∆ x ( M ) indeed is roughly the cost an agent suffer each time it plays it.
Thanks to this Lemma, controlling the regret is equivalent to controlling N x ( T ) for each sub-optimal state-action pair x ∈ X M , which we do in Appendix D.
Definition 4 introduces notations used for expressing concentration results of events studied in the main proof (see Appendix D).
Those are inspired from Honda and Takemura [2015] and lies on the assumptions made in this paper.
Lemma 2 states that under the ergodic Assumption 2, any learning agent is guaranteed to visit every states sufficiently often with a large enough probability.
This lemma is proved in Proposition 2 of
Burnetas and Katehakis [1997].
Definition 5 introduces the notion of M -adapted sub-division that split interval [0 , t ] into Π +sub-intervals.
This notion follows from the developed intuition about Policy Improvement and we will prove that with high probability, a policy improvement occurs (if possible) in-between each sub-interval.
Lemma 3 uses the previous lemma to derive a stronger result by proving that a linear number of visits in each state and in each linear sub-interval of a M -adapted subdivision can be obtained with high probability.
It is useful to derive improvement of the skeleton between the beginning and end of a sub-interval.
Lemma 4 states that, by definition of the skeleton, by Lemma 3, and by the reward Assumptions 1, 3, empirical quantities defined on the MDP restricted by the skeleton are well approximated.
Lemma 5 expresses how well the gain g π and bias b π of every policy π ∈ Π( M ) can be estimated given a precision on the mean rewards and transitions of the original MDP.
Lemma 6 expresses how well the optimal gain g M and optimal bias b M defined by the optimal
Poisson Equation 6 can be estimated given a precision on the mean rewards and transitions of the original MDP.
Lemma 7 expresses how well the optimal gain g M ( A ( t )) and optimal bias b M ( A ( t )) defined by the optimal Poisson Equation 6 can be estimated given a precision on the mean rewards and transitions of the MDP M ( A ( t )) prescribed by a good event.
Lemma 1 (Regret decomposition) .
Under the ergodic assumption 2, for all optimal policy ⋆ ∈ O ( M ) , the regret of any policy π = ( π t ) t can be decomposed as
[Equation]
Proof of Lemma 1.
It holds by the Poisson equation that m π
= g π
+ ( I − p π ) b π .
Hence, the cumulative reward of a strategy playing policy π t at time t until time T and starting from state s 1 is given by
[Equation]
Under the ergodic Assumption 2, for all optimal policy ⋆ , g ⋆ ( s 1 ) takes the same value g ⋆ for all state s 1 .
In this case, ( p π g ⋆ )( s 1 ) =
g ⋆ for all π, s 1 .
Using this property it comes
[Equation]
At this point
[Equation]
we note that
[Equation]
) =
E π [∆( s, a )] .
To conclude, we note that
[Equation]
Definition 4.
Let M be an MDP satisfying Assumption 2 and whose reward distribution r satisfy
Assumptions 1 and
[Equation]
then the following quantities 8 are well defined,
[Equation]
where B = m max ( s, a ) + max s ′ ∈S b M ( s ) and µ sa =
φ M ( r ( s, a ) ⊗ p ( s, a )) .
By Sections 6 and 7 of
Honda and Takemura [2015], x ∗ sa, M exists uniquely when φ M ( r ( s, a ) ⊗ p ( s, a ))
⩽ γ s ( M ) and λ sa,ε > 1 for ε > 0 .
In Section D.1, we drop the explicit mention to M as we are referring to the original MDP and because it makes the equations easier to read.
Lemma 2 (Proposition 2 of Burnetas and Katehakis [1997]) .
Let M be an MDP satisfying the ergodic Assumption 2.
There exists B M > 0 and β M such that for all all β > 0 , for all s ∈ S , t > |S| and policy π = ( π k )
⩽ k ⩽ t ,
[Equation]
In particular, for β < β M , the probability that any given state has been visited less than a linear amount of time is exponentially small.
Such a β M that satisfies this proposition controls the rate at which all states are visited.
Definition 5 ( M -adapted sub-division) .
Let M be an MDP and denote by Π
= | Π ( M ) | the number of deterministic policies Π ( M ) on M .
Let t > Π + 2 , and I t = { 1 , · · · , t } the discrete time steps from 1 to t .
For all ν ∈ { 0 , · · · , Π } , let
[Equation]
22) with σ Π+1 t = t
+ 1 , σ νt =
σ ν +1 t − ⌊ t Π+1 ⌋ for all ν ∈ { 1 , · · · , Π } and let σ 0 t = 1 .
The sub-division (cid:83) ν
I νt of I t induced by ( σ νt )
ν is called an M -adapted sub-division at time t .
It follows immediately from the definition that
[Equation]
24) and | I νt
[Equation]
=
[Equation] [Equation]
Lemma 3 (Linear visit in each interval of a M -adapted sub-division) .
Let M be an MDP and denote by Π
= | Π ( M ) | the number of deterministic policies Π ( M ) on M .
Let t > Π + 2 , and ( I νt , σ νt )
ν be an M -adapted sub-division of M at time t , i.e. a sub-division of I t
= { 1 , · · · , t } .
Let π =
( π k ) k be a policy.
Let N νs ( t )
=
(cid:80)
k ∈
I νt 1 { s t = s } be the number of time the agent visit state s during the sub-interval I νt .
Let κ be such that 0 < κ < 1Π+1 and let 0 < β < β M with β M as in Lemma 2.
Let V νt
=
(cid:84) s ∈S { N νs ( t ) ⩾ κβt } the event that all states are visited more than κβt times during interval ν .
Finally, denote V t =
Π (cid:84) ν
=0 V νt the event that all states are visited more than κβt times during each sub-interval of the sub-division.
Then,
E
[Equation] [Equation]
Proof.
By applying Proposition 2 and using a union bound on all states and sub-intervals, the
Lemma 3 follows.
Lemma 4 (Uniform concentration on the skeleton) .
Let M be an MDP and denote by Π
= | Π ( M ) | the number of deterministic policies Π ( M ) on M .
Let t > Π + 2 , and ( I νt , σ νt )
ν be an M -adapted sub-division of M at time t , i.e. a sub-division of I t
= { 1 , · · · , t } .
Let κ , β , V νt and V t be as in
Lemma 3.
Let δ > 0 be a positive number representing a precision on the skeleton, let
[Equation]
be the event of uniform δ -good approximation on the skeleton for sub-interval ν and let
[Equation]
be the event of uniform δ -good approximation on the skeleton for all time steps after the first sub-interval.
Then, for all policy π = ( π k ) k , it holds that
[Equation]
where S M ( δ ) is decomposed as
[Equation] [Equation]
with
[Equation]
For all state-action pair ( s, a ) , S sa, M is expressed as the limit of a convergent series.
Proof.
First, we remark that V t ∩ S t ( δ ) =
(cid:83) Π ν =1 V t ∩ S νt ( δ ) so that, by a simple union bound, we only need to control the probability of the event V t
∩ S νt ( δ ) .
We then remark that, since for all state s and for all action
a ∈
A s ( k ) , N sa ( k ) ⩾ log 2 max a ′ ∈A s N sa ′
( k ) and max N sa ′
( k )
⩾ N s ( k ) /
|A
s | , we have N sa ( k ) ⩾ log 2 ( N s ( k ) / |A
s | ) .
Combining with V t , for all k ∈ I νt , ( s, a ) ∈ X M ( A ( k )) , the number of samples of ( s, a ) is lower bounded by
[Equation]
The event V t ∩ S νt ( δ ) therefore satisfies the following inclusion,
[Equation]
Again
[Equation]
by a union bound on state-action pairs
[Equation]
we are interested in controlling the event,
[Equation]
that is to say, the probability of
[Equation]
Using the light-tail Assumption 1 and the fact that p ( s
[Equation]
a ) is a multinomial (hence light-tailed too)
[Equation]
we get from Equations (2.2.12) and (
[Equation]
2.13) of Dembo and Zeitouni [1998],
[Equation] [Equation]
from which we deduce that the probability of Equation 32 is upper bounded by
[Equation]
which is the term of a convergent series (in t ).
Denoting S sa, M the limit of the series
[Equation]
and then combining all the union bound
[Equation]
we deduce that
[Equation]
Lemma 5 (Sensibility of the Poisson equation, Lemma 7 (i) Burnetas and Katehakis [1997]) .
Let ε > 0 a real positive number.
Let M =
( S , A , p , r ) be an MDP and M δ =
( S , A , p δ , r δ ) be another
MDP such that for all x ∈ X M = X M δ , max ( ∥ p ( x ) − p δ ( x ) ∥ , | m ( x ) − m δ ( x ) | ) ⩽ δ .
There exits δ M ( ε ) such that for all δ < δ M ( ε ) , for all policy π ∈ Π ( M ) ,
[Equation]
The proof of this fact is given by Burnetas and Katehakis [1997] (Lemma 7 (i)) and a more modern proof to this result is given in Section 1, Chapter 4 of the book Cao [2007] under the name of perturbation analysis.
Lemma 6 (Sensibility of the optimal Poisson equation, Lemma 8 (ii) Burnetas and Katehakis [1997]) .
Let ε > 0 a real positive number.
Let M =
( S , A , p , r ) be an MDP and M δ =
( S , A , p δ , r δ ) be another MDP such that for all x ∈ X M
= X M δ ,
[Equation]
There exits δ M ( ε ) such that for all δ < δ M ( ε ) ,
[Equation]
It follows immediately that forall ν ⊗ q ∈ P ( R ) ⊗ P ( S ))
(with ν having an expected value), | φ M ( ν ⊗ q ) − φ M δ ( ν ⊗ q ) | ⩽ ε .
Furthermore, for ε such that 0 < ε < ε M , O ( M δ ) ⊆
O ( M ) , where
[Equation]
Proof.
The first part is proved in Lemma 8 (ii) of Burnetas and Katehakis [1997] and we prove the last claim for the sake of introducing ε M .
Let ε M = 13 min (cid:8) | g M π − g M π ′ | : π, π ′ ∈ Π ( M ) , g M π ̸ =
g M π ′ (cid:9) and ε be such that 0 < ε < ε M .
Let δ < δ M ( ε ) where δ M is defined in Lemma 5.
Let ⋆ ∈
O
( M δ ) , then ⋆ also is optimal in MDP M .
Indeed, for all π ′ ∈ Π ( M ) ,
[Equation]
which implies that g M ⋆ ⩾ g M π ′ by definition of ε < ε M which separates policies by at least 3 ε M .
Therefore, O ( M δ ) ⊆
O ( M ) .
Lemma 7.
Let σ νt , and S t ( δ ) be defined as in Lemma 4, let δ < δ M ( ε ) with δ M and ε < ε M defined in Lemma 6, then, under S t ( δ ) , for all k ⩾ σ 1 t ,
[Equation]
[Equation] [Equation]
Proof.
This Lemma is a direct consequence of Lemma 6 and definition of S t ( δ ) in Lemma 4.
D Proof of Theorem (REGRET)
In this appendix, we prove our main Theorem 2 that asses the optimality of the IMED-RL algorithm.
Theorem 2 (Regret upper bound for Ergodic MDPs) .
Let M =
( S , A , p , r ) be an MDP satisfying
Assumptions 1, 2, 3.
Let 0 < ε ⩽ 13 min π ∈ Π( M ) min
( s,a )
∈X M {| ∆ s,a ( M ( π ))
| : | ∆ s,a ( M ( π )) | > 0 } .
The regret of IMED-RL is upper bounded as
[Equation]
where Γ s ( M ) is constant that depends on the MDP M and state s ; it is made explicit in the proof below.
A Taylor expansion allows to write the regret upper bound as
[Equation]
Were the semi-bounded reward assumption changed to a bounded reward one with known upper and lower bound, and the O (cid:16) (log T ) 10 / 11 (cid:17) could be made a O (1) as explained in Appendix E.
Outline The proof combines the concentration results obtained by Honda and Takemura [2015] for the family of rewards we study and the skeleton improvement idea from Burnetas and Katehakis [1997].
Because we define a new notion of skeleton and that IMED-RL does not require forced exploration, we specifically derive the Lemma 9 that is at the heart of the proof that φ M ( A ( t )) converges fast enough to φ M , thus allowing to optimally leverage the IMED algorithm in the MDP setting.
Proposition 3.
For all state-action pair x ∈ X M , for all ε > 0 ,
N
[Equation]
) where x t = ( s t , a t ) .
Proof.
The proof is immediate by decomposing the event { x t = x } on
[Equation] [Equation]
∥ ∞ ⩽
[Equation]
b M ∥
[Equation]
⩽
[Equation]
⩽
[Equation]
O (cid:16)(cid:99)
[Equation] [Equation]
∥
[Equation]
(cid:99)
[Equation]
A
[Equation]
− b M ∥ ∞ ⩽
[Equation]
(cid:111)
[Equation]
1 (cid:26) O (cid:16)(cid:99)
M t (
[Equation]
D.1 Bandit term
Let us recall the definition of the quantity we called the "bandit term".
[Equation]
In order to control this quantity
[Equation]
we first make the following useful observation.
Lemma 8.
Whenever the inequality ∥ b (cid:99) M t ( A ( t )) − b M ∥ ∞ ⩽ ε holds true
[Equation]
it implies that for all ν , q ,
[Equation]
Proof.
Let us assume that ∥ b (cid:99) M t ( A ( t )) − b M ∥ ∞ ⩽ ε .
Recall that φ M ( ν ⊗ q ) =
E R ∼
ν
[ R ] + q b M .
Then, for all ν , q
[Equation]
In order to control (44), we further split the considered event depending on whether the threshold appearing in the complexity term is (subsection D.1.2) or not (subsection D.1.1) underestimated.
D.1.1 The threshold is not underestimated
In this subsection, we assume that the threshold is not underestimated, that is to say,
[Equation]
Non-critical state-action pair
First, we study the case where the state-action pair ( s, a ) is not critical, that is to say C sa, M ( F sa ⊗ P ( S ))
def = { ν ⊗ q ∈ F sa ⊗ P ( S ) :
φ M ( ν ⊗ q ) > γ s ( M ) } = ∅ .
Proposition 4.
For all non-critical state-action pair,
[Equation]
  = 0 .
[Equation]
Proof.
Let C sa, M ( F sa ⊗ P ( S ) , δ ) def = { ν ⊗ q ∈ F sa ⊗ P ( S ) :
φ M ( ν ⊗ q ) > γ s ( M ) − ε } .
There exists δ sa small enough such that C s, M ( F sa ⊗ P ( S ) , δ sa ) = ∅ .
Let ε be strictly smaller than δ sa /(which is the case for ε defined as in the statement of Theorem 2),
[Equation]
Therefore, for all distributions in F sa ⊗ P ( S ) , φ (cid:99) M t ( A ( t ))
( ν ⊗ q ) <
ˆ γ s ( t ) meaning that the empirical set of critical distributions is empty, C s, (cid:99) M t ( A ( t ))
( F sa ⊗ P ( S ))
= ∅ .
Remembering that we define inf ∅ = + ∞ by convention, it implies that K s,a ( t ) =
+ ∞ and therefore, H s,a ( t ) =
+ ∞ .
A state-action pair with infinite index can never be sampled since at least one action at that state, the current empirical best one, has a finite, therefore strictly smaller, index.
Proposition 5.
Following immediately from Proposition 4, for all sub-optimal state-action pair ( s, a ) ∈ X M that are not critical, it holds
[Equation]
Critical state-action pair We now study the case were ( s, a ) is critical,
i
[Equation]
e. it can be made optimal under the distributions assumptions, formally C sa, M ( F sa ⊗ P ( S ))
̸
= ∅ .
Proposition 6 (Number of pulls of critical state-action pair in the bandit term) .
For all sub-optimal state-action pair ( s, a ) ∈
X M that are critical, it holds
E
[Equation]

[Equation]
where Γ sa ( M ) =
[Equation]
Proof.
For all ν ⊗ q ∈ F sa ⊗ P ( S ) ,
[Equation]
therefore, C sa, (cid:99) M t ( A ( t ))
( F sa ⊗ P ( S )) ⊆
C sa, M ( F sa ⊗ P ( S ) , ε ) .
Because the infimum over a larger set is smaller than the infimum over a smaller (for the inclusion order),
[Equation]
where
[Equation]
We recall that r t ( s, a ) ⊗ ˆ p t ( s, a ) =
r t ( s, a, N s,a ( t )) ⊗ ˆ p t ( s, a, N s,a ( t )) and we denote
[Equation]
the random variable associated to n samples of state-action pair ( s, a ) .
Because ( s t , a t ) =
( s, a ) , under the the studied event,
[Equation]
Therefore
[Equation]
for a critical state action pair,
[Equation]
It remains to control the expected value of the term
[Equation]
that is to say, control
[Equation]
a quantity that has attracted a lot of attention from the theoretical Bandit community.
In particular, under the Assumption 3 (semi-bounded) and Assumption 1 (light-tail)
[Equation]
one can apply Lemma 7 of
Honda and Takemura [2015] (as in their Theorem 3) to deduce that,
[Equation]
where Γ sa ( M ) =
[Equation]
D.1.2 The threshold is underestimated
In this subsection
[Equation]
we now turn to the case when the threshold is underestimated, that is to say,
[Equation]
In particular, it means that the gain is underestimated since γ s ( M )
=
g M + b M ( s ) and that for the studied bandit term, the bias is ε -well estimated and the empirical set of optimal state-action pairs is included in the true one.
Because all empirically optimal actions belong to the skeleton, those are bound to have been sampled enough.
Further using the ergodic Assumption
[Equation]
we get the concentration we need to bound the expected value of
[Equation]
Proposition 7.
For all sub-optimal state-action pair,
[Equation]
where
[Equation]
Proof.
Let ⋆ ∈
O s ( M ) be an optimal action in state s , and denote H
⋆s ( t ) =
min H sa ( t ) the value of the minimal index.
We note that H ⋆s ( t ) ⩾ log N x t ( t )
= log N sa ( t ) since x t =
( s, a ) under the current studied event.
In particular, H s,⋆ ( t ) ⩾ log N s,a ( t ) .
For all ν ⊗ q ∈ F s⋆ ⊗ P ( S ) ,
[Equation]
This implies that C sa, M ( F sa ⊗ P
[Equation]
S ) , ε ) ⊆
C sa, (cid:99) M t ( A ( t ))
( F sa ⊗ P ( S )) and, using notation K s⋆ introduced Equation 49, proves that K s⋆ ( t ) ⩽ K εs⋆ ( t ) , i.e. , combining with a previous inequality,
[Equation]
Furthermore, because ˆ γ s ( t ) < γ s ( M ) − 2 ε , we have that
[Equation]
This implies that
[Equation]
  is smaller than or equal to
[Equation]
Recalling that, by definition, φ M ( r ( s, ⋆ ) ⊗ p ( s, ⋆ )) is an expected value, this quantity is controlled by Lemma 14 of Honda and Takemura [2015] with
[Equation]
56) with
[Equation] [Equation]
Finally
[Equation]
taking the minimum over all optimal arm ⋆
[Equation]
we get the result of the proposition,
[Equation]
D.1.3 Bandit term: upper bound
Proposition 8 (Upper bound for bandit term) .
Wrapping up, Propositions 6 and 7, if ( s, a ) is a critical state-action pair,
[Equation]
and by Propositions 5 and 7, if ( s, a ) is not a critical state-action pair,
[Equation]
D.2 Skeleton improvement term
In this part of the main proof, we aim at controlling the expected value of the sum
[Equation]
and prove that E ( S ( T ))
=
O (1) .
For readability, we denote
[Equation]
in the rest of proof.
Proposition 9.
Let V t and S t ( δ ) as in Lemma 3, then
[Equation]
Proof.
Recall that we denote ( σ νt , I νt )
the M -adapted sub-division of I t = { 1 , · · · , t } used in V t and
S t ( δ ) as in Definition 5, 0 < β < β M as in Proposition 2, and V t and S t ( δ ) are the events defined in
Lemma 4.
We first decompose { W t ( ε ) } on V t ∩ S t ( δ ) and V t ∩ S t ( δ ) by the law of total probability, and deduce the inequality
[Equation]
By Lemma 3,
[Equation]
and by Lemma 4,
[Equation]
therefore,
[Equation]
Outline - Intuition
The intuition for controlling the remaining term,
[Equation]
is the following.
There are Π + 1 sub-intervals and Π policies.
At the end of the first sub-interval, all states have been visited linear amount of time and from that point, the gain and bias are well estimated on the skeleton, i.e. , the bandit problem defined by φ M ( A ( k )) for all k ⩾ σ 1 t .
Because we play a bandit strategy, sub-optimal actions for the current bandit problem are guaranteed to be played a logarithmic amount of time while at least one improving action will be played a linear amount of time until it belongs to the skeleton, defining a new well-estimated bandit problem.
Because the condition to belong to the skeleton is in log 2 which is greatly sub-linear, an improving action will quickly makes its way to the skeleton.
Each interval lasting a linear amount of time, the probability that a skeleton improvement, if one is possible, occurred between the beginning and end of the sub-interval is high.
Since there are Π policies and Π sub-intervals after the first one, by the pigeonhole principle, the probability that the skeleton contains an optimal policy by the end of the last sub-interval is large.
The two main propositions for controlling this term are Propositions 10 and 11.
Lemma 9 (Skeleton coherence) .
Under the event V t ∩ S t ( δ ) with δ < δ M ( ε M ) , for all k ⩾ σ 1 t ,
[Equation]
Proof.
It is sufficient to show that A ( k + 1) ∩ O (cid:16)(cid:99)
M k ( A ( k )) (cid:17) ̸ = ∅ , i.e. to prove that no empirically optimal policy at time k disappear from the skeleton after the action taken at that time.
We distinguish between two cases.
If the sampled action a k / ∈ A s k ( k ) , i.e. the chosen action does not belong to the current skeleton, then the skeleton at s k can only grow in size, as a k / ∈ arg max a max a N s k a ( k ) , A s k ( k +
[Equation]
⊆
[Equation]
s k
[Equation]
k )
[Equation]
k
[Equation]
and therefore, in that case, g M ( A ( k +1)) ⩾ g M ( A ( k )) .
• If the sampled action a k ∈ A s k ( k ) , i.e. the chosen action belongs to the current skeleton, then the size of the skeleton may decrease if a k / ∈ arg max a max a N s k a ( k ) .
We distinguish again between two cases.
– If the sampled action a k ∈ arg max a φ (cid:99) M k ( A ( k )) (cid:0) ˆ r k
( s k , a ) ⊗ p k
( s k , a ) (cid:1) , i.e. the chosen action is empirically optimal and belongs to O (cid:16)(cid:99)
M k ( A ( k )) (cid:17) , then this action will belong to the skeleton at time k
+ 1 (whether a k / ∈ arg max a max a N s k a ( k ) or not).
By Lemma 7, O (cid:16)(cid:99)
M k ( A ( k )) (cid:17) ⊆
O ( M ( A ( k ))) and therefore, the true gain on the skeleton will remain the same.
– If the sampled action a k / ∈ arg max a φ (cid:99) M k ( A ( k )) (cid:0) ˆ r k
( s k , a ) ⊗ p k
( s k , a ) (cid:1) , then we show that it cannot belong to arg max a max a N s k a ( k ) and thus that the skeleton remains the same between times k and k + 1 .
We show this fact by contradiction.
If a ′ is an action that is not empirically optimal and belongs to arg max a max a N s k a ( k ) , then H s k a ′ ( k ) > log max a N s k a ( k ) .
On the other hand, for all empirically optimal action ⋆ , H s k ⋆ ( k ) = log N s k
⋆ ⩽ log max a N s k a ( k ) .
Therefore, H s k a ′ ( k ) >
H s k ⋆ ( k ) and action a ′ cannot be sampled.
Therefore, in that case,
g M
A k +1))
=
g M ( A ( k )) .
We proved that under the event V t ∩ S t ( δ ) with δ < δ M ( ε M ) , for all k ⩾ σ 1 t , g M ( A ( k +1))
⩾ g M ( A ( k )) .
where strict improvement can only occur when the sampled action a k does not belong to the skeleton.
If the action a k belongs to the current skeleton, the gain on the skeleton can only remains the same.
An immediate consequence of Lemma 9 is that under the event V t ∩ S t ( δ ) , g M ( A ( σ ν +1 t )) ⩾
g M ( A ( σ νt )) .
The aim is to prove that the inequality is strict unless the optimal gain has already been reached, i.e. an optimal policy belongs to the skeleton.
Another consequence is that if the skeleton contains an optimal policy at some time k , then this policy will remain in the skeleton for all subsequent step (under the event V t ∩ S t ( δ ) ).
For all ν ∈ { 1 , · · · , Π − 1 } , let
[Equation]
and let
[Equation]
where G t is an event in which there is a still skeleton improvement between each interval until optimality is reached.
Proposition 10.
[Equation]
Proof.
One one hand, there are Π policies, therefore the pigeonholes principle implies that under
G t , by the end of the last sub-interval, O ( M ( A ( t ))) ⊆
O ( M ) .
Furthermore, under V t ∩ S t ( δ ) , by
Lemma 7, O (cid:16)(cid:99)
M t ( A ( t )) (cid:17) ⊆
O ( M ( A ( t ))) .
Thus, O (cid:16)(cid:99)
M t ( A ( t )) (cid:17) ⊆
O ( M ) .
Furthermore, under the event S t ( δ ) , by Lemma 7, ∥ b (cid:99) M t
( A ( t ))
− b M ( A ( t ))
∥ ⩽ ε 2 and since O ( M ( A ( t ))) ⊆
O ( M ) , it follows that b M ( A ( t )) =
b M by the optimal Poisson equation and therefore, ∥ b (cid:99) M t ( A ( t ))
− b M ∥ ⩽ ε 2 .
Therefore, event { V t , S t ( δ ) , G t } implies W t ( ε ) .
Thus, { W t ( ε ) , V t , S t ( δ ) , G t } = ∅ , the indicator of such an event is always 0 and the sum equally.
The last proposition lets us to study, for all ν ∈ { 1 , · · · , Π − 1 } ,
[Equation]
because G
[Equation]
Lemma 10.
The equality,
[Equation]
is true.
Proof.
Under V t ∩ S t ( δ ) , Lemma 9 implies
that, g M ( A ( σ ν +1 t )) ⩾
g M ( A ( σ νt )) .
G νt implies that g M ( A ( σ ν +1 t )) <
g M ( A ( σ νt )) or g M ( A ( σ ν +1 t )) =
g M ( A ( σ νt )) <
g M .
Therefore the result.
In the last part of the proof, we aim to control the probability that, under good event V t ∩ S t ( δ ) , no improvement occurs during sub-interval ν .
Denote by I ( s, M , γ ) the set of improving actions over a threshold γ ,
[Equation]
(67) and I + ( s, M , γ ) the set of maximally improving actions over a threshold (it may be empty),
[Equation]
Lemma 11.
On the event,
[Equation]
we have that for all k ∈
I νt ∪ (cid:8)
σ ν +1 t (cid:9)
[Equation] [Equation]
and for all s ∈ S ,
[Equation]
Proof.
The first three equations result from the, definition of the event, the unicity of the optimal
Poisson equation (see Equation 5.2.18, Chapter 5, Hernández-Lerma and Lasserre [1996]) and the definition of φ M .
The set of improving actions cannot change during I νt ∪ (cid:8) σ ν +1 t (cid:9) because if it were, it would mean that an improving action now belong to the skeleton.
This would lead to a strict increase of the gain during I νt , which is in contradiction with the studied event.
Proposition 11 (Expected time before policy improvement) .
Under the event V t ∩ S t ( δ ) , the expected time during interval ν without skeleton improvement is bounded, i.e.,
[Equation]
where ˜ ζ ( ε ) is defined by Equation 77, τ ( M ) is defined in Equation 83 and A = max s
|A s | .
Outline of the proof We aim to control the number of pulls within the improving set during interval ν .
The threshold to belong to the skeleton will be at most log 2 σ ν +1 t
and I tν is of linear length.
Furthermore, the applied bandit strategy on the well estimated problem given by φ M ( A ( σ νt )) will sample roughly a linear number of times an optimal action for that problem, i.e. an action in the improving set.
This will imply that such an action belong to the skeleton by the end of the interval, contradicting the non-increasing gain assumed by the studied event.
We will have to control the probability that no optimal action is sampled more than log 2 (cid:0) σ ν +1 t (cid:1) during interval ν , which is very unlikely.
Proof.
First, we remark that there exits a state s ∈ S such that I (cid:0) s, M ( A ( σ νt )) , γ M ( A ( σ νt )) (cid:1) is not empty because the gain g M ( A ( σ νt )) <
g M and an improving action is bound to exist by the policy improvement theorem.
Let s be such a state.
We denote by P νt ( s ) =
(cid:80)
k ∈
I kt 1 { s k = s, a k ∈ A s ( k ) } the number of times the pulled action belongs to the skeleton at state s , while an improving action outside of it exists.
We have,
[Equation]
The sum corresponding to Equation 75 is equal to 0 because under V t ∩ S t ( δ ) , for all action
a ∈
A s ( k ) ,
[Equation]
By definition of s , no empirically optimal action ⋆ in state s belong to the skeleton and ˆ γ s ( k ) is realized by an action
⋆ / ∈
A s ( k ) .
Let ⋆ be such an action, the its IMED-RL index is log N s⋆ ( k ) < log max
a ′ N sa ′
( k ) .
Further, for all action a ∈
A s ( k ) , its IMED-RL index is strictly larger than log N sa ( k ) ⩾ log max a ′ N sa ′
( k ) .
Therefore, a k cannot belong to A s ( k ) if ˆ γ s ( k ) ⩾ γ s ( k ) − 2 ε under the favorable event V t ∩ S t ( δ ) .
Thus,
[Equation]
For all a ∈ I + ( s, M , γ s ( k )) , φ M ( A ( k ))
( r ( s, a ) ⊗ p ( s, a ))
= γ s ( k ) and is an optimal choice of action for the bandit problem defined by φ M ( A ( k )) =
φ M ( A ( σ νt )) .
While this fact is not used, we still note that any action a ∈ I ( s, M , γ s ( k )) that is sampled enough will make the gain increase.
For all a ∈ I + ( s, M , γ s ( k )) , φ M ( A ( k ))
( r ( s, a ) ⊗ p ( s, a ))
=
γ s ( k ) , and therefore φ M
[Equation]
A
[Equation]
k
[Equation]
By a union bound, for all ⋆ ∈ I + ( s, γ s ( σ νt )) ,
P νt ( s ) ⩽ (cid:88) a/ ∈
I + ( s,γ s (
[Equation]
νt
[Equation]
(cid:88)
[Equation]
ε   , a quantity that is similar to the one controlled in Equation 55.
In particular, this quantity is similarly controlled by Lemma 14 of Honda and Takemura [2015] with
E
[Equation]
where
[Equation]
This proves that E [ P νt ( s )] ⩽ A ˜ ζ ( ε ) where
A = max s |A s | ,
i
[Equation]
e. , P νt ( s ) is a positive random variable with finite expected value.
This implies that, P ( P νt ( s ) ⩾ αt ) =
o (1 /t ) for all α > 0 .
To end the proof
[Equation]
we decompose,
[Equation]
as the sum
[Equation] [Equation]
We control the term 80.
This term is upper bounded by 1 (cid:110) P νt ( s ) ⩾ κβt |A
s | (cid:111) , and because
[Equation]
by using again the fact that E
[Equation]
P
[Equation]
Finally, we control the term 79.
The event (cid:110) P νt ( s ) < κβt |A
s | (cid:111) implies that the number of times the sampled action belong to the skeleton during I νt is only a fraction of that sub-interval.
The remaining sample must therefore not belong to the skeleton.
Sub-optimal actions outside the skeleton at time σ νt will be sampled at most log 2 σ ν +1 t times and there are at most A such actions.
Therefore, there are at most κβt |A
s |
+ A log 2 ( νβt ) samples that are not in the improving set
I ( s, γ s ( σ νt )) .
The improving set is at most of size A and therefore, at least one action in that set is sampled more than
[Equation]
times.
This quantity is linear and for t larger than a constant that depends on κ , β , A ,
|A s | and ν , we have that,
[Equation]
We denote by τ ( M ) , the maximum on s ∈ S and ν of these constants.
Therefore,
[Equation]
D.2.1 Skeleton improvement: upper bound
Proposition 12.
By combining Propositions 9 and 11, then
[Equation]
D.3 Regret upper bound
Finally
[Equation]
one can express the full regret upper bound on the regret of IMED-RL by combining the decomposition of Proposition D
[Equation]
1, the result of Proposition 8 and the one of Proposition 12.
Theorem 4 (Regret upper bound for Ergodic MDPs) .
Let M =
( S , A , p , r ) be an MDP satisfying
Assumptions 1, 2, 3.
Let 0 < ε ⩽
[Equation]
π ∈ Π( M )
[Equation]
s
[Equation]
M
[Equation]
π )) | [Equation]
The regret of IMED-RL is upper bounded as
[Equation]
where Γ s ( M ) is constant that depends on the MDP M and state s ; it is made explicit in the proof below.
A Taylor expansion allows to write the regret upper bound as
[Equation]
The Taylor expansion is a direct application of Equation 7, Corollary 4 of Honda and Takemura [2015].
E Assumptions
In this section, we discussion a variant of the considered setup when the support of transitions is considered known, and then a possible relaxation of the ergodic assumption.
Known support of transitions As quickly explained in the main article, when the support of transition is known, the infimum in sub-optimality cost K s,a defined by equation 9 is redefined as one over the set { q ∈ P ( S ) :
Supp ( q ) =
Supp ( p ( ·| s, a )) } , modifying the lower bound.
Without the knowledge of the support,
[Equation]
and with the knowledge of the support,
[Equation]
Hence, two similar but different lower bounds can be derived depending on whether or not, one assumes to know the support Supp ( p ) of the transitions.
In both cases, it can be written
[Equation]
where C ( M ) = (cid:8) ( s, a ) : 0 < K s,a ( M ) < ∞ (cid:9) , the set of critical state-action pairs, depends on the made hypothesis.
Since the lower bound obtained with the knowledge of the support is smaller than without this knowledge, it is a priori not trivial that an algorithm originally designed for the case when support is unknown can indeed exploit this knowledge.
Fortunately, due to form of the IMED-RL , it is enough to use the same restriction on the set { q ∈ P ( S ) :
Supp ( q ) =
Supp ( p ( ·| s, a )) } in the definition of the index to leverage this knowledge.
The resulting algorithm slightly differs from
IMED-RL and it can be checked easily that the regret analysis for this modified version can be done similarly to IMED-RL , and leads to algorithm matching the asymptotic lower bound knowing the support of transitions.
Please refer to Subsection F.1 to see how the hereafter term denoted B is modified by the knowledge of the support.
Bounded support Similarly, one can modify the Assumption 3 to be one with a bounded reward assumption.
In this case, Theorem 5 of Honda and Takemura [2015] shows that the Taylor expansion made in the regret upper bound has the aforementioned form in Theorem 2.
Communicating MDPs and ε -soft policies The ergodic assumption can be limiting in practice, since most common MDPs are not ergodic but only communicating.
Interestingly, in a communicating MDP, every stochastic policy π : s ∈ S (cid:55)→ π ( ·| s ) ∈ P ( A s ) , with full-support (that is
Supp ( π ( ·| s ))
=
A s for each s ∈ S ) is ergodic.
In particular, the uniform policy is ergodic.
Also, ε -soft policies, that satisfy π ( a | s ) ⩾ ε for all s, a , are ergodic.
When restricting to the class of ε -soft policies in a communicating MDPs, it seems that modifying IMED-RL to be also ε -soft should lead to a strategy competitive with an optimal ε -soft policy.
For ε < 1 / |A
s | , the modification is to sample the chosen action with probability 1 − ( |A s | − 1) ε and any other action with probability ε .
Now, a precise analysis of this modification is postponed to further work, and going beyond this case to handle the full-blown communicating assumption seem to require other ideas, especially since the lower bound for non-ergodic MDPs is expected to be much different from that of ergodic MDPs.
F Numerical Experiments
In this section, we first discuss a few implementation details of the IMED-RL index, then present additional numerical experiments as well as some extensions.
F.1 Solving the optimization problem K s,a
Although the IMED index involves only a single optimization problem (unlike KL-UCB that requires two), computing the Kullback-Leibler projection is not obvious in general.
The same remark holds for IMED-RL .
Luckily, inspired from the work of Honda and Takemura [2015], the IMED-RL index can be computed easily when restricting to some families of distributions.
In particular, when the set F sa of reward distributions is a set of multinomial distributions over a finite set with largest element < m max , then the computation of K s,a can be done easily, owing to the rewriting Theorem 3 from Honda and Takemura [2012].
Lemma 12.
Let M be an MDP satisfying Assumptions 2, B = m max ( s, a ) + max s ′ ∈S b M ( s ) with m max ( s, a ) as in Assumption 3 and rewards satisfying Assumption 1.
Then,
[Equation]
which is a finite convex optimisation problem in x ∈ R .
For the computation of K s,a ( t ) , we use the empirical support of r ( s, a ) computed with the gathered samples.
Theoretical guarantees comes from the finitness of the support of the original distribution but most importantly, from the fact that is upper bounded by known constant.
If
E R ∼ r
( s,a ) ,
S ∼ p ( s,a ) (cid:16)
B
[Equation]
B −
[Equation]
− b M ( S ) (cid:17) ⩽ 1 , then this optimisation problem even has a closed form formula and the maximum of the right-hand-side is obtained for x =
[Equation]
In a run of IMED-RL , we compute the solution of the empirical problem the same way,
[Equation]
with B t = m max ( s, a ) + max s ′ ∈S b (cid:99) M t ( A ( t )) .
In the general case, the problem given in Proposition 2 is still convex and can be numerically solved as long as one can correctly approximate an expected value, i.e. an integral.
Known support of transition We follow the discussion started in Appendix E.
If the support of the transition is known, then the cost is computed as
[Equation]
where B sa = m max ( s, a )+
max s ′ ∈ Supp ( p ( ·| s,a ))
b M ( s ) replacing the initial B in Lemma 12, thus making
IMED-RL takes the knowledge of the support of the transition into account.
Of course, this problem is still a finite convex optimisation problem in x ∈ R .
Lazy updates Numerically, IMED-RL benefits from this fast computation and the fact that it employs a Value Iteration in lieu of an Extended Value Iteration for instance used in UCRL 3.
On the other hand, IMED-RL updates its policy at each time step, unlike UCRL 3 that proceeds into episodes.
On our numerical experiments, the overall running time of IMED-RL is only about 5 times that of UCRL 3, despite updating its policy at each time step.
Interestingly, it may be possible to further reduce the numerical complexity of IMED-RL by performing lazy computation of the indexes after some time.
Indeed, by design, with high probability, the potential function φ (cid:99)
M ( A ( t )) is not destined to change nor to be much different from the true φ M once an optimal policy belongs to A ( t ) .
As the number of samples increase, the magnitude of the updates decreases and φ (cid:99) M ( A ( t )) roughly remains the same, thus allowing the practitioner to perform value iteration every once in a while, when at least one estimate shifted by more than a fraction of the minimal sub-optimality gap for instance.
Of course this modification requires to update the regret analysis accordingly.
F.2 Additional experiments
In this section, we detail a few more experiments.
In all experiments, we used environments with maximal reward 0 .
99 and bound m max = 1 given to the learner.
The code of the experiments is available on the github repository 10 of this paper.
Experiments are conducted using 256 replications (independent run), with horizon specified in case.
River-swim We consider one experimentation with a river-swim with 25 states.
River-swim environments are sometimes considered hard instances for strategies such as PSRL , as the reward signal is sparse.
We observe in Figure 4 that indeed PSRL struggles in such an environment.
The three other strategies work well, with some advantage for IMED-RL on the long run.
[Figure]
For the sake of completeness, we present the average runtime for completing a trajectory of the tested algorithms on both ergodic and non-ergodic RiverSwim.
[Table] [Table]
Apart from Q-learning, all algorithms seem to benefit a numerical boost from the ergodicity of the environment.
Two-room grid-world
The two-room environment we consider in this experiment consists of a 9 × 11 grid and 4 actions, and is actually a larger state-action space than the four-room MDP considered in the main text, Section 5.
Also, it contains a bottleneck state, which is sometimes considered as a hard instance.
Note that since the considered grid-worlds are slippery (frozenlake style, with 0 .
1 probability of visiting executing nearby actions), this also means that from the bottleneck state, it is actually possible to enter the bottom room not only with action down, but also left and right.
Hence, this MDP does not contain a bottleneck state-action pair.
In such environments, although not being ergodic, we expect the IMED-RL strategy to work reasonably well, which is confirmed by the experiment in Figure 5.
[Figure]
Another grid-world We further provide below complementary experiments in Figure 6 and Figurewith other randomly generated frozen-lake grid-worlds with a unique goal state.
The learner jumps to a random initial state each time the goal is reached.
The frozen lake part is implemented as slippery actions, where for instance choosing action up has some small probability to move the learner also left or right, or action left has some probability to move the learner up or down, as long as there is no wall (note also that they are coded as toric environments).
Although these environments are not ergodic but only communicating.
we can observe the striking performance of IMED-RL against the state-of-the-art UCRL 3 or related PSRL and Q-learning strategies.
Note that these other strategies eventually learn as well
[Figure]
but for larger time horizon.
In Figure
[Figure]
we did not report UCRL 3 and PSRL as their computation time were prohibitive compare to IMED-RL and Q-learning in this setup
[Figure]
We present again the average runtime for completing a trajectory of the tested algorithms on such a grid-world environment.
[Table]
We can see that the performances of IMED-RL and UCRL3 were exchanged.
Generally, our experiments tends to show that the performances of IMED-RL are quite good on grid-worlds, both from a regret minimization viewpoint and a numerical complexity viewpoint.
1.4 7LPHVWHSV ×10         5 HJ U H W 7 J   V X P B W U B W ×
RandomGridworld-16x16-s7-v
IMED-RL Q-learning
A reward-rich environment While the previous examples were considering environments with a sparse reward signal, it is interesting to test the behavior of the algorithm in other types of environments.
In the following experiment, we consider a reward-rich environment, where about 80% of state-action pairs generate a reward of at least 0 .
4 (and the maximal reward is 0 . 99 ).
Such environments are known to favor the PSRL strategy as well as optimistically initialized strategies, that benefit from a reduced burn-in phase thanks to their prior.
In Figure 8, we observe that IMED-RL outperforms the UCRL 3 strategy, but is indeed beaten by PSRL (while PSRL had poor regret in reward-scarce environments, see Figure 4), as well as the Q-learning algorithm initialized with γ = 0 .
99 and initial value 1 / (1 − γ ) in each state.
When the MDP is modified to have minimal pass p ( s ′ | s, a ) ⩾ 0 .
01 for each s, a, s ′ , the performance of IMED-RL improves and becomes more stable (as well as that of other strategies), as seen in Figure 8-right.
[Figure]
A nasty case In order to better understand the limitation of the IMED-RL algorithm, we tried (but did not succeed) to craft an environment that would make the IMED-RL algorithm fail.
The analysis reveals that we should consider a non-ergodic MDP for this purpose.
Importantly, the index for pair ( s, a ) is based on building a modified MDP with unmodified reward and transitions for pairs different than ( s, a ) , which is a feature coming from the ergodic property.
However, in a non-ergodic
MDP, an optimal policy and a policy playing a in state s may have different recurrent classes, say class ⋆ and ⋆ a .
It is not difficult to show that when all paths from a state in ⋆ to a state in ⋆ a must contain ( s, a ) , that is ( s, a ) is a bottleneck pair, then changing the MDP only in pair ( s, a ) to build a "confusing instance" isn’t sound anymore, hence the construction of the IMED-RL index is no longer justified in such cases.
Inspired from this intuition, we build in Figure 9 a specific nasty MDP with such a bottleneck state-action pair, separating two cycles with close value.
We further remark that this structure, two promising cycles at two ends of a chain with less rewards in between, may induce an "oscillation" of a learning agent between the two cycles, paying the cost of the travel along the chain each time it "decides" to change cycle.
We observe that the quantile tube of IMED-RL is larger than before and indeed indicates more struggles but not enough that the IMED-RL fails the task.
Still, we remark a small advantage of PSRL over IMED-RL in this environment.
Note that the environment is reward rich with rewards close to 1 , which also favors PSRL and Q-learning with optimistic initialization.
[Figure]
[Figure]
( r = 0 . 99) ( r = 0 . 99) ( r = 0 . 99) s 0 ssssss( r = 0 .
( r = 0 . 94)
( r = 0 . 94) 0 .
[Figure]
Conclusion The numerical experiments presented in the main paper and this section show that
the
IMED-RL algorithm seems to deliver the promise of the premise.
On ergodic environment, its regret is empirically very low and often the smallest in tested environment.
As explained in this section, it is pleasing to see that its numerical guarantees seems to go beyond the ergodic assumption with particularly good performances in grid-world.
In order to better understand the role of the made assumptions, we specifically designed a "nasty case" built to make IMED-RL fail.
This case paves the way for future work.
G A note on the seminal paper of Burnetas and Katehakis [1997]
In this last appendix, we discuss the subtle but key modification that we made to the notion of skeleton introduced in the seminal paper of Burnetas and Katehakis [1997] and defined, for each state s and time t , by,
[Equation]
In contrast, the skeleton used in IMED-RL is defined replacing the sum N s ( t )
=
(cid:80) a ′ ∈A s N sa ′
( t ) with a maximum as follows
[Equation]
Correctness The restricted MDP defined by IMED-RL , M A ( t ) , is well defined in the sense that at least one action is available in each state, that is for all t , for all s , A s ( t ) ̸ = ∅ .
On the other hand, especially at the beginning, A BKs ( t ) could very well be empty.
Without saying anything about their algorithm, we just highlight that 11 it must explore all actions in each state at least once before proceeding with non-trivial allocation (this is because the index is −∞ when an arm has not been pulled).
Suppose that there are 4 actions in a state s .
After the first 3 visits in s , whatever the current time t , N s ( t ) = 3 , N sa ( t ) ⩽ 1 (as it is 0 for the only unsampled action or 1 for the three others).
Because log 2 3 ≃ 1 . 2 > 1 , the skeleton at state s is hence empty and therefore, no action belong to the skeleton.
Note that this situation does not happen when using the definition of skeleton used by
IMED-RL , since x ⩾ ln 2 ( x ) for all x ⩾ 0 .
5 and at least one action must be sampled ( N s,a ( t ) ⩾ 1 ).
In this case, the behaviour of the algorithm presented in the paper of Burnetas and Katehakis [1997] is undefined as it is not specified how to compute the bias and gain on the lacking restricted MDP.
Now, in and MDP with a larger number of actions, say 100 , the same argument shows that between the 3 rd and 100 th visit of state s , the skeleton at s is empty and the behaviour undefined.
This means that if there are only 20 states in the MDP, the behaviour of the algorithm is undefined for at least about |S| × ( A − log 2 ( A )) ≃ 2000 steps (and possibly much more, since one would need all states to be visited about A − log 2 ( A ) time and it is unlikely that all states are visited equally often).
Incoherence The skeleton as defined in (85) is "incoherent" in the sense that actions may be removed from it for no "justified" reason.
In the worst case, all actions may be removed in one step.
Assume a state s with 2 actions, one having been sampled 3 times and the other 2 times, i.e. N s ( t )
= 5 .
Because 2 < log 2 5 ≃ 2 . 6 < 3 , one action belongs to the skeleton and the other does not.
Assume that the action that have been sampled 2 times is now sampled at time k > t .
Then both actions have been sampled 3 times, N s ( k ) = 6 and the skeleton at s is now empty since log 2 6 ≃ 3 . 2 > 3 .
While this kind of behaviour disappear for large number of samples, it is not desirable in finite time and introduces incoherence that makes the algorithm undefined and the learning less efficient if we were to resolve undefined behaviour by random choices.
Forced exploration Because of their definition of skeleton, forced exploration is necessary in the analysis of Burnetas and Katehakis [1997] meaning that their algorithm is not purely based on a computed index.
While forced exploration and tracking is not inherently an unwanted feature, we think that it should be avoided when possible, hence leaning towards our IMED-RL skeleton.
Measuring accuracy The skeleton is used to build a restricted MDP on which the gain and bias can be controlled.
This control is due to the fact that, on the skeleton, state-action pairs have been sampled enough.
In each state, we are interested in actions with the largest number of pulls amongst all available actions.
The most sampled action in each state should therefore obviously belong to the skeleton.
Furthermore, it seems natural that the skeleton at a state does not change if the maximal precision in that state, given by the action that has been sampled the most in that state, does not change.
This is mainly the rationale behind our subtle but key modification of the notion of skeleton.