Original: Reinforcement Learning in a Birth and Death Process: Breaking the Dependence on the State Space
Revised: Reinforcement Learning in a Birth and Death Process: Breaking the Dependence on the State Space

Original: Abstract
Revised: Abstract

Original: In this paper, we revisit the regret of undiscounted reinforcement learning in MDPswith a birth and death structure.
Revised: In this paper, we revisit the regret of undiscounted reinforcement learning in MDPs with a birth and death structure.

Original: Speciﬁcally, we consider a controlled queue withimpatient jobs and the main objective is to optimize a trade-off between energyconsumption and user-perceived performance.
Revised: Speciﬁcally, we consider a controlled queue with impatient jobs and the main objective is to optimize a trade-off between energy consumption and user-perceived performance.

Original: Within this setting, the diameter Dof the MDP is Ω( S S ) , where S is the number of states.
Revised: Within this setting, the diameter D of the MDP is Ω( S S ) , where S is the number of states.

Original: Therefore, the existinglower and upper bounds on the regret at time T , of order O ( √ DSAT ) for MDPswith S states and A actions, may suggest that reinforcement learning is inefﬁcienthere.
Revised: Therefore, the existing lower and upper bounds on the regret at time T , of order O ( √ DSAT ) for MDPs with S states and A actions, may suggest that reinforcement learning is inefﬁcient here.

Original: In our main result however, we exploit the structure of our MDPs to show thatthe regret of a slightly-tweaked version of the classical learning algorithm U CRL 2is in fact upper bounded by ˜ O ( √ E 2 AT ) where E 2 is related to the weightedsecond moment of the stationary measure of a reference policy.
Revised: In our main result however, we exploit the structure of our MDPs to show that the regret of a slightly-tweaked version of the classical learning algorithm U CRLis in fact upper bounded by ˜ O ( √ E 2 AT ) where E 2 is related to the weighted second moment of the stationary measure of a reference policy.

Original: Importantly, E 2 isbounded independently of S .
Revised: Importantly, E 2 is bounded independently of S .

Original: Thus, our bound is asymptotically independent ofthe number of states and of the diameter.
Revised: Thus, our bound is asymptotically independent of the number of states and of the diameter.

Original: This result is based on a careful study ofthe number of visits performed by the learning algorithm to the states of the MDP,which is highly non-uniform.
Revised: This result is based on a careful study of the number of visits performed by the learning algorithm to the states of the MDP, which is highly non-uniform.

Original:  Introduction
Revised:  Introduction

Original: In the context of undiscounted reinforcement learning in Markov decision processes (MDPs), it hasbeen shown in the seminal work [10] that the total regret of any learning algorithm with respect to anoptimal policy is lower bounded by Ω( √ DSAT ) , where S is the number of states, A the numberof actions, T the time horizon and D the diameter of the MDP.
Revised: In the context of undiscounted reinforcement learning in Markov decision processes (MDPs), it has been shown in the seminal work [11] that the total regret of any learning algorithm with respect to an optimal policy is lower bounded by Ω( √ DSAT ) , where S is the number of states, A the number of actions, T the time horizon and D the diameter of the MDP.

Original: Roughly speaking, the diameter isthe mean time to move from any state s to any other state s (cid:48) within an appropriate policy.
Revised: Roughly speaking, the diameter is the mean time to move from any state s to any other state s (cid:48) within an appropriate policy.

Original: In theliterature, several efforts have been dedicated to approach this lower bound.
Revised: In the literature, several efforts have been dedicated to approach this lower bound.

Original: As a result, learningalgorithms have been developed with a total regret of ˜ O ( DS √ AT ) in [10], ˜ O
Revised: As a result, learning algorithms have been developed with a total regret of ˜ O ( DS √ AT ) in [11], ˜ O

Original: ( D √ SAT ) in [3] andeven ˜ O ( √ DSAT )
Revised: ( D √ SAT ) in [3] and even ˜ O ( √ DSAT )

Original: according to [20, 24].
Revised: according to [21, 25].

Original: These results may give a sense of optimality since thelower bound is attained up to some universal constant.
Revised: These results may give a sense of optimality since the lower bound is attained up to some universal constant.

Original: However, lower bounds are based on theminimax approach, which relies on the worst possible MDP with given D , A and S .
Revised: However, lower bounds are based on the minimax approach, which relies on the worst possible MDP with given D , A and S .

Original: This meansthat when a reinforcement learning algorithm is used on a given MDP, one can expect a much betterperformance.
Revised: This means that when a reinforcement learning algorithm is used on a given MDP, one can expect a much better performance.

Original: One way to alleviate the minimax lower bound is to consider structured reinforcement learning , orequivalently MDPs with some speciﬁc structure.
Revised: One way to alleviate the minimax lower bound is to consider structured reinforcement learning , or equivalently MDPs with some speciﬁc structure.

Original: The exploitation of such structure may yield moreefﬁcient learning algorithms or tighter regret analyses of existing learning algorithms.
Revised: The exploitation of such structure may yield more efﬁcient learning algorithms or tighter regret analyses of existing learning algorithms.

Original: In this context,a ﬁrst example is to consider factored MDPs [6, 9], i.e., MDPs where the state space can be factoredinto a number of components; in this case, roughly speaking, S = K n where n is the number of“factors” and K is the number of states in each factor.
Revised: In this context, a ﬁrst example is to consider factored MDPs [6, 9], i.e., MDPs where the state space can be factored into a number of components; in this case, roughly speaking, S = K n where n is the number of “factors” and K is the number of states in each factor.

Original: The regret of learning algorithms in factored
Revised: The regret of learning algorithms in factored

Original: MDPs has been analyzed in [19, 16, 23, 13] and it is found that the S term of existing upper boundscan be replaced by nK .
Revised: MDPs has been analyzed in [20, 17, 24, 14] and it is found that the S term of existing upper bounds can be replaced by nK .

Original: A similar approach is used in [8] to learn the optimal policy in stochastic bandits with a regret that is logarithmic in the number of states.
Revised: A similar approach is used in [8] to learn the optimal policy in stochastic bandits with a regret that is logarithmic in the number of states.

Original: There is also a line of research worksthat exploit the parametric nature of MDPs.
Revised: There is also a line of research works that exploit the parametric nature of MDPs.

Original: Inspired by parametric bandits, a d -linear additive modelwas introduced in [11], where it is shown that an optimistic modiﬁcation of Least-Squares Value
Revised: Inspired by parametric bandits, a d -linear additive model was introduced in [12], where it is shown that an optimistic modiﬁcation of Least-Squares Value

Original: Iteration, see [14], achieves a regret over a ﬁnite horizon H of ˜ O ( √ d 3 H 3 T ) where d is the ambientdimension of the feature space (the number of unknown parameters).
Revised: Iteration, see [15], achieves a regret over a ﬁnite horizon H of ˜ O ( √ d 3 H 3 T ) where d is the ambient dimension of the feature space (the number of unknown parameters).

Original: In this case, the regret does notdepend on the number of states and actions and the diameter is replaced by the horizon.
Revised: In this case, the regret does not depend on the number of states and actions and the diameter is replaced by the horizon.

Original: A discussionabout the inapplicability of this approach to our case is postponed to Section 4.2.
Revised: A discussion about the inapplicability of this approach to our case is postponed to Section 4.2.

Original: Learning in Queueing Systems.
Revised: Learning in Queueing Systems.

Original: The control of queueing systems is undoubtedly one of the mainapplication areas of MDPs; see, e.g., [15, Chapters 1–3] and [12].
Revised: The control of queueing systems is undoubtedly one of the main application areas of MDPs; see, e.g., [16, Chapters 1–3] and [13].

Original: Within the rich literature ofstructured reinforcement learning however, few papers are dedicated to reinforcement learning inqueueing systems, see [21, Section 5], and this motivates us to examine the total regret in this context.
Revised: Within the rich literature of structured reinforcement learning however, few papers are dedicated to reinforcement learning in queueing systems, see [22, Section 5], and this motivates us to examine the total regret in this context.

Original: Typical control problems on queues have the following distinguishing characteristics:
Revised: Typical control problems on queues have the following distinguishing characteristics:

Original: No discount .
Revised: No discount .

Original: Discounting costs or rewards is common practice in the reinforcement learning literature, especially in Q-learning algorithms [18].
Revised: Discounting costs or rewards is common practice in the reinforcement learning literature, especially in Q-learning algorithms [19].

Original: However, in queues one is typically interested in optimizing with respect to the average cost.
Revised: However, in queues one is typically interested in optimizing with respect to the average cost.

Original: Large diameter.
Revised: Large diameter.

Original: Queueing systems are usually investigated under a drift condition that makes thesystem stable , i.e., positive recurrent.
Revised: Queueing systems are usually investigated under a drift condition that makes the system stable , i.e., positive recurrent.

Original: This condition implies that some states are hard to reach.
Revised: This condition implies that some states are hard to reach.

Original: In fact, for many queueing control problems, the diameter D is exponential in the size of the state space.
Revised: In fact, for many queueing control problems, the diameter D is exponential in the size of the state space.

Original: Even in the simple case of an M/M/1 queue with a ﬁnite buffer, or equivalently a birth–death process with a ﬁnite state space and constant birth and death rates, the diameter is exponential in the size of the state space.
Revised: Even in the simple case of an M/M/1 queue with a ﬁnite buffer, or equivalently a birth–death process with a ﬁnite state space and constant birth and death rates, the diameter is exponential in the size of the state space.

Original: Structured transition matrices.
Revised: Structured transition matrices.

Original: Queueing models describe how jobs join and leave queues, andthis yields bounded state transitions.
Revised: Queueing models describe how jobs join and leave queues, and this yields bounded state transitions.

Original: As a result, MDPs on queues have sparse and structuredtransition matrices.
Revised: As a result, MDPs on queues have sparse and structured transition matrices.

Original: The regret bounds discussed above and item 2 may suggest that the total regret of existing learningalgorithm, when applied to queueing systems, is large.
Revised: The regret bounds discussed above and item 2 may suggest that the total regret of existing learning algorithm, when applied to queueing systems, is large.

Original: However, they often work well in practice andthis bring us to consider the following question: When the underlying MDP has the structure of aqueueing system, does the diameter D or the number of states S actually play a role in the regret?
Revised: However, they often work well in practice and this bring us to consider the following question: When the underlying MDP has the structure of a queueing system, do the diameter D or the number of states S actually play a role in the regret?

Original: Our Contribution.
Revised: Our Contribution.

Original: In this paper, we examine the previous question with respect to the class ofcontrol problems presented in [1].
Revised: In this paper, we examine the previous question with respect to the class of control problems presented in [1].

Original: Speciﬁcally, an inﬁnite sequence of jobs joins a service systemover time to receive some processing according to the ﬁrst-come ﬁrst-served scheduling rule; thesystem can buffer at most S − 1 jobs and in fact it corresponds to an M/M/S-1/S-1 queue.
Revised: Speciﬁcally, an inﬁnite sequence of jobs joins a service system over time to receive some processing according to the ﬁrst-come ﬁrst-served scheduling rule; the system can buffer at most S − 1 jobs and in fact it corresponds to an M/M/1/S-1 queue.

Original: In addition,each job comes with a deadline constraint, and if a job is not completed before its deadline, thenit becomes obsolete and is removed from the system.
Revised: In addition, each job comes with a deadline constraint, and if a job is not completed before its deadline, then it becomes obsolete and is removed from the system.

Original: The controller chooses the server processingspeed and the objective is to design a speed policy for the server that minimizes its average energyconsumption plus an obsolescence cost per deadline miss.
Revised: The controller chooses the server processing speed and the objective is to design a speed policy for the server that minimizes its average energy consumption plus an obsolescence cost per deadline miss.

Original: Although this may look quite speciﬁc, thisproblem captures the typical characteristics of a controlled queue: i) the transition matrix has thestructure of a birth and death process with jump probabilities that are afﬁne functions of the state andii) the reward is linear in the state and convex in the action.
Revised: Although this may look quite speciﬁc, this problem captures the typical characteristics of a controlled queue: i) the transition matrix has the structure of a birth and death process with jump probabilities that are afﬁne functions of the state and ii) the reward is linear in the state and convex in the action.

Original: For any MDP in this class, deﬁned in fulldetails in Section 3, we show that the diameter is D = Ω( S S − 2 ) ; see Appendix B.3.
Revised: For any MDP in this class, deﬁned in full details in Section 3, we show that the diameter is D = Ω( S S − 2 ) ; see Appendix B.3.

Original: Thus, withoutexploiting the particular structure of this MDP, the existing lower and upper bounds do not justify thereason why standard learning algorithms work efﬁciently here.
Revised: Thus, without exploiting the particular structure of this MDP, the existing lower and upper bounds do not justify the reason why standard learning algorithms work efﬁciently here.

Original: We provide a slight variation of the learning algorithm U CRL 2, introduced in [10], and show inour main result that the resulting regret is upper bounded by ˜ O ( √ E 2 AT ) where E 2 is a term thatdepends on the stationary measure of a reference policy deﬁned in Section 3.1.
Revised: We provide a slight variation of the learning algorithm U CRL 2, introduced in [11], and show in our main result that the resulting regret is upper bounded by ˜ O ( √ E 2 AT ) where E 2 is a term that depends on the stationary measure of a reference policy deﬁned in Section 3.1.

Original: Importantly, E 2 doesnot depend on S .
Revised: Importantly, E 2 does not depend on S .

Original: Thus, efﬁcient reinforcement learning can be achieved independently of the numberof states by exploiting the stationary structure of the MDP.
Revised: Thus, efﬁcient reinforcement learning can be achieved independently of the number of states by exploiting the stationary structure of the MDP.

Original: Let us provide some intuition about ourresult.
Revised: Let us provide some intuition about our result.

Original: First, one may think that any learning algorithm should visit each state a sufﬁcient numberof times, which justiﬁes why the diameter of an MDP appears in existing regret analyses.
Revised: First, one may think that any learning algorithm should visit each state a sufﬁcient number of times, which justiﬁes why the diameter of an MDP appears in existing regret analyses.

Original: However,this point of view does not take into account the fact that the value of an MDP is the scalar productof the reward and the stationary measure of the optimal policy.
Revised: However, this point of view does not take into account the fact that the value of an MDP is the scalar product of the reward and the stationary measure of the optimal policy.

Original: If this stationary measure is “highlynon-uniform”, then some states are rarely visited under the optimal policy and barely contribute tothe value.
Revised: If this stationary measure is “highly non-uniform”, then some states are rarely visited under the optimal policy and barely contribute to the value.

Original: In this case, we claim that the learner may not need to visit the rare states that often to geta good estimation of the value, and thus it may not need to pay for the diameter.
Revised: In this case, we claim that the learner may not need to visit the rare states that often to get a good estimation of the value, and thus it may not need to pay for the diameter.

Original:  Reinforcement Learning Framework
Revised:  Reinforcement Learning Framework

Original: We consider a unichain Markov Decision Process (MDP) M
Revised: We consider a unichain Markov Decision Process (MDP) M

Original: =
Revised: =

Original: ( S , A , P, r ) in discrete time where Sis the ﬁnite state space,
Revised: ( S , A , P, r ) in discrete time where S is the ﬁnite state space,

Original: A the ﬁnite action space, P the transition probabilities and r the expectedreward function [15].
Revised: A the ﬁnite action space, P the transition probabilities and r the expected reward function [16].

Original: Let also S := |S| and A := |A| where | · | is the set cardinality operator.
Revised: Let also S := |S| and A := |A| where | · | is the set cardinality operator.

Original: Themodel-based reinforcement learning problem consists in ﬁnding a learning algorithm , or learner,that chooses actions to maximize a cumulative reward over a ﬁnite time horizon T .
Revised: The model-based reinforcement learning problem consists in ﬁnding a learning algorithm , or learner, that chooses actions to maximize a cumulative reward over a ﬁnite time horizon T .

Original: At each timestep t ∈ N , the system is in state s t ∈ S and the learner chooses an action a t ∈ A .
Revised: At each time step t ∈ N , the system is in state s t ∈ S and the learner chooses an action a t ∈ A .

Original: When executing a t ,the learner receives a random reward r t ( s t , a t ) with mean r ( s t , a t ) and the system moves, at timestep t +1 , to state s (cid:48) with probability P ( s (cid:48) | s t , a t ) .
Revised: When executing a t , the learner receives a random reward r t ( s t , a t ) with mean r ( s t , a t ) and the system moves, at time step t +1 , to state s (cid:48) with probability P ( s (cid:48) | s t , a t ) .

Original: The learning algorithm does not know the MDP Mexcept for the sets S and A .
Revised: The learning algorithm does not know the MDP M except for the sets S and A .

Original: For simplicity, in the following we consider weakly communicating MDPs.
Revised: For simplicity, in the following we consider weakly communicating MDPs.

Original: Since we will be interestedin the long-run average cost, this will let us remove the dependence on the initial state for severalquantities of interest.
Revised: Since we will be interested in the long-run average cost, this will let us remove the dependence on the initial state for several quantities of interest.

Original: 2.1 Undiscounted Regret
Revised: 2.1 Undiscounted Regret

Original: Given an MDP M , let Π :=
Revised: Given an MDP M , let Π :=

Original: { π : S → A} denote the set of stationary and deterministic policies andlet
Revised: { π : S → A} denote the set of stationary and deterministic policies and let

Original: [Equation]
Revised: [Equation]

Original: denote the average reward induced by policy π .
Revised: denote the average reward induced by policy π .

Original: Since M has ﬁnite state and action spaces, wenotice that i)
Revised: Since M has ﬁnite state and action spaces, we notice that i)

Original: The limit in (1) always exists, ii)
Revised: The limit in (1) always exists, ii)

Original: It does not depend on the initial state s 0 when Mis unichain [15] and iii)
Revised: It does not depend on the initial state s 0 when M is unichain [16] and iii)

Original: The restriction to stationary and deterministic policies is not a loss ofoptimality [15, Theorem 8.4.5].
Revised: The restriction to stationary and deterministic policies is not a loss of optimality [16, Theorem 8.4.5].

Original: Let also ρ ∗ :
Revised: Let also ρ ∗ :

Original: = ρ ∗ ( M ) := max π ∈ Π ρ ( M, π ) be the optimal average reward.
Revised: = ρ ∗ ( M ) := max π ∈ Π ρ ( M, π ) be the optimal average reward.

Original: Deﬁnition 2.1 (Regret) .
Revised: Deﬁnition 2.1 (Regret) .

Original: The regret at time T of the learning algorithm L is
Revised: The regret at time T of the learning algorithm L is

Original: [Equation]
Revised: [Equation]

Original: The regret 2 is a natural benchmark for evaluating the performance of a learning algorithm.
Revised: The regret 2 is a natural benchmark for evaluating the performance of a learning algorithm.

Original: In [10],a universal lower bound on Reg( M, L , T ) has been developed in terms of the diameter of theunderlying MDP.
Revised: In [11], a universal lower bound on Reg( M, L , T ) has been developed in terms of the diameter of the underlying MDP.

Original: Deﬁnition 2.2 (Diameter of an MDP) .
Revised: Deﬁnition 2.2 (Diameter of an MDP) .

Original: Let π : S
Revised: Let π : S

Original: → A be a stationary policy of M with initial state s .
Revised: → A be a stationary policy of M with initial state s .

Original: Let T ( s (cid:48) | M, π, s ) :
Revised: Let T ( s (cid:48) | M, π, s ) :

Original: = min { t ≥ 0 : s t =
Revised: = min { t ≥ 0 : s t =

Original: s (cid:48) | s 0 = s } be the random variable for the ﬁrst time step in which s (cid:48) is reached from s under π .
Revised: s (cid:48) | s 0 = s } be the random variable for the ﬁrst time step in which s (cid:48) is reached from s under π .

Original: Then, we say that the diameter of M is
Revised: Then, we say that the diameter of M is

Original: [Equation]
Revised: [Equation]

Original: It should be clear that the diameter of an MDP can be large if there exist states that are hard to reach.
Revised: It should be clear that the diameter of an MDP can be large if there exist states that are hard to reach.

Original: Within the set of structured MDPs considered in this paper, this will be the case and we will show that
Revised: Within the set of structured MDPs considered in this paper, this will be the case and we will show that

Original: D = Ω( S S − 2 ) .
Revised: D = Ω( S S − 2 ) .

Original: The following result shows that all learning algorithms have a regret that eventuallyincrease with√ D .
Revised: The following result shows that all learning algorithms have a regret that eventually increase with D .

Original: Theorem 2.3 (Universal lower bound [10]) .
Revised: Theorem 2.3 (Universal lower bound [11]) .

Original: For any learning algorithm L , any natural numbers
Revised: For any learning algorithm L , any natural numbers

Original: S, A ≥ 10 , D ≥ 20 log A S , and T ≥ DSA , there is an MDP M with S states, A actions, anddiameter D such that for any initial state s ∈ S ,
Revised: S, A ≥ 10 , D ≥ 20 log A S , and T ≥ DSA , there is an MDP M with S states, A actions, and diameter D such that for any initial state s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: In view of this result, the diameter of an MDP and its state space appear to be critical parameters forevaluating the performance of a learning algorithm.
Revised: In view of this result, the diameter of an MDP and its state space appear to be critical parameters for evaluating the performance of a learning algorithm.

Original: 2.2 The U CRL 2 Algorithm
Revised: 2.2 The U CRL 2 Algorithm

Original: We now focus on U CRL 2, a classical reinforcement learning algorithm introduced in [10] that is avariant of U CRL [2].
Revised: We now focus on U CRL 2, a classical reinforcement learning algorithm introduced in [11] that is a variant of U CRL [2].

Original: While more efﬁcient algorithms have been proposed for the general case (seefor example [3, 20]), we will show that U CRL 2 already achieves a very low regret, namely ˜ O ( √ AT ) ,independent of S so using more reﬁned algorithms can only bring marginal gains.
Revised: While more efﬁcient algorithms have been proposed for the general case (see for example [3, 21]), we will show that U CRL 2 already achieves a very low regret, namely ˜ O ( √ AT ) , independent of S so using more reﬁned algorithms can only bring marginal gains.

Original: U CRL 2 is based on episodes .
Revised: U CRL 2 is based on episodes .

Original: For each episode k , let t k denote its start time.
Revised: For each episode k , let t k denote its start time.

Original: For each state s andaction a , let ν k ( s, a ) denote the number of visits of ( s, a ) during episode k and let N t ( s, a ) :
Revised: For each state s and action a , let ν k ( s, a ) denote the number of visits of ( s, a ) during episode k and let N t ( s, a ) :

Original: =# { τ < t : s τ = s, a τ = a } denote the number of visits of ( s, a ) until timestep t .
Revised: = # { τ < t : s τ = s, a τ = a } denote the number of visits of ( s, a ) until timestep t .

Original: Let M k be theconﬁdence set of MDPs with transition probabilities ˜ p and rewards ˜ r that are “close” to the empirical
Revised: Let M k be the conﬁdence set of MDPs with transition probabilities ˜ p and rewards ˜ r that are “close” to the empirical

Original: MDP at episode k , ˆ p k and ˆ r k , i.e., ˜ p and ˜ r satisfy
Revised: MDP at episode k , ˆ p k and ˆ r k , i.e., ˜ p and ˜ r satisfy

Original: [Equation]
Revised: [Equation]

Original: [Equation] With these quantities, a pseudocode for U CRL 2 is given in Algorithm 1.
Revised: With these quantities, a pseudocode for U CRL 2 is given in Algorithm 1.

Original: We notice that U CRL 2relies on Extended Value Iteration (EVI), that is a variant of the celebrated Value Iteration (VI)algorithm [15]; for further details about EVI, we point the reader to [10, Section 3.1].
Revised: We notice that U CRLrelies on Extended Value Iteration (EVI), that is a variant of the celebrated Value Iteration (VI) algorithm [16]; for further details about EVI, we point the reader to [11, Section 3.1].

Original: Let us commenton how U CRL 2 works.
Revised: Let us comment on how U CRL 2 works.

Original: There are three main steps.
Revised: There are three main steps.

Original: First, at the start of each episode, U CRL
Revised: First, at the start of each episode, U CRL

Original: Algorithm 
Revised: Algorithm 1:

Revised: The

Original: Compute the estimates s a and p k 
Revised: 2 algorithm. Input: A conﬁdence parameter δ ∈ (0 , 1) , S and A . Output: . 1 Set t := 1 and observe s2 for episodes k = 1 , 2 , . . .

Original: ( s
Revised: do ˆ

Original:  | s a )as in 
Revised: (cid:48) | s, a ) in (7).

Original: Use “Extended Value Iteration” to ﬁnd a policy ˜ π k and an optimistic MDP˜ M k ∈ M k such that
Revised: optimistic MDP that

Original: [Equation]
Revised: [Equation]

Original: while νk s t
Revised: while ν

Original:  ˜ π k  s t ) < max { 
Revised: ( s , ˜ π k ( s t )) < max { 1 , N t k ( s t , ˜ π k ( s t )) } do ( s

Original: N t k ( s t ,˜ π k ( s t )) } do a 
Revised: , a t ) : = ν ( s t , a t ) + 1 ; t :=

Original:  end 10 end139 computes the empirical estimates
Revised: t + 1 ; end 10 end computes the empirical estimates

Original: [Equation]
Revised: [Equation]

Original: of the reward and probability transitions
Revised: of the reward and probability transitions

Original: , respectively, where 1 E is the indicator function of E .
Revised: , respectively, where 1 E is the indicator function of E .

Original: Then,it applies Extended Value Iteration (EVI) to ﬁnd a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k suchthat (6) holds true.
Revised: Then, it applies Extended Value Iteration (EVI) to ﬁnd a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k such that (6) holds true.

Original: Finally, it executes policy ˜
Revised: Finally, it executes policy ˜

Original: π k until it ﬁnds a state-action pair ( s, a ) whose countwithin episode k is greater than the corresponding state-action count prior to episode k .
Revised: π k until it ﬁnds a state-action pair ( s, a ) whose count within episode k is greater than the corresponding state-action count prior to episode k .

Original:  Controlled Birth and Death Processes for Energy Minimization
Revised:  Controlled Birth and Death Processes for Energy Minimization

Original: Now, we focus on a speciﬁc class of MDPs that has been introduced in [1], which provides a rathergeneral example of a controlled birth and death process with convex costs on the actions and linearrates.
Revised: Now, we focus on a speciﬁc class of MDPs that has been introduced in [1], which provides a rather general example of a controlled birth and death process with convex costs on the actions and linear rates.

Original: We will denote by M the set of MDPs with the structure described below.
Revised: We will denote by M the set of MDPs with the structure described below.

Original: The MDPsin M have been proposed to represent a Dynamic Voltage and Frequency Scaling (DVFS) processor executing jobs with soft obsolescence deadlines.
Revised: The MDPs in M have been proposed to represent a Dynamic Voltage and Frequency Scaling (DVFS) processor executing jobs with soft obsolescence deadlines.

Original: Here, jobs arrive according to a Poisson processwith rate λ ∈ [0 , λ max ] in a buffer of size S − 1 .
Revised: Here, jobs arrive according to a Poisson process with rate λ ∈ [0 , λ max ] in a buffer of size S − 1 .

Original: If the buffer is full and a job arrives, then the job isrejected.
Revised: If the buffer is full and a job arrives, then the job is rejected.

Original: Each job has a deadline and a size, i.e., amount of work, which are exponentially distributedrandom variables with rates µ ∈ [0 , µ max ] and one, respectively.
Revised: Each job has a deadline and a size, i.e., amount of work, which are exponentially distributed random variables with rates µ ∈ [0 , µ max ] and one, respectively.

Original: Job deadlines and sizes are allindependent random variables.
Revised: Job deadlines and sizes are all independent random variables.

Original: If a job misses its deadline, which is a real time constraint activated atthe moment of its arrival, it is removed from the queue without being served and a cost C is paid.
Revised: If a job misses its deadline, which is a real time constraint activated at the moment of its arrival, it is removed from the queue without being served and a cost C is paid.

Original: Theprocessor serves jobs under any work-conserving scheduling discipline, e.g., ﬁrst-come ﬁrst-served,with a processing speed that belongs to the ﬁnite set { 0 , . . .
Revised: The processor serves jobs under any work-conserving scheduling discipline, e.g., ﬁrst-come ﬁrst-served, with a processing speed that belongs to the ﬁnite set { 0 , . . .

Original: , A max } .
Revised: , A max } .

Original: The objective is to design aspeed policy that minimizes the sum of the long term power dissipation and the cost induced by jobsmissing their deadlines.
Revised: The objective is to design a speed policy that minimizes the sum of the long term power dissipation and the cost induced by jobs missing their deadlines.

Original: When the processor works at speed a ∈ { 0 , . .
Revised: When the processor works at speed a ∈ { 0 , . .

Original: A max } , it processes a unitsof work per second while its power dissipation is w ( a ) .
Revised: A max } , it processes a units of work per second while its power dissipation is w ( a ) .

Original: After uniformization, it is shown in [1] that this control problem can be modeled as an MDP indiscrete time with a “birth-and-death” transition matrix of size S .
Revised: After uniformization, it is shown in [1] that this control problem can be modeled as an MDP in discrete time with a “birth-and-death” transition matrix of size S .

Original: Speciﬁcally, we have an MDP
Revised: Speciﬁcally, we have an MDP

Original: M = ( S , A , P, r ) where
Revised: M = ( S , A , P, r ) where

Original: S = { 0 , . .
Revised: S = { 0 , . .

Original: , S − 1 } , with s ∈ S representing the number of jobs in thesystem, and A = { 0 , . . .
Revised: , S − 1 } , with s ∈ S representing the number of jobs in the system, and A = { 0 , . . .

Original: , A max } , with a ∈ A representing the processor speed.
Revised: , A max } , with a ∈ A representing the processor speed.

Original: Then, the transitionprobabilities under policy π are given by
Revised: Then, the transition probabilities under policy π are given by

Original: [Equation]
Revised: P i,j ( π ) =   1 U

Original:  if [Equation]
Revised: λ i if i < S − 1 and j = i +1 U ( π ( i ) + iµ ) if i > 0 and j = i −

Original:  otherwise , where U := 
Revised: P ii if j = i otherwise , where U := λ max

Original: λ max +( S − 1) µ max + A max is a uniformization constant, P ii = 1 U
Revised: +( S − 1) µ max + A max is a uniformization constant, P ii = 1 U

Original: ( U − λ
Revised: ( U − λ

Original: i − µi − π ( i ))and λ
Revised: i − µi − π ( i )) and λ

Original: i := λ (cid:16) 1 − iS − 1 (cid:17) is the decaying arrival rate.
Revised: i := λ (cid:16) 1 − iS − 1 (cid:17) is the decaying arrival rate.

Original: We have replaced the constant arrival rate λ bya decaying arrival rate
Revised: We have replaced the constant arrival rate λ by a decaying arrival rate

Original: λ i because we want to learn an optimal policy that does not exploit the buffersize S − 1 ; see [1] for further details.
Revised: λ i because we want to learn an optimal policy that does not exploit the buffer size S − 1 ; see [1] for further details.

Original: For conciseness, Figure 1 displays the transition diagram of the
Revised: For conciseness, Figure 1 displays the transition diagram of the

Original: Markov chain induced by policy π .
Revised: Markov chain induced by policy π .

Original: [Equation]
Revised: [Equation]

Original: Finally, the reward is a combination of C , the constant cost due to a departing job missing its deadlineand w ( a ) , an arbitrary convex function of a , giving the energy cost for using speed a .
Revised: Finally, the reward is a combination of C , the constant cost due to a departing job missing its deadline and w ( a ) , an arbitrary convex function of a , giving the energy cost for using speed a .

Original: The immediatecost c ( s, a ) in state s under action a is a random variable whose value is w ( a ) + C with probabilityiµ/U (missed deadline) and w ( a ) otherwise.
Revised: The immediate cost c ( s, a ) in state s under action a is a random variable whose value is w ( a ) + C with probability iµ/U (missed deadline) and w ( a ) otherwise.

Original: To keep in line with the use of rewards instead of costs,we introduce a bound on the cost, r max :=
Revised: To keep in line with the use of rewards instead of costs, we introduce a bound on the cost, r max :=

Original: C + w ( A max ) so that the reward in state s under action ais a positive and bounded random variable given by
Revised: C + w ( A max ) so that the reward in state s under action a is a positive and bounded random variable given by

Original: [Equation]
Revised: [Equation]

Original: As in Section 2.1, ρ ∗ ( M ) is the optimal average cost and ρ ( M, π ) is the average cost induced bypolicy π , where π belongs to the set of deterministic and stationary policies Π .
Revised: As in Section 2.1, ρ ∗ ( M ) is the optimal average cost and ρ ( M, π ) is the average cost induced by policy π , where π belongs to the set of deterministic and stationary policies Π .

Original: Since the underlying
Revised: Since the underlying

Original: Markov chain induced by any policy is ergodic, we observe that
Revised: Markov chain induced by any policy is ergodic, we observe that

Original: [Equation]
Revised: [Equation]

Original: ) where m π is the stationary measure under policy π .
Revised: where m π is the stationary measure under policy π .

Original: In [1], it has been shown that the optimal policyis unique and will be 
Revised: In [1], it has been shown that the optimal policy is unique and will be denoted by π ∗ .

Original: 3.1 Properties of M
Revised: 3.1 Properties of M

Original: In the following, we will use the “reference” (or bounding) policy π 0 ( s ) = 0 for all s ∈ S , which thusassigns speed 0 to all states.
Revised: In the following, we will use the “reference” (or bounding) policy π 0 ( s ) = 0 for all s ∈ S , which thus assigns speed 0 to all states.

Original: This policy provides a stochastic bound on all policies in the followingsense.
Revised: This policy provides a stochastic bound on all policies in the following sense.

Original: Let s πt be the state under policy π and let ≤ st denote the stochastic order [17]; given tworandom variables X and Y on R + , we recall that X ≤ st Y if P ( X ≥ s ) ≤ P ( Y ≥ s ) for all s .
Revised: Let s πt be the state under policy π and let ≤ st denote the stochastic order [18]; given two random variables X and Y on R + , we recall that X ≤ st Y if P ( X ≥ s ) ≤ P ( Y ≥ s ) for all s .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Consider an MDP in M .
Revised: Consider an MDP in M .

Original: For all t and policy π ∈ Π , s πt ≤ st s π 0 t , provided thats π 0 ≤ st s π 0 0 .
Revised: For all t and policy π ∈ Π , s πt ≤ st s π 0 t , provided that s π 0 ≤ st s π 0 0 .

Original: Proof.
Revised: Proof.

Original: (sketch)
Revised: (sketch)

Original: The proof follows by a simple coupling argument between the two policies.
Revised: The proof follows by a simple coupling argument between the two policies.

Original: Roughlyspeaking, each time the Markov chain under π decreases from s to s − 1 because of the speed π ( s ) ,it stays in state s under policy π 0 .
Revised: Roughly speaking, each time the Markov chain under π decreases from s to s − 1 because of the speed π ( s ) , it stays in state s under policy π 0 .

Original: Therefore, P ( s πt ≥ s ) ≤ P ( s π 0 t ≥ s ) for all s and t , which also implies that the respective stationarymeasures are comparable, i.e., (cid:80) S − 1 i = s m πi ≤ (cid:80) S − 1 i = s m π
Revised: Therefore, P ( s πt ≥ s ) ≤ P ( s π 0 t ≥ s ) for all s and t , which also implies that the respective stationary measures are comparable, i.e., (cid:80) S − 1 i = s m πi ≤ (cid:80) S − 1 i = s m π

Original: 0 i .
Revised: 0 i .

Original: Let us now consider H ( s ) , the bias at state s of the optimal policy π ∗ , deﬁned by
Revised: Let us now consider H ( s ) , the bias at state s of the optimal policy π ∗ , deﬁned by

Original: [Equation]
Revised: [Equation]

Revised: s

Revised: ≤ S − 1 , (10)

Original: Let also ∂H ( s ) :
Revised: Let also ∂H ( s ) :

Original: =
Revised: =

Original: H ( s ) − H ( s − 1) be the local variation of the bias.
Revised: H ( s ) − H ( s − 1) be the local variation of the bias.

Original: The following result was shown in [1, Lemma 3.8].
Revised: The following result was shown in [1, Lemma 3.8].

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: The local variation of the bias, ∂H ( s ) , is negative, decreasing in s , and bounded:− ∂H ( s ) ≤ ∆( s ) with 0 < ∆( s ) ≤ C for all 1 ≤ s ≤ S − 1 .
Revised: The local variation of the bias, ∂H ( s ) , is negative, decreasing in s , and bounded: − ∂H ( s ) ≤ ∆( s ) with 0 < ∆( s ) ≤ C for all 1 ≤ s ≤ S − 1 .

Original: Both m π 0 and ∆ will play a major role in our analysis of the regret.
Revised: Both m π 0 and ∆ will play a major role in our analysis of the regret.

Original: 3.2 Applying U CRL 2 in M
Revised: 3.2 Applying U CRL 2 in M

Original: We assume that the bounds λ max and µ max are ﬁxed so that r max is known to the learner.
Revised: We assume that the bounds λ max and µ max are ﬁxed so that r max is known to the learner.

Original: This is aclassical assumption, often replaced by assuming that rewards live in [0 , 1] .
Revised: This is a classical assumption, often replaced by assuming that rewards live in [0 , 1] .

Original: In the remainder, we will apply U CRL 2 over an MDP in M with a change in the conﬁdence boundsto take into account the support of P .
Revised: In the remainder, we will apply U CRL 2 over an MDP in M with a change in the conﬁdence bounds to take into account the support of P .

Original: The conﬁdence bounds in (4) (resp.
Revised: The conﬁdence bounds in (4) (resp.

Original: (5)) are replaced byr max (cid:114)
Revised: (5)) are replaced by r max (cid:114)

Original: [Equation]
Revised: [Equation]

Original: We also impose that the conﬁdence set M k onlycontains matrices with the same support as P .
Revised: We also impose that the conﬁdence set M k only contains matrices with the same support as P .

Original: Removing S in the conﬁdence bounds does help toreduce the regret.
Revised: Removing S in the conﬁdence bounds does help to reduce the regret.

Original: However, by using existing analysis, this only removes a factor √ S in the regretbound (for example, see [4]).
Revised: However, by using existing analysis, this only removes a factor √ S in the regret bound (for example, see [4]).

Original: Finally, note that U CRL 2 does not beneﬁt from the parametric nature of the MDPs in M , which isessentially deﬁned by three parameters ( λ , µ and C ) and the real convex function w ( · ) .
Revised: Finally, note that U CRL 2 does not beneﬁt from the parametric nature of the MDPs in M , which is essentially deﬁned by three parameters ( λ , µ and C ) and the real convex function w ( · ) .

Original:  Regret of U CRL 2 on M
Revised:  Regret of U CRL 2 on M

Original: Our objective is to develop an upper bound on the regret of the learning algorithm
Revised: Our objective is to develop an upper bound on the regret of the learning algorithm

Original: U CRL 2 whenapplied to MDPs in our class M .
Revised: U CRL 2 when applied to MDPs in our class M .

Original: The driving idea is to construct a bound that exploits the structureof the stationary measure of all policies, as they all make some states hard to reach, and to control thenumber of visits to these states to get a new type of bound.
Revised: The driving idea is to construct a bound that exploits the structure of the stationary measure of all policies, as they all make some states hard to reach, and to control the number of visits to these states to get a new type of bound.

Original: 4.1 Main Result
Revised: 4.1 Main Result

Original: The following theorem gives an upper bound on the regret that does not depend on the classicalparameters such as the size of the state space nor on global quantities such as the diameter of the
Revised: The following theorem gives an upper bound on the regret that does not depend on the classical parameters such as the size of the state space nor on global quantities such as the diameter of the

Original: MDP nor the span of the bias of some policy.
Revised: MDP nor the span of the bias of some policy.

Original: Instead, the regret bound below mainly depends onthe weighted second moment of the stationary measure of the reference policy π 0 , which is boundedindependently of the size of the state space.
Revised: Instead, the regret bound below mainly depends on the weighted second moment of the stationary measure of the reference policy π 0 , which is bounded independently of the size of the state space.

Original: We consider the policy π max such that π max ( s ) =
Revised: We consider the policy π max such that π max ( s ) =

Original: A max for all s and m max its stationary measure.
Revised: A max for all s and m max its stationary measure.

Original: Let us also recall that m π 0 is the stationary measure of the Markov chain under policy π 0 ( s ) =
Revised: Let us also recall that m π 0 is the stationary measure of the Markov chain under policy π 0 ( s ) =

Original: 0for all s and that ∆ : S → R + is a function bounding the local variations of the optimal bias.
Revised: for all s and that ∆ : S → R + is a function bounding the local variations of the optimal bias.

Original: Let
Revised: Let

Original: E 2 :=
Revised: E 2 :=

Original: F E m π [-
 [(∆-] + r max )  · f ] with f : s (cid:55)→ max { 1 ,s ( s − 1) } ∆( s )+ r max  and F := (cid:80) s ∈S f ( s ) − 1 .
Revised: F E m π [-
 [(∆-]0 (cid:2) (∆ + r max ) 2 · f (cid:3) with f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max ) 2 and F := (cid:80) s ∈S f ( s ) − 1 .

Original: Here, E 2 isclosely related to the second moment of the measure m π 0 weighted by the bias variations and themaximal reward.
Revised: Here, Eis closely related to the second moment of the measure m π 0 weighted by the bias variations and the maximal reward.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let M ∈ M .
Revised: Let M ∈ M .

Original: Deﬁne 
Revised: Deﬁne Q max := (cid:16) [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Revised: E [Reg

Original: [Figure] [Table]
Revised: [Equation] [Equation] [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: AT

Original: [Equation]
Revised: [Equation]

Original:  Here, E 2 ≤ 12 r 2max (cid:16) 1 + λ 2 µ 2 (cid:17) , so that the regret satisﬁes
Revised: AT ) + [Equation] Here, E 2 ≤ 12 r 2max (cid:16) 1 + λ 2 µ 2 (cid:17) , so that the regret satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: (cid:33) .

Original: Before giving a sketch of the proof, let us comment on the bound (11).
Revised: Before giving a sketch of the proof, let us comment on the bound (11).

Original: Although the ﬁrst term is oforder √ T with a multiplicative constant independent of S - as desired - the second term, of order
Revised: Although the ﬁrst term is of order √ T with a multiplicative constant independent of S - as desired - the second term, of order

Original: T 1 / 4 , contains very large terms.
Revised: T 1 / 4 , contains very large terms.

Original: Its interest however, lies in the novel approach used in the proof thatuses the stationary behavior of the algorithm.
Revised: Its interest however, lies in the novel approach used in the proof that uses the stationary behavior of the algorithm.

Original: 4.2 Comparison with 
Revised: 4.2 Comparison with Other Bounds

Original: Other Bounds

Original: Let us compare our upper bound with the ones existing in the literature, as we claim that ours is of adifferent nature.
Revised: Let us compare our upper bound with the ones existing in the literature, as we claim that ours is of a different nature.

Original: First, let us compare with the bound given in [10], which states that with probability 1 − δ ,
Revised: First, let us compare with the bound given in [11], which states that with probability 1 − δ ,

Original: Reg( M, T ) ≤ 34 DS (cid:113) AT log (cid:0) Tδ (cid:1) for any T > 1 .
Revised: Reg( M, T ) ≤ 34 DS (cid:113) AT log (cid:0) Tδ (cid:1) for any T > 1 .

Original: For any M ∈ M , the diameter grows as
Revised: For any M ∈ M , the diameter grows as

Original: S S (see Appendix B.3), thus this bound is very loose here.
Revised: S S (see Appendix B.3), thus this bound is very loose here.

Original: More recent works have improved thisbound by replacing the term D by the local diameter of the MDP [5].
Revised: More recent works have improved this bound by replacing the term D by the local diameter of the MDP [5].

Original: In Appendix B.3, we showthat the local diameter grows again as S S for any M ∈ M , and thus these results do not yieldsigniﬁcant improvements.
Revised: In Appendix B.3, we show that the local diameter grows again as S S for any M ∈ M , and thus these results do not yield signiﬁcant improvements.

Original: Other papers show that the diameter can be replaced by the span of thebias, see [7, 24].
Revised: Other papers show that the diameter can be replaced by the span of the bias, see [7, 25].

Original: This has a big impact because the span of the bias, for any M ∈ M , is linear in S(instead of S S for the diameter); see Appendix B.3.
Revised: This has a big impact because the span of the bias, for any M ∈ M , is linear in S (instead of S S for the diameter); see Appendix B.3.

Original: However, this is still not as good as the boundgiven in Theorem 4.1, which is independent of S .
Revised: However, this is still not as good as the bound given in Theorem 4.1, which is independent of S .

Original: Now, let us compare with existing bounds for parametric MDPs, as mentioned in the introduction.
Revised: Now, let us compare with existing bounds for parametric MDPs, as mentioned in the introduction.

Original: The d -linear additive model, d < S , introduced in [11] assumes that P ( ·| s, a )
Revised: The d -linear additive model, d < S , introduced in [12] assumes that P ( ·| s, a )

Original: = (cid:104) φ ( s, a ) , θ ( · ) (cid:105) ,where φ ( s, a ) is a known feature mapping and θ is an unknown measure on R d .
Revised: = (cid:104) φ ( s, a ) , θ ( · ) (cid:105) , where φ ( s, a ) is a known feature mapping and θ is an unknown measure on R d .

Original: This form of P ( ·| s, a )implies that the transition kernel is of rank d .
Revised: This form of P ( ·| s, a ) implies that the transition kernel is of rank d .

Original: Unfortunately, this property does not hold true inbirth and death processes.
Revised: Unfortunately, this property does not hold true in birth and death processes.

Original: In fact, the kernel of any M ∈ M has almost full rank under all policies.
Revised: In fact, the kernel of any M ∈ M has almost full rank under all policies.

Original: The linear mixture model introduced in [25] assumes instead that P ( s (cid:48) | s, a )
Revised: The linear mixture model introduced in [26] assumes instead that P ( s (cid:48) | s, a )

Original: = (cid:104) φ ( s (cid:48) | s, a ) , θ (cid:105) ,θ ∈ R d .
Revised: = (cid:104) φ ( s (cid:48) | s, a ) , θ (cid:105) , θ ∈ R d .

Original: This is more adapted to our case, which can be (almost) seen as a linear mixture modelof dimension d = 3 .
Revised: This is more adapted to our case, which can be (almost) seen as a linear mixture model of dimension d = 3 .

Original: The bound on the discounted regret of the algorithm proposed in [25] is
Revised: The bound on the discounted regret of the algorithm proposed in [26] is

Original: Reg( M, T ) ≤ d √ T/ (1 − γ ) 2 where γ is a discount factor.
Revised: Reg( M, T ) ≤ d √ T/ (1 − γ ) 2 where γ is a discount factor.

Original: In contrast to our work, this regretanalysis holds for discounted problems, where we remark that both the diameter and the span areirrelevant.
Revised: In contrast to our work, this regret analysis holds for discounted problems, where we remark that both the diameter and the span are irrelevant.

Original: On the other hand, both are replaced by a term of the form (1 − γ ) − 2 , which impliesthat the previous bound grows to inﬁnity as γ ↑ 1 .
Revised: On the other hand, both are replaced by a term of the form (1 − γ ) − 2 , which implies that the previous bound grows to inﬁnity as γ ↑ 1 .

Original: More recently, a regret bound of O ( d √ DT ) hasbeen proven in [22] in the undiscounted case, that is the case considered in our work.
Revised: More recently, a regret bound of O ( d √ DT ) has been proven in [23] in the undiscounted case, that is the case considered in our work.

Original: However, thealgorithm presented in that reference highly depends on the diameter and is unsuitable for MDPswith a birth and death structure.
Revised: However, the algorithm presented in that reference highly depends on the diameter and is unsuitable for MDPs with a birth and death structure.

Original: Finally, our bound depends on the second moment of the stationary measure of a reference policy,i.e., E 2 , which can be bounded independently of the state space size.
Revised: Finally, our bound depends on the second moment of the stationary measure of a reference policy, i.e., E 2 , which can be bounded independently of the state space size.

Original: This is structurally different from the ones existing in the literature.
Revised: This is structurally different from the ones existing in the literature.

Original: We believe that this structure holds as well in a class of
Revised: We believe that this structure holds as well in a class of

Original: MDPs much larger than M .
Revised: MDPs much larger than M .

Original: In particular, if m is the stationary measure of some bounding/referencepolicy, and if the critical quantity E m [∆ · f ] is small for a well chosen function f , then the regret of alearning algorithm navigating the MDP should also be small.
Revised: In particular, if m is the stationary measure of some bounding/reference policy, and if the critical quantity E m [∆ · f ] is small for a well chosen function f , then the regret of a learning algorithm navigating the MDP should also be small.

Original: A deeper analysis is left as future work.
Revised: A deeper analysis is left as future work.

Original: 4.3 Sketch of the Proof
Revised: 4.3 Sketch of the Proof

Original: Our proof for Theorem 4.1 is technical and is provided in the supplementary material.
Revised: Our proof for Theorem 4.1 is technical and is provided in the supplementary material.

Original: In this section,we present the main ideas and its general structure.
Revised: In this section, we present the main ideas and its general structure.

Original: It initially relies on the regret analysis of U CRL 2developed in [10], and the differences are highlighted below.
Revised: It initially relies on the regret analysis of U CRLdeveloped in [11], and the differences are highlighted below.

Original: First, we consider the mean rewardsand split the regret into episodes to separately treat the cases where the true MDP is in the conﬁdenceset of optimistic MDPs M k or not.
Revised: First, we consider the mean rewards and split the regret into episodes to separately treat the cases where the true MDP is in the conﬁdence set of optimistic MDPs M k or not.

Original: Thus, let R k := (cid:80) s,a ν k ( s, a )(
Revised: Thus, let R k := (cid:80) s,a ν k ( s, a )(

Original: ρ ∗ − r ( s, a )) denote the regret inepisode k .
Revised: ρ ∗ − r ( s, a )) denote the regret in episode k .

Original: Then, Hoeffding’s inequality gives the following ﬁrst bound of the expected regret (see Appendix A.2):
Revised: This split can be written:

Original: [Equation]
Revised: [Equation]

Original: where R in := (cid:80) k
Revised: where R in := (cid:80) k

Original: R k 1 M ∈M k
Revised: R k 1 M ∈M k

Original: and R out := (cid:80)
Revised: and R out := (cid:80)

Original: k R k 1 M / ∈M k .
Revised: k R k 1 M / ∈M k .

Original: To control R out , we use, as in [10], the stopping criterion and the conﬁdence bounds.
Revised: To control R out , we use, as in [11], the stopping criterion and the conﬁdence bounds.

Original: This gives277 E [ R out ] ≤ r max S , so that the regret due to episodes where the conﬁdence regions fail will benegligible next to the main terms.
Revised: This gives E [ R out ] ≤ r max S , so that the regret due to episodes where the conﬁdence regions fail will be negligible next to the main terms.

Original: Then, when the true MDP belongs to the conﬁdence region, we usethe properties of Extended Value Iteration (EVI) to decompose R in into(cid:88)
Revised: Then, when the true MDP belongs to the conﬁdence region, we use the properties of Extended Value Iteration (EVI) to decompose R in into (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where v k is the vector of the state-action counts ν k ’s,
Revised: where v k is the vector of the state-action counts ν k ’s,

Original: ˜ P k and ˜h k are respectively the transition matrixand the bias in ˜ M k under policy ˜ π k , and d k is the proﬁle difference between the last step of EVI andthe bias (see Appendix A.3).
Revised: ˜ P k and ˜h k are respectively the transition matrix and the bias in ˜ M k under policy ˜ π k , and d k is the proﬁle difference between the last step of EVI and the bias (see Appendix A.3).

Original: We now show how to handle R rewards , R EVI , R bias .
Revised: We now show how to handle R rewards , R EVI , R bias .

Original: First, we deal with the terms that do not involvethe bias.
Revised: First, we deal with the terms that do not involve the bias.

Original: Using the conﬁdence bound on the rewards (see Appendix A.3.1:
Revised: Using the conﬁdence bound on the rewards (see Appendix A.3.1:

Original: [Equation]
Revised: [Equation]

Original: Now, let us consider R EVI .
Revised: Now, let us consider R EVI .

Original: Since d k becomes arbitrarily small after enough iterations of EVI (see
Revised: Since d k becomes arbitrarily small after enough iterations of EVI (see

Original: Appendix A.1), for T ≥ e 8 2 AT , we get
Revised: Appendix A.1), for T ≥ e 8 2 AT , we get

Original: [Equation]
Revised: [Equation]

Original: The analysis of R bias is different from the one in [10]: While in [10] the bias is directly bounded bythe diameter, we can use the variations of the bias to control the regret more precisely.
Revised: The analysis of R bias is different from the one in [11]: While in [11] the bias is directly bounded by the diameter, we can use the variations of the bias to control the regret more precisely.

Original: Using P k , i.e.,the transitions in the true MDP under ˜ π k , R bias is further decomposed into:
Revised: Using P k , i.e., the transitions in the true MDP under ˜ π k , R bias is further decomposed into:

Original: [Equation]
Revised: [Equation]

Original: The term R ep can be treated in a similar manner as in [10] by bounding the bias terms with thediameter to apply an Azuma-Hoeffding inequality (see Appendix A.3.5).
Revised: The term R ep can be treated in a similar manner as in [11] by bounding the bias terms with the diameter to apply an Azuma-Hoeffding inequality (see Appendix A.3.5).

Original: Here, we obtain
Revised: Here, we obtain

Revised: [Equation]

Revised: [Equation]

Revised: SAD

Original: [Equation]
Revised: [Equation]

Original: Next, we show in A.3.2 that R diﬀ does not contribute to the main term of the regret.
Revised: Next, we show in A.3.2 that R diﬀ does not contribute to the main term of the regret.

Original: This is oneof the hard point in our proof.
Revised: This is one of the hard point in our proof.

Original: First, linear algebra techniques are used to bound ||
Revised: First, linear algebra techniques are used to bound ||

Original: ˜ h
Revised: ˜ h

Original: k − h ∗ || ∞ by
Revised: k − h ∗ || ∞ by

Original: D (2 r max D ||
Revised: D (2 r max D ||

Original: ˜
Revised: ˜

Original: P k − P ∗ || ∞
Revised: P k − P ∗ || ∞

Original: + ||
Revised: + ||

Original: ˜
Revised: ˜

Original: r k − r ∗ || ∞ ) .
Revised: r k − r ∗ || ∞ ) .

Original: Each norm is then bounded using Hoeffding inequality.
Revised: Each norm is then bounded using Hoeffding inequality.

Original: This introduces the special quantity N t k ( x k , π k ( x k )) that yields to the worst conﬁdence bound in episode k .
Revised: This introduces the special quantity N t k ( x k , π k ( x k )) that yields to the worst conﬁdence bound in episode k .

Original: Then, an adaptation of McDiarmid’s inequality to Markov chains is used to show that
Revised: Then, an adaptation of McDiarmid’s inequality to Markov chains is used to show that

Original: N t k ( x k , π k ( x k )) ≥ ( t k +1 − t k ) m max ( S − 1) / 2 with high probability, where m max ( S − 1) is thestationary measure of state S − 1 under the uniform policy π max ( s )
Revised: N t k ( x k , π k ( x k )) ≥ ( t k +1 − t k ) m max ( S − 1) / 2 with high probability, where m max ( S − 1) is the stationary measure of state S − 1 under the uniform policy π max ( s )

Original: =
Revised: =

Original: A max .
Revised: A max .

Original: This eventually impliesthat
Revised: This eventually implies that

Original: [Equation]
Revised: [Equation]

Original: where Q [Equation] := 
Revised: where Q max := (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Then, to deal with the main term R trans , we exploit the optimal bias.
Revised: Then, to deal with the main term R trans , we exploit the optimal bias.

Original: The unit vector being in the kernel of ˜ P k − P k , we can rewrite:
Revised: The unit vector being in the kernel of ˜ P k − P k , we can rewrite:

Original: [Equation]
Revised: [Equation]

Original: and, thus, using the conﬁdence bound and the bounded variations of the bias,
Revised: and, thus, using the conﬁdence bound and the bounded variations of the bias,

Original: [Equation]
Revised: [Equation]

Original: We can now aggregate R trans , R rewards and R EVI to compute the main term of the regret (see
Revised: We can now aggregate R trans , R rewards and R EVI to compute the main term of the regret (see

Original: Appendix A.3.4).
Revised: Appendix A.3.4).

Original: Here, the key ingredient is to bound
Revised: Here, the key ingredient is to bound

Original: [Equation]
Revised: [Equation]

Original: s [Equation] a ) } independently of S .
Revised: independently of S .

Original: This is the second main difference with [10].
Revised: This is the second main difference with [11].

Original: Instead of exploring the MDP uniformly, we know that the algorithm will mostly visit the ﬁrst states of the MDP, regardless of the chosen policy.
Revised: Instead of exploring the MDP uniformly, we know that the algorithm will mostly visit the ﬁrst states of the MDP, regardless of the chosen policy.

Original: As shown in [10], for a ﬁxed state s :
Revised: As shown in [11], for a ﬁxed state s :

Original: [Equation]
Revised: [Equation]

Original: Now, instead of summing over the states, we can use properties of stochastic ordering to compare themean number of visits of a state with the probability measure m π 0 ; here, we strongly rely on the birthand death structure of the MDPs in M .
Revised: Now, instead of summing over the states, we can use properties of stochastic ordering to compare the mean number of visits of a state with the probability measure m π 0 ; here, we strongly rely on the birth and death structure of the MDPs in M .

Original: For any non-negative non-decreasing function f : S → R + ,we obtain 
Revised: For any non-negative non-decreasing function f : S → R + , we obtain E

Original: [Equation]
Revised: [Equation]

Original: Let us choose f : s (cid:55)→ max { 1 ,s ( s − 1) } ∆( s )+ r max  and let F := (cid:80) s f ( s ) − 1 ≤ 3( C + r max )  .
Revised: Let us choose f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max ) 2 and let F := (cid:80) s f ( s ) − 1 ≤ 3( C + r max ) 2 .

Original: Let also E 2 :=
Revised: Let also E 2 :=

Original: F E m π [-
 [(∆-] + r max )  f ] .
Revised: F E m π [-
 [(∆-]0 (cid:2) (∆ + r max ) 2 · f (cid:3) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: In Appendix A.3.4, we further show that E 2 ≤ 3( C + r max ) 2 (cid:16) 1 + λ 2 µ 2 (cid:17) .
Revised: In Appendix A.3.4, we further show that E 2 ≤ 3( C + r max ) 2 (cid:16) 1 + λ 2 µ 2 (cid:17) .

Original: Therefore, for the threemain terms, we obtain
Revised: Therefore, for the three main terms, we obtain

Original: [Equation]
Revised: [Equation]

Original: and we conclude our proof by combining all of these terms.
Revised: and we conclude our proof by combining all of these terms.

Original:  Conclusions
Revised:  Conclusions

Original: For learning in a class of birth and death processes, we have shown that exploiting the stationarymeasure in the analysis of classical learning algorithms yields a K √ T regret, where K only dependson the stationary measure of the system under a well chosen policy.
Revised: For learning in a class of birth and death processes, we have shown that exploiting the stationary measure in the analysis of classical learning algorithms yields a K √ T regret, where K only depends on the stationary measure of the system under a well chosen policy.

Original: Thus, the dependence on thesize of the state space as well as on the diameter of the MDP or its span disappears.
Revised: Thus, the dependence on the size of the state space as well as on the diameter of the MDP or its span disappears.

Original: We believe thatthis type of results can be generalized to other cases such as optimal routing, admission control andallocation problems in queuing systems, as the stationary distribution under all policies is unevenbetween the states.Checklist For all authors...
Revised: We believe that this type of results can be generalized to other cases such as optimal routing, admission control and allocation problems in queuing systems, as the stationary distribution under all policies is uneven between the states.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] The abstract and the introduction do provide a concise

Original: statement of the original result proved in this paper.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] The limitations of our work areexplicit as we described in 3 the subclass of MDPs on which the theorem applies.(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [No] This work

Original: is mainly of mathematical nature and we hope that it has a very little direct societal

Original: impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes] We conﬁrm that our paper complies with the ethical guidelines of Neurips.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] All assumptionsand the context of the theorem are provided in 2 and 3.(b)

Original: Did you include complete proofs of all theoretical results?

Original: [Yes] A sketch of the proofis given in 4.3 and the complete proof is given in the supplemental material.3.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A]

Original: There is no

Original: experiment in this paper.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]

Original: There is no experiment in this paper.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]

Original: There is no experiment in this paper.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]

Original: There is no experiment in this

Original: paper.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]

Original: We did not work

Original: with existing assets or new assets.

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: We did not work with existing assets

Original: or new assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: We did not work with existing assets or new assets.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: We did not work with existing assets or new assets.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: We did not work with existing assets or new

Original: assets.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: We did not work with human subjects.

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: We did not work with human subjects.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: We did not work with human subjects.

Original: Errata: During the submission of the supplementary material, we realized that there were typosin Theorem 4.1.

Original: There is a typo in Theorem 4.1 in the constants: the constant 11 should be 19,while the constant 96 should be 97.

Original: There is also a typo in the bound given in the theorem: alog term was written twice and must be removed.

Original: These are corrected in the new version.

Original: Also,there was a typo in the deﬁnition of Q max where a logarithmic term is missing.

Original: The correctvalue is Q max := (cid:16)

Original: [Equation]

Original: This has also been corrected in this version.

Original: The rest of the appendix is organized as follows: We ﬁrst provide some insights on extended valueiterations useful in our construction of the regret.

Original: Then, the detailed proof of theorem 4.1 is given with bounds on the ﬁve terms in our decompositionof the regret.

Original: A ﬁnal appendix provides technical lemmas about MDPs in M .

Original: A Proof of Theorem 4.1A.1 Extended value iteration
Revised: A Proof of Theorem 4. A.1 Extended value iteration

Original: For each episode k , we use the extended value iteration algorithm described in [10] to compute˜ π k and ˜ M ∈ M k , an optimistic policy and MDP.
Revised: For each episode k , we use the extended value iteration algorithm described in [11] to compute ˜ π k and ˜ M ∈ M k , an optimistic policy and MDP.

Original: The values we iteratively get are deﬁned in thefollowing way:
Revised: The values we iteratively get are deﬁned in the following way:

Original: [Equation]
Revised: [Equation]

Original: 16) where ˜ r is the maximal reward from (4) and P ( s, a ) is the set of probabilities from (5).
Revised: where ˜ r is the maximal reward from (4) and P ( s, a ) is the set of probabilities from (5).

Original: Now, from [10, Theorem 7], we obtain the following lemma on the iterations of extended valueiteration.
Revised: Now, from [11, Theorem 7], we obtain the following lemma on the iterations of extended value iteration.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For episode k , denote by i the last step of extended value iteration, stopped when:
Revised: For episode k , denote by i the last step of extended value iteration, stopped when:

Original: [Equation]
Revised: [Equation]

Original: The optimistic MDP ˜ M k and the optimistic policy ˜ π k that we choose are so that the gain is 1 √ t k −close to the optimal gain:
Revised: The optimistic MDP ˜ M k and the optimistic policy ˜ π k that we choose are so that the gain is 1 √ t k − close to the optimal gain:

Original: [Equation]
Revised: [Equation]

Original: Moreover from [15, Theorem 8.5.6]:
Revised: Moreover from [16, Theorem 8.5.6]:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 19) and when the optimal policy yields an irreducible and aperiodic Markov chain, we have that ˜ ρ k
Revised: and when the optimal policy yields an irreducible and aperiodic Markov chain, we have that ˜ ρ k

Original: =ρ ( ˜ M k , ˜
Revised: = ρ ( ˜ M k , ˜

Original: π k , s ) for any s , so that we can deﬁne the bias:
Revised: π k , s ) for any s , so that we can deﬁne the bias:

Original: [Equation]
Revised: [Equation]

Original: [Equation] By choosing iteration i large enough, from [15, Equation 8.2.5], we can also ensure that:
Revised: By choosing iteration i large enough, from [16, Equation 8.2.5], we can also ensure that:

Original: [Equation]
Revised: [Equation]

Original: so that we can deﬁne the following difference
Revised: so that we can deﬁne the following difference

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A.2 Regret when M is out of the conﬁdence bound
Revised: A.2 Regret when M is out of the conﬁdence bound

Original: Let us compute E
Revised: Let us compute E

Original: [Reg] , the expected regret.
Revised: [Reg] , the expected regret.

Original: We will mainly follow the approach in [10, Section 4],with a few tweaks.
Revised: We will mainly follow the approach in [11, Section 4], with a few tweaks.

Original: We start by splitting the regret into a sum over episodes and states.
Revised: We start by splitting the regret into a sum over episodes and states.

Original: We remind that r ( s, a ) is the overall mean reward and N T ( s, a ) the total count of visits.
Revised: We remind that r ( s, a ) is the overall mean reward and N T ( s, a ) the total count of visits.

Original: We alsodeﬁne R k ( s ) :
Revised: We also deﬁne R k ( s ) :

Original: = (cid:80) a ν k
Revised: = (cid:80) a ν k

Original: ( s, a )(
Revised: ( s, a )(

Original: ρ ∗ − r ( s, a ))
Revised: ρ ∗ − r ( s, a ))

Original: the regret at episodek induced by state s , with ν k ( s, a )the number of visit of ( s, a ) during episode k . Let E T denote the event [Equation] Then, using a Hoeffding inequality for a ﬁxed count N T ( s, a ) we have that
Revised: the regret at episode

Original: [Equation]

Original: (23) and then by the union bound:

Original: [Equation]

Original: (24) so that by splitting into events where the previous condition is either true or false:

Original: [Equation]

Original: and then taking the expectation:

Original: [Equation]

Original: ≤ E (cid:34)

Original: [Equation]

Original: as T = (cid:80) s,a N T ( s, a ) .

Original: Let R in :=
Revised: Let R in :=

Original: (cid:80) s (cid:80) mk
Revised: (cid:80) s (cid:80) mk

Original: =1 R k ( s ) 1 M ∈M k and R out := (cid:80) s (cid:80) mk
Revised: =1 R k ( s ) 1 M ∈M k and R out := (cid:80) s (cid:80) mk

Original: =1 R k ( s ) 1 M / ∈M k .
Revised: =1 R k ( s ) 1 M / ∈M k .

Original: We therefore have thesplit, for T ≥ 2 , as log T ≥ 1 4 :
Revised: We therefore have the split, for T ≥ 2 , as log T ≥ 14 :

Original: E [ Reg ] ≤ E [ R in ] + E [ R out ] + r max (cid:112) T log ( T ) .
Revised: [Equation]

Original: Now, let ν k ( s )
Revised: Now, let ν k ( s )

Original: =
Revised: =

Original: (cid:80) a ν k
Revised: (cid:80) a ν k

Original: ( s, a ) and denote by M ( t )
Revised: ( s, a ) and denote by M ( t )

Original: the set of MDPs M k such that t k ≤ t < t k +1 .
Revised: the set of MDPs M k such that t k ≤ t < t k +1 .

Original: For the terms out of the conﬁdence sets, we have:
Revised: For the terms out of the conﬁdence sets, we have:

Original: [Equation]

Original: N

Original: [Equation]
Revised: [Equation]

Original: using the stopping criterion
Revised: the stopping criterion

Original: [Equation]
Revised: [Equation]

Original: Taking the expectations:
Revised: Taking the expectations:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Thus, we have dealt with the cases where the MDP M did not belong to any conﬁdence set, for someepisodes.
Revised: Thus, we have dealt with the cases where the MDP M did not belong to any conﬁdence set, for some episodes.

Original: We now need to deal with the rest.
Revised: We now need to deal with the rest.

Original: A.3 Regret terms when M is in the conﬁdence bound
Revised: A.3 Regret terms when M is in the conﬁdence bound

Original: We now assume that M ∈ M k and deal with the terms in the conﬁdence bound, so that we can omitthe repetitions of the indicator functions.
Revised: We now assume that M ∈ M k and deal with the terms in the conﬁdence bound, so that we can omit the repetitions of the indicator functions.

Original: For each episode k , let R in ,k := (cid:80) s R k .
Revised: For each episode k , let R in ,k := (cid:80) s R k .

Original: We deﬁned ˜
Revised: We deﬁned ˜

Original: π k the optimistic policy computed at episode k , now deﬁne ˜
Revised: π k the optimistic policy computed at episode k , now deﬁne ˜

Original: P
Revised: P

Original: k :=
Revised: k :=

Original: (˜ p k ( s (cid:48) | s, ˜ π k ( s ))) thetransition matrix of that policy on the optimistic MDP ˜ M k .
Revised: (˜ p k ( s (cid:48) | s, ˜ π k ( s ))) the transition matrix of that policy on the optimistic MDP ˜ M k .

Original: Deﬁne also v k :=
Revised: Deﬁne also v k :=

Original: ( ν k ( s, ˜ π k ) the rowvector of visit counts during episode k .
Revised: ( ν k ( s, ˜ π k ) the row vector of visit counts during episode k .

Original: Following the same steps as in [10], we get the inequality onthe regret of episode k , assuming M ∈ M k , using Lemma A.1:
Revised: Following the same steps as in [11], we get the inequality on the regret of episode k , assuming M ∈ M k , using Lemma A.1:

Original: [Equation]
Revised: [Equation]

Original: Then with (19) and using the deﬁnition of the iterated values from EVI, we have for a given state s and a s :=
Revised: Then with (19) and using the deﬁnition of the iterated values from EVI [Equation] we have for a given state s and a s :=

Original: ˜ π k ( s ) :
Revised: ˜ π k ( s ) :

Original: [Equation]
Revised: [Equation]

Original: so that:
Revised: so that:

Original: [Equation]
Revised: [Equation]

Original: Remember that for any state s : | d k ( s ) | ≤ r max √ t k , where ˜h k is the bias of the average optimal policy for the optimist MDP, and:
Revised: Remember that for any state s : | d k ( s ) | ≤ r max √ t k , where ˜h k is the bias of the average optimal policy for the optimist MDP, and:

Original: [Equation]
Revised: [Equation]

Original: Notice that the unit vector is in the kernel of (cid:16)
Revised: Notice that the unit vector is in the kernel of (cid:16)

Original: ˜
Revised: ˜

Original: P k − I (cid:17) .
Revised: P k − I (cid:17) .

Original: Therefore, in the ﬁrst term, we can replace u i by any translation of it.
Revised: Therefore, in the ﬁrst term, we can replace u i by any translation of it.

Original: We get:
Revised: We get:

Original: [Equation]
Revised: [Equation]

Original: so that:
Revised: so that:

Original: [Equation]
Revised: [Equation]

Original: Then, using the assumption on empirical rewards (4), as M ∈ M k, and noticing that N t k ≤ t k :
Revised: Then, using the assumption on empirical rewards (4) [Equation] as M ∈ M k

Original: [Equation]
Revised: [Equation]

Revised: and noticing that N t k ≤ t k :

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: For the term v k (cid:16) ˜
Revised: For the term v k (cid:16) ˜

Original: P k − I (cid:17) d k , which does not appear in the analysis of [10], we obtain
Revised: P k − I [Equation] d k , which does not appear in the analysis of [11], we obtain

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we used that max { 1 , N t k ( s, a ) } ≤ t k ≤ T .
Revised: where in the last inequality we used that max { 1 , N t k ( s, a ) } ≤ t k ≤ T .

Original: Thus, for T ≥ e 2 [-
-] AT theregret term coming from the consequences and approximations of EVI satisﬁes
Revised: Thus, for T ≥ e 2 [-
-]2 AT the regret term coming from the consequences and approximations of EVI satisﬁes

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, by deﬁning P k the transition matrix of the optimistic policy ˜ π k in the true MDP M , we havethe following decomposition of the middle term:
Revised: Now, by deﬁning P k the transition matrix of the optimistic policy ˜ π k in the true MDP M , we have the following decomposition of the middle term:

Original: [Equation]
Revised: [Equation]

Original: Overall: R in ≤ (cid:88)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: A.3.1 Bound on R trans
Revised: A.3.1 Bound on R trans

Original: Let us deal with the ﬁrst term R trans .
Revised: Let us deal with the ﬁrst term R trans .

Original: To bound this term, we will use our knowledge of the optimalbias h ∗ and the control of the difference of the transition matrices, and for the second term we willcontrol the difference of the biases.
Revised: To bound this term, we will use our knowledge of the optimal bias h ∗ and the control of the difference of the transition matrices, and for the second term we will control the difference of the biases.

Original: Notice that for a ﬁxed state 1 ≤ s ≤ N − 1 :(cid:88)
Revised: Notice that for a ﬁxed state 1 ≤ s ≤ N − 1 :

Original: [Equation]
Revised: [Equation]

Original: The same is true for ˜ p k , and knowing the MDP is a birth and death process:
Revised: The same is true for ˜ p k , and knowing the MDP is a birth and death process:

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality, we used the knowledge on the bounded variations of the optimal biasfrom Lemma 3.2, and that the optimistic MDP has transitions close to the true transitions.
Revised: where in the last inequality, we used the knowledge on the bounded variations of the optimal bias from Lemma 3.2, and that the optimistic MDP has transitions close to the true transitions.

Original: A.3.2 Bound on R diﬀ
Revised: A.3.2 Bound on R diﬀ

Original: We now deal with the term involving the difference of bias, R diﬀ .
Revised: We now deal with the term involving the difference of bias, R diﬀ .

Original: For each episode k with policy π k , denote by x k the state such that the conﬁdence bounds are at their worst and denote by a k :=
Revised: For each episode k with policy π k , denote by x k the state such that the conﬁdence bounds are at their worst and denote by a k :=

Original: π k ( x k ) the corresponding action used at this state, so that N t k ( x k , a k ) is minimal.
Revised: π k ( x k ) the corresponding action used at this state, so that N t k ( x k , a k ) is minimal.

Original: We therefore have that (cid:113) [-log(
-]
Revised: We therefore have that (cid:113) [-log(
-]log

Original:  At k ) max { 1 ,N tk ( x k ,a k ) } is maximal for episode k .
Revised: [Equation] 2 At k [Equation] max { 1 ,N tk ( x k ,a k ) } is maximal for episode k .

Original: The true MDP being within the conﬁdence bounds, with a triangle inequality:
Revised: The true MDP being within the conﬁdence bounds, with a triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then using Lemma C.4, and noticing that to bound the biases ˜h k , h ∗ and the quantity (cid:107) (cid:80) 
Revised: Then using Lemma C.4, and noticing that to bound the biases ˜h k , h ∗ and the quantity (cid:107) (cid:80) T t [Equation]

Original: Tt =1 ˜ P 
Revised: 1 ˜ P t k ˜

Original: tk ˜ r k (cid:107)is bounded by the same diameter D , using the same argument as in [10] (Equation (11)), and noticingthat D ≥ 1 : 
Revised: [Equation] k (cid:107) is bounded by the same diameter D , using the same argument as in [11] (Equation (11)), and noticing that D ≥ 1 : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we have used (29) and that by deﬁnition of D
Revised: where in the last inequality we have used (27) and that by deﬁnition of D

Original: T hit := inf s (cid:48)

Original: [Equation]
Revised: [Equation]

Original: and we called
Revised: and we called

Revised: [Equation]

Revised: By the choice of x k

Original: [Equation]
Revised: [Equation]

Original: By the choice of x k , N t k ( x k , a k ) ≤ N t k ( s, a ) for any state-action pair ( s, a ) , so that we can rewrite,with I k :
Revised: N t k ( x k , a k ) ≤ N t k ( s, a ) for any state-action pair ( s, a ) [Equation] so that we can rewrite, with I k :

Original: =
Revised: =

Original: t k +1 − t k the length of episode k :
Revised: t k +1 − t k the length of episode k :

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne Q max [Equation] =
Revised: Now deﬁne Q max := (cid:16)

Original: [Equation]
Revised: [Equation]

Original: and I ( T ) :
Revised: and I ( T ) :

Original: =
Revised: =

Original: max (cid:8) Q max , T 1 / 4 (cid:9) .
Revised: max (cid:8) Q max , T 1 / 4 (cid:9) .

Original: We split the sum depending on whether the episodes are shorter than I ( T ) or not, and call K ≤ I the number of such episodes.
Revised: We split the sum depending on whether the episodes are shorter than I ( T ) or not, and call K ≤ I the number of such episodes.

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: Using the stopping criterion for episodes:
Revised: Using the stopping criterion for episodes:

Original: [Equation]
Revised: [Equation]

Original: Now denote by E the event:
Revised: Now denote by E the event:

Original: [Equation]
Revised: [Equation]

Original: By splitting the sum, using the above event, we get:
Revised: By splitting the sum, using the above event, we get:

Original: [Equation]
Revised: [Equation]

Original: We use Corollary C.6 to get P (cid:0) ¯ E (cid:1) ≤ 14 T , so that when taking the expectation:
Revised: We use Corollary C.6 to get P (cid:0) ¯ E (cid:1) ≤ 14 T , so that when taking the expectation:

Original: [Equation]
Revised: [Equation]

Original: Now using Lemma B.3, SA ≥ 4 , I ( T ) ≥ 2 m max ( S − 1) and that 1 log2 + 14 ≤ 2 :
Revised: Now using Lemma B.3, SA ≥ 4 , I ( T ) ≥ 2 m max ( S − 1) and that 1 log2 + [Equation]

Original: [Equation]
Revised: [Equation]

Original: We therefore have that:
Revised: We therefore have that:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A.3.3 Bound on the main terms: Exploiting the stochastic ordering
Revised: A.3.3 Bound on the main terms: Exploiting the stochastic ordering

Original: In Section 4.3 we have shown that:
Revised: In Section 4.3 we have shown that:

Original: [Equation]
Revised: [Equation]

Original: To control this term as well as R EVI (28) and R rewards (27), we need to control the terms in the sumin a way that does not make the parameters D or S appear, as this will be one of the main contributingterms.
Revised: To control this term as well as R EVI (26) and R rewards (25), we need to control the terms in the sum in a way that does not make the parameters D or S appear, as this will be one of the main contributing terms.

Original: To do so, we need to sum over the episodes and take the expectation, so that with Lemma B.4,we get:[Equation]
Revised: To do so, we need to sum over the episodes and take the expectation, so that with Lemma B.4, we get:

Original: Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: We will use the following lemma to carry on the computations:
Revised: We will use the following lemma to carry on the computations:

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let m π 0 be the stationary measure of the Markov chain under policy π 0 , such that forevery state s : π 0 ( s ) = 0 .
Revised: Let m π 0 be the stationary measure of the Markov chain under policy π 0 , such that for every state s : π 0 ( s ) = 0 .

Original: Let f : S
Revised: Let f : S

Original: → R + be a non-negative non-decreasing function on the statespace.
Revised: → R + be a non-negative non-decreasing function on the state space.

Original: Then for any state s ∈ S ,
Revised: Then for any state s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let s ∈ S .
Revised: Let s ∈ S .

Original: For any state s (cid:48) , deﬁne N m π 0 ,π 0 t ( s(cid:48) )
Revised: For any state s (cid:48) , deﬁne N m π 0 ,π 0 t [Equation] s

Original:  the number of visits when the starting state follows the initial distribution m π 0 , and the MDP always executes the policy π 0 at every timestep instead of the policy determined by the algorithm U CRL 2.
Revised: [Equation] ) the number of visits when the starting state follows the initial distribution m π 0 , and the MDP always executes the policy π 0 at every timestep instead of the policy determined by the algorithm U CRL 2.

Original: Notice already that for any state s (cid:48) :
Revised: Notice already that for any state s (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for x ∈ S , we have the stochastic ordering:(cid:88)
Revised: On the other hand, for x ∈ S , we have the stochastic ordering:

Original: [Equation]
Revised: [Equation]

Original: so that for any non-negative non-decreasing function f , with the convention f ( − 1) = 0 :
Revised: so that for any non-negative non-decreasing function f [Equation] with the convention f ( − 1) = 0 :

Original: [Equation]
Revised: [Equation]

Original: and then summing the equation above for s ≤ x ≤ S − 1 and switching the sums yields:
Revised: and then summing the equation above for s ≤ x ≤ S − 1 and switching the sums yields:

Original: [Equation]
Revised: [Equation]

Original: which simpliﬁes to:
Revised: which simpliﬁes to:

Original: [Equation]
Revised: [Equation]

Original: Now summing with the second equation in (33)
Revised: Now summing with the second equation in (31)

Original: [Equation]
Revised: [Equation]

Original: we get the following equation:
Revised: we get the following equation:

Original: [Equation]
Revised: [Equation]

Original: Taking the expectation in this last inequality ﬁnishes the proof
Revised: Taking the expectation in this last inequality ﬁnishes the proof.

Original: [Equation] Now, we can conclude our bound on R trans .
Revised: Now, we can conclude our bound on R trans .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: let f be a non-negative non-decreasing function over the state space, such that F := (cid:80) s ≥ 

Revised: let f be a non-negative non-decreasing function over the state space, such that F := (cid:80) s ≥

Original: f ( s ) − 1exists.
Revised: f ( s ) −exists.

Original: Then by concavity:
Revised: Then by concavity:

Original: [Equation]
Revised: [Equation]

Original: ) using Lemma A.2 , so that overall, (34) becomes:
Revised: Lemma A. [Equation] so that overall, (32) becomes:

Original: [Equation]
Revised: [Equation]

Original: This is the term mainly contributing to the regret.
Revised: This is the term mainly contributing to the regret.

Original: A.3.4 Bound on the main terms: Introducing E
Revised: A.3.4 Bound on the main terms: Introducing E

Original: Now, using Lemma B.5 which gives the stationary distribution of m 0 , we can compute the expectationunder m 0 of a well-chosen function f :
Revised: Now, using Lemma B.5 which gives the stationary distribution of m 0 , we can compute the expectation under m 0 of a well-chosen function f :

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Let us choose the increasing function f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max )  .
Revised: Let us choose the increasing function f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max ) 2 .

Original: Then F ≤ 3( C + r max ) and (cid:80) s ≥ 0 (∆( s ) + r max )  f ( s ) m π 0 ( s )=
Revised: Then F ≤ 3( C + r max )and (cid:80) s ≥ 0 (∆( s ) + r max ) 2 f ( s ) m π 0 ( s )

Original: [-E m π 
-]
Revised: [-E m π 
-]=

Original: [(∆ + r max )  · f ] ≤( C + r max ) (cid:16) 1 + λ 2 µ2 (cid:17) , so that:
Revised: E m π 0 (cid:2) (∆ + r max ) [Equation] · f (cid:3) ≤ (cid:16) 1 + λ 2 µ

Original: [Equation]
Revised: [Equation]

Revised: , so that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For F , we obtain:
Revised: For F , we obtain:

Original: [Equation]
Revised: [Equation]

Original: Using the following computations:
Revised: Using the following computations:

Original: [Equation]
Revised: [Equation]

Original: S −s (cid:19) (cid:18) λ ( S − 1) µ (cid:19) s =

Original: ( S − 2)( S − 1)

Original: S (cid:88) s

Original: = (cid:18) S −s −(cid:19) (cid:18)

Original: λ
Revised: λ

Original:  ( S− 1) µ (cid:19) s =
Revised: [Equation] [Equation] = ( S

Original:  ( S − 2)( S − 1) (cid:18)
Revised: [Equation] ( S [Equation]

Original:  λ ( S − 1) µ (cid:19)2 S − 3 (cid:88) s
Revised: [Equation] (cid:18) λ ( S − 1) µ (cid:19)

Original: =
Revised: [Equation]

Original: (cid:18) S −s (cid:19) (cid:18) λ ( S − 1) µ (cid:19) s =
Revised: (cid:18) 1 + λ ( S − 1) µ (cid:19) S −≤ (cid:18) λ µ (cid:19)

Original: ( S − 2)( S − 1) (cid:18) λ ( S − 1) µ (cid:19) 2 (cid:18) 1 + λ ( S − 1) µ (cid:19) S
Revised: [Equation]

Original: −≤ (cid:18) λ µ (cid:19) 2 (cid:18) 1 + λ ( S − 1) µ (cid:19) S −, and using that 1 + λµ ≤ (cid:16) 1 + λ ( S − 1) µ (cid:17) S − 1 , we get: (cid:16) 1 + λ 
Revised: (cid:18) 1 + λ ( S − 1) µ (cid:19) S −, and using that 1 + λµ ≤ (cid:16) 1 + λ ( S − 1) µ (cid:17) S − 1 , we get: [Equation] λ [Equation]

Original: ( S − 1) µ (cid:17) S −( C + r max ) E m π 0 [(∆ + r max ) · f ] ≤ (cid:18) 1 + λµ(cid:19) (cid:18) 1 + λ ( S − 1) µ (cid:19) S −, so that ﬁnally
Revised: so that ﬁnally

Original: E m π 0 [(∆ + r max ) · f ] ≤ ( C + r max ) (cid:18) 1 + λµ(cid:19) , which concludes the proof.
Revised: [Equation] which concludes the proof.

Original: Finally (35) becomes:
Revised: Finally (33) becomes:

Original: [Equation]
Revised: [Equation]

Original: and thus:
Revised: and thus:

Original: [Equation]
Revised: [Equation]

Original: In particular:
Revised: In particular:

Original: [Equation]
Revised: [Equation]

Original: A.3.5 Bound on R ep
Revised: A.3.5 Bound on R ep

Original: It remains to deal with the following regret term:
Revised: It remains to deal with the following regret term:

Original: [Equation]
Revised: [Equation]

Original: We will follow the core of the proof from [10].
Revised: We will follow the core of the proof from [11]

Revised: [Equation]

Original: Deﬁne X t := ( p ( ·| s t , a t ) − e s t )
Revised: Deﬁne X t := ( p ( ·| s t , a t ) − e s t )

Original: ˜h k ( t ) 1 M ∈M k ( t ) ,where k ( t ) is the episode containing step t and e i the vector with i -th coordinate 1 and 0 for the othercoordinates.
Revised: ˜h k ( t ) 1 M ∈M k ( t ) , where k ( t ) is the episode containing step t and e i the vector with i -th coordinate 1 and 0 for the other coordinates.

Original: [Equation]
Revised: [Equation]

Original: By summing over the episodes we get:
Revised: By summing over the episodes we get:

Original: [Equation]
Revised: [Equation]

Original: Notice that E [ X t | s 1 , a 1 , . . .
Revised: Notice that E [ X t | s 1 , a 1 , . . .

Original: , s t , a t ] = 0 , so that when taking the expectations, only the term in thenumber of episodes remains.
Revised: , s t , a t ] = 0 , so that when taking the expectations, only the term in the number of episodes remains.

Original: On the other side, using Lemma B.3, we get when taking the expectation:
Revised: On the other side, using Lemma B.3, we get when taking the expectation:

Original: [Equation]
Revised: [Equation]

Original: Assuming SA ≥ 4 , and using log(2) ≥ 12 :
Revised: Assuming SA ≥ 4 , and using log(2) ≥ 12 :

Original: [Equation]
Revised: [Equation]

Original: We can now gather the expected regret terms when the true MDP is within the conﬁdence bounds.
Revised: We can now gather the expected regret terms when the true MDP is within the conﬁdence bounds.

Original: Using (30), (37) and (39):
Revised: Using (28), (35) and (37):

Original: [-E [ R in ] ≤ 96 r max SAD 
 I ( T ) log 2 (2 AT ) + 12 (cid:112) 2 E 2 AT log (2 AT )-]
Revised: [-E [ R in ] ≤ 96 r max SAD 
 I ( T ) log 2 (2 AT ) + 12 (cid:112) 2 E 2 AT log (2 AT )-][Equation]

Original: + 2 r max SAD log(2 AT ) , which gives with (25) and (26), assuming that T ≥ S 2 :
Revised: which gives with (23) and (24), assuming that T ≥ S 2 :

Original: [-E [ Reg ] ≤ 97 r max SAD 
 I ( T ) log 2 (2 AT ) + 2 r max (cid:112) T log T + 12 (cid:112) 2 E 2 AT log (2 AT ) .-]
Revised: [-E [ Reg ] ≤ 97 r max SAD 
 I ( T ) log 2 (2 AT ) + 2 r max (cid:112) T log T + 12 (cid:112) 2 E 2 AT log (2 AT ) .-][Equation]

Original: which ﬁnally gives, assuming that SA ≥ 4 , and since r max ≤ √ E 2 ,
Revised: which ﬁnally gives:

Original: [-E [ Reg ] ≤ 97 r max SAD 
 I ( T ) log 2 (2 AT ) + 19 (cid:112) E 2 AT log (2 AT ) .-]
Revised: [-E [ Reg ] ≤ 97 r max SAD 
 I ( T ) log 2 (2 AT ) + 19 (cid:112) E 2 AT log (2 AT ) .-][Equation]

Original: B Technical Lemmas 
Revised: B Technical Lemmas B.1 Probability of the conﬁdence bounds

Original: B.1 Probability of the conﬁdence bounds

Original: This ﬁrst lemma is from [10, Lemma 17] and adapted to our conﬁdence bounds.
Revised: This ﬁrst lemma is from [11, Lemma 17] and adapted to our conﬁdence bounds.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: For t > 1 , the probability that the MDP M is not within the sate of plausible MDPs
Revised: For t > 1 , the probability that the MDP M is not within the sate of plausible MDPs

Original: M t is bounded by:
Revised: M t is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix a state action pair ( s, a ) , and n the number of visits of this pair before time t .
Revised: Fix a state action pair ( s, a ) , and n the number of visits of this pair before time t .

Original: Recall thatˆ p and ˆ r are the empirical transition probabilities and rewards from the n observations.
Revised: Recall that ˆ p and ˆ r are the empirical transition probabilities and rewards from the n observations.

Original: Knowing thatfrom each pair, there are at most 3 transitions, a Weissman’s inequality gives for any ε p > 0 :
Revised: Knowing that from each pair, there are at most 3 transitions, a Weissman’s inequality gives for any ε p > 0 :

Original: [Equation]
Revised: [Equation]

Original: So that for the choice of ε p = (cid:113) 2 n log (16 At 4 ) ≤ (cid:113) 8 n log (2 At ) , we get: 
Revised: So that for the choice of ε p = (cid:113) 2 n log (16 At 4 ) ≤ (cid:113) 8 n log (2 At ) , we get: P

Original: [Equation]
Revised: [Equation]

Original: We can do similar computations for the conﬁdence on rewards, with a Hoeffding inequality:
Revised: We can do similar computations for the conﬁdence on rewards, with a Hoeffding inequality:

Original: [Equation]
Revised: [Equation]

Original:  and choosing ε r = r max (cid:113) 
Revised: (cid:19) , and choosing ε r = r max (cid:113) 12 n log (16 At 4 ) ≤ r max (cid:113) 2 n log (2 At ) , so that: P

Original: [Equation]
Revised: [Equation]

Original: log [Equation] [Equation] 4 ) ≤ r
Revised: (cid:41) ≤ 8 At 4 .

Original: [Equation]
Revised: Now with a union bound for all values of n ∈ { 0 , 1 , · · · , t − 1 } , we get: P

Original: [Equation]
Revised: [Equation]

Original: ) , so that:
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: Now with a union bound for all values of n ∈ { 0 , 1 , · · · , t − 1 } , we get:
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: a ) }
Revised: 1 ≥ (cid:115) 8 log (2 At ) [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  8 At 3 , and
Revised: } (cid:41) ≤ 8 At 3 , and

Original: P (cid:40)
Revised: P (cid:40)

Revised: [Equation]

Revised: [Equation]

Revised: max

Original: [Equation]
Revised: [Equation]

Revised: At ) max {

Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: [Equation]

Revised: } (cid:41)

Original:  and ﬁnally, when summing over all state-action pairs, P { M / ∈ M ( t ) } < S 2 t 3 .
Revised: [Equation] 8 At [Equation] , and ﬁnally, when summing over all state-action pairs, P { M / ∈ M ( t ) } < S 2 t 3 .

Original: B.2 Number of visits for an MDP in M
Revised: B.2 Number of visits for an MDP in M

Original: This lemma is needed in the proof of Lemma C.5.
Revised: This lemma is needed in the proof of Lemma C.5.

Original: Lemma B.2 (Azuma-Hoeffding inequality) .
Revised: Lemma B.2 (Azuma-Hoeffding inequality) .

Original: Let X 1 , X 2 , . .
Revised: Let X 1 , X 2 , . .

Original: . be a martingale difference sequence with | X i | ≤ RD for all i and some R > 0 .
Revised: . be a martingale difference sequence with | X i | ≤ RD for all i and some R > 0 .

Original: Then for all ε > 0 and n ∈ N :
Revised: Then for all ε > 0 and n ∈ N :

Original: [Equation]
Revised: [Equation]

Original: The two following lemmas are proved in [10, Appendix C.2 and Appendix C.3] respectively.
Revised: The two following lemmas are proved in [11, Appendix C.2 and Appendix C.3] respectively.

Original: Bounding the number of episodes is notably useful to obtain equation (30).
Revised: Bounding the number of episodes is notably useful to obtain equation (28).

Original: Lemma B.3.
Revised: Lemma B.3.

Original: Denote by K t the number of episodes up to time t , and let t > SA .
Revised: Denote by K t the number of episodes up to time t , and let t > SA .

Original: It is bounded by:
Revised: It is bounded by:

Original: [Equation]
Revised: [Equation]

Original: The following lemma is used to simplify regret terms, notably (31).
Revised: The following lemma is used to simplify regret terms, notably (29).

Original: Lemma B.4.
Revised: Lemma B.4.

Original: For any ﬁxed state action pair ( s, a ) and time T , we have:
Revised: For any ﬁxed state action pair ( s, a ) and time T , we have:

Original: [Equation]
Revised: [Equation]

Original: B.3 Diameter and Span of MDPs in M
Revised: B.3 Diameter and Span of MDPs in M

Original: For completeness, and to support the discussion in Section 4.2, the section details the behavior of thediameter and the span of MDPs in M , as functions of S .
Revised: For completeness, and to support the discussion in Section 4.2, the section details the behavior of the diameter and the span of MDPs in M , as functions of S .

Original: Under policy π 0 , it is possible to get en explicit expression for the stationary distribution of the states.
Revised: Under policy π 0 , it is possible to get en explicit expression for the stationary distribution of the states.

Original: Lemma B.5.
Revised: Lemma B.5.

Original: Under the stationary policy π 0 , the stationary measure m π 0 ( s ) of the MDP is given by:
Revised: Under the stationary policy π 0 , the stationary measure m π 0 ( s ) of the MDP is given by:

Original: [Equation]
Revised: [Equation]

Original: This lemma is shown in the proof of [1, Lemma 3.3].
Revised: This lemma is shown in the proof of [1, Lemma 3.3].

Original: First, it should be clear that under any policy π , the diameter of the MDP under π is extremely largebecause the probability to move from state s to state s + 1 is smaller and smaller as s grows.
Revised: First, it should be clear that under any policy π , the diameter of the MDP under π is extremely large because the probability to move from state s to state s + 1 is smaller and smaller as s grows.

Original: Actually,this is also true for the local diameter, more precisely the expected time to go from s to s + 1 growsextremely fast with s .
Revised: Actually, this is also true for the local diameter, more precisely the expected time to go from s to s + 1 grows extremely fast with s .

Original: This discussion is formalized in the following result.
Revised: This discussion is formalized in the following result.

Original: Lemma B.6.
Revised: Lemma B.6.

Original: For any M ∈ M and any policy π , the diameter D π as well as the local diameter
Revised: For any M ∈ M and any policy π , the diameter D π as well as the local diameter

Original: D π ( s − 1 , s ) grow as S S − 2 .
Revised: D π ( s − 1 , s ) grow as S S − 2 .

Original: Proof.
Revised: Proof.

Original: Under policy π , the following sequence of inequalities follows from the stochastic comparisonwith π 0 and monotonicity under π 0 .
Revised: Under policy π , the following sequence of inequalities follows from the stochastic comparison with π 0 and monotonicity under π 0 .

Original: [Equation]
Revised: [Equation]

Original: where τ π ( x, y ) is the expected time to go from x to y under policy π .
Revised: where τ π ( x, y ) is the expected time to go from x to y under policy π .

Original: Now, starting from S − 2 , the Markov chain moves to S − 1 with probability p :=
Revised: Now, starting from S − 2 , the Markov chain moves to S − 1 with probability p :=

Original: λ/ ( U ( S − 1)) andthe time to reach S − 1 is equal to 1 or moves to S − 2 or S − 3 with probability 1 − p .
Revised: λ/ ( U ( S − 1)) and the time to reach S − 1 is equal to 1 or moves to S − 2 or S − 3 with probability 1 − p .

Original: Therefore,τ π 0 ( S − 2 , S − 1) is bounded by 1 − p times the return time to S − 1 , bounded in turn by the inverseof the stationary measure of state S − 2 in the chain truncated at S − 2 .
Revised: Therefore, τ π 0 ( S − 2 , S − 1) is bounded by 1 − p times the return time to S − 1 , bounded in turn by the inverse of the stationary measure of state S − 2 in the chain truncated at S − 2 .

Original: Using Lemma B.5,
Revised: Using Lemma B.5,

Original: [Equation]
Revised: [Equation]

Original: As for the maximal local diameter,max s D π ( s [Equation] [Equation] s
Revised: As for the maximal local diameter,

Original: [Equation]
Revised: [Equation]

Original: ≥ max s

Original: [Equation]

Original: s

Original: [Equation]

Original: [Equation]

Original: s

Original: [Equation] S [Equation] − 1)and the same argument as before applies.
Revised: and the same argument as before applies.

Original: Let us now consider the bias of the optimal policy in M .
Revised: Let us now consider the bias of the optimal policy in M .

Original: Using Lemma 3.2, the bias h ∗ ( s ) isdecreasing and concave in s , with bounded increments.
Revised: Using Lemma 3.2, the bias h ∗ ( s ) is decreasing and concave in s , with bounded increments.

Original: Therefore, its span, deﬁned as span ( h ∗ ) :
Revised: Therefore, its span, deﬁned as span ( h ∗ ) :

Original: =max s h ∗ ( s ) − min s h ∗ ( s ) , satisﬁes
Revised: = [Equation] , satisﬁes

Original: [Equation]
Revised: [Equation]

Original: This implies that the span of the bias behaves as a linear function of S for all M .
Revised: This implies that the span of the bias behaves as a linear function of S for all M .

Original: C Generic lemmas on ergodic MDPs 
Revised: C Generic lemmas on ergodic MDPs C.1 From bias variations to probability transition variations

Original: C.1 From bias variations to probability transition variations

Original: The three ﬁrst lemmas of this subsection are used in the proof of Lemma C.4.
Revised: The three ﬁrst lemmas of this subsection are used in the proof of Lemma C.4.

Original: This lemma is neededto obtain equation (29).
Revised: This lemma is needed to obtain equation (27).

Original: Lemma C.1.
Revised: Lemma C.1.

Original: For a MDP with rewards r ∈ [0 , r max ] and transition matrix P , denote by J s ( π, T ) :
Revised: For a MDP with rewards r ∈ [0 , r max ] and transition matrix P , denote by J s ( π, T ) :

Original: =
Revised: =

Original: E (cid:104)(cid:80) Tt =

Revised: E (cid:104)(cid:80) Tt =

Original: r
Revised: r

Original: ( s t , π ( s t ))
Revised: ( s t , π ( s t ))

Original: (cid:105) the expected cumulative rewards until time T starting from state s , underpolicy π .
Revised: (cid:105) the expected cumulative rewards until time T starting from state s , under policy π .

Original: Let D π be the diameter under policy π .
Revised: Let D π be the diameter under policy π .

Original: The following inequality holds: span ( J ( π, T )) ≤r max D π .
Revised: The following inequality holds: span ( J ( π, T )) ≤ r max D π .

Original: Proof.
Revised: Proof.

Original: Let s, s (cid:48) ∈ S .
Revised: Let s, s (cid:48) ∈ S .

Original: Call τ s → s (cid:48)
Revised: Call τ s → s (cid:48)

Original: the random time needed to reach state s (cid:48) from state s under policyπ .
Revised: the random time needed to reach state s (cid:48) from state s under policy π .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: which proves the lemma.
Revised: which proves the lemma.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Consider two ergodic MDPs M and M (cid:48) .
Revised: Consider two ergodic MDPs M and M (cid:48) .

Original: For i ∈ 1 , 2 , let r i ∈
Revised: For i ∈ 1 , 2 , let r i ∈

Original: [0 , r max ] and P
Revised: [0 , r max ] and P

Original: i be the rewards and transition matrix of MDP M i under policy π i , where both MDPs have the same state and action spaces.
Revised: i be the rewards and transition matrix of MDP M i under policy π i , where both MDPs have the same state and action spaces.

Original: Denote by g i the average reward obtained under policy π i in the MDP M i .
Revised: Denote by g i the average reward obtained under policy π i in the MDP M i .

Original: Then the difference of the gains is upper bounded.
Revised: Then the difference of the gains is upper bounded.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne for any state s the following correction term b ( s ) :
Revised: Deﬁne for any state s the following correction term b ( s ) :

Original: = r max D π (cid:107) p ( ·| s ) − p (cid:48) ( ·| s ) (cid:107) .
Revised: = r max D π (cid:107) p ( ·| s ) − p (cid:48) ( ·| s ) (cid:107) .

Original: Let us show by induction that for T ≥ 0 ,
Revised: Let us show by induction that for T ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Revised: t =

Revised: P t r ≤

Revised: T − 1 (cid:88) t =

Revised: P (cid:48) t ( r + b ) .

Original: This is true for T = 0 .
Revised: This is true for T = 0 .

Original: Assume that the inequality is true for some T ≥ 0 , then
Revised: Assume that the inequality is true for some T ≥ 0 , then

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: P t r −
Revised: P t r −

Original: T (cid:88) 
Revised: T (cid:88) t =

Revised: P

Original: [Equation]
Revised: [Equation]

Revised: t ( r

Revised: [Equation]

Revised: b ) =

Revised: [Equation]

Original: P
Revised: P

Revised: T − 1 (cid:88) t =

Revised: P t r − P (cid:48)

Revised: T − 1 (cid:88) t =

Revised: P (cid:48) t ( r + b ) =

Original: [Equation]
Revised: [Equation]

Original:  rby induction hypothesis
Revised: P (cid:48) (cid:32) T − 1 (cid:88) t = P t r − T − 1 (cid:88) t = P (cid:48) t ( r

Revised: [Equation]

Original: Notice that, for any state s :(cid:32) ( P − P (cid:48) ) 
Revised: b ) (cid:33) + ( P − P (cid:48) ) T (cid:88) t =

Original: T (cid:88) t = P 
Revised: P t r ≤ −

Original: [Equation]
Revised: [Equation]

Original: r [Equation]
Revised: ( P − P (cid:48) )

Original: ≤ [Equation] |
Revised: T (cid:88) t =

Original: [Equation] − p [Equation] s
Revised: P t r by induction hypothesis

Original: [Equation] · span ( J [Equation]
Revised: Notice that, for any state s : (cid:32) ( P − P (cid:48) )

Original: T [Equation]
Revised: T (cid:88) t =

Original: ≤
Revised: P t

Original: [Equation]
Revised: [Equation]

Original: p (·| s 
Revised: (cid:33) ( s )

Original: [Equation]
Revised: [Equation]

Original:  − p 
Revised: (cid:107) p ( ·| s ) − p (cid:48) ( ·| s ) (cid:107) · span ( J ( T ))

Original: [Equation] ( ·| s [Equation] (cid:107) by Lemma C.
Revised: ≤ r max D π (cid:107) p ( ·| s ) − p (cid:48) ( ·| s ) (cid:107) by Lemma C.

Original: = b ( s )
Revised: = b ( s )

Original: In the same manner we show that:
Revised: In the same manner we show that:

Original: T (cid:88) 
Revised: T (cid:88) t = P

Original: [Equation]
Revised: [Equation]

Original: P t r ≥

Original: T (cid:88) t =
Revised: (cid:88) t =

Original: P (cid:48) t ( r − b ) .
Revised: P (cid:48) t ( r − b ) .

Original: Hence, as P (cid:48) has non-negative coefﬁcients, denoting by e the unit vector:
Revised: Hence, as P (cid:48) has non-negative coefﬁcients, denoting by e the unit vector:

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Revised: T (cid:88) t =

Revised: P t r −

Revised: T (cid:88) t =

Revised: P (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We can also show that:
Revised: We can also show that:

Original: [Equation]
Revised: [Equation]

Original: And therefore with a multiplication by 1 T +and by taking the Cesáro limit in (cid:13)(cid:13)(cid:13)(cid:80)
Revised: And therefore with a multiplication by 1 T +

Original: [Equation]
Revised: [Equation]

Original: {+and by taking the Cesáro limit in (cid:13)(cid:13)(cid:13)(cid:80) Tt =
 P t r − (cid:80) Tt =
 P (cid:48) t r (cid:48) (cid:13)(cid:13)(cid:13)+} ∞ , and with a triangle inequality:
Revised: {+and by taking the Cesáro limit in (cid:13)(cid:13)(cid:13)(cid:80) Tt =
 P t r − (cid:80) Tt =
 P (cid:48) t r (cid:48) (cid:13)(cid:13)(cid:13)+} ∞ , and with a triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) b (cid:107) ∞ =
Revised: where (cid:107) b (cid:107) ∞ =

Original: r max D π (cid:107) P − P (cid:48) (cid:107) ∞ .
Revised: r max [Equation]

Original: Lemma C.3.
Revised: Lemma C.3.

Original: Let P be the stochastic matrix of an ergodic Markov chain with state space 1 , . . .
Revised: Let P be the stochastic matrix of an ergodic Markov chain with state space 1 , . . .

Original: , S .
Revised: , S .

Original: The matrix A :
Revised: The matrix A :

Original: = I − P has a block decomposition
Revised: = I − P has a block decomposition

Original: [Equation]
Revised: [Equation]

Original: then A S , of size ( S − 1) × ( S − 1) is invertible and (cid:107)
Revised: then A S , of size ( S − 1) × ( S − 1) is invertible and (cid:107)

Original: A − 1 S (cid:107) ∞ =
Revised: A − 1 S (cid:107) ∞ =

Original: sup i ∈S E i τ S , where E i τ S is theexpected time to reach state S from state i .
Revised: sup i ∈S E i τ S , where E i τ S is the expected time to reach state S from state i .

Original: Remark that this lemma is true for any state in S .
Revised: Remark that this lemma is true for any state in S .

Original: Proof.
Revised: Proof.

Original: ( E i τ S )
Revised: ( E i τ S )

Original: i is the unique vector solution to the system:
Revised: i is the unique vector solution to the system:

Original: [Equation]
Revised: [Equation]

Original: We can rewrite this system of equations as: ˜
Revised: We can rewrite this system of equations as: ˜

Original: Av = e − e S , where ˜ A is the matrix
Revised: Av = e − e S , where ˜ A is the matrix

Original: [Equation]
Revised: [Equation]

Original: e the unit vector and e S the vector with value 1 for the last state and 0 otherwise.
Revised: e the unit vector and e S the vector with value 1 for the last state and 0 otherwise.

Original: Then (cid:101) A and A S are invertible and we write:
Revised: Then (cid:101) A and A S are invertible and we write:

Original: [Equation]
Revised: [Equation]

Original: Thus, by computing ˜ A − 1 ( e − e S ) , for i (cid:54) =
Revised: Thus, by computing ˜ A − 1 ( e − e S ) , for i (cid:54) =

Original: S , ( E i τ S )
Revised: S , ( E i τ S )

Original: i
Revised: i

Original: =
Revised: =

Original: A − 1 S e .
Revised: A − 1 S e .

Original: By deﬁnition of the inﬁnite normand using that A S is an M-matrix and that its inverse has non-negative components, (cid:107)
Revised: By deﬁnition of the inﬁnite norm and using that A S is an M-matrix and that its inverse has non-negative components, (cid:107)

Original: A − 1 S (cid:107) ∞ =
Revised: A − 1 S (cid:107) ∞ =

Original: sup i ∈S E i τ S .
Revised: sup i ∈S E i τ S .

Original: In the following lemma, we use the same notations as in Lemma C.2 with a common state space{ 1 , . . .
Revised: In the following lemma, we use the same notations as in Lemma C.2 with a common state space { 1 , . . .

Original: S } .
Revised: S } .

Original: Lemma C.4.
Revised: Lemma C.4.

Original: Let the biases h , h (cid:48) be the biases of the two MDPs verify their respective Bellman equations with the renormalization choice h ( S )
Revised: Let the biases h , h (cid:48) be the biases of the two MDPs verify their respective Bellman equations with the renormalization choice h ( S )

Original: = h (cid:48) ( S ) = 0 .
Revised: = h (cid:48) ( S ) = 0 .

Original: Let sup s ∈S E s τ πs (cid:48) be the worst expected hitting time to reach the state s (cid:48) with policy π , and call T hit :=
Revised: Let sup s ∈S E s τ πs (cid:48) be the worst expected hitting time to reach the state s (cid:48) with policy π , and call T hit :=

Original: inf s (cid:48) ∈S sup s ∈S E s τ s (cid:48) .
Revised: inf s (cid:48) ∈S sup s ∈S E s τ s (cid:48) .

Original: We have the following control of the difference:
Revised: We have the following control of the difference:

Original: [Equation]
Revised: [Equation]

Original: Notice that although the biases are unique up to a constant additive term, the renormalization choicedoes not matter as the unit vector is in the kernel of ( P − P (cid:48) ) .
Revised: Notice that although the biases are unique up to a constant additive term, the renormalization choice does not matter as the unit vector is in the kernel of ( P − P (cid:48) ) .

Original: Proof.
Revised: Proof.

Revised: The computations in this proof follow the same idea as in the proof of [10, Theorem 4.2].

Original: The biases verify the following Bellman equations r − ge =
Revised: The biases verify the following Bellman equations r − ge =

Original: ( I − P ) h , and also the arbitraryrenormalization equations, thanks to the previous remark: h ( S ) = 0 .
Revised: ( I − P ) h , and also the arbitrary renormalization equations, thanks to the previous remark: h ( S ) = 0 .

Original: Using the same notations as inthe proof of Lemma C.3, we can write the system of equations ˜
Revised: Using the same notations as in the proof of Lemma C.3, we can write the system of equations ˜

Original: Ah =
Revised: Ah =

Original: ˜ r − ˜ g , with ˜ r and ˜ g respectivelyequal to r and g everywhere but on the last state, where their value is replaced by 0 .
Revised: ˜ r − ˜ g , with ˜ r and ˜ g respectively equal to r and g everywhere but on the last state, where their value is replaced by 0 .

Original: We therefore have that h =
Revised: We therefore have that h =

Original: ˜
Revised: ˜

Original: A − 1 (˜ r − ˜ g ) , and with identical computations, h (cid:48)
Revised: A − 1 (˜ r − ˜ g ) , and with identical computations, h (cid:48)

Original: =
Revised: =

Original: ˜ A (cid:48)− 1 (˜ r (cid:48) − ˜ g (cid:48) ) .
Revised: ˜ A (cid:48)− 1 (˜ r (cid:48) − ˜ g (cid:48) ) .

Original: By denoting dX := X − X (cid:48) for any vector or matrix X , we get:
Revised: By denoting dX := X − X (cid:48) for any vector or matrix X , we get:

Original: [Equation]
Revised: [Equation]

Original: The previously deﬁned block decompositions are:
Revised: The previously deﬁned block decompositions are:

Original: [Equation]
Revised: [Equation]

Original: and d ˜

Original: [Equation]

Original: =

Original: [Equation]

Original: b − b

Original: [Equation]

Original: (cid:19) .

Original: For s < S , dh ( s ) = − e Ts A − 1 S
Revised: For s < S , dh ( s ) = − e Ts A − 1 S

Original: ( dA S h (cid:48)
Revised: ( dA S h (cid:48)

Original: + d ˜ r − d ˜ g ) and dh ( S ) = 0 .
Revised: + d ˜ r − d ˜ g ) and dh ( S ) = 0 .

Original: Now by taking the norm and using C.1:
Revised: Now by taking the norm and using C.1:

Original: [Equation]
Revised: [Equation]

Original: Notice that (cid:107) dA S (cid:107) ∞ ≤ (cid:107) dP (cid:107) ∞ , (cid:107) d ˜ r (cid:107) ≤ (cid:107) dr (cid:107) and (cid:107) d ˜ g (cid:107) =
Revised: Notice that (cid:107) dA S (cid:107) ∞ ≤ (cid:107) dP (cid:107) ∞ , (cid:107) d ˜ r (cid:107) ≤ (cid:107) dr (cid:107) and (cid:107) d ˜ g (cid:107) =

Original: | dg | .
Revised: | dg | .

Original: Using Lemma C.2 and Lemma C.3,and taking the inﬁmum for the choice of the state of renormalization implies the claimed inequalityfor the biases.
Revised: Using Lemma C.2 and Lemma C.3, and taking the inﬁmum for the choice of the state of renormalization implies the claimed inequality for the biases.

Original: C.2 A McDiarmid’s inequality
Revised: C.2 A McDiarmid’s inequality

Original: Lemma C.5.
Revised: Lemma C.5.

Original: Recall that m max is the stationary measure of the Markov chain under policy π max ,such that for every state s : π max ( s ) =
Revised: Recall that m max is the stationary measure of the Markov chain under policy π max , such that for every state s : π max ( s ) =

Original: A max .
Revised: A max .

Original: Let k be an episode, and assume that the length of this episode I k is at least I ( T ) = 1 +max (cid:8) Q max , T 1 / 4 (cid:9) , with Q max := (cid:16)
Revised: Let k be an episode, and assume that the length of this episode I k is at least I ( T ) = 1 + max (cid:8) Q max , T 1 / 4 (cid:9) , with Q max := (cid:16)

Original: [Equation]
Revised: [Equation]

Revised: (cid:19) .

Original: Then, with probability atleast 1 − 14 T :
Revised: Then, with probability at least 1 − 14 T :

Original: [Equation]
Revised: [Equation]

Original: We will now prove Lemma C.5:
Revised: We will now prove Lemma C.5:

Original: Proof.
Revised: Proof.

Original: Let k be an episode such that I k ≥ I ( T ) , and ﬁrst consider it is of ﬁxed length I .
Revised: Let k be an episode such that I k ≥ I ( T ) , and ﬁrst consider it is of ﬁxed length I .

Original: Denoteby ˚ r the vector of reward equal to 1 if the current state is x k and 0 otherwise.
Revised: Denote by ˚ r the vector of reward equal to 1 if the current state is x k and 0 otherwise.

Original: Denote by ˚ g π k thegain associated to the policy π k for the transitions p and rewards ˚ r , and similarly deﬁne ˚ h π k thebias, translated so that ˚ h π k ( S − 1) = 0 .
Revised: Denote by ˚ g π k the gain associated to the policy π k for the transitions p and rewards ˚ r , and similarly deﬁne ˚ h π k the bias, translated so that ˚ h π k ( S − 1) = 0 .

Original: Notice in that case, that if we denote by m k the stationarydistribution under policy π k , that m max ( S − 1) ≤ m k
Revised: Notice in that case, that if we denote by m k the stationary distribution under policy π k , that m max ( S − 1) ≤ m k

Original: ( s ) for any state s , assuming that S ≥ λµ + 1 .
Revised: ( s ) for any state s , assuming that S ≥ λµ + 1 .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: ˚ g π k +˚ h

Original: [Equation] s u ) [Equation] p ( ·| s u , π k ( s u )) , ˚ h π k (cid:69) using a Bellman’s equation=
Revised: using a Bellman’s equation

Original: [Equation]
Revised: [Equation]

Original: |
Revised: By Azuma-Hoeffding inequality B.2, following the same proof as in section 4.3.2 of [11], notice that X u = ˚

Original: [Equation]
Revised: [Equation]

Original: u )
Revised: π

Original: [Equation]
Revised: [Equation]

Original: ˚ h
Revised: π k

Original: [Equation]By Azuma-Hoeffding inequality B.2, following the same proof as in section 4.3.2 of [10], notice that X u = ˚
Revised: [Equation]

Original: h π k ( s u +1 ) − (cid:68) p ( ·| s u , π k ( s u )) , ˚ h π k (cid:69) form a martingale difference sequence with | X u| ≤ D :
Revised: u

Original: [Equation]
Revised: [Equation]

Original: Using that (cid:12)(cid:12)(cid:12) ˚ h π k ( s t k ) − ˚ h π k ( s t k +1 )
Revised: ˚ [Equation] k [Equation]

Original: (cid:12)(cid:12)(cid:12) ≤ D, with probability at least 1 − 1 t 2 :
Revised: form a martingale difference sequence with | X u | ≤ D :

Original: [Equation]
Revised: [Equation]

Original: u
Revised: Using that (cid:12)(cid:12)(cid:12) ˚

Original: [Equation]
Revised: [Equation]

Revised: π k ( s t k ) − ˚ h π k

Revised: ( s t k +1 )

Revised: (cid:12)(cid:12)(cid:12) ≤ D , with probability at least 1 − 1 I 2 : ν k ( x k , a k ) ≥

Revised: t k +1 − 1 (cid:88) u

Original:  On the other hand: 
Revised: = t k ˚ g π k − 5 D (cid:112) I log I. On the other hand: t k +

Original: [Equation]
Revised: [Equation]

Original: so that, using that m k ( a k ) ≥ m max ( S − 1) , with probability at least 1 − 1 I 5 :ν k ( x k , a k
Revised: 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ≥ m max ( S
Revised: k

Original: [Equation]
Revised: [Equation]

Original:  I 
Revised: so that, using that m k ( a k ) ≥ m max ( S − 1) , with probability at least 1 − 1 I 5 : ν k ( x k , a k ) ≥ m max ( S −

Original: [Equation]
Revised: [Equation]

Original:  5  (cid:112) I log I.
Revised: I − 5 D (cid:112) I log I.

Original: We now use a union bound over the possible values of the episode lengths I k , between I ( T ) + 1 and
Revised: We now use a union bound over the possible values of the episode lengths I k , between I ( T ) + 1 and

Original: T :
Revised: T :

Original: [Equation]
Revised: [Equation]

Original: T (cid:88)

Original: I =

Original: I ( T )+

Original: I 5 ≤

Original: T (cid:88)

Original: I = T1 / 4 + I
Revised: I = T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: so that we now have that with probability at least 1 − 14 T :
Revised: so that we now have that with probability at least 1 − 14 T :

Original: [Equation]
Revised: [Equation]

Original: We can show a corollary of Lemma C.5 that we will use for the regret computations:
Revised: We can show a corollary of Lemma C.5 that we will use for the regret computations:

Original: Corollary C.6.
Revised: Corollary C.6.

Original: For an episode k such that its length I k is greater than I ( T ) ,with probability at least1 − 14 T :
Revised: For an episode k such that its length I k is greater than I ( T ) ,with probability at least 1 − 14 T :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: With Lemma C.5, it is enough to show that 5  √ I k log I k ≤ m max ( S − 1)
Revised: With Lemma C.5, it is enough to show that 5 D √ I k log I k ≤ m max ( S − 1)

Original: I k , i.e. that 
Revised: I k , i.e. that (cid:113) I k log I k ≥ 10 D m max ( S − 1) =: B .

Original: By monotonicity, as I k ≥ Q max =
Revised: By monotonicity, as I k ≥ Q max =

Original: B 2 log B 4 we can show in- stead that B  log  B  (cid:1) .
Revised: B 2 log B 4 we can show instead that B 2 log B 4 ≥ B 2 log (cid:0) B 2 log B 4 (cid:1) .

Original: This last inequality is true, using that log x ≥ log(2 log x ) for x > 1 .
Revised: This last inequality is true, using that log x ≥ log(2 log x ) for x > 1 .

Original: This proves the corollary.
Revised: This proves the corollary.