Original: Reinforcement Learning in a Birth and Death Process: Breaking the Dependence on the State Space
Revised: Reinforcement Learning in a Birth and Death Process: Breaking the Dependence on the State Space

Original: Abstract
Revised: Abstract

Original: In this paper, we revisit the regret of undiscounted reinforcement learning in MDPs with a birth and death structure.
Revised: In this paper, we revisit the regret of undiscounted reinforcement learning in MDPs with a birth and death structure.

Original: Speci ﬁ cally, we consider a controlled queue with impatient jobs and the main objective is to optimize a trade-off between energy consumption and user-perceived performance.
Revised: Specifically, we consider a controlled queue with impatient jobs and the main objective is to optimize a trade-off between energy consumption and user-perceived performance.

Original: Within this setting, the diameter D of the MDP is Ω ( S S ) , where S is the number of states.
Revised: Within this setting, the diameter D of the MDP is Ω( S S ) , where S is the number of states.

Original: Therefore, the existing lower and upper bounds on the regret at time T , of order O ( √ DSAT ) for MDPs with S states and A actions, may suggest that reinforcement learning is inef ﬁ cient here.
Revised: Therefore, the existing lower and upper bounds on the regret at time T , of order O ( √ DSAT ) for MDPs with S states and A actions, may suggest that reinforcement learning is inefficient here.

Original: In our main result however, we exploit the structure of our MDPs to show that the regret of a slightly-tweaked version of the classical learning algorithm U CRLis in fact upper bounded by ˜ O ( √ E 2 AT ) where E 2 is related to the weighted second moment of the stationary measure of a reference policy.
Revised: In our main result however, we exploit the structure of our MDPs to show that the regret of a slightly-tweaked version of the classical learning algorithm U CRLis in fact upper bounded by ˜ O ( √ E 2 AT ) where E 2 is related to the weighted second moment of the stationary measure of a reference policy.

Original: Importantly, E 2 is bounded independently of S .
Revised: Importantly, E 2 is bounded independently of S .

Original: Thus, our bound is asymptotically independent of the number of states and of the diameter.
Revised: Thus, our bound is asymptotically independent of the number of states and of the diameter.

Original: This result is based on a careful study of the number of visits performed by the learning algorithm to the states of the MDP, which is highly non-uniform.
Revised: This result is based on a careful study of the number of visits performed by the learning algorithm to the states of the MDP, which is highly non-uniform.

Original:  Introduction
Revised:  Introduction

Original: In the context of undiscounted reinforcement learning in Markov decision processes (MDPs), it has been shown in the seminal work [11] that the total regret of any learning algorithm with respect to an optimal policy is lower bounded by Ω ( √ DSAT ) , where S is the number of states, A the number of actions, T the time horizon and D the diameter of the MDP.
Revised: In the context of undiscounted reinforcement learning in Markov decision processes (MDPs), it has been shown in the seminal work [11] that the total regret of any learning algorithm with respect to an optimal policy is lower bounded by Ω( √ DSAT ) , where S is the number of states, A the number of actions, T the time horizon and D the diameter of the MDP.

Original: Roughly speaking, the diameter is the mean time to move from any state s to any other state s ′ within an appropriate policy.
Revised: Roughly speaking, the diameter is the mean time to move from any state s to any other state s ′ within an appropriate policy.

Original: In the literature, several efforts have been dedicated to approach this lower bound.
Revised: In the literature, several efforts have been dedicated to approach this lower bound.

Original: As a result, learning algorithms have been developed with a total regret of ˜ O ( DS √ AT ) in [11], ˜ O
Revised: As a result, learning algorithms have been developed with a total regret of ˜ O ( DS √ AT ) in [11], ˜ O

Original: ( D √ SAT ) in [3] and even ˜ O ( √ DSAT )
Revised: ( D √ SAT ) in [3] and even ˜ O ( √ DSAT )

Original: according to [21, 25].
Revised: according to [21, 25].

Original: These results may give a sense of optimality since the lower bound is attained up to some universal constant.
Revised: These results may give a sense of optimality since the lower bound is attained up to some universal constant.

Original: However, lower bounds are based on the minimax approach, which relies on the worst possible MDP with given D , A and S .
Revised: However, lower bounds are based on the minimax approach, which relies on the worst possible MDP with given D , A and S .

Original: This means that when a reinforcement learning algorithm is used on a given MDP, one can expect a much better performance.
Revised: This means that when a reinforcement learning algorithm is used on a given MDP, one can expect a much better performance.

Original: One way to alleviate the minimax lower bound is to consider structured reinforcement learning , or equivalently MDPs with some speci ﬁ c structure.
Revised: One way to alleviate the minimax lower bound is to consider structured reinforcement learning , or equivalently MDPs with some specific structure.

Original: The exploitation of such structure may yield more ef ﬁ cient learning algorithms or tighter regret analyses of existing learning algorithms.
Revised: The exploitation of such structure may yield more efficient learning algorithms or tighter regret analyses of existing learning algorithms.

Original: In this context, a ﬁ rst example is to consider factored MDPs [6, 9], i.e., MDPs where the state space can be factored into a number of components; in this case, roughly speaking, S = K n where n is the number of “factors” and K is the number of states in each factor.
Revised: In this context, a first example is to consider factored MDPs [6, 9], i.e., MDPs where the state space can be factored into a number of components; in this case, roughly speaking, S = K n where n is the number of “factors” and K is the number of states in each factor.

Original: The regret of learning algorithms in factored
Revised: The regret of learning algorithms in factored

Original: MDPs has been analyzed in [20, 17, 24, 14] and it is found that the S term of existing upper bounds can be replaced by nK .
Revised: MDPs has been analyzed in [20, 17, 24, 14] and it is found that the S term of existing upper bounds can be replaced by nK .

Original: A similar approach is used in [8] to learn the optimal policy in stochastic bandits with a regret that is logarithmic in the number of states.
Revised: A similar approach is used in [8] to learn the optimal policy in stochastic bandits with a regret that is logarithmic in the number of states.

Original: There is also a line of research works that exploit the parametric nature of MDPs.
Revised: There is also a line of research works that exploit the parametric nature of MDPs.

Original: Inspired by parametric bandits, a d -linear additive model was introduced in [12], where it is shown that an optimistic modi ﬁ cation of Least-Squares Value
Revised: Inspired by parametric bandits, a d -linear additive model was introduced in [12], where it is shown that an optimistic modification of Least-Squares Value

Original: Iteration, see [15], achieves a regret over a ﬁ nite horizon H of ˜ O ( √ d 3 H 3 T ) where d is the ambient dimension of the feature space (the number of unknown parameters).
Revised: Iteration, see [15], achieves a regret over a finite horizon H of ˜ O ( √ d 3 H 3 T ) where d is the ambient dimension of the feature space (the number of unknown parameters).

Original: In this case, the regret does not depend on the number of states and actions and the diameter is replaced by the horizon.
Revised: In this case, the regret does not depend on the number of states and actions and the diameter is replaced by the horizon.

Original: A discussion about the inapplicability of this approach to our case is postponed to Section 4.2.
Revised: A discussion about the inapplicability of this approach to our case is postponed to Section 4.2.

Original: Learning in Queueing Systems.
Revised: Learning in Queueing Systems.

Original: The control of queueing systems is undoubtedly one of the main application areas of MDPs; see, e.g., [16, Chapters 1–3] and [13].
Revised: The control of queueing systems is undoubtedly one of the main application areas of MDPs; see, e.g., [16, Chapters 1–3] and [13].

Original: Within the rich literature of structured reinforcement learning however, few papers are dedicated to reinforcement learning in queueing systems, see [22, Section 5], and this motivates us to examine the total regret in this context.
Revised: Within the rich literature of structured reinforcement learning however, few papers are dedicated to reinforcement learning in queueing systems, see [22, Section 5], and this motivates us to examine the total regret in this context.

Original: Typical control problems on queues have the following distinguishing characteristics:
Revised: Typical control problems on queues have the following distinguishing characteristics:

Original: No discount .
Revised: No discount .

Original: Discounting costs or rewards is common practice in the reinforcement learning literature, especially in Q-learning algorithms [19].
Revised: Discounting costs or rewards is common practice in the reinforcement learning literature, especially in Q-learning algorithms [19].

Original: However, in queues one is typically interested in optimizing with respect to the average cost.
Revised: However, in queues one is typically interested in optimizing with respect to the average cost.

Original: Large diameter.
Revised: Large diameter.

Original: Queueing systems are usually investigated under a drift condition that makes the system stable ,
Revised: Queueing systems are usually investigated under a drift condition that makes the system stable ,

Original: i
Revised: i

Original: e., positive recurrent.
Revised: e., positive recurrent.

Original: This condition implies that some states are hard to reach.
Revised: This condition implies that some states are hard to reach.

Original: In fact, for many queueing control problems, the diameter D is exponential in the size of the state space.
Revised: In fact, for many queueing control problems, the diameter D is exponential in the size of the state space.

Original: Even in the simple case of an M/M/1 queue with a ﬁ nite buffer, or equivalently a birth–death process with a ﬁ nite state space and constant birth and death rates, the diameter is exponential in the size of the state space.
Revised: Even in the simple case of an M/M/1 queue with a finite buffer, or equivalently a birth–death process with a finite state space and constant birth and death rates, the diameter is exponential in the size of the state space.

Original: Structured transition matrices.
Revised: Structured transition matrices.

Original: Queueing models describe how jobs join and leave queues, and this yields bounded state transitions.
Revised: Queueing models describe how jobs join and leave queues, and this yields bounded state transitions.

Original: As a result, MDPs on queues have sparse and structured transition matrices.
Revised: As a result, MDPs on queues have sparse and structured transition matrices.

Original: The regret bounds discussed above and item 2 may suggest that the total regret of existing learning algorithm, when applied to queueing systems, is large.
Revised: The regret bounds discussed above and item 2 may suggest that the total regret of existing learning algorithm, when applied to queueing systems, is large.

Original: However, they often work well in practice and this bring us to consider the following question: When the underlying MDP has the structure of a queueing system, do the diameter D or the number of states S actually play a role in the regret?
Revised: However, they often work well in practice and this bring us to consider the following question: When the underlying MDP has the structure of a queueing system, do the diameter D or the number of states S actually play a role in the regret?

Original: Our Contribution.
Revised: Our Contribution.

Original: In this paper, we examine the previous question with respect to the class of control problems presented in [1].
Revised: In this paper, we examine the previous question with respect to the class of control problems presented in [1].

Original: Speci ﬁ cally, an in ﬁ nite sequence of jobs joins a service system over time to receive some processing according to the ﬁ rst-come ﬁ rst-served scheduling rule; the system can buffer at most S − 1 jobs and in fact it corresponds to an M/M/1/S-1 queue.
Revised: Specifically, an infinite sequence of jobs joins a service system over time to receive some processing according to the first-come first-served scheduling rule; the system can buffer at most S − 1 jobs and in fact it corresponds to an M/M/1/S-1 queue.

Original: In addition, each job comes with a deadline constraint, and if a job is not completed before its deadline, then it becomes obsolete and is removed from the system.
Revised: In addition, each job comes with a deadline constraint, and if a job is not completed before its deadline, then it becomes obsolete and is removed from the system.

Original: The controller chooses the server processing speed and the objective is to design a speed policy for the server that minimizes its average energy consumption plus an obsolescence cost per deadline miss.
Revised: The controller chooses the server processing speed and the objective is to design a speed policy for the server that minimizes its average energy consumption plus an obsolescence cost per deadline miss.

Original: Although this may look quite speci ﬁ c, this problem captures the typical characteristics of a controlled queue: i) the transition matrix has the structure of a birth and death process with jump probabilities that are af ﬁ ne functions of the state and ii) the reward is linear in the state and convex in the action.
Revised: Although this may look quite specific, this problem captures the typical characteristics of a controlled queue: i) the transition matrix has the structure of a birth and death process with jump probabilities that are affine functions of the state and ii) the reward is linear in the state and convex in the action.

Original: For any MDP in this class, de ﬁ ned in full details in Section 3, we show that the diameter is D = Ω ( S S − 2 ) ; see Appendix B.3.
Revised: For any MDP in this class, defined in full details in Section 3, we show that the diameter is D = Ω( S S − 2 ) ; see Appendix B.3.

Original: Thus, without exploiting the particular structure of this MDP, the existing lower and upper bounds do not justify the reason why standard learning algorithms work ef ﬁ ciently here.
Revised: Thus, without exploiting the particular structure of this MDP, the existing lower and upper bounds do not justify the reason why standard learning algorithms work efficiently here.

Original: We provide a slight variation of the learning algorithm U CRL 2, introduced in [11], and show in our main result that the resulting regret is upper bounded by ˜ O ( √ E 2 AT ) where E 2 is a term that depends on the stationary measure of a reference policy de ﬁ ned in Section 3.1.
Revised: We provide a slight variation of the learning algorithm U CRL 2, introduced in [11], and show in our main result that the resulting regret is upper bounded by ˜ O ( √ E 2 AT ) where E 2 is a term that depends on the stationary measure of a reference policy defined in Section 3.1.

Original: Importantly, E 2 does not depend on S .
Revised: Importantly, E 2 does not depend on S .

Original: Thus, ef ﬁ cient reinforcement learning can be achieved independently of the number of states by exploiting the stationary structure of the MDP.
Revised: Thus, efficient reinforcement learning can be achieved independently of the number of states by exploiting the stationary structure of the MDP.

Original: Let us provide some intuition about our result.
Revised: Let us provide some intuition about our result.

Original: First, one may think that any learning algorithm should visit each state a suf ﬁ cient number of times, which justi ﬁ es why the diameter of an MDP appears in existing regret analyses.
Revised: First, one may think that any learning algorithm should visit each state a sufficient number of times, which justifies why the diameter of an MDP appears in existing regret analyses.

Original: However, this point of view does not take into account the fact that the value of an MDP is the scalar product of the reward and the stationary measure of the optimal policy.
Revised: However, this point of view does not take into account the fact that the value of an MDP is the scalar product of the reward and the stationary measure of the optimal policy.

Original: If this stationary measure is “highly non-uniform”, then some states are rarely visited under the optimal policy and barely contribute to the value.
Revised: If this stationary measure is “highly non-uniform”, then some states are rarely visited under the optimal policy and barely contribute to the value.

Original: In this case, we claim that the learner may not need to visit the rare states that often to get a good estimation of the value, and thus it may not need to pay for the diameter.
Revised: In this case, we claim that the learner may not need to visit the rare states that often to get a good estimation of the value, and thus it may not need to pay for the diameter.

Original:  Reinforcement Learning Framework
Revised:  Reinforcement Learning Framework

Original: We consider a unichain Markov Decision Process (MDP) M
Revised: We consider a unichain Markov Decision Process (MDP) M

Original: =
Revised: =

Original: ( S , A , P, r ) in discrete time where S is the ﬁ nite state space,
Revised: ( S , A , P, r ) in discrete time where S is the finite state space,

Original: A the ﬁ nite action space, P the transition probabilities and r the expected reward function [16].
Revised: A the finite action space, P the transition probabilities and r the expected reward function [16].

Original: Let also S := |S| and A := |A| where | · | is the set cardinality operator.
Revised: Let also S := |S| and A := |A| where | · | is the set cardinality operator.

Original: The model-based reinforcement learning problem consists in ﬁ nding a learning algorithm , or learner, that chooses actions to maximize a cumulative reward over a ﬁ nite time horizon T .
Revised: The model-based reinforcement learning problem consists in finding a learning algorithm , or learner, that chooses actions to maximize a cumulative reward over a finite time horizon T .

Original: At each time step t ∈ N , the system is in state s t ∈ S and the learner chooses an action a t ∈ A .
Revised: At each time step t ∈ N , the system is in state s t ∈ S and the learner chooses an action a t ∈ A .

Original: When executing a t , the learner receives a random reward r t ( s t , a t ) with mean r ( s t , a t ) and the system moves, at time step t +1 , to state s ′ with probability P ( s ′ | s t , a t ) .
Revised: When executing a t , the learner receives a random reward r t ( s t , a t ) with mean r ( s t , a t ) and the system moves, at time step t +1 , to state s ′ with probability P ( s ′ | s t , a t ) .

Original: The learning algorithm does not know the MDP M except for the sets S and A .
Revised: The learning algorithm does not know the MDP M except for the sets S and A .

Original: For simplicity, in the following we consider weakly communicating MDPs.
Revised: For simplicity, in the following we consider weakly communicating MDPs.

Original: Since we will be interested in the long-run average cost, this will let us remove the dependence on the initial state for several quantities of interest.
Revised: Since we will be interested in the long-run average cost, this will let us remove the dependence on the initial state for several quantities of interest.

Original: 2.1 Undiscounted Regret
Revised: 2.1 Undiscounted Regret

Original: Given an MDP M , let Π :=
Revised: Given an MDP M , let Π :=

Original: { π : S → A} denote the set of stationary and deterministic policies and let
Revised: { π : S → A} denote the set of stationary and deterministic policies and let

Original: [Equation]
Revised: [Equation]

Original: denote the average reward induced by policy π .
Revised: denote the average reward induced by policy π .

Original: Since M has ﬁ nite state and action spaces, we notice that i)
Revised: Since M has finite state and action spaces, we notice that i)

Original: The limit in (1) always exists, ii)
Revised: The limit in (1) always exists, ii)

Original: It does not depend on the initial state s 0 when M is unichain [16] and iii)
Revised: It does not depend on the initial state s 0 when M is unichain [16] and iii)

Original: The restriction to stationary and deterministic policies is not a loss of optimality [16, Theorem 8.4.5].
Revised: The restriction to stationary and deterministic policies is not a loss of optimality [16, Theorem 8.4.5].

Original: Let also ρ ∗ :
Revised: Let also ρ ∗ :

Original: = ρ ∗ ( M ) := max π ∈ Π ρ ( M, π ) be the optimal average reward.
Revised: = ρ ∗ ( M ) := max π ∈ Π ρ ( M, π ) be the optimal average reward.

Original: De ﬁ nition 2.1 (Regret) .
Revised: Definition 2.1 (Regret) .

Original: The regret at time T of the learning algorithm L is
Revised: The regret at time T of the learning algorithm L is

Original: [Equation]
Revised: [Equation]

Original: The regret 2 is a natural benchmark for evaluating the performance of a learning algorithm.
Revised: The regret 2 is a natural benchmark for evaluating the performance of a learning algorithm.

Original: In [11], a universal lower bound on Reg( M, L , T ) has been developed in terms of the diameter of the underlying MDP.
Revised: In [11], a universal lower bound on Reg( M, L , T ) has been developed in terms of the diameter of the underlying MDP.

Original: De ﬁ nition 2.2 (Diameter of an MDP) .
Revised: Definition 2.2 (Diameter of an MDP) .

Original: Let π : S
Revised: Let π : S

Original: → A be a stationary policy of M with initial state s .
Revised: → A be a stationary policy of M with initial state s .

Original: Let T ( s ′ | M, π , s ) :
Revised: Let T ( s ′ | M, π, s ) :

Original: = min { t ≥ 0 : s t = s ′
Revised: = min { t ≥ 0 : s t = s ′

Original: | s 0 = s } be the random variable for the ﬁ rst time step in which s ′ is reached from s under π .
Revised: | s 0 = s } be the random variable for the first time step in which s ′ is reached from s under π .

Original: Then, we say that the diameter of M is
Revised: Then, we say that the diameter of M is

Original: [Equation]
Revised: [Equation]

Original: It should be clear that the diameter of an MDP can be large if there exist states that are hard to reach.
Revised: It should be clear that the diameter of an MDP can be large if there exist states that are hard to reach.

Original: Within the set of structured MDPs considered in this paper, this will be the case and we will show that
Revised: Within the set of structured MDPs considered in this paper, this will be the case and we will show that

Original: D = Ω ( S S − 2 ) .
Revised: D = Ω( S S − 2 ) .

Original: The following result shows that all learning algorithms have a regret that eventually increase with √ D .
Revised: The following result shows that all learning algorithms have a regret that eventually increase with √ D .

Original: Theorem 2.3 (Universal lower bound [11]) .
Revised: Theorem 2.3 (Universal lower bound [11]) .

Original: For any learning algorithm L , any natural numbers
Revised: For any learning algorithm L , any natural numbers

Original: S, A ≥ 10 , D ≥ 20 log A S , and T ≥ DSA , there is an MDP M with S states, A actions, and diameter D such that for any initial state s 
Revised: S, A ≥ 10 , D ≥ 20 log A S , and T ≥ DSA , there is an MDP M with S states, A actions, and diameter D such that for any initial state s ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: In view of this result, the diameter of an MDP and its state space appear to be critical parameters for evaluating the performance of a learning algorithm.
Revised: In view of this result, the diameter of an MDP and its state space appear to be critical parameters for evaluating the performance of a learning algorithm.

Original: 2.2 The U CRL 2 Algorithm
Revised: 2.2 The U CRL 2 Algorithm

Original: We now focus on U CRL 2, a classical reinforcement learning algorithm introduced in [11] that is a variant of U CRL [2].
Revised: We now focus on U CRL 2, a classical reinforcement learning algorithm introduced in [11] that is a variant of U CRL [2].

Original: While more ef ﬁ cient algorithms have been proposed for the general case (see for example [3, 21]), we will show that U CRL 2 already achieves a very low regret, namely ˜ O ( √ AT ) , independent of S so using more re ﬁ ned algorithms can only bring marginal gains.
Revised: While more efficient algorithms have been proposed for the general case (see for example [3, 21]), we will show that U CRL 2 already achieves a very low regret, namely ˜ O ( √ AT ) , independent of S so using more refined algorithms can only bring marginal gains.

Original: U CRL 2 is based on episodes .
Revised: U CRL 2 is based on episodes .

Original: For each episode k , let t k denote its start time.
Revised: For each episode k , let t k denote its start time.

Original: For each state s and action a , let ν k ( s, a ) denote the number of visits of ( s, a ) during episode k and let N t ( s, a ) :
Revised: For each state s and action a , let ν k ( s, a ) denote the number of visits of ( s, a ) during episode k and let N t ( s, a ) :

Original: = # { τ < t : s τ = s, a τ = a } denote the number of visits of ( s, a ) until timestep t .
Revised: = # { τ < t : s τ = s, a τ = a } denote the number of visits of ( s, a ) until timestep t .

Original: Let M k be the con ﬁ dence set of MDPs with transition probabilities ˜ p and rewards ˜ r that are “close” to the empirical
Revised: Let M k be the confidence set of MDPs with transition probabilities ˜ p and rewards ˜ r that are “close” to the empirical

Original: MDP at episode k , ˆ p k and ˆ r k , i.e., ˜ p and ˜ r satisfy
Revised: MDP at episode k , ˆ p k and ˆ r k , i.e., ˜ p and ˜ r satisfy

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: With these quantities, a pseudocode for U CRL 2 is given in Algorithm 1.
Revised: With these quantities, a pseudocode for U CRL 2 is given in Algorithm 1.

Original: We notice that U CRLrelies on Extended Value Iteration (EVI), that is a variant of the celebrated Value Iteration (VI) algorithm [16]; for further details about EVI, we point the reader to [11, Section 3.1].
Revised: We notice that U CRLrelies on Extended Value Iteration (EVI), that is a variant of the celebrated Value Iteration (VI) algorithm [16]; for further details about EVI, we point the reader to [11, Section 3.1].

Original: Let us comment on how U CRL 2 works.
Revised: Let us comment on how U CRL 2 works.

Original: There are three main steps.
Revised: There are three main steps.

Original: First, at the start of each episode, U CRL
Revised: First, at the start of each episode, U CRL

Original: Algorithm 1:

Original: The Input: A con ﬁ dence parameter δ ∈ (0 , 1) , S and A .
Revised: Input: A confidence parameter δ ∈ (0 , 1) , S and A .

Original: Output: .
Revised: Output: .

Original: 1 Set t := 1 and observe s2 for episodes k = 1 , 2 , . . .
Revised: 1 Set t := 1 and observe s2 for episodes k = 1 , 2 , . . .

Original: do
Revised: do

Original: Compute the estimates ˆ r ( s, a ) and ˆ p k
Revised: Compute the estimates ˆ r ( s, a ) and ˆ p k

Original: ( s ′ | s, a ) as in (7).
Revised: ( s ′ | s, a ) as in (7).

Original: Use “Extended Value Iteration” to ﬁ nd a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k such that
Revised: Use “Extended Value Iteration” to find a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k such that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: while ν k ( s t , ˜ π k ( s t )) < max { 1 , N t k ( s t , ˜ π k ( s t )) } do
Revised: while ν k ( s t , ˜ π k ( s t )) < max { 1 , N t k ( s t , ˜ π k ( s t )) } do

Original: Choose action a t = ˜
Revised: Choose action a t = ˜

Original: π k ( s t ) , obtain reward r t and observe s t +1 ;
Revised: π k ( s t ) , obtain reward r t and observe s t +1 ;

Original: ν k ( s t , a t ) :
Revised: ν k ( s t , a t ) :

Original: =
Revised: =

Original: ν k
Revised: ν k

Original: ( s t , a t ) + 1 ; t :=
Revised: ( s t , a t ) + 1 ; t :=

Original: t + 1 ; end 10 end computes the empirical estimates
Revised: t + 1 ; end 10 end computes the empirical estimates

Original: [Equation]
Revised: [Equation]

Original: of the reward and probability transitions, respectively, where � E is the indicator function of E .
Revised: of the reward and probability transitions , respectively, where 1 E is the indicator function of E .

Original: Then, it applies Extended Value Iteration (EVI) to ﬁ nd a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k such that (6) holds true.
Revised: Then, it applies Extended Value Iteration (EVI) to find a policy ˜ π k and an optimistic MDP ˜ M k ∈ M k such that (6) holds true.

Original: Finally, it executes policy ˜ π k until it ﬁ nds a state-action pair ( s, a ) whose count within episode k is greater than the corresponding state-action count prior to episode k .
Revised: Finally, it executes policy ˜ π k until it finds a state-action pair ( s, a ) whose count within episode k is greater than the corresponding state-action count prior to episode k .

Original:  Controlled Birth and Death Processes for Energy Minimization
Revised:  Controlled Birth and Death Processes for Energy Minimization

Original: Now, we focus on a speci ﬁ c class of MDPs that has been introduced in [1], which provides a rather general example of a controlled birth and death process with convex costs on the actions and linear rates.
Revised: Now, we focus on a specific class of MDPs that has been introduced in [1], which provides a rather general example of a controlled birth and death process with convex costs on the actions and linear rates.

Original: We will denote by M the set of MDPs with the structure described below.
Revised: We will denote by M the set of MDPs with the structure described below.

Original: The MDPs in M have been proposed to represent a Dynamic Voltage and Frequency Scaling (DVFS) processor executing jobs with soft obsolescence deadlines.
Revised: The MDPs in M have been proposed to represent a Dynamic Voltage and Frequency Scaling (DVFS) processor executing jobs with soft obsolescence deadlines.

Original: Here, jobs arrive according to a Poisson process with rate λ ∈ [0 , λ max ] in a buffer of size S − 1 .
Revised: Here, jobs arrive according to a Poisson process with rate λ ∈ [0 , λ max ] in a buffer of size S − 1 .

Original: If the buffer is full and a job arrives, then the job is rejected.
Revised: If the buffer is full and a job arrives, then the job is rejected.

Original: Each job has a deadline and a size, i.e., amount of work, which are exponentially distributed random variables with rates µ ∈ [0 , µ max ] and one, respectively.
Revised: Each job has a deadline and a size, i.e., amount of work, which are exponentially distributed random variables with rates µ ∈ [0 , µ max ] and one, respectively.

Original: Job deadlines and sizes are all independent random variables.
Revised: Job deadlines and sizes are all independent random variables.

Original: If a job misses its deadline, which is a real time constraint activated at the moment of its arrival, it is removed from the queue without being served and a cost C is paid.
Revised: If a job misses its deadline, which is a real time constraint activated at the moment of its arrival, it is removed from the queue without being served and a cost C is paid.

Original: The processor serves jobs under any work-conserving scheduling discipline, e.g., ﬁ rst-come ﬁ rst-served, with a processing speed that belongs to the ﬁ nite set { 0 , . . .
Revised: The processor serves jobs under any work-conserving scheduling discipline, e.g., first-come first-served, with a processing speed that belongs to the finite set { 0 , . . .

Original: , A max } .
Revised: , A max } .

Original: The objective is to design a speed policy that minimizes the sum of the long term power dissipation and the cost induced by jobs missing their deadlines.
Revised: The objective is to design a speed policy that minimizes the sum of the long term power dissipation and the cost induced by jobs missing their deadlines.

Original: When the processor works at speed a ∈ { 0 , . .
Revised: When the processor works at speed a ∈ { 0 , . .

Original: A max } , it processes a units of work per second while its power dissipation is w ( a ) .
Revised: A max } , it processes a units of work per second while its power dissipation is w ( a ) .

Original: After uniformization, it is shown in [1] that this control problem can be modeled as an MDP in discrete time with a “birth-and-death” transition matrix of size S .
Revised: After uniformization, it is shown in [1] that this control problem can be modeled as an MDP in discrete time with a “birth-and-death” transition matrix of size S .

Original: Speci ﬁ cally, we have an MDP
Revised: Specifically, we have an MDP

Original: M = ( S , A , P, r ) where
Revised: M = ( S , A , P, r ) where

Original: S = { 0 , . .
Revised: S = { 0 , . .

Original: , S − 1 } , with s ∈ S representing the number of jobs in the system, and A = { 0 , . . .
Revised: , S − 1 } , with s ∈ S representing the number of jobs in the system, and A = { 0 , . . .

Original: , A max } , with a ∈ A representing the processor speed.
Revised: , A max } , with a ∈ A representing the processor speed.

Original: Then, the transition probabilities under policy π are given by
Revised: Then, the transition probabilities under policy π are given by

Original: P i,j ( π ) =   1 U 
Revised: [Equation]  1 U λ i

Original: λ i if i < S − 1 and j = i [-+ 
-]
Revised: if i < S − 1 and j = i [-+ 
-]+1 U

Original: U ( π ( i ) + iµ ) if i > 0 and j = i −
Revised: ( π ( i ) + iµ ) if i > 0 and j = i −

Original: P ii if j = i otherwise , where U := λ max
Revised: P ii if j = i otherwise , where U := λ max

Original: +( S − 1) µ max + A max is a uniformization constant, P ii = 1 U
Revised: +( S − 1) µ max + A max is a uniformization constant, P ii = 1 U

Original: ( U − λ
Revised: ( U − λ

Original: i − µi − π ( i )) and λ
Revised: i − µi − π ( i )) and λ

Original: i := λ  1 − iS − is the decaying arrival rate.
Revised: i := λ (cid:16) 1 − iS − 1 (cid:17) is the decaying arrival rate.

Original: We have replaced the constant arrival rate λ by a decaying arrival rate
Revised: We have replaced the constant arrival rate λ by a decaying arrival rate

Original: λ i because we want to learn an optimal policy that does not exploit the buffer size S − 1 ; see [1] for further details.
Revised: λ i because we want to learn an optimal policy that does not exploit the buffer size S − 1 ; see [1] for further details.

Original: For conciseness, Figure 1 displays the transition diagram of the
Revised: For conciseness, Figure 1 displays the transition diagram of the

Original: Markov chain induced by policy π .
Revised: Markov chain induced by policy π .

Original: [Equation]
Revised: [Equation]

Original: [Figure] Finally, the reward is a combination of C , the constant cost due to a departing job missing its deadline and w ( a ) , an arbitrary convex function of a , giving the energy cost for using speed a .
Revised: Finally, the reward is a combination of C , the constant cost due to a departing job missing its deadline and w ( a ) , an arbitrary convex function of a , giving the energy cost for using speed a .

Original: The immediate cost c ( s, a ) in state s under action a is a random variable whose value is w ( a ) + C with probability iµ/U (missed deadline) and w ( a ) otherwise.
Revised: The immediate cost c ( s, a ) in state s under action a is a random variable whose value is w ( a ) + C with probability iµ/U (missed deadline) and w ( a ) otherwise.

Original: To keep in line with the use of rewards instead of costs, we introduce a bound on the cost, r max :=
Revised: To keep in line with the use of rewards instead of costs, we introduce a bound on the cost, r max :=

Original: C + w ( A max ) so that the reward in state s under action a is a positive and bounded random variable given by 
Revised: C + w ( A max ) so that the reward in state s under action a is a positive and bounded random variable given by r ( s, a ) :

Original: [Equation]
Revised: = r max − c ( s, a ) .

Original: As in Section 2.1, ρ ∗ ( M ) is the optimal average cost and ρ ( M, π ) is the average cost induced by policy π , where π belongs to the set of deterministic and stationary policies Π .
Revised: As in Section 2.1, ρ ∗ ( M ) is the optimal average cost and ρ ( M, π ) is the average cost induced by policy π , where π belongs to the set of deterministic and stationary policies Π .

Original: Since the underlying
Revised: Since the underlying

Original: Markov chain induced by any policy is ergodic, we observe that
Revised: Markov chain induced by any policy is ergodic, we observe that

Original: [Equation]
Revised: [Equation]

Original: where m π is the stationary measure under policy π .
Revised: where m π is the stationary measure under policy π .

Original: In [1], it has been shown that the optimal policy is unique and will be denoted by π ∗ .
Revised: In [1], it has been shown that the optimal policy is unique and will be denoted by π ∗ .

Original: 3.1 Properties of M
Revised: 3.1 Properties of M

Original: In the following, we will use the “reference” (or bounding) policy π 0 ( s ) = 0 for all s ∈ S , which thus assigns speed 0 to all states.
Revised: In the following, we will use the “reference” (or bounding) policy π 0 ( s ) = 0 for all s ∈ S , which thus assigns speed 0 to all states.

Original: This policy provides a stochastic bound on all policies in the following sense.
Revised: This policy provides a stochastic bound on all policies in the following sense.

Original: Let s π t be the state under policy π and let ≤ st denote the stochastic order [18]; given two random variables X and Y on R + , we recall that X ≤ st Y if P ( X ≥ s ) ≤ P ( Y ≥ s ) for all s .
Revised: Let s πt be the state under policy π and let ≤ st denote the stochastic order [18]; given two random variables X and Y on R + , we recall that X ≤ st Y if P ( X ≥ s ) ≤ P ( Y ≥ s ) for all s .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Consider an MDP in M .
Revised: Consider an MDP in M .

Original: For all t and policy π ∈ Π , s π t ≤ st s π 0 t , provided that s π 0 ≤ st s π 0 0 .
Revised: For all t and policy π ∈ Π , s πt ≤ st s π 0 t , provided that s π 0 ≤ st s π 0 0 .

Original: Proof.
Revised: Proof.

Original: (sketch)
Revised: (sketch)

Original: The proof follows by a simple coupling argument between the two policies.
Revised: The proof follows by a simple coupling argument between the two policies.

Original: Roughly speaking, each time the Markov chain under π decreases from s to s − 1 because of the speed π ( s ) , it stays in state s under policy π 0 .
Revised: Roughly speaking, each time the Markov chain under π decreases from s to s − 1 because of the speed π ( s ) , it stays in state s under policy π 0 .

Original: Therefore, P ( s π t ≥ s ) ≤ P ( s π 0 t ≥ s ) for all s and t , which also implies that the respective stationary measures are comparable, i.e., 
Revised: Therefore, P ( s πt ≥ s ) ≤ P ( s π 0 t ≥ s ) for all s and t , which also implies that the respective stationary measures are comparable, i.e., (cid:80) S − 1 i = s m πi ≤ (cid:80) S − 1 i = s m π

Original: [Equation]
Revised: 0 i .

Original: Let us now consider H ( s ) , the bias at state s of the optimal policy π ∗ , de ﬁ ned by
Revised: Let us now consider H ( s ) , the bias at state s of the optimal policy π ∗ , defined by

Original: [Equation]
Revised: [Equation]

Revised: S − 1 , (10)

Original: Let also ∂ H ( s ) :
Revised: Let also ∂H ( s ) :

Original: =
Revised: =

Original: H ( s ) − H ( s − 1) be the local variation of the bias.
Revised: H ( s ) − H ( s − 1) be the local variation of the bias.

Original: The following result was shown in [1, Lemma 3.8].
Revised: The following result was shown in [1, Lemma 3.8].

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: The local variation of the bias, ∂ H ( s ) , is negative, decreasing in s , and bounded: 
Revised: The local variation of the bias, ∂H ( s ) , is negative, decreasing in s , and bounded: − ∂H ( s )

Original: − ∂ H ( s ) ≤ ∆ ( s ) with 0 < ∆ ( s ) ≤ C for all 1 ≤ s ≤ S − 1 .
Revised: ≤ ∆( s ) with 0 < ∆( s ) ≤ C for all 1 ≤ s ≤ S − 1 .

Original: Both m π 0 and ∆ will play a major role in our analysis of the regret.
Revised: Both m π 0 and ∆ will play a major role in our analysis of the regret.

Original: 3.2 Applying U CRL 2 in M
Revised: 3.2 Applying U CRL 2 in M

Original: We assume that the bounds λ max and µ max are ﬁ xed so that r max is known to the learner.
Revised: We assume that the bounds λ max and µ max are fixed so that r max is known to the learner.

Original: This is a classical assumption, often replaced by assuming that rewards live in [0 , 1] .
Revised: This is a classical assumption, often replaced by assuming that rewards live in [0 , 1] .

Original: In the remainder, we will apply U CRL 2 over an MDP in M with a change in the con ﬁ dence bounds to take into account the support of P .
Revised: In the remainder, we will apply U CRL 2 over an MDP in M with a change in the confidence bounds to take into account the support of P .

Original: The con ﬁ dence bounds in (4) (resp.
Revised: The confidence bounds in (4) (resp.

Original: (5)) are replaced by r max [Equation] [Equation] s
Revised: (5)) are replaced by r max (cid:114)

Original: [Equation]
Revised: [Equation]

Original: We also impose that the con ﬁ dence set M k only contains matrices with the same support as P .
Revised: We also impose that the confidence set M k only contains matrices with the same support as P .

Original: Removing S in the con ﬁ dence bounds does help to reduce the regret.
Revised: Removing S in the confidence bounds does help to reduce the regret.

Original: However, by using existing analysis, this only removes a factor √ S in the regret bound (for example, see [4]).
Revised: However, by using existing analysis, this only removes a factor √ S in the regret bound (for example, see [4]).

Original: Finally, note that U CRL 2 does not bene ﬁ t from the parametric nature of the MDPs in M , which is essentially de ﬁ ned by three parameters ( λ , µ and C ) and the real convex function w ( · ) .
Revised: Finally, note that U CRL 2 does not benefit from the parametric nature of the MDPs in M , which is essentially defined by three parameters ( λ , µ and C ) and the real convex function w ( · ) .

Original:  Regret of U CRL 2 on M
Revised:  Regret of U CRL 2 on M

Original: Our objective is to develop an upper bound on the regret of the learning algorithm
Revised: Our objective is to develop an upper bound on the regret of the learning algorithm

Original: U CRL 2 when applied to MDPs in our class M .
Revised: U CRL 2 when applied to MDPs in our class M .

Original: The driving idea is to construct a bound that exploits the structure of the stationary measure of all policies, as they all make some states hard to reach, and to control the number of visits to these states to get a new type of bound.
Revised: The driving idea is to construct a bound that exploits the structure of the stationary measure of all policies, as they all make some states hard to reach, and to control the number of visits to these states to get a new type of bound.

Original: 4.1 Main Result
Revised: 4.1 Main Result

Original: The following theorem gives an upper bound on the regret that does not depend on the classical parameters such as the size of the state space nor on global quantities such as the diameter of the
Revised: The following theorem gives an upper bound on the regret that does not depend on the classical parameters such as the size of the state space nor on global quantities such as the diameter of the

Original: MDP nor the span of the bias of some policy.
Revised: MDP nor the span of the bias of some policy.

Original: Instead, the regret bound below mainly depends on the weighted second moment of the stationary measure of the reference policy π 0 , which is bounded independently of the size of the state space.
Revised: Instead, the regret bound below mainly depends on the weighted second moment of the stationary measure of the reference policy π 0 , which is bounded independently of the size of the state space.

Original: We consider the policy π max such that π max ( s ) =
Revised: We consider the policy π max such that π max ( s ) =

Original: A max for all s and m max its stationary measure.
Revised: A max for all s and m max its stationary measure.

Original: Let us also recall that m π 0 is the stationary measure of the Markov chain under policy π 0 ( s ) =
Revised: Let us also recall that m π 0 is the stationary measure of the Markov chain under policy π 0 ( s ) =

Original: for all s and that ∆ : S → R + is a function bounding the local variations of the optimal bias.
Revised: for all s and that ∆ : S → R + is a function bounding the local variations of the optimal bias.

Original: Let
Revised: Let

Original: E [Equation]
Revised: E 2 :=

Original:  with f : s 7 → max { 1 ,s ( s − 1) } ( ∆ ( s )+ r max ) 2 and F := P s ∈ S f ( s ) − 1 .
Revised: F E m π 0 (cid:2) (∆ + r max ) 2 · f (cid:3) with f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max ) 2 and F := (cid:80) s ∈S f ( s ) − 1 .

Original: Here, Eis closely related to the second moment of the measure m π 0 weighted by the bias variations and the maximal reward.
Revised: Here, Eis closely related to the second moment of the measure m π 0 weighted by the bias variations and the maximal reward.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let M ∈ M .
Revised: Let M ∈ M .

Original: De ﬁ ne Q max := 
Revised: Define Q max := (cid:16) [Figure] [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Original: Here, E2 ≤ 12 r 2max
Revised: E

Original: [Equation]
Revised: [Equation]

Original:  , so that the regret satis ﬁ es
Revised: Here, E 2 ≤ 12 r 2max (cid:16) 1 + λ 2 µ 2 (cid:17) , so that the regret satisfies

Original: [Equation]
Revised: [Equation]

Original: Before giving a sketch of the proof, let us comment on the bound (11).
Revised: Before giving a sketch of the proof, let us comment on the bound (11).

Original: Although the ﬁ rst term is of order √ T with a multiplicative constant independent of S - as desired - the second term, of order
Revised: Although the first term is of order √ T with a multiplicative constant independent of S - as desired - the second term, of order

Original: T 1 / 4 , contains very large terms.
Revised: T 1 / 4 , contains very large terms.

Original: Its interest however, lies in the novel approach used in the proof that uses the stationary behavior of the algorithm.
Revised: Its interest however, lies in the novel approach used in the proof that uses the stationary behavior of the algorithm.

Original: 4.2 Comparison with Other Bounds
Revised: 4.2 Comparison with Other Bounds

Original: Let us compare our upper bound with the ones existing in the literature, as we claim that ours is of a different nature.
Revised: Let us compare our upper bound with the ones existing in the literature, as we claim that ours is of a different nature.

Original: First, let us compare with the bound given in [11], which states that with probability 1 − δ ,
Revised: First, let us compare with the bound given in [11], which states that with probability 1 − δ ,

Original: Reg( M, [Equation] for any T > 1 .
Revised: Reg( M, T ) ≤ 34 DS (cid:113) AT log (cid:0) Tδ (cid:1) for any T > 1 .

Original: For any M ∈ M , the diameter grows as
Revised: For any M ∈ M , the diameter grows as

Original: S S (see Appendix B.3), thus this bound is very loose here.
Revised: S S (see Appendix B.3), thus this bound is very loose here.

Original: More recent works have improved this bound by replacing the term D by the local diameter of the MDP [5].
Revised: More recent works have improved this bound by replacing the term D by the local diameter of the MDP [5].

Original: In Appendix B.3, we show that the local diameter grows again as S S for any M ∈ M , and thus these results do not yield signi ﬁ cant improvements.
Revised: In Appendix B.3, we show that the local diameter grows again as S S for any M ∈ M , and thus these results do not yield significant improvements.

Original: Other papers show that the diameter can be replaced by the span of the bias, see [7, 25].
Revised: Other papers show that the diameter can be replaced by the span of the bias, see [7, 25].

Original: This has a big impact because the span of the bias, for any M ∈ M , is linear in S (instead of S S for the diameter); see Appendix B.3.
Revised: This has a big impact because the span of the bias, for any M ∈ M , is linear in S (instead of S S for the diameter); see Appendix B.3.

Original: However, this is still not as good as the bound given in Theorem 4.1, which is independent of S .
Revised: However, this is still not as good as the bound given in Theorem 4.1, which is independent of S .

Original: Now, let us compare with existing bounds for parametric MDPs, as mentioned in the introduction.
Revised: Now, let us compare with existing bounds for parametric MDPs, as mentioned in the introduction.

Original: The d -linear additive model, d < S , introduced in [12] assumes that P ( ·| s, a ) =
Revised: The d -linear additive model, d < S , introduced in [12] assumes that P ( ·| s, a ) =

Original: ⟨ ϕ ( s, a ) , θ ( · ) ⟩ , where ϕ ( s, a ) is a known feature mapping and θ is an unknown measure on R d .
Revised: ⟨ ϕ ( s, a ) , θ ( · ) ⟩ , where ϕ ( s, a ) is a known feature mapping and θ is an unknown measure on R d .

Original: This form of P ( ·| s, a ) implies that the transition kernel is of rank d .
Revised: This form of P ( ·| s, a ) implies that the transition kernel is of rank d .

Original: Unfortunately, this property does not hold true in birth and death processes.
Revised: Unfortunately, this property does not hold true in birth and death processes.

Original: In fact, the kernel of any M ∈ M has almost full rank under all policies.
Revised: In fact, the kernel of any M ∈ M has almost full rank under all policies.

Original: The linear mixture model introduced in [26] assumes instead that P ( s ′ | s, a ) = ⟨ ϕ ( s ′ | s, a ) , θ ⟩ , θ ∈ R d .
Revised: The linear mixture model introduced in [26] assumes instead that P ( s ′ | s, a ) = ⟨ ϕ ( s ′ | s, a ) , θ ⟩ , θ ∈ R d .

Original: This is more adapted to our case, which can be (almost) seen as a linear mixture model of dimension d = 3 .
Revised: This is more adapted to our case, which can be (almost) seen as a linear mixture model of dimension d = 3 .

Original: The bound on the discounted regret of the algorithm proposed in [26] is
Revised: The bound on the discounted regret of the algorithm proposed in [26] is

Original: Reg( M, T ) ≤ d √ T/ (1 − γ ) 2 where γ is a discount factor.
Revised: Reg( M, T ) ≤ d √ T/ (1 − γ ) 2 where γ is a discount factor.

Original: In contrast to our work, this regret analysis holds for discounted problems, where we remark that both the diameter and the span are irrelevant.
Revised: In contrast to our work, this regret analysis holds for discounted problems, where we remark that both the diameter and the span are irrelevant.

Original: On the other hand, both are replaced by a term of the form (1 − γ ) − 2 , which implies that the previous bound grows to in ﬁ nity as γ ↑ 1 .
Revised: On the other hand, both are replaced by a term of the form (1 − γ ) − 2 , which implies that the previous bound grows to infinity as γ ↑ 1 .

Original: More recently, a regret bound of O ( d √ DT ) has been proven in [23] in the undiscounted case, that is the case considered in our work.
Revised: More recently, a regret bound of O ( d √ DT ) has been proven in [23] in the undiscounted case, that is the case considered in our work.

Original: However, the algorithm presented in that reference highly depends on the diameter and is unsuitable for MDPs with a birth and death structure.
Revised: However, the algorithm presented in that reference highly depends on the diameter and is unsuitable for MDPs with a birth and death structure.

Original: Finally, our bound depends on the second moment of the stationary measure of a reference policy, i.e., E 2 , which can be bounded independently of the state space size.
Revised: Finally, our bound depends on the second moment of the stationary measure of a reference policy, i.e., E 2 , which can be bounded independently of the state space size.

Original: This is structurally different from the ones existing in the literature.
Revised: This is structurally different from the ones existing in the literature.

Original: We believe that this structure holds as well in a class of
Revised: We believe that this structure holds as well in a class of

Original: MDPs much larger than M .
Revised: MDPs much larger than M .

Original: In particular, if m is the stationary measure of some bounding/reference policy, and if the critical quantity E m [ ∆ · f ] is small for a well chosen function f , then the regret of a learning algorithm navigating the MDP should also be small.
Revised: In particular, if m is the stationary measure of some bounding/reference policy, and if the critical quantity E m [∆ · f ] is small for a well chosen function f , then the regret of a learning algorithm navigating the MDP should also be small.

Original: A deeper analysis is left as future work.
Revised: A deeper analysis is left as future work.

Original: 4.3 Sketch of the Proof
Revised: 4.3 Sketch of the Proof

Original: Our proof for Theorem 4.1 is technical and is provided in the supplementary material.
Revised: Our proof for Theorem 4.1 is technical and is provided in the supplementary material.

Original: In this section, we present the main ideas and its general structure.
Revised: In this section, we present the main ideas and its general structure.

Original: It initially relies on the regret analysis of U CRLdeveloped in [11], and the differences are highlighted below.
Revised: It initially relies on the regret analysis of U CRLdeveloped in [11], and the differences are highlighted below.

Original: First, we consider the mean rewards and split the regret into episodes to separately treat the cases where the true MDP is in the con ﬁ dence set of optimistic MDPs M k or not.
Revised: First, we consider the mean rewards and split the regret into episodes to separately treat the cases where the true MDP is in the confidence set of optimistic MDPs M k or not.

Original: Thus, let R k := 
Revised: Thus, let R k := (cid:80) s,a ν k ( s, a )(

Original: P s,a ν k ( s, a )( ρ ∗ − r ( s, a )) denote the regret in episode k .
Revised: ρ ∗ − r ( s, a )) denote the regret in episode k .

Original: This split can be written:
Revised: This split can be written:

Original: [Equation]
Revised: [Equation]

Original: where R in := 
Revised: where R in := (cid:80) k

Original: P k R k �
Revised: R k 1 M ∈M k

Original: M ∈ M k and R out := 
Revised: and R out := (cid:80)

Original: P k R k � M / ∈ M k .
Revised: k R k 1 M / ∈M k .

Original: To control R out , we use, as in [11], the stopping criterion and the con ﬁ dence bounds.
Revised: To control R out , we use, as in [11], the stopping criterion and the confidence bounds.

Original: This gives E [ R out ] ≤ r max S , so that the regret due to episodes where the con ﬁ dence regions fail will be negligible next to the main terms.
Revised: This gives E [ R out ] ≤ r max S , so that the regret due to episodes where the confidence regions fail will be negligible next to the main terms.

Original: Then, when the true MDP belongs to the con ﬁ dence region, we use the properties of Extended Value Iteration (EVI) to decompose R in into
Revised: Then, when the true MDP belongs to the confidence region, we use the properties of Extended Value Iteration (EVI) to decompose R in into

Original: X k,s,a
Revised: (cid:88) k

Original: [Equation]
Revised: [Equation]

Original: where v k is the vector of the state-action counts ν k ’s,
Revised: where v k is the vector of the state-action counts ν k ’s,

Original: ˜ P k and ˜h k are respectively the transition matrix and the bias in ˜ M k under policy ˜ π k , and d k is the pro ﬁ le difference between the last step of EVI and the bias (see Appendix A.3).
Revised: ˜ P k and ˜h k are respectively the transition matrix and the bias in ˜ M k under policy ˜ π k , and d k is the profile difference between the last step of EVI and the bias (see Appendix A.3).

Original: We now show how to handle R rewards , R EVI , R bias .
Revised: We now show how to handle R rewards , R EVI , R bias .

Original: First, we deal with the terms that do not involve the bias.
Revised: First, we deal with the terms that do not involve the bias.

Original: Using the con ﬁ dence bound on the rewards (see Appendix A.3.1:
Revised: Using the confidence bound on the rewards (see Appendix A.3.1:

Original: [Equation]
Revised: [Equation]

Original: Now, let us consider R EVI .
Revised: Now, let us consider R EVI .

Original: Since d k becomes arbitrarily small after enough iterations of EVI (see
Revised: Since d k becomes arbitrarily small after enough iterations of EVI (see

Original: Appendix A.1), for T ≥ e 8 2 AT , we get
Revised: Appendix A.1), for T ≥ e 8 2 AT , we get

Original: [Equation]
Revised: [Equation]

Original:  The analysis of R bias is different from the one in [11]: While in [11] the bias is directly bounded by the diameter, we can use the variations of the bias to control the regret more precisely.
Revised: [Equation] The analysis of R bias is different from the one in [11]: While in [11] the bias is directly bounded by the diameter, we can use the variations of the bias to control the regret more precisely.

Original: Using P k , i.e., the transitions in the true MDP under ˜ π k , R bias is further decomposed into:
Revised: Using P k , i.e., the transitions in the true MDP under ˜ π k , R bias is further decomposed into:

Original: [Equation]
Revised: [Equation]

Original: The term R ep can be treated in a similar manner as in [11] by bounding the bias terms with the diameter to apply an Azuma-Hoeffding inequality (see Appendix A.3.5).
Revised: The term R ep can be treated in a similar manner as in [11] by bounding the bias terms with the diameter to apply an Azuma-Hoeffding inequality (see Appendix A.3.5).

Original: Here, we obtain
Revised: Here, we obtain

Original: [Equation]
Revised: [Equation]

Original: Next, we show in A.3.2 that R di ﬀ does not contribute to the main term of the regret.
Revised: Next, we show in A.3.2 that R diff does not contribute to the main term of the regret.

Original: This is one of the hard point in our proof.
Revised: This is one of the hard point in our proof.

Original: First, linear algebra techniques are used to bound ||
Revised: First, linear algebra techniques are used to bound ||

Original: ˜ h
Revised: ˜ h

Original: k − h ∗ || ∞ by
Revised: k − h ∗ || ∞ by

Original: D (2 r max D ||
Revised: D (2 r max D ||

Original: ˜
Revised: ˜

Original: P k − P ∗ || ∞
Revised: P k − P ∗ || ∞

Original: + ||
Revised: + ||

Original: ˜
Revised: ˜

Original: r k − r ∗ || ∞ ) .
Revised: r k − r ∗ || ∞ ) .

Original: Each norm is then bounded using Hoeffding inequality.
Revised: Each norm is then bounded using Hoeffding inequality.

Original: This introduces the special quantity N t k ( x k , π k ( x k )) that yields to the worst con ﬁ dence bound in episode k .
Revised: This introduces the special quantity N t k ( x k , π k ( x k )) that yields to the worst confidence bound in episode k .

Original: Then, an adaptation of McDiarmid’s inequality to Markov chains is used to show that
Revised: Then, an adaptation of McDiarmid’s inequality to Markov chains is used to show that

Original: N t k ( x k , π k ( x k )) ≥ ( t k +1 − t k ) m max ( S − 1) / 2 with high probability, where m max ( S − 1) is the stationary measure of state S − 1 under the uniform policy π max ( s )
Revised: N t k ( x k , π k ( x k )) ≥ ( t k +1 − t k ) m max ( S − 1) / 2 with high probability, where m max ( S − 1) is the stationary measure of state S − 1 under the uniform policy π max ( s )

Original: =
Revised: =

Original: A max .
Revised: A max .

Original: This eventually implies that
Revised: This eventually implies that

Original: [Equation]
Revised: [Equation]

Original: where Q max := 
Revised: where Q max := (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Then, to deal with the main term R trans , we exploit the optimal bias.
Revised: Then, to deal with the main term R trans , we exploit the optimal bias.

Original: The unit vector being in the kernel of ˜ P k − P k , we can rewrite:
Revised: The unit vector being in the kernel of ˜ P k − P k , we can rewrite:

Original: [Equation]
Revised: [Equation]

Original: and, thus, using the con ﬁ dence bound and the bounded variations of the bias,
Revised: and, thus, using the confidence bound and the bounded variations of the bias,

Original: [Equation]
Revised: [Equation]

Original: We can now aggregate R trans , R rewards and R EVI to compute the main term of the regret (see
Revised: We can now aggregate R trans , R rewards and R EVI to compute the main term of the regret (see

Original: Appendix A.3.4).
Revised: Appendix A.3.4).

Original: Here, the key ingredient is to bound
Revised: Here, the key ingredient is to bound

Original: [Equation]
Revised: [Equation]

Original: independently of S .
Revised: independently of S .

Original: This is the second main difference with [11].
Revised: This is the second main difference with [11].

Original: Instead of exploring the MDP uniformly, we know that the algorithm will mostly visit the ﬁ rst states of the MDP, regardless of the chosen policy.
Revised: Instead of exploring the MDP uniformly, we know that the algorithm will mostly visit the first states of the MDP, regardless of the chosen policy.

Original: As shown in [11], for a ﬁ xed state s :
Revised: As shown in [11], for a fixed state s :

Original: [Equation]
Revised: [Equation]

Original: Now, instead of summing over the states, we can use properties of stochastic ordering to compare the mean number of visits of a state with the probability measure m π 0 ; here, we strongly rely on the birth and death structure of the MDPs in M .
Revised: Now, instead of summing over the states, we can use properties of stochastic ordering to compare the mean number of visits of a state with the probability measure m π 0 ; here, we strongly rely on the birth and death structure of the MDPs in M .

Original: For any non-negative non-decreasing function f : S → R + , we obtain
Revised: For any non-negative non-decreasing function f : S → R + , we obtain

Original: [Equation]
Revised: [Equation]

Original: Let us choose f : s 7 → max { 1 ,s ( s − 1) } ( ∆ ( s )+ r max ) 2 and let F := P s f ( s ) − 1 ≤ 3( C + r max ) 2 .
Revised: Let us choose f : s (cid:55)→ max { 1 ,s ( s − 1) } (∆( s )+ r max ) 2 and let F := (cid:80) s f ( s ) − 1 ≤ 3( C + r max ) 2 .

Original: Let also
Revised: Let also

Original: E 2 :=
Revised: E 2 :=

Original: F E m π [Equation]  .
Revised: F E m π 0 (cid:2) (∆ + r max ) 2 · f (cid:3) .

Original: Then,
Revised: Then,

Original: E

Original: [Equation]
Revised: [Equation]

Original: In Appendix A.3.4, we further show that E 2 ≤ 3( C + r max ) 1 + λ 2 µ .
Revised: In Appendix A.3.4, we further show that E 2 ≤ 3( C + r max ) 2 (cid:16) 1 + λ 2 µ 2 (cid:17) .

Original: Therefore, for the three main terms, we obtain
Revised: Therefore, for the three main terms, we obtain

Original: [Equation]
Revised: [Equation]

Original: and we conclude  these terms.
Revised: and we conclude our proof by combining all of these terms.

Original:  Conclusions
Revised:  Conclusions

Original: For learning in a class of birth and death processes, we have shown that exploiting the stationary measure in the analysis of classical learning algorithms yields a K √ T regret, where K only depends on the stationary measure of the system under a well chosen policy.
Revised: For learning in a class of birth and death processes, we have shown that exploiting the stationary measure in the analysis of classical learning algorithms yields a K √ T regret, where K only depends on the stationary measure of the system under a well chosen policy.

Original: Thus, the dependence on the size of the state space as well as on the diameter of the MDP or its span disappears.
Revised: Thus, the dependence on the size of the state space as well as on the diameter of the MDP or its span disappears.

Original: We believe that this type of results can be generalized to other cases such as optimal routing, admission control and allocation problems in queuing systems, as the stationary distribution under all policies is uneven between the states.
Revised: We believe that this type of results can be generalized to other cases such as optimal routing, admission control and allocation problems in queuing systems, as the stationary distribution under all policies is uneven between the states.