Original: Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regret
Revised: Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regret

Original: Abstract
Revised: Abstract

Original: We propose a new learning framework that captures the tiered structure of many real-world user-interaction applications, where the users can be divided into two groups based on their different tolerance on exploration risks and should be treated separately.
Revised: We propose a new learning framework that captures the tiered structure of many real-world user-interaction applications, where the users can be divided into two groups based on their different tolerance on exploration risks and should be treated separately.

Original: In this setting, we simultaneously maintain two policies π O and π E : π O (“O” for “online”) interacts with more risk-tolerant users from the first tier and minimizes regret by balancing exploration and exploitation as usual, while π E (“E” for “exploit”) exclusively focuses on exploitation for risk-averse users from the second tier utilizing the data collected so far.
Revised: In this setting, we simultaneously maintain two policies π O and π E : π O (“O” for “online”) interacts with more risk-tolerant users from the first tier and minimizes regret by balancing exploration and exploitation as usual, while π E (“E” for “exploit”) exclusively focuses on exploitation for risk-averse users from the second tier utilizing the data collected so far.

Original: An important question is whether such a separation yields advantages over the standard online setting (i.e., π E = π O ) for the risk-averse users.
Revised: An important question is whether such a separation yields advantages over the standard online setting (i.e., π E = π O ) for the risk-averse users.

Original: We individually consider the gap-independent vs. gap-dependent settings.
Revised: We individually consider the gap-independent vs. gap-dependent settings.

Original: For the former, we prove that the separation is indeed not beneficial from a minimax perspective.
Revised: For the former, we prove that the separation is indeed not beneficial from a minimax perspective.

Original: For the latter, we show that if choosing Pessimistic Value
Revised: For the latter, we show that if choosing Pessimistic Value

Original: Iteration as the exploitation algorithm to produce π E , we can achieve a constant regret for risk-averse users independent of the number of episodes K , which is in sharp contrast to the Ω(log K ) regret for any online RL algorithms in the same setting, while the regret of π O (almost) maintains its online regret optimality and does not need to compromise for the success of π E .
Revised: Iteration as the exploitation algorithm to produce π E , we can achieve a constant regret for risk-averse users independent of the number of episodes K , which is in sharp contrast to the Ω(log K ) regret for any online RL algorithms in the same setting, while the regret of π O (almost) maintains its online regret optimality and does not need to compromise for the success of π E .

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) has been applied to many real-world user-interaction applications to provide users with better services, such as in recommendation systems [Afsar et al., 2021] and medical treatment [Yu et al., 2021, Lipsky and Sharp, 2001].
Revised: Reinforcement learning (RL) has been applied to many real-world user-interaction applications to provide users with better services, such as in recommendation systems [Afsar et al., 2021] and medical treatment [Yu et al., 2021, Lipsky and Sharp, 2001].

Original: In those scenarios, the users take the role of the environments and the interaction strategies (e g. recommendation or medical treatment) correspond to the agents in RL.
Revised: In those scenarios, the users take the role of the environments and the interaction strategies (e g. recommendation or medical treatment) correspond to the agents in RL.

Original: In the theoretical study of such problems, most of the existing literature adopts the online interaction protocol, where in each episode k ∈
Revised: In the theoretical study of such problems, most of the existing literature adopts the online interaction protocol, where in each episode k ∈

Original: [ K ] , the learning agent executes a policy π k to interact with users (i.e. environments), receives new data to update the policy, and moves on to the next episode.
Revised: [ K ] , the learning agent executes a policy π k to interact with users (i.e. environments), receives new data to update the policy, and moves on to the next episode.

Original: While this formulation treats each user equivalently when optimizing the regret, many scenarios have a special “ Tiered Structure ” 2 : users can be divided into multiple groups depending on their different preference and tolerance about the risk that results from the necessary exploration to improve the policy , and such grouping is available to the learner in advance so it would be better to treat them separately.
Revised: While this formulation treats each user equivalently when optimizing the regret, many scenarios have a special “ Tiered Structure ” 2 : users can be divided into multiple groups depending on their different preference and tolerance about the risk that results from the necessary exploration to improve the policy , and such grouping is available to the learner in advance so it would be better to treat them separately.

Original: As a concrete example, in medical treatment, after a new treatment plan comes out, some courageous patients or paid volunteers (denoted as G O ; “O” for “Online”) may prefer it given the potential risks, while some conservative patients (denoted as G E ; “E” for “Exploit”) may tend to receive mature and well-tested plans, even if the new one is promising to be more effective.
Revised: As a concrete example, in medical treatment, after a new treatment plan comes out, some courageous patients or paid volunteers (denoted as G O ; “O” for “Online”) may prefer it given the potential risks, while some conservative patients (denoted as G E ; “E” for “Exploit”) may tend to receive mature and well-tested plans, even if the new one is promising to be more effective.

Original: As another example, companies offering recommendation services may recruit paid testers or use bonus to attract customers (G O ) to interact with the system to shoulder the majority of the exploration risk during policy improvement, which may result in better service (low regret) for the remaining customers (G E ).
Revised: As another example, companies offering recommendation services may recruit paid testers or use bonus to attract customers (G O ) to interact with the system to shoulder the majority of the exploration risk during policy improvement, which may result in better service (low regret) for the remaining customers (G E ).

Original: Moreover, many online platforms have free service open for everyone (G O ), while some users are willing to pay for enhanced service (G E ).
Revised: Moreover, many online platforms have free service open for everyone (G O ), while some users are willing to pay for enhanced service (G E ).

Original: If we follow the traditional online setting and treat the users in these two groups equivalently, then in expectation each group will suffer the same regret and risk.
Revised: If we follow the traditional online setting and treat the users in these two groups equivalently, then in expectation each group will suffer the same regret and risk.

Original: In contrast, if we leverage the group information by using policies with different risk levels to interact with different groups, it is potentially possible to transfer some exploration risks from users in G E to G O , while the additional risks suffered by G O will be compensated in other forms (such as payment, the users’ inherent motivation, or the free service itself).
Revised: In contrast, if we leverage the group information by using policies with different risk levels to interact with different groups, it is potentially possible to transfer some exploration risks from users in G E to G O , while the additional risks suffered by G O will be compensated in other forms (such as payment, the users’ inherent motivation, or the free service itself).

Original: To make our objective more clear, we abstract the problem setting into Frw.
Revised: To make our objective more clear, we abstract the problem setting into Frw.

Original: and compare it with the standard online setting in Fig.
Revised: and compare it with the standard online setting in Fig.

Original: 1, where we use Alg O and Alg E to denote the two algorithms producing policies π E and π O to interact with users in G O and G E , respectively.
Revised: 1, where we use Alg O and Alg E to denote the two algorithms producing policies π E and π O to interact with users in G O and G E , respectively.

Original: To enable theoretical analyses, we do adopt a few simplification assumptions while still modelling the core challenges in the aforementioned scenarios: firstly, at each iteration of Frw.
Revised: To enable theoretical analyses, we do adopt a few simplification assumptions while still modelling the core challenges in the aforementioned scenarios: firstly, at each iteration of Frw.

Original: 1, the algorithms will interact with and collect one trajectory from each group, which assumes that users from two groups will come to seek for service in pair with the same frequency.
Revised: 1, the algorithms will interact with and collect one trajectory from each group, which assumes that users from two groups will come to seek for service in pair with the same frequency.

Original: In practice, usually, the users come in random order and the frequencies from different groups are not the same; see Appx.
Revised: In practice, usually, the users come in random order and the frequencies from different groups are not the same; see Appx.

Original: B for why our abstraction is still a valid surrogate and how our results can be generalized.
Revised: B for why our abstraction is still a valid surrogate and how our results can be generalized.

Original: Secondly, for convenience, we only use the samples generated from G O , because Alg E is expected to best exploit the available information and not encouraged to perform intelligent exploration.
Revised: Secondly, for convenience, we only use the samples generated from G O , because Alg E is expected to best exploit the available information and not encouraged to perform intelligent exploration.

Original: Nonetheless, our results hold with minor modifications if one also chooses to use trajectories from G E .
Revised: Nonetheless, our results hold with minor modifications if one also chooses to use trajectories from G E .

Original: Thirdly, we assume that the dynamics and rewards during the interactions with users in different groups are all the same (i.e. Env = Env O = Env E ).
Revised: Thirdly, we assume that the dynamics and rewards during the interactions with users in different groups are all the same (i.e. Env = Env O = Env E ).

Original: It is possible that the users in different tiers can behave differently, and we leave the relaxation of such an assumption to future work.
Revised: It is possible that the users in different tiers can behave differently, and we leave the relaxation of such an assumption to future work.

Original: Framework 1: The Tiered RL Framework
Revised: [Table]

Original: 1 Input: Env O and Env E // Note that Env O =
Revised: 1 Input: Env O and Env E // Note that Env O =

Original: Env E 2 Initialize D 1 ← {

Original: [Equation]

Original: = 1 , [Equation] [Equation] 3 for k = 1 , 2 , ..., K do π O k ← Alg O
Revised: Env E 2 Initialize D 1 ← {} . 3 for k = 1 , 2 , ..., K do π O k ← Alg O

Original: ( D k ) ; π E k ← Alg E ( D k ) .
Revised: ( D k ) ; π E k ← Alg E ( D k ) .

Original: π O k interacts with customers/users/patients in G O
Revised: π O k interacts with customers/users/patients in G O

Original: (i
Revised: (i

Original: e. Env O ), and collect data τ O k .
Revised: e. Env O ), and collect data τ O k .

Original: π E k interacts with customers/users/patients in G E
Revised: π E k interacts with customers/users/patients in G E

Original: (i
Revised: (i

Original: e. Env E ), and collect data τ E k .
Revised: e. Env E ), and collect data τ E k .

Original: D k
Revised: D k

Original: +1 =
Revised: +1 =

Original: D k ∪ { τ O k } .
Revised: D k ∪ { τ O k } .

Original: We do not consider to use τ E k in this paper.
Revised: We do not consider to use τ E k in this paper.

Original: 8 end
Revised: 8 end

Original: Similar to the online setting, we use the expected pseudo-regret to measure the performance of the algorithm, which is formalized in Def. 2.1.
Revised: Similar to the online setting, we use the expected pseudo-regret to measure the performance of the algorithm, which is formalized in Def. 2.1.

Original: The key problem we would like to investigate is provable benefits of leveraging the tiered structure by Frw.
Revised: The key problem we would like to investigate is provable benefits of leveraging the tiered structure by Frw.

Original: 1 comparing with the standard online setting:
Revised: 1 comparing with the standard online setting:

Original: [Equation] Is it possible for Regret ( Alg E ) to be strictly lower than any online learning algorithms in certain scenarios, while keeping Regret ( Alg O ) near-optimal?
Revised: Is it possible for Regret ( Alg E ) to be strictly lower than any online learning algorithms in certain scenarios, while keeping Regret ( Alg O ) near-optimal?

Original: Note that we still expect regret of Alg O to enjoy near-optimal regret guarantees, which is a reasonable requirement as the experience of users in G O also matters in many of our motivating applications.
Revised: Note that we still expect regret of Alg O to enjoy near-optimal regret guarantees, which is a reasonable requirement as the experience of users in G O also matters in many of our motivating applications.

Original: We regard the above problem formulation as our first contribution , which is mainly conceptual.
Revised: We regard the above problem formulation as our first contribution , which is mainly conceptual.

Original: As our second contribution , Sec. 3 shows that Alg E has the same minimax gap-independent lower bound as online learning algorithms.
Revised: As our second contribution , Sec. 3 shows that Alg E has the same minimax gap-independent lower bound as online learning algorithms.

Original: This result reveals the difficulty to leverage tiered structure in standard tabular MDPs, and motivates us to investigate the benefits under the gap-dependent setting, which is frequently considered in the Multi-Armed Bandit (MAB)
Revised: This result reveals the difficulty to leverage tiered structure in standard tabular MDPs, and motivates us to investigate the benefits under the gap-dependent setting, which is frequently considered in the Multi-Armed Bandit (MAB)

Original: [Lattimore and Szepesvári, 2020,
Revised: [Lattimore and Szepesvári, 2020,

Original: Rouyer and Seldin, 2020] and RL literature [Xu et al., 2021, Simchowitz and Jamieson, 2019].
Revised: Rouyer and Seldin, 2020] and RL literature [Xu et al., 2021, Simchowitz and Jamieson, 2019].

Original: As our third contribution and our main technical results, Sec. 4 establishes provable benefits of Frw. 1 by proposing a new algorithmic framework and showing Regret ( Alg E ) is constant and independent of the number of episodes K , which is in sharp contrast with the Ω(log K ) regret lower bound for any algorithms in the standard online setting that do not leverage the tiered structure.
Revised: As our third contribution and our main technical results, Sec. 4 establishes provable benefits of Frw. 1 by proposing a new algorithmic framework and showing Regret ( Alg E ) is constant and independent of the number of episodes K , which is in sharp contrast with the Ω(log K ) regret lower bound for any algorithms in the standard online setting that do not leverage the tiered structure.

Original: Specifically, we use Pessimistic Value Iteration (PVI) as Alg E for exploitation to interact with G E , while Alg O can be arbitrary online algorithms with near-optimal regret.
Revised: Specifically, we use Pessimistic Value Iteration (PVI) as Alg E for exploitation to interact with G E , while Alg O can be arbitrary online algorithms with near-optimal regret.

Original: Concretely, we first study stochastic MABs as a warm-up, where we choose Alg E to be LCB (Lower Confidence Bonus), a degenerated version of PVI in bandits, and choose UCB (Upper Confidence Bonus) as Alg O for a concrete case study.
Revised: Concretely, we first study stochastic MABs as a warm-up, where we choose Alg E to be LCB (Lower Confidence Bonus), a degenerated version of PVI in bandits, and choose UCB (Upper Confidence Bonus) as Alg O for a concrete case study.

Original: We prove that Alg E can achieve constant pseudo-regret (cid:101) O
Revised: We prove that Alg E can achieve constant pseudo-regret (cid:101) O

Original: ( (cid:80) ∆ i > 

Revised: ( (cid:80) ∆ i >

Original: ( A − i ) (cid:16) 1∆ i −
Revised: ( A − i ) (cid:16) 1∆ i −

Original: [Table]
Revised: [Table]

Original: (cid:17) ) with A being the number of actions and ∆ i ’s being the gaps with ∆ 1 ≥ ... ≥ ∆ A − 1 ≥ ∆ A = 0 , while Alg O is near-optimal due to the regret guarantee of UCB.
Revised: (cid:17) ) with A being the number of actions and ∆ i ’s being the gaps with ∆ 1 ≥ ... ≥ ∆ A − 1 ≥ ∆ A = 0 , while Alg O is near-optimal due to the regret guarantee of UCB.

Original: After that, Sec. 4.2 extends the success of PVI to tabular MDPs, and establishes results that apply to a wide range of online algorithms Alg O with near-optimal regret.
Revised: After that, Sec. 4.2 extends the success of PVI to tabular MDPs, and establishes results that apply to a wide range of online algorithms Alg O with near-optimal regret.

Original: Although the benefits of pessimism have been widely recognized in offline RL [Jin et al., 2021], to our knowledge, we are the first to study PVI in a gapdependent online setting.
Revised: Although the benefits of pessimism have been widely recognized in offline RL [Jin et al., 2021], to our knowledge, we are the first to study PVI in a gapdependent online setting.

Original: We also contribute several novel techniques for overcoming the difficulties in achieving constant regret, and defer their summary to Sec. 4.2.
Revised: We also contribute several novel techniques for overcoming the difficulties in achieving constant regret, and defer their summary to Sec. 4.2.

Original: Moreover, in Appx.
Revised: Moreover, in Appx.

Original: H, we report experiment results to demostrate the advantage of leveraging tiered structure as predicted by theory.
Revised: H, we report experiment results to demostrate the advantage of leveraging tiered structure as predicted by theory.

Original: Closest Related Work Due to space limit, we only discuss the closest related work here and defer the rest to Appx.
Revised: Closest Related Work Due to space limit, we only discuss the closest related work here and defer the rest to Appx.

Original: A.
Revised: A.

Original: To our knowledge, there is no previous works on leveraging tiered structure in
Revised: To our knowledge, there is no previous works on leveraging tiered structure in

Original: MDPs.
Revised: MDPs.

Original: In the bandit setting, there is a line of related works studying decoupling exploration and exploitation [Avner et al., 2012, Rouyer and Seldin, 2020], where [Rouyer and Seldin, 2020] studied “best of both worlds” methods and reported a similar constant regret.
Revised: In the bandit setting, there is a line of related works studying decoupling exploration and exploitation [Avner et al., 2012, Rouyer and Seldin, 2020], where [Rouyer and Seldin, 2020] studied “best of both worlds” methods and reported a similar constant regret.

Original: First, in stochastic bandits, there are many cases when our result is tighter than theirs, (see a detailed comparison in Sec. 4.1), and more importantly, our methods can naturally extend to RL (i.e., MDPs), whereas a similar extension of their techniques can run into serious difficulties: they relied on importance sampling to provide unbiased off-policy estimation for policy value, which incurs the infamous “curse of horizon”, a.k.a., a sample complexity exponential in the planning horizon H in long-horizon RL (see examples in
Revised: First, in stochastic bandits, there are many cases when our result is tighter than theirs, (see a detailed comparison in Sec. 4.1), and more importantly, our methods can naturally extend to RL (i.e., MDPs), whereas a similar extension of their techniques can run into serious difficulties: they relied on importance sampling to provide unbiased off-policy estimation for policy value, which incurs the infamous “curse of horizon”, a.k.a., a sample complexity exponential in the planning horizon H in long-horizon RL (see examples in

Original: Sec. 2 in [Liu et al., 2018]).
Revised: Sec. 2 in [Liu et al., 2018]).

Original: Our approach overcomes this difficulty by developing a pessimism-based learning framework, which is fundamentally different from their approach and requires several novel techniques in the analyses.
Revised: Our approach overcomes this difficulty by developing a pessimism-based learning framework, which is fundamentally different from their approach and requires several novel techniques in the analyses.

Original: Second, they did not provide any guarantee for the regret of exploration algorithm, whereas in our results the regret of Alg O can be near-optimal, which we believe is also important as the experience of users in G O also matters in many of our motivating applications.
Revised: Second, they did not provide any guarantee for the regret of exploration algorithm, whereas in our results the regret of Alg O can be near-optimal, which we believe is also important as the experience of users in G O also matters in many of our motivating applications.

Original: Third, their bandit results require a unique best arm, whereas we allow the optimal arms/policies to be non-unique, which can cause non-trivial difficulties in the analyses as we will discuss in Sec.
Revised: Third, their bandit results require a unique best arm, whereas we allow the optimal arms/policies to be non-unique, which can cause non-trivial difficulties in the analyses as we will discuss in Sec.

Original:  Preliminary and Problem formulation
Revised:  Preliminary and Problem formulation

Original: Stochastic Multi-Armed Bandits (MABs)
Revised: Stochastic Multi-Armed Bandits (MABs)

Original: The MAB model consists of a set of arms A = { 1 , 2 , ..., A } .
Revised: The MAB model consists of a set of arms A = { 1 , 2 , ..., A } .

Original: When sampling an arm i ∈ A , the agent observes a random variable r i ∈ [0 , 1] .
Revised: When sampling an arm i ∈ A , the agent observes a random variable r i ∈ [0 , 1] .

Original: We use µ i =
Revised: We use µ i =

Original: E [ r i ] to denote the mean value for arm i for each i ∈ A .
Revised: E [ r i ] to denote the mean value for arm i for each i ∈ A .

Original: We allow the optimal arms to be non-unique.
Revised: We allow the optimal arms to be non-unique.

Original: For simplicity of notation, we assume the arms are ordered such that µ 1 ≤ µ 2 ... ≤ µ A .
Revised: For simplicity of notation, we assume the arms are ordered such that µ 1 ≤ µ 2 ... ≤ µ A .

Original: Finite-Horizon Tabular Markov Decision Processes (MDPs)
Revised: Finite-Horizon Tabular Markov Decision Processes (MDPs)

Original: For the reinforcement learning (RL) setting, we consider the episodic tabular MDPs denoted by M ( S , A , H, P, r ) , where S is the finite state space, A is the finite action space, H is the horizon length, and P =
Revised: For the reinforcement learning (RL) setting, we consider the episodic tabular MDPs denoted by M ( S , A , H, P, r ) , where S is the finite state space, A is the finite action space, H is the horizon length, and P =

Original: { P h } Hh =1 and r = { r h } Hh = are the time-dependent transition and reward functions, respectively.
Revised: { P h } Hh =1 and r = { r h } Hh = are the time-dependent transition and reward functions, respectively.

Original: We assume all steps share the state and action space
Revised: We assume all steps share the state and action space

Original: (i.e. S 1 = S 2 ...
Revised: (i.e. S 1 = S 2 ...

Original: =
Revised: =

Original: S H = S ,
Revised: S H = S ,

Original: A 1 =
Revised: A 1 =

Original: A 2 ...
Revised: A 2 ...

Original: =
Revised: =

Original: A H =
Revised: A H =

Original: A ) while the transition and reward functions can be different.
Revised: A ) while the transition and reward functions can be different.

Original: At the beginning of each episode, the environment will start from a fixed initial state s 1 (w.l.o.g.).
Revised: At the beginning of each episode, the environment will start from a fixed initial state s 1 (w.l.o.g.).

Original: Then, for each time step h ∈
Revised: Then, for each time step h ∈

Original: [ H ] , the agent selects an action a h ∈ A based on the current state s h , receives the reward r h ( s h , a h ) , and observes the system transition to the next state s h +1 , until s H +1 is reached.
Revised: [ H ] , the agent selects an action a h ∈ A based on the current state s h , receives the reward r h ( s h , a h ) , and observes the system transition to the next state s h +1 , until s H +1 is reached.

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , we assume the reward function r is deterministic and our results can be easily extended to handle stochastic rewards.
Revised: , we assume the reward function r is deterministic and our results can be easily extended to handle stochastic rewards.

Original: A time-dependent policy is specified as π = { π 1 , π 2 , ..., π H } with π h : S → ∆( A ) for all h ∈ [ H ] .
Revised: A time-dependent policy is specified as π = { π 1 , π 2 , ..., π H } with π h : S → ∆( A ) for all h ∈ [ H ] .

Original: Here ∆( A ) denotes the probability simplex over the action space.
Revised: Here ∆( A ) denotes the probability simplex over the action space.

Original: With a slight abuse of notation, when π h is a deterministic policy, we use π h : S
Revised: With a slight abuse of notation, when π h is a deterministic policy, we use π h : S

Original: → A to refer to a deterministic mapping.
Revised: → A to refer to a deterministic mapping.

Original: V πh ( s ) and
Revised: V πh ( s ) and

Original: Q πh ( s, a ) denote the value function and Q-function at
Revised: Q πh ( s, a ) denote the value function and Q-function at

Original: step h ∈
Revised: step h ∈

Original: [ H ] , which are defined as: V πh ( s ) =
Revised: [ H ] , which are defined as: V πh ( s ) =

Original: E [ (cid:80) Hh
Revised: E [ (cid:80) Hh

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: ′ , a
Revised: ′ , a

Original: [Equation]
Revised: [Equation]

Original: ) | s h
Revised: ) | s h

Original: = s, π ] ,
Revised: = s, π ] ,

Original: [Equation]
Revised: [Equation]

Original: π ] .
Revised: π ] .

Original: We use V ∗ h ( · ) :
Revised: We use V ∗ h ( · ) :

Original: = max π V πh ( · ) and Q ∗ h ( · , · ) = max π Q πh ( · , · ) to refer to the optimal state/action-value functions, and Π ∗ ( s h ) :
Revised: = max π V πh ( · ) and Q ∗ h ( · , · ) = max π Q πh ( · , · ) to refer to the optimal state/action-value functions, and Π ∗ ( s h ) :

Original: = { a h | Q ∗ ( s h , a h ) =
Revised: = { a h | Q ∗ ( s h , a h ) =

Original: V ∗ h ( s h ) } to denote the collection of all optimal actions at state s h .
Revised: V ∗ h ( s h ) } to denote the collection of all optimal actions at state s h .

Original: With an abuse of notation, we define Π ∗ :
Revised: With an abuse of notation, we define Π ∗ :

Original: = { π : V π 1 ( s 1 ) =
Revised: = { π : V π 1 ( s 1 ) =

Original: V ∗ 1 ( s 1 ) } , i.e., the set of policies that maximize the total expected return.
Revised: V ∗ 1 ( s 1 ) } , i.e., the set of policies that maximize the total expected return.

Original: In this paper, when we say that the MDP has “unique optimal (deterministic) policy”, it is up to the occupancy measure, that is, all policies in Π ∗ share the same state-action occupancy d π
Revised: In this paper, when we say that the MDP has “unique optimal (deterministic) policy”, it is up to the occupancy measure, that is, all policies in Π ∗ share the same state-action occupancy d π

Original: ( s h , a h ) :
Revised: ( s h , a h ) :

Original: =
Revised: =

Original: Pr( S h
Revised: Pr( S h

Original: =
Revised: =

Original: s h , A h = a h | S 1 = s 1 , π ) for all h ∈
Revised: s h , A h = a h | S 1 = s 1 , π ) for all h ∈

Original: [ H ] , s h ∈ S h , a h ∈
Revised: [ H ] , s h ∈ S h , a h ∈

Original: A h .
Revised: A h .

Original: In the following, we use | Π ∗ | = 1 to refer to the case of unique optimal (deterministic) policy, where the cardinality of Π ∗ is counted up to the equivalence of occupancies.
Revised: In the following, we use | Π ∗ | = 1 to refer to the case of unique optimal (deterministic) policy, where the cardinality of Π ∗ is counted up to the equivalence of occupancies.

Original: Besides, for any function V : S → R , we denote P h V ( s h , a h ) :
Revised: Besides, for any function V : S → R , we denote P h V ( s h , a h ) :

Original: =
Revised: =

Original: E s h +1 ∼ P h ( ·| s h ,a h )
Revised: E s h +1 ∼ P h ( ·| s h ,a h )

Original: [ V ( s h +1 )] .
Revised: [ V ( s h +1 )] .

Original: Gap-Dependent Setting We follow the standard formulation of gap-dependent setting in previous bandits [Lattimore and Szepesvári, 2020] and RL literature [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021].
Revised: Gap-Dependent Setting We follow the standard formulation of gap-dependent setting in previous bandits [Lattimore and Szepesvári, 2020] and RL literature [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021].

Original: In bandits, the gap w.r.t.
Revised: In bandits, the gap w.r.t.

Original: arm i is defined as ∆ i := max j ∈
Revised: arm i is defined as ∆ i := max j ∈

Original: [ A ] µ j − µ i , ∀ i ∈
Revised: [ A ] µ j − µ i , ∀ i ∈

Original: [ A ] , and we assume that there exists a strictly positive value ∆ min such that, either ∆ i = 0 or ∆
Revised: [ A ] , and we assume that there exists a strictly positive value ∆ min such that, either ∆ i = 0 or ∆

Original: i ≥ ∆ min .
Revised: i ≥ ∆ min .

Original: For tabular RL setting, we define ∆ h ( s h , a h ) :
Revised: For tabular RL setting, we define ∆ h ( s h , a h ) :

Original: = V ∗ ( s h ) − Q ∗ ( s h , a h ) , ∀ h ∈
Revised: = V ∗ ( s h ) − Q ∗ ( s h , a h ) , ∀ h ∈

Original: [ H ] , s h ∈
Revised: [ H ] , s h ∈

Original: S h , a h ∈
Revised: S h , a h ∈

Original: A h .
Revised: A h .

Original: We use the same notation ∆ min to refer to the minimal gap in tabular setting and assume that either ∆ h ( s h , a h ) = 0 or ∆ h ( s h , a h ) ≥ ∆ min .
Revised: We use the same notation ∆ min to refer to the minimal gap in tabular setting and assume that either ∆ h ( s h , a h ) = 0 or ∆ h ( s h , a h ) ≥ ∆ min .

Original: Performance Measure We use Pseudo-Regret defined below to measure the performance of Alg O and Alg E .
Revised: Performance Measure We use Pseudo-Regret defined below to measure the performance of Alg O and Alg E .

Original: In the following, we will also use “exploitation regret” to refer to Regret K ( Alg E ) .
Revised: In the following, we will also use “exploitation regret” to refer to Regret K ( Alg E ) .

Original: Definition 2.1 (Pseudo-Regret) .
Revised: Definition 2.1 (Pseudo-Regret) .

Original: We define the regret of Alg O and Alg E to be:
Revised: We define the regret of Alg O and Alg E to be:

Original: [Equation]
Revised: [Equation]

Original: where π O k and π E k are generated according to the procedure in Framework 1 and the expectation is taken over the randomness in data generation and algorithms.
Revised: where π O k and π E k are generated according to the procedure in Framework 1 and the expectation is taken over the randomness in data generation and algorithms.

Original:  Lower Bound of Regret ( Alg E ) without Gap Assumption
Revised:  Lower Bound of Regret ( Alg E ) without Gap Assumption

Original: In this section, we show that, in normal tabular RL setting, for arbitrary algorithm pair ( Alg O , Alg E ) , even if we do not constrain Alg O to be near-optimal, the regret of Alg E has the same minimax lower bound as algorithms in online setting.
Revised: In this section, we show that, in normal tabular RL setting, for arbitrary algorithm pair ( Alg O , Alg E ) , even if we do not constrain Alg O to be near-optimal, the regret of Alg E has the same minimax lower bound as algorithms in online setting.

Original: We defer the formal statement and proof to Appendix C.1.
Revised: We defer the formal statement and proof to Appendix C.1.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: [Lower Bound for Alg E without Gap Assumption] There exist positive constants c, ε 0 , δ 0 , such that, for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 2 , K ≥ cε 20 H 3 SA , and arbitrary algorithm pair ( Alg O , Alg E ) , there must exist a hard tabular MDP M hard , E ( Alg O , Alg E ) ,M hard (cid:104)(cid:80)
Revised: [Lower Bound for Alg E without Gap Assumption] There exist positive constants c, ε 0 , δ 0 , such that, for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 2 , K ≥ cε 20 H 3 SA , and arbitrary algorithm pair ( Alg O , Alg E ) , there must exist a hard tabular MDP M hard , E ( Alg O , Alg E ) ,M hard (cid:104)(cid:80)

Original: Kk =1 V ∗ − V π E k (cid:105) ≥ δ 0 √ cH 3 SAK, where the expectation is taken over the randomness of algorithms and MDP.
Revised: Kk =1 V ∗ − V π E k (cid:105) ≥ δ 0 √ cH 3 SAK, where the expectation is taken over the randomness of algorithms and MDP.

Original: The theorem above is stating that, comparing with the regret lower bound for online algorithms (cid:101) O ( √ H 3 SAK ) in Theorem 9 of Domingues et al.
Revised: The theorem above is stating that, comparing with the regret lower bound for online algorithms (cid:101) O ( √ H 3 SAK ) in Theorem 9 of Domingues et al.

Original: [2021], the exploitation algorithm cannot reduce the dependence on any of parameters H, S, A, K in hard MDPs, even if we allow Alg O to sacrifice its performance to gather the best possible data for Alg E .
Revised: [2021], the exploitation algorithm cannot reduce the dependence on any of parameters H, S, A, K in hard MDPs, even if we allow Alg O to sacrifice its performance to gather the best possible data for Alg E .

Original: Also, the lower bound would still hold even if we allow both Alg O and Alg E to additionally use the data τ E generated by π E .
Revised: Also, the lower bound would still hold even if we allow both Alg O and Alg E to additionally use the data τ E generated by π E .

Original: This negative result implies that without any further assumptions, the separation is not beneficial from a minimax optimality perspective, and we can simply choose both Alg E and Alg O to be the same near-optimal online algorithm as without worrying about separating them.
Revised: This negative result implies that without any further assumptions, the separation is not beneficial from a minimax optimality perspective, and we can simply choose both Alg E and Alg O to be the same near-optimal online algorithm as without worrying about separating them.

Original: However, in the next section, we will show that, in tabular MDPs with strictly positive gaps, in contrast with the Ω(log K ) lower bound for online algorithms, we can have Alg E such that its regret is constant and independent on the number of time horizon K , which reveals the fundamental differences between the pure online setting and the Tiered RL setting considered in this paper.
Revised: However, in the next section, we will show that, in tabular MDPs with strictly positive gaps, in contrast with the Ω(log K ) lower bound for online algorithms, we can have Alg E such that its regret is constant and independent on the number of time horizon K , which reveals the fundamental differences between the pure online setting and the Tiered RL setting considered in this paper.

Original:  Pessimism in the Face of Uncertainty and Constant Regret
Revised:  Pessimism in the Face of Uncertainty and Constant Regret

Original: In this section, we consider the gap-dependent setting and contribute to identifying the possibility to achieve constant regret by using pessimistic algorithms for Alg E .
Revised: In this section, we consider the gap-dependent setting and contribute to identifying the possibility to achieve constant regret by using pessimistic algorithms for Alg E .

Original: Intuitively, the main reason why PVI can lead to a constant regret is that the quality of the policy returned by PVI is positively correlated to the accumulation of optimal trajectories in the dataset D , which is directly connected with
Revised: Intuitively, the main reason why PVI can lead to a constant regret is that the quality of the policy returned by PVI is positively correlated to the accumulation of optimal trajectories in the dataset D , which is directly connected with

Original: Regret ( Alg O ) .
Revised: Regret ( Alg O ) .

Original: As a result, on the one hand, the regret minimization objective of Alg E coincidentally aligns with the optimality constraint of Alg O .
Revised: As a result, on the one hand, the regret minimization objective of Alg E coincidentally aligns with the optimality constraint of Alg O .

Original: On the other hand, thanks to the positive gap assumption, π E will gradually converge to the optimal policy with high probability when Alg E is PVI, so there will be no regret after that.
Revised: On the other hand, thanks to the positive gap assumption, π E will gradually converge to the optimal policy with high probability when Alg E is PVI, so there will be no regret after that.

Original: In Sec. 4.1, we start with stochastic MAB as a warm-up, and in Sec. 4.
Revised: In Sec. 4.1, we start with stochastic MAB as a warm-up, and in Sec. 4.

Original: we extend our success to tabular RL setting.
Revised: we extend our success to tabular RL setting.

Original: We defer the proofs in this section to Appendix D.
Revised: We defer the proofs in this section to Appendix D.

Original: 4.1 Warm-Up: Gap-Dependent Regret Bound for Stochastic Multi-Armed Bandits
Revised: 4.1 Warm-Up: Gap-Dependent Regret Bound for Stochastic Multi-Armed Bandits

Original: Algorithm
Revised: Algorithm

Original: 1 Initilize :
Revised: 1 Initilize :

Original: [Equation]
Revised: [Equation]

Original: > 1;
Revised: > 1;

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 for k = 1 , 2 , ..., K do
Revised: 2 for k = 1 , 2 , ..., K do

Original: [Equation]
Revised: [Equation]

Original: O
Revised: O

Original: k ← arg max i (cid:98) µ i ( k ) + (cid:113) 2 α log f ( k ) N
Revised: k ← arg max i (cid:98) µ i ( k ) + (cid:113) 2 α log f ( k ) N

Original: i
Revised: i

Original: ( k ) , π E k ← arg max i (cid:98) µ i ( k ) − (cid:113) 

Revised: ( k ) , π E k ← arg max i (cid:98) µ i ( k ) − (cid:113)

Original: α log f ( k ) N i ( k ) .
Revised: α log f ( k ) N i ( k ) .

Original: do
Revised: do

Original: N i ( k + 1) ← N i ( k ) + I [ π O k
Revised: N i ( k + 1) ← N i ( k ) + I [ π O k

Original: = i ]; (cid:98) µ
Revised: = i ]; (cid:98) µ

Original: [Equation]
Revised: [Equation]

Original: ) ←
Revised: ) ←

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: N i (
Revised: N i (

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: I [ π O k
Revised: I [ π O k

Original: = i ] N i ( k
Revised: = i ] N i ( k

Original: [Equation]
Revised: [Equation]

Original: end 8 end
Revised: end 8 end

Original: Our main algorithm for bandit setting is shown in Alg 1, where we consider the UCB algorithm [Lattimore and Szepesvári, 2020] as Alg O and choose the LCB as Alg E , which flips the sign of the bonus term in UCB.
Revised: Our main algorithm for bandit setting is shown in Alg 1, where we consider the UCB algorithm [Lattimore and Szepesvári, 2020] as Alg O and choose the LCB as Alg E , which flips the sign of the bonus term in UCB.

Original: We use N i ( k ) to denote the number of times that arm i was pulled previous to step k , and use (cid:98) µ i to record the empirical average of arm i .
Revised: We use N i ( k ) to denote the number of times that arm i was pulled previous to step k , and use (cid:98) µ i to record the empirical average of arm i .

Original: Besides, we assume 1 /N i ( · ) =
Revised: Besides, we assume 1 /N i ( · ) =

Original: + ∞ if N i ( · ) = 0 , which implies that at the first |A| steps the algorithm will pull each arm one by one.
Revised: + ∞ if N i ( · ) = 0 , which implies that at the first |A| steps the algorithm will pull each arm one by one.

Original: Moreover, as we will show later, the choice of α > 1 is crucial to avoiding dependence on K in
Revised: Moreover, as we will show later, the choice of α > 1 is crucial to avoiding dependence on K in

Original: Regret ( Alg E ) with our techniques.
Revised: Regret ( Alg E ) with our techniques.

Original: For Alg. 2, we have the following guarantee:
Revised: For Alg. 2, we have the following guarantee:

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: [Exploitation Regret]
Revised: [Exploitation Regret]

Original: In Algorithm 2, by choosing arbitrary α > 1 , there exists an absolute constant c , such that, for arbitrary K ≥ 1 , the pseduo-regret of Alg E is upper bounded by:
Revised: In Algorithm 2, by choosing arbitrary α > 1 , there exists an absolute constant c , such that, for arbitrary K ≥ 1 , the pseduo-regret of Alg E is upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: where ∆
Revised: where ∆

Original: [Equation]
Revised: [Equation]

Original: ∞ so
Revised: ∞ so

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Our result implies that by choosing PVI as Alg E , we can achieve constant regret while keeping
Revised: Our result implies that by choosing PVI as Alg E , we can achieve constant regret while keeping

Original: Alg O near-optimal.
Revised: Alg O near-optimal.

Original: Besides the advantages discussed in the related work paragraph in Sec. 1, our guarantee is also more favorable in certain cases compared to the O ( (cid:113)
Revised: Besides the advantages discussed in the related work paragraph in Sec. 1, our guarantee is also more favorable in certain cases compared to the O ( (cid:113)

Original: A ∆ min (cid:113)(cid:80) ∆ i > 0 1∆ i ) result in Rouyer and Seldin [2020]: while it is not easy to verify whether our guarantee dominates theirs, in many cases ours can be strictly better (or at least no worse) than theirs.
Revised: A ∆ min (cid:113)(cid:80) ∆ i > 0 1∆ i ) result in Rouyer and Seldin [2020]: while it is not easy to verify whether our guarantee dominates theirs, in many cases ours can be strictly better (or at least no worse) than theirs.

Original: For example, consider the following two representative cases: ∆ 1 = ∆ 2 = ...
Revised: For example, consider the following two representative cases: ∆ 1 = ∆ 2 = ...

Original: ∆ A − 1 = ∆ min (uniform gap) and ∆ 1 = ∆ 2 = ... =
Revised: ∆ A − 1 = ∆ min (uniform gap) and ∆ 1 = ∆ 2 = ... =

Original: ∆ A − 2 ≫ ∆ A − 1 = ∆ min (small last gap); our result achieves (cid:101) O
Revised: ∆ A − 2 ≫ ∆ A − 1 = ∆ min (small last gap); our result achieves (cid:101) O

Original: ( A ∆ min ) and (cid:101) O
Revised: ( A ∆ min ) and (cid:101) O

Original: ( 1∆ min ) , respectively, in contrast to their (cid:101)
Revised: ( 1∆ min ) , respectively, in contrast to their (cid:101)

Original: O
Revised: O

Original: ( A ∆ min ) and (cid:101) O ( √ A ∆ min ) .
Revised: ( A ∆ min ) and (cid:101) O ( √ A ∆ min ) .

Original: Proof Sketch :
Revised: Proof Sketch :

Original: The proof consists of two novel technique lemmas with a carefully chosen failure rate δ k ∼ O (1 /k Θ( α ) )
Revised: The proof consists of two novel technique lemmas with a carefully chosen failure rate δ k ∼ O (1 /k Θ( α ) )

Original: so that the accumulative failure probability (cid:80) ∞ k =1 δ k < + ∞ .
Revised: so that the accumulative failure probability (cid:80) ∞ k =1 δ k < + ∞ .

Original: The first one is Lem.
Revised: The first one is Lem.

Original: 4.2, where we show that w.p. 1 − δ k , LCB will not prefer i with ∆ i > 0 as long as another better arm has been visited enough times in the dataset.
Revised: 4.2, where we show that w.p. 1 − δ k , LCB will not prefer i with ∆ i > 0 as long as another better arm has been visited enough times in the dataset.

Original: The second step is to identify a key property of UCB algorithm as stated in Lem.
Revised: The second step is to identify a key property of UCB algorithm as stated in Lem.

Original: 4.3, where we provide a high probability upper bound that N i ( k ) ≤ k/λ if k ≥ (cid:101)
Revised: 4.3, where we provide a high probability upper bound that N i ( k ) ≤ k/λ if k ≥ (cid:101)

Original: Θ( λ/ ∆ 2 i ) for arbitrary λ ∈ [1 , 4 A ] , and it serves to indicate that the condition required by the success of LCB is achievable as long as k is large enough 3 .
Revised: Θ( λ/ ∆ 2 i ) for arbitrary λ ∈ [1 , 4 A ] , and it serves to indicate that the condition required by the success of LCB is achievable as long as k is large enough 3 .

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: [Blessing of Pessimism] With the choice that f ( k ) = 

Revised: [Blessing of Pessimism] With the choice that f ( k ) =

Original: + 

Revised: +

Original: A 2 ( k + 1) 2 , for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, and arbitrary j satisfying ∆
Revised: A 2 ( k + 1) 2 , for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, and arbitrary j satisfying ∆

Original: j < ∆ i , we have:
Revised: j < ∆ i , we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: [Property of UCB]
Revised: [Property of UCB]

Original: With the choice that f ( k ) = 1 + 

Revised: With the choice that f ( k ) = 1 +

Original: A 2 ( k + 1) 

Revised: A 2 ( k + 1)

Original: , there exists a constant c , for arbitrary i with ∆ i > 0 and arbitrary λ ∈ [1 , 4 A ] , in UCB algorithm, we have:
Revised: , there exists a constant c , for arbitrary i with ∆ i > 0 and arbitrary λ ∈ [1 , 4 A ] , in UCB algorithm, we have:

Original: [Equation]
Revised: [Equation]

Original: Directly combining the above two results, we can obtain an upper bound for Regret ( Alg E ) of order (cid:101)
Revised: Directly combining the above two results, we can obtain an upper bound for Regret ( Alg E ) of order (cid:101)

Original: O
Revised: O

Original: ( A/ ∆ − 2min ) , which is already independent of K .
Revised: ( A/ ∆ − 2min ) , which is already independent of K .

Original: To achieve better dependence on ∆ min in the regret, we conduct a finer analysis.
Revised: To achieve better dependence on ∆ min in the regret, we conduct a finer analysis.

Original: For each arm i with ∆ i > 0 , we separate all the arms including i into two groups based on whether its gap exceeds ∆ i / 2 : G lower i = { j : ∆ j > ∆ i / 2 } and G upper i = { j : ∆ j ≤ ∆ i / 2 } .
Revised: For each arm i with ∆ i > 0 , we separate all the arms including i into two groups based on whether its gap exceeds ∆ i / 2 : G lower i = { j : ∆ j > ∆ i / 2 } and G upper i = { j : ∆ j ≤ ∆ i / 2 } .

Original: As a result of Lem.
Revised: As a result of Lem.

Original: 4.2, we know that π E k will not prefer arm i as long as there exists j ∈ G upper i such that N j ( k )
Revised: 4.2, we know that π E k will not prefer arm i as long as there exists j ∈ G upper i such that N j ( k )

Original: = (cid:101)
Revised: = (cid:101)

Original: Ω(4∆ − 2 i ) =
Revised: Ω(4∆ − 2 i ) =

Original: (cid:101) Ω(∆ − 2 i ) .
Revised: (cid:101) Ω(∆ − 2 i ) .

Original: Based on Lem.
Revised: Based on Lem.

Original: 4.3, we know it is true with high probability, as long as k ≥ (cid:101)
Revised: 4.3, we know it is true with high probability, as long as k ≥ (cid:101)

Original: Θ( A · ∆ − 2 i ) , since at that time N l ( k ) ≤ k/A holds for arbitrary l ∈ G lower i , which directly implies that max j : j ∈ G upper
Revised: Θ( A · ∆ − 2 i ) , since at that time N l ( k ) ≤ k/A holds for arbitrary l ∈ G lower i , which directly implies that max j : j ∈ G upper

Original: i
Revised: i

Original: N j ( k ) ≥ (cid:101) Ω(∆
Revised: N j ( k ) ≥ (cid:101) Ω(∆

Original: − 2 i ) .
Revised: − 2 i ) .

Original: Then, combining Lem.
Revised: Then, combining Lem.

Original: 4.2, with high probability, the regret resulting from taken arm i cannot be higher than (cid:101) Θ( A · ∆ − 2 i ) · ∆ i = (cid:101) Θ( A · ∆ − 1 i ) , which results in a (cid:101)
Revised: 4.2, with high probability, the regret resulting from taken arm i cannot be higher than (cid:101) Θ( A · ∆ − 2 i ) · ∆ i = (cid:101) Θ( A · ∆ − 1 i ) , which results in a (cid:101)

Original: O
Revised: O

Original: ( (cid:80) ∆ i > 

Revised: ( (cid:80) ∆ i >

Original: A/ ∆ i ) regret bound.
Revised: A/ ∆ i ) regret bound.

Original: As for the techniques leading to the further improvement in our final result, please refer to Lem.
Revised: As for the techniques leading to the further improvement in our final result, please refer to Lem.

Original: D.1 and the proof of Thm.
Revised: D.1 and the proof of Thm.

Original: 4.1 in Appx.
Revised: 4.1 in Appx.

Original: D.
Revised: D.

Original: 4.2 Constant Regret of Alg E in Tabular MDPs
Revised: 4.2 Constant Regret of Alg E in Tabular MDPs

Original: In this section, we establish constant regret of Alg E based on realistic conditions for Alg O and Alg E .
Revised: In this section, we establish constant regret of Alg E based on realistic conditions for Alg O and Alg E .

Original: We highlight the key steps of our analysis and our technical contributions here.
Revised: We highlight the key steps of our analysis and our technical contributions here.

Original: First of all, in Sec.4.2.1, we propose the concrete PVI algorithm, and inspired by the clipping trick used for optimistic online algorithms [Simchowitz and Jamieson, 2019], we develop a high-probability gap-dependent upper bound for the sub-optimality of π E , which is related to the accumulation of the optimal trajectories in dataset D k .
Revised: First of all, in Sec.4.2.1, we propose the concrete PVI algorithm, and inspired by the clipping trick used for optimistic online algorithms [Simchowitz and Jamieson, 2019], we develop a high-probability gap-dependent upper bound for the sub-optimality of π E , which is related to the accumulation of the optimal trajectories in dataset D k .

Original: Secondly, in Sec.
Revised: Secondly, in Sec.

Original: 4.2.2, we first introduce a general condition (Cond. 4.6) for the chocie of Alg O , based on which we quantify the accumulation of optimal trajectories in D k with the regret of Alg O , and connect the exploration by Alg O and the optimality of
Revised: 4.2.2, we first introduce a general condition (Cond. 4.6) for the chocie of Alg O , based on which we quantify the accumulation of optimal trajectories in D k with the regret of Alg O , and connect the exploration by Alg O and the optimality of

Original: Alg E .
Revised: Alg E .

Original: We also supplement some details about how to relax such a condition and inherit the guarantees by the doubling-trick in Appx.
Revised: We also supplement some details about how to relax such a condition and inherit the guarantees by the doubling-trick in Appx.

Original: G, which may be of independent interest.
Revised: G, which may be of independent interest.

Original: In Sec.
Revised: In Sec.

Original: 4.2.3, i.e. the last part of analysis, we bring the above two steps together and complete the proof.
Revised: 4.2.3, i.e. the last part of analysis, we bring the above two steps together and complete the proof.

Original: However, there is an additional challenge when the tabular MDP has multiple deterministic optimal policies, which is possible when there are non-unique optimal actions at some states.
Revised: However, there is an additional challenge when the tabular MDP has multiple deterministic optimal policies, which is possible when there are non-unique optimal actions at some states.

Original: We overcome this difficulty by
Revised: We overcome this difficulty by

Original: Thm. 4.8 about policy coverage.
Revised: Thm. 4.8 about policy coverage.

Original: To our knowledge, the only paper that runs into a similar challenge is [Papini et al., 2021], and they bypass the difficulty by assuming the uniqueness of optimal policy.
Revised: To our knowledge, the only paper that runs into a similar challenge is [Papini et al., 2021], and they bypass the difficulty by assuming the uniqueness of optimal policy.

Original: Finally, Section 4.2.4 provide some interpretation and implications of our results.
Revised: Finally, Section 4.2.4 provide some interpretation and implications of our results.

Original: 4.2.1 Pessimistic Value Iteration as Alg E and its Property
Revised: 4.2.1 Pessimistic Value Iteration as Alg E and its Property

Original: The full details of our algorithm for tiered RL setting is provided in Alg.
Revised: The full details of our algorithm for tiered RL setting is provided in Alg.

Original: 3, where we use PVI as
Revised: 3, where we use PVI as

Original: Alg E .
Revised: Alg E .

Original: Here we do not specify a concrete Bonus function, but provide general results for a range of
Revised: Here we do not specify a concrete Bonus function, but provide general results for a range of

Original: Alg O ( D k ) .
Revised: Alg O ( D k ) .

Original: // Step 3 : Sample trajectories and collect new data
Revised: // Step 3 : Sample trajectories and collect new data

Original: Interact with G E and G O by π E k and π O k , and observe τ E k and τ O k , respectively.
Revised: Interact with G E and G O by π E k and π O k , and observe τ E k and τ O k , respectively.

Original: D k +1 ← D k ∪ { τ O k } .
Revised: D k +1 ← D k ∪ { τ O k } .

Original: 19 end qualified bonus functions satisfying Cond.
Revised: 19 end qualified bonus functions satisfying Cond.

Original: 4.4 below.
Revised: 4.4 below.

Original: Cond.
Revised: Cond.

Original: 4.4 can be satisfied by many bonus term considered in online literatures, and we briefly dicuss some examples in Appx. F.1.
Revised: 4.4 can be satisfied by many bonus term considered in online literatures, and we briefly dicuss some examples in Appx. F.1.

Original: Condition 4.4 (Condition on Bonus Term for Alg E ) .
Revised: Condition 4.4 (Condition on Bonus Term for Alg E ) .

Original: We define the following event at iteration k ∈
Revised: We define the following event at iteration k ∈

Original: [ K ] during the running of Alg. 3: E Bonus ,k := (cid:84) h ∈
Revised: [ K ] during the running of Alg. 3: E Bonus ,k := (cid:84) h ∈

Original: [ H ] ,s h ∈S h ,a h ∈A h (cid:110) {| (cid:98) P k,h (cid:98) V k,
Revised: [ H ] ,s h ∈S h ,a h ∈A h (cid:110) {| (cid:98) P k,h (cid:98) V k,

Original: h +1 ( s h , a h ) −
Revised: h +1 ( s h , a h ) −

Original: P h (cid:98) V k,h
Revised: P h (cid:98) V k,h

Original: +

Revised: +

Original: ( s h , a h ) | < b k,h
Revised: ( s h , a h ) | < b k,h

Original: ( s h , a h ) } ∩ { b k,h
Revised: ( s h , a h ) } ∩ { b k,h

Original: ( s h , a h ) ≤ B 1 (cid:113) log( B 2 /δ k ) N k,h
Revised: ( s h , a h ) ≤ B 1 (cid:113) log( B 2 /δ k ) N k,h

Original: ( s h ,a h ) } (cid:111) where B 1 and B 2 are parameters depending on S, A, H and ∆ but independent of δ k , k .
Revised: ( s h ,a h ) } (cid:111) where B 1 and B 2 are parameters depending on S, A, H and ∆ but independent of δ k , k .

Original: We assume that, Bonus function satisfies that, in Alg. 3, given arbitrary sequence { δ k } Kk =1 with δ 1 , δ 2 , ..., δ K ∈ (0 , 1 / 2) , at arbitrary iteration k ∈ [ K ] , we have Pr( E Bonus ,k ) ≥ 1 − δ k .
Revised: We assume that, Bonus function satisfies that, in Alg. 3, given arbitrary sequence { δ k } Kk =1 with δ 1 , δ 2 , ..., δ K ∈ (0 , 1 / 2) , at arbitrary iteration k ∈ [ K ] , we have Pr( E Bonus ,k ) ≥ 1 − δ k .

Original: Next, we provide an upper bound for the sub-optimality gap of π PVI k with the clipping operator
Revised: Next, we provide an upper bound for the sub-optimality gap of π PVI k with the clipping operator

Original: Clip [ x | ε ] := x · I [ x ≥ ε ] .
Revised: Clip [ x | ε ] := x · I [ x ≥ ε ] .

Original: Previous upper bounds of PVI [e.g., Theorem 4.4 of Jin et al., 2021] do not leverage the strictly positive gap and can be much looser when N k,h is large, and directly applying those results to our analysis would result in a regret scaling with √ K .
Revised: Previous upper bounds of PVI [e.g., Theorem 4.4 of Jin et al., 2021] do not leverage the strictly positive gap and can be much looser when N k,h is large, and directly applying those results to our analysis would result in a regret scaling with √ K .

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: By running Algorithm 3 with confidence level δ k , a function Bonus satisfying Condition 4.4, and a dataset D = { τ 1 , ...τ k } consisting of k complete trajectories generated by executing a sequence of policies π 1 , ..., π k , on the event E Bonus defined in Condition 4.4:
Revised: By running Algorithm 3 with confidence level δ k , a function Bonus satisfying Condition 4.4, and a dataset D = { τ 1 , ...τ k } consisting of k complete trajectories generated by executing a sequence of policies π 1 , ..., π k , on the event E Bonus defined in Condition 4.4:

Original: [Equation]
Revised: [Equation]

Original: where π ∗ can be an arbitrary optimal policy, ε Clip :
Revised: where π ∗ can be an arbitrary optimal policy, ε Clip :

Original: = ∆ min 2 H +2 if | Π ∗ | = 1 and ε Clip := d min ∆ min 2 SAH if | Π ∗ | > 1 , where d min :=
Revised: = ∆ min 2 H +2 if | Π ∗ | = 1 and ε Clip := d min ∆ min 2 SAH if | Π ∗ | > 1 , where d min :=

Original: min π ∈ Π ∗ ,h ∈
Revised: min π ∈ Π ∗ ,h ∈

Original: [ H ] ,s h ∈S h ,a h ∈A h
Revised: [ H ] ,s h ∈S h ,a h ∈A h

Original: d π ( s h , a h ) subject to d π ( s h , a h ) > 0 .
Revised: d π ( s h , a h ) subject to d π ( s h , a h ) > 0 .

Original: 4.2.2 Choice and Analysis of Alg O
Revised: 4.2.2 Choice and Analysis of Alg O

Original: Next, we introduce our general condition for Alg O that the Alg O can achieve O (log k ) -regret with high probability.
Revised: Next, we introduce our general condition for Alg O that the Alg O can achieve O (log k ) -regret with high probability.

Original: It is worth noting that many existing near-optimal online RL algorithms for gapdependent settings may not directly satisfy the condition [Simchowitz and Jamieson, 2019, Xu et al.,
Revised: It is worth noting that many existing near-optimal online RL algorithms for gapdependent settings may not directly satisfy the condition [Simchowitz and Jamieson, 2019, Xu et al.,

Original: 2021, Dann et al., 2021] since they use a fixed confidence interval δ .
Revised: 2021, Dann et al., 2021] since they use a fixed confidence interval δ .

Original: In Appx.
Revised: In Appx.

Original: G, we will introduce a more realistic abstraction of those algorithms in Cond.
Revised: G, we will introduce a more realistic abstraction of those algorithms in Cond.

Original: G.1, and discuss in more details about how to close this gap with an algorithm framework inspired by the doubling trick.
Revised: G.1, and discuss in more details about how to close this gap with an algorithm framework inspired by the doubling trick.

Original: Condition 4.6 (Condition on Alg O ) .
Revised: Condition 4.6 (Condition on Alg O ) .

Original: Alg O is an algorithm which returns deterministic policies at each iteration, and for arbitrary k ≥ 2 , we have: Pr
Revised: Alg O is an algorithm which returns deterministic policies at each iteration, and for arbitrary k ≥ 2 , we have: Pr

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 are parameters only depending on S, A, H and gap ∆ and independent of k .
Revised: where C 1 , C 2 are parameters only depending on S, A, H and gap ∆ and independent of k .

Original: Implication of Condition 4.6 for Alg O Intuitively, low regret implies high accumulation of optimal trajectories in the dataset collected by Alg O .
Revised: Implication of Condition 4.6 for Alg O Intuitively, low regret implies high accumulation of optimal trajectories in the dataset collected by Alg O .

Original: We formalize this intuition in Thm. 4.7 by establishing the relationship between the regret of Alg O , d π ∗ and (cid:80) k (cid:101) k
Revised: We formalize this intuition in Thm. 4.7 by establishing the relationship between the regret of Alg O , d π ∗ and (cid:80) k (cid:101) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: , a h
Revised: , a h

Original: [Equation]
Revised: [Equation]

Original: (the expectation of N k,h ).
Revised: (the expectation of N k,h ).

Original: Theorem 4.7.
Revised: Theorem 4.7.

Original: For an arbitrary sequence of deterministic policies π 1 , π 2 , ..., π k , there must exist a sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ..., π ∗ k , such that ∀ h ∈
Revised: For an arbitrary sequence of deterministic policies π 1 , π 2 , ..., π k , there must exist a sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ..., π ∗ k , such that ∀ h ∈

Original: [ H ] , s h ∈ S h , a h ∈
Revised: [ H ] , s h ∈ S h , a h ∈

Original: A h :
Revised: A h :

Original: [Equation]
Revised: [Equation]

Original: 4.2.3 Main Results and Analysis
Revised: 4.2.3 Main Results and Analysis

Original: The main analysis is based on our discussion about the properties of Alg E and Alg O in previous sub-sections.
Revised: The main analysis is based on our discussion about the properties of Alg E and Alg O in previous sub-sections.

Original: In the following, we first discuss the proof sketch for the case when | Π ∗ | = 1 .
Revised: In the following, we first discuss the proof sketch for the case when | Π ∗ | = 1 .

Original: The main idea is to show that the unique π ∗ will be “well-covered” by dataset, where we say a policy π ∗ is “well-covered” if for each ( s h , a h ) ∈ S
Revised: The main idea is to show that the unique π ∗ will be “well-covered” by dataset, where we say a policy π ∗ is “well-covered” if for each ( s h , a h ) ∈ S

Original: h ×
Revised: h ×

Original: A h with d π ∗ ( s h , a h ) > 0 , N k,h
Revised: A h with d π ∗ ( s h , a h ) > 0 , N k,h

Original: ( s h , a h ) can strictly increase so that the RHS of Eq.(1) in Thm.
Revised: ( s h , a h ) can strictly increase so that the RHS of Eq.(1) in Thm.

Original: 4.5 will gradually decay to zero (e.g. N k,h
Revised: 4.5 will gradually decay to zero (e.g. N k,h

Original: ( s h , a h ) ≥ (cid:101) O ( k ) ).
Revised: ( s h , a h ) ≥ (cid:101) O ( k ) ).

Original: To show this, the key observation is that, with high probability, N k,h
Revised: To show this, the key observation is that, with high probability, N k,h

Original: ( s h , a h ) will not deviate too much from its expectation (cid:80) (cid:101) k d π (cid:101) k
Revised: ( s h , a h ) will not deviate too much from its expectation (cid:80) (cid:101) k d π (cid:101) k

Original: ( s h , a h ) (Lem. F.8), and can be lower bounded by (cid:80) k (cid:101) k
Revised: ( s h , a h ) (Lem. F.8), and can be lower bounded by (cid:80) k (cid:101) k

Original: =1 d π ∗ (cid:101) k
Revised: =1 d π ∗ (cid:101) k

Original: ( s h , a h ) − O (log k ) =
Revised: ( s h , a h ) − O (log k ) =

Original: kd π ∗ ( s h , a h ) − O (log k ) as a result of Thm. 4.7.
Revised: kd π ∗ ( s h , a h ) − O (log k ) as a result of Thm. 4.7.

Original: As a result, the clipping operator in Eq.(1) will take effects as long as k is large enough, and π PVI k will converge to the optimal policy with no regret.
Revised: As a result, the clipping operator in Eq.(1) will take effects as long as k is large enough, and π PVI k will converge to the optimal policy with no regret.

Original: All that remains is to show the regret under failure events is also at the constant level because we choose a gradually decreasing failure rate O ( 1 k α ) , and lim
Revised: All that remains is to show the regret under failure events is also at the constant level because we choose a gradually decreasing failure rate O ( 1 k α ) , and lim

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: O (
Revised: O (

Original: [Equation]
Revised: [Equation]

Original: ) < ∞ as long as α > 1 .
Revised: ) < ∞ as long as α > 1 .

Original: However, when | Π ∗ | > 1 , the analysis becomes more challenging.
Revised: However, when | Π ∗ | > 1 , the analysis becomes more challenging.

Original: The main difficulty is that, when the optimal policy is not unique, it is not obvious about the existence of “well-covered” π ∗ , since it is not guarantee that how much similarity is shared by the sequence of policies π ∗ 1 , ...
Revised: The main difficulty is that, when the optimal policy is not unique, it is not obvious about the existence of “well-covered” π ∗ , since it is not guarantee that how much similarity is shared by the sequence of policies π ∗ 1 , ...

Original: π ∗ k , especially when | Π ∗ | is exponentially large (e.g. | Π ∗ |
Revised: π ∗ k , especially when | Π ∗ | is exponentially large (e.g. | Π ∗ |

Original: = Ω(( SA ) H ) ).
Revised: = Ω(( SA ) H ) ).

Original: We overcome this difficulty by proving the existence of “well-covered” policy in the theorem stated below:
Revised: We overcome this difficulty by proving the existence of “well-covered” policy in the theorem stated below:

Original: Theorem 4.8.
Revised: Theorem 4.8.

Original: [The existance of well-covered optimal policy] Given an arbitrary tabular MDP, and an arbitrary sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ...
Revised: [The existance of well-covered optimal policy] Given an arbitrary tabular MDP, and an arbitrary sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ...

Original: π ∗ k ( π ∗ i may not equal to π ∗ j for arbitrary 1 ≤ i < j ≤ k when there are multiple deterministic optimal policies), there exists a (possibly stochastic) policy π ∗ cover
Revised: π ∗ k ( π ∗ i may not equal to π ∗ j for arbitrary 1 ≤ i < j ≤ k when there are multiple deterministic optimal policies), there exists a (possibly stochastic) policy π ∗ cover

Original: such that ∀ h ∈
Revised: such that ∀ h ∈

Original: [ H ] , ∀
Revised: [ H ] , ∀

Original: ( s h , a h ) ∈ S
Revised: ( s h , a h ) ∈ S

Original: h ×
Revised: h ×

Original: A h with d π ∗ cover ( s h , a h ) > 0 : k (cid:88)
Revised: A h with d π ∗ cover ( s h , a h ) > 0 : k (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( s h , a h ) , with (cid:101) d π ∗ cover ( · , ·
Revised: ( s h , a h ) , with (cid:101) d π ∗ cover ( · , ·

Original: [Equation]
Revised: [Equation]

Original: := max (cid:26) d
Revised: := max (cid:26) d

Original: [Equation]
Revised: [Equation]

Original: h, min
Revised: h, min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∗ cover ( · ,
Revised: ∗ cover ( · ,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:27) .
Revised: ) (cid:27) .

Original: where Z ∗ h, div :=
Revised: where Z ∗ h, div :=

Original: { ( s h , a h ) ∈ S
Revised: { ( s h , a h ) ∈ S

Original: h ×
Revised: h ×

Original: A h |∃ π ∗ , (cid:101)
Revised: A h |∃ π ∗ , (cid:101)

Original: π ∗ ∈
Revised: π ∗ ∈

Original: Π ∗ , s.t.
Revised: Π ∗ , s.t.

Original: d π ∗ ( s h ) > 0 , d (cid:101) π ∗ ( s h ) = 0 } , and d ∗ h, min ( s h , a h ) :
Revised: d π ∗ ( s h ) > 0 , d (cid:101) π ∗ ( s h ) = 0 } , and d ∗ h, min ( s h , a h ) :

Original: =
Revised: =

Original: min
Revised: min

Original: π ∗ ∈ Π ∗ d π ∗ ( s h , a h ) subject to d π ∗ ( s h , a h ) > 0 .
Revised: π ∗ ∈ Π ∗ d π ∗ ( s h , a h ) subject to d π ∗ ( s h , a h ) > 0 .

Original: Here we provide some explanation to the above result.
Revised: Here we provide some explanation to the above result.

Original: According to the definition, Z ∗ h,div is the set including the state-action pairs which can be covered by some deterministic policies but is not reachable by some other deterministic policies, and therefore |Z ∗ h,div | ≤ SA (or even |Z ∗ h,div | ≪ SA ).
Revised: According to the definition, Z ∗ h,div is the set including the state-action pairs which can be covered by some deterministic policies but is not reachable by some other deterministic policies, and therefore |Z ∗ h,div | ≤ SA (or even |Z ∗ h,div | ≪ SA ).

Original: Besides, d ∗ h, min ( s h , a h ) denotes the minimal occupancy over all possible deterministic optimal policies which can hit s h , a h , and therefore, is no less than d min defined in Thm.
Revised: Besides, d ∗ h, min ( s h , a h ) denotes the minimal occupancy over all possible deterministic optimal policies which can hit s h , a h , and therefore, is no less than d min defined in Thm.

Original: As a result, we know there exists a “well-covered” π ∗ cover , since the accumulative density of its arbitrary reachable states can be lower bounded by O ( k ) .
Revised: As a result, we know there exists a “well-covered” π ∗ cover , since the accumulative density of its arbitrary reachable states can be lower bounded by O ( k ) .

Original: Then, following a similar discussion as the case | Π ∗ | = 1 , we can finish the proof.
Revised: Then, following a similar discussion as the case | Π ∗ | = 1 , we can finish the proof.

Original: We summarize our main result below.
Revised: We summarize our main result below.

Original: Theorem 4.9.
Revised: Theorem 4.9.

Original: By running an Algorithm satisfying Condition 4.6 as Alg O , running Alg 3 as Alg E with a bonus term function Bonus satisfying Condition 4.4 and δ k = 1 /k α , for some constant α > 1 , for arbitrary K ≥ 1 , the exploitation regret of Alg E can be upper bounded by:
Revised: By running an Algorithm satisfying Condition 4.6 as Alg O , running Alg 3 as Alg E with a bonus term function Bonus satisfying Condition 4.4 and δ k = 1 /k α , for some constant α > 1 , for arbitrary K ≥ 1 , the exploitation regret of Alg E can be upper bounded by:

Original: (i) When | Π ∗ | = 1 (unique optimal deterministic policy): Regret K ( Alg E )
Revised: (i) When | Π ∗ | = 1 (unique optimal deterministic policy): Regret K ( Alg E )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (ii) When | Π ∗ | > 1 (non-unique optimal deterministic policies): Regret K ( Alg E ) ≤ O (cid:16)
Revised: (ii) When | Π ∗ | > 1 (non-unique optimal deterministic policies): Regret K ( Alg E ) ≤ O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: where π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are introduced in Theorem 4.
Revised: where π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are introduced in Theorem 4.

Original: [Equation]
Revised: [Equation]

Original: 4.2.4 Interpretation of Results in Tabular RL
Revised: 4.2.4 Interpretation of Results in Tabular RL

Original: Recall our objective in Sec. 1 is to establish the benefits of leveraging the tiered structure by showing
Revised: Recall our objective in Sec. 1 is to establish the benefits of leveraging the tiered structure by showing

Original: Regret K ( Alg E ) is constant.
Revised: Regret K ( Alg E ) is constant.

Original: This contrasts the lower bound of online algorithms that continuously increases with the episode number K , which corresponds to the regret suffered by users in G E without leveraging the tiered structure, while Regret K ( Alg O ) keeps (near-)optimal as before.
Revised: This contrasts the lower bound of online algorithms that continuously increases with the episode number K , which corresponds to the regret suffered by users in G E without leveraging the tiered structure, while Regret K ( Alg O ) keeps (near-)optimal as before.

Original: In Appx.
Revised: In Appx.

Original: H, we also provide some simulation results as a verification of our theoretical discovery.
Revised: H, we also provide some simulation results as a verification of our theoretical discovery.

Original: One limitation of our results is that our bounds have additional dependence on d π ∗ (or even 1 /d π ∗ ) compared to most of the regret bounds in the online setting, although similar dependence on log d π ∗ also appeared in a few recent works [e.g., λ + h in Thms.
Revised: One limitation of our results is that our bounds have additional dependence on d π ∗ (or even 1 /d π ∗ ) compared to most of the regret bounds in the online setting, although similar dependence on log d π ∗ also appeared in a few recent works [e.g., λ + h in Thms.

Original: 8 and 9 of Papini et al., 2021].
Revised: 8 and 9 of Papini et al., 2021].

Original: Besides, according to the lower and upper bound of online RL in gap-dependent settings [Simchowitz and
Revised: Besides, according to the lower and upper bound of online RL in gap-dependent settings [Simchowitz and

Original: Jamieson, 2019], C 1 + C 2 in Cond.
Revised: Jamieson, 2019], C 1 + C 2 in Cond.

Original: 4.6 have dependence on O (∆ − 1min ) , which implies that in the regret bound in Thm. 4.9, the dependence on ∆ min would be O (∆ − 2min ) .
Revised: 4.6 have dependence on O (∆ − 1min ) , which implies that in the regret bound in Thm. 4.9, the dependence on ∆ min would be O (∆ − 2min ) .

Original: For the former, in Appx.
Revised: For the former, in Appx.

Original: C.2, we prove a lower bound, showing that log 1 d π ∗ is unavoidable when Alg O is allowed to behave adversarially without violating Cond.
Revised: C.2, we prove a lower bound, showing that log 1 d π ∗ is unavoidable when Alg O is allowed to behave adversarially without violating Cond.

Original: 4.6; for the latter, we note that in the analysis of MAB setting (Sec.4.1), specifying the detailed behavior of Alg O can help tighten the bound.
Revised: 4.6; for the latter, we note that in the analysis of MAB setting (Sec.4.1), specifying the detailed behavior of Alg O can help tighten the bound.

Original: Therefore, we conjecture that our results can be improved by putting more constraints on the behavior of Alg O , which we leave to future work.
Revised: Therefore, we conjecture that our results can be improved by putting more constraints on the behavior of Alg O , which we leave to future work.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we identify the tiered structure in many real-world applications and study the potential advantages of leveraging it by interacting with users from different groups with different strategies.
Revised: In this paper, we identify the tiered structure in many real-world applications and study the potential advantages of leveraging it by interacting with users from different groups with different strategies.

Original: Under the gap-dependent setting, we provide theoretical evidence of benefits by deriving constant regret for the exploitation policy while maintaining the optimality of the online learning policy.
Revised: Under the gap-dependent setting, we provide theoretical evidence of benefits by deriving constant regret for the exploitation policy while maintaining the optimality of the online learning policy.

Original: As for the future work, we propose several potentially interesting directions.
Revised: As for the future work, we propose several potentially interesting directions.

Original: (i)
Revised: (i)

Original: As we mentioned in Section 4.2.4, it is worth investigating the possibility of improving the regret bound of Alg E by considering a more concrete choice of Alg O , or maybe other choices for Alg E .
Revised: As we mentioned in Section 4.2.4, it is worth investigating the possibility of improving the regret bound of Alg E by considering a more concrete choice of Alg O , or maybe other choices for Alg E .

Original: (ii) It would be interesting to relax our constraint on the optimality of Alg O by introducing the notion of budget C as the tolerance on the sub-optimality of Alg O .
Revised: (ii) It would be interesting to relax our constraint on the optimality of Alg O by introducing the notion of budget C as the tolerance on the sub-optimality of Alg O .

Original: As a result, our setting and the decoupling exploration and exploitation setting can be regarded as special cases of a more general framework when C =and C = ∞ .
Revised: As a result, our setting and the decoupling exploration and exploitation setting can be regarded as special cases of a more general framework when C =and C = ∞ .

Original: (iii)
Revised: (iii)

Original: We assume that the users from different groups share the same transition and reward function, and it would also be interesting to extend our results to more general settings, where the group ID serves as context and will affect the dynamics [Abbasi-Yadkori and Neu, 2014, Modi et al., 2018].
Revised: We assume that the users from different groups share the same transition and reward function, and it would also be interesting to extend our results to more general settings, where the group ID serves as context and will affect the dynamics [Abbasi-Yadkori and Neu, 2014, Modi et al., 2018].

Original: (iv) We only consider the setting with two tiers, and it may be worth studying the possibility and potential benefits under the setting with multiple tiers.
Revised: (iv) We only consider the setting with two tiers, and it may be worth studying the possibility and potential benefits under the setting with multiple tiers.

Original: Acknowledgements
Revised: Acknowledgements

Original: JH’s research activities on this work were conducted during his internship at MSRA.
Revised: JH’s research activities on this work were conducted during his internship at MSRA.

Original: NJ’s last involvement was in December 2021.
Revised: NJ’s last involvement was in December 2021.

Original: NJ also acknowledges funding support from ARL Cooperative
Revised: NJ also acknowledges funding support from ARL Cooperative

Original: Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF CAREER award, and Adobe Data Science
Revised: Agreement W911NF-17-2-0196, NSF IIS-2112471, NSF CAREER award, and Adobe Data Science

Original: Research Award.
Revised: Research Award.

Original: The authors thank Yuanying Cai for valuable discussion.
Revised: The authors thank Yuanying Cai for valuable discussion.

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: A Detailed Related Work
Revised: [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original:  Online RL The online RL/MAB is the most basic framework studying the trade-off between exploration and exploitation [Auer et al., 2002, Slivkins, 2019, Lattimore and Szepesvári, 2020], where the agent targets at exploring the MDP to identify good actions as fast as possible to minimize the accumulative regrets.
Revised: [N/A] A Detailed Related Work Online RL The online RL/MAB is the most basic framework studying the trade-off between exploration and exploitation [Auer et al., 2002, Slivkins, 2019, Lattimore and Szepesvári, 2020], where the agent targets at exploring the MDP to identify good actions as fast as possible to minimize the accumulative regrets.

Original: In tabular MDPs [Jaksch et al., 2010, Dann et al., 2017, Jin et al., 2018], the regret lower bounds for non-stationary MDPs [Domingues et al., 2021] have been achieved by
Revised: In tabular MDPs [Jaksch et al., 2010, Dann et al., 2017, Jin et al., 2018], the regret lower bounds for non-stationary MDPs [Domingues et al., 2021] have been achieved by

Original: Azar et al.
Revised: Azar et al.

Original: [2017], Zanette and Brunskill [2019].
Revised: [2017], Zanette and Brunskill [2019].

Original: Recently, there has been interests in studying the gap-dependent regret [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021, Yang et al., 2021, He et al., 2021], where the agent can achieve log dependence on the number of episodes K under additional dependence on (the inverse of) the minimal gap 1∆ min .
Revised: Recently, there has been interests in studying the gap-dependent regret [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021, Yang et al., 2021, He et al., 2021], where the agent can achieve log dependence on the number of episodes K under additional dependence on (the inverse of) the minimal gap 1∆ min .

Original: Simchowitz and Jamieson [2019] reports that, similar to stochastic MABs [Lattimore and Szepesvári, 2020], the regret in the gap-dependent setting must scale as Ω(log K ) , which implies that the log K upper bound is asymptotically tight.
Revised: Simchowitz and Jamieson [2019] reports that, similar to stochastic MABs [Lattimore and Szepesvári, 2020], the regret in the gap-dependent setting must scale as Ω(log K ) , which implies that the log K upper bound is asymptotically tight.

Original: However, all of these works treat the customers equivalently and ignore the opportunities of leveraging the tiered structure.
Revised: However, all of these works treat the customers equivalently and ignore the opportunities of leveraging the tiered structure.

Original: Recently, [Papini et al., 2021] achieved similar constant regret in online setting with linear function approximation.
Revised: Recently, [Papini et al., 2021] achieved similar constant regret in online setting with linear function approximation.

Original: Comparing with ours, they investigated the benefits of good features, while we focus on the benefits of considering a different learning protocol.
Revised: Comparing with ours, they investigated the benefits of good features, while we focus on the benefits of considering a different learning protocol.

Original: Besides, although their results were established on a more general linear setting, their assumptions on the feature and the uniqueness of optimal policy are quite restrictive in tabular setting.
Revised: Besides, although their results were established on a more general linear setting, their assumptions on the feature and the uniqueness of optimal policy are quite restrictive in tabular setting.

Original: Offline RL Offline RL considers how to learn a good policy with a fixed dataset [Levine et al., 2020].
Revised: Offline RL Offline RL considers how to learn a good policy with a fixed dataset [Levine et al., 2020].

Original: Without the requirement of exploration, offline RL prefers algorithms with strong guarantees for exploitation and safety, and Pessimism in the Face of Uncertainty (PFU) becomes a major principle for achieving this both theoretically and empirically [Yin and Wang, 2021, Uehara and Sun, 2022, Liu et al., 2020, Xie et al., 2021, Buckman et al., 2020, Kumar et al., 2020, Fujimoto and Gu, 2021, Yu et al., 2020].
Revised: Without the requirement of exploration, offline RL prefers algorithms with strong guarantees for exploitation and safety, and Pessimism in the Face of Uncertainty (PFU) becomes a major principle for achieving this both theoretically and empirically [Yin and Wang, 2021, Uehara and Sun, 2022, Liu et al., 2020, Xie et al., 2021, Buckman et al., 2020, Kumar et al., 2020, Fujimoto and Gu, 2021, Yu et al., 2020].

Original: Similar to the offline setting, we choose Alg E to be a pessimistic algorithm.
Revised: Similar to the offline setting, we choose Alg E to be a pessimistic algorithm.

Original: However, we still consider to interact the environment with Alg E , although we ignore the data collected by
Revised: However, we still consider to interact the environment with Alg E , although we ignore the data collected by

Original: Alg E for now and leave the investigation of its value to future work.
Revised: Alg E for now and leave the investigation of its value to future work.

Original: As another difference, offline
Revised: As another difference, offline

Original: RL assumes the dataset is fixed and only the final performance matters, whereas we evaluate the accumulative regret of Alg E .
Revised: RL assumes the dataset is fixed and only the final performance matters, whereas we evaluate the accumulative regret of Alg E .

Original: B More Discussion about Framework. 1 and Motivating Examples
Revised: B More Discussion about Framework. 1 and Motivating Examples

Original: In this section, we try to justify that our Frw.
Revised: In this section, we try to justify that our Frw.

Original: 1 is an appropriate abstraction for our motivating examples and user-interaction real-world applications.
Revised: 1 is an appropriate abstraction for our motivating examples and user-interaction real-world applications.

Original: In the standard online learning protocol, at iteration k , the algorithm Alg compute a policy π k based on previous exploration data, the environment samples a user u k from G O and G E according to the probability where P ( u k ∈ G O ) and P ( u k ∈ G E ) , respectively (note that P ( u k ∈ G O ) + P ( u k ∈
Revised: In the standard online learning protocol, at iteration k , the algorithm Alg compute a policy π k based on previous exploration data, the environment samples a user u k from G O and G E according to the probability where P ( u k ∈ G O ) and P ( u k ∈ G E ) , respectively (note that P ( u k ∈ G O ) + P ( u k ∈

Original: G E )
Revised: G E )

Original: = 1 ).
Revised: = 1 ).

Original: After that, π k will interact with u k and obtain a new trajectory for the future learning, while u k suffers loss V ∗ 1 − V π k , and the expected accumulative loss suffered by users from two groups till step K is
Revised: After that, π k will interact with u k and obtain a new trajectory for the future learning, while u k suffers loss V ∗ 1 − V π k , and the expected accumulative loss suffered by users from two groups till step K is

Original: Regret K ( Alg) :
Revised: Regret K ( Alg

Original: = E [
Revised: [Equation]

Original: K
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now, we consider a realistic assumption about the probability of u k from different groups:
Revised: Now, we consider a realistic assumption about the probability of u k from different groups:

Original: Assumption 1 (Assumption on the Ratio between Users from Different Groups) .
Revised: Assumption 1 (Assumption on the Ratio between Users from Different Groups) .

Original: We assume that:
Revised: We assume that:

Original: [Equation]
Revised: [Equation]

Original: for some constant C .
Revised: for some constant C .

Original: Based on the assumption above, if we do not leverage the tiered structure, and treat the users from different groups equivalently, then the loss suffered by each group will be proportional to the size of that group.
Revised: Based on the assumption above, if we do not leverage the tiered structure, and treat the users from different groups equivalently, then the loss suffered by each group will be proportional to the size of that group.

Original: Therefore, even if we assume Alg is near-optimal, the expected loss for each group will scale with log K , i.e.:
Revised: Therefore, even if we assume Alg is near-optimal, the expected loss for each group will scale with log K , i.e.:

Original: [Equation]
Revised: [Equation]

Original: Besides, we can also leverage the tiered structure, and consider an alternative protocol below:
Revised: Besides, we can also leverage the tiered structure, and consider an alternative protocol below:

Original: Algorithm 4:
Revised: Algorithm 4:

Original: Leveraging
Revised: Leveraging

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: end 12 end
Revised: end 12 end

Original: In another word, in this new protocol, we use two policies at different exploitation level to interact with users from different groups, and only update policies if user comes from group G O .
Revised: In another word, in this new protocol, we use two policies at different exploitation level to interact with users from different groups, and only update policies if user comes from group G O .

Original: Note that in expectation, { u k ∈ G O } will happen for K 1+ C times, and therefore we have:
Revised: Note that in expectation, { u k ∈ G O } will happen for K 1+ C times, and therefore we have:

Original: [Equation]
Revised: [Equation]

Original: where Regret ( · ) ( Alg O ) and Regret ( · ) ( Alg E ) are originally defined in Def. 2.1, and they are exactly the metric we used to measure the performance of Alg E and Alg O under our Frw.
Revised: where Regret ( · ) ( Alg O ) and Regret ( · ) ( Alg E ) are originally defined in Def. 2.1, and they are exactly the metric we used to measure the performance of Alg E and Alg O under our Frw.

Original: Based on our results in Sec. 4.1 and 4.2, we know that under our framework, it is possible to achieve that:
Revised: Based on our results in Sec. 4.1 and 4.2, we know that under our framework, it is possible to achieve that:

Original: [Equation]
Revised: [Equation]

Original: where constant means independence of K but may include dependence on other parameters such as
Revised: where constant means independence of K but may include dependence on other parameters such as

Original: S, A, H, ∆ .
Revised: S, A, H, ∆ .

Original: Comparing with Eq.(2), (3), (4), and (5), we can see that users from G O will suffer less regret than before because we “transfer” some the regret from G E to G O , and the additional regret suffered by G O can be compensated in other forms as we discussed in Sec. 1.
Revised: Comparing with Eq.(2), (3), (4), and (5), we can see that users from G O will suffer less regret than before because we “transfer” some the regret from G E to G O , and the additional regret suffered by G O can be compensated in other forms as we discussed in Sec. 1.

Original: Remark Besides, our methods and results can be applied to those scenarios suggested by the decoupling setting [Avner et al., 2012, Rouyer and Seldin, 2020], where Alg E does not necessarily interact with the environment, and we omit the discussion here.
Revised: Remark Besides, our methods and results can be applied to those scenarios suggested by the decoupling setting [Avner et al., 2012, Rouyer and Seldin, 2020], where Alg E does not necessarily interact with the environment, and we omit the discussion here.

Original: C Lower Bounds C.1 Regret Lower Bounds for Tabular MDP without Strictly Positive Gap Assumption
Revised: C Lower Bounds C.1 Regret Lower Bounds for Tabular MDP without Strictly Positive Gap Assumption

Original: We first recall a Theorem from [Dann et al., 2017]:
Revised: We first recall a Theorem from [Dann et al., 2017]:

Original: Theorem C.1 (Theorem C.1 in [Dann et al., 2017]) .
Revised: Theorem C.1 (Theorem C.1 in [Dann et al., 2017]) .

Original: There exist positive constant c , δ 0 > 0 , ε 0 > 0 , such that for every ε ∈ (0 , ε 0 ) , S ≥ 4 , A ≥ 2 and for every algorithm Alg and n ≤ cASH 3 εthere is a fixed-horizon episodic MDP M hard with time-dependent transition probabilities and S states and A actions so that returning an ε -optimal policy after n episodes is at most 1 − δ 0 .
Revised: There exist positive constant c , δ 0 > 0 , ε 0 > 0 , such that for every ε ∈ (0 , ε 0 ) , S ≥ 4 , A ≥ 2 and for every algorithm Alg and n ≤ cASH 3 εthere is a fixed-horizon episodic MDP M hard with time-dependent transition probabilities and S states and A actions so that returning an ε -optimal policy after n episodes is at most 1 − δ 0 .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: [Lower Bound for Alg E without Gap Assumption] There exist positive constants c, ε 0 , δ 0 , such that, for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 2 , K ≥ cε 20 H 3 SA , and arbitrary algorithm pair ( Alg O , Alg E ) , there must exist a hard tabular MDP M hard , E ( Alg O , Alg E ) ,M hard (cid:104)(cid:80)
Revised: [Lower Bound for Alg E without Gap Assumption] There exist positive constants c, ε 0 , δ 0 , such that, for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 2 , K ≥ cε 20 H 3 SA , and arbitrary algorithm pair ( Alg O , Alg E ) , there must exist a hard tabular MDP M hard , E ( Alg O , Alg E ) ,M hard (cid:104)(cid:80)

Original: Kk =1 V ∗ − V π E k (cid:105) ≥ δ 0 √ cH 3 SAK, where the expectation is taken over the randomness of algorithms and MDP.
Revised: Kk =1 V ∗ − V π E k (cid:105) ≥ δ 0 √ cH 3 SAK, where the expectation is taken over the randomness of algorithms and MDP.

Original: Proof.
Revised: Proof.

Original: Suppose we have a pair algorithm ( Alg O , Alg E ) , we can construct a PAC algorithm with Alg O and Alg E , in the following way:
Revised: Suppose we have a pair algorithm ( Alg O , Alg E ) , we can construct a PAC algorithm with Alg O and Alg E , in the following way:

Original: • Input: K .
Revised: • Input: K .

Original: • For k = 1 , 2 , ..., K , run Alg O to collect data and run Alg E to generate a sequence of policies π E 1 , ..., π E K .
Revised: • For k = 1 , 2 , ..., K , run Alg O to collect data and run Alg E to generate a sequence of policies π E 1 , ..., π E K .

Original: • Uniformly randomly select an index from { 1 , 2 , ..., K } , and denote it as K PAC •
Revised: • Uniformly randomly select an index from { 1 , 2 , ..., K } , and denote it as K PAC •

Original: Output π E K PAC .
Revised: Output π E K PAC .

Original: In the following, we denote such an algorithm as Alg PAC .
Revised: In the following, we denote such an algorithm as Alg PAC .

Original: Then, for an arbitrary MDP M , we must have:
Revised: Then, for an arbitrary MDP M , we must have:

Original: [Equation]
Revised: [Equation]

Original: As a result of Markov inequality and that V ∗ − V π ≥ 0 for arbitrary π , for arbitrary ε > 0 , we have:
Revised: As a result of Markov inequality and that V ∗ − V π ≥ 0 for arbitrary π , for arbitrary ε > 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: Since the above holds for arbitrary ε ∈ (0 , ε 0 ) , by choosing ε
Revised: Since the above holds for arbitrary ε ∈ (0 , ε 0 ) , by choosing ε

Original: = ¯ ε := (cid:112) cH 3 SA/K , since K > cε 20 H 3 SA , we have ¯ ε < ε 0 and
Revised: = ¯ ε := (cid:112) cH 3 SA/K , since K > cε 20 H 3 SA , we have ¯ ε < ε 0 and

Original: [Equation]
Revised: [Equation]

Original: Because K ≤ cH 3 SA/ ¯ ε 2 , Thm.
Revised: Because K ≤ cH 3 SA/ ¯ ε 2 , Thm.

Original: C.1 implies that, for Alg PAC , for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 1 , there must exists a hard MDP M hard , such that:
Revised: C.1 implies that, for Alg PAC , for arbitrary S ≥ 4 , A ≥ 2 , H ≥ 1 , there must exists a hard MDP M hard , such that:

Original: [Equation]
Revised: [Equation]

Original: and it is equivalent to:
Revised: and it is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: which finishes the proof.
Revised: which finishes the proof.

Original: Remark C.2 (Regret lower bound when τ E is used in Framework 1) .
Revised: Remark C.2 (Regret lower bound when τ E is used in Framework 1) .

Original: Our techniques can be extended to the case when τ E are used by Alg O and Alg E , and establish the same O ( √ H 3 SAK ) lower bound.
Revised: Our techniques can be extended to the case when τ E are used by Alg O and Alg E , and establish the same O ( √ H 3 SAK ) lower bound.

Original: Because the only difference would be in this new setting, at iteration k , Alg O and Alg E will use 2 k trajectories to compute π O k and π E k , which only double the sample size comparing with k trajectories in Framework 1.
Revised: Because the only difference would be in this new setting, at iteration k , Alg O and Alg E will use 2 k trajectories to compute π O k and π E k , which only double the sample size comparing with k trajectories in Framework 1.

Original: Therefore, with the same techniques (and choosing ¯ ε = (cid:112) cH 3 SA/ (2 K ) ), one can obtain a lower bound which differs from Eq.
Revised: Therefore, with the same techniques (and choosing ¯ ε = (cid:112) cH 3 SA/ (2 K ) ), one can obtain a lower bound which differs from Eq.

Original: (8) by constant.
Revised: (8) by constant.

Original: C.2 Lower Bound for the Dependence on log d min when | Π ∗ | =
Revised: C.2 Lower Bound for the Dependence on log d min when | Π ∗ | =

Original: In this section, because we will conduct discussion on multiple different MDPs, and in order to distinguish them, we will introduce a subscript of M to highlight which MDP we are discussing.
Revised: In this section, because we will conduct discussion on multiple different MDPs, and in order to distinguish them, we will introduce a subscript of M to highlight which MDP we are discussing.

Original: Therefore, we revise some key notations and re-introduce them here.
Revised: Therefore, we revise some key notations and re-introduce them here.

Original: Notation Given arbitrary tabular MDP M = {S , A , P , r, H } .
Revised: Notation Given arbitrary tabular MDP M = {S , A , P , r, H } .

Original: We use Π ∗ M to denote the set of deterministic optimal policies of M .
Revised: We use Π ∗ M to denote the set of deterministic optimal policies of M .

Original: For each deterministic optimal policy π ∗ M ∈ Π ∗ M , we define d π ∗ M min to be the minimal non-zero occupancy of the reachable state action by π ∗ M , i.e.
Revised: For each deterministic optimal policy π ∗ M ∈ Π ∗ M , we define d π ∗ M min to be the minimal non-zero occupancy of the reachable state action by π ∗ M , i.e.

Original: [Equation]
Revised: [Equation]

Original: Then, we define:
Revised: Then, we define:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Different from regret analysis in online setting [Simchowitz and Jamieson, 2019, Xu et al., 2021,
Revised: Different from regret analysis in online setting [Simchowitz and Jamieson, 2019, Xu et al., 2021,

Original: Dann et al., 2021], our regret bound in Thm. 4.9 has additional dependence on log d π ∗ , even if the optimal policy is unique.
Revised: Dann et al., 2021], our regret bound in Thm. 4.9 has additional dependence on log d π ∗ , even if the optimal policy is unique.

Original: In Thm.
Revised: In Thm.

Original: C.3, we show that the dependence of log d π ∗ is unavoidable if we do not have other assumptions about the behavior of Alg O besides Cond. 4.6.
Revised: C.3, we show that the dependence of log d π ∗ is unavoidable if we do not have other assumptions about the behavior of Alg O besides Cond. 4.6.

Original: In another word, even if constrained by satisfies Cond. 4.6, Alg O can be arbitrarily adversarial so that log d π ∗ exists in the lower bound.
Revised: In another word, even if constrained by satisfies Cond. 4.6, Alg O can be arbitrarily adversarial so that log d π ∗ exists in the lower bound.

Original: We defer the proof to Appx.
Revised: We defer the proof to Appx.

Original: C.
Revised: C.

Original: Theorem C.3.
Revised: Theorem C.3.

Original: For arbitrary S, A, H ≥ 3 , arbitrary ∆ min > 0 and d min > 0 , if there exists
Revised: For arbitrary S, A, H ≥ 3 , arbitrary ∆ min > 0 and d min > 0 , if there exists

Original: an
Revised: an

Original: MDP M = {S , A , P , r, H } such that |S| = S , |A| =
Revised: MDP M = {S , A , P , r, H } such that |S| = S , |A| =

Original: A d M, min =
Revised: A d M, min =

Original: d min and the minimal gap is lower bound by ∆ min , then there exists a hard MDP M +
Revised: d min and the minimal gap is lower bound by ∆ min , then there exists a hard MDP M +

Original: =
Revised: =

Original: {S + , A , P + , r + , H } with |S + | =
Revised: {S + , A , P + , r + , H } with |S + | =

Original: S + 1 , minimal gap lower bounded by 3∆ min / 4 and d M + , min =
Revised: S + 1 , minimal gap lower bounded by 3∆ min / 4 and d M + , min =

Original: d M, min / 4 , and an adversarial choice of
Revised: d M, min / 4 , and an adversarial choice of

Original: Alg O satisfying Cond.
Revised: Alg O satisfying Cond.

Original: 4.6, such that when K is large enough, the expected Pseudo-Regret of Alg E is lower bounded by:
Revised: 4.6, such that when K is large enough, the expected Pseudo-Regret of Alg E is lower bounded by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is divided into three steps.
Revised: The proof is divided into three steps.

Original: Step 1: Construction of the Hard MDP Instance Now, we construct a hard MDP instance
Revised: Step 1: Construction of the Hard MDP Instance Now, we construct a hard MDP instance

Original: M + := {S + , A + , P + , r + , H } based on M by expanding the state space with an absorbing state s absorb for layer h ≥ 2 (we use h in s h,absorb to distinguish the absorbing state at different time step), and define the transition and reward function by:
Revised: M + := {S + , A + , P + , r + , H } based on M by expanding the state space with an absorbing state s absorb for layer h ≥ 2 (we use h in s h,absorb to distinguish the absorbing state at different time step), and define the transition and reward function by:

Original: [Equation]
Revised: [Equation]

Original: , a 1 ) r + ( s 1 , a 1 ) =
Revised: , a 1 ) r + ( s 1 , a 1 ) =

Original: (1 − d M, min) r ( s 1 , a 1 ) ∀ h ≥ 2 , s h ∈ S h , a h ∈
Revised: (1 − d M, min) r ( s 1 , a 1 ) ∀ h ≥ 2 , s h ∈ S h , a h ∈

Original: A h , P + ( ·| s h , a h ) =
Revised: A h , P + ( ·| s h , a h ) =

Original: P ( ·| s h , a h ) , r + ( s h , a h )
Revised: P ( ·| s h , a h ) , r + ( s h , a h )

Original: =
Revised: =

Original: r ( s h , a h ) ∀ h ≥ 2 , a h ∈
Revised: r ( s h , a h ) ∀ h ≥ 2 , a h ∈

Original: A h , P + ( s h +1 ,absorb | s h,absorb , a h )
Revised: A h , P + ( s h +1 ,absorb | s h,absorb , a h )

Original: =
Revised: =

Original: ∀ H ≥ h ≥ 2 , a h ∈
Revised: ∀ H ≥ h ≥ 2 , a h ∈

Original: A h , r ( s h,absorb , a h ) =
Revised: A h , r ( s h,absorb , a h ) =

Original: ∆ min I [ a h = a ∗ h ]
Revised: ∆ min I [ a h = a ∗ h ]

Original: Briefly speaking, at the initial state, by taking arbitrary action, with probability d M, min / 2 , it will transit to absorbing state at layer 2, and the agent can not escape from the absorbing state till the end of the episodes.
Revised: Briefly speaking, at the initial state, by taking arbitrary action, with probability d M, min / 2 , it will transit to absorbing state at layer 2, and the agent can not escape from the absorbing state till the end of the episodes.

Original: Besides, at the absorbing states, for each layer 2 ≤ h ≤ H , there always exists an optimal action a ∗ h with reward ∆ min and taking any the other actions will lead to 0 reward.
Revised: Besides, at the absorbing states, for each layer 2 ≤ h ≤ H , there always exists an optimal action a ∗ h with reward ∆ min and taking any the other actions will lead to 0 reward.

Original: Moreover,
Revised: Moreover,

Original: M + agrees with M for all the transition and rewards when h ≥ 2 .
Revised: M + agrees with M for all the transition and rewards when h ≥ 2 .

Original: Easy to see that:
Revised: Easy to see that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, if V ∗ M ( s 1 ) − Q ∗ M ( s 1 , a 1 ) > 0 , we still have:
Revised: Therefore, if V ∗ M ( s 1 ) − Q ∗ M ( s 1 , a 1 ) > 0 , we still have:

Original: [Equation]
Revised: [Equation]

Original: Combining with the transition and reward functions in absorbing states, we can conclude that the gap of M + is still O (∆ min ) .
Revised: Combining with the transition and reward functions in absorbing states, we can conclude that the gap of M + is still O (∆ min ) .

Original: Step 2: Construction of Adversarial Alg O
Revised: Step 2: Construction of Adversarial Alg O

Original: Let’s use Π ∗ M + to denote the set of deterministic optimal policies at MDP M + .
Revised: Let’s use Π ∗ M + to denote the set of deterministic optimal policies at MDP M + .

Original: It’s easy to see that, for arbitrary π ∗ M + ∈ Π ∗ M + , there must exists an optimal policy
Revised: It’s easy to see that, for arbitrary π ∗ M + ∈ Π ∗ M + , there must exists an optimal policy

Original: π ∗ M ∈ Π ∗ M agrees with π ∗ M + at all non-absorbing states (and vice versa), i.e.
Revised: π ∗ M ∈ Π ∗ M agrees with π ∗ M + at all non-absorbing states (and vice versa), i.e.

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: Then, for arbitrary π ∗ M + ∈ Π ∗ M + , we have:
Revised: Then, for arbitrary π ∗ M + ∈ Π ∗ M + , we have:

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: and s h,absorb are the hardest state to reach for all deterministic optimal policies.
Revised: and s h,absorb are the hardest state to reach for all deterministic optimal policies.

Original: In the following, we randomly choose an optimal deterministic policy π ∗ M + from Π ∗ M + , and randomly select one action ¯ a H from A H with ¯ a H ̸
Revised: In the following, we randomly choose an optimal deterministic policy π ∗ M + from Π ∗ M + , and randomly select one action ¯ a H from A H with ¯ a H ̸

Original: = a ∗ H and fix them in the following discussion.
Revised: = a ∗ H and fix them in the following discussion.

Original: Based on the definition above, we define a deterministic policy in M + , which agree with π ∗ M + for all states except s H :
Revised: Based on the definition above, we define a deterministic policy in M + , which agree with π ∗ M + for all states except s H :

Original: [Equation]
Revised: [Equation]

Original: Now, we are ready to design the adversarial choice of Alg O satisfying the condition 4.6.
Revised: Now, we are ready to design the adversarial choice of Alg O satisfying the condition 4.6.

Original: We consider the following algorithm:
Revised: We consider the following algorithm:

Original: [Equation]
Revised: [Equation]

Original: where k sup is defined to be:
Revised: where k sup is defined to be:

Original: [Equation]
Revised: [Equation]

Original: We can easily verify that Cond.
Revised: We can easily verify that Cond.

Original: 4.6 will not be violated, since ∀ k ≥ 1 ,
Revised: 4.6 will not be violated, since ∀ k ≥ 1 ,

Original: K (cid:88)
Revised: K (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Step 3: Lower Bound of Alg E under the Choice of Adversarial Alg O
Revised: Step 3: Lower Bound of Alg E under the Choice of Adversarial Alg O

Original: Now, we can derive an lower bound for Alg E .
Revised: Now, we can derive an lower bound for Alg E .

Original: Since in the first k sup steps, Alg E can only observe what happens if action ¯ a H is taken at s H,absorb , and therefore, it has no idea about which action among A H \ ¯ a H is the optimal action a ∗ H .
Revised: Since in the first k sup steps, Alg E can only observe what happens if action ¯ a H is taken at s H,absorb , and therefore, it has no idea about which action among A H \ ¯ a H is the optimal action a ∗ H .

Original: We use M + to denote a set of MDPs by permuting the position of a ∗ H in M + .
Revised: We use M + to denote a set of MDPs by permuting the position of a ∗ H in M + .

Original: Since |A
Revised: Since |A

Original: H | =
Revised: H | =

Original: A , we have |M + |
Revised: A , we have |M + |

Original: =
Revised: =

Original: A − 1 and M + ∈ M + .
Revised: A − 1 and M + ∈ M + .

Original: Then, we uniformly sample an MDP from M + and run the adversarial Alg O above to generate the data for Alg E to learn.
Revised: Then, we uniformly sample an MDP from M + and run the adversarial Alg O above to generate the data for Alg E to learn.

Original: We use M + i with i = 1 , 2 ..., A − 1 to refer to the MDPs in M + and use index i to refer to the position of the optimal action at s H,absorb in each MDP.
Revised: We use M + i with i = 1 , 2 ..., A − 1 to refer to the MDPs in M + and use index i to refer to the position of the optimal action at s H,absorb in each MDP.

Original: For the simplicity of the notation, we use A as the index to refer to the position of ¯ a H .
Revised: For the simplicity of the notation, we use A as the index to refer to the position of ¯ a H .

Original: Because Alg E do not have prior knowledge about which MDP in M + is sampled, we have:
Revised: Because Alg E do not have prior knowledge about which MDP in M + is sampled, we have:

Original: [Equation]
Revised: [Equation]

Original: (Drop the probability that π E k is sub-optimal at non-absorbing states)
Revised: (Drop the probability that π E k is sub-optimal at non-absorbing states)

Original: [Equation]
Revised: [Equation]

Original: A −k sup (cid:88) k
Revised: A −k sup (cid:88) k

Original: = (cid:16) (cid:88)
Revised: = (cid:16) (cid:88)

Original: j ∈
Revised: j ∈

Original: [ A ] ,j ̸ =
Revised: [ A ] ,j ̸ =

Original: A −
Revised: A −

Original: Pr Alg O ,M i ( π E k ( s H,absorb ) =
Revised: Pr Alg O ,M i ( π E k ( s H,absorb ) =

Original: j ) + (cid:88) i ∈
Revised: j ) + (cid:88) i ∈

Original: [ A − 2] (cid:88)
Revised: [ A − 2] (cid:88)

Original: j ∈
Revised: j ∈

Original: [ A ] ,j ̸ = i
Revised: [ A ] ,j ̸ = i

Original: Pr Alg O ,M i ( π E k ( s H,absorb ) =
Revised: Pr Alg O ,M i ( π E k ( s H,absorb ) =

Original: j ) (cid:17) =
Revised: j ) (cid:17) =

Original: d M, min ∆ min
Revised: d M, min ∆ min

Original: A −k sup (cid:88) k
Revised: A −k sup (cid:88) k

Original: = (cid:16) (cid:88) i ∈
Revised: = (cid:16) (cid:88) i ∈

Original: [ A − 2]
Revised: [ A − 2]

Original: Pr Alg O ,M i ( π E k ( s H,absorb )
Revised: Pr Alg O ,M i ( π E k ( s H,absorb )

Original: = i )
Revised: = i )

Original: + (cid:88)
Revised: + (cid:88)

Original: i ∈
Revised: i ∈

Original: [ A − 2] (cid:88)
Revised: [ A − 2] (cid:88)

Original: j ∈
Revised: j ∈

Original: [ A ] ,j ̸ = i
Revised: [ A ] ,j ̸ = i

Original: Pr Alg O ,M i ( π E k ( s H,absorb ) =
Revised: Pr Alg O ,M i ( π E k ( s H,absorb ) =

Original: j ) (cid:17) (Alg E can not distinguish between M + i ) =
Revised: j ) (cid:17) (Alg E can not distinguish between M + i ) =

Original: d M, min ∆ min
Revised: d M, min ∆ min

Original: A −k sup (cid:88) k
Revised: A −k sup (cid:88) k

Original: = (cid:16) (cid:88) i ∈
Revised: = (cid:16) (cid:88) i ∈

Original: [ A − 2] (cid:88)
Revised: [ A − 2] (cid:88)

Original: j ∈
Revised: j ∈

Original: [ A ]
Revised: [ A ]

Original: Pr Alg O ,M i ( π E k ( s H,absorb ) =
Revised: Pr Alg O ,M i ( π E k ( s H,absorb ) =

Original: j ) =
Revised: j ) =

Original: A −

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: 1 d M, min
Revised: A

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: sup =
Revised: sup =

Original: O (cid:16) (
Revised: O (cid:16) (

Original: [Equation]
Revised: [Equation]

Original: ) log
Revised: ) log

Original: C

Original: [Equation]
Revised: [Equation]

Original: min (cid:17)
Revised: min (cid:17)

Original: [Equation]
Revised: [Equation]

Original: ) log
Revised: log

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: min (cid:17)
Revised: (cid:17)

Original: D Analysis for Bandit Setting D.1 The Optimality of Alg O in Alg
Revised: D Analysis for Bandit Setting D.1 The Optimality of Alg O in Alg

Original: From Theorem 8.1 of [Lattimore and Szepesvári, 2020], we can show the following guarantee for the
Revised: From Theorem 8.1 of [Lattimore and Szepesvári, 2020], we can show the following guarantee for the

Original: UCB algorithm in Alg 2 with revised bonus function:
Revised: UCB algorithm in Alg 2 with revised bonus function:

Original: [Equation]
Revised: [Equation]

Original: where we assume K ≥ A .
Revised: where we assume K ≥ A .

Original: Since α is just at the constant level, the above regret still matches the lower bound.
Revised: Since α is just at the constant level, the above regret still matches the lower bound.

Original: D.2 Analysis for LCB
Revised: D.2 Analysis for LCB

Original: Outline In this section, we establish regret bound for Alg.
Revised: Outline In this section, we establish regret bound for Alg.

Original: We first provide the proof of two key
Revised: We first provide the proof of two key

Original: Lemma: Lem.
Revised: Lemma: Lem.

Original: 4.2 and Lem.
Revised: 4.2 and Lem.

Original: After that, in Lem.
Revised: After that, in Lem.

Original: 4.3, we try to combine the above two results and prove that for those arm i with ∆ i > 0 , when k is large enough, we are almost sure (with high probability) that LCB will not take arm i .
Revised: 4.3, we try to combine the above two results and prove that for those arm i with ∆ i > 0 , when k is large enough, we are almost sure (with high probability) that LCB will not take arm i .

Original: Finally, we conclude this section with the proof of Thm.
Revised: Finally, we conclude this section with the proof of Thm.

Original: 4.1. Definition of c f
Revised: 4.1. Definition of c f

Original: Under the choice f ( k ) = 1 + 16 A 2 ( k + 1) 2 , we use c f to denote the minimal positive constant independent with α, A and arbitrary ∆ i with ∆ i > 0 , such that ∀
Revised: Under the choice f ( k ) = 1 + 16 A 2 ( k + 1) 2 , we use c f to denote the minimal positive constant independent with α, A and arbitrary ∆ i with ∆ i > 0 , such that ∀

Original: [Equation]
Revised: [Equation]

Original: i > 0 ,
Revised: i > 0 ,

Original: [Equation]
Revised: [Equation]

Original: ∈ [1 , 4 A ] , as long as k ≥ c f αλ ∆ 2 i log(1 + αA ∆ i ) , we have k ≥ 32 αλ ∆i
Revised: ∈ [1 , 4 A ] , as long as k ≥ c f αλ ∆ 2 i log(1 + αA ∆ i ) , we have k ≥ 32 αλ ∆i

Original: [Equation]
Revised: [Equation]

Original: f ( k ) (11)
Revised: f ( k ) (11)

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: [Blessing of Pessimism] With the choice that f ( k ) = 

Revised: [Blessing of Pessimism] With the choice that f ( k ) =

Original: + 

Revised: +

Original: A 2 ( k + 1) 2 , for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, and arbitrary j satisfying ∆
Revised: A 2 ( k + 1) 2 , for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, and arbitrary j satisfying ∆

Original: j < ∆ i , we have:
Revised: j < ∆ i , we have:

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: < ∆ i } ∩ (cid:110) N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 (cid:111)(cid:17) ≤ 2 k 2 α .
Revised: < ∆ i } ∩ (cid:110) N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 (cid:111)(cid:17) ≤ 2 k 2 α .

Original: Proof.
Revised: Proof.

Original: Pr( { i = π E k } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )
Revised: Pr( { i = π E k } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )

Original: ≤ Pr( { (cid:98) µ i ( k ) − (cid:115) 

Revised: ≤ Pr( { (cid:98) µ i ( k ) − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N i ( k ) ≥ (cid:98) µ j ( k ) − (cid:115) 

Revised: N i ( k ) ≥ (cid:98) µ j ( k ) − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } ) =
Revised: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } ) =

Original: Pr( { (cid:98) µ i ( k ) − µ i − (cid:115) 

Revised: Pr( { (cid:98) µ i ( k ) − µ i − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (∆ j − ∆ i ) − (cid:115) 

Revised: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (∆ j − ∆ i ) − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )
Revised: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )

Original: ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115) 

Revised: ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (cid:115) 

Revised: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )
Revised: N j ( k ) } ∩ { ∆ j < ∆ i } ∩ { N j ( k ) ≥ 8 α log f ( k ) (∆ j − ∆ i ) 2 } )

Original: ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115) 

Revised: ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (cid:115) 

Revised: N i ( k ) ≥ (cid:98) µ j ( k ) − µ j + (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N j ( k ) } ) ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115) 

Revised: N j ( k ) } ) ≤ Pr( { (cid:98) µ i ( k ) − µ i − (cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N i ( k ) ≥ 0 } )
Revised: N i ( k ) ≥ 0 } )

Original: + Pr( { 0 ≥ (cid:98) µ j ( k ) − µ j
Revised: + Pr( { 0 ≥ (cid:98) µ j ( k ) − µ j

Original: +
Revised: +

Original: cid:115) 

Revised: cid:115)

Original: α log f ( k )
Revised: α log f ( k )

Original: N j ( k ) } )
Revised: N j ( k ) } )

Original: ≤ 2 /f ( k ) α ≤ 2 /k 2 α where the last but two step is because of the Azuma-Hoeffding’s inequality.
Revised: ≤ 2 /f ( k ) α ≤ 2 /k 2 α where the last but two step is because of the Azuma-Hoeffding’s inequality.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: [Property of UCB]
Revised: [Property of UCB]

Original: With the choice that f ( k ) = 1 + 

Revised: With the choice that f ( k ) = 1 +

Original: A 2 ( k + 1) 

Revised: A 2 ( k + 1)

Original: , there exists a constant c , for arbitrary i with ∆ i > 0 and arbitrary λ ∈ [1 , 4 A ] , in UCB algorithm, we have:
Revised: , there exists a constant c , for arbitrary i with ∆ i > 0 and arbitrary λ ∈ [1 , 4 A ] , in UCB algorithm, we have:

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We choose c f defined in Eq.(11) to be the constant c in this Lemma.
Revised: We choose c f defined in Eq.(11) to be the constant c in this Lemma.

Original: The key idea of the proof is that, because N i ( k ) ≤ k for all k , if N i ( k ) ≥ ⌈ k/λ ⌉ , there must exists an iteration (cid:101) k between ⌈ k/λ ⌉ − 1 and k , such that { N i ( (cid:101) k ) =
Revised: The key idea of the proof is that, because N i ( k ) ≤ k for all k , if N i ( k ) ≥ ⌈ k/λ ⌉ , there must exists an iteration (cid:101) k between ⌈ k/λ ⌉ − 1 and k , such that { N i ( (cid:101) k ) =

Original: ⌈ k/λ ⌉ − 1 } ∩ { N i ( (cid:101) k ) =
Revised: ⌈ k/λ ⌉ − 1 } ∩ { N i ( (cid:101) k ) =

Original: ⌈ k/λ ⌉} (i.e. (cid:101) k is the time step that UCB takes arm i for the ⌈ k/λ ⌉ -th time).
Revised: ⌈ k/λ ⌉} (i.e. (cid:101) k is the time step that UCB takes arm i for the ⌈ k/λ ⌉ -th time).

Original: Therefore, for arbitrary fixed λ ∈ [1 , A 2 ] , when k ≥ λ + c f · αλ ∆ 2 i log(1 + αλ ∆ min ) , we have:
Revised: Therefore, for arbitrary fixed λ ∈ [1 , A 2 ] , when k ≥ λ + c f · αλ ∆ 2 i log(1 + αλ ∆ min ) , we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ∗ + (cid:115) 

Revised: ∗ + (cid:115)

Original: α log f ( (cid:101) k )
Revised: α log f ( (cid:101) k )

Original: N i ∗ ( (cid:101) k ) ≤ 0 } )
Revised: N i ∗ ( (cid:101) k ) ≤ 0 } )

Original: + Pr( { N i ( (cid:101) k )
Revised: + Pr( { N i ( (cid:101) k )

Original: =
Revised: =

Original: ⌈ k/λ ⌉ − 1) , N i ( (cid:101) k + 1)
Revised: ⌈ k/λ ⌉ − 1) , N i ( (cid:101) k + 1)

Original: =
Revised: =

Original: ⌈ k/λ ⌉ ) } ∩ { 0 ≤ (cid:98) µ i ( (cid:101) k ) − µ i − (cid:115) 

Revised: ⌈ k/λ ⌉ ) } ∩ { 0 ≤ (cid:98) µ i ( (cid:101) k ) − µ i − (cid:115)

Original: α log f ( (cid:101) k )
Revised: α log f ( (cid:101) k )

Original: N i ( (cid:101) k ) } )
Revised: N i ( (cid:101) k ) } )

Original: (Under our choice of k , and N i ( (cid:101) k ) =
Revised: (Under our choice of k , and N i ( (cid:101) k ) =

Original: ⌈ k/λ ⌉ − 1 , log f ( (cid:101) k ) N i ( (cid:101) k ) ≤ log f ( k ) k/
Revised: ⌈ k/λ ⌉ − 1 , log f ( (cid:101) k ) N i ( (cid:101) k ) ≤ log f ( k ) k/

Original: λ − 1 ≤ ∆ 2 i 8 α ) ≤ k
Revised: λ − 1 ≤ ∆ 2 i 8 α ) ≤ k

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:88)
Revised: 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:115)
Revised: (cid:115)

Original: [Equation]
Revised: [Equation]

Original: α log
Revised: α log

Original: [Equation]
Revised: [Equation]

Original: ( (cid:101) k )
Revised: ( (cid:101) k )

Original: N i ( (cid:101) k ) } )
Revised: N i ( (cid:101) k ) } )

Original: [Equation]
Revised: [Equation]

Original: (Azuma-Hoeffding Inequality)
Revised: (Azuma-Hoeffding Inequality)

Original: [Equation]
Revised: [Equation]

Original: ( λ ∈ [
Revised: ( λ ∈ [

Original: [Equation]
Revised: [Equation]

Original: where the step (12) is because:
Revised: where the step (12) is because:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma D.1.
Revised: Lemma D.1.

Original: Given an arm i , we separate all the arms into two parts depending on whether its gap is larger than ∆ i and define G lower i :=
Revised: Given an arm i , we separate all the arms into two parts depending on whether its gap is larger than ∆ i and define G lower i :=

Original: { ι | ∆ ι > ∆ i / 2 } and G
Revised: { ι | ∆ ι > ∆ i / 2 } and G

Original: [Equation]
Revised: [Equation]

Original: i := { ι | ∆ ι ≤ ∆ i / 2 } .
Revised: i := { ι | ∆ ι ≤ ∆ i / 2 } .

Original: With the choice that f ( k ) = 1 + 

Revised: With the choice that f ( k ) = 1 +

Original: A 2 ( k + 1) 

Revised: A 2 ( k + 1)

Original: , there is a constant c , such that for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, we have:
Revised: , there is a constant c , such that for arbitrary i with ∆ i > 0 , for the LCB algorithm in Alg 2, we have:

Original: [Equation]
Revised: [Equation]

Original: where c is the constant considered in Lem.
Revised: where c is the constant considered in Lem.

Original: 4.3 (i.e. c f defined in Eq.
Revised: 4.3 (i.e. c f defined in Eq.

Original: Proof.
Revised: Proof.

Original: We want to remark that the constants in the definition of k i (i.e. 8 in “ 8 αc ” and 4 in “” 4 | G upper i | ”) can be replaced by others, but we choose them carefully in order to make sure some steps in the proof of this Lemma and Thm.
Revised: We want to remark that the constants in the definition of k i (i.e. 8 in “ 8 αc ” and 4 in “” 4 | G upper i | ”) can be replaced by others, but we choose them carefully in order to make sure some steps in the proof of this Lemma and Thm.

Original: 4.1 can go through.
Revised: 4.1 can go through.

Original: The main idea of the proof is to use Lem.
Revised: The main idea of the proof is to use Lem.

Original: 4.3 to show that, for those arm i with ∆ i > 0 , when k ≥ k
Revised: 4.3 to show that, for those arm i with ∆ i > 0 , when k ≥ k

Original: i ,
Revised: i ,

Original: N ι ( k ) will be small for those ι ∈ G lower i .
Revised: N ι ( k ) will be small for those ι ∈ G lower i .

Original: As a result, there must exist an arm j ∈ G upper i , such that
Revised: As a result, there must exist an arm j ∈ G upper i , such that

Original: N j ( k ) is large than the threshold considered in Lem.
Revised: N j ( k ) is large than the threshold considered in Lem.

Original: 4.2 and therefore, with high probability, arm i will not be preferred.
Revised: 4.2 and therefore, with high probability, arm i will not be preferred.

Original: First, we try to apply Lem.
Revised: First, we try to apply Lem.

Original: 4.3 to upper bound the quantity N j ( k ) for those arm j ∈ G lower i .
Revised: 4.3 to upper bound the quantity N j ( k ) for those arm j ∈ G lower i .

Original: For each j ∈ G lower i , we define the following quantity, which measures the magnitude of 1 / ∆ 2 j with k :
Revised: For each j ∈ G lower i , we define the following quantity, which measures the magnitude of 1 / ∆ 2 j with k :

Original: [Equation]
Revised: [Equation]

Original: We only consider k ≥ k i , where we always have γ k,j ≥ 1 based on the definition of k i .
Revised: We only consider k ≥ k i , where we always have γ k,j ≥ 1 based on the definition of k i .

Original: Next, we separately consider two cases depending on whether γ k,j > 2 A or not.
Revised: Next, we separately consider two cases depending on whether γ k,j > 2 A or not.

Original: Case 1: γ k,j > 2 A :
Revised: Case 1: γ k,j > 2 A :

Original: In this case, ∆ j is relatively large (or say more sub-optimal) comparing with iteration k .
Revised: In this case, ∆ j is relatively large (or say more sub-optimal) comparing with iteration k .

Original: For arbitrary k ≥ k i , we have
Revised: For arbitrary k ≥ k i , we have

Original: [Equation]
Revised: [Equation]

Original: which implies that k satisfying the condition of applying Lemma 4.3 with λ = 2 A , and we can conclude that:
Revised: which implies that k satisfying the condition of applying Lemma 4.3 with λ = 2 A , and we can conclude that:

Original: [Equation]
Revised: [Equation]

Original: Case 2: γ k,j ≤ 2 A : Note that
Revised: Case 2: γ k,j ≤ 2 A : Note that

Original: [Equation]
Revised: [Equation]

Original: Since 2 γ k,j locates in the interval [1 , 4 A ] and:
Revised: Since 2 γ k,j locates in the interval [1 , 4 A ] and:

Original: [Equation]
Revised: [Equation]

Original: which satisfies the condition of applying Lem.
Revised: which satisfies the condition of applying Lem.

Original: 4.3 with λ = 2 γ k,j .
Revised: 4.3 with λ = 2 γ k,j .

Original: Therefore, we have:
Revised: Therefore, we have:

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases, we can conclude that, for arbitrary j ∈ G lower
Revised: Combining the above two cases, we can conclude that, for arbitrary j ∈ G lower

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: which reflects that with high probability, (cid:80) j ∈ G lower
Revised: which reflects that with high probability, (cid:80) j ∈ G lower

Original: i
Revised: i

Original: N j ( k ) is small:
Revised: N j ( k ) is small:

Original: [Equation]
Revised: [Equation]

Original: Since (cid:80) j ∈ G upper
Revised: Since (cid:80) j ∈ G upper

Original: i
Revised: i

Original: N j ( k ) + (cid:80)
Revised: N j ( k ) + (cid:80)

Original: [Equation]
Revised: [Equation]

Original: lower
Revised: lower

Original: i
Revised: i

Original: N j ( k ) =
Revised: N j ( k ) =

Original: k , and note that, k − ( k
Revised: k , and note that, k − ( k

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: G lower
Revised: G lower

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, w.p, 1 − 2 A k 2 α − 1 , there exists j ∈ G upper i , such that
Revised: Therefore, w.p, 1 − 2 A k 2 α − 1 , there exists j ∈ G upper i , such that

Original: [Equation]
Revised: [Equation]

Original: Recall our choice of c (Eq.(11))
Revised: Recall our choice of c (Eq.(11))

Original: [Equation]
Revised: [Equation]

Original: the above implies that:
Revised: the above implies that:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: { i
Revised: { i

Original: [Equation]
Revised: [Equation]

Original: π E k } ∩ { k
Revised: E k } [Equation] { k

Original: [Equation]
Revised: [Equation]

Original: k i }
Revised: i }

Original: [Equation]
Revised: [Equation]

Original: Lemma D.2 (Integral Lemma) .
Revised: Lemma D.2 (Integral Lemma) .

Original: For arbitrary k 0 ≥ 1 and β >
Revised: For arbitrary k 0 ≥ 1 and β >

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: [Exploitation Regret]
Revised: [Exploitation Regret]

Original: In Algorithm 2, by choosing arbitrary α > 1 , there exists an absolute constant c , such that, for arbitrary K ≥ 1 , the pseduo-regret of Alg E is upper bounded by:
Revised: In Algorithm 2, by choosing arbitrary α > 1 , there exists an absolute constant c , such that, for arbitrary K ≥ 1 , the pseduo-regret of Alg E is upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: ∆
Revised: ∆

Original: [Equation]
Revised: [Equation]

Original: = ∞
Revised: = ∞

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the definition of k i in Eq.(13) in Lem
Revised: Recall the definition of k i in Eq.(13) in Lem

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: 1 above.
Revised: 1 above.

Original: For i ≥
Revised: For i ≥

Original: [Equation]
Revised: [Equation]

Original: if ∆ i ̸ = ∆ i − 1 , we have:
Revised: if ∆ i ̸ = ∆ i − 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: and if ∆ i = ∆ i − 1 , we also have:
Revised: and if ∆ i = ∆ i − 1 , we also have:

Original: [Equation]
Revised: [Equation]

Original: Moreover, for i = 1 , with the extended definition that ∆ 0 = ∞ (so that 1 / ∆ 20 = 0 ) and | G upper|
Revised: Moreover, for i = 1 , with the extended definition that ∆ 0 = ∞ (so that 1 / ∆ 20 = 0 ) and | G upper|

Original: =
Revised: =

Original: A , we also have:
Revised: A , we also have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have (we denote k A := ∞ and k 0 := 0 ): lim K →∞ Regret K ( Alg E ) = ∞ (cid:88) k
Revised: Therefore, we have (we denote k A := ∞ and k 0 := 0 ): lim K →∞ Regret K ( Alg E ) = ∞ (cid:88) k

Original: = (cid:88) j :∆ j >
Revised: = (cid:88) j :∆ j >

Original: Pr( j
Revised: Pr( j

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: A (cid:88) i
Revised: A (cid:88) i

Original: =
Revised: =

Original: k
Revised: k

Original: i (cid:88) k
Revised: i (cid:88) k

Original: = k
Revised: = k

Original: i − 1 + (cid:88) ∆
Revised: i − 1 + (cid:88) ∆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: A (cid:88) i
Revised: A (cid:88) i

Original: =
Revised: =

Original: k
Revised: k

Original: i (cid:88) k
Revised: i (cid:88) k

Original: = k
Revised: = k

Original: i − 1 + (cid:16) (cid:88)
Revised: i − 1 + (cid:16) (cid:88)

Original: ∆ j ≥
Revised: ∆ j ≥

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: term is maximized when Pr( i = π E k ) = 1 .)
Revised: term is maximized when Pr( i = π E k ) = 1 .)

Original: [Equation]
Revised: [Equation]

Original: (First term: Lemma D.2 and some simplification; Second term: Definition of k i .)
Revised: (First term: Lemma D.2 and some simplification; Second term: Definition of k i .)

Original: [Equation]
Revised: [Equation]

Original: According to the definition, we always have | G upper i − 1 | ≤ A − i + 1 , therefore, lim
Revised: According to the definition, we always have | G upper i − 1 | ≤ A − i + 1 , therefore, lim

Original: [Table]
Revised: [Table]

Original: ( Alg
Revised: ( Alg

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: E Behavior Analysis of Optimistic Algorithm
Revised: E Behavior Analysis of Optimistic Algorithm

Original: Definition E.1 (Definition of Events) .
Revised: Definition E.1 (Definition of Events) .

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: In another word, E k,h,π means π k disagrees with π at state s h which occurs at step h , (cid:101) E k,h,π means the first disagreement between π k and π occurs at step h , and ¯ E k,π denotes the event that there exists one state s h at some time step
Revised: In another word, E k,h,π means π k disagrees with π at state s h which occurs at step h , (cid:101) E k,h,π means the first disagreement between π k and π occurs at step h , and ¯ E k,π denotes the event that there exists one state s h at some time step

Original: h ∈
Revised: h ∈

Original: [ H ] such that π k agrees with π k at s h .
Revised: [ H ] such that π k agrees with π k at s h .

Original: Besides, E k,h denotes the events that π k,h
Revised: Besides, E k,h denotes the events that π k,h

Original: ( s h ) will not be taken by any optimal policy.
Revised: ( s h ) will not be taken by any optimal policy.

Original: Note that here we use Π ∗ h ( s h ) to denote the set of all possible optimal actions at state s h .
Revised: Note that here we use Π ∗ h ( s h ) to denote the set of all possible optimal actions at state s h .

Original: Given a deterministic optimal policy π ∗ , in general E k,h ̸ = E k,h,π ∗ when there are multiple optimal actions at one state.
Revised: Given a deterministic optimal policy π ∗ , in general E k,h ̸ = E k,h,π ∗ when there are multiple optimal actions at one state.

Original: Lemma E.2.
Revised: Lemma E.2.

Original: For arbitrary reward function R , given a fixed deterministic policy π , we have:
Revised: For arbitrary reward function R , given a fixed deterministic policy π , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma E
Revised: Lemma E

Original: [Equation]
Revised: [Equation]

Original: 3 (Relationship between Density Difference and Policy Disagreement Probability)
Revised: 3 (Relationship between Density Difference and Policy Disagreement Probability)

Original: [Equation]
Revised: [Equation]

Original: where we use Pr( ¯ E k,π | π k ) as a short note of E s
Revised: where we use Pr( ¯ E k,π | π k ) as a short note of E s

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: By applying Lemma E.2 with δ s h ,a h := I [ S h =
Revised: By applying Lemma E.2 with δ s h ,a h := I [ S h =

Original: s h , A h = a h ] as reward function, we have:
Revised: s h , A h = a h ] as reward function, we have:

Original: [Equation]
Revised: [Equation]

Original: which implies that,
Revised: which implies that,

Original: [Equation]
Revised: [Equation]

Original: Combining with d π k ≥ 0 , we finish the proof
Revised: Combining with d π k ≥ 0 , we finish the proof

Original: [Equation]
Revised: [Equation]

Original: Definition E.4 (Conversion to Optimal Deterministic Policy) .
Revised: Definition E.4 (Conversion to Optimal Deterministic Policy) .

Original: Given arbitrary deterministic policy π
Revised: Given arbitrary deterministic policy π

Original: = { π 1 , ..., π H } , we use Π ∗ ◦
Revised: = { π 1 , ..., π H } , we use Π ∗ ◦

Original: π = { π ∗ 1 , ..., π ∗ H } to denote an optimal deterministic policy, such that:
Revised: π = { π ∗ 1 , ..., π ∗ H } to denote an optimal deterministic policy, such that:

Original: [Equation]
Revised: [Equation]

Original: where Select is a function which returns the first optimal action from Π ∗ h ( s h ) .
Revised: where Select is a function which returns the first optimal action from Π ∗ h ( s h ) .

Original: In another word, Π ∗ ◦ π agrees with π if π h ( s h ) is one of the optimal action at state s h .
Revised: In another word, Π ∗ ◦ π agrees with π if π h ( s h ) is one of the optimal action at state s h .

Original: Otherwise, Π ∗ ◦ π takes one of a fixed optimal action from Π ∗ h ( s h ) .
Revised: Otherwise, Π ∗ ◦ π takes one of a fixed optimal action from Π ∗ h ( s h ) .

Original: In order to make sure Π ∗ ◦ is a deterministic mapping, we assume function Select only choose the first optimal action in Π ∗ ( s h ) (ordered by index of action).
Revised: In order to make sure Π ∗ ◦ is a deterministic mapping, we assume function Select only choose the first optimal action in Π ∗ ( s h ) (ordered by index of action).

Original: Theorem 4.7.
Revised: Theorem 4.7.

Original: For an arbitrary sequence of deterministic policies π 1 , π 2 , ..., π k , there must exist a sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ..., π ∗ k , such that ∀ h ∈
Revised: For an arbitrary sequence of deterministic policies π 1 , π 2 , ..., π k , there must exist a sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ..., π ∗ k , such that ∀ h ∈

Original: [ H ] , s h ∈ S h , a h ∈
Revised: [ H ] , s h ∈ S h , a h ∈

Original: A h :
Revised: A h :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For each π k , we construct an optimal deterministic policy π ∗ k :=
Revised: For each π k , we construct an optimal deterministic policy π ∗ k :=

Original: Π ∗ ◦ π k , where Π ∗ ◦ is defined in Def.
Revised: Π ∗ ◦ π k , where Π ∗ ◦ is defined in Def.

Original: E.4.
Revised: E.4.

Original: By applying Lemma E.2 with the reward function in MDP, and π = π ∗ k , we have:
Revised: By applying Lemma E.2 with the reward function in MDP, and π = π ∗ k , we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have:
Revised: Therefore, we have:

Original: [Equation]
Revised: [Equation]

Original: By applying Lemma E.3, we have:
Revised: By applying Lemma E.3, we have:

Original: [Equation]
Revised: [Equation]

Original: After the same discussion for all k ∈ [ K ]
Revised: After the same discussion for all k ∈ [ K ]

Original: [Equation]
Revised: [Equation]

Original: and the above inequality of each k together, we have:
Revised: and the above inequality of each k together, we have:

Original: [Equation]
Revised: [Equation]

Original: Corollary E.5 (Unique Optimal Policy) .
Revised: Corollary E.5 (Unique Optimal Policy) .

Original: When | Π ∗ | = 1 , Thm. 4.7 implies that:
Revised: When | Π ∗ | = 1 , Thm. 4.7 implies that:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.8.
Revised: Theorem 4.8.

Original: [The existance of well-covered optimal policy] Given an arbitrary tabular MDP, and an arbitrary sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ...
Revised: [The existance of well-covered optimal policy] Given an arbitrary tabular MDP, and an arbitrary sequence of deterministic optimal policies π ∗ 1 , π ∗ 2 , ...

Original: π ∗ k ( π ∗ i may not equal to π ∗ j for arbitrary 1 ≤ i < j ≤ k when there are multiple deterministic optimal policies), there exists a (possibly stochastic) policy π ∗ cover
Revised: π ∗ k ( π ∗ i may not equal to π ∗ j for arbitrary 1 ≤ i < j ≤ k when there are multiple deterministic optimal policies), there exists a (possibly stochastic) policy π ∗ cover

Original: such that ∀ h ∈
Revised: such that ∀ h ∈

Original: [ H ] , ∀
Revised: [ H ] , ∀

Original: ( s h , a h ) ∈ S
Revised: ( s h , a h ) ∈ S

Original: h ×
Revised: h ×

Original: A h with d π ∗ cover ( s h , a h ) > 0 : k
Revised: A h with d π ∗ cover ( s h , a h ) > 0 : k

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: where Z ∗ h, div :=
Revised: where Z ∗ h, div :=

Original: { ( s h , a h ) ∈ S
Revised: { ( s h , a h ) ∈ S

Original: h ×
Revised: h ×

Original: A h |∃ π ∗ , (cid:101)
Revised: A h |∃ π ∗ , (cid:101)

Original: π ∗ ∈
Revised: π ∗ ∈

Original: Π ∗ , s.t.
Revised: Π ∗ , s.t.

Original: d π ∗ ( s h ) > 0 , d (cid:101) π ∗ ( s h ) = 0 } , and d ∗ h, min ( s h , a h ) :
Revised: d π ∗ ( s h ) > 0 , d (cid:101) π ∗ ( s h ) = 0 } , and d ∗ h, min ( s h , a h ) :

Original: =
Revised: =

Original: min
Revised: min

Original: π ∗ ∈ Π ∗ d π ∗ ( s h , a h ) subject to d π ∗ ( s h , a h ) > 0 .
Revised: π ∗ ∈ Π ∗ d π ∗ ( s h , a h ) subject to d π ∗ ( s h , a h ) > 0 .

Original: Proof.
Revised: Proof.

Original: For arbitrary h ∈ [ H ] , we define:
Revised: For arbitrary h ∈ [ H ] , we define:

Original: [Equation]
Revised: [Equation]

Original: h , [Equation] In another word, N I ∗ K ( · , · ) denotes the number of optimal policies in the sequence, which can hit state s h and take action a h .
Revised: In another word, N I ∗ K ( · , · ) denotes the number of optimal policies in the sequence, which can hit state s h and take action a h .

Original: Next, we define Z ∗ h , Z insuff h and Π h, insuff as
Revised: Next, we define Z ∗ h , Z insuff h and Π h, insuff as

Original: [Equation]
Revised: [Equation]

Original: In a word, Z ∗ h is the collection of states actions reachable by at least one optimal policy, Z insuff h is a collection of “insufficiently hitted” states actions at step h , which are only covered by a small portion of optimal policies in the sequence, and I insuff h is a collection of the index of the optimal policies in the sequence, which cover at least one state action pair in Z insuff h .
Revised: In a word, Z ∗ h is the collection of states actions reachable by at least one optimal policy, Z insuff h is a collection of “insufficiently hitted” states actions at step h , which are only covered by a small portion of optimal policies in the sequence, and I insuff h is a collection of the index of the optimal policies in the sequence, which cover at least one state action pair in Z insuff h .

Original: Note that we must have Z insuff h ⊂
Revised: Note that we must have Z insuff h ⊂

Original: Z h, div ) , because if one state action pair s h , a h is reachable by arbitrary deterministic policy, then N I ∗ k
Revised: Z h, div ) , because if one state action pair s h , a h is reachable by arbitrary deterministic policy, then N I ∗ k

Original: ( s h , a h ) = K .
Revised: ( s h , a h ) = K .

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: We define I suff 1:
Revised: We define I suff 1:

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Intuitively, I suff 1: H is the set including the indices of optimal policies in the sequence only hitting those states which are covered by most of the other optimal policies.
Revised: Intuitively, I suff 1: H is the set including the indices of optimal policies in the sequence only hitting those states which are covered by most of the other optimal policies.

Original: In fact, I suff 1: H is non-empty since:
Revised: In fact, I suff 1: H is non-empty since:

Original: [Equation]
Revised: [Equation]

Original: We use π ∗ I suff 1: H to denote the average mixture policy over { π ∗ i : i ∈ I suff 1: H } , a direct result is that:
Revised: We use π ∗ I suff 1: H to denote the average mixture policy over { π ∗ i : i ∈ I suff 1: H } , a direct result is that:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, for all s h , a h such that d π ∗ I suff 1: H ( s h , a h ) > 0 , we must have ( s h , a h )
Revised: On the other hand, for all s h , a h such that d π ∗ I suff 1: H ( s h , a h ) > 0 , we must have ( s h , a h )

Original: ̸∈ Z insuff h , and therefore:
Revised: ̸∈ Z insuff h , and therefore:

Original: [Equation]
Revised: [Equation]

Original: Combining the above two inequalities, we finish the proof.
Revised: Combining the above two inequalities, we finish the proof.

Original: F Analysis of Pessimistic Value Iteration
Revised: F Analysis of Pessimistic Value Iteration

Original: In this section, we provide analysis for Alg. 3.
Revised: In this section, we provide analysis for Alg. 3.

Original: Our analyses base on an extension of the Clipping
Revised: Our analyses base on an extension of the Clipping

Original: Trick in [Simchowitz and Jamieson, 2019] into our setting.
Revised: Trick in [Simchowitz and Jamieson, 2019] into our setting.

Original: F.1 Underestimation and Some Concrete Choices of Bonus Term
Revised: F.1 Underestimation and Some Concrete Choices of Bonus Term

Original: Lemma F.1 (Underestimation) .
Revised: Lemma F.1 (Underestimation) .

Original: Given a Bonus satisfying Cond.
Revised: Given a Bonus satisfying Cond.

Original: 4.4, for arbitrary dataset D k consisting of k trajectories by a sequence of policies π 1 , ..., π k , by running Alg 3 with D k and the bonus term b ( · , · ) returned by Bonus ( D k , δ k ) , on the events E Bonus defined in Cond.
Revised: 4.4, for arbitrary dataset D k consisting of k trajectories by a sequence of policies π 1 , ..., π k , by running Alg 3 with D k and the bonus term b ( · , · ) returned by Bonus ( D k , δ k ) , on the events E Bonus defined in Cond.

Original: [Equation]
Revised: [Equation]

Original: where we use π (cid:98) Q = { π (cid:98) Q, 1 , ..., π (cid:98) Q,H } to denote the greedy policy w.r.t.
Revised: where we use π (cid:98) Q = { π (cid:98) Q, 1 , ..., π (cid:98) Q,H } to denote the greedy policy w.r.t.

Original: (cid:98) Q .
Revised: (cid:98) Q .

Original: Proof.
Revised: Proof.

Original: We only prove the first inequality holds, since the second one holds directly because of the definition of optimal policy.
Revised: We only prove the first inequality holds, since the second one holds directly because of the definition of optimal policy.

Original: First of all, V H +1 = 

Revised: First of all, V H +1 =

Original: ≤ V π (cid:98) Q H +1 holds directly, which implies that Eq.(16) holds at step h
Revised: ≤ V π (cid:98) Q H +1 holds directly, which implies that Eq.(16) holds at step h

Original: = H as a result of the deterministic reward function.
Revised: = H as a result of the deterministic reward function.

Original: Now, we conduct the induction.
Revised: Now, we conduct the induction.

Original: Suppose Eq.(16) already holds for h + 1 , which implies that:
Revised: Suppose Eq.(16) already holds for h + 1 , which implies that:

Original: [Equation]
Revised: [Equation]

Original: then, at step h , we have:
Revised: then, at step h , we have:

Original: [Equation]
Revised: [Equation]

Original: As we can see, part 1 is non-positive with probability 1 − δ as a result of Cond.
Revised: As we can see, part 1 is non-positive with probability 1 − δ as a result of Cond.

Original: 4.4, while part 2 is also less than or equal to zero because of the induction condition in Eq.(17).
Revised: 4.4, while part 2 is also less than or equal to zero because of the induction condition in Eq.(17).

Original: Choice 1: Naive Bound
Revised: Choice 1: Naive Bound

Original: According to Hoeffding inequality, with probability 1 − δ/ ( SAH ) , we have the following holds for each s h , a h , h
Revised: According to Hoeffding inequality, with probability 1 − δ/ ( SAH ) , we have the following holds for each s h , a h , h

Original: [Equation]
Revised: [Equation]

Original: which implies that condition 4.4 holds with probability 1 − δ as long as:
Revised: which implies that condition 4.4 holds with probability 1 − δ as long as:

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: Choice 2: Adaptive Bonus Term based on the “Bernstein Trick” One can also consider an analogue of the bonus term functions in Alg.
Revised: Choice 2: Adaptive Bonus Term based on the “Bernstein Trick” One can also consider an analogue of the bonus term functions in Alg.

Original: 3 of [Simchowitz and Jamieson, 2019], which is originally designed for optimistic algorithms.
Revised: 3 of [Simchowitz and Jamieson, 2019], which is originally designed for optimistic algorithms.

Original: We omit the discussions here.
Revised: We omit the discussions here.

Original: F.2 Definition of “Surplus” in Pessimistic Algorithms and the Clipping Trick
Revised: F.2 Definition of “Surplus” in Pessimistic Algorithms and the Clipping Trick

Original: We consider the pessimistic algorithm, and denote the estimation of value function as (cid:98) Q , (cid:98) V .
Revised: We consider the pessimistic algorithm, and denote the estimation of value function as (cid:98) Q , (cid:98) V .

Original: We assume they are pessimistic estimation, i.e.:
Revised: We assume they are pessimistic estimation, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Definition F.2 (Definition of Surplus in Pessimistic Algorithm setting) .
Revised: Definition F.2 (Definition of Surplus in Pessimistic Algorithm setting) .

Original: We define the surplus in
Revised: We define the surplus in

Original: Pessimistic Algorithm setting:
Revised: Pessimistic Algorithm setting:

Original: [Equation]
Revised: [Equation]

Original: Because of the underestimation, different from the surplus in overestimation cases [Simchowitz and Jamieson, 2019], here we flip the role between (cid:98) Q and r + P (cid:98) V to make sure the quantity is non-negative (with high probability).
Revised: Because of the underestimation, different from the surplus in overestimation cases [Simchowitz and Jamieson, 2019], here we flip the role between (cid:98) Q and r + P (cid:98) V to make sure the quantity is non-negative (with high probability).

Original: Based on our definition, we have the following lemma:
Revised: Based on our definition, we have the following lemma:

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Under the same condition as Lemma F.1, for arbitrary h, s h , the policy π PVI k returned by Alg.3 satisfying:
Revised: Under the same condition as Lemma F.1, for arbitrary h, s h , the policy π PVI k returned by Alg.3 satisfying:

Original: [Equation]
Revised: [Equation]

Original: Moreover, for arbitrary optimal deterministic or non-deterministic policy π ∗ , we have:
Revised: Moreover, for arbitrary optimal deterministic or non-deterministic policy π ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Besides, given arbitrary optimal policy π ∗ , we have:
Revised: Besides, given arbitrary optimal policy π ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: E a h ∼ π ∗
Revised: E a h ∼ π ∗

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: s h , a h ) + P h (cid:98) V k
Revised: s h , a h ) + P h (cid:98) V k

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) −
Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: Q k
Revised: Q k

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: a h ) + P h
Revised: a h ) + P h

Original: [Equation]
Revised: [Equation]

Original: V ∗ h
Revised: V ∗ h

Original: +1 −
Revised: +1 −

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: a h )] ≤ E π ∗ [
Revised: a h )] ≤ E π ∗ [

Original: H (cid:88) h ′ =
Revised: H (cid:88) h ′ =

Original: h
Revised: h

Original: E k,h ′ ( s h ′ , a h ′ )]
Revised: E k,h ′ ( s h ′ , a h ′ )]

Original: Lemma F.4.
Revised: Lemma F.4.

Original: Under the same condition as Lemma F.1, we have:
Revised: Under the same condition as Lemma F.1, we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: On the other hand, because the reward function is always locates in [0 , 1] and (cid:98) Q is always larger than zero, we have E k,
Revised: On the other hand, because the reward function is always locates in [0 , 1] and (cid:98) Q is always larger than zero, we have E k,

Original: h ( s h , a h ) ≤ H − h + 1 ≤ H .
Revised: h ( s h , a h ) ≤ H − h + 1 ≤ H .

Original: In the following, we define
Revised: In the following, we define

Original: [Equation]
Revised: [Equation]

Original: where ε Clip :=
Revised: where ε Clip :=

Original: ∆ min 2 H +2 , and Clip [ x | ε ] := x · I [ x ≥ ε ] .
Revised: ∆ min 2 H +2 , and Clip [ x | ε ] := x · I [ x ≥ ε ] .

Original: Then, we recursively define ¨ Q
Revised: Then, we recursively define ¨ Q

Original: [Equation]
Revised: [Equation]

Original: Note that although different optimal policies π ∗ and (cid:101) π ∗ have the same optimal value V ∗ , ¨ V π ∗ may no longer equal to ¨ V (cid:101) π ∗ because they may have different state occupancy and ¨ V depends on E .
Revised: Note that although different optimal policies π ∗ and (cid:101) π ∗ have the same optimal value V ∗ , ¨ V π ∗ may no longer equal to ¨ V (cid:101) π ∗ because they may have different state occupancy and ¨ V depends on E .

Original: Therefore, in the following, when we consider the ¨ V for optimal policies, we will always specify which optimal policy we are referring to.
Revised: Therefore, in the following, when we consider the ¨ V for optimal policies, we will always specify which optimal policy we are referring to.

Original: Lemma F.5 (Relationship between ¨ V π ∗ , V π PVI k and (cid:98) V k,h ) .
Revised: Lemma F.5 (Relationship between ¨ V π ∗ , V π PVI k and (cid:98) V k,h ) .

Original: Under the same condition as Lemma F.1, for arbitrary optimal policy π ∗ , we have:
Revised: Under the same condition as Lemma F.1, for arbitrary optimal policy π ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that:
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: (Underestimation (Lemma F.1))
Revised: (Underestimation (Lemma F.1))

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: F.3 Additional Lemma for the Analysis of the Regret of Alg E when Optimal Deterministic Policies are non-unique
Revised: F.3 Additional Lemma for the Analysis of the Regret of Alg E when Optimal Deterministic Policies are non-unique

Original: We first introduce a useful Lemma related to the clipping operator from [Simchowitz and Jamieson, 2019]
Revised: We first introduce a useful Lemma related to the clipping operator from [Simchowitz and Jamieson, 2019]

Original: Lemma F.6 (Lemma B.3 in [Simchowitz and Jamieson, 2019]) .
Revised: Lemma F.6 (Lemma B.3 in [Simchowitz and Jamieson, 2019]) .

Original: Let M ≥ 2 , a 1 , ...a m ≥ 0 and ε ≥ 0 .
Revised: Let M ≥ 2 , a 1 , ...a m ≥ 0 and ε ≥ 0 .

Original: Clip [ (cid:80) mi =1 a i | ε ] ≤ 2 (cid:80) mi =1 Clip [ a i | ε 2 m ] .
Revised: Clip [ (cid:80) mi =1 a i | ε ] ≤ 2 (cid:80) mi =1 Clip [ a i | ε 2 m ] .

Original: Next, based on definition of d min in Eq.(10), we have the following Lemma:
Revised: Next, based on definition of d min in Eq.(10), we have the following Lemma:

Original: Lemma F.7.
Revised: Lemma F.7.

Original: Given arbitrary deterministic policy π , if π ̸∈ Π ∗ , we have:
Revised: Given arbitrary deterministic policy π , if π ̸∈ Π ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use π ∗ :
Revised: We use π ∗ :

Original: = Π ∗ ◦ π to denote the converted deterministic optimal policy, where Π ∗ ◦ is defined in Def.
Revised: = Π ∗ ◦ π to denote the converted deterministic optimal policy, where Π ∗ ◦ is defined in Def.

Original: E.4.
Revised: E.4.

Original: As a direct application of Lemma E.2, we have:
Revised: As a direct application of Lemma E.2, we have:

Original: [Equation]
Revised: [Equation]

Original: where we use (cid:101) E h to denote the event that at step h , π first disagrees with π ∗ , or equivalently, π first take non-optimal action; in the second inequality, we define h init :
Revised: where we use (cid:101) E h to denote the event that at step h , π first disagrees with π ∗ , or equivalently, π first take non-optimal action; in the second inequality, we define h init :

Original: = min h ∈
Revised: = min h ∈

Original: [ H ] , s.t
Revised: [ H ] , s.t

Original: [Equation]
Revised: [Equation]

Original: Pr( (cid:101) E h ) > 0 .
Revised: Pr( (cid:101) E h ) > 0 .

Original: Besides, the last inequality is because:
Revised: Besides, the last inequality is because:

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: where the last step is because, according to the definition of h init , there is at least one s h ∈ S h
Revised: where the last step is because, according to the definition of h init , there is at least one s h ∈ S h

Original: such that I [ π ∗ h ( s h ) ̸
Revised: such that I [ π ∗ h ( s h ) ̸

Original: = π h
Revised: = π h

Original: ( s h )]
Revised: ( s h )]

Original: = 1 .
Revised: = 1 .

Original: F.4 Upper Bound for the Regret of Alg E
Revised: F.4 Upper Bound for the Regret of Alg E

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: By running Algorithm 3 with confidence level δ k , a function Bonus satisfying Condition 4.4, and a dataset D = { τ 1 , ...τ k } consisting of k complete trajectories generated by executing a sequence of policies π 1 , ..., π k , on the event E Bonus defined in Condition 4.4:
Revised: By running Algorithm 3 with confidence level δ k , a function Bonus satisfying Condition 4.4, and a dataset D = { τ 1 , ...τ k } consisting of k complete trajectories generated by executing a sequence of policies π 1 , ..., π k , on the event E Bonus defined in Condition 4.4:

Original: [Equation]
Revised: [Equation]

Original: Clip
Revised: Clip

Original: [Equation]
Revised: [Equation]

Original: where π ∗ can be an arbitrary optimal policy, ε Clip :
Revised: where π ∗ can be an arbitrary optimal policy, ε Clip :

Original: = ∆ min 2 H +2 if | Π ∗ | = 1 and ε Clip := d min ∆ min 2 SAH if | Π ∗ | > 1 , where d min :=
Revised: = ∆ min 2 H +2 if | Π ∗ | = 1 and ε Clip := d min ∆ min 2 SAH if | Π ∗ | > 1 , where d min :=

Original: min π ∈ Π ∗ ,h ∈
Revised: min π ∈ Π ∗ ,h ∈

Original: [ H ] ,s h ∈S h ,a h ∈A h
Revised: [ H ] ,s h ∈S h ,a h ∈A h

Original: d π ( s h , a h ) subject to d π ( s h , a h ) > 0 .
Revised: d π ( s h , a h ) subject to d π ( s h , a h ) > 0 .

Original: Proof.
Revised: Proof.

Original: We separately discuss the cases when there are unique or multiple deterministic optimal policies.
Revised: We separately discuss the cases when there are unique or multiple deterministic optimal policies.

Original: Case 1: Unique Deterministic Optimal Policy For arbitrary h, s h , suppose π PVI k
Revised: Case 1: Unique Deterministic Optimal Policy For arbitrary h, s h , suppose π PVI k

Original: ( s h )
Revised: ( s h )

Original: ̸∈ Π ∗ ( s h ) , we have:
Revised: ̸∈ Π ∗ ( s h ) , we have:

Original: [Equation]
Revised: [Equation]

Original: Recall the definition of Events in Def.
Revised: Recall the definition of Events in Def.

Original: E.1, and note that when the optimal policy is unique, the events (cid:101) E k,h,π ∗ , ¯ E k,h,π ∗ collapse to (cid:101) E k,h , ¯ E k,h , respectively.
Revised: E.1, and note that when the optimal policy is unique, the events (cid:101) E k,h,π ∗ , ¯ E k,h,π ∗ collapse to (cid:101) E k,h , ¯ E k,h , respectively.

Original: For arbitrary optimal policy π ∗ , we have:
Revised: For arbitrary optimal policy π ∗ , we have:

Original: [Equation]
Revised: [Equation]

Original: Besides
Revised: Besides

Original: [Equation]
Revised: [Equation]

Original: on the other hand,
Revised: on the other hand,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two results and Lemma F.
Revised: Combining the above two results and Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we finish the discussion for Case 1.
Revised: we finish the discussion for Case 1.

Original: Case 2: Non-unique Optimal Deterministic Policies From Lemma F.3, we know that,
Revised: Case 2: Non-unique Optimal Deterministic Policies From Lemma F.3, we know that,

Original: [Equation]
Revised: [Equation]

Original:  s h , a h )] where π ∗ can be arbitrary optimal policy.
Revised: ( s h , a h )] where π ∗ can be arbitrary optimal policy.

Original: Combining with Lemma F.7, we know that:
Revised: Combining with Lemma F.7, we know that:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because Clip [ αx | ε ] < α Clip [ x | ε ] as long as α < 1 .
Revised: where the last inequality is because Clip [ αx | ε ] < α Clip [ x | ε ] as long as α < 1 .

Original: Combining with
Revised: Combining with

Original: Lemma F.4, we finish the proof.
Revised: Lemma F.4, we finish the proof.

Original: Next, we introduce a useful Lemma from [Dann et al., 2017]:
Revised: Next, we introduce a useful Lemma from [Dann et al., 2017]:

Original: Lemma F.8 (Lemma 7.4 in [Dann et al., 2017]) .
Revised: Lemma F.8 (Lemma 7.4 in [Dann et al., 2017]) .

Original: Let F i for i, 1 ... be a filtration and X 1 , ...
Revised: Let F i for i, 1 ... be a filtration and X 1 , ...

Original: X n be a sequence of Bernoulli random variables with Pr( X i − 1 |F i − 1 )
Revised: X n be a sequence of Bernoulli random variables with Pr( X i − 1 |F i − 1 )

Original: =
Revised: =

Original: P i with P i being F i − 1 -measurable
Revised: P i with P i being F i − 1 -measurable

Original: and X i being F i measurable.
Revised: and X i being F i measurable.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Definition of Good Events We first introduce some notations about good events which holds with high probability.
Revised: Definition of Good Events We first introduce some notations about good events which holds with high probability.

Original: We override the definition in Cond.
Revised: We override the definition in Cond.

Original: 4.4 by assigning δ = δ k
Revised: 4.4 by assigning δ = δ k

Original: = 1 /k α at iteration k , i.e.:
Revised: = 1 /k α at iteration k , i.e.:

Original: [Equation]
Revised: [Equation]

Original: with b k = { b k, 1 , ..., b k,H } ← Bonus ( D k , 1 /k α ) .
Revised: with b k = { b k, 1 , ..., b k,H } ← Bonus ( D k , 1 /k α ) .

Original: Besides, we use E Con ,k to denote the concentration event that
Revised: Besides, we use E Con ,k to denote the concentration event that

Original: [Equation]
Revised: [Equation]

Original: Finally, we use E Alg O ,k to denote the good events that the regret of Alg O is only at the level log k :
Revised: Finally, we use E Alg O ,k to denote the good events that the regret of Alg O is only at the level log k :

Original: [Equation]
Revised: [Equation]

Original: Based on Cond.
Revised: Based on Cond.

Original: 4.4, Cond.
Revised: 4.4, Cond.

Original: 4.6 and Lemma F.8, we have:
Revised: 4.6 and Lemma F.8, we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma F.9.
Revised: Lemma F.9.

Original: [One Step Sub-optimality Gap Conditioning on Good Events] At iteration k , on the good events E Bonus ,k , E Con ,k and E Alg O ,k , the sub-optimality gap of π E k can be upper bounded by: i when | Π ∗ | = 1 (i.e. the optimal deterministic policy is unique):
Revised: [One Step Sub-optimality Gap Conditioning on Good Events] At iteration k , on the good events E Bonus ,k , E Con ,k and E Alg O ,k , the sub-optimality gap of π E k can be upper bounded by: (i) when | Π ∗ | = 1 (i.e. the optimal deterministic policy is unique):

Original: [Equation]
Revised: [Equation]

Original: (ii) when | Π ∗ | > 1 (i
Revised: (ii) when | Π ∗ | > 1 (i

Original: e. there are multiple optimal deterministic policy
Revised: e. there are multiple optimal deterministic policy):

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: where ε Clip :=
Revised: where ε Clip :=

Original: ∆ min 2 H +2 and ε ′
Revised: ∆ min 2 H +2 and ε ′

Original: Clip :
Revised: Clip :

Original: = d min ∆ min 2 SAH ; π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are defined in Thm. 4.8; besides, ¯
Revised: = d min ∆ min 2 SAH ; π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are defined in Thm. 4.8; besides, ¯

Original: [Equation]
Revised: [Equation]

Original: for some constant c τ , c ′ τ .
Revised: for some constant c τ , c ′ τ .

Original: Proof.
Revised: Proof.

Original: We first discuss the case when | Π ∗ | = 1 .
Revised: We first discuss the case when | Π ∗ | = 1 .

Original: Case 1: unique optimal deterministic policy As a result of Thm. 4.5, on the event E Bonus ,k , we show that the sub-optimality gap of π E k can be upper bounded by:
Revised: Case 1: unique optimal deterministic policy As a result of Thm. 4.5, on the event E Bonus ,k , we show that the sub-optimality gap of π E k can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Because of Lemma F.4, the above further implies that:
Revised: Because of Lemma F.4, the above further implies that:

Original: [Equation]
Revised: [Equation]

Original: Because of Thm. 4.7, on the event E Con ,k and E Alg O ,k , we further have:
Revised: Because of Thm. 4.7, on the event E Con ,k and E Alg O ,k , we further have:

Original: [Equation]
Revised: [Equation]

Original: Now, we define that,
Revised: Now, we define that,

Original: [Equation]
Revised: [Equation]

Original: there must exists a constant c τ independent with C 1 , C 2 , α, d π ∗ ( s h , a h ) and ∆ min , such that: ∀ h ∈ [ H ] , s h ∈ S h , a h ∈
Revised: there must exists a constant c τ independent with C 1 , C 2 , α, d π ∗ ( s h , a h ) and ∆ min , such that: ∀ h ∈ [ H ] , s h ∈ S h , a h ∈

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: Easy to check that, for arbitrary k ≥ ¯
Revised: Easy to check that, for arbitrary k ≥ ¯

Original: τ π ∗ s h ,a h , on the good events
Revised: τ π ∗ s h ,a h , on the good events

Original: , we can verify that N k,h ≥ k 4 d π ∗ ( s h , a h ) ≥ ¯ τ π ∗ sh,ahd π ∗ ( s h , a h ) ≥ c τ 4 > 0 , and as a result, we have:
Revised: , we can verify that N k,h ≥ k 4 d π ∗ ( s h , a h ) ≥ ¯ τ π ∗ sh,ahd π ∗ ( s h , a h ) ≥ c τ 4 > 0 , and as a result, we have:

Original: [Equation]
Revised: [Equation]

Original: h =
Revised: h =

Original: Clip (cid:104) min { H − h
Revised: Clip (cid:104) min { H − h

Original: + 1 , 2 B(cid:115)
Revised: + 1 , 2 B(cid:115)

Original: [Equation]
Revised: [Equation]

Original: B 2 k )
Revised: B 2 k )

Original: [Equation]
Revised: [Equation]

Original: Clip (cid:105)(cid:105)
Revised: Clip (cid:105)(cid:105)

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: Clip (cid:104) I
Revised: Clip (cid:104) I

Original: [ k
Revised: [ k

Original: [Equation]
Revised: [Equation]

Original: ¯ τ
Revised: ¯ τ

Original: [Equation]
Revised: [Equation]

Original: s h ,a h
Revised: s h ,a h

Original: [Equation]
Revised: [Equation]

Original: · H
Revised: · H

Original: [Equation]
Revised: [Equation]

Original: ¯
Revised: ¯

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: B(cid:115)
Revised: B(cid:115)

Original: [Equation]
Revised: [Equation]

Original: log( B 2 k )
Revised: log( B 2 k )

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: h )
Revised: h )

Original: [Equation]
Revised: [Equation]

Original: 4 (cid:12)(cid:12)(cid:12) ε Clip (cid:105)(cid:105)
Revised: 4 (cid:12)(cid:12)(cid:12) ε Clip (cid:105)(cid:105)

Original: Case 2: multiple optimal deterministic policies The discussion are similar.
Revised: Case 2: multiple optimal deterministic policies The discussion are similar.

Original: As a result of Thm.
Revised: As a result of Thm.

Original: 4.8, on the event E Con ,k and E Alg O ,k , we further have:
Revised: 4.8, on the event E Con ,k and E Alg O ,k , we further have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we define that,
Revised: we define that,

Original: [Equation]
Revised: [Equation]

Original: there must exists a constant c ′ τ independent with C 1 , C 2 , α, (cid:101) d π ∗ cover ( s h , a h ) and ∆ min
Revised: there must exists a constant c ′ τ independent with C 1 , C 2 , α, (cid:101) d π ∗ cover ( s h , a h ) and ∆ min

Original: [Equation]
Revised: [Equation]

Original: such that: ∀
Revised: such that: ∀

Original: [Equation]
Revised: [Equation]

Original: ∈ [ H ] , s h ∈ S h , a h ∈
Revised: ∈ [ H ] , s h ∈ S h , a h ∈

Original: A h ,
Revised: A h ,

Original: [Equation]
Revised: [Equation]

Original: For arbitrary k ≥ ¯ τ π ∗ cover s h ,a h , on the good events, we can verify that N k,h ≥ k 8 d π ∗ cover ( s h , a h ) > 0 , and as a result, we have:
Revised: For arbitrary k ≥ ¯ τ π ∗ cover s h ,a h , on the good events, we can verify that N k,h ≥ k 8 d π ∗ cover ( s h , a h ) > 0 , and as a result, we have:

Original: [Equation]
Revised: [Equation]

Original: Now, we are ready to prove the main theorem.
Revised: Now, we are ready to prove the main theorem.

Original: Theorem 4.9.
Revised: Theorem 4.9.

Original: By running an Algorithm satisfying Condition 4.6 as Alg O , running Alg 3 as Alg E with a bonus term function Bonus satisfying Condition 4.4 and δ k = 1 /k α , for some constant α > 1 , for arbitrary K ≥ 1 , the exploitation regret of Alg E can be upper bounded by:
Revised: By running an Algorithm satisfying Condition 4.6 as Alg O , running Alg 3 as Alg E with a bonus term function Bonus satisfying Condition 4.4 and δ k = 1 /k α , for some constant α > 1 , for arbitrary K ≥ 1 , the exploitation regret of Alg E can be upper bounded by:

Original: (i) When | Π ∗ | = 1 (unique optimal deterministic policy):
Revised: (i) When | Π ∗ | = 1 (unique optimal deterministic policy):

Original: [Equation]
Revised: [Equation]

Original: (ii) When | Π ∗ | > 1 (non-unique optimal deterministic policies): Regret K ( Alg E ) ≤ O
Revised: (ii) When | Π ∗ | > 1 (non-unique optimal deterministic policies): Regret K ( Alg E ) ≤ O

Original: [Equation]
Revised: [Equation]

Original: where π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are introduced in Theorem 4.8.
Revised: where π ∗ cover and (cid:101) d π ∗ cover ( s h , a h ) are introduced in Theorem 4.8.

Original: Proof.
Revised: Proof.

Original: Because the expectation and summation are linear, we have:
Revised: Because the expectation and summation are linear, we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, in the following, we first provide an upper bound for each E
Revised: Therefore, in the following, we first provide an upper bound for each E

Original: [ V ∗ − V π E k ] .
Revised: [ V ∗ − V π E k ] .

Original: Note that the expected regert at step k can be upper bounded by:
Revised: Note that the expected regert at step k can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: Easy to see that lim K →∞ (cid:80)
Revised: Easy to see that lim K →∞ (cid:80)

Original: Kk =1 1 k α < αα − 1 < ∞ as long as α > 1 , therefore, in the following, we mainly focus on the first part, and separately discuss its upper bound for the case when | Π ∗ | = 1 or | Π ∗ | > 1 .
Revised: Kk =1 1 k α < αα − 1 < ∞ as long as α > 1 , therefore, in the following, we mainly focus on the first part, and separately discuss its upper bound for the case when | Π ∗ | = 1 or | Π ∗ | > 1 .

Original: Case 1: | Π ∗ | = 1 (Unique Optimal Policy)
Revised: Case 1: | Π ∗ | = 1 (Unique Optimal Policy)

Original: We use π ∗ to denote the unique optimal policy and define:
Revised: We use π ∗ to denote the unique optimal policy and define:

Original: [Equation]
Revised: [Equation]

Original: Recall that ε Clip :=
Revised: Recall that ε Clip :=

Original: ∆ min / (2 H + 2) , it’s easy to verify that, there exists a constant c Clip such that,
Revised: ∆ min / (2 H + 2) , it’s easy to verify that, there exists a constant c Clip such that,

Original: [Equation]
Revised: [Equation]

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: For the first part
Revised: For the first part

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: For the second part
Revised: For the second part

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: where in the last step
Revised: where in the last step

Original: [Equation]
Revised: [Equation]

Original: we drop the term − (cid:112) ¯ τ π ∗ s h ,a h log B 2 ¯
Revised: we drop the term − (cid:112) ¯ τ π ∗ s h ,a h log B 2 ¯

Original: τ π ∗
Revised: τ π ∗

Original: s h ,a h
Revised: s h ,a h

Original: [Equation]
Revised: [Equation]

Original: and c 2 is a constant.
Revised: and c 2 is a constant.

Original: Combining the above results
Revised: Combining the above results

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: (cid:17) where c Alg E is some constant.
Revised: (cid:17) where c Alg E is some constant.

Original: Case 2: | Π ∗ | > 1 (Non-Unique Optimal Policy) Similar to the discussion above, we define:
Revised: Case 2: | Π ∗ | > 1 (Non-Unique Optimal Policy) Similar to the discussion above, we define:

Original: [Equation]
Revised: [Equation]

Original: Recall that ε ′
Revised: Recall that ε ′

Original: Clip :
Revised: Clip :

Original: = d min ∆ min / (2 SAH ) , it’s easy to verify that, there exists a constant c Clip such that,
Revised: = d min ∆ min / (2 SAH ) , it’s easy to verify that, there exists a constant c Clip such that,

Original: [Equation]
Revised: [Equation]

Original: Following a similar discussion, we have:
Revised: Following a similar discussion, we have:

Original: [Equation]
Revised: [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Lemma F.10 (Computation of Integral) .
Revised: Lemma F.10 (Computation of Integral) .

Original: Suppose p ≥ 1 , b ≥ a ≥ e/p , then we have:
Revised: Suppose p ≥ 1 , b ≥ a ≥ e/p , then we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: b a
Revised: b a

Original: [Equation]
Revised: [Equation]

Original: G Doubling Trick for Alg O Satisfying Cond. G. 
Revised: G Doubling Trick for Alg O Satisfying Cond. G.

Original: As we briefly mentioned in Sec.4.2.2, Cond.4.6 may not holds for some algorithms with near-optimal regret guarantees.
Revised: As we briefly mentioned in Sec.4.2.2, Cond.4.6 may not holds for some algorithms with near-optimal regret guarantees.

Original: For example, in [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021], although these algorithms are anytime, they require a confidence interval δ as input at the beginning of the algorithm and fix it during the running, which we abstract into the Cond.
Revised: For example, in [Simchowitz and Jamieson, 2019, Xu et al., 2021, Dann et al., 2021], although these algorithms are anytime, they require a confidence interval δ as input at the beginning of the algorithm and fix it during the running, which we abstract into the Cond.

Original: G.1 below:
Revised: G.1 below:

Original: Condition G.1 (Alternative Condition of Alg O ) .
Revised: Condition G.1 (Alternative Condition of Alg O ) .

Original: Alg O is an algorithm which returns a deterministic policies π O (cid:101) k at each iteration (cid:101) k , and for arbitrary fixed k ≥ 2 , with probability 1 − δ , we have the following holds:
Revised: Alg O is an algorithm which returns a deterministic policies π O (cid:101) k at each iteration (cid:101) k , and for arbitrary fixed k ≥ 2 , with probability 1 − δ , we have the following holds:

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 are some parameters depending on S, A, H and ∆ h ( s h , a h ) and independent with k .
Revised: where C 1 , C 2 are some parameters depending on S, A, H and ∆ h ( s h , a h ) and independent with k .

Original: As a result, no matter how small δ is chosen at the beginning, when k ≥ ⌈ 1 /δ ⌉ , the Cond.
Revised: As a result, no matter how small δ is chosen at the beginning, when k ≥ ⌈ 1 /δ ⌉ , the Cond.

Original: 4.6 can not be directly guaranteed.
Revised: 4.6 can not be directly guaranteed.

Original: To overcome this issue, we present a new framework in Alg 5 inspired by doubling trick.
Revised: To overcome this issue, we present a new framework in Alg 5 inspired by doubling trick.

Original: Algorithm 5:
Revised: Algorithm 5:

Original: RL Algorithm with
Revised: RL Algorithm with

Original: 1 Input : α > 1 .
Revised: 1 Input : α > 1 .

Original: 2 K 0 = 1 , k = 1 , π E 1 , 1 ← Alg E ( {} ) .
Revised: 2 K 0 = 1 , k = 1 , π E 1 , 1 ← Alg E ( {} ) .

Original: 3 for n = 1 , 2 , ... do
Revised: 3 for n = 1 , 2 , ... do

Original: K n ← 2 K n , δ n − 1 = 1 /K αn ,
Revised: K n ← 2 K n , δ n − 1 = 1 /K αn ,

Original: D n, 1 ← {} for k = 1 , ..., K n do //
Revised: D n, 1 ← {} for k = 1 , ..., K n do //

Original: Here we do not update π E π O n,k +1 ← Alg O ( D n,k , δ n ) .
Revised: Here we do not update π E π O n,k +1 ← Alg O ( D n,k , δ n ) .

Original: π E n,k +1 = (cid:40) π E n − 1 ,K n − 1 / 2+ ⌈ k/ 2 ⌉ , If k ≤ K n / 2 ,
Revised: π E n,k +1 = (cid:40) π E n − 1 ,K n − 1 / 2+ ⌈ k/ 2 ⌉ , If k ≤ K n / 2 ,

Original: Alg E ( D n,k , 1 /k α ) ,
Revised: Alg E ( D n,k , 1 /k α ) ,

Original: Otherwise .
Revised: Otherwise .

Original: τ k +1 ∼ π O n,k +
Revised: τ k +1 ∼ π O n,k +

Original: D n,k
Revised: D n,k

Original: = D n,k ∪ τ n k +  end 12 end
Revised: = D n,k ∪ τ n,k +  end 12 end

Original: The basic idea is to iteratively run Alg O satisfying Cond.
Revised: The basic idea is to iteratively run Alg O satisfying Cond.

Original: G.1 from scratch while gradually doubling the number of iterations (i.e. K n ) and shrinking the confidence level δ n rather than runnning with a fixed δ forever.
Revised: G.1 from scratch while gradually doubling the number of iterations (i.e. K n ) and shrinking the confidence level δ n rather than runnning with a fixed δ forever.

Original: Besides, another crucial part is the computation of π E k .
Revised: Besides, another crucial part is the computation of π E k .

Original: Instead of continuously updating π E with the data collected before, we only update the exploitation policy when k ≥ K n /for each outer loop n .
Revised: Instead of continuously updating π E with the data collected before, we only update the exploitation policy when k ≥ K n /for each outer loop n .

Original: As we will discuss in Lemma G.2, Alg E n,k will behave as if the dataset is generated by another online algorithm satisfying Cond. 4.6, and therefore, the analysis based on
Revised: As we will discuss in Lemma G.2, Alg E n,k will behave as if the dataset is generated by another online algorithm satisfying Cond. 4.6, and therefore, the analysis based on

Original: Cond.
Revised: Cond.

Original: 4.6 can be adapted here, which we summarize to Thm.
Revised: 4.6 can be adapted here, which we summarize to Thm.

Original: G.3 below.
Revised: G.3 below.

Original: Lemma G.2.
Revised: Lemma G.2.

Original: By running an algorithm satisfying Cond.
Revised: By running an algorithm satisfying Cond.

Original: G.1 in Alg.
Revised: G.1 in Alg.

Original: 5 as Alg O , for arbitrary n ≥and K n / 2 + 1 ≤ k < K n / 2 , we have:
Revised: 5 as Alg O , for arbitrary n ≥and K n / 2 + 1 ≤ k < K n / 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: with C ′ 1 = C 1 + ( α + 1) C 2 log 2 and C ′ 2 = α +1 α C 2 .
Revised: with C ′ 1 = C 1 + ( α + 1) C 2 log 2 and C ′ 2 = α +1 α C 2 .

Original: Proof.
Revised: Proof.

Original: Based on Cond.
Revised: Based on Cond.

Original: G.1, we know that:
Revised: G.1, we know that:

Original: [Equation]
Revised: [Equation]

Original: Since δ n = 1 /K αn and k ≥ K n / 2 , we have:
Revised: Since δ n = 1 /K αn and k ≥ K n / 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we are ready to upper bound the regret of Alg E :
Revised: we are ready to upper bound the regret of Alg E :

Original: Theorem G
Revised: Theorem G

Original: [Equation]
Revised: [Equation]

Original: By choosing an arbitrary algorithm satisfying Cond.
Revised: By choosing an arbitrary algorithm satisfying Cond.

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: 1 as Alg O , choosing Alg.
Revised: 1 as Alg O , choosing Alg.

Original: 3 as
Revised: 3 as

Original: Alg E and choosing a bonus function satisfying Cond.
Revised: Alg E and choosing a bonus function satisfying Cond.

Original: [Equation]
Revised: [Equation]

Original: 4 as Bonus
Revised: 4 as Bonus

Original: [Equation]
Revised: [Equation]

Original: the Pseudo regret of π E n,k in Alg.
Revised: the Pseudo regret of π E n,k in Alg.

Original: 5 can be upper bounded by:
Revised: 5 can be upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 1 (unique optimal deterministic policy):
Revised: 1 (unique optimal deterministic policy):

Original: [Equation]
Revised: [Equation]

Original: (ii) | Π ∗ | > 1 (non-unique optimal deterministic policies):
Revised: (ii) | Π ∗ | > 1 (non-unique optimal deterministic policies):

Original: [Equation]
Revised: [Equation]

Original: where C ′ 

Revised: where C ′

Original: =
Revised: =

Original: C 1 + ( α + 1) C 2 log 2 and C ′ 

Revised: C 1 + ( α + 1) C 2 log 2 and C ′

Original: = α +1 α C 2 .
Revised: = α +1 α C 2 .

Original: Remark G.4 ( O (log 2 K ) -Regret of Alg O ) .
Revised: Remark G.4 ( O (log 2 K ) -Regret of Alg O ) .

Original: Although the regret of Alg E stays constant under this framework, it is easy to verify that the pseudo-regret of Alg O will be O (log 2 K ) as a result of the doubling trick, which is worse than O (log K ) up to a factor of log K .
Revised: Although the regret of Alg E stays constant under this framework, it is easy to verify that the pseudo-regret of Alg O will be O (log 2 K ) as a result of the doubling trick, which is worse than O (log K ) up to a factor of log K .

Original: Therefore, more rigorously speaking, the regret of Alg O will be almost near-optimal.
Revised: Therefore, more rigorously speaking, the regret of Alg O will be almost near-optimal.

Original: Proof.
Revised: Proof.

Original: The key observation is that one can decompose the total expected regret into two parts:
Revised: The key observation is that one can decompose the total expected regret into two parts:

Original: [Equation]
Revised: [Equation]

Original: Therefore, all we need to do is to upper bound the second part of Eq.(18).
Revised: Therefore, all we need to do is to upper bound the second part of Eq.(18).

Original: As a result of Lemma G.2, we can apply Lemma F.9 to upper bound the regret of the policy sequence {{ π E n,k } Nn =1 } K n k = K n / 2+1 , since the Cond.
Revised: As a result of Lemma G.2, we can apply Lemma F.9 to upper bound the regret of the policy sequence {{ π E n,k } Nn =1 } K n k = K n / 2+1 , since the Cond.

Original: 4.6 is satisfied when generating those policies.
Revised: 4.6 is satisfied when generating those policies.

Original: Therefore, the Pseudo regret
Revised: Therefore, the Pseudo regret

Original: E [ (cid:80) N n =1 (cid:80) K n k = K n / 2+1 V ∗ − V π E n,k ] can be upper bounded by extending the results in Thm.
Revised: E [ (cid:80) N n =1 (cid:80) K n k = K n / 2+1 V ∗ − V π E n,k ] can be upper bounded by extending the results in Thm.

Original: 4.9 here, and we finish the proof.
Revised: 4.9 here, and we finish the proof.

Original: H Experiments H.1 Experiment Setup
Revised: H Experiments H.1 Experiment Setup

Original: Environment We test our algorithms in tabular MDPs with randomly generated transition and rewards functions.
Revised: Environment We test our algorithms in tabular MDPs with randomly generated transition and rewards functions.

Original: To generate the MDP, for each layer h and each state action pair ( s h , a h ) , we first sample a random vector P ( ·| s h , a h ) , where each element is uniformly sampled from { 1 , 2 , 3 ..., 10 } , and then normalize it to a valid probability vector.
Revised: To generate the MDP, for each layer h and each state action pair ( s h , a h ) , we first sample a random vector P ( ·| s h , a h ) , where each element is uniformly sampled from { 1 , 2 , 3 ..., 10 } , and then normalize it to a valid probability vector.

Original: Besides, the reward function is set to ξ/ 10 where ξ is randomly generated from { 1 , 2 , ..., 10 } to make sure it locates in [0 , 1] .
Revised: Besides, the reward function is set to ξ/ 10 where ξ is randomly generated from { 1 , 2 , ..., 10 } to make sure it locates in [0 , 1] .

Original: Algorithm We implement the StrongEuler algorithm in [Simchowitz and Jamieson, 2019] as Alg O and construct the same adaptive bonus term (Alg. 3 in [Simchowitz and Jamieson, 2019]) for Alg E to match Cond. 4.4.
Revised: Algorithm We implement the StrongEuler algorithm in [Simchowitz and Jamieson, 2019] as Alg O and construct the same adaptive bonus term (Alg. 3 in [Simchowitz and Jamieson, 2019]) for Alg E to match Cond. 4.4.

Original: Although for the convenience of analysis, in our Framework 1, we do not consider to use the data generated by Alg E , in experiments, we use both τ O and τ E , which slightly improves the performance.
Revised: Although for the convenience of analysis, in our Framework 1, we do not consider to use the data generated by Alg E , in experiments, we use both τ O and τ E , which slightly improves the performance.

Original: Besides, in practice, we observe that the bonus term is quite loose, and it will take a long time before the estimated Q/V value fallen in the interval [0 , H ] , which is the value range of true value functions.
Revised: Besides, in practice, we observe that the bonus term is quite loose, and it will take a long time before the estimated Q/V value fallen in the interval [0 , H ] , which is the value range of true value functions.

Original: Therefore, we introduce a multiplicator α and adjust the bonus term from b k,h to α · b k,h , and set α = 0 .
Revised: Therefore, we introduce a multiplicator α and adjust the bonus term from b k,h to α · b k,h , and set α = 0 .

Original: 25 for both Alg O and Alg E .
Revised: 25 for both Alg O and Alg E .

Original: H.2 Results
Revised: H.2 Results

Original: We test the algorithms in tabular MDPs with S =
Revised: We test the algorithms in tabular MDPs with S =

Original: A = H = 5 5 .
Revised: A = H = 5 5 .

Original: Although the minimal gap ∆ min is hard to control since we generate the MDP in a random way, we filter out three random seeds in
Revised: Although the minimal gap ∆ min is hard to control since we generate the MDP in a random way, we filter out three random seeds in

Original: MDP construction, which correspond to minimal gaps (approximately) equal to 0.0015, 0.003 and 0.009, respectively.
Revised: MDP construction, which correspond to minimal gaps (approximately) equal to 0.0015, 0.003 and 0.009, respectively.

Original: We report the accumulative regret in Fig.
Revised: We report the accumulative regret in Fig.

Original: As predicted by our theory, Alg E can indeed achieve constant regret in contrast with the continuously increasing regret of Alg O , which demonstrates the advantage of leveraging tiered structure.
Revised: As predicted by our theory, Alg E can indeed achieve constant regret in contrast with the continuously increasing regret of Alg O , which demonstrates the advantage of leveraging tiered structure.