Original: O N THE P OWER OF P RE - TRAINING FOR G ENERALIZATION IN RL: P ROVABLE B ENEFITS AND H ARDNESS
Revised: O N THE P OWER OF P RE - TRAINING FOR G ENERALIZATION IN RL: P ROVABLE B ENEFITS AND H ARDNESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generalization in Reinforcement Learning (RL) aims to train an agent during training that generalizes to the target environment.
Revised: Generalization in Reinforcement Learning (RL) aims to train an agent during training that generalizes to the target environment.

Original: In this work, we first point out that RL generalization is fundamentally different from the generalization in supervised learning, and fine-tuning on the target environment is necessary for good test performance.
Revised: In this work, we first point out that RL generalization is fundamentally different from the generalization in supervised learning, and fine-tuning on the target environment is necessary for good test performance.

Original: Therefore, we seek to answer the following question: how much can we expect pre-training over training environments to be helpful for efficient and effective fine-tuning?
Revised: Therefore, we seek to answer the following question: how much can we expect pre-training over training environments to be helpful for efficient and effective fine-tuning?

Original: On one hand, we give a surprising result showing that asymptotically, the improvement from pre-training is at most a constant factor.
Revised: On one hand, we give a surprising result showing that asymptotically, the improvement from pre-training is at most a constant factor.

Original: On the other hand, we show that pre-training can be indeed helpful in the non-asymptotic regime by designing a policy collection-elimination (PCE) algorithm and proving a distribution-dependent regret bound that is independent of the state-action space.
Revised: On the other hand, we show that pre-training can be indeed helpful in the non-asymptotic regime by designing a policy collection-elimination (PCE) algorithm and proving a distribution-dependent regret bound that is independent of the state-action space.

Original: We hope our theoretical results can provide insight towards understanding pre-training and generalization in RL.
Revised: We hope our theoretical results can provide insight towards understanding pre-training and generalization in RL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) is concerned with sequential decision making problems in which the agent interacts with the environment aiming to maximize its cumulative reward.
Revised: Reinforcement learning (RL) is concerned with sequential decision making problems in which the agent interacts with the environment aiming to maximize its cumulative reward.

Original: This framework has achieved tremendous successes in various fields such as game playing (Mnih et al., 2013; Silver et al., 2017; Vinyals et al., 2019), resource management (Mao et al., 2016), recommendation systems (Shani et al., 2005; Zheng et al., 2018) and online advertising (Cai et al., 2017).
Revised: This framework has achieved tremendous successes in various fields such as game playing (Mnih et al., 2013; Silver et al., 2017; Vinyals et al., 2019), resource management (Mao et al., 2016), recommendation systems (Shani et al., 2005; Zheng et al., 2018) and online advertising (Cai et al., 2017).

Original: However, many empirical applications of RL algorithms are typically restricted to the single environment setting.
Revised: However, many empirical applications of RL algorithms are typically restricted to the single environment setting.

Original: That is, the RL policy is learned and evaluated in the exactly same environment.
Revised: That is, the RL policy is learned and evaluated in the exactly same environment.

Original: This learning paradigm can lead to the issue of overfitting in RL (Sutton, 1995; Farebrother et al., 2018), and may have degenerate performance when the agent is deployed to an unseen (but similar) environment.
Revised: This learning paradigm can lead to the issue of overfitting in RL (Sutton, 1995; Farebrother et al., 2018), and may have degenerate performance when the agent is deployed to an unseen (but similar) environment.

Original: The ability to generalize to test environments is important to the success of reinforcement learning algorithms, especially in the real applications such as autonomous driving (Shalev-Shwartz et al., 2016; Sallab et al., 2017), robotics (Kober et al., 2013; Kormushev et al., 2013) and health care (Yu et al., 2021).
Revised: The ability to generalize to test environments is important to the success of reinforcement learning algorithms, especially in the real applications such as autonomous driving (Shalev-Shwartz et al., 2016; Sallab et al., 2017), robotics (Kober et al., 2013; Kormushev et al., 2013) and health care (Yu et al., 2021).

Original: In these real-world tasks, the environment can be dynamic, open-ended and always changing.
Revised: In these real-world tasks, the environment can be dynamic, open-ended and always changing.

Original: We hope the agent can learn meaningful skills in the training stage and be robust to the variation during the test stage.
Revised: We hope the agent can learn meaningful skills in the training stage and be robust to the variation during the test stage.

Original: Furthermore, in applications such as robotics where we have a simulator to efficiently and safely generate unlimited data, we can firstly train the agent in the randomized simulator models and then generalize it to the real environment (Rusu et al., 2017; Peng et al., 2018; Andrychowicz et al., 2020).
Revised: Furthermore, in applications such as robotics where we have a simulator to efficiently and safely generate unlimited data, we can firstly train the agent in the randomized simulator models and then generalize it to the real environment (Rusu et al., 2017; Peng et al., 2018; Andrychowicz et al., 2020).

Original: An RL algorithm with good generalization ability can greatly reduce the demand of real-world data and improve test-time performance.
Revised: An RL algorithm with good generalization ability can greatly reduce the demand of real-world data and improve test-time performance.

Original: Generalization in supervised learning has been widely studied for decades (Mitchell et al., 1986;
Revised: Generalization in supervised learning has been widely studied for decades (Mitchell et al., 1986;

Original: Bousquet & Elisseeff, 2002; Kawaguchi et al., 2017).
Revised: Bousquet & Elisseeff, 2002; Kawaguchi et al., 2017).

Original: For a typical supervised learning task such as classification, given a hypothesis space H and a loss function ℓ , the agent aims to find an optimal solution in the average manner.
Revised: For a typical supervised learning task such as classification, given a hypothesis space H and a loss function ℓ , the agent aims to find an optimal solution in the average manner.

Original: That is, we hope the solution is near-optimal compared with the optimal hypothesis h ∗ in expectation over the data distribution, which is formally defined as h ∗ =
Revised: That is, we hope the solution is near-optimal compared with the optimal hypothesis h ∗ in expectation over the data distribution, which is formally defined as h ∗ =

Original: arg min h ∈H E (cid:2) ℓ ( h ( X ) , Y ) (cid:3) .
Revised: arg min h ∈H E (cid:2) ℓ ( h ( X ) , Y ) (cid:3) .

Original: From this perspective, generalization in RL is fundamentally different.
Revised: From this perspective, generalization in RL is fundamentally different.

Original: Once the agent is deployed in the test environment M sampled from distribution D , it is expected to achieve comparable performance with the optimal policy in M .
Revised: Once the agent is deployed in the test environment M sampled from distribution D , it is expected to achieve comparable performance with the optimal policy in M .

Original: In other words, we hope the learned policy can perform near-optimal compared with the optimal value V ∗M in instance for the sampled test environment M .
Revised: In other words, we hope the learned policy can perform near-optimal compared with the optimal value V ∗M in instance for the sampled test environment M .

Original: Unfortunately, as discussed in many previous works (Malik et al., 2021; Ghosh et al., 2021), the instance-optimal solution in the target environment can be statistically intractable without additional assumptions.
Revised: Unfortunately, as discussed in many previous works (Malik et al., 2021; Ghosh et al., 2021), the instance-optimal solution in the target environment can be statistically intractable without additional assumptions.

Original: We formulate this intractability into a lower bound (Proposition 1) to show that it is impractical to directly obtain a near-optimal policy for the test environment M ∗ with high probability.
Revised: We formulate this intractability into a lower bound (Proposition 1) to show that it is impractical to directly obtain a near-optimal policy for the test environment M ∗ with high probability.

Original: This motivates us to ask: in what settings can the generalization problem in RL be tractable?
Revised: This motivates us to ask: in what settings can the generalization problem in RL be tractable?

Original: Targeting on RL generalization, the agent is often allowed to further interact with the test environment to improve its policy.
Revised: Targeting on RL generalization, the agent is often allowed to further interact with the test environment to improve its policy.

Original: For example, many previous results in robotics have demonstrated that fine-tuning in the test environment can greatly improve the test performance for sim-to-real transfer (Rusu et al., 2017; James et al., 2019; Rajeswaran et al., 2016).
Revised: For example, many previous results in robotics have demonstrated that fine-tuning in the test environment can greatly improve the test performance for sim-to-real transfer (Rusu et al., 2017; James et al., 2019; Rajeswaran et al., 2016).

Original: Therefore, one possible way to formulate generalization is to allow further interaction with the target environment during the test stage.
Revised: Therefore, one possible way to formulate generalization is to allow further interaction with the target environment during the test stage.

Original: Specifically, suppose the agent interacts with MDP M ∼ D in the test stage, and we measure the performance of the fine-tuning algorithm
Revised: Specifically, suppose the agent interacts with MDP M ∼ D in the test stage, and we measure the performance of the fine-tuning algorithm

Original: A using the expected regret in K episodes, i.e.
Revised: A using the expected regret in K episodes, i.e.

Original: Reg
Revised: Reg

Original: [Equation]
Revised: [Equation]

Original: In this setting, can the information obtained from pre-training 1 help reduce the regret suffered during the test stage?
Revised: In this setting, can the information obtained from pre-training 1 help reduce the regret suffered during the test stage?

Original: In addition, when the test-time fine-tuning is not allowed, what can we expect the pre-training to be helpful?
Revised: In addition, when the test-time fine-tuning is not allowed, what can we expect the pre-training to be helpful?

Original: As discussed above, we can no longer demand instance-optimality in this setting, but can only step back and pursue a near-optimal policy in expectation .
Revised: As discussed above, we can no longer demand instance-optimality in this setting, but can only step back and pursue a near-optimal policy in expectation .

Original: Specifically, our goal is to perform near-optimal in terms of the optimal policy with maximum value in expectation, i.e. π ∗ ( D ) =
Revised: Specifically, our goal is to perform near-optimal in terms of the optimal policy with maximum value in expectation, i.e. π ∗ ( D ) =

Original: arg max π ∈ Π E
Revised: arg max π ∈ Π E

Original: M∼ D V π M .
Revised: M∼ D V π M .

Original: Here V π M is the value function of the policy π in MDP M .
Revised: Here V π M is the value function of the policy π in MDP M .

Original: We seek to answer: is it possible to design a sample-efficient training algorithm that returns a ϵ -optimal policy π in expectation , i.e. E M∼ D (cid:104)
Revised: We seek to answer: is it possible to design a sample-efficient training algorithm that returns a ϵ -optimal policy π in expectation , i.e. E M∼ D (cid:104)

Original: V π ∗ ( D ) M
Revised: V π ∗ ( D ) M

Original: [Equation]
Revised: [Equation]

Original: Main contributions.
Revised: Main contributions.

Original: In this paper, we theoretically study RL generalization in the above two settings.
Revised: In this paper, we theoretically study RL generalization in the above two settings.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • When fine-tuning is allowed, we study the benefit of pre-training for the test-time performance.
Revised: • When fine-tuning is allowed, we study the benefit of pre-training for the test-time performance.

Original: Since
Revised: Since

Original: all
Revised: all

Original: information
Revised: information

Original: we
Revised: we

Original: can
Revised: can

Original: gain from
Revised: gain from

Original: training
Revised: training

Original: is no
Revised: is no

Original: more than the distribution
Revised: more than the distribution

Original: D
Revised: D

Original: itself, we start with a
Revised: itself, we start with a

Original: somewhat
Revised: somewhat

Original: surprising theorem
Revised: surprising theorem

Original: showing the limitation of this benefit: there exists hard cases where, even if
Revised: showing the limitation of this benefit: there exists hard cases where, even if

Original: the
Revised: the

Original: agent
Revised: agent

Original: has
Revised: has

Original: exactly learned the environment distribution D in the training stage, it
Revised: exactly learned the environment distribution D in the training stage, it

Original: cannot
Revised: cannot

Original: improve the
Revised: improve the

Original: test
Revised: test

Original: -time
Revised: -time

Original: regret
Revised: regret

Original: up to a universal factor in the asymptotic setting ( K → ∞ ).
Revised: up to a universal factor in the asymptotic setting ( K → ∞ ).

Original: In other words, knowing the distribution D cannot provide more information in consideration of the regret asymptotically.
Revised: In other words, knowing the distribution D cannot provide more information in consideration of the regret asymptotically.

Original: Our theorem is proved by using Radon transform and Lebesgue
Revised: Our theorem is proved by using Radon transform and Lebesgue

Original: integral analysis
Revised: integral analysis

Original: to
Revised: to

Original: give a global level information limit
Revised: give a global level information limit

Original: , which we believe are novel techniques for RL communities
Revised: , which we believe are novel techniques for RL communities

Original: • Inspired by this lower bound, we focus on the non-asymptotic setting
Revised: • Inspired by this lower bound, we focus on the non-asymptotic setting

Original: and study whether and how much we can reduce the regret in this case.
Revised: and study whether and how much we can reduce the regret in this case.

Original: We propose an efficient pre-training and test-time finetuning algorithm called PCE (Policy Collection-Elimination).
Revised: We propose an efficient pre-training and test-time finetuning algorithm called PCE (Policy Collection-Elimination).

Original: By maintaining a minimum policy set that generalizes well, it achieves a regret upper bound ˜ O
Revised: By maintaining a minimum policy set that generalizes well, it achieves a regret upper bound ˜ O

Original: (cid:16)(cid:112) C ( D ) K (cid:17) in the test stage, where C ( D ) is a complexity measure of the distribution D .
Revised: (cid:16)(cid:112) C ( D ) K (cid:17) in the test stage, where C ( D ) is a complexity measure of the distribution D .

Original: This bound removes the polynomial dependence on the cardinality of state-action space by leveraging the information obtained from pre-training.
Revised: This bound removes the polynomial dependence on the cardinality of state-action space by leveraging the information obtained from pre-training.

Original: We give a fine-grained analysis on the value of C ( D ) and show that our bound can be significantly smaller than state-action space dependent bound in many settings.
Revised: We give a fine-grained analysis on the value of C ( D ) and show that our bound can be significantly smaller than state-action space dependent bound in many settings.

Original: • When the agent cannot interact with the test environment, we propose an efficient algorithm called OMERM (Optimistic Model-based Empirical Risk Minimization) to find a near-optimal policy in expectation .
Revised: • When the agent cannot interact with the test environment, we propose an efficient algorithm called OMERM (Optimistic Model-based Empirical Risk Minimization) to find a near-optimal policy in expectation .

Original: This algorithm is guaranteed to return a ϵ -optimal policy with O (cid:16) log (cid:16) N Π ϵ/ (12 H )
Revised: This algorithm is guaranteed to return a ϵ -optimal policy with O (cid:16) log (cid:16) N Π ϵ/ (12 H )

Original: (cid:17) /ϵ 2 (cid:17) sampled MDP tasks in the training stage where N Π ϵ/ (12 H ) is the complexity of the policy class.
Revised: (cid:17) /ϵ 2 (cid:17) sampled MDP tasks in the training stage where N Π ϵ/ (12 H ) is the complexity of the policy class.

Original: This rate matches the traditional generalization rate in many supervised learning results (Mohri et al., 2018; Kawaguchi et al., 2017).
Revised: This rate matches the traditional generalization rate in many supervised learning results (Mohri et al., 2018; Kawaguchi et al., 2017).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Generalization and Multi-task RL.
Revised: Generalization and Multi-task RL.

Original: Many empirical works study how to improve generalization for deep RL algorithms (Packer et al., 2018; Zhang et al., 2020; Ghosh et al., 2021).
Revised: Many empirical works study how to improve generalization for deep RL algorithms (Packer et al., 2018; Zhang et al., 2020; Ghosh et al., 2021).

Original: We refer readers to a recent survey Kirk et al.
Revised: We refer readers to a recent survey Kirk et al.

Original: (2021) for more discussion on empirical results.
Revised: (2021) for more discussion on empirical results.

Original: Our paper is more closely related to the recent works towards understanding RL generalization from the theoretical perspective.
Revised: Our paper is more closely related to the recent works towards understanding RL generalization from the theoretical perspective.

Original: Wang et al.
Revised: Wang et al.

Original: (2019) focused on a special class of reparameterizable RL problems, and derive generalization bounds based on Rademacher complexity and the PAC-Bayes bound.
Revised: (2019) focused on a special class of reparameterizable RL problems, and derive generalization bounds based on Rademacher complexity and the PAC-Bayes bound.

Original: Malik et al.
Revised: Malik et al.

Original: ; Duan et al.
Revised: ; Duan et al.

Original: (2021) also provided lower bounds showing that instance-optimal solution is statistically difficult for RL generalization when we cannot access the sampled test environment.
Revised: (2021) also provided lower bounds showing that instance-optimal solution is statistically difficult for RL generalization when we cannot access the sampled test environment.

Original: Further, they proposed efficient algorithms which is guaranteed to return a near-optimal policy for deterministic MDPs under the strong proximity condition they introduced.
Revised: Further, they proposed efficient algorithms which is guaranteed to return a near-optimal policy for deterministic MDPs under the strong proximity condition they introduced.

Original: Our paper is also related to recent works studying multi-task learning in RL (Tirinzoni et al., 2020; 
Revised: Our paper is also related to recent works studying multi-task learning in RL (Tirinzoni et al., 2020; Hu et al., 2021;

Original: Hu et al., 2021; Zhang & Wang, 2021; Lu et al., 2021), in which they studied how to transfer the knowledge learned from previous tasks to new tasks.
Revised: Zhang & Wang, 2021; Lu et al., 2021), in which they studied how to transfer the knowledge learned from previous tasks to new tasks.

Revised: Their problem formulation is different from ours since they study the multi-task setting where the MDP is selected from a given MDP set without probability mechanism(Brunskill & Li, 2013).

Original: Their settings are different from ours since they typically assume that all the tasks have similar transition dynamics or share common representations.
Revised: In addition, they typically assume that all the tasks have similar transition dynamics or share common representations.

Original: Provably Efficient Exploration in RL.
Revised: Provably Efficient Exploration in RL.

Original: Recent years have witnessed many theoretical results studying provably efficient exploration in RL (Osband et al., 2013; Azar et al., 2017; Osband & Van Roy, 2017; Jin et al., 2018; 2020b; Wang et al., 2020; Zhang et al., 2021) with the minimax regret for tabular MDPs with non-stationary transition being ˜ O ( √ HSAK ) .
Revised: Recent years have witnessed many theoretical results studying provably efficient exploration in RL (Osband et al., 2013; Azar et al., 2017; Osband & Van Roy, 2017; Jin et al., 2018; 2020b; Wang et al., 2020; Zhang et al., 2021) with the minimax regret for tabular MDPs with non-stationary transition being ˜ O ( √ HSAK ) .

Original: These results indicate that polynomial dependence on the whole state-action space is unavoidable without additional assumptions.
Revised: These results indicate that polynomial dependence on the whole state-action space is unavoidable without additional assumptions.

Original: Their formulation corresponds to the single-task setting where the agent only interacts with a single environment aiming to maximize its cumulative rewards without pre-training.
Revised: Their formulation corresponds to the single-task setting where the agent only interacts with a single environment aiming to maximize its cumulative rewards without pre-training.

Original: The regret defined in the fine-tuning setting coincides with the concept of Bayesian regret in the previous literature (Osband et al., 2013; Osband & Van Roy, 2017; O’Donoghue, 2021).
Revised: The regret defined in the fine-tuning setting coincides with the concept of Bayesian regret in the previous literature (Osband et al., 2013; Osband & Van Roy, 2017; O’Donoghue, 2021).

Original: The best-known Bayesian regret for tabular RL is ˜ O ( √ HSAK ) when applied to our setting (O’Donoghue, 2021).
Revised: The best-known Bayesian regret for tabular RL is ˜ O ( √ HSAK ) when applied to our setting (O’Donoghue, 2021).

Original:  P RELIMINARY AND F RAMEWORK
Revised:  P RELIMINARY AND F RAMEWORK

Original: Notations Throughout the paper, we use [ N ] to denote the set { 1 , · · · , N } where N ∈ N + .
Revised: Notations Throughout the paper, we use [ N ] to denote the set { 1 , · · · , N } where N ∈ N + .

Original: For an event E , let I [ E ] be the indicator function of event E , i.e. I [ E ] = 

Revised: For an event E , let I [ E ] be the indicator function of event E , i.e. I [ E ] =

Original: if and only if E is true.
Revised: if and only if E is true.

Original: For any domain Ω , we use C (Ω) to denote the continuous function on Ω .
Revised: For any domain Ω , we use C (Ω) to denote the continuous function on Ω .

Original: We use O ( · ) to denote the standard big O notation, and ˜ O ( · ) to denote the big O notation with log( · ) term omitted.
Revised: We use O ( · ) to denote the standard big O notation, and ˜ O ( · ) to denote the big O notation with log( · ) term omitted.

Original: 3. E PISODIC MDP S
Revised: 3. E PISODIC MDP S

Original: An episodic MDP M is specified as a tuple ( S , A , P M , R M , H ) , where S , A are the state and action space with cardinality S and A respectively, and H is the steps in one episode.
Revised: An episodic MDP M is specified as a tuple ( S , A , P M , R M , H ) , where S , A are the state and action space with cardinality S and A respectively, and H is the steps in one episode.

Original: P M ,h : S × A (cid:55)→ ∆( S ) is the transition such that P M ,h ( s ′ | s, a ) denotes the probability to transit to state s ′ if action a is taken in state s in step h .
Revised: P M ,h : S × A (cid:55)→ ∆( S ) is the transition such that P M ,h ( s ′ | s, a ) denotes the probability to transit to state s ′ if action a is taken in state s in step h .

Original: R M ,h : S × A (cid:55)→ ∆( R ) is the reward function such that R M ,h ( s, a ) is the distribution of reward with non-negative mean r M ,h ( s, a ) when action a is taken in state s at step h .
Revised: R M ,h : S × A (cid:55)→ ∆( R ) is the reward function such that R M ,h ( s, a ) is the distribution of reward with non-negative mean r M ,h ( s, a ) when action a is taken in state s at step h .

Original: In order to compare with traditional generalization, we make the following assumption:
Revised: In order to compare with traditional generalization, we make the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: The total mean reward is bounded by 1 , i.e. ∀M ∈ Ω , (cid:80)H h
Revised: The total mean reward is bounded by 1 , i.e. ∀M ∈ Ω , (cid:80)

Original:  =1 r M ,h ( s h , a h ) ≤ 1 for all trajectory ( s 1 , a 1 , · · · , s H , a H ) with positive probability in M ; The reward mechanism
Revised: Hh =1 r M ,h ( s h , a h ) ≤ 1 for all trajectory ( s 1 , a 1 , · · · , s H , a H ) with positive probability in M ; The reward mechanism

Original: R M ( s, a ) is 1 -subgaussian, i.e. E X ∼ R M ,h
Revised: R M ( s, a ) is 1 -subgaussian, i.e. E X ∼ R M ,h

Original: ( s,a ) [exp( λ [ X − r M ,h
Revised: ( s,a ) [exp( λ [ X − r M ,h

Original: ( s, a )])]
Revised: ( s, a )])]

Original: ≤ exp λ 2 2 for all λ ∈ R .
Revised: ≤ exp λ 2 2 for all λ ∈ R .

Original: The total reward assumption follows the previous works on horizon-free RL (Ren et al., 2021; Zhang et al., 2021; Li et al., 2022) and covers the traditional setting where r M ,h ( s, a ) ∈ [0 , 1] by scaling
Revised: The total reward assumption follows the previous works on horizon-free RL (Ren et al., 2021; Zhang et al., 2021; Li et al., 2022) and covers the traditional setting where r M ,h ( s, a ) ∈ [0 , 1] by scaling

Original: H , and it is more natural in environments with sparse rewards (Vecerik et al., 2017; Riedmiller et al., 2018).
Revised: H , and it is more natural in environments with sparse rewards (Vecerik et al., 2017; Riedmiller et al., 2018).

Original: In addition, it allows us to compare with supervised learning bound where
Revised: In addition, it allows us to compare with supervised learning bound where

Original: H =
Revised: H =

Original: and the loss is bounded by [0 , 1] .
Revised: and the loss is bounded by [0 , 1] .

Original: The subgaussian assumption is more common in practice and is widely used in bandits (Lattimore & Szepesv´ari, 2020).
Revised: The subgaussian assumption is more common in practice and is widely used in bandits (Lattimore & Szepesv´ari, 2020).

Original: It also covers traditional RL setting where R M ,h ( s, a ) ∈ ∆([0 , 1]) , and allows us to study MDP environment with a wider range.
Revised: It also covers traditional RL setting where R M ,h ( s, a ) ∈ ∆([0 , 1]) , and allows us to study MDP environment with a wider range.

Original: For the convenience of explanation, we assume the agent always starts from the same state s 1 .
Revised: For the convenience of explanation, we assume the agent always starts from the same state s 1 .

Original: It is straightforward to recover the initial state distribution µ from this setting by adding an initial state s 0 with transition µ (Du et al., 2019; Chen et al., 2021).
Revised: It is straightforward to recover the initial state distribution µ from this setting by adding an initial state s 0 with transition µ (Du et al., 2019; Chen et al., 2021).

Original: Policy and Value Function.
Revised: Policy and Value Function.

Original: A policy π is set of H functions where each maps a state to an action distribution, i.e. π = { π h } Hh =1 , π h : S (cid:55)→ ∆( A ) and π can be stochastic.
Revised: A policy π is set of H functions where each maps a state to an action distribution, i.e. π = { π h } H h =1 , π h : S (cid:55)→ ∆( A ) and π can be stochastic.

Original: We denote the set of all policies described above as Π .
Revised: We de- note the set of all policies described above as Π .

Original: We define N Π ϵ as the ϵ -covering number of the policy space Π w.r.t.
Revised: We define N Π ϵ as the ϵ -covering number of the policy space Π w.r.t.

Original: distance d( π 1 , π 2 ) = max s ∈S ,h ∈
Revised: distance d( π 1 , π 2 ) = max s ∈S ,h ∈

Original: [ H ] ∥ π 1 h ( ·| s ) − π 2 h ( ·| s ) ∥ 1 .
Revised: [ H ] ∥ π 1 h ( ·| s ) − π 2 h ( ·| s ) ∥ 1 .

Original: Given π and h ∈ [ H ] , we define the Q-function Q π M ,h : S × A
Revised: Given π and h ∈ [ H ] , we define the Q-function Q π M ,h : S × A

Original: (cid:55)→ R + , where Q π M ,h ( s, a ) =
Revised: (cid:55)→ R + , where Q π M ,h ( s, a ) =

Original: r M ,h ( s, a ) + (cid:80) s ′ ∈S P M ,h ( s ′ | s, a ) V π M ,h +1 ( s ′ ) , and the V-function V π M ,h : S (cid:55)→ R + , where
Revised: r M ,h ( s, a ) + (cid:80) s ′ ∈S P M ,h ( s ′ | s, a ) V π M ,h +1 ( s ′ ) , and the V-function V π M ,h : S (cid:55)→ R + , where

Original: V π M ,h ( s ) =
Revised: V π M ,h ( s ) =

Original: E a ∼ π h ( ·| s ) Q π M ,h ( s, a ) for h ≤ H and V π M ,
Revised: E a ∼ π h ( ·| s ) Q π M ,h ( s, a ) for h ≤ H and V π M ,

Original: H +

Revised: H +

Original: ( s )
Revised: ( s )

Original: = 0 .
Revised: = 0 .

Original: We abbreviate V π M , 1 ( s 1 ) as
Revised: We abbreviate V π M , 1 ( s 1 ) as

Original: V π M , which can be interpreted as the value when executing policy π in M .
Revised: V π M , which can be interpreted as the value when executing policy π in M .

Original: Following the notations in previous works, we use P h V ( s, a ) as the shorthand of (cid:80) s ′ ∈S P h
Revised: Following the notations in previous works, we use P h V ( s, a ) as the shorthand of (cid:80) s ′ ∈S P h

Original: ( s ′ | s, a ) V ( s ′ ) in our analysis.
Revised: ( s ′ | s, a ) V ( s ′ ) in our analysis.

Original: 3. RL G ENERALIZATION F ORMULATION
Revised: 3. RL G ENERALIZATION F ORMULATION

Original: We mainly study the setting where all MDP instances we face in training and testing stages are i.i.d.
Revised: We mainly study the setting where all MDP instances we face in training and testing stages are i.i.d.

Original: sampled from a distribution D supported on a (possibly infinite) countable set Ω .
Revised: sampled from a distribution D supported on a (possibly infinite) countable set Ω .

Original: For an MDP
Revised: For an MDP

Original: M ∈ Ω , we use P ( M ) to denote the probability of sampling M according to distribution D .
Revised: M ∈ Ω , we use P ( M ) to denote the probability of sampling M according to distribution D .

Original: For an
Revised: For an

Original: MDP set ˜Ω ⊆ Ω , we similarly define P (˜Ω) =
Revised: MDP set ˜Ω ⊆ Ω , we similarly define P (˜Ω) =

Original: (cid:80)
Revised: (cid:80)

Original: M∈ ˜Ω P ( M ) .
Revised: M∈ ˜Ω P ( M ) .

Original: We assume that S , A , H is shared by all MDPs, while the transition and reward are different.
Revised: We assume that S , A , H is shared by all MDPs, while the transition and reward are different.

Original: When interacting with a sampled instance
Revised: When interacting with a sampled instance

Original: M , one does not know which instance it is, but can only identify its model through interactions.
Revised: M , one does not know which instance it is, but can only identify its model through interactions.

Original: In the training (pre-training) stage, the agent can sample i.i.d.
Revised: In the training (pre-training) stage, the agent can sample i.i.d.

Original: MDP instances from the unknown distribution D .
Revised: MDP instances from the unknown distribution D .

Original: The overall goal is to perform well in the test stage with the information learned in the training stage.
Revised: The overall goal is to perform well in the test stage with the information learned in the training stage.

Original: Define the optimal policy as
Revised: Define the optimal policy as

Original: [Equation]
Revised: [Equation]

Original: We say a policy π is ϵ -optimal in expectation , if E M∼ D (cid:2)
Revised: We say a policy π is ϵ -optimal in expectation , if E M∼ D (cid:2)

Original: V π ∗ ( D ) M − V π M (cid:3) ≤ ϵ.
Revised: V π ∗ ( D [Equation] M − V π M (cid:3) ≤ ϵ.

Original: We say a policy π is ϵ -optimal in instance , if E M∼ D (cid:2) V π ∗ ( M ) M
Revised: We say a policy π is ϵ -optimal in instance , if E M∼ D (cid:2) V π ∗ ( M ) M

Original: [Equation]
Revised: [Equation]

Original: Without Test-time Interaction.
Revised: Without Test-time Interaction.

Original: When the interaction with the test environment is unavailable, optimality in instance can be statistically intractable, and we can only pursue optimality in expectation .
Revised: When the interaction with the test environment is unavailable, optimality in instance can be statistically intractable, and we can only pursue optimality in expectation .

Original: We formulate this difficulty into the following proposition.
Revised: We formulate this difficulty into the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: There exists an MDP support Ω , such that for any distribution D with positive p.d.f.
Revised: There exists an MDP support Ω , such that for any distribution D with positive p.d.f.

Original: p ( r ) , ∃ ϵ 0 > 0 , and for any deployed policy ˆ π ,
Revised: p ( r ) , ∃ ϵ 0 > 0 , and for any deployed policy ˆ π ,

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 is proved by constructing Ω as a set of MDPs with opposed optimal action, and the complete proof can be found in Appendix A.
Revised: Proposition 1 is proved by constructing Ω as a set of MDPs with opposed optimal action, and the complete proof can be found in Appendix A.

Original: When Ω is discrete, there exists hard instances where the proposition holds for ϵ 0 ≥ 12 .
Revised: When Ω is discrete, there exists hard instances where the proposition holds for ϵ 0 ≥ 12 .

Original: This implies that without test-time interactions or special knowledge on the structure of Ω and D , it is impractical to be near optimal in instance .
Revised: This implies that without test-time interactions or special knowledge on the structure of Ω and D , it is impractical to be near optimal in instance .

Original: This intractability arises from the demand on instance optimal policy, which is never asked in supervised learning.
Revised: This intractability arises from the demand on instance optimal policy, which is never asked in supervised learning.

Original: With Test-time Interaction.
Revised: With Test-time Interaction.

Original: To pursue the optimality in instance , we study the problem of RL generalization with test-time interaction.
Revised: To pursue the optimality in instance , we study the problem of RL generalization with test-time interaction.

Original: When our algorithm is allowed to interact with the target
Revised: When our algorithm is allowed to interact with the target

Original: MDP M ∗ ∼ D for K episodes in the test stage, we want to reduce the regret, which is defined as
Revised: MDP M ∗ ∼ D for K episodes in the test stage, we want to reduce the regret, which is defined as

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ≜

Original: [Equation]
Revised: [Equation]

Original: where π k is the policy that A deploys in episode k .
Revised: where π k is the policy that A deploys in episode k .

Original: Here M ∗ is unknown and unchanged during all
Revised: Here M ∗ is unknown and unchanged during all

Original: K episodes.
Revised: K episodes.

Original: The choice of Bayesian regret is more natural in generalization, and can better evaluate the performance of an algorithm in practice.
Revised: The choice of Bayesian regret is more natural in generalization, and can better evaluate the performance of an algorithm in practice.

Original: From the standard Regret-to-PAC technique (Jin et al., 2018;
Revised: From the standard Regret-to-PAC technique (Jin et al., 2018;

Original: Dann et al., 2017), an algorithm with ˜ O
Revised: Dann et al., 2017), an algorithm with ˜ O

Original: ( √ K ) regret can be transformed to an algorithm that returns an ϵ -optimal policy with ˜ O
Revised: ( √ K ) regret can be transformed to an algorithm that returns an ϵ -optimal policy with ˜ O

Original: (1 /ϵ 2 ) trajectories.
Revised: (1 /ϵ 2 ) trajectories.

Original: Therefore, we believe regret can also be a good criterion to measure the sample efficiency of fine-tuning algorithms in the test stage.
Revised: Therefore, we believe regret can also be a good criterion to measure the sample efficiency of fine-tuning algorithms in the test stage.

Original:  R ESULTS FOR THE S ETTING WITH T EST - TIME I NTERACTION
Revised:  R ESULTS FOR THE S ETTING WITH T EST - TIME I NTERACTION

Original: In this section, we study the setting where the agent is allowed to interact with the sampled test
Revised: In this section, we study the setting where the agent is allowed to interact with the sampled test

Original: MDP M ∗ .
Revised: MDP M ∗ .

Original: When there is no pre-training stage, the typical regret bound in the test stage is
Revised: When there is no pre-training stage, the typical regret bound in the test stage is

Original: ˜ O ( √ SAHK ) (Zhang et al., 2021).
Revised: ˜ O ( √ SAHK ) (Zhang et al., 2021).

Original: For generalization in RL, we mainly care about the performance in the test stage, and hope the agent can reduce test regret by leveraging the information learned in the pre-training stage.
Revised: For generalization in RL, we mainly care about the performance in the test stage, and hope the agent can reduce test regret by leveraging the information learned in the pre-training stage.

Original: Obviously, when Ω is the set of all tabular MDPs and the distribution D is uniform over Ω , pre-training can do nothing on improving the test regret, since it provides no extra information for the test stage.
Revised: Obviously, when Ω is the set of all tabular MDPs and the distribution D is uniform over Ω , pre-training can do nothing on improving the test regret, since it provides no extra information for the test stage.

Original: Therefore, we seek a distribution-dependent improvement that is better than traditional upper bound in most of benign settings.
Revised: Therefore, we seek a distribution-dependent improvement that is better than traditional upper bound in most of benign settings.

Original: 4. L OWER B OUND
Revised: 4. L OWER B OUND

Original: We start by understanding how much information the pre-training stage can provide at most .
Revised: We start by understanding how much information the pre-training stage can provide at most .

Original: One natural focus is on the MDP distribution D , which is a sufficient statistic of the possible environment that the agent will encounter in the test stage.
Revised: One natural focus is on the MDP distribution D , which is a sufficient statistic of the possible environment that the agent will encounter in the test stage.

Original: We strengthen the algorithm by directly telling it the accurate distribution D , and analyze how much this extra information can help to improve the regret.
Revised: We strengthen the algorithm by directly telling it the accurate distribution D , and analyze how much this extra information can help to improve the regret.

Original: Specifically, we ask: Is there a multiplied factor C ( D ) that is small when D enjoys some benign properties (e.g. D is sharp and concentrated), such that when knowing D , there exists an algorithm that can reduce the regret by a factor of C ( D ) for large enough K ?
Revised: Specifically, we ask: Is there a multiplied factor C ( D ) that is small when D enjoys some benign properties (e.g. D is sharp and concentrated), such that when knowing D , there exists an algorithm that can reduce the regret by a factor of C ( D ) for large enough K ?

Original: Perhaps surprisingly, our answer towards this question is negative for all large enough K in the asymptotic case.
Revised: Perhaps surprisingly, our answer towards this question is negative for all large enough K in the asymptotic case.

Original: As is formulated in Theorem 1, the importance of D is constrained by a universal factor c 0 asymptotically.
Revised: As is formulated in Theorem 1, the importance of D is constrained by a universal factor c 0 asymptotically.

Original: Here c 0 = 116 holds universally and does not depend on D .
Revised: Here c 0 = 116 holds universally and does not depend on D .

Original: This theorem implies that no matter what distribution D is, for sufficiently large K , any algorithm can only reduce the total regret by at most a constant factor with the extra knowledge of D .
Revised: This theorem implies that no matter what distribution D is, for sufficiently large K , any algorithm can only reduce the total regret by at most a constant factor with the extra knowledge of D .

Original: Theorem 1.
Revised: Theorem 1.

Original: There exists an MDP instance set Ω , a universal constant c 0 = 116 , and an algorithm ˆ A that only inputs the episode K , such that for any distribution D with positive p.d.f.
Revised: There exists an MDP instance set Ω , a universal constant c 0 = 116 , and an algorithm ˆ A that only inputs the episode K , such that for any distribution D with positive p.d.f.

Original: p ( r ) ∈ C (Ω) (which ˆ A does NOT know), any algorithm A that inputs D and the episode K ,
Revised: p ( r ) ∈ C (Ω) (which ˆ A does NOT know), any algorithm A that inputs D and the episode K ,

Original: Ω is not degraded, i.e. lim K →∞ Reg K ( D , A ( D , K )) =
Revised: Ω is not degraded, i.e. lim K →∞ Reg K ( D , A ( D , K )) =

Original: + ∞ .
Revised: + ∞ .

Original: Knowing the distribution is useless up to a constant, i.e. lim
Revised: Knowing the distribution is useless up to a constant, i.e. lim

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: In Theorem 1, Point (1) avoids any trivial support Ω where ∃ π ∗ that is optimal for all M ∈ Ω , in which case the distribution is of course useless since ˆ A can be optimal by simply following π ∗ even it does not know D .
Revised: In Theorem 1, Point (1) avoids any trivial support Ω where ∃ π ∗ that is optimal for all M ∈ Ω , in which case the distribution is of course useless since ˆ A can be optimal by simply following π ∗ even it does not know D .

Original: Note that our bounds hold for any distribution D , which indicates that even a very sharp distribution cannot provide useful information in the asymptotic case where K → ∞ .
Revised: Note that our bounds hold for any distribution D , which indicates that even a very sharp distribution cannot provide useful information in the asymptotic case where K → ∞ .

Original: We point out that the value of c 0 depends on the coefficient of previous upper and lower bound, and we conjecture that it could be arbitrarily close to 1 .
Revised: We point out that the value of c 0 depends on the coefficient of previous upper and lower bound, and we conjecture that it could be arbitrarily close to 1 .

Original: We defer the complete proof to Appendix B, and briefly sketch the intuition here.
Revised: We defer the complete proof to Appendix B, and briefly sketch the intuition here.

Original: The key observation is that the information provided in the training stage (prior) is fixed, while the required information gradually increase as K increases.
Revised: The key observation is that the information provided in the training stage (prior) is fixed, while the required information gradually increase as K increases.

Original: When K = 1 , the agent can clearly benefit from the knowledge of D .
Revised: When K = 1 , the agent can clearly benefit from the knowledge of D .

Original: Without this knowledge, all it can do is a random guess since it has never interacted with M ∗ before.
Revised: Without this knowledge, all it can do is a random guess since it has never interacted with M ∗ before.

Original: However, when K is large, the algorithm can interact with M ∗ many times and learn M ∗ more accurately, while the prior D will become relatively less informative.
Revised: However, when K is large, the algorithm can interact with M ∗ many times and learn M ∗ more accurately, while the prior D will become relatively less informative.

Original: As a result, the benefits of knowing D vanishes eventually.
Revised: As a result, the benefits of knowing D vanishes eventually.

Original: Theorem 1 lower bounds the improvement of regret by a constant.
Revised: Theorem 1 lower bounds the improvement of regret by a constant.

Original: As is commonly known, the regret bound can be converted into a PAC-RL bound (Jin et al., 2018; Dann et al., 2017).
Revised: As is commonly known, the regret bound can be converted into a PAC-RL bound (Jin et al., 2018; Dann et al., 2017).

Original: This implies that when δ, ϵ → 0 , in terms of pursuing a ϵ -optimal policy to π ∗ ( M ∗ ) , pre-training cannot help reduce the sample complexity.
Revised: This implies that when δ, ϵ → 0 , in terms of pursuing a ϵ -optimal policy to π ∗ ( M ∗ ) , pre-training cannot help reduce the sample complexity.

Original: Despite negative, we point out that this theorem only describe the asymptotic setting where K → ∞ , but it imposes no constraint when K is fixed.
Revised: Despite negative, we point out that this theorem only describe the asymptotic setting where K → ∞ , but it imposes no constraint when K is fixed.

Original: 4. N ON - ASYMPTOTIC U PPER B OUND
Revised: 4. N ON - ASYMPTOTIC U PPER B OUND

Original: In the last subsection, we provide a lower bound showing that the information obtained from the training stage can be useless in the asymptotic setting where K → ∞ .
Revised: In the last subsection, we provide a lower bound showing that the information obtained from the training stage can be useless in the asymptotic setting where K → ∞ .

Original: In practice, a near-optimal regret in the non-asymptotic setting is also desirable in many applications.
Revised: In practice, a near-optimal regret in the non-asymptotic setting is also desirable in many applications.

Original: In this section, we fix the value of K and seek to design an algorithm such that it can leverage the pre-training information and reduce K -episode test regret.
Revised: In this section, we fix the value of K and seek to design an algorithm such that it can leverage the pre-training information and reduce K -episode test regret.

Original: To avoid redundant explanation for single MDP learning, we formulate the following oracles.
Revised: To avoid redundant explanation for single MDP learning, we formulate the following oracles.

Original: Definition 1.
Revised: Definition 1.

Original: (Policy learning oracle)
Revised: (Policy learning oracle)

Original: We define O l ( M , ϵ, log(1 /δ )) as the policy learning oracle which can return a policy π that is ϵ -optimal w.r.t.
Revised: We define O l ( M , ϵ, log(1 /δ )) as the policy learning oracle which can return a policy π that is ϵ -optimal w.r.t.

Original: MDP M with probability at least 1 − δ , i.e.
Revised: MDP M with probability at least 1 − δ , i.e.

Original: V ∗M ( s 1 ) − V π M ( s 1 ) ≤ ϵ .
Revised: V ∗M ( s 1 ) − V π M ( s 1 ) ≤ ϵ .

Original: The randomness of the policy π is due to the randomness of both the oracle algorithm and the environment.
Revised: The randomness of the policy π is due to the randomness of both the oracle algorithm and the environment.

Original: Definition 2.
Revised: Definition 2.

Original: (Policy evaluation oracle)
Revised: (Policy evaluation oracle)

Original: We define O e ( M , π, ϵ, log(1 /δ )) as the policy evaluation oracle which can return a value v that is ϵ -close to the value function V π M ( s 1 ) with probability at least 1 − δ , i.e. | v − V π M ( s 1 ) | ≤ ϵ .
Revised: We define O e ( M , π, ϵ, log(1 /δ )) as the policy evaluation oracle which can return a value v that is ϵ -close to the value function V π M ( s 1 ) with probability at least 1 − δ , i.e. | v − V π M ( s 1 ) | ≤ ϵ .

Original: The randomness of the value v is due to the randomness of both the oracle algorithm and the environment.
Revised: The randomness of the value v is due to the randomness of both the oracle algorithm and the environment.

Original: Both oracles can be efficiently implemented using the previous algorithms for single-task MDPs.
Revised: Both oracles can be efficiently implemented using the previous algorithms for single-task MDPs.

Original: Specifically, we can implement the policy learning oracle using algorithms such as UCBVI (Azar et al., 2017), LSVI-UCB (Jin et al., 2020b) and GOLF (Jin et al., 2021) with polynomial sample complexities, and the policy evaluation oracle can be achieved by the standard Monte Carlo method (Sutton & Barto, 2018).
Revised: Specifically, we can implement the policy learning oracle using algorithms such as UCBVI (Azar et al., 2017), LSVI-UCB (Jin et al., 2020b) and GOLF (Jin et al., 2021) with polynomial sample complexities, and the policy evaluation oracle can be achieved by the standard Monte Carlo method (Sutton & Barto, 2018).

Original: 4.2. A LGORITHM
Revised: 4.2. A LGORITHM

Original: There are two major difficulties in designing the algorithm.
Revised: There are two major difficulties in designing the algorithm.

Original: First, what do we want to learn during the pre-training process and how to learn it?
Revised: First, what do we want to learn during the pre-training process and how to learn it?

Original: One idea is to directly learn the whole distribution D , which is all that we can obtain for the test stage.
Revised: One idea is to directly learn the whole distribution D , which is all that we can obtain for the test stage.

Original: However, this requires ˜ O ( | Ω | 2 /δ 2 ) samples for a required accuracy δ , and is unacceptable when | Ω | is large or even infinite.
Revised: However, this requires ˜ O ( | Ω | 2 /δ 2 ) samples for a required accuracy δ , and is unacceptable when | Ω | is large or even infinite.

Original: Second, how to design the test-stage algorithm to leverage the learned information effectively?
Revised: Second, how to design the test-stage algorithm to leverage the learned information effectively?

Original: If we cannot effectively use the information from the pre-training, the regret or samples required in the test stage can be ˜
Revised: If we cannot effectively use the information from the pre-training, the regret or samples required in the test stage can be ˜

Original: O ( poly ( S, A )) in the worst case.
Revised: O ( poly ( S, A )) in the worst case.

Original: Pre-training Stage
Revised: Pre-training Stage

Original: the
Revised: the

Original: Test Stage
Revised: Test Stage

Original: [Table]
Revised: [Table]

Original: Eliminate ( π l , v l ) from the instance set ˆΠ l , denote the remaining set as ˆΠ l + 8:
Revised: Eliminate ( π l , v l ) from the instance set ˆΠ l , denote the remaining set as ˆΠ l + 8:

Original: Set k 

Revised: Set k

Original: = k + 1 , and l = l +
Revised: = k + 1 , and l = l +

Original: To tackle the above difficulties, we formulate this problem as a policy candidate collectionelimination process.
Revised: To tackle the above difficulties, we formulate this problem as a policy candidate collectionelimination process.

Original: Our intuition is to find a minimum policy set that can generalize to most
Revised: Our intuition is to find a minimum policy set that can generalize to most

Original: MDPs sampled from D .
Revised: MDPs sampled from D .

Original: In the pre-training stage, we maintain a policy set that can perform well on most MDP instances.
Revised: In the pre-training stage, we maintain a policy set that can perform well on most MDP instances.

Original: This includes policies that are the near-optimal policy for an MDP M with relatively large P ( M ) , or that can work well on different MDPs.
Revised: This includes policies that are the near-optimal policy for an MDP M with relatively large P ( M ) , or that can work well on different MDPs.

Original: In the test stage, we sequentially execute policies in this set.
Revised: In the test stage, we sequentially execute policies in this set.

Original: Once we realize that current policy is not near-optimal for M ∗ , we eliminate it and switch to another.
Revised: Once we realize that current policy is not near-optimal for M ∗ , we eliminate it and switch to another.

Original: This helps reduce the regret from the cardinality of the whole state-action space to the size of policy covering set.
Revised: This helps reduce the regret from the cardinality of the whole state-action space to the size of policy covering set.

Original: The pseudo code is in Algorithm 1.
Revised: The pseudo code is in Algorithm 1.

Original: Pre-training Stage.
Revised: Pre-training Stage.

Original: In the pre-training stage, we say a policy-value pair ( π, v ) that covers an MDP
Revised: In the pre-training stage, we say a policy-value pair ( π, v ) that covers an MDP

Original: M if π is O ( ϵ ) -optimal for the MDP M and v is an estimation of the optimal value V ∗M ( s 1 ) with at most O ( ϵ ) error.
Revised: M if π is O ( ϵ ) -optimal for the MDP M and v is an estimation of the optimal value V ∗M ( s 1 ) with at most O ( ϵ ) error.

Original: For a policy-value set ˆΠ , we say the policy set covers the distribution D with probability at least 1 − O ( δ ) if Pr M∼ D (cid:104) ∃
Revised: For a policy-value set ˆΠ , we say the policy set covers the distribution D with probability at least 1 − O ( δ ) if Pr M∼ D (cid:104) ∃

Original: ( π, v ) ∈ ˆΠ , ( π, v ) covers M (cid:105) ≥ 1 − O ( δ ) .
Revised: ( π, v ) ∈ ˆΠ , ( π, v ) covers M (cid:105) ≥ 1 − O ( δ ) .

Original: The basic goal in the pre-training phase is to find a policy-value set ˆΠ with bounded cardinality that covers D with high probability.
Revised: The basic goal in the pre-training phase is to find a policy-value set ˆΠ with bounded cardinality that covers D with high probability.

Original: The pre-training stage contains several phases.
Revised: The pre-training stage contains several phases.

Original: In each phase, we sample N
Revised: In each phase, we sample N

Original: MDPs from the distribution D and obtain an MDP set {M 1 , M 2 , · · · , M N } .
Revised: MDPs from the distribution D and obtain an MDP set {M 1 , M 2 , · · · , M N } .

Original: We call the oracle
Revised: We call the oracle

Original: O l to calculate the near-optimal policy π j for each MDP M j ,
Revised: O l to calculate the near-optimal policy π j for each MDP M j ,

Original: and we calculate the value estimation v i,j for each policy π j on MDP M i using oracle O e .
Revised: and we calculate the value estimation v i,j for each policy π j on MDP M i using oracle O e .

Original: We use the following condition to indicate whether the pair ( π j , v j,j ) covers the MDP M i :
Revised: We use the following condition to indicate whether the pair ( π j , v j,j ) covers the MDP M i :

Original: [Equation]
Revised: [Equation]

Original: The above condition indicates that π j is a near-optimal policy for M i , and v j,j is an accurate estimation of the value V π j M i , 1 ( s 1 ) .
Revised: The above condition indicates that π j is a near-optimal policy for M i , and v j,j is an accurate estimation of the value V π j M i , 1 ( s 1 ) .

Original: With this condition, we construct a policy-value set ˆΩ that covers (1 − 3 δ ) -fraction of the MDPs in the sampled MDP set by calling Subroutine 4.
Revised: With this condition, we construct a policy-value set ˆΠ that covers (1 − 3 δ ) -fraction of the MDPs in the sampled MDP set by calling Subroutine 4.

Original: We output the policy-value set once the distribution estimation error (cid:114)| [Equation]
Revised: We output the policy-value set once the distribution estimation error (cid:114)

Original: | log [Equation] 2 N/δ 
Revised: | ˆΩ | log(2 N/δ )

Original: [Equation]

Original: N −| ˆΩ | is less than δ , and double the number of the sampled MDPs N to increase the accuracy of the distribution estimation otherwise.
Revised: N −| ˆΩ | is less than δ , and double the number of the sampled MDPs N to increase the accuracy of the distribution estimation otherwise.

Original: After the pre-training phase, we can guarantee that the returned policy set can cover D with probability at least 1 − O ( δ ) , i.e.
Revised: After the pre-training phase, we can guarantee that the returned policy set can cover D with probability at least 1 − O ( δ ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: Fine-tuning Stage.
Revised: Fine-tuning Stage.

Original: We start with the policy-value set ˆΠ from the pre-training stage and eliminate the policy-value pairs until we reach a ( π, v ) ∈ ˆΠ that covers the test MDP M ∗ .
Revised: We start with the policy-value set ˆΠ from the pre-training stage and eliminate the policy-value pairs until we reach a ( π, v ) ∈ ˆΠ that covers the test MDP M ∗ .

Original: Specifically, we split all episodes into different phases.
Revised: Specifically, we split all episodes into different phases.

Original: In phase l , we maintain a set ˆΠ l that covers the real environment M ∗ with high probability.
Revised: In phase l , we maintain a set ˆΠ l that covers the real environment M ∗ with high probability.

Original: We select ( π l , v l ) with the most optimistic value v l in ˆΠ l and execute the policy π l for several episodes.
Revised: We select ( π l , v l ) with the most optimistic value v l in ˆΠ l and execute the policy π l for several episodes.

Original: During execution, we also evaluate the policy π l on the MDP M ∗ (i.e. V π l M ∗ , 1 ( s 1 ) ) and maintain the empirical estimation
Revised: During execution, we also evaluate the policy π l on the MDP M ∗ (i.e. V π l M ∗ , 1 ( s 1 ) ) and maintain the empirical estimation

Original: [Equation]
Revised: [Equation]

Original: kτ
Revised: kτ

Original: [Equation]
Revised: [Equation]

Original:  G k .
Revised: k 0 G k .

Original: Once we identify that π l is not near-optimal for M ∗ , we end this phase and eliminate ( π l , v l ) from ˆΠ l .
Revised: Once we identify that π l is not near-optimal for M ∗ , we end this phase and eliminate ( π l , v l ) from ˆΠ l .

Original: 4.2. R EGRET
Revised: 4.2. R EGRET

Original: The efficiency of Algorithm 1 is summarized in Theorem 2, and the proof is in Appendix C.
Revised: The efficiency of Algorithm 1 is summarized in Theorem 2, and the proof is in Appendix C.

Original: Theorem 2.
Revised: Theorem 2.

Original: The regret of Algorithm 1 is at most
Revised: The regret of Algorithm 1 is at most

Original: [Equation]
Revised: [Equation]

Original: where C ( D ) ≜ min P (˜Ω) ≥ 1 − δ | ˜Ω | is a complexity measure of D and δ = 1 / √ K .
Revised: where C ( D ) ≜ min P (˜Ω) ≥ 1 − δ | ˜Ω | is a complexity measure of D and δ = 1 / √ K .

Original: In addition, with probability at least 1 − O ( δ log( C ( D ) /δ )) , samples required in the pre-training stage is O ( C ( D ) δ 2 ) .
Revised: In addition, with probability at least 1 − O ( δ log( C ( D ) /δ )) , samples required in the pre-training stage is O ( C ( D ) δ 2 ) .

Original: Different from the previous regret bound when the pre-training is unavailable (Azar et al., 2017;
Revised: Different from the previous regret bound when the pre-training is unavailable (Azar et al., 2017;

Original: Osband et al., 2013; Zhang et al., 2021), this distribution-dependent upper bound improves the dependence on S, A to a complexity measure of D defined as C ( D ) .
Revised: Osband et al., 2013; Zhang et al., 2021), this distribution-dependent upper bound improves the dependence on S, A to a complexity measure of D defined as C ( D ) .

Original: C ( · ) serves as a multiplied factor, and can be small when D enjoys benign properties as shown below.
Revised: C ( · ) serves as a multiplied factor, and can be small when D enjoys benign properties as shown below.

Original: First, when the cardinality of Ω is small, i.e. M ≪ SA , we have C ( D ) ≤ M and the pre-training can greatly reduce the instance space via representation learning or multi-task learning (Agarwal et al., 2020; Brunskill & Li, 2013).
Revised: First, when the cardinality of Ω is small, i.e. | Ω | ≪ SA , we have C ( D ) ≤ | Ω | and the pre-training can greatly reduce the instance space via representation learning or multi-task learning (Agarwal et al., 2020; Brunskill & Li, 2013).

Original: Specifically, the regret in the test stage is reduced from ˜ O ( √ SAHK ) to ˜ O ( √ MK ) .
Revised: Specifically, the regret in the test stage is reduced from ˜ O ( √ SAHK ) to ˜ O ( (cid:112) | Ω | K ) .

Original: To the best of our knowledge, this is the first result that achieves dependence only on M for any general distribution in RL generalization. 
Revised: To the best of our knowledge, this is the first result that achieves dependence only on M for any general distribution in RL generalization. We provide a lower bound in

Revised: Appendix C.4 to show that our regret in the test stage is near-optimal except for logarithmic factors.

Original: When | Ω | is large or even infinite, C ( D ) is still bounded and can be significantly smaller when D enjoys benign properties.
Revised: When | Ω | is large or even infinite, C ( D ) is still bounded and can be significantly smaller when D enjoys benign properties.

Original: In fact, in the worst case, it is not hard to find that the dependence on C ( D ) is unavoidable 2 , and there is no way to expect any improvement.
Revised: In fact, in the worst case, it is not hard to find that the dependence on C ( D ) is unavoidable 2 , and there is no way to expect any improvement.

Original: However, in practice where | Ω | is large, the probability will typically be concentrated on several subset region of Ω and decay quickly outside, e.g. when D is subgaussian or mixtures of subgaussian.
Revised: However, in practice where | Ω | is large, the probability will typically be concentrated on several subset region of Ω and decay quickly outside, e.g. when D is subgaussian or mixtures of subgaussian.

Original: Specifically, if the probability of the i -th MDP p i ≤ c 1 e − λi for some positive constant c 1 , λ , then C ( D ) ≤ O (log 1 δ ) , which gives the upper bound O (cid:18)(cid:113) K log 3 ( K ) (cid:19) .
Revised: Specifically, if the probability of the i -th MDP p i ≤ c 1 e − λi for some positive constant c 1 , λ , then C ( D ) ≤ O (log 1 δ ) , which gives the upper bound O (cid:18)(cid:113) K log 3 ( K ) (cid:19) .

Original: It is worthwhile to mention that our algorithm in the pre-training stage actually finds a policy covering set, rather than an MDP covering set, despite that the regret in Theorem 2 depends on the cardinality of the MDP covering set.
Revised: It is worthwhile to mention that our algorithm in the pre-training stage actually finds a policy covering set, rather than an MDP covering set, despite that the regret in Theorem 2 depends on the cardinality of the MDP covering set.

Original: This dependence can possibly be improved to the cardinality of the policy covering set by adding other assumptions to our policy learning oracle, which we leave as an interesting problem for the future research.
Revised: This dependence can possibly be improved to the cardinality of the policy covering set by adding other assumptions to our policy learning oracle, which we leave as an interesting problem for the future research.

Original:  R ESULTS FOR THE S ETTING WITHOUT T EST - TIME I NTERACTION
Revised:  R ESULTS FOR THE S ETTING WITHOUT T EST - TIME I NTERACTION

Original: In this section, we study the benefits of pre-training without test-time interaction.
Revised: In this section, we study the benefits of pre-training without test-time interaction.

Original: As illustrated by
Revised: As illustrated by

Original: Proposition 1, we only pursue the optimality in expectation , which is in line with supervised learning.
Revised: Proposition 1, we only pursue the optimality in expectation , which is in line with supervised learning.

Original: Traditional Empirical Risk Minimization algorithm can be ϵ -optimal with O ( log |H| ϵ 2 ) samples, and we expect this to be true in RL generalization as well.
Revised: Traditional Empirical Risk Minimization algorithm can be ϵ -optimal with O ( log |H| ϵ 2 ) samples, and we expect this to be true in RL generalization as well.

Original: However, a policy in RL needs to sequentially interact with the environment, which is not captured by pure ERM algorithm.
Revised: However, a policy in RL needs to sequentially interact with the environment, which is not captured by pure ERM algorithm.

Original: In addition, different MDPs in Ω can have distinct optimal actions, making it hard to determine which action is better even in expectation.
Revised: In addition, different MDPs in Ω can have distinct optimal actions, making it hard to determine which action is better even in expectation.

Original: To overcome these issues, we design an algorithm called OMERM in
Revised: To overcome these issues, we design an algorithm called OMERM in

Original: Algorithm 2.
Revised: Algorithm 2.

Original: Our algorithm is designed for tabular MDPs with finite state-actions space.
Revised: Our algorithm is designed for tabular MDPs with finite state-actions space.

Original: Nevertheless, it can be extended to the non-tabular setting by combining our ideas with previous algorithms for efficient RL with function approximation (Jin et al., 2020a; Wang et al., 2020).
Revised: Nevertheless, it can be extended to the non-tabular setting by combining our ideas with previous algorithms for efficient RL with function approximation (Jin et al., 2020a; Wang et al., 2020).

Original: In Algorithm 2, we first sample N tasks from the distribution D as the input.
Revised: In Algorithm 2, we first sample N tasks from the distribution D as the input.

Original: The goal of this algorithm is to find a near-optimal policy in expectation w.r.t.
Revised: The goal of this algorithm is to find a near-optimal policy in expectation w.r.t.

Original: the task set {M 1 , M 2 , · · · , M N } .
Revised: the task set {M 1 , M 2 , · · · , M N } .

Original: In each episode, the algorithm estimates each MDP instance using history trajectories, and calculates the optimistic value function of each instance.
Revised: In each episode, the algorithm estimates each MDP instance using history trajectories, and calculates the optimistic value function of each instance.

Original: Based on this, it selects the policy from Π that maximizes the average optimistic value,
Revised: Based on this, it selects the policy from Π that maximizes the average optimistic value,

Original: i.e. 1 N (cid:80) Ni =
Revised: i.e. 1 N (cid:80) Ni =

Original: 1 ˆ V π M i , 1 ( s 1 ) .
Revised: 1 ˆ V π M i , 1 ( s 1 ) .

Original: This selection objective is inspired by ERM, with the difference that we require this estimation to be optimistic.
Revised: This selection objective is inspired by ERM, with the difference that we require this estimation to be optimistic.

Original: It can be achieved by a planning oracle when Π is all stochastic maps from ( S , H ) to ∆( A ) , or by gradient descent when Π is a parameterized model.
Revised: It can be achieved by a planning oracle when Π is all stochastic maps from ( S , H ) to ∆( A ) , or by gradient descent when Π is a parameterized model.

Original: For each sampled M i , our algorithm needs to interact with it for poly ( S, A, H, 1 ϵ ) times, which is a typical requirement to learn a model well.
Revised: For each sampled M i , our algorithm needs to interact with it for poly ( S, A, H, 1 ϵ ) times, which is a typical requirement to learn a model well.

Original: Theorem 3.
Revised: Theorem 3.

Original: With probability 3 at least 2 / 3 , Algorithm 2 can output a policy ˆ π satisfying
Revised: With probability 3 at least 2 / 3 , Algorithm 2 can output a policy ˆ π satisfying

Original: E 
Revised: E M ∗ ∼ D [ V π ∗

Original: [Equation]
Revised: [Equation]

Original:  with O 
Revised: D ) M ∗ − V [Equation] M ∗ ] ≤ ϵ with O (cid:18)

Original: [Equation]
Revised: [Equation]

Revised: (cid:19)

Original: MDP instance samples during training.
Revised: MDP instance samples during training.

Original: The number of episodes collected for each task is bounded by O (cid:16) H 2 S 2 A log( SAH ) ϵ(cid:17) .
Revised: The number of episodes collected for each task is bounded by O (cid:16) H 2 S 2 A log( SAH ) ϵ(cid:17) .

Original: We defer the proof of Theorem 3 to Appendix D.
Revised: We defer the proof of Theorem 3 to Appendix D.

Original: This theorem implies that Algorithm 2 needs approximately O (log (cid:0) N Π ϵ 12 H (cid:1) /ϵ 2 ) samples to return an ϵ -optimal policy in expectation .
Revised: This theorem implies that Algorithm 2 needs approximately O (log (cid:0) N Π ϵ 12 H (cid:1) /ϵ 2 ) samples to return an ϵ -optimal policy in expectation .

Original: Recall that log N Π
Revised: Recall that log N Π

Original: ϵ 12 H is the log-covering number of Π .
Revised: ϵ 12 H is the log-covering number of Π .

Original: When Π is all stochastic maps from S , H to ∆( A ) , it is bounded by ˜ O
Revised: When Π is all stochastic maps from S , H to ∆( A ) , it is bounded by ˜ O

Original: ( HSA ) .
Revised: ( HSA ) .

Original: When Π is a parameterized model where the parameter θ ∈ R d has finite norm and π θ satisfies some smoothness condition on θ , log( N Π ϵ 12 H ) ≤ ˜ O
Revised: When Π is a parameterized model where the parameter θ ∈ R d has finite norm and π θ satisfies some smoothness condition on θ , log( N Π ϵ 12 H ) ≤ ˜ O

Original: ( d ) .
Revised: ( d ) .

Original: This result for episode k = 1 , 2 , · ·
Revised: This result for episode k = 1 , 2 , · ·

Original: , K do 5: for i = 1 , 2 , · · · , N do
Revised: , K do 5: for i = 1 , 2 , · · · , N do

Original: Denote N M i ,k,h ( s, a, s ′ ) and N M i ,k,h ( s, a ) as the counter that agent encounters ( s, a, s ′ ) and ( s, a ) at step h in M i till step k − 1 , respectively
Revised: Denote N M i ,k,h ( s, a, s ′ ) and N M i ,k,h ( s, a ) as the counter that agent encounters ( s, a, s ′ ) and ( s, a ) at step h in M i till step k − 1 , respectively

Original: Estimate ˆ P M i ,k,h ( s ′ | s, a ) =
Revised: Estimate ˆ P M i ,k,h ( s ′ | s, a ) =

Original: N M i,k,h ( s,a,s ′ ) max { 1 ,N M i,k,h ( s,a ) } for step h ∈
Revised: N M i,k,h ( s,a,s ′ ) max { 1 ,N M i,k,h ( s,a ) } for step h ∈

Original: [ H ]
Revised: [ H ]

Original: Estimate ˆ R M
Revised: Estimate ˆ R M

Original: i ,k,h ( s, a ) =
Revised: i ,k,h ( s, a ) =

Original: (cid:80)
Revised: (cid:80)

Original: k − 1 τ =1 r M i,τ,h 1 ( s M i,τ,h = s,a M i,τ,h = a ) max { 1 ,N M i,k,h ( s,a ) } for step h ∈
Revised: k − 1 τ =1 r M i,τ,h 1 ( s M i,τ,h = s,a M i,τ,h = a ) max { 1 ,N M i,k,h ( s,a ) } for step h ∈

Original: [ H ]
Revised: [ H ]

Original: Define the UCB bonus b M i ,k,h ( s, a )
Revised: Define the UCB bonus b M i ,k,h ( s, a )

Original: =
Revised: =

Original: (cid:113) 8 S log(8 SANHK ) max { 1 ,N M i,k,h ( s,a ) } 10:
Revised: (cid:113) 8 S log(8 SANHK ) max { 1 ,N M i,k,h ( s,a ) } 10:

Original: Initialize ˆ V π M i ,k,H +1 ( s, a ) = 0 , ∀ s, a for h = H, H − 1 , · · · , 1 do ˆ Q π M i ,k,h ( s, a ) = min (cid:110) 1 , ˆ R M i ,k,h ( s, a )
Revised: Initialize ˆ V π M i ,k,H +1 ( s, a ) = 0 , ∀ s, a for h = H, H − 1 , · · · , 1 do ˆ Q π M i ,k,h ( s, a ) = min (cid:110) 1 , ˆ R M i ,k,h ( s, a )

Original: + b M i ,k ( s, a ) + ˆ P M i ,k,
Revised: + b M i ,k ( s, a ) + ˆ P M i ,k,

Original: h ˆ V π M i ,k,h +1 ( s, a ) (cid:111)
Revised: h ˆ V π M i ,k,h +1 ( s, a ) (cid:111)

Original: ˆ V π M i ,k,h ( s ) =
Revised: ˆ V π M i ,k,h ( s ) =

Original: (cid:80) a π h ( a | s )
Revised: (cid:80) a π h ( a | s )

Original: ˆ Q π M i ,k,h ( s, a )
Revised: ˆ Q π M i ,k,h ( s, a )

Original: Calculate the optimistic policy π k = arg max π ∈ Π 1 N (cid:80) Ni
Revised: Calculate the optimistic policy π k = arg max π ∈ Π 1 N (cid:80) Ni

Original: =

Revised: =

Original: ˆ V π M i , 1 ( s 1 ) .
Revised: ˆ V π M i , 1 ( s 1 ) .

Original: 15: for i = 1 , 2 , · · · , N do
Revised: 15: for i = 1 , 2 , · · · , N do

Original: Execute the policy π k on MDP M i for one episode, and observe the trajectory ( s M i ,k,h , a M i ,k,h , r M i ,k,h ) Hh =1 .
Revised: Execute the policy π k on MDP M i for one episode, and observe the trajectory ( s M i ,k,h , a M i ,k,h , r M i ,k,h ) Hh =1 .

Original: Output : a policy selected uniformly randomly from the policy set { π k } Kk =1 .
Revised: Output : a policy selected uniformly randomly from the policy set { π k } Kk =1 .

Original: matches traditional bounds in supervised learning, and it implies that when we pursue the optimal in expectation , generalization in RL enjoys quantitatively similar upper bound to supervised learning.
Revised: matches traditional bounds in supervised learning, and it implies that when we pursue the optimal in expectation , generalization in RL enjoys quantitatively similar upper bound to supervised learning.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: This work theoretically studies how much pre-training can improve test performance under different generalization settings.
Revised: This work theoretically studies how much pre-training can improve test performance under different generalization settings.

Original: we first point out that RL generalization is fundamentally different from the generalization in supervised learning, and fine-tuning on the target environment is necessary for good generalization.
Revised: we first point out that RL generalization is fundamentally different from the generalization in supervised learning, and fine-tuning on the target environment is necessary for good generalization.

Original: When the agent can interact with the test environment to update the policy, we first prove that the prior information obtained in the pre-training stage can be theoretically useless in the asymptotic setting, and show that in non-asymptotic setting we can reduce the test-time regret to ˜ O (cid:0)(cid:112) C ( D ) K (cid:1) by designing an efficient learning algorithm.
Revised: When the agent can interact with the test environment to update the policy, we first prove that the prior information obtained in the pre-training stage can be theoretically useless in the asymptotic setting, and show that in non-asymptotic setting we can reduce the test-time regret to ˜ O (cid:0)(cid:112) C ( D ) K (cid:1) by designing an efficient learning algorithm.

Original: In addition, when the agent cannot interact with the test environment, we provide an efficient algorithm called OMERM which returns a near-optimal policy in expectation by interacting with O (cid:0) log( N Π ϵ/ (12 H ) )
Revised: In addition, when the agent cannot interact with the test environment, we provide an efficient algorithm called OMERM which returns a near-optimal policy in expectation by interacting with O (cid:0) log( N Π ϵ/ (12 H ) )

Original: /ϵ 2 (cid:1)
Revised: /ϵ 2 (cid:1)

Original: MDP instances.
Revised: MDP instances.

Original: Our work seeks a comprehensive understanding on how much pre-training can be helpful to test performance theoretically, and it also provides insights on real RL generalization application.
Revised: Our work seeks a comprehensive understanding on how much pre-training can be helpful to test performance theoretically, and it also provides insights on real RL generalization application.

Original: For example, when test time interactions are not allowed, one cannot guarantee to be near optimal in instance .
Revised: For example, when test time interactions are not allowed, one cannot guarantee to be near optimal in instance .

Original: Therefore, for a task where large regret is not tolerable, instead of designing good algorithm, it is more important to find an environment that is close to the target environment and improve policies there, rather than to train a policy in a diverse set of MDPs and hope it can generalize to the target.
Revised: Therefore, for a task where large regret is not tolerable, instead of designing good algorithm, it is more important to find an environment that is close to the target environment and improve policies there, rather than to train a policy in a diverse set of MDPs and hope it can generalize to the target.

Original: In addition, for tasks where we can improve policies on the fly, we can try to pre-train our algorithm in advance to reduce the regret suffered.
Revised: In addition, for tasks where we can improve policies on the fly, we can try to pre-train our algorithm in advance to reduce the regret suffered.

Original: This corresponds to the many applications where test time interactions are very expensive, such as autonomous driving and robotics.
Revised: This corresponds to the many applications where test time interactions are very expensive, such as autonomous driving and robotics.

Original: There are still problems remaining open.
Revised: There are still problems remaining open.

Original: Firstly, we mainly study the i.i.d. case where the training
Revised: Firstly, we mainly study the i.i.d. case where the training

Original: MDPs and test MDPs are sampled from the same distribution.
Revised: MDPs and test MDPs are sampled from the same distribution.

Original: It is an interesting problem to study the out-of-distribution generalization in RL under certain distribution shifting.
Revised: It is an interesting problem to study the out-of-distribution generalization in RL under certain distribution shifting.

Original: Secondly, there is still room for improving our instance dependent bound possibly by leveraging ideas from recent
Revised: Secondly, there is still room for improving our instance dependent bound possibly by leveraging ideas from recent

Original: Bayesian-optimal algorithms such as Thompson sampling (Osband et al., 2013).
Revised: Bayesian-optimal algorithms such as Thompson sampling (Osband et al., 2013).

Original: We hope these problems can be addressed in the future research.
Revised: We hope these problems can be addressed in the future research.

Original: A O MITTED PROOF FOR P ROPOSITION
Revised: A O MITTED PROOF FOR P ROPOSITION

Original: Assume Ω is the set of M MDP instances M 1 , · · · , M M , where all instances consist only one identical state s 1 and their horizon H = 1 .
Revised: Assume Ω is the set of M MDP instances M 1 , · · · , M M , where all instances consist only one identical state s 1 and their horizon H = 1 .

Original: In M i and state s 1 , the reward is 1 for action a i andotherwise.
Revised: In M i and state s 1 , the reward is 1 for action a i andotherwise.

Original: The optimal policy in M i is therefore taking action a t and V π ∗ ( M i )
Revised: The optimal policy in M i is therefore taking action a t and V π ∗ ( M i )

Original: M i = 1 .
Revised: M i = 1 .

Original: For any distribution D , assume the probability to sample M i is p i > 0 .
Revised: For any distribution D , assume the probability to sample M i is p i > 0 .

Original: For any deployed policy ˆ π , assume the probability that ˆ π takes action
Revised: For any deployed policy ˆ π , assume the probability that ˆ π takes action

Original: a
Revised: a

Original: i is q
Revised: i is q

Original: i
Revised: i

Original: , then
Revised: , then

Original: [Equation]
Revised: [Equation]

Original: Therefore, denote ϵ 0 = 1 − max i ∈
Revised: Therefore, denote ϵ 0 = 1 − max i ∈

Original: [ M ] p i > 0 , we have
Revised: [ M ] p i > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Notice that when D is a uniform distribution, ϵ 

Revised: Notice that when D is a uniform distribution, ϵ

Original: = M − 1 M , which can be arbitrarily close to 1 .
Revised: = M − 1 M , which can be arbitrarily close to 1 .

Original: B O MITTED PROOF FOR T HEOREM
Revised: B O MITTED PROOF FOR T HEOREM

Original: To prove the theorem, we let Ω be a subset of MDP with H
Revised: To prove the theorem, we let Ω be a subset of MDP with H

Original: = S
Revised: = S

Original: = 1 , under which it becomes a bandit problem, and it suffices to prove the theorem in this setting.
Revised: = 1 , under which it becomes a bandit problem, and it suffices to prove the theorem in this setting.

Original: Below we first introduce the bandits notations, and give the complete proof.
Revised: Below we first introduce the bandits notations, and give the complete proof.

Original: B. N OTATIONS
Revised: B. N OTATIONS

Original: To be consistent with traditional K -arm bandits problem, in this section we use a slightly different notations from the main paper.
Revised: To be consistent with traditional K -arm bandits problem, in this section we use a slightly different notations from the main paper.

Original: An bandit instance can be represented as a vector r ∈ Ω ⊂ [0 , 1] K , where r k is the mean reward when arm k is pulled.
Revised: An bandit instance can be represented as a vector r ∈ Ω ⊂ [0 , 1] K , where r k is the mean reward when arm k is pulled.

Original: Inherited from previous MDP settings, we also assume that this reward is sampled from an 1 -subgaussian distribution with mean r k .
Revised: Inherited from previous MDP settings, we also assume that this reward is sampled from an 1 -subgaussian distribution with mean r k .

Original: In each episode t ∈ [ T ] , based on initial input and the history, an algorithm A chooses an arm a t to pull, and obtain a reward y a t ∼ D a t .
Revised: In each episode t ∈ [ T ] , based on initial input and the history, an algorithm A chooses an arm a t to pull, and obtain a reward y a t ∼ D a t .

Original: Similarly, assume r is sampled from a distribution D supported on Ω , and we want to minimize the Bayesian regret 
Revised: Similarly, assume r is sampled from a distribution D supported on Ω , and we want to minimize the Bayesian regret [Equation] , where

Original: [Equation]
Revised: [Equation]

Original: Here r ∗ = max k r k is the optimal arm.
Revised: Here r ∗ = max k r k is the optimal arm.

Original: If we define S T k ( r, A ) = (cid:80) [-T t =
-]
Revised: If we define S Tk ( r, A ) = (cid:80) [-T t =
-]Tt

Original: {+=
+} I [ a t = k ] as the r.v. of how many times A has pulled arm k in T episodes and ∆ k
Revised: {+=
+} I [ a t = k ] as the r.v. of how many times A has pulled arm k in T episodes and ∆ k

Original: = r ∗ − r k as the sub-optimal gap, we can decompose regret
Revised: = r ∗ − r k as the sub-optimal gap, we can decompose regret

Original: [Equation]
Revised: [Equation]

Original: This identity is frequently used in the subsequent proof.
Revised: This identity is frequently used in the subsequent proof.

Original: B. P ROOF
Revised: B. P ROOF

Original: We first specify the choice of support, constant and algorithm.
Revised: We first specify the choice of support, constant and algorithm.

Original: Without loss of generality, we set Ω =
Revised: Without loss of generality, we set Ω =

Original: [0 , 1] K , which is quite common and general in bandit tasks.
Revised: [0 , 1] K , which is quite common and general in bandit tasks.

Original: Let c 

Revised: Let c

Original: = 116 , and ˆ A be the Asymptotically Optimal UCB Algorithm defined in Algorithm 3.
Revised: = 116 , and ˆ A be the Asymptotically Optimal UCB Algorithm defined in Algorithm 3.

Original: On the other hand, let ˜ A be uniformly optimal,
Revised: On the other hand, let ˜ A be uniformly optimal,

Original: i.e. ˜ A ( D , T ) =
Revised: i.e. ˜ A ( D , T ) =

Original: arg min
Revised: arg min

Original: A Reg T ( D , A ) .
Revised: A Reg T ( D , A ) .

Original: To prove the theorem, we only need to show that
Revised: To prove the theorem, we only need to show that

Original: [Equation]
Revised: [Equation]

Original: which is the major result (point (2) ).
Revised: which is the major result (point (2) ).

Original: Later, we show that lim T →∞ Reg T ( D , ˜
Revised: Later, we show that lim T →∞ Reg T ( D , ˜

Original: A ( D , T )) =
Revised: A ( D , T )) =

Original: + ∞ in
Revised: + ∞ in

Original: Lemma 9, which proves point (1) in the theorem.
Revised: Lemma 9, which proves point (1) in the theorem.

Original: When T is fixed, we abbreviate ˜ A
Revised: When T is fixed, we abbreviate ˜ A

Original: ( D , T ) , ˆ A ( T ) as ˜ A , ˆ A .
Revised: ( D , T ) , ˆ A ( T ) as ˜ A , ˆ A .

Original: ˆ A enjoys a well known instance dependent regret upper bound, which is copied below:
Revised: ˆ A enjoys a well known instance dependent regret upper bound, which is copied below:

Original: Lemma 4 (Theorem 8.1 in Lattimore & Szepesv´ari (2020) ) .
Revised: Lemma 4 (Theorem 8.1 in Lattimore & Szepesv´ari (2020) ) .

Original: For all r ∈ Ω , ∀ k ∈
Revised: For all r ∈ Ω , ∀ k ∈

Original: [ K ] ,
Revised: [ K ] ,

Revised: E [

Revised: [Equation]

Revised: r

Original: [Equation]
Revised: [Equation]

Revised: A

Revised: [Equation]

Revised: ] ≤ min { T

Revised: [Equation]

Original: By Holder Inequality, we immediately have
Revised: By Holder Inequality, we immediately have

Original: [Equation]
Revised: [Equation]

Original: where the coefficient function
Revised: where the coefficient function

Original: [Equation]
Revised: [Equation]

Original: being non-increasing and lim T →∞ u ( T )
Revised: t log 2 t + 1) + 1) 1 / 3 (cid:3)being non-increasing and lim T →∞ u ( T )

Original: = 2 .
Revised: = 2 .

Original: For simplicity, we define e ( T ) ≜ (cid:113) u ( T )log T T −and s (∆ , T ) ≜ min { ∆ T, ∆ + u ( T ) ∆ log T } .
Revised: For simplicity, we define e ( T ) ≜ (cid:113) u ( T )log T T −and s (∆ , T ) ≜ min { ∆ T, ∆ + u ( T ) ∆ log T } .

Original: We can upper bound Reg T ( r, ˆ A ) ≤ (cid:80)
Revised: We can upper bound Reg T ( r, ˆ A ) ≤ (cid:80)

Original: Kk =1 s (∆ k , T ) .
Revised: Kk =1 s (∆ k , T ) .

Original: Decomposition of Inq.
Revised: Decomposition of Inq.

Original: For k ∈ [ K ] , define Ω kT = { r ∈ Ω : ∆ k ≥ T − p 0 } , where p 0 ∈ (0 , 12 ) is a universal constant to be specified later.
Revised: For k ∈ [ K ] , define Ω kT = { r ∈ Ω : ∆ k ≥ T − p 0 } , where p 0 ∈ (0 , 12 ) is a universal constant to be specified later.

Original: Further define Λ ϵk = { r ∈ Ω , r k + ∆ k (1 + ϵ ) < 1 } .
Revised: Further define Λ ϵk = { r ∈ Ω , r k + ∆ k (1 + ϵ ) < 1 } .

Original: Using
Revised: Using

Original: Eq. 1, it suffices to prove the lemma if ∀ k ∈ [ K ] ,
Revised: Eq. 1, it suffices to prove the lemma if ∀ k ∈ [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: Fix k ∈
Revised: Fix k ∈

Original: [ K ] and ϵ > 0 that is sufficiently small, we decompose Inq.
Revised: [ K ] and ϵ > 0 that is sufficiently small, we decompose Inq.

Original: 3 as below:
Revised: 3 as below:

Original: [Equation]
Revised: [Equation]

Original: Sequentially denote the four terms in the right hand side as M 1 ∼ M 4 .
Revised: Sequentially denote the four terms in the right hand side as M 1 ∼ M 4 .

Original: M 4 can be lower bounded by 1 based on the Lemma 4.
Revised: M 4 can be lower bounded by 1 based on the Lemma 4.

Original: The rest three terms is bounded by the subsequent three lemmas.
Revised: The rest three terms is bounded by the subsequent three lemmas.

Original: Lemma 5.
Revised: Lemma 5.

Original: ∀ k ∈
Revised: ∀ k ∈

Original: [ K ] ,
Revised: [ K ] ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 5 lower bounds M 3 , saying that the influence of instance in Ω \ Ω kT is negligible.
Revised: Lemma 5 lower bounds M 3 , saying that the influence of instance in Ω \ Ω kT is negligible.

Original: The proof of this lemma needs theory on Lebesgue Integral and Radon Transform, which will be introduced in the Section B.3.
Revised: The proof of this lemma needs theory on Lebesgue Integral and Radon Transform, which will be introduced in the Section B.3.

Original: Lemma 6.
Revised: Lemma 6.

Original: For any k ∈ [ K ] , ϵ ∈ (0 , 1) ,
Revised: For any k ∈ [ K ] , ϵ ∈ (0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 implies that M 2 → 1 , allowing us to focus on the integral on a calibrated smaller set Ω kT (cid:84) Λ ϵk , in which we can use information theory to control E [ S Tk ( r, ˜ A )] .
Revised: Lemma 6 implies that M 2 → 1 , allowing us to focus on the integral on a calibrated smaller set Ω kT (cid:84) Λ ϵk , in which we can use information theory to control E [ S Tk ( r, ˜ A )] .

Original: The following lemma is the major lemma in our proof, which use the optimality of ˜ A to analyze the global structure of
Revised: The following lemma is the major lemma in our proof, which use the optimality of ˜ A to analyze the global structure of

Original: Reg T ( r, ˜ A ) for r ∈ Ω k T (cid:84) Λ ϵ k , and lower bound term M 1 :
Revised: Reg T ( r, ˜ A ) for r ∈ Ω kT (cid:84) Λ ϵk , and lower bound term M 1 :

Original: Lemma 7.
Revised: Lemma 7.

Original: For any ϵ ∈ (0 , 1) , we have
Revised: For any ϵ ∈ (0 , 1) , we have

Original: [Equation]
Revised: [Equation]

Original: Combined together,
Revised: Combined together,

Original: [Equation]
Revised: [Equation]

Original: The proof of Inq. 2 is finished by selecting p 0 = 18 and letting ϵ → 0 . 
Revised: The proof of Inq. 2 is finished by selecting p 0 = 18 and letting ϵ → 0 . : Input

Revised: , total episode T

Original: Algorithm
Revised: : for step t = 1 , · · · , K do 3: Choose arm a and obtain reward y t 4: Set ˆ R t = y t , ˆ S t = for step

Original: B.
Revised: K + 1 , · · · , T do 6: f ( t ) = 1 + t log 2 ( t ) 7: Choose a t = arg max k (cid:16) ˆ R

Revised: k

Revised: ˆ S k

Revised: + (cid:113) 2log f ( t ) ˆ S k (cid:17) 8:

Revised: Set

Revised: ˆ R k

Revised: = R k + y t , S k = S k

Revised: +

Original:  P ROOF OF L EMMAS
Revised: B. P ROOF OF L EMMAS

Original: Before proving all lemmas above, we need to introduce theory on Lebesgue Integral and Radon
Revised: Before proving all lemmas above, we need to introduce theory on Lebesgue Integral and Radon

Original: Transform.
Revised: Transform.

Original: In R K space where K ≥ 2 , when a function is Riemann integrable, it is also Lebesgue integrable, and two integrals are equal.
Revised: In R K space where K ≥ 2 , when a function is Riemann integrable, it is also Lebesgue integrable, and two integrals are equal.

Original: Since we assume p ( r ) ∈ C (Ω) , the integral always exists.
Revised: Since we assume p ( r ) ∈ C (Ω) , the integral always exists.

Original: Below we always consider the Lebesgue integral.
Revised: Below we always consider the Lebesgue integral.

Original: For a compact measurable set S , define L p ( S ) as the space of all measurable function in S with standard p -norm.
Revised: For a compact measurable set S , define L p ( S ) as the space of all measurable function in S with standard p -norm.

Original: Since the p.d.f of D is continuous in compact set Ω and is positive, ∃ L, U ∈ R + , ∀ r ∈ Ω , p ( r ) ∈
Revised: Since the p.d.f of D is continuous in compact set Ω and is positive, ∃ L, U ∈ R + , ∀ r ∈ Ω , p ( r ) ∈

Original: [ L, U ] .
Revised: [ L, U ] .

Original: Denote T k = { r ∈ Ω : r k = max( r ) } .
Revised: Denote T k = { r ∈ Ω : r k = max( r ) } .

Original: Clearly, (cid:82) Ω f ( r )d r = (cid:80) k ∈
Revised: Clearly, (cid:82) Ω f ( r )d r = (cid:80) k ∈

Original: [ K ] (cid:82) T k f ( r )d r since m ( T i (cid:84) T j ) = 0 , ∀T
Revised: [ K ] (cid:82) T k f ( r )d r since m ( T i (cid:84) T j ) = 0 , ∀T

Original: i ̸ = T j .
Revised: i ̸ = T j .

Original: Here m ( · ) is the Lebesgue measure in R K .
Revised: Here m ( · ) is the Lebesgue measure in R K .

Original: If we define P t,γ = { r ∈ Ω , γ · r = t } and γ ik =
Revised: If we define P t,γ = { r ∈ Ω , γ · r = t } and γ ik =

Original: ( e i − e k ) , i ̸
Revised: ( e i − e k ) , i ̸

Original: =
Revised: =

Original: k , then P t,γ ik (cid:84) T i = { r ∈ T i :
Revised: k , then P t,γ ik (cid:84) T i = { r ∈ T i :

Original: ∆ k = t } .
Revised: ∆ k = t } .

Original: According to
Revised: According to

Original: Radon Transform theory, since T i is compact and p ( r )
Revised: Radon Transform theory, since T i is compact and p ( r )

Original: ∈ C ( T i ) ,
Revised: ∈ C ( T i ) ,

Original: [Equation]
Revised: [Equation]

Original: , is also continuous w.r.t variable t ∈ [0 , 1] for all k ̸ = i ∈
Revised: p [Equation] r )d r, is also continuous w.r.t variable t ∈ [0 , 1] for all k ̸ = i ∈

Original: [ K ] .
Revised: [ K ] .

Original: Here the integration is perform in the corresponding R K − 1 space, i.e. the plane P t,γ ik, and when m K − 1 ( P t,γ ik (cid:84) T i ) >0 ,
Revised: Here the integration is perform in the corresponding R K − 1 space, i e. the plane P t,γ ik and when m K − 1 ( P t,γ ik (cid:84) T i ) >

Original: [Equation]

Original: We further define q k ( t ) =
Revised: We further define q k ( t ) =

Original: (cid:80) i ̸
Revised: (cid:80) i ̸

Original: = k ρ ik ( t ) , which also belongs to C ([0 , 1]) .
Revised: = k ρ ik ( t ) , which also belongs to C ([0 , 1]) .

Original: The continuity of q k ( t ) helps derive the following equation.
Revised: The continuity of q k ( t ) helps derive the following equation.

Original: For any f ∈ L 1 (Ω) that relies only on ∆ k , i.e. f ( r ) = ˜ f (∆ k ) , we have
Revised: For any f ∈ L 1 (Ω) that relies only on ∆ k , i.e. f ( r ) = ˜ f (∆ k ) , we have

Original: [Equation]
Revised: [Equation]

Original: Here the last line is because of Fubini Theorem, which states that the integral of a function can be computed by iterating lower-dimensional integrals in any order.
Revised: Here the last line is because of Fubini Theorem, which states that the integral of a function can be computed by iterating lower-dimensional integrals in any order.

Original: The factor √ 22 arises because in traditional Radon Transform we require ∥ γ ∥ 2 = 1 , but here ∥ γ ik ∥ 2 = ∥ e i − e k ∥ 2 = √ 2 .
Revised: The factor √ 22 arises because in traditional Radon Transform we require ∥ γ ∥ 2 = 1 , but here ∥ γ ik ∥ 2 = ∥ e i − e k ∥ 2 = √ 2 .

Original: B.3. P ROOF OF L EMMA
Revised: B.3. P ROOF OF L EMMA

Original: Recall that s (∆ , T ) =
Revised: Recall that s (∆ , T ) =

Original: min { ∆ T, ∆ + u ( T ) ∆ log T } , and ∆ T < ∆ + u ( T ) ∆ log T ↔ ∆ <
Revised: min { ∆ T, ∆ + u ( T ) ∆ log T } , and ∆ T < ∆ + u ( T ) ∆ log T ↔ ∆ <

Original: e ( T ) , where e ( T ) is defined as (cid:113) u ( T )log T T
Revised: e ( T ) , where e ( T ) is defined as (cid:113) u ( T )log T T

Original: −.
Revised: −.

Original: When e ( T ) ≤ T − p 0 , Eq. 6 implies that ∀ r ∈ Ω kT , 
Revised: When e ( T ) ≤ T − p 0 , Eq. 6 implies that ∀ r ∈ Ω kT , lim inf T →∞ [Equation] lim

Original: [Equation]
Revised: [Equation]

Original: To prove the lemma, it suffices to show that
Revised: To prove the lemma [Equation] it suffices to show that

Original: [Equation]
Revised: [Equation]

Original: Define E T = { x ∈ [0 , 1] : x ≤ e ( T ) } , F T = { x ∈ [0 , 1] : x ≤ T − p 0 } , thenlim T →∞
Revised: Define E T = { x ∈ [0 , 1] : x ≤ e ( T ) } , F T = { x ∈ [0 , 1] : x ≤ T − p 0 } , then

Original: [Equation]
Revised: [Equation]

Original: We have shown that ρ ik ( t ) m K − 1 ( P t,γik ) ∈
Revised: We have shown that ρ ik ( t ) m K − 1 ( P t,γik ) ∈

Original: [ L, U ] .
Revised: [ L, U ] .

Original: With some calculation, m ( t ) ≜ m K − 1 ( P t,γ ik (cid:84) T i ) =
Revised: With some calculation, m ( t ) ≜ m K − 1 ( P t,γ ik (cid:84) T i ) =

Original: 1 − t K − 1 K − 1 .
Revised: 1 − t K − 1 K − 1 .

Original: Therefore, q k ( t ) ∈ [ m ( t )( K − 1) L, m ( t )( K − 1) U ] , and q k (0) =
Revised: Therefore, q k ( t ) ∈ [ m ( t )( K − 1) L, m ( t )( K − 1) U ] , and q k (0) =

Original: lim t → 0 + q k ( t ) ≥
Revised: lim t → 0 + q k ( t ) ≥

Original: L > 0 .
Revised: L > 0 .

Original: By this continuity, for small enough ε 1 > 0 , ∃ δ 1 > 0 , ∀ t ∈ [0 , δ 1 ] , q k ( t ) ∈
Revised: By this continuity, for small enough ε 1 > 0 , ∃ δ 1 > 0 , ∀ t ∈ [0 , δ 1 ] , q k ( t ) ∈

Original: [(1 − ε 1 ) q k (0) , (1 + ε 1 ) q k (0)] .
Revised: [(1 − ε 1 ) q k (0) , (1 + ε 1 ) q k (0)] .

Original: As a result,
Revised: As a result,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This implies that the limitation of 8 exists and equals to 0 .
Revised: This implies that the limitation of 8 exists and equals to 0 .

Original: Similarly, define G T = { x ∈ [0 , 1] : x ≤log T } , then G T , F T ⊂ [0 , δ 1 ] for sufficiently large T .
Revised: Similarly, define G T = { x ∈ [0 , 1] : x ≤log T } , then G T , F T ⊂ [0 , δ 1 ] for sufficiently large T .

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: Here the second equation comes from Inq.
Revised: Here the second equation comes from Inq.

Original: The proof of Inq.
Revised: The proof of Inq.

Original: 7 is finished by letting ε 1 → 0 .
Revised: 7 is finished by letting ε 1 → 0 .

Original: B.3.
Revised: B.3.

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Recall that Λ ϵk is defined as { r ∈ Ω , r k + ∆ k
Revised: Recall that Λ ϵk is defined as { r ∈ Ω , r k + ∆ k

Original: (1 + ϵ ) < 1 } .
Revised: (1 + ϵ ) < 1 } .

Original: For all t ∈ [0 , 1) , i ̸
Revised: For all t ∈ [0 , 1) , i ̸

Original: = k ,
Revised: = k ,

Original: P t,γ ik (cid:92) T i (cid:92) Λ ϵk = { r ∈ P t,γ ik : r i = max j r j , r ∈ Λ ϵk } = { r ∈ Ω : r i = max j r j , r
Revised: P t,γ ik (cid:92) T i (cid:92) Λ ϵk = { r ∈ P t,γ ik : r i = max j r j , r ∈ Λ ϵk } = { r ∈ Ω : r i = max j r j , r

Original: i − r k = t, r ∈ Λ ϵk } = { r ∈ Ω : r i = max j r j , r i − r k = t, r i + ϵt < 1 } = { r ∈ P t,γ ik (cid:92) T i : r i < 1 − ϵt } .
Revised: i − r k = t, r ∈ Λ ϵk } = { r ∈ Ω : r i = max j r j , r i − r k = t, r i + ϵt < 1 } = { r ∈ P t,γ ik (cid:92) T i : r i < 1 − ϵt } .

Original: This implies that m
Revised: This implies that m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ik
Revised: ik

Original: [Equation]
Revised: [Equation]

Original: = (cid:40)
Revised: = (cid:40)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: 11+ ϵ 1 − t K
Revised: 11+ ϵ 1 − t K

Original: [Equation]
Revised: [Equation]

Original: K −o.w.
Revised: K −o.w.

Original: Notice that Λ ϵk is open, T i \ Λ ϵk is compact, and ( S 1 (cid:84) S 2 ) \ S 

Revised: Notice that Λ ϵk is open, T i \ Λ ϵk is compact, and ( S 1 (cid:84) S 2 ) \ S

Original: = S 1 (cid:84) ( S 2 \ S 3 ) .
Revised: = S 1 (cid:84) ( S 2 \ S 3 ) .

Original: We have ˜ ρ ik ( t ) ≜ (cid:90)
Revised: We have ˜ ρ ik ( t ) ≜ (cid:90)

Original: P t,γik (cid:84) ( T
Revised: P t,γik (cid:84) ( T

Original: i \ Λ ϵk ) p
Revised: i \ Λ ϵk ) p

Original: ( r )d r ∈ C
Revised: ( r )d r ∈ C

Original: ( T i \ Λ ϵk ) .
Revised: ( T i \ Λ ϵk ) .

Original: For all i ̸
Revised: For all i ̸

Original: = k (Define O =
Revised: = k (Define O =

Original: [ T − p 0 , 1] ), lim T →∞ (cid:82)
Revised: [ T − p 0 , 1] ), lim T →∞ (cid:82)

Original: [Equation]
Revised: [Equation]

Original: ϵk p
Revised: ϵk p

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: ∆
Revised: ∆

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: d r
Revised: d r

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: lim
Revised: lim

Original: [Equation]
Revised: [Equation]

Original: = lim
Revised: = lim

Original: [Equation]
Revised: [Equation]

Original: ≤ lim
Revised: ≤ lim

Original: [Equation]
Revised: [Equation]

Original: U
Revised: U

Original: L · (cid:82)
Revised: L · (cid:82)

Original: [Equation]
Revised: [Equation]

Original: Here the first equation is based on Eq.6, and the second last Inq. comes from Bernoulli inequality.
Revised: Here the first equation is based on Eq.6, and the second last Inq. comes from Bernoulli inequality.

Original: B.3. P ROOF OF L EMMA
Revised: B.3. P ROOF OF L EMMA

Original: To prove the lemma, We first reiterate an important lemma that lower bounds E [ S Tk ( r, A )] .
Revised: To prove the lemma, We first reiterate an important lemma that lower bounds E [ S Tk ( r, A )] .

Original: Lemma 8 (Lemma 16.3 in Lattimore & Szepesv´ari (2020)) .
Revised: Lemma 8 (Lemma 16.3 in Lattimore & Szepesv´ari (2020)) .

Original: Let r, r ′ ∈
Revised: Let r, r ′ ∈

Original: Ω be 2 instances that differs only in one arm k ∈
Revised: Ω be 2 instances that differs only in one arm k ∈

Original: [ K ] , where ∆ k > 0 in r and k uniquely optimal in r ′ .
Revised: [ K ] , where ∆ k > 0 in r and k uniquely optimal in r ′ .

Original: Then for any algorithm
Revised: Then for any algorithm

Original: A , ∀ T ,
Revised: A , ∀ T ,

Original: [Equation]
Revised: [Equation]

Original: For fixed ϵ and r we set r ′ k
Revised: For fixed ϵ and r we set r ′ k

Original: = r k + (1 + ∆ k ) ϵ ,
Revised: = r k + (1 + ∆ k ) ϵ ,

Original: so for r ∈ Λ ϵk , r ′ ∈ Ω .
Revised: so for r ∈ Λ ϵk , r ′ ∈ Ω .

Original: On the other hand, when
Revised: On the other hand, when

Original: T −
Revised: T −

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k .
Revised: k .

Original: According to Lemma 8, for all r ∈ Λ ϵk ,
Revised: According to Lemma 8, for all r ∈ Λ ϵk ,

Original: [Equation]
Revised: [Equation]

Original: Define I kT ≜ (cid:82) Ω kT (cid:84) Λ ϵk p ( r ) ∆ k log T d r < ∞ , and we have q kT ( r )
Revised: Define I kT ≜ (cid:82) Ω kT (cid:84) Λ ϵk p ( r ) ∆ k log T d r < ∞ , and we have q kT ( r )

Original: = p ( r ) /
Revised: = p ( r ) /

Original: ∆ k I kn is thus a p.d.f.
Revised: ∆ k I kn is thus a p.d.f.

Original: in Ω kT (cid:84) Λ ϵk .
Revised: in Ω kT (cid:84) Λ ϵk .

Original: We already know that for any 1-subgaussian bandit instance, the regret of UCB algorithm is bounded by 8 √ KT log T + 3 (cid:80) Kk =
Revised: We already know that for any 1-subgaussian bandit instance, the regret of UCB algorithm is bounded by 8 √ KT log T + 3 (cid:80) Kk =

Original: [Equation]
Revised: [Equation]

Original: ∆ k ≤ 9 √ KT log T .
Revised: ∆ k ≤ 9 √ KT log T .

Original: The optimality of ˜ A
Revised: The optimality of ˜ A

Original: ( D , T ) =
Revised: ( D , T ) =

Original: arg min
Revised: arg min

Original: A Reg T ( D , A ) implies that Reg T ( D , ˜ A ) ≤ 9 √ KT log T .
Revised: A Reg T ( D , A ) implies that Reg T ( D , ˜ A ) ≤ 9 √ KT log T .

Original: Due to the concavity of log( · ) ,
Revised: Due to the concavity of log( · ) ,

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have lim inf
Revised: we have lim inf

Original: [Equation]
Revised: [Equation]

Original: 11) ≥
Revised: 11) ≥

Original: [Equation]
Revised: [Equation]

Original: Here the last third line is because r ≥ T − p 0 , ∀ r ∈ Ω k T
Revised: Here the last third line is because r ≥ T − p 0 , ∀ r ∈ Ω k T

Original: [Equation]
Revised: [Equation]

Original: B.
Revised: B.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: P ROOF
Revised: P ROOF

Original: IN
Revised: IN

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: In Lemma 7 we already show that
Revised: In Lemma 7 we already show that

Original: [Equation]
Revised: [Equation]

Original: To prove this lemma
Revised: To prove this lemma

Original: [Equation]
Revised: [Equation]

Original: if suffices to show that
Revised: if suffices to show that

Original: [Equation]
Revised: [Equation]

Original: Notice that Ω kT (cid:84) Λ ϵk is non-decreasing in terms of T .
Revised: Notice that Ω kT (cid:84) Λ ϵk is non-decreasing in terms of T .

Original: [Equation]
Revised: [Equation]

Original: C O MITTED D ETAILS FOR T HEOREMC. S UBROUTINE FOR FINDING THE COVER SET
Revised: C O MITTED D ETAILS FOR T HEOREMC. S UBROUTINE FOR FINDING THE COVER SET

Original: This subroutine is used to find a policy-value set ˆΠ such that ˆΠ covers (1 − 3 δ ) -fraction of the MDPs in the sampled MDP set,
Revised: This subroutine is used to find a policy-value set ˆΠ such that ˆΠ covers (1 − 3 δ ) -fraction of the MDPs in the sampled MDP set,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.
Revised: e.

Original: (cid:80) Ni
Revised: (cid:80) Ni

Original: =

Revised: =

Original: I [ ∃ ( π, v ) ∈ ˆΠ , s.t.
Revised: I [ ∃ ( π, v ) ∈ ˆΠ , s.t.

Original: ( π, v ) covers M i ]
Revised: ( π, v ) covers M i ]

Original: N ≥ 1 − 3 δ.
Revised: N ≥ 1 − 3 δ.

Original: The algorithm is a greedy algorithm consisting of at most N steps.
Revised: The algorithm is a greedy algorithm consisting of at most N steps.

Original: As the beginning of the algorithm, we calculate a matrix A , where A i,j indicates whether ( π j , v j ) covers the MDP M i .
Revised: As the beginning of the algorithm, we calculate a matrix A , where A i,j indicates whether ( π j , v j ) covers the MDP M i .

Original: In each step t , we find a policy-value pair ( π j t , v j t ) with the maximum cover number in the uncovered MDP set T t − 1 .
Revised: In each step t , we find a policy-value pair ( π j t , v j t ) with the maximum cover number in the uncovered MDP set T t − 1 .

Original: We update the index set U t and T t according to the selected index j t .
Revised: We update the index set U t and T t according to the selected index j t .

Original: We output the policy-value set ˆΠ once the cover size (cid:80) tτ =1 n τ ≥ (1 − 3 δ ) N .
Revised: We output the policy-value set ˆΠ once the cover size (cid:80) tτ =1 n τ ≥ (1 − 3 δ ) N .

Original: Calculate
Revised: Calculate

Original: C. P ROOF FOR THE P RE - TRAINING S TAGE
Revised: C. P ROOF FOR THE P RE - TRAINING S TAGE

Original: During the proof, we use Ω ∗ to denote the MDP set satisfying the (1 − δ ) -cover condition with minimum cardinality, i.e. Ω ∗ = arg min P (˜Ω) ≥ 1 − δ | ˜Ω | .
Revised: During the proof, we use Ω ∗ to denote the MDP set satisfying the (1 − δ ) -cover condition with minimum cardinality, i.e. Ω ∗ = arg min P (˜Ω) ≥ 1 − δ | ˜Ω | .

Original: As defined in Algorithm 4, we use U to denote the index set of ˆΩ , which has the same cardinality as ˆΩ .
Revised: As defined in Algorithm 4, we use U to denote the index set of ˆΩ , which has the same cardinality as ˆΩ .

Original: We have the following lemma for the pre-training stage.
Revised: We have the following lemma for the pre-training stage.

Original: Lemma 10 (Pre-training algorithm) .
Revised: Lemma 10 (Pre-training algorithm) .

Original: With probability at least 1 − O (cid:16) δ log C ( D ) δ (cid:17) , the pre-training stage algorithm returns within log (cid:16) ˜
Revised: With probability at least 1 − O (cid:16) δ log C ( D ) δ (cid:17) , the pre-training stage algorithm returns within log (cid:16) ˜

Original: O
Revised: O

Original: ( C ( D ))
Revised: ( C ( D ))

Original: (cid:17) phases with total MDP sample complexity bounded by ˜ O (cid:16) C ( D ) δ 2 (cid:17) .
Revised: (cid:17) phases with total MDP sample complexity bounded by ˜ O (cid:16) C ( D ) δ 2 (cid:17) .

Original: The return set ˆΠ satisfies
Revised: The return set ˆΠ satisfies

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and the size is bounded by | ˆΠ | ≤ 2 C ( D ) log 1 δ .
Revised: and the size is bounded by | ˆΠ | ≤ 2 C ( D ) log 1 δ .

Original: Proof.
Revised: Proof.

Original: For each phase, we first define the high-probability event in Lemma 11.
Revised: For each phase, we first define the high-probability event in Lemma 11.

Original: We prove the lemma in Appendix C.2.1.
Revised: We prove the lemma in Appendix C.2.1.

Original: Lemma 11 (High Probability Events) .
Revised: Lemma 11 (High Probability Events) .

Original: For all phases, with probability at least 1 − 4 δ , the following events hold:
Revised: For all phases, with probability at least 1 − 4 δ , the following events hold:

Original: (cid:12)(cid:12)(cid:12) 1 N (cid:80) Ni
Revised: (cid:12)(cid:12)(cid:12) 1 N (cid:80) Ni

Original: =

Revised: =

Original: I [ M i ∈
Revised: I [ M i ∈

Original: Ω ∗ ] − P (Ω ∗ ) (cid:12)(cid:12)(cid:12) ≤ δ .
Revised: Ω ∗ ] − P (Ω ∗ ) (cid:12)(cid:12)(cid:12) ≤ δ .

Original: ∀ i ∈
Revised: ∀ i ∈

Original: [ N ] , π i is ϵ 2 -optimal for M i .
Revised: [ N ] , π i is ϵ 2 -optimal for M i .

Original: ∀
Revised: ∀

Original: i, j ∈ [ N ] , (cid:12)(cid:12)(cid:12)
Revised: i, j ∈ [ N ] , (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: For all index set U ′ ⊂
Revised: For all index set U ′ ⊂

Original: [ N ] , we have
Revised: [ N ] , we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 11, the three events defined in Lemma 11 hold with probability 1 − 4 δ .
Revised: According to Lemma 11, the three events defined in Lemma 11 hold with probability 1 − 4 δ .

Original: As stated in Lemma 12, condition on the first and second events, we have |U| ≤ 2 C
Revised: As stated in Lemma 12, condition on the first and second events, we have |U| ≤ 2 C

Original: ( D ) log 1 δ for each phase.
Revised: ( D ) log 1 δ for each phase.

Original: We prove Lemma 12 in Appendix C.2.2.
Revised: We prove Lemma 12 in Appendix C.2.2.

Original: Note that these lemmas also hold for the last phase in which we return the policy-value set ˆΠ .
Revised: Note that these lemmas also hold for the last phase in which we return the policy-value set ˆΠ .

Original: Lemma 12.
Revised: Lemma 12.

Original: For all phases, if the first and second events in Lemma 11 hold, then the size of candidate set U satisfies
Revised: For all phases, if the first and second events in Lemma 11 hold, then the size of candidate set U satisfies

Original: [Equation]
Revised: [Equation]

Original: Since the stopping condition is
Revised: Since the stopping condition is

Original: [Equation]
Revised: [Equation]

Original: ≤ δ
Revised: ≤ δ

Original: and we already know |U| ≤ 2 C
Revised: and we already know |U| ≤ 2 C

Original: ( D ) log 1 δ , the stopping condition is satisfied for N ≥ 4 C ( D )log 2 ( C ( D ) /δ )
Revised: ( D ) log 1 δ , the stopping condition is satisfied for N ≥ 4 C ( D )log 2 ( C ( D ) /δ )

Original: δ.
Revised: δ.

Original: Based on the doubling trick, we know that the number of total phases is bounded by log (cid:16)
Revised: Based on the doubling trick, we know that the number of total phases is bounded by log (cid:16)

Original: ˜ O ( C ( D )) (cid:17) and the sample complexity is bounded by 2 N = ˜
Revised: ˜ O ( C ( D )) (cid:17) and the sample complexity is bounded by 2 N = ˜

Original: O ( C ( D ) /δ 2 ) .
Revised: O ( C ( D ) /δ 2 ) .

Original: Therefore, by union bound across all phases, with probability at least 1 − O (log C ( D ) δ δ ) , Lemmaholds for all phases.
Revised: Therefore, by union bound across all phases, with probability at least 1 − O (log C ( D ) δ δ ) , Lemmaholds for all phases.

Original: Using the third event on the return set U , we know that
Revised: Using the third event on the return set U , we know that

Original: [Equation]
Revised: [Equation]

Original: Therefore, the property on ˆΠ holds.
Revised: Therefore, the property on ˆΠ holds.

Original: C.2. P ROOF OF L EMMA
Revised: C.2. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: To prove this lemma, we sequentially bound the failure probability for each event
Revised: To prove this lemma, we sequentially bound the failure probability for each event

Original: [Equation]
Revised: [Equation]

Original: Empirical probability of Ω ∗ .
Revised: Empirical probability of Ω ∗ .

Original: Notice that
Revised: Notice that

Original: since M i ∼ D , the expectation of
Revised: since M i ∼ D , the expectation of

Original: r
Revised: r

Original: .v.
Revised: .v.

Original: I
Revised: I

Original: [ M i ∈ Ω ∗ ] is exactly P (Ω ∗ ) .
Revised: [ M i ∈ Ω ∗ ] is exactly P (Ω ∗ ) .

Original: According to the Chernoff Bound, we have
Revised: According to the Chernoff Bound, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the failure rate of the first event is bounded by δ .
Revised: Therefore, the failure rate of the first event is bounded by δ .

Original: Oracle error.
Revised: Oracle error.

Original: For each i ∈ [ N ] , the failure rate of O l is at most δN ; For all i, j ∈ [ N ] , the failure rate of O e is at most δN 2 .
Revised: For each i ∈ [ N ] , the failure rate of O l is at most δN ; For all i, j ∈ [ N ] , the failure rate of O e is at most δN 2 .

Original: Therefore, with probability at least 1 − 2 δ , we know that π i is indeed ϵ 2 -optimal for M i , and (cid:12)(cid:12)(cid:12) v ij − V π j M i , 1 ( s 1 ) (cid:12)(cid:12)(cid:12) <
Revised: Therefore, with probability at least 1 − 2 δ , we know that π i is indeed ϵ 2 -optimal for M i , and (cid:12)(cid:12)(cid:12) v ij − V π j M i , 1 ( s 1 ) (cid:12)(cid:12)(cid:12) <

Original: ϵ 2 for all i, j ∈ [ N ] .
Revised: ϵ 2 for all i, j ∈ [ N ] .

Original: This implies that the failure rate of the second event is bounded by 2 δ .
Revised: This implies that the failure rate of the second event is bounded by 2 δ .

Original: Covering probability of U ′ .
Revised: Covering probability of U ′ .

Original: We first fix any index set U ′ ⊂
Revised: We first fix any index set U ′ ⊂

Original: [ N ] and define U c =
Revised: [ N ] and define U c =

Original: [ N ] \ U ′ .
Revised: [ N ] \ U ′ .

Original: For a policy-value pair ( π, v ) and an MDP M , we define the following random variable
Revised: For a policy-value pair ( π, v ) and an MDP M , we define the following random variable

Original: [Equation]
Revised: [Equation]

Original: Notice that A i,j is exactly an instance of r.v. χ ( π j , v j , M i ) .
Revised: Notice that A i,j is exactly an instance of r.v. χ ( π j , v j , M i ) .

Original: For a fixed index set U ′ ⊂
Revised: For a fixed index set U ′ ⊂

Original: [ N ] , each
Revised: [ N ] , each

Original: MDP M i with index i ∈
Revised: MDP M i with index i ∈

Original: [ N ] \ U ′ can be regarded as an i.i.d.
Revised: [ N ] \ U ′ can be regarded as an i.i.d.

Original: sample from the distribution D .
Revised: sample from the distribution D .

Original: According to Chernoff Bound, with probability at least 1 − δ (2 N ) |U′| we have
Revised: According to Chernoff Bound, with probability at least 1 − δ (2 N ) |U′| we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, we can use χ to control the probability that π j is near optimal for M i .
Revised: On the other hand, we can use χ to control the probability that π j is near optimal for M i .

Original: Specifically,
Revised: Specifically,

Original: [Equation]
Revised: [Equation]

Original: Similar to the analysis in the second event of Lemma
Revised: Similar to the analysis in the second event of Lemma

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ , for all j ∈ U ′ , the return π ′ is ϵ 2 -optimal for M
Revised: with probability at least 1 − 2 δ , for all j ∈ U ′ , the return π ′ is ϵ 2 -optimal for M

Original: [Equation]
Revised: [Equation]

Original: and the estimated value v and v ′ in the RHS is ϵ 2 -close to their mean.
Revised: and the estimated value v and v ′ in the RHS is ϵ 2 -close to their mean.

Original: Assume this event hold, we have
Revised: Assume this event hold, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can substitute the RHS in Eqn.
Revised: we can substitute the RHS in Eqn.

Original: 25 as (where 2 δ is the oracle failure probability)
Revised: 25 as (where 2 δ is the oracle failure probability)

Original: [Equation]
Revised: [Equation]

Original: Combining Eqn.
Revised: Combining Eqn.

Original: 24 and Eqn. 31, by the union bound, with probability at least
Revised: 24 and Eqn. 31, by the union bound, with probability at least

Original: [Equation]
Revised: [Equation]

Original: for all U ′ ⊂ [ N ] , we have
Revised: for all U ′ ⊂ [ N ] , we have

Original: [Equation]
Revised: [Equation]

Original: C.2. P ROOF OF L EMMA
Revised: C.2. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: For a certain fixed phase, we define N M ,t = (cid:80) i ∈T t
Revised: For a certain fixed phase, we define N M ,t = (cid:80) i ∈T t

Original: I [ M i = M ] as the population of M in
Revised: I [ M i = M ] as the population of M in

Original: T t , and ˆ C ≜ (cid:80)
Revised: T t , and ˆ C ≜ (cid:80)

Original: Ni
Revised: Ni

Original: =

Revised: =

Original: I [ M i ∈ Ω ∗ ] .
Revised: I [ M i ∈ Ω ∗ ] .

Original: We have ˆ C = (cid:80) M∈ Ω ∗ N M , 0 .
Revised: We have ˆ C = (cid:80) M∈ Ω ∗ N M , 0 .

Original: Thanks to the conditional events in Lemma 11, we have (cid:12)(cid:12)(cid:12) ˆ CN − P (Ω ∗ ) (cid:12)(cid:12)(cid:12) ≤ δ and ˆ CN ≥ 1 − 2 δ .
Revised: Thanks to the conditional events in Lemma 11, we have (cid:12)(cid:12)(cid:12) ˆ CN − P (Ω ∗ ) (cid:12)(cid:12)(cid:12) ≤ δ and ˆ CN ≥ 1 − 2 δ .

Original: Notice that U is generated by greedily selecting the best policy under current MDP remaining set
Revised: Notice that U is generated by greedily selecting the best policy under current MDP remaining set

Original: T t − 1 , i.e. the policy that covers most MDP in T t − 1 .
Revised: T t − 1 , i.e. the policy that covers most MDP in T t − 1 .

Original: On the other hand, since the second event in
Revised: On the other hand, since the second event in

Original: Lemma 11 holds, for M i =
Revised: Lemma 11 holds, for M i =

Original: M j , we have Cnd ( v i,j , v
Revised: M j , we have Cnd ( v i,j , v

Original: i,i , v j,j ) and Cnd ( v j,i , v i,i , v j,j ) are true and thus A i,j =
Revised: i,i , v j,j ) and Cnd ( v j,i , v i,i , v j,j ) are true and thus A i,j =

Original: A j,i = 1 .
Revised: A j,i = 1 .

Original: Therefore, for each step t and each M ∈ Ω ∗ , if we have M i =
Revised: Therefore, for each step t and each M ∈ Ω ∗ , if we have M i =

Original: M for some i ∈ T t − 1 , then in this round, we have
Revised: M for some i ∈ T t − 1 , then in this round, we have

Original: [Equation]
Revised: [Equation]

Original: Since we choose policy π j t instead of π i in step t according to the greedy strategy, we have
Revised: Since we choose policy π j t instead of π i in step t according to the greedy strategy, we have

Original: [Equation]
Revised: [Equation]

Original: for all M ∈ Ω ∗ .
Revised: for all M ∈ Ω ∗ .

Original: The right term is 0 if M no longer exists in the remaining set, and the inequality still holds.
Revised: The right term is 0 if M no longer exists in the remaining set, and the inequality still holds.

Original: Summing over all M ∈ Ω ∗ , we obtain
Revised: Summing over all M ∈ Ω ∗ , we obtain

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because N M ,t is monotonically decreasing in t and the second last equation is because (cid:80) tτ
Revised: where the last inequality is because N M ,t is monotonically decreasing in t and the second last equation is because (cid:80) tτ

Original: =
Revised: =

Original: 1 n t represents the population of MDPs that are covered in the first t rounds.
Revised: 1 n t represents the population of MDPs that are covered in the first t rounds.

Original: This implies that (notice that n 0 = 0 )
Revised: This implies that (notice that n 0 = 0 )

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: When t ≥ ( C ( D ) + 1) log 1 δ , we have |
Revised: When t ≥ ( C ( D ) + 1) log 1 δ , we have |

Original: [Equation]
Revised: [Equation]

Original: Therefore, upon breaking, the size of U satisfies |U| =
Revised: Therefore, upon breaking, the size of U satisfies |U| =

Original: |U t | ≤ ( C ( D ) + 1) log 1 δ .
Revised: |U t | ≤ ( C ( D ) + 1) log 1 δ .

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: In Lemma 10, we prove that with probability at least 1 − O (cid:16) δ log C ( D ) δ (cid:17) , the policy set ˆΠ returned in the pre-training stage covers the MDPs M ∼ D with probability at least 1 − 6 δ , i.e.
Revised: In Lemma 10, we prove that with probability at least 1 − O (cid:16) δ log C ( D ) δ (cid:17) , the policy set ˆΠ returned in the pre-training stage covers the MDPs M ∼ D with probability at least 1 − 6 δ , i.e.

Original: [Equation]
Revised: [Equation]

Original: Note that this event happens with high probability.
Revised: Note that this event happens with high probability.

Original: If this event does not happen (w.p.
Revised: If this event does not happen (w.p.

Original: O (cid:16) δ log C
Revised: O (cid:16) δ log C

Original: ( D ) δ (cid:17) ), the regret can still be upper bounded by K , which leads to an additional term of K · O (cid:16) δ log C ( D ) δ (cid:17)
Revised: ( D ) δ (cid:17) ), the regret can still be upper bounded by K , which leads to an additional term of K · O (cid:16) δ log C ( D ) δ (cid:17)

Original: =
Revised: =

Original: O (cid:16) √ K log( K C ( D )) (cid:17) in the final bound.
Revised: O (cid:16) √ K log( K C ( D )) (cid:17) in the final bound.

Original: This term is negligible compared with the dominant term in the regret.
Revised: This term is negligible compared with the dominant term in the regret.

Original: In the following analysis, we only discuss the case where the statement in Lemma 10 holds.
Revised: In the following analysis, we only discuss the case where the statement in Lemma 10 holds.

Original: We also assume that for the test MDP M ∗ ,
Revised: We also assume that for the test MDP M ∗ ,

Original: [Equation]
Revised: [Equation]

Original: which will happen with probability 1 − 6 δ under the event defined in Lemma 10.
Revised: which will happen with probability 1 − 6 δ under the event defined in Lemma 10.

Original: We use L to denote the maximum epoch counter, which satisfies L ≤ | ˆΠ | .
Revised: We use L to denote the maximum epoch counter, which satisfies L ≤ | ˆΠ | .

Original: Lemma 13.
Revised: Lemma 13.

Original: (Optimism)
Revised: (Optimism)

Original: With probability at least 1 − δ/ 2 , we have v l ≥ V ∗M ∗ , 1 ( s 1 ) − 2 ϵ,
Revised: With probability at least 1 − δ/ 2 , we have v l ≥ V ∗M ∗ , 1 ( s 1 ) − 2 ϵ,

Original: ∀ l ∈
Revised: ∀ l ∈

Original: [ L ] .
Revised: [ L ] .

Original: Proof.
Revised: Proof.

Original: To prove the lemma, we need to show that the optimal policy-value pair (ˆ π ∗ , ˆ v ∗ ) for M ∗ will never be eliminated from the set ˆΠ l with high probability.
Revised: To prove the lemma, we need to show that the optimal policy-value pair (ˆ π ∗ , ˆ v ∗ ) for M ∗ will never be eliminated from the set ˆΠ l with high probability.

Original: Condition on the sampled M ∗ , for a fixed episode k ∈
Revised: Condition on the sampled M ∗ , for a fixed episode k ∈

Original: [ K ] , by Azuma’s inequality, we have:
Revised: [ K ] , by Azuma’s inequality, we have:

Original: [Equation]
Revised: [Equation]

Original: By union bound over all k ∈ [ K ] , we know that
Revised: By union bound over all k ∈ [ K ] , we know that

Original: [Equation]
Revised: [Equation]

Original: By Inq.
Revised: By Inq.

Original: 38, we know that (cid:12)(cid:12)
Revised: 38, we know that (cid:12)(cid:12)

Original: V ˆ π ∗ M ∗ , 1 ( s 1 ) − ˆ v ∗ (cid:12)(cid:12) < 4 ϵ .
Revised: V ˆ π ∗ M ∗ , 1 ( s 1 ) − ˆ v ∗ (cid:12)(cid:12) < 4 ϵ .

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the elimination condition defined in line 6 of Algorithm 1 in the fine-tuning stage, (ˆ π ∗ , ˆ v ∗ ) will never be eliminated from the set ˆΠ l with probability at least 1 − δ/ 2 .
Revised: By the elimination condition defined in line 6 of Algorithm 1 in the fine-tuning stage, (ˆ π ∗ , ˆ v ∗ ) will never be eliminated from the set ˆΠ l with probability at least 1 − δ/ 2 .

Original: By the definition that v l = max ( π,v ) ∈ ˆΠ l v , we have v l ≥ ˆ v ∗ ≥ V ∗M ∗ , 1 ( s 1 ) − 2 ϵ .
Revised: By the definition that v l = max ( π,v ) ∈ ˆΠ l v , we have v l ≥ ˆ v ∗ ≥ V ∗M ∗ , 1 ( s 1 ) − 2 ϵ .

Original: Now we are ready to prove Theorem 2.
Revised: Now we are ready to prove Theorem 2.

Original: Proof.
Revised: Proof.

Original: We use τ l to denote the starting episode of epoch l .
Revised: We use τ l to denote the starting episode of epoch l .

Original: Without loss of generality, we set τ L +

Revised: Without loss of generality, we set τ L +

Original: = K + 1 .
Revised: = K + 1 .

Original: By lemma 13, we know that v l ≥ V ∗M ∗ , 1 ( s 1 ) with high probability.
Revised: By lemma 13, we know that v l ≥ V ∗M ∗ , 1 ( s 1 ) with high probability.

Original: Under this event, we can decompose the value gap in the following way:
Revised: Under this event, we can decompose the value gap in the following way:

Original: [Equation]
Revised: [Equation]

Original: For the first term, by the elimination condition defined in line 6 of Algorithm 1 in the fine-tuning stage, we have
Revised: For the first term, by the elimination condition defined in line 6 of Algorithm 1 in the fine-tuning stage, we have

Original: [Equation]
Revised: [Equation]

Original: For the second term, by Azuma’s inequality and union bound over all episodes, with probability at least 1 − δ/ 2 ,
Revised: For the second term, by Azuma’s inequality and union bound over all episodes, with probability at least 1 − δ/ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, by Cauchy-Schwarz inequality,
Revised: Therefore, by Cauchy-Schwarz inequality,

Original: [Equation]
Revised: [Equation]

Original: The last inequality is due to | ˆΠ | ≤ 2 C ( D ) log(1 /δ ) by Lemma 10.
Revised: The last inequality is due to | ˆΠ | ≤ 2 C ( D ) log(1 /δ ) by Lemma 10.

Original: Finally
Revised: Finally, we take expectation over all possible M ∗ , and we get

Original: [Equation]
Revised: [Equation]

Original:  we take expectation over all possible M ∗
Revised: In this subsection, we provide a lower bound to show that the regret upper bound in Theorem 2 is tight except for logarithmic factors. The lower bound is stated as follows. Theorem 14.

Original:  [Equation] 
Revised: Suppose | Ω | ≥ 2 and K ≥ 5 . For any pre-training and fine-tuning algorithm Alg, there exists a distribution D over the MDP class Ω , such that the regret in the fine-tuning stage is at least [Equation] This lower bound states that no matter how many samples are collected in the pre-training stage, the regret in the fine-tuning stage is at least Ω (cid:16)(cid:112) C

Revised: ( D ) K (cid:17) , which indicates that our upper bound is near-optimal except for logarithmic factors.

Revised: The proof is from an information theoretical perspective, which shares the similar idea with the lower bound proof for bandits (e.g. Theorem 15.2 in Lattimore & Szepesv ´ ari (2020) and Theorem 5.1 in

Revised: Auer et al.

Revised: We first construct the hard instance and then prove the theorem.

Revised: Since the bandit problem can be regarded as an MDP with horizon 1 , our hard instance is constructed using a distribution over bandit instances.

Revised: For a fixed algorithm Alg, we define Ω to be a set of M multi-armed bandit instances.

Revised: For bandit instance ν i ∈ Ω , there are M arms.

Revised: The reward of arm j is a Guassian distribution with unit variance and mean reward 12 + ∆

Revised: I { i = j } .

Revised: The parameter ∆ ∈ [0 , 1 / 2] will be defined later.

Revised: We use ν 0 to denote the bandit instance where the reward of each arm is a Guassian distribution with unit variance and mean reward 12 .

Revised: We set D to be a uniform distribution over the MDP set Ω .

Revised: We use r k = ⟨ r 1 , · · · , r k ⟩ to denote the sequence of rewards received up through step k ∈ [ K ] .

Revised: Note that any randomized algorithm Alg is equivalent to an a-prior random choice from the set of all deterministic algorithms.

Revised: We can formally regard the algorithm Alg as a fixed function which maps the reward history r k − 1 to the action a k in each step k .

Revised: This technique is not crucial for the proof but simplifies the notations, which has also been applied in Auer et al.

Revised: With a slight abuse of notation, we use Reg K ( ν i , Alg ) to denote the regret of algorithm Alg in bandit instance ν i .

Revised: Therefore, we have Reg K ( D , Alg )

Revised: = 1 M (cid:80) Mi

Revised: =1 Reg K ( ν i , Alg ) .

Revised: We use P ν i and E ν i to denote the probability and the expectation under the condition that the bandit instance in the test stage is ν i , respectively.

Revised: We use D KL ( P 1 , P 2 ) to denote the KL-divergence between the probability measure P 1 and P 2 .

Revised: We use T i ( K ) to denote the number of times arm i is pulled in the

Revised: K steps.

Revised: We first provide the following lemma to upper bound the difference between expectations when measured using E ν i and E ν 0 .

Revised: Lemma 15.

Revised: Let f : R K

Revised: → [0 , K ] be any function defined on the reward sequence r .

Revised: Then for any action i ,

Revised: [Equation]

Revised: Proof.

Revised: We upper bound the difference E

Revised: ν

Original: {+i [ f ( r )] − E ν 
+}
Revised: {+i [ f ( r )] − E ν 
+}

Revised: [ f ( r )] by calculating the expectation w.r.t.

Revised: different probability measure.

Revised: [Equation]

Revised: Here (cid:12)(cid:12)(cid:82) r

Revised: ( d P ν i ( r ) − d P ν 0 ( r )) (cid:12)(cid:12) is the TV-distance between the two probability measure P ν

Original:  andwe get
Revised: i ( r ) and

Revised: P ν 0 ( r ) .

Revised: Note that ν

Revised: i and ν 0 only differs in the expected reward of arm i .

Revised: By Pinsker’s inequality and

Revised: Lemma 15.1 in Lattimore & Szepesv´ari (2020), we have

Revised: [Equation]

Revised: The lemma can be proved by combining the above two inequalities.

Revised: Now we can prove Theorem 14.

Revised: Proof.

Revised: By definition, we have

Revised: [Equation]

Revised: We apply Lemma 15 to T i ( K ) , which is a function of the reward sequence r since the actions of the algorithm Alg are determined by the past rewards.

Revised: We have

Revised: [Equation]

Revised: We sum the above inequality over all ν i ∈ Ω , then we have

Original: [Equation]
Revised: [Equation]

Revised: where the second inequality is due to the Cauchy-Schwarz inequality and the fact that (cid:80) ν i ∈

Revised: [Equation]

Revised: E ν

Revised: [Equation]

Revised: [ T i ( K )]

Revised: = K .

Revised: Plgging this inequality back to Inq.

Revised: 43, we have

Revised: [Equation]

Revised: Since K ≥ 5 , we know that C ( D ) has the same order as M .

Revised: If K ≤ M , we know that (cid:112)

Revised: C ( D ) K

Revised: = Ω( K ) .

Revised: We choose ∆ = 1 / 2 and know that Reg K ( D , Alg ) ≥ Ω( K ) .

Revised: Therefore, we have Reg K ( D , Alg ) ≥ Ω (cid:16) min (cid:16)(cid:112) C

Revised: ( D ) K, K (cid:17)(cid:17) .

Revised: If K ≥ M , we choose ∆ = MK .

Revised: Since

Revised: M ≥ 2 , we can also prove that

Revised: [Equation]

Original: D O MITTED PROOF FOR T HEOREM
Revised: D O MITTED PROOF FOR T HEOREM

Original: We define π ∗ = arg max π ∈ Π E
Revised: We define π ∗ = arg max π ∈ Π E

Original: M∼ D V π M , 1 ( s 1 ) and ˆ π ∗ =
Revised: M∼ D V π M , 1 ( s 1 ) and ˆ π ∗ =

Original: arg max π ∈ Π 1 N (cid:80) Ni
Revised: arg max π ∈ Π 1 N (cid:80) Ni

Original: =1 V π M i , 1 ( s 1 ) .
Revised: =1 V π M i , 1 ( s 1 ) .

Original: For the returned policy ˆ π 
Revised: For the returned policy ˆ π , we can decompose the value gap into the following terms:

Original: [Equation]
Revised: [Equation]

Original: we can decompose the value gap into the following terms:

Original: [Equation]
Revised: [Equation]

Original: Note that the first and the second terms are generalization gap for a given policy, which can be upper bounded by Chernoff bound and union bound.
Revised: Note that the first and the second terms are generalization gap for a given policy, which can be upper bounded by Chernoff bound and union bound.

Original: The third term is the value gap during the training phase.
Revised: The third term is the value gap during the training phase.

Original: The last term is less than 0 by the optimality of ˆ π ∗ .
Revised: The last term is less than 0 by the optimality of ˆ π ∗ .

Original: Upper bounds on the first and the second terms We can bound these terms following the generalization technique.
Revised: Upper bounds on the first and the second terms We can bound these terms following the generalization technique.

Original: Define the distance between polices d ( π 1 , π 2 ) ≜ max s ∈S ,h ∈ [ H ] ∥ π 1 h ( ·| s ) − π 2 h ( ·| s ) ∥ 1 .
Revised: Define the distance between polices d ( π 1 , π 2 ) ≜ max s ∈S ,h ∈ [ H ] ∥ π 1 h ( ·| s ) − π 2 h ( ·| s ) ∥ 1 .

Original: We construct the ϵ 0 -covering set ˜Π w.r.t.
Revised: We construct the ϵ 0 -covering set ˜Π w.r.t.

Original: d such that
Revised: d such that

Original: [Equation]
Revised: [Equation]

Original: By Inq. 47, we have
Revised: By Inq. 50, we have

Original: [Equation]
Revised: [Equation]

Original: By definition of the covering number,(cid:12) (cid:12)(cid:12)˜Π (cid:12)  (cid:12)(cid:12)  =
Revised: By definition of the covering number, (cid:12)(cid:12) (cid:12) ˜Π (cid:12)(cid:12) (cid:12) =

Original: N (Π , ϵ 0 , d ) .
Revised: N (Π , ϵ 0 , d ) .

Original: By Chernoff bound and union bound over the policy set ˜Π , we have with prob.
Revised: By Chernoff bound and union bound over the policy set ˜Π , we have with prob.

Original: 1 − δ 1 , for any ˜ π ∈ ˜Π ,
Revised: 1 − δ 1 , for any ˜ π ∈ ˜Π ,

Original: [Equation]
Revised: [Equation]

Revised: By Inq. 51 and Inq. 52, for ∀ π ∈ Π , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: N
Revised: N

Original: [Equation] [Equation]
Revised: N (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: We can set ϵ 

Revised: We can set ϵ

Original: = ϵ 12 H and δ 1 = 1 / 6 .
Revised: = ϵ 12 H and δ 1 = 1 / 6 .

Original: Since N = C 1 log ( N (Π , ϵ/ (12 H ) , d )) /ϵ 2 and π ∗ , ˆ π ∈ Π , we know that with probability at least 5 / 6 ,[Equation] N
Revised: Since N = C 1 log ( N (Π , ϵ/ (12 H ) , d )) /ϵ 2 and π ∗ , ˆ π ∈ Π , we know that with probability at least 5 / 6 ,

Original: [Equation]
Revised: [Equation]

Original: Upper bound on the third term We have the following lemma, which is proved in the following subsections.
Revised: Upper bound on the third term We have the following lemma, which is proved in the following subsections.

Original: Lemma 14.
Revised: Lemma 16.

Original: With probability at least 5 / 6 , Algorithm 2 can return a policy ˆ π satisfying
Revised: With probability at least 5 / 6 , Algorithm 2 can return a policy ˆ π satisfying

Original: [Equation]
Revised: [Equation]

Original: where ˆ π ∗ is the empirical maximizer, i.e. ˆ π ∗ = arg max π ∈ Π 1 N (cid:80) 
Revised: where ˆ π ∗ is the empirical maximizer, i.e. ˆ π ∗ = arg max π ∈ Π 1 N (cid:80) N i =1 V π M

Original: Ni
Revised: i

Original: =1 V π M i , 1 ( s 1 ) .
Revised: [Equation] 1 ( s 1 ) [Equation]

Original: Pluging the results in Lemma 14, Inq. 54 and Inq.
Revised: Pluging the results in Lemma 16, Inq. 57 and Inq.

Original: 55 back into Eq.
Revised: 58 back into Eq.

Original: 43, we know that with probability at least 2 / 3 ,
Revised: 46, we know that with probability at least 2 / 3 ,

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF L EMMA
Revised: D. P ROOF OF L EMMA

Original: We first state the following high-probability events.
Revised: We first state the following high-probability events.

Original: Lemma 15.
Revised: Lemma 17.

Original: With probability at least 1 − 1 2 HK , the following inequality holds for any i ∈ [ N ] , k ∈
Revised: With probability at least 1 − 1 2 HK , the following inequality holds for any i ∈ [ N ] , k ∈

Original: [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A ,(cid:13)(cid:13)(cid:13)
Revised: [ K ] , h ∈ [ H ] , s ∈ S , a ∈ A ,

Original: ˆ P M i ,k,h ( ·| s, a ) − P M i ,h ( ·| s, a ) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Weissman et al.
Revised: According to Weissman et al.

Original: (2003), the L 1 -deviation of the true distribution and the empirical distribution over m distinct events from n samples is bounded by
Revised: (2003), the L 1 -deviation of the true distribution and the empirical distribution over m distinct events from n samples is bounded by

Original: [Equation]
Revised: [Equation]

Original: In our case where m = S , for any fixed i, k, h, s, a , we have
Revised: In our case where m = S , for any fixed i, k, h, s, a , we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ .
Revised: with probability at least 1 − δ .

Original: Taking union bound over all possible i, k, h, s, a , we know that Inq.
Revised: Taking union bound over all possible i, k, h, s, a , we know that Inq.

Original: 58 holds for any i, k, h, s, a with probability at least 1 − NKHSAδ .
Revised: 61 holds for any i, k, h, s, a with probability at least 1 − NKHSAδ .

Original: We reach Inq.
Revised: We reach Inq.

Original: by setting δ = 1 4 NK 2 H 2 SA .
Revised: by setting δ = 1 4 NK 2 H 2 SA .

Original: For the reward estimation, we know that the reward is 1 -subgaussian by definition.
Revised: For the reward estimation, we know that the reward is 1 -subgaussian by definition.

Original: By Hoeffding’s inequality, we have
Revised: By Hoeffding’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ for any fixed i, k, h, s, a .
Revised: with probability at least 1 − δ for any fixed i, k, h, s, a .

Original: Inq.
Revised: Inq.

Original: 57 can be similarly proved by union bound over all possible i, k, h, s, a and setting δ =
Revised: 60 can be similarly proved by union bound over all possible i, k, h, s, a and setting δ =

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Lemma 16.
Revised: Lemma 18.

Original: The following inequality holds with probability at least 1 −
Revised: The following inequality holds with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the above two inequalities, the RHS can be regarded as the summation of Martingale differences.
Revised: For the above two inequalities, the RHS can be regarded as the summation of Martingale differences.

Original: Therefore, the above inequalities hold by applying Azuma’s inequality.
Revised: Therefore, the above inequalities hold by applying Azuma’s inequality.

Original: We use Λ 1 to denote the events defined in the above lemmas.
Revised: We use Λ 1 to denote the events defined in the above lemmas.

Original: Now we prove the optimism of our algorithm under event Λ 1 .
Revised: Now we prove the optimism of our algorithm under event Λ 1 .

Original: Lemma 17.
Revised: Lemma 19.

Original: Under event Λ 1 , we have ˆ V π M i ,k, 1 ( s 1 ) ≥ V π M i , 1 ( s 1 ) for any i ∈ [ N ] , k ∈
Revised: Under event Λ 1 , we have ˆ V π M i ,k, 1 ( s 1 ) ≥ V π M i , 1 ( s 1 ) for any i ∈ [ N ] , k ∈

Original: [ K ] , π ∈ Π .
Revised: [ K ] , π ∈ Π .

Original: Proof.
Revised: Proof.

Original: We prove the lemma by induction.
Revised: We prove the lemma by induction.

Original: Suppose ˆ Q π M ,k,h +1 ( s, a ) ≥ Q π M ,h +1 ( s, a ) .
Revised: Suppose ˆ Q π M ,k,h +1 ( s, a ) ≥ Q π M ,h +1 ( s, a ) .

Original: For h ∈ [ H ] , if ˆ Q π M ,k,h ( s, a ) = 1 , then we trivially have ˆ Q π M ,k,h ( s, a ) ≥ Q π M ,h ( s, a ) .
Revised: For h ∈ [ H ] , if ˆ Q π M ,k,h ( s, a ) = 1 , then we trivially have ˆ Q π M ,k,h ( s, a ) ≥ Q π M ,h ( s, a ) .

Original: If ˆ Q π M ,k,h ( s, a ) < 1 , then ˆ Q π M ,k,h ( s, a ) − Q π M ,h ( s, a ) ≥ ˆ R M
Revised: If ˆ Q π M ,k,h ( s, a ) < 1 , then ˆ Q π M ,k,h ( s, a ) − Q π M ,h ( s, a ) ≥ ˆ R M

Original: i
Revised: i

Original: k,h ( s, a ) + b M i ,k,h ( s, a ) + ˆ P M i ,k,h
Revised: k,h ( s, a ) + b M i ,k,h ( s, a ) + ˆ P M i ,k,h

Original: ˆ V π M i ,k,h +1 ( s, a ) − r M i ,h ( s, a ) − P M i ,h V π M i ,h +1 ( s, a ) ≥ b M 
Revised: ˆ V π M i ,k,h +1 ( s, a ) − r M i ,h ( s, a ) − P M i ,h V π M i ,h +1 ( s, a ) ≥ b M i ,k,h ( s, a ) − (cid:12)(cid:12)(cid:12) ˆ R M i ,k,h ( s, a ) − r M i ,h ( s, a ) (cid:12)(cid:12)(cid:12) − (cid:13)(cid:13)(cid:13) ˆ P M i ,k,h ( ·| s, a ) − P M i ,h ( ·| s, a ) (cid:13)(cid:13)(cid:13)+ P M i

Original: i ,k,h ( s, a ) − (cid:12)(cid:12) (cid:12) ˆ R M i ,k,h ( s, a ) − r M i ,h ( s, a ) (cid:12)(cid:12)
Revised: [Equation] h (cid:16) ˆ V [Equation] M i ,k

Original: [Equation]
Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: [Equation]

Revised: V

Revised: [Equation]

Original:  ≥0 , where the second inequality is due to Lemma 15 and the third inequality is derived from the definition of b M i ,k,h ( s, a ) .
Revised: M i ,h [Equation] 1 (cid:17) ( s, a ) ≥ b M i ,k,h ( s, a ) − (cid:115) 2 log(8 SANHK ) max { 1 , N M

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: [Equation]

Revised: a

Original: The last inequality is from the induction condition that ˆ Q π M ,k,h +1 ( s, a ) ≥
Revised: [Equation] − (cid:115) [Equation] S log(8 SANHK ) max { 1 , N M

Original: Q π M  ,h +1  ( s, a ) .
Revised: [Equation] h ( s, a ) } + P M i ,h (cid:16) ˆ V π M i ,k,h +1 − V π M i ,h +1 (cid:17) ( s, a ) ≥ P M i ,h (cid:16) ˆ V π M i ,k [Equation] ≥ 0 , where the second inequality is due to Lemma 17 and the third inequality is derived from the definition of b M i ,k,h ( s, a ) .

Revised: The last inequality is from the induction condition that ˆ Q π M ,k,h +1 ( s, a ) ≥

Original:  Therefore, for step h , we also have ˆ Q π M ,k,h ( s, a ) ≥ Q π M ,h ( s, a ) .
Revised: Q π M ,h +1 ( s, a ) . Therefore, for step h , we also have ˆ Q π M ,k,h ( s, a ) ≥ Q π M ,h ( s, a ) .

Original: From the induction, we know that ˆ Q π M ,k, 1 ( s, a ) ≥ Q π M , 1 ( s, a ) . 
Revised: From the induction, we know that ˆ Q π M ,k, 1 ( s, a ) ≥ Q π M , 1 ( s, a ) . By definition, we have ˆ V π M i ,k [Equation] ( s

Original: By definition, we have ˆ V π M i ,k, 1 ( s 1 ) ≥ V π M i, 1 ( s 1 ) .
Revised: [Equation] ≥ V [Equation] M i

Revised: [Equation]

Revised: [Equation]

Original: Now we prove Lemma 14.
Revised: Now we prove Lemma 16.

Original: Proof.
Revised: Proof.

Original: (Proof of Lemma 14) By Lemma 17 and the optimality of ˆ π k ,
Revised: (Proof of Lemma 16) By Lemma 19 and the optimality of ˆ π k ,

Original: [Equation]
Revised: [Equation]

Original: We decompose the value difference using Bellman equation

Original: [Equation]
Revised: [Equation]

Original: For each episode k ∈ [ K ] :

Original: [Equation]

Original: Therefore
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: K [Equation] ≤ [Equation]
Revised: k,h )

Original: +
Revised: +

Original: K

Original: [Equation]

Original: k =

Original: N (cid:88) i =

Original: H (cid:88) h

Original: = (cid:16) ˆ R M i ,k,h ( s M i ,k,h , a M i ,k,h ) −

Original: [Equation]

Original: M i ,h (

Original: [Equation]

Original: i ,k,h

Original: [Equation]

Original: M i ,k,h ) (cid:17) +

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: N(cid:88) i = H (cid:88) h = (cid:16)
Revised: N

Original: [Equation]
Revised: [Equation]

Original: i [Equation] +
Revised: i =

Original: K
Revised: H

Original: [Equation]
Revised: [Equation]

Original: k
Revised: h

Original: [Equation]
Revised: [Equation]

Original: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]

Original: k =

Original: N (cid:88) i = [Equation] H [Equation]
Revised: K (cid:88) k

Original: h =[Equation] (cid:16) [Equation]
Revised: =

Original: [Equation]

Original: i

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: h

Original: [Equation]
Revised: [Equation]

Original: 1 ) (cid:17)(cid:17) ≤

Original: K (cid:88) k
Revised: N [Equation]

Original: = N (cid:88) i = 
Revised: i = [Equation]

Original: H
Revised: + [Equation] h +1 )

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: K (cid:88) k

Revised: =

Revised: N (cid:88) i =

Original: h
Revised: H

Original: [Equation]
Revised: [Equation]

Original: h )
Revised: h =

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: N (cid:88) i = H (cid:88) h =
Revised: [Equation]

Original: b M
Revised: N

Original: [Equation]
Revised: [Equation]

Original: ,k
Revised: i = H

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: k
Revised: b

Original: [Equation]
Revised: [Equation]

Original: h , a M
Revised: log

Original: [Equation]
Revised: [Equation]

Original: k,h )
Revised: KH )) .

Original: + O ( (cid:112)

Original: NHK log( KH )) .

Original: The last inequality is due to Lemma 17 under event Λ 1 .
Revised: The last inequality is due to Lemma 19 under event Λ 1 .

Original: By Lemma 15, we have
Revised: By Lemma 17, we have

Original: [Equation]
Revised: K (cid:88) k = N (cid:88) i = H (cid:88) h

Original:  ≤ 
Revised: = (cid:16) ˆ R M i ,k,h ( s M i ,k,h , a M i ,k,h ) − r M i ,h ( s M i ,k,h , a M i ,k,h ) + (cid:16) ˆ P M i ,k,h − P M i ,h (cid:17) ˆ V π k M i ,k,h +1 ( s M i ,k,h , a M i ,k,h ) (cid:17) ≤ K (cid:88) k =

Original: [Equation]
Revised: N (cid:88) i = H (cid:88) h = (cid:16) ˆ R M i ,k,h ( s M i ,k,h , a M i ,k,h ) − r M i ,h ( s M i ,k,h , a M i ,k,h ) (cid:17) + K (cid:88) k

Original: +
Revised: =

Original: [Equation]
Revised: N (cid:88) i = H (cid:88) h = (cid:13)(cid:13)(cid:13) ˆ P M i ,k,h ( ·| s M i ,k,h , a M i ,k,h ) − P M i ,h ( ·| s M i ,k,h , a M i ,k,h ) (cid:13)(cid:13)(cid:13)≤ b M i ,k,h ( s M i ,k,h , a M i ,k,h )

Original: We can upper bound
Revised: We can upper bound

Original: [Equation]
Revised: [Equation]

Original: by the summation of [Equation] h ) .
Revised: by the summation of b M i ,k,h ( s M i ,k,h , a M i ,k,h ) .

Original: By definition, we have
Revised: By definition, we have

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: By Cauchy-Schwarz inequality, we have 
Revised: By Cauchy-Schwarz inequality, we have K

Original: [Equation]
Revised: [Equation]

Original: Since ˆ π is uniformly selected from the policy set { π k } Kk =1 .
Revised: Since ˆ π is uniformly selected from the policy set { π k } Kk =1 .

Original: By Markov’s inequality [Equation] the following inequality holds with probability at least 5 / 6 ,
Revised: By Markov’s inequality, the following inequality holds with probability at least 5 / 6 ,

Original: [Equation]
Revised: [Equation]

Original: With our choice of K = C 2 S 2 AH 2 log( SAH/ϵ ) /ϵ [Equation] we have
Revised: With our choice of K = C 2 S 2 AH 2 log( SAH/ϵ ) /ϵ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: D. H IGH PROBABILITY BOUND
Revised: D. H IGH PROBABILITY BOUND

Original: To obtain a high probability bound with probability at least 1 − δ .
Revised: To obtain a high probability bound with probability at least 1 − δ .

Original: Our idea is to first execute Algorithm 2 independently for O (log(1 /δ ))
Revised: Our idea is to first execute Algorithm 2 independently for O (log(1 /δ ))

Original: times and obtains a policy set with cardinality O (log(1 /δ )) .
Revised: times and obtains a policy set with cardinality O (log(1 /δ )) .

Original: We evaluate the policies in the policy set on the sampled M MDPs, and then return the policy with the maximum empirical value.
Revised: We evaluate the policies in the policy set on the sampled M MDPs, and then return the policy with the maximum empirical value.

Original: The algorithm is described in Algorithm
Revised: The algorithm is described in Algorithm

Original: We have the following theorem for Algorithm 5.

Original: Theorem 18.

Original: With probability at least 1 − δ , Algorithm 5 can output a policy ˆ π satisfying

Original: [Equation]

Original: ∗ [Equation] with O [Equation]
Revised: Algorithm 5 OMERM with High Probability

Original: during training

Original: [Equation]

Original: The number of episodes collected for each task is bounded by O

Original: [Equation]

Original: The proof follows the proof idea of Theorem 3, with only the difference in the bound on the empirical risk.

Original: We first prove the following lemma.

Original: Lemma 19.

Original: With probability at least 1 − δ 2 , Algorithm 5 can return a policy π ξ ∗ satisfying

Original: [Equation]

Original: where ˆ π ∗ is the empirical maximizer, i.e. ˆ π ∗ = arg max π ∈ Π 1 N (cid:80)

Original: Ni

Original: =1 V π M i , 1 ( s 1 ) .

Original: Input : target accuracy ϵ > 0 , high probability parameter δ
Revised: Input : target accuracy ϵ > 0 , high probability parameter δ

Original: N = C 1 log ( N (Π , ϵ/ (12 H ) , d ) /δ )
Revised: N = C 1 log ( N (Π , ϵ/ (12 H ) , d ) /δ )

Original: /ϵ 2 for a constant C 1 >
Revised: /ϵ 2 for a constant C 1 >

Original: N 1 = log(2 /δ )
Revised: N 1 = log(2 /δ )

Original: / log(1 / 6) , N 2 =
Revised: / log(1 / 6) , N 2 =

Original: C 2 log( NN 1 /δ )
Revised: C 2 log( NN 1 /δ )

Original: /ϵ 2 for a constant C 2 >
Revised: /ϵ 2 for a constant C 2 >

Original: Sample N tasks from the distribution D , denoted as {M 1 , M 2 , · · · , M N } 5: for ξ = 1 , 2 , · · · , N 1 do
Revised: Sample N tasks from the distribution D , denoted as {M 1 , M 2 , · · · , M N } 5: for ξ = 1 , 2 , · · · , N 1 do

Original: Execute Algorithm 2 with target accuracy ϵ/ 2 and task set {M 1 , M 2 , · · · , M N } , and obtain a policy π ξ for task index i = 1 , 2 , · · · , N do
Revised: Execute Algorithm 2 with target accuracy ϵ/ 2 and task set {M 1 , M 2 , · · · , M N } , and obtain a policy π ξ for task index i = 1 , 2 , · · · , N do

Original: Execute π ξ on task M i for N 2 times, denoted the average total rewards as V ξ,i
Revised: Execute π ξ on task M i for N 2 times, denoted the average total rewards as V ξ,i

Original: Calculate the average value V ξ = 1 N (cid:80) Ni =1 V ξ,i 10: Output: the policy π ξ ∗ with ξ ∗ = arg max ξ ∈
Revised: Calculate the average value V ξ = 1 N (cid:80) Ni =1 V ξ,i 10: Output: the policy π ξ ∗ with ξ ∗ = arg max ξ ∈

Original: [ N 1 ] V ξ
Revised: [ N 1 ] V ξ

Revised: We have the following theorem for Algorithm 5.

Original: Proof.
Revised: Theorem 20. With probability at least 1 − δ , Algorithm 5 can output a policy ˆ π satisfying E [Equation]

Revised: with O

Revised: [Equation]

Revised: The number of episodes collected for each task is bounded by O (cid:16)

Revised: [Equation]

Revised: The proof follows the proof idea of Theorem 3, with only the difference in the bound on the empirical risk.

Revised: We first prove the following lemma.

Revised: Lemma 21.

Revised: With probability at least 1 − δ 2 , Algorithm 5 can return a policy π ξ ∗ satisfying

Revised: [Equation]

Revised: where ˆ π ∗ is the empirical maximizer, i.e. ˆ π ∗ = arg max π ∈ Π 1 N (cid:80)

Original:  By Lemma 14, for each ξ ∈ [ N 1 ] , the following inequality holds with probability at least 5 / 6 ,
Revised: Ni =1 V π M i , 1 ( s 1 ) . Proof. By Lemma 16, for each ξ ∈ [ N 1 ] , the following inequality holds with probability at least 5 / 6 ,

Original: [Equation]
Revised: [Equation]

Original: In Algorithm 5, we evaluate the policies in the MDPs by executing the policies for N 2 times.
Revised: In Algorithm 5, we evaluate the policies in the MDPs by executing the policies for N 2 times.

Original: By
Revised: By

Original: Hoeffding’s inequality and the union bound over all ξ ∈ [ N 1 ] and i ∈
Revised: Hoeffding’s inequality and the union bound over all ξ ∈ [ N 1 ] and i ∈

Original: [ N ] , with probability at least 1 − δ 2 / 2 , we have
Revised: [ N ] , with probability at least 1 − δ 2 / 2 , we have

Original: [Equation]
Revised: [Equation]

Original: We denote the above event as Λ 2 .
Revised: We denote the above event as Λ 2 .

Original: For each ξ ∈ [ N 1 ] , we define η i as the event that
Revised: For each ξ ∈ [ N 1 ] , we define η i as the event that

Original: N (cid:80) Ni
Revised: N (cid:80) Ni

Original: =1 V ˆ π ∗ M
Revised: =1 V ˆ π ∗ M

Original: i , 1 ( s 1 ) − V ξ ≤ 2 ϵ 9 .
Revised: i , 1 ( s 1 ) − V ξ ≤ 2 ϵ 9 .

Original: By Inq. 60, we have Pr { η i } ≥ 5 / 6 under event Λ 2 .
Revised: By Inq. 63, we have Pr { η i } ≥ 5 / 6 under event Λ 2 .

Original: Note that the event { η i } N 1 i =1 are independent with each other.
Revised: Note that the event { η i } N 1 i =1 are independent with each other.

Original: Therefore, with probability at least 1 − (1 / 6) N 1 = 1 − δ 2 / 2 , there exists ξ 0 , such that the event η ξ 0 happens.
Revised: Therefore, with probability at least 1 − (1 / 6) N 1 = 1 − δ 2 / 2 , there exists ξ 0 , such that the event η ξ 0 happens.

Original: That is, there exists a policy π ξ 0 such that
Revised: That is, there exists a policy π ξ 0 such that

Original: [Equation]
Revised: [Equation]

Original: By definition, ξ ∗ = arg max ξ ∈ [ N 1 ] V ξ .
Revised: By definition, ξ ∗ = arg max ξ ∈ [ N 1 ] V ξ .

Original: Therefore, we have 1 N (cid:80) Ni
Revised: Therefore, we have 1 N (cid:80) Ni

Original: =1 V ˆ π ∗ M
Revised: =1 V ˆ π ∗ M

Original: i , 1 ( s 1 ) − V ξ ∗ ≤ 2 ϵ 9 .
Revised: i , 1 ( s 1 ) − V ξ ∗ ≤ 2 ϵ 9 .

Original: Under the event of Λ 2 , we have the following inequality holds with probability at least 1 − δ 2 / 2 ,
Revised: Under the event of Λ 2 , we have the following inequality holds with probability at least 1 − δ 2 / 2 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: (Proof of Theorem 18)
Revised: (Proof of Theorem 20)

Original: The proof follows the proof idea of Theorem 3.
Revised: The proof follows the proof idea of Theorem 3.

Original: We bound the empirical risk by Lemma 19.
Revised: We bound the empirical risk by Lemma 21.

Original: With our choice of δ 2 = δ/ 3 , we have with probability at least 1 − δ/ 2 ,
Revised: With our choice of δ 2 = δ/ 3 , we have with probability at least 1 − δ/ 2 ,

Original: [Equation]
Revised: [Equation]

Revised: Following the proof of Theorem 3, we can similarly show that the following inequalities hold with probability 1 − δ/ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Following the proof of Theorem 3, we can similarly show thatthe following inequalities hold with probability  1 − δ/ 2 ,
Revised: Combining the results in Inq. 65, Inq.66 and Inq. 64, we know that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: [Table]