Original: P ROVABLE B ENEFITS OF R EPRESENTATIONAL T RANSFER IN R EINFORCEMENT L EARNING
Revised: Provable Benefits of Representational Transfer in Reinforcement Learning

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: Abstract

Original: We study the problem of representational transfer in RL, where an agent first pretrainsoffline in a number of source tasks to discover a shared representation which is subsequently used to learn a good policyonline in a target task
Revised: We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation, which is subsequently used to learn a good policy in a target task.

Original: We propose a new notion of task relatedness between source and target tasks, and develop a novel approach for representational transfer under this assumption
Revised: We propose a new notion of task relatedness between source and target tasks, and develop a novel approach for representational transfer under this assumption.

Original: Concretely , we show that given  generative access toa set of source tasks we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy, with only online access to the target task.
Revised: Concretely, we show that given a generative access to source tasks, we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy, with only online access to the target task.

Original: The sample complexity is close to knowing the ground truth features in the target task, and comparable to prior representation learning results in the source tasks.
Revised: The sample complexity is close to knowing the ground truth features in the target task, and comparable to prior representation learning results in the source tasks.

Original: We complement our positive results with lower bounds without generative access, and validate our findings with empirical evaluation on rich observation MDPs that require deep exploration.
Revised: We complement our positive results with lower bounds without generative access, and validate our findings with empirical evaluation on rich observation MDPs that require deep exploration

Original:  I NTRODUCTION
Revised:  Introduction

Original: Leveraging historical experiences acquired in learning past skills to accelerate the learning of a new skill is a hallmark of intelligent behavior.
Revised: Leveraging historical experiences acquired in learning past skills to accelerate the learning of a new skill is a hallmark of intelligent behavior.

Original: In this paper, we study this question in the context of reinforcement learning (RL).
Revised: In this paper, we study this question in the context of reinforcement learning (RL).

Original: Specifically, we consider a setting where the learner is exposed to multiple tasks and ask the following question:
Revised: Specifically, we consider a setting where the learner is exposed to multiple tasks and ask the following question:

Original: Can we accelerate RL by sharing representations across multiple related tasks?
Revised: Can we accelerate RL by sharing representations across multiple related tasks?

Original: There is rich empirical literature which studies multiple approaches to this question and various paradigms for instantiating it.
Revised: There is rich empirical literature which studies multiple approaches to this question and various paradigms for instantiating it.

Original: For instance, in a multi-task learning scenario, the learner has simultaneous access to different tasks and tries to improve the sample complexity by sharing data across them (Caruana, 1997).
Revised: For instance, in a multi-task learning scenario, the learner has simultaneous access to different tasks and tries to improve the sample complexity by sharing data across them (Caruana, 1997).

Original: Other works study a transfer learning setting, where the learner has access to multiple source tasks during a pre-training phase, followed byonline learning in a target task (Pan and Yang, 2009).
Revised: Other works study a transfer learning setting, where the learner has access to multiple source tasks during a pre-training phase, followed by a target task (Pan and Yang, 2009).

Original: The goal is to learn features and/or a policy which can be quickly adapted to succeed in the target task.
Revised: The goal is to learn features and/or a policy which can be quickly adapted to succeed in the target task.

Original: More generally, the paradigms of meta-learning (Finn et al., 2017), lifelong learning (Parisi et al., 2019) and curriculum learning (Bengio et al., 2009) also consider related questions.
Revised: More generally, the paradigms of meta-learning (Finn et al., 2017), lifelong learning (Parisi et al., 2019) and curriculum learning (Bengio et al., 2009) also consider related questions.

Original: On the theoretical side, questions of representation learning have received an increased recent emphasis owing to their practical significance, both in supervised learning and RL settings.
Revised: On the theoretical side, questions of representation learning have received an increased recent emphasis owing to their practical significance, both in supervised learning and RL settings.

Original: In RL, a limited form of transfer learning across multiple downstream reward functions is enabled by several recent reward-free representation learning approaches (Jin et al., 2020a; Zhang et al., 2020;
Revised: In RL, a limited form of transfer learning across multiple downstream reward functions is enabled by several recent reward-free representation learning approaches (Jin et al., 2020a; Zhang et al., 2020;

Original: Wang et al., 2020; Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020; Modi et al., 2021).
Revised: Wang et al., 2020; Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020; Modi et al., 2021).

Original: Inspired by recent treatments of representation transfer in supervised (Maurer et al., 2016; Du et al., 2020) and imitation learning (Arora et al., 2020), some works also study more general task collections in bandits (Hu et al., 2021;
Revised: Inspired by recent treatments of representation transfer in supervised (Maurer et al., 2016; Du et al., 2020) and imitation learning (Arora et al., 2020), some works also study more general task collections in bandits (Hu et al., 2021;

Original: Yang et al., 2020, 2022) and RL (Hu et al., 2021;
Revised: Yang et al., 2020, 2022) and RL (Hu et al., 2021;

Original: Lu et al., 2021).
Revised: Lu et al., 2021).

Original: Almost all these works study settings where the representation is frozen after pre-training in the source tasks, and a linear policy or optimal value function approximation is trained in the target task using these learned features.
Revised: Almost all these works study settings where the representation is frozen after pre-training in the source tasks, and a linear policy or optimal value function approximation is trained in the target task using these learned features.

Original: This setting, which we call representational transfer , is the main focus of our paper.
Revised: This setting, which we call representational transfer , is the main focus of our paper.

Original: A crucial question in formalizing representational transfer settings is the notion of similarity between source and target tasks.
Revised: A crucial question in formalizing representational transfer settings is the notion of similarity between source and target tasks.

Original: Prior works in supervised learning make the stringent assumption that the covariates x follow the same underlying distribution in all the tasks, and only the conditional P ( y | x ) can vary across tasks (Du et al., 2020).
Revised: Prior works in supervised learning make the stringent assumption that the covariates x follow the same underlying distribution in all the tasks, and only the conditional P ( y | x ) can vary across tasks (Du et al., 2020).

Original: This assumption does not nicely generalize to RL settings, where state distributions are typically policy dependent, and prior attempts to extend this assumption to RL (Lu et al., 2021) result in strong assumptions on the learning setup.
Revised: This assumption does not nicely generalize to RL settings, where state distributions are typically policy dependent, and prior attempts to extend this assumption to RL (Lu et al., 2021) result in strong assumptions on the learning setup.

Original: Other works (Hu et al., 2021; Yang et al., 2020, 2022) focus on linear representations only, which limits the expressivity of the feature maps, and does not adequately represent the empirical literature in the field.
Revised: Other works (Hu et al., 2021; Yang et al., 2020, 2022) focus on linear representations only, which limits the expressivity of the feature maps, and does not adequately represent the empirical literature in the field.

Original: Our contributions.
Revised: Our contributions.

Original: In this context, our work makes the following contributions
Revised: In this context, our work makes the following contributions

Original: : • We propose a new linear span assumption of task relatedness for representational transfer, where the target task dynamics can be expressed as a (state-dependent) linear span of source task dynamics, in addition to the dynamics being low-rank under a shared representation.
Revised: : • We propose a new linear span assumption of task relatedness for representational transfer, where the target task dynamics can be expressed as a (state-dependent) linear span of source task dynamics, in addition to the dynamics being low-rank under a shared representation.

Original: We give examples captured by this assumption, and it generalizes all prior settings for representational transfer in RL.
Revised: We give examples captured by this assumption, and it generalizes all prior settings for representational transfer in RL.

Original: We do not make any linearity assumptions on our feature maps.
Revised: We do not make any linearity assumptions on our feature maps.

Original: • When we have(offline) generative access to source tasks, we provide a novel algorithm R EPT RANSFER that successfully pretrains a representation for downstream online learning in any target task (i.e., no generative access in target task) satisfying the linear span assumption, when the source tasks satisfy a common latent reachability assumption.
Revised: • When we have generative access to source tasks, we provide a novel algorithm R EP T RANSFER that successfully pretrains a representation for downstream online learning in any target task (i e., no generative access in target task) satisfying the linear span assumption, when the source tasks satisfy a common latent reachability assumption.

Original: The regret bound of learning in the target task is close to that of learning in a linear MDP equipped with the ground truth features, the strongest possible yardstick in our setup.
Revised: The regret bound of learning in the target task is close to that of learning in a linear MDP equipped with the ground truth features, the strongest possible yardstick in our setup.

Revised: The additional terms in our regret largely arise out of the distributional mismatch between source and target tasks, which is expected.

Original: We complement the theory with an empirical validation of R EP T RANSFER on the challenging rich observation combination lock benchmarks (Misra et al., 2020), confirming our theoretical findings.
Revised: We complement the theory with an empirical validation of R EP T RANSFER on the challenging rich observation combination lock benchmarks (Misra et al., 2020), confirming our theoretical findings.

Original: • Without generative access to source tasks, we show the statistical hardness of representational transfer under the linear span assumption, and confirm this hardness in our empirical evaluation.
Revised: • Without generative access to source tasks, we show the statistical hardness of representational transfer under the linear span assumption, and confirm this hardness in our empirical evaluation.

Revised: We show that an additional assumption that every observed state is reachable in every source task is sufficient for allowing fully online learning in source tasks.

Original: The new task relatedness assumption, reward-free learning result for low-rank MDPs and our analysis of L SVI -U CB under average case misspecification may be of independent interest.  
Revised: The new task relatedness assumption, reward-free learning result for low-rank MDPs and our analysis of L SVI -U CB under average case misspecification may be of independent interest.  Related Work

Revised: Multi-task and Transfer Learning in Supervised Learning.

Revised: The theoretical benefit of representation learning are well studied under conditions such as the i.i.d.

Revised: task assumption (Maurer et al., 2016) and the diversity assumption (Du et al., 2020; Tripuraneni et al., 2020).

Original: Due to space constraint, we defer a detailed discussion of related work to the Section B and empirical evaluation to 
Revised: Many works below successfully adopt the frameworks and assumptions to sequential decision making problems. Multi-task and Transfer Learning in Bandit and small-size MDPs. A sequence of recent works study multi-task linear bandit problems with linear representation functions ( ϕ ( s ) =

Revised: A s )

Revised: (Hu et al., 2021; Yang et al., 2020, 2022).

Revised: The techniques developed in these works crucially rely on the linear representation function structure and can not be applied to nonlinear function classes.

Revised: Lazaric et al.

Revised: (2013) study spectral techniques for online sequential transfer learning.

Revised: Brunskill and Li (2013) consider multi-task RL under a fixed distribution over finitely many diverse MDPs, while Brunskill and Li (2014) considers transfer learning in semi-MDPs by learning options (temporally extended actions).

Revised: All these works consider small size tabular models while we focus on large-scale MDPs.

Revised: Multi-task and Transfer Learning in RL via representation learning.

Revised: Beyond tabular MDPs,

Revised: Arora et al.

Revised: (2020) and D’Eramo et al.

Revised: (2019) show benefits of representation learning in imitation learning and planning, but do not address exploration.

Original: Section J.
Revised: Lu et al. (2021) study transfer learning in low-rank MDPs with general nonlinear representations, but make a generative model assumption on both the source tasks and the target task, along with other distributional and structural assumptions. We do not require generative access to the target task and make much weaker structural assumptions on the source-target relatedness. Recently and independently, Cheng et al.

Original: P RELIMINARIES
Revised: (2022) also studied transfer learning in low-rank MDPs in the online learning setting, identical to the setting we study in Section 5. However, their analysis relies on an additional assumption that bounds the point-wise TV error with the population TV error, which we show is in fact not necessary. We refer readers to Appendix B for a more detailed comparison with the above two closely related works. Efficient Representation Learning in RL.

Revised: Even in the single task setting, efficient representation learning is an active area of study that has seen many recent advances with exploration (Agarwal et al., 2020;

Revised: Modi et al., 2021; Uehara et al., 2021; Zhang et al., 2022) or without (Ren et al., 2021).

Revised: Other papers study feature selection (e.g. Farahmand and Szepesvári, 2011; Jiang et al., 2015;

Revised: Pacchiano et al., 2020; Cutkosky et al., 2021; Lee et al., 2021; Zhang et al., 2021), or focus on learning sparse models (Hao et al., 2021a,b).

Original: Notations: We denote total variation distance of P 1 and P 2 by ∥ P 1 − P 2 ∥ TV .
Revised: Notations: We denote total variation distance of P 1 and P 2 by ∥ P 1 − P 2 ∥ TV .

Original: Given a vector a , we define ∥ a ∥ B = √ a ⊤ Ba .
Revised: Given a vector a , we define ∥ a ∥ B = √ a ⊤ Ba .

Original: c 0 , c 1 , · · · are universal constants.
Revised: c 0 , c 1 , · · · are universal constants.

Original: We use a ≲ b to mean a ≤ Cb for some universal constant C .
Revised: We use a ≲ b to mean a ≤ Cb for some universal constant C .

Original: Also, [ K ] = { 1 , . . .
Revised: Also, [ K ] = { 1 , . . .

Original: , K } and λ min ( A ) is the smallest eigenvalue of matrix A .
Revised: , K } and λ min ( A ) is the smallest eigenvalue of matrix A .

Original: Please see Table 1 for a full list of notations and Table 2 for a list of algorithms.
Revised: Please see Table 1 for a full list of notations and Table 2 for a list of algorithms.

Original: Low-rank Markov Decision Processes: We consider a finite-horizon episodic Markov Decision
Revised: Low-rank Markov Decision Processes: We consider a finite-horizon episodic Markov Decision

Original: Process M = ⟨ H, S , A ,P ⋆ = { P ⋆h } 0: H − 1 ,r = { r h } 0: H − 1 , d 0 ⟩ , specified by the episode length 
Revised: Process M = ⟨ H, S , A , { P ⋆h } 0: H − 1 , { r h } 0: H − 1 , d 0 ⟩ , specified by the episode length H , state space

Original: H , state space S , discrete action space A of size A , transition models P ⋆h : S ×
Revised: S , discrete action space A of size A , transition models P ⋆h : S ×

Original: A → ∆( S ) , reward functions r h :S ×
Revised: A → ∆( S ) , reward functions r h :

Original: A → [0 , 1] , and an initial distribution d 0 ∈ ∆( S ) .
Revised: S ×A → [0 , 1] , and an initial distribution d 0 ∈ ∆( S ) .

Revised: Occasionally, we use P ⋆ to denote { P ⋆h } 0: H −and r to denote { r h } 0: H − 1 .

Original: Starting from an initial state s 0 ∼ d 0 , an agent chooses actions a h at step h upon observing the state s h , receives a reward r h ( s h , a h ) , and transitions s h +1 ∼ P ⋆h
Revised: Starting from an initial state s 0 ∼ d 0 , an agent chooses actions a h at step h upon observing the state s h , receives a reward r h ( s h , a h ) , and transitions s h +1 ∼ P ⋆h

Original: ( ·| s h , a h ) .
Revised: ( ·| s h , a h ) .

Original: We assume that r and d 0 are known.
Revised: We assume that r and d 0 are known.

Original: Given a policy π : S → ∆( A ) and transition kernel P , we use the notation E π,P [ · ] to denote expectation under the distribution of trajectories when π is executed in P .
Revised: Given a policy π : S → ∆( A ) and transition kernel P , we use the notation E π,P [ · ] to denote expectation under the distribution of trajectories when π is executed in P .

Original: The value function V πP,r ; h ( s ) =
Revised: The value function V πP,r ; h ( s ) =

Original: E π,P (cid:104) (cid:80) H − 1 τ = h r τ ( s τ , a τ ) | s h
Revised: E π,P (cid:104)(cid:80) H − 1 τ = h r τ ( s τ , a τ ) | s h

Original: = s (cid:105) gives the expected reward-to-go of π under the MDP with transitions P , rewards r , starting at state s in step h .
Revised: = s (cid:105) gives the expected reward-to-go of π under the MDP with transitions P , rewards r , starting at state s in step h .

Original: Similarly, we define the Q function Q πP,r ; h ( s, a ) :
Revised: Similarly, we define the Q function Q π P,r ; h ( s, a ) :

Original: = r h ( s, a ) + E s ′ ∼ P h ( ·| s,a ) V πP,r ; h +1 ( s ′ ) .
Revised: = r h ( s, a ) + E s ′ ∼ P h ( ·| s,a ) V π P,r ; h +1 ( s ′ ) .

Original: The expected total reward of a policy π under transitions P and rewards r is denoted as V πP,r :=
Revised: The expected total reward of a policy π under transitions P and rewards r is denoted as V πP,r :=

Original: E s 0 ∼ d 0 V πP,r ;0 ( s 0 ) .
Revised: E s 0 ∼ d 0 V πP,r ;0 ( s 0 ) .

Original: We define the state-action occupancy distribution
Revised: We define the state-action occupancy distribution

Original: d
Revised: d

Original: πP ; h ( s, a ) as the probability of π visiting ( s, a ) at time step h under π and P .
Revised: πP ; h ( s, a ) as the probability of π visiting ( s, a ) at time step h under π and P .

Original: d πP ; h ( s ) is the marginal state visitation, which is equal to (cid:80) a ∈A d πP ; h ( s, a ) .
Revised: d πP ; h ( s ) is the marginal state visitation, which is equal to (cid:80) a ∈A d πP ; h ( s, a ) .

Original: We study low-rank MDPs defined as follows (Jiang et al., 2017; Agarwal et al., 2020).
Revised: We study low-rank MDPs defined as follows (Jiang et al., 2017; Agarwal et al., 2020).

Original: The conditions on the upper bounds of the norm of ϕ ⋆ , µ ⋆ are just for normalization.
Revised: The conditions on the upper bounds of the norm of ϕ ⋆ , µ ⋆ are just for normalization.

Original: Definition 2.1 (Low-rank MDP)
Revised: Definition 3.1 (Low-rank MDP)

Original: A transition model P ⋆h : S ×
Revised: A transition model P ⋆h : S ×

Original: A → ∆( S ) admits a low rank decomposition with rank d ∈ N if there exist two unknown embedding functions ϕ
Revised: A → ∆( S ) admits a low rank decomposition with rank d ∈ N if there exist two unknown embedding functions ϕ

Original: ⋆ h : S × A (cid:55)→ R d , µ ⋆h : S (cid:55)→ R d such that ∀ s, s ′ ∈ S , a ∈ A : P ⋆h ( s ′ | s, a )
Revised: ⋆h : S × A (cid:55)→ R d , µ ⋆h : S (cid:55)→ R d such that ∀ s, s ′ ∈ S , a ∈ A : P ⋆h ( s ′ | s, a )

Original: = µ ⋆h ( s ′ )
Revised: = µ ⋆h ( s ′ )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ⋆h ( s, a ) , where ∥ ϕ ∗ h ( s, a ) ∥ 2 ≤for all ( s, a ) and for any function g : S → [0 , 1] , (cid:13)(cid:13)(cid:82) g ( s )d µ ⋆h ( s ) (cid:13)(cid:13) 2 ≤ √ d .
Revised: ⋆h ( s, a ) , where ∥ ϕ ∗ h ( s, a ) ∥ 2 ≤for all ( s, a ) and for any function g : S → [0 , 1] , (cid:13)(cid:13)(cid:82) g ( s )d µ ⋆h ( s ) (cid:13)(cid:13) 2 ≤ √ d .

Original: An MDP is a low rank
Revised: An MDP is a low rank

Original: MDP if P ⋆h admits such a low rank decomposition for all h = 0 , 1 , . . .
Revised: MDP if P ⋆h admits such a low rank decomposition for all h = 0 , 1 , . . .

Original: , H − 1 .
Revised: , H − 1 .

Original: Low-rank MDPs capture the latent variable model (Agarwal et al., 2020)
Revised: Low-rank MDPs capture the latent variable model (Agarwal et al., 2020)

Original: where ϕ
Revised: where ϕ

Original: ⋆ ( s, a ) is a distribution over a discrete latent state space Z , and the block-MDP model (Du et al., 2019)
Revised: ⋆ ( s, a ) is a distribution over a discrete latent state space Z , and the block-MDP model (Du et al., 2019)

Original: where ϕ
Revised: where ϕ

Original: ⋆ ( s, a ) is a one-hot encoding vector.
Revised: ⋆ ( s, a ) is a one-hot encoding vector.

Original: Note that the linear MDP model (Yang and Wang, 2020; Jin et al., 2020b) assumes a known ϕ ⋆ , which significantly simplifies the algorithm design.
Revised: Note that the linear MDP model (Yang and Wang, 2020; Jin et al., 2020b) assumes a known ϕ ⋆ , which significantly simplifies the algorithm design.

Original: Transfer Learning:
Revised: Transfer Learning:

Original: In contrast to the classic single-task learning setting, in this paper, we explore the setting of transfer learning , where learning consists of two phases: (1) the pre-training phase where the agent interacts with K − 1 source tasks with dynamics P ⋆k , and (2) the deployment phase where the agent is deployed into the K -th target task and no longer has access to the source tasks.
Revised: In contrast to the classic single-task learning setting, in this paper, we explore the setting of transfer learning , where learning consists of two phases: (1) the pre-training phase where the agent interacts with K − 1 source tasks with dynamics P ⋆k , and (2) the deployment phase where the agent is deployed into the K -th target task and no longer has access to the source tasks.

Original: The performance is measured mainly by the regret incurred in the target task upon deployment, while we also desire small sample complexity in the source tasks.
Revised: The performance is measured mainly by the regret incurred in the target task upon deployment, while we also desire small sample complexity in the source tasks.

Original: We denote d πk ; h :=
Revised: We denote d πk ; h :=

Original: d πP ⋆k ; h .
Revised: d πP ⋆k ; h .

Original: In order for the pre-training phase to help with learning in the target task, we must make assumptions on the connections between tasks.
Revised: In order for the pre-training phase to help with learning in the target task, we must make assumptions on the connections between tasks.

Original: In this work, we make the following fundamental structural assumption on all the tasks at hand, namely that they share the same underlying representation ϕ ⋆ .
Revised: In this work, we make the following fundamental structural assumption on all the tasks at hand, namely that they share the same underlying representation ϕ ⋆ .

Original: Assumption 2.1 (Common representation)
Revised: Assumption 3.1 (Common representation)

Original: We assume that all tasks P ⋆k are low-rank MDPs with a shared representation ϕ
Revised: We assume that all tasks P ⋆k are low-rank MDPs with a shared representation ϕ

Original: ⋆h ( s, a ) but distinct µ ⋆k ; h ( s ′ ) , that is P
Revised: ⋆h ( s, a ) but distinct µ ⋆k ; h ( s ′ ) , that is P

Original: ⋆k ; h ( s ′ | s, a ) =
Revised: ⋆k ; h ( s ′ | s, a ) =

Original: ϕ ⋆h ( s, a ) ⊤ µ ⋆k ; h ( s ′ ) .
Revised: ϕ ⋆h ( s, a ) ⊤ µ ⋆k ; h ( s ′ ) .

Original: Assumption 2.2 (Realizability) For any source
Revised: Assumption 3.2 (Realizability) For any source

Original: task k ∈
Revised: task k ∈

Original: [ K − 1] and any h ∈ [ H ] , we assume that the agent has access to realizable function classes Φ h and Υ k ; h , such that ϕ ⋆h ∈
Revised: [ K − 1] and any h ∈ [ H ] , we assume that the agent has access to realizable function classes Φ h and Υ k ; h , such that ϕ ⋆h ∈

Original: Φ h and µ ⋆k ; h ∈ Υ k ; h .
Revised: Φ h and µ ⋆k ; h ∈ Υ k ; h .

Original: For normalization, we assume that for all k, h , all ϕ ∈ Φ h satisfy ∥
Revised: For normalization, we assume that for all k, h , all ϕ ∈ Φ h satisfy ∥

Original: ϕ
Revised: ϕ

Original: ( s, a ) ∥ 2 ≤ 1 , and for all µ ∈ Υ k ; h and any function g : S →
Revised: ( s, a ) ∥ 2 ≤ 1 , and for all µ ∈ Υ k ; h and any function g : S →

Original: [0 , 1] , (cid:13) (cid:13)(cid:82) g ( s )d µ ⋆h ( s ) (cid:13)(cid:13) 2 ≤ √ d .
Revised: [0 , 1] , (cid:13)(cid:13)(cid:82) g ( s )d µ ⋆h ( s ) (cid:13)(cid:13) 2 ≤ √ d .

Original: Assumption 2.2 is standard realizability condition in function class which is made in almost all prior works on RL with linear and nonlinear function approximation (e.g., Jiang et al.
Revised: Assumption 3.2 is standard realizability condition in function class which is made in almost all prior works on RL with linear and nonlinear function approximation (e.g

Original: (2017); Sun et al. (2019); Jin et al.
Revised: Jiang

Original: (2020b);
Revised: [Table]

Original: Agarwal et al. Du et al. ; Uehara et al.
Revised: [Table] [Table]

Original: Since all the tasks share a common representation, we expect the agent to learn a good representation during its initial interactions with the source tasks, and then learn a good policy for the target task using this representation in the deployment phase.
Revised: Since all the tasks share a common representation, we expect the agent to learn a good representation during its initial interactions with the source tasks, and then learn a good policy for the target task using this representation in the deployment phase.

Original: The availability of multiple source tasks is beneficial as the agent might not be able to learn a good representation for all parts of the observation space of the target task from one source only.
Revised: The availability of multiple source tasks is beneficial as the agent might not be able to learn a good representation for all parts of the observation space of the target task from one source only.

Original: In order to realize this intuition though, we need two additional assumptions which we describe next.
Revised: In order to realize this intuition though, we need two additional assumptions which we describe next.

Revised: Assumptions for representational transfer.

Revised: In addition to standard assumptions in the source tasks, we make the following structural and relatedness assumptions on the source and target tasks.

Original: Assumption 2.3 (Feature reachability in the source tasks)
Revised: Assumption 3.3 (Feature reachability in the source tasks)

Original: We assume that ψ :=
Revised: We assume that ψ :=

Original: min k 
Revised: min k ∈ [ K − 1] ,h [Equation] [ H ] max [Equation]

Revised: min (cid:0)

Original: [Equation]
Revised: [Equation]

Revised: ,P ⋆k

Original: [Equation]
Revised: [Equation]

Revised: ϕ ⋆h ( s h , a h

Original: [Equation]
Revised: [Equation]

Revised: ⋆h ( s

Original:  is strictly positive.
Revised: [Equation] , a h ) [Equation] (cid:1) is strictly positive.

Original: Assumption 2.3 intuitively requires that no subspace in R d is unreachable in the source tasks, as otherwise it’s impossible to guarantee the quality of the learned representation in that subspace which may contain high rewards in the target task.
Revised: Assumption 3.3 intuitively requires that no subspace in R d is unreachable in the source tasks, as otherwise it’s impossible to guarantee the quality of the learned representation in that subspace which may contain high rewards in the target task.

Original: Note that no reachability is required in the target task.
Revised: Note that no reachability is required in the target task.

Original: The next assumption quantifies the relatedness of the target task and the source tasks.
Revised: The next assumption quantifies the relatedness of the target task and the source tasks.

Original: Assumption 2.4 (Relatedness: Point-wise Linear span)
Revised: Assumption 3.4 (Relatedness: Point-wise Linear span)

Original: For any h ∈ [ H ] and s ′ ∈ S
Revised: For any h ∈ [ H ] and s ′ ∈ S

Original: h
Revised: h

Original: +1 , there is a vector
Revised: +1 , there is a vector

Original: α h
Revised: α h

Original: ( s ′ ) ∈
Revised: ( s ′ ) ∈

Original: R K − 1 such that µ ⋆K ; h ( s ′ )
Revised: R K − 1 such that µ ⋆K ; h ( s ′ )

Original: = (cid:80) K − 1 k =1 α k ; h ( s ′ )
Revised: = (cid:80) K − 1 k =1 α k ; h ( s ′ )

Original: µ ⋆k ; h ( s ′ ) , with α max = max h ; k,s ′ ∈S | α k ; h ( s ′ ) | and ¯ α = max h (cid:80) K − 1 k =1 max s ′ ∈S | α k ; h ( s ′ ) | .
Revised: µ ⋆k ; h ( s ′ ) , with α max = max h ; k,s ′ ∈S | α k ; h ( s ′ ) | and ¯ α = max h (cid:80) K − 1 k =1 max s ′ ∈S | α k ; h ( s ′ ) | .

Original: Assumption 2.4 ensures that if s ′ is reachable from an ( s, a ) pair in the target task, then it must be reachable from the same ( s, a ) pair in at least one of the source tasks.
Revised: Assumption 3.4 ensures that if s ′ is reachable from an ( s, a ) pair in the target task, then it must be reachable from the same ( s, a ) pair in at least one of the source tasks.

Original: This intuitively is also necessary for transfer learning, as s ′ could be a high rewarding state in the target.
Revised: This intuitively is also necessary for transfer learning, as s ′ could be a high rewarding state in the target.

Original: A special case is the convex combination, i.e., for any s ′ ∈ S , h , (cid:80) k
Revised: A special case is the convex combination, i.e., for any s ′ ∈ S , h , (cid:80) k

Original: α k ; h ( s ′ ) =
Revised: α k ; h ( s ′ ) =

Original: p k with p k ≥ 0 , (cid:80) p p k = 1 , which implies α max ≤ 1 and ¯ α = 1 .
Revised: p k with p k ≥ 0 , (cid:80) p p k = 1 , which implies α max ≤ 1 and ¯ α = 1 .

Original: On the other hand, if the target task largely focuses on observations quite rare under the source tasks, then ¯ α can grow large, and this is unavoidable in a transfer learning setting.
Revised: On the other hand, if the target task largely focuses on observations quite rare under the source tasks, then ¯ α can grow large, and this is unavoidable in a transfer learning setting.

Original: We remark that unlike prior work (Du et al., 2020; Tripuraneni et al., 2020; Lu et al., 2021), we do not make any assumption on the data generating distribution in either the source or the target tasks.
Revised: We remark that unlike prior work (Du et al., 2020; Tripuraneni et al., 2020; Lu et al., 2021), we do not make any assumption on the data generating distribution in either the source or the target tasks.

Original: Instead, our approach is end-to-end, i.e., we collect our own data from scratch for representation learning by doing strategic exploration in the source tasks (see Section C for a detailed discussion).
Revised: Instead, our approach is end-to-end, i.e., we collect our own data from scratch for representation learning by doing strategic exploration in the source tasks (see Section B for a detailed discussion).

Original: In fact, in Theorem D.1 we show that the above assumptions are too weak to guarantee successful transfer in the supervised learning setting, thus establishing an interesting separation between supervised and reinforcement learning.
Revised: In fact, in Theorem C.1 we show that the above assumptions do not permit successful transfer in the supervised learning setting, thus establishing an interesting separation between supervised and reinforcement learning.

Original: [Equation]
Revised: We conclude this section with a couple of examples where our assumptions are satisfied. Example 3.1 (Mixture of source tasks) . Perhaps the simplest example of our assumptions is where P ⋆K ( s ′ | s, a ) = (cid:80) K − 1 k

Original: [Equation]{+=
 α k P+}
Revised: {+=
 α k P+}

Original: [Equation]
Revised: ⋆k

Original: [Equation]{+( s ′ | s, a ) with the coefficients independent of s ′ , α k ≥ 
 and (cid:80) K − 1 k =1 α k = 1 . Such mixtures of base models have been considered in several prior works (Modi et al., 2020; Ayoub et al., 2020). While prior works study the case of arbitrary, but known base models, we instead allow structured, unknown base models with a shared representation.+}
Revised: {+( s ′ | s, a ) with the coefficients independent of s ′ , α k ≥ 
 and (cid:80) K − 1 k =1 α k = 1 . Such mixtures of base models have been considered in several prior works (Modi et al., 2020; Ayoub et al., 2020). While prior works study the case of arbitrary, but known base models, we instead allow structured, unknown base models with a shared representation.+}

Original: [Equation]
Revised: Here ¯ α = 1 .

Original: [Equation]
Revised: Example 3.2 (Block MDPs with shared latent dynamics) . In this example, each MDP P ⋆k is a Block MDP (Du et al., 2019) with a shared latent space Z and a shared decoder ψ ⋆ : S → Z .

Original: [Equation]
Revised: In a block

Original: [Equation]
Revised: MDP, given state action pair ( s, a ) , the decoder ψ ⋆ maps s to a latent state z , the next latent state is sampled from the latent transition z ′ ∼ P ( ·| z, a ) , and the next state is generated from an emission distribution s ′ ∼ o ( ·| z ′ ) . Recall that o

Original: [Equation] [Equation] [Equation]
Revised: ( s ′ | z ′ ) > 0 at only one

Original: [Equation]
Revised: z ′ ∈ Z for any s ′ ∈ S for a block MDP. We assume that the latent transition model P ( z ′ | z, a ) is shared across all the tasks, but the emission process differs across the MDPs.

Original: [Equation]
Revised: For instance, in a typical navigation example used to motivate Block MDPs, the latent dynamics might correspond to navigating in a shared 2-D map, while emission distributions capture different wall colors or lighting conditions across multiple rooms. Then Assumption 3.3 requires that the agent can visit the entire 2-D map, while Assumption 3. requires that the color/lighting conditions of the target task resemble that of at least one source task. The coefficients α for any s ′ are non-zero on the source tasks which can generate that observation.

Original:  T RANSFER L EARNING WITH G ENERATIVE A CCESS TO S OURCE T ASKS
Revised:  Transfer Learning with Generative Access to Source Tasks

Original: In this paper we study the setting where we assume simulator access to the source environments, before going online in the target environment .
Revised: We first study a setting where we assume generative model access to the source tasks, while having only online access to the target task .

Original: Assumption 3.1 (Generative access to the source tasks)
Revised: Assumption 4.1 (Generative access to the source tasks)

Original: We assume that we have access to generative models for the K − 1 source tasks.
Revised: We assume that we have access to generative models for the K − 1 source tasks.

Original: Specifically, for any P ⋆k with k ∈ [ K − 1] , we can query any ( s h , a h ) pair and the generative model will return a next state sample
Revised: Specifically, for any P ⋆k with k ∈ [ K − 1] , we can query any ( s h , a h ) pair, and the generative model will return a next state sample

Original: s h +1 ∼ P
Revised: s h +1 ∼ P

Original: ⋆k ; h ( s h , a h ) .
Revised: ⋆k ; h ( s h , a h ) .

Revised: D EPLOYMENT P HASE

Original: Having generative model access is not unrealistic, especially in applications where a high-quality simulation environment is available.
Revised: Having generative model access is not unrealistic, especially in applications where a high-quality simulation environment is available.

Original: Generative access also does not trivialize the challenge of efficient exploration in the source tasks, since there are a potentially infinite number of states and the ground truth representation ϕ
Revised: Generative access also does not trivialize the challenge of efficient exploration in the source tasks, since there are a potentially infinite number of states and the ground truth representation ϕ

Original: ⋆ is unknown.
Revised: ⋆ is unknown.

Original: Prior works using generative access typically require either a known representation ϕ
Revised: Prior works using generative access typically require either a known representation ϕ

Original: ⋆ (so that one can perform D-optimal design to construct an exploratory state-action distribution (Agarwal et al., 2019)), or directly assume access to a diverse state-action distribution which provides coverage and from which one can sample.
Revised: ⋆ (so that one can perform D-optimal design to construct an exploratory state-action distribution (Agarwal et al., 2019)), or directly assume access to a diverse state-action distribution which provides coverage and from which one can sample.

Original: Neither ϕ ⋆ nor such a diverse sampling distribution is given in our case.
Revised: Neither ϕ ⋆ nor such a diverse sampling distribution is given in our case.

Revised: We also note that in Section 5 we will show that without any additional assumptions, generative access in source tasks is necessary.

Original: Algorithm overview.
Revised: Algorithm overview.

Original: Given the above setup, now we present our algorithm R EP T RANSFER , detailed in Algorithm 1.
Revised: Given the above setup, now we present our algorithm R EP T RANSFER , detailed in Algorithm 1.

Original: R EP T RANSFER takes a representation learning approach to the transfer learning problem and operates in two phases.
Revised: R EP T RANSFER takes a representation learning approach to the transfer learning problem and operates in two phases.

Original: During the pre-training phase, R EP T RANSFER performs rewardfree exploration in each of the source tasks to learn task-specific policies π k , which satisfies for all h = 0 , 1 , ..., H − 1 , that E π k ,P ⋆k (cid:2) ϕ
Revised: During the pre-training phase, R EP T RANSFER performs rewardfree exploration in each of the source tasks to learn task-specific policies π k , which satisfies for all h = 0 , 1 , ..., H − 1 , that E π k ,P ⋆k (cid:2) ϕ

Original: ⋆h ( s h , a h ) ϕ
Revised: ⋆h ( s h , a h ) ϕ

Original: ⋆ ⊤ h ( s h , a h ) ⊤ (cid:3) ⪰ λ min
Revised: ⋆ ⊤ h ( s h , a h ) ⊤ (cid:3) ⪰ λ min

Original: I for some λ min > 0 .
Revised: I for some λ min > 0 .

Original: We call such policies λ min - exploratory and will use this definition in Lemma 3.1 and Theorem 3.1.
Revised: We call such policies λ min - exploratory and will use this definition in Lemma 4.1 and Theorem 4.1.

Original: In
Revised: In

Original: Section C.1 we present one particular algorithm that finds such π k ’s with a specific λ min , but our main analysis below is modular to the choice of the reward-free exploration algorithm.
Revised: Section 4.1 we present one particular algorithm that finds such π k ’s with a specific λ min , but our main analysis below is modular to the choice of the reward-free exploration algorithm.

Original: Given the exploratory policies { π k } k ∈ [ K − 1] , R EP T RANSFER collects a joint dataset across the source tasks using these policies and cross-sampling across pairs of tasks (1), and then learns a single representation (cid:98) ϕ .
Revised: Given the exploratory policies { π k } k ∈ [ K − 1] , R EP T RANSFER collects a joint dataset across the source tasks using these policies and cross-sampling across pairs of tasks (1), and then learns a single representation (cid:98) ϕ .

Original: In the deployment phase, R EP T RANSFER runs optimistic least squares value iteration (Algorithm 6) using the learned representation (cid:98) ϕ in the target task 1 .
Revised: In the deployment phase, R EP T RANSFER runs optimistic least squares value iteration (Algorithm 6) using the learned representation (cid:98) ϕ in the target task 2 .

Original: The cross sampling procedure and learning a shared representation.
Revised: The cross sampling procedure and learning a shared representation.

Original: We now describe the cross sampling procedure in detail.
Revised: We now describe the cross sampling procedure in detail.

Original: Given an exploratory policy π k for each source task k , the next step is to sample fresh data under a cross-sampling procedure using the generative model.
Revised: Given an exploratory policy π k for each source task k , the next step is to sample fresh data under a cross-sampling procedure using the generative model.

Original: Consider a particular ( k, j ) pair of source environments.
Revised: Consider a particular ( k, j ) pair of source environments.

Original: For each h ∈ [ H ] , we first sample s h − 1 , a h − 1 ∼ d π k k ; h − 1 .
Revised: For each h ∈ [ H ] , we first sample s h − 1 , a h − 1 ∼ d π k k ; h − 1 .

Original: Then, in the simulator of task j , we reset to ( s h − 1 , a h − 1 ) and perform a transition step to s h , i.e., s h ∼ P j ; h − 1 ( s h − 1 , a h − 1 ) .
Revised: Then, in the simulator of task j , we reset to ( s h − 1 , a h − 1 ) and perform a transition step to s h , i.e., s h ∼ P j ; h − 1 ( s h − 1 , a h − 1 ) .

Original: Then we reset the simulator for task k to state s h , sample a uniformly random action a h and then perform another transition to s
Revised: Then we reset the simulator for task k to state s h , sample a uniformly random action a h and then perform another transition to s

Original: h +1 , i.e., s h
Revised: h +1 , i.e., s h

Original: +1 ∼ P ⋆ k
Revised: +1 ∼ P ⋆k

Original: ( s h , a h ) .
Revised: ( s h , a h ) .

Original: Such a procedure is only possible under the generative model setting.
Revised: Such a procedure is only possible under the generative model setting.

Original: Given datasets, each of size n , collected using the cross-sampling procedure, we perform an MLE-based representation learning procedure (Algorithm 3) jointly on all source tasks.
Revised: Given datasets, each of size n , collected using the cross-sampling procedure, we perform an MLE-based representation learning procedure (Algorithm 3) jointly on all source tasks.

Original: An adaptation of the result of Agarwal et al.
Revised: An adaptation of the result of Agarwal et al.

Original: (2020, Theorem 21) to the multi-task setting shows that Multi-task R EP L EARN (Algorithm 3) achieves the following total variation guarantee with probability at least 1 − δ :
Revised: (2020, Theorem 21) to the multi-task setting shows that Multi-task R EP L EARN (Algorithm 3) achieves the following total variation guarantee with probability at least 1 − δ :

Original: [Equation]
Revised: [Equation]

Original: where ν k ; h denotes the generating distribution of inputted datasets D k ; h , each with size N .
Revised: where ν k ; h denotes the generating distribution of inputted datasets D k ; h , each with size N .

Original: For example, in R EP T RANSFER we have D k ; h = ∪ K − 1 j =1 D kj ; h and N = n
Revised: For example, in R EP T RANSFER we have D k ; h = ∪ K − 1 j =1 D kj ; h and N = n

Original: ( K − 1) .
Revised: ( K − 1) .

Original: Representational transfer.
Revised: Representational transfer.

Original: Next we transfer this guarantee to the target task via Assumption 2.4, a key insight of our work.
Revised: Next we transfer this guarantee to the target task via Assumption 3.4, a key insight of our work.

Original: Specifically, we show that under cross sampling and Assumption 2.4, (cid:98) ϕ also linearly approximates the true transition in the target task well, under the occupancy measure of any policy .
Revised: Specifically, we show that under cross sampling and Assumption 3.4, (cid:98) ϕ also linearly approximates the true transition in the target task well, under the occupancy measure of any policy .

Original: Remarkably, this holds before the agent has ever interacted with the target task.
Revised: Remarkably, this holds before the agent has ever interacted with the target task.

Original: Lemma 3.1 Suppose Assumption 2.4 and that for all source tasks k ∈
Revised: Lemma 4.1 Suppose Assumption 3.4 and that for all source tasks k ∈

Original: [ K − 1] we have a λ minexploratory policy π k .
Revised: [ K − 1] we have a λ minexploratory policy π k .

Original: Then, for any δ ∈ (0 , 1) , learning features (cid:98) ϕ using the cross-sampling procedure of Algorithm 1 satisfies (2) w.p. 1 − δ .
Revised: Then, for any δ ∈ (0 , 1) , learning features (cid:98) ϕ using the cross-sampling procedure of Algorithm 1 satisfies (2) w.p. 1 − δ .

Original: Furthermore, for any h = 0 , 1 , ..., H − 1 , there exists (cid:101) µ h : S → R d such that for any function g : S → [0 , 1] , ∥ (cid:82) g ( s )d (cid:101)
Revised: Furthermore, for any h = 0 , 1 , ..., H − 1 , there exists (cid:101) µ h : S → R d such that for any function g : S → [0 , 1] , ∥ (cid:82) g ( s )d (cid:101)

Original: µ h ( s ) ∥ 2 ≤ ¯ α √ d and sup π E π,P ⋆K (cid:13)(cid:13)(cid:13)(cid:98)
Revised: µ h ( s ) ∥ 2 ≤ ¯ α √ d and sup π E π,P ⋆K (cid:13)(cid:13)(cid:13)(cid:98)

Original: ϕ h ( s h , a h ) ⊤ (cid:101) µ h ( · ) − ϕ ⋆h
Revised: ϕ h ( s h , a h ) ⊤ (cid:101) µ h ( · ) − ϕ ⋆h

Original: ( s h , a h ) ⊤ µ ⋆K ; h ( · ) (cid:13) (cid:13)(cid:13) TV ≤ ε TV := (cid:112) |A| α 3max Kζ n /λ min .
Revised: ( s h , a h ) ⊤ µ ⋆K ; h ( · ) (cid:13)(cid:13)(cid:13) TV ≤ ε TV := (cid:112) |A| α 3max Kζ n /λ min .

Original: Lemma 3.1 implies that (cid:98) ϕ is a feature such that P ⋆K is an approximatly linear MDP in (cid:98) ϕ .
Revised: Lemma 4.1 implies that (cid:98) ϕ is a feature such that P ⋆K is an approximatly linear MDP in (cid:98) ϕ .

Original: Learning in approximately linear MDPs has been studied in Jin et al.
Revised: Learning in approximately linear MDPs has been studied in Jin et al.

Original: (2020b), but under a much stronger ℓ ∞ error bound rather than the average misspecification here.
Revised: (2020b), but under a much stronger ℓ ∞ error bound rather than the average misspecification here.

Original: Our result for downstream task learning under such an average model misspecification case presents the strongest result for learning in an approximately linear MDP, and the result might be of independent interest (see Section H for details).
Revised: Our result for downstream task learning under such an average model misspecification case presents the strongest result for learning in an approximately linear MDP, and the result might be of independent interest.

Original: Regret bound for R EP T RANSFER .
Revised: Regret bound for R EP T RANSFER .

Original: Putting things together, we obtain the following regret bound for online learning in the target task using our learned representation (cid:98) ϕ .
Revised: Putting things together, we obtain the following regret bound for online learning in the target task using our learned representation (cid:98) ϕ .

Original: Theorem 3.1 (Regret under generative source access)
Revised: Theorem 4.1 (Regret under generative source access)

Original: Suppose Assumptions 2.1-2.4 and 3.1, and suppose the input policies π k are λ min -exploratory.
Revised: Suppose Assumptions 3.1-3.4 and 4.1, and suppose the input policies π k are λ min -exploratory.

Original: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EP T RANSFER when deployed in the target task has regret at most (cid:101) O (cid:16) ¯ αH 2 d 1 .
Revised: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EP T RANSFER when deployed in the target task has regret at most (cid:101) O (cid:16) ¯ αH 2 d 1 .

Original: 5 (cid:112) T log(1 /δ )
Revised: 5 (cid:112) T log(1 /δ )

Original: (cid:17) , with at most Kn generative accesses per source task, with n = O (cid:16) λ − 1min
Revised: (cid:17) , with at most Kn generative accesses per source task, with n = O (cid:16) λ − 1min

Original: Aα 3max KT (cid:16) log | Φ | δ + K log | Υ | (cid:17)(cid:17) .
Revised: Aα 3max KT (cid:16) log | Φ | δ + K log | Υ | (cid:17)(cid:17) .

Original: Remarkably, Theorem 3.1 shows that with the pre-trained features, we achieve the same regret bound on the target task to the setting of linear MDP with known ϕ
Revised: Remarkably, Theorem 4.1 shows that with the pre-trained features, we achieve the same regret bound on the target task to the setting of linear MDP with known ϕ

Original: ⋆
Revised: ⋆

Original: (Jin et al., 2020b), up to the additional ¯ α factor.
Revised: (Jin et al., 2020b), up to the additional ¯ α factor.

Original: The scaling factor ¯ α only depends on α itself and captures the hardness of transfer learning.
Revised: The scaling factor ¯ α only depends on α itself and captures the hardness of transfer learning.

Original: For special cases such as convex combination, i.e., α is state-independent and α h ∈ ∆( K ) , then ¯ α = 1 .
Revised: For special cases such as convex combination, i.e., α is state-independent and α h ∈ ∆( K ) , then ¯ α = 1 .

Original: In the worst-case, some dependence on the scale of α seems unavoidable as we can have a state s ′ such that µ K ( s ′ )
Revised: In the worst-case, some dependence on the scale of α seems unavoidable as we can have a state s ′ such that µ K ( s ′ )

Original: = 1 and µ 1 ( s ′ ) ≪ 1 with α 1 ( s ′ )
Revised: = 1 and µ 1 ( s ′ ) ≪ 1 with α 1 ( s ′ )

Original: ≫ 1 .
Revised: ≫ 1 .

Original: This corresponds to a rarely observed state for the source task encountered often in the target, and our estimates of transitions involving this state can be highly unreliable if it is not seen in any other source, roughly scaling the error between target and source tasks as | α 1 ( s ′ ) | .
Revised: This corresponds to a rarely observed state for the source task encountered often in the target, and our estimates of transitions involving this state can be highly unreliable if it is not seen in any other source, roughly scaling the error between target and source tasks as | α 1 ( s ′ ) | .

Original: Obtaining formal lower bounds that capture a matching dependence on structural properties of α is an interesting question for future research.
Revised: Obtaining formal lower bounds that capture a matching dependence on structural properties of α is an interesting question for future research.

Revised: 4.1 Reward-free exploration in the source environments.

Revised: So far we have assumed that we have a reward-free exploration black box algorithm that can provide an exploratory policy π k for each source task k .

Revised: In this section, we give a detailed algorithm that achieves this goal.

Original: One natural question is whether transfer learning is possible with only online access to  the source tasks.
Revised: Recall that our transfer learning algorithm, for source task P ⋆k , relies on an exploratory policy π k such that the empirical covariance with respect to ϕ ⋆h is lower bounded. This ensures good exploration in the underlying ground truth feature space in P ⋆K . R EWARD F REE Algorithm 2 achieves this goal by first invoking the R EWARD F REE R EP -UCB algorithm (Algorithm 4) to learn an estimated linear

Revised: MDP model (cid:98) P k for P ⋆k .

Revised: Subsequently, we perform reward-free exploration (e.g., using L SVI -U CB with zero reward) within each (cid:98) P k which involves no further environment interactions.

Revised: Lemma 4.2 (Reward-free Exploration) Fix any source task k ∈

Revised: [ K − 1] .

Revised: Suppose Assumptions 3.2 and 3.3.

Revised: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EWARD F REE (Algorithm 2) with

Revised: N L SVI -U CB

Revised: = (cid:101)

Revised: Θ (cid:0)

Revised: A 3 d 6 H 8 ψ − 2 (cid:1) and N R EWARD F REE = (cid:101) O (cid:16) A 3 d 4 H 6 log (cid:16) | Φ || Υ | δ (cid:17) N 2 L SVI -U CB (cid:17) returns a λ min -exploratory policy π k where λ min = (cid:101) Ω (cid:0)

Revised: A − 3 d − 5 H − 7 ψ 2 (cid:1) .

Revised: The sample complexity here is

Revised: N R EWARD F REE episodes in the source task.

Revised: To the best of our knowledge, Lemma 4.2 is the first result that finds a full-rank policy cover in the low-rank MDP setting, and might be of independent interest.

Revised: Wagenmaker et al.

Revised: (2022) recently obtained a related guarantee in the linear MDP setting with known features ϕ ⋆ .

Revised: Incorporating

Revised: Lemma 4.2 and Theorem 4.1, we get our final sample complexity bound.

Revised: Theorem 4.2 (Regret for R EP T RANSFER with R EWARD F REE subroutine)

Revised: Suppose Assumptions 3.1-3.4 and 4.1 hold, and let δ ∈ (0 , 1) .

Original: {+Let { π k } K − 1 k =
+}
Revised: {+Let { π k } K − 1 k =
+}

Revised: be exploratory policies learned from running R EWARD F REE on each source task with N L SVI -U CB and N R EWARD F REE set as in

Revised: Lemma 4.2.

Revised: Then, w.p. 1 − δ , running R EP T RANSFER with { π k } K − 1 k =1 has regret in the target task of (cid:101) O (cid:16) ¯ αH 2 d 1 .

Revised: 5 (cid:112) T log(1 /δ )

Revised: (cid:17) , with at most (cid:101) O (cid:0) A 4 α 3 max d 5 H 7 K 2 Tψ − 2 (log( | Φ | /δ )

Revised: + K log | Υ | ) (cid:1) generative accesses per source task.

Revised:  Transfer Learning with Online Access to Source Tasks

Revised: In the previous section, we show that efficient transfer learning is possible under very weak structural assumptions, assuming the help of generative access to the source tasks.

Revised: One natural question is whether transfer learning is possible with only online access to the source tasks.

Original: In that case, the cross-sampling procedure no longer works.
Revised: In that case, the cross-sampling procedure no longer works.

Original: Somewhat surprisingly, we show that this is impossible without significantly stronger assumption.
Revised: Somewhat surprisingly , we show that this is impossible without significantly stronger assumption.

Original: Theorem 3.2 (Impossibility Result) Let M K be the set of K-task
Revised: Theorem 5.1 (Impossibility Result) Let M K be the set of K-task

Original: multi-set that satisfies (1) all tasks are Block MDPs; (2) all tasks satisfy Assumption 2.3 and Assumption 2.4; (3) the latent dynamics are exactly the same for all source and target tasks.
Revised: multi-set that satisfies (1) all tasks are Block MDPs; (2) all tasks satisfy Assumption 3.3 and Assumption 3.4; (3) the latent dynamics are exactly the same for all source and target tasks.

Original: For any pre-training algorithm A which outputs a feature ˆ ϕ by interacting with the source tasks k ∈
Revised: For any pre-training algorithm A which outputs a feature ˆ ϕ by interacting with the source tasks k ∈

Original: [ K − 1] , there exists { P ⋆k } k ∈ [ K ] ∈ M K , such that with probability at least 1 / 2 , A will output a feature ˆ ϕ , such that for any policy taking the functional form of π ( s ) =
Revised: [ K − 1] , there exists { P ⋆k } k ∈ [ K ] ∈ M K , such that with probability at least 1 / 2 , A will output a feature ˆ ϕ , such that for any policy taking the functional form of π ( s ) =

Original: f (cid:16) { ˆ ϕ ( s, a ) } a ∈A , { r ( s, a ) } a ∈A (cid:17) , we have V ⋆K − V πK ≥ 1 / 2 .
Revised: f (cid:16) { ˆ ϕ ( s, a ) } a ∈A , { r ( s, a ) } a ∈A (cid:17) , we have V ⋆K − V πK ≥ 1 / 2 .

Original: The above theorem implies that a representation learned only from online access to source tasks does not enable learning in downstream tasks if the downstream task algorithm is restricted to use the representation as the only information of the state-action pairs (e.g., running L SVI -U CB with ˆ ϕ ). 
Revised: The above theorem implies that a representation learned only from online access to source tasks does not enable learning in downstream tasks if the downstream task algorithm is restricted to use the representation as the only information of the state-action pairs (e.g., running L SVI -U CB with ˆ ϕ ). We briefly explain the intuition behind the above lower bound. For a Block MDP, for any ( s, a ) , we can model the ground-truth ϕ ⋆ as a one-hot encoding e

Original: Appendices
Revised: ( z,a ) corresponding to the latent state-action pair ( z, a ) with z = ψ ⋆ ( s ) being the encoded latent state. The key observation here is that any permutation of ϕ ⋆ will also be a perfect feature in terms of characterizing the Block MDPs, since it corresponds to simply permuting the indices of the latent states. Therefore, without cross referencing, the agent could potentially learn different permutations in different source tasks, which would collapse in the target task. A precise constructive proof of Theorem 5.1 can be found in Section C.

Revised: Part of the reason that the above example fails is that each source task has its own observed subset of raw states, which permits such permutation to happen.

Revised: In what follows, we show that, indeed, under an additional assumption on the reachability of raw states, a slight variant of the same algorithm (Algorithm 5 in Section F) can achieve the same regret with only online access to the source tasks.

Revised: Assumption 5.1 (Reachability in the raw states)

Revised: For all source tasks k ∈

Revised: [ K − 1] , any policy π and h = 0 , 1 , ..., H − 1 , we have inf s ∈S ,a ∈A d πk

Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: Assumption 5.1 implies that for each source task, any policy that achieves a full-rank covariance matrix also achieves global coverage across the raw state-action space.

Revised: In addition, in order to apply importance sampling (IS) to transfer the TV error from source task to target task, we need to assume

Revised: [Figure]

Revised: O-

Revised: [Figure]

Revised: EP T RANSFER

Revised: G-R EP T RANSFER

Revised: Oracle

Revised: Target

Revised: [Figure]

Revised: that the target task distribution has bounded density.

Revised: This is true, for example, when S is discrete or when S is compact and has bounded measure.

Revised: Assumption 5.2 (Bounded density)

Revised: For all ( π, h, s, a ) , we have d πK ; h ( s, a ) ≤ 1 .

Revised: Theorem 5.2 (Regret with online access)

Revised: Suppose Assumptions 3.1-3.4,5.1,5.2 hold.

Revised: For any δ ∈ (0 , 1) , w.p.

Revised: 1 − δ , Algorithm 5 with appropriately set parameters achieves a regret in the target task at most (cid:101) O (cid:16) ¯ αd 1 .

Revised: 5 H 2 (cid:112) T log(1 /δ )

Revised: (cid:17) , with at most poly (cid:0) A, α max , d, H, K, T, ψ − 1 , ψ − 1 raw , log( | Φ || Υ | /δ ) (cid:1) online queries in the source tasks.

Revised: Assumption 5.1 is satisfied in a Block MDP, when, for example, the emission function o ( s | z ) satisfies that ∀ s, ∃ z, s.t., o

Revised: ( s | z ) ≥ c .

Revised: That is, for any source task, any state in the state space can be generated by at least one latent state.

Revised:  Experiments

Revised: For the empirical study, we investigate the benefit of transfer learning under the Block MDP setting, on the challenging Rich Observation Combination Lock (comblock) benchmark.

Revised: We choose this environment because one must recover the correct feature from rich observations and perform strategic exploration or otherwise pay for exponential sample complexity.

Revised: We include a visual overview in

Revised: Fig. 1(a).

Revised: This environment is uniquely challenging since it requires strategic exploration and latent state discovery at the same time, which results the failures of common deep RL methods (Misra et al., 2020), and theoretical RL approaches based on linear function approximation (and reproducing kernel Hilbert space) (Zhang et al., 2022).

Revised: In particular, in this section we study the following questions: i) what are the benefits of representational transfer using multiple source tasks, and ii) whether generative access to source tasks is needed.

Revised: We design two sets of experiments with various source and target environment configurations.

Revised: We defer the details of the experiments (the design of vanilla comblock, hyperparameters, etc.) in

Revised: Appendix I and include the essential design information in the following two sections.

Revised: Baselines.

Revised: We denote Source as the smallest sample complexity of L SVI -U CB using learned features from any of the source tasks

Revised: R EP T RANSFER

Revised: only online access to the source

Revised: ; G-R EP T RANSFER

Revised: R EP T RANSFER

Revised: generative access to the source tasks; Oracle

Revised: in the target

Revised: ground truth features

Revised: and Target as

Revised: B

Revised: — the SOTA Block MDP algorithm

Revised: in the target task

Revised: no

Revised: [Table]

Revised: [Figure]

Revised: 6.1 Comblock without partitioned observations

Revised: In this section we start off with an easier setting: we use 5 source tasks, each with horizon H = 25 ,latent states per level and 10 actions.

Revised: The latent transition dynamics are different for each source task, but notice that for comblock, the reachability assumption (c r. Assumption 3.3) is always satisfied.

Revised: The emission distribution of all the source and target tasks is identical, so Assumption 5.1 holds here.

Revised: For the construction of the target task, for each timestep h , we choose the latent transition dynamics from one of the sources uniformly at random (thus Assumption 3.4 is satisfied).

Revised: We record the number of episodes in the target environment that each method takes to solve the target environment in Table. 1(b).

Revised: We first observe that R EP T RANSFER with either online or generative access can solve the target task (since.

Revised: Assumption 5.1 holds).

Revised: Second, we observe that directly applying the learned feature from any single source task does not suffice to solve the target environment.

Revised: This is because the representation learned from a single source task may collapse two latent states into a single one during encoding (e.g., if two latent states at the same time step have exactly identical latent transitions).

Revised: See the visualization of comparisons between the learned decoders from a single source task and

Revised: R EP T RANSFER in Fig.

Revised: 2(a).

Revised: Third, the result shows that R EP T RANSFER saves order of magnitude of target samples compared with training in the target environment from scratch using the SOTA

Revised: Block MDPs algorithm BRIEE.

Revised: This set of results verifies the empirical benefits of representation learning from multiple tasks, i.e., resolves ambiguity and speeds up downstream task learning .

Revised: 6.2 Comblock with partitioned observation space (Comblock-PO)

Revised: In this section, following the intuition of our lower bound (Theorem 5.1), we construct a setting where the supports of the emission distributions from each task are completely disjoint, while the emission distribution in the target task is a mixture of all source emissions and the latent dynamics are identical across tasks.

Original: [Table]
Revised: Hence Assumption 3.4 holds while Assumption 5.1 fails. So we expect that an algorithm without generative access to source tasks will fail based on Theorem 5.1. Specifically, under disjoint emission supports, a representation can decode the latent state correctly for each source, but permute latent state labels across sources, causing decoding errors on the target. We record the number of target episodes for each method to solve the target task in Table. 1(b).

Original: N
Revised: We observe that indeed the online version fails while the generative version still succeeds. We show the visualizations of O-R EP T RANSFER and G-R EP T RANSFER in Fig. 2(b), and we note that the empirical results exactly match our theoretical results. This ablation verifies that source generative model access is needed , unless one has additional stronger distributional assumptions. Notations

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: We
Revised: Transfer learning with generative access.

Original: [Table]
Revised: [Table]

Revised: Free exploration in low-rank MDP.

Original: [Table]
Revised: [Table]

Original: Simplified version of R EP T RANSFER if we have reachability under raw states (see Theorem ?? )
Revised: Multi-task Maximum Likelihood (MLE) Representation Learning.

Original: [Table]B R ELATED W ORK
Revised: [Table] R EP -UCB

Original: Multi-task and Transfer Learning in Supervised Learning.

Original: The theoretical benefit of representation learning are well studied under conditions such as the i.i.d.

Original: task assumption (Maurer et al., 2016) and the diversity assumption (Du et al., 2020; Tripuraneni et al., 2020).

Original: Many works below successfully adopt the frameworks and assumptions to sequential decision making problems.

Original: Multi-task and Transfer Learning in Bandit and small-size MDPs.

Original: A sequence of recent works study multi-task linear bandit problems with linear representation functions ( ϕ ( s ) = A s ) (Hu et al., 2021; Yang et al., 2020, 2022).
Revised: for

Original: The techniques developed in these works crucially rely on the linear representation function structure and can not be applied to nonlinear function classes. Lazaric et al. (2013) study spectral techniques for online sequential transfer learning. Brunskill and Li (2013) consider multi-task RL under a fixed distribution over finitely many diverse MDPs, while Brunskill and Li (2014) considers transfer learning in semi-MDPs by learning options (temporally extended actions).
Revised: reward-free

Original: All these works consider small size tabular models while we focus on large-scale MDPs. Multi-task and Transfer Learning in RL via representation learning.Beyond tabular MDPs, Arora et al.
Revised: model learning.

Original: (2020) and D’Eramo et al. (2019) show benefits of representation learning in imitation learning and planning, but do not address exploration. Lu et al. (2021) study transfer learning in low-rank MDPs with general nonlinear representations, but make a generative model assumption on both the source tasks and the target task, along with other distributional and structural assumptions.
Revised: [Table]

Original: We do not require generative access to the target task and make much weaker structural assumptions on the source-target relatedness.

Original: Recently and independently, Cheng et al. (2022) also studied transfer learning in low-rank MDPs in the online learning setting, identical to the setting we study in Section ?? . However, their analysis relies on an additional assumption that bounds the point-wise
Revised: Simplified version of R EP T RANSFER if we have reachability under raw states (see Theorem 5.2).

Original: TV error with the population TV error, which we show is in fact not necessary. We refer readers to Appendix C for a more detailed comparison with the above two closely related works. Efficient Representation Learning in RL.
Revised: [Table]

Original: Even in the single task setting, efficient representation learning is an active area of study that has seen many recent advances with exploration (Agarwal et al., 2020; Modi et al., 2021; Uehara et al., 2021; Zhang et al., 2022) or without (Ren et al., 2021). Other papers study feature selection (e.g. Farahmand and Szepesvári, 2011; Jiang et al., 2015; Pacchiano et al., 2020; Cutkosky et al., 2021; Lee et al., 2021; Zhang et al., 2021), or focus on learning sparse models (Hao et al., 2021a,b).
Revised: Optimistmic Least Squares Value Iteration (with bonus).

Original: C C OMPARISONS TO CLOSELY RELATED WORKS
Revised: B Comparisons to closely related works

Original: Recall that in addition to the commonly made shared representation assumption (Assumption 2.1), we made two additional structural assumptions: reachability (Assumption 2.3) and linear span on µ ⋆ (Assumption 2.4).
Revised: Recall that in addition to the commonly made shared representation assumption (Assumption 3.1), we made two additional structural assumptions: reachability (Assumption 3.3) and linear span on µ ⋆ (Assumption 3.4).

Original: The reachability assumption is commonly made in prior works even in single agent RL, e.g. (Modi et al., 2021; Misra et al., 2020).
Revised: The reachability assumption is commonly made in prior works even in single agent RL, e.g. (Modi et al., 2021; Misra et al., 2020).

Original: It ensures that there is no redundant dimensions in the ground-truth representation, which is a reasonable requirement.
Revised: It ensures that there is no redundant dimensions in the ground-truth representation, which is a reasonable requirement.

Original: The linear span assumption is closely related to the diversity assumptions made in prior works of transfer learning in both supervised learning (Tripuraneni et al., 2020) and reinforcement learning (Lu et al., 2021).
Revised: The linear span assumption is closely related to the diversity assumptions made in prior works of transfer learning in both supervised learning (Tripuraneni et al., 2020) and reinforcement learning (Lu et al., 2021).

Original: Lu et al.
Revised: Lu et al.

Original: In the prior work of Lu et al. (2021), which also studies transfer learning in lowrank MDPs with nonlinear function approximations, they need to make the following assumptions:
Revised: In the prior work of Lu et al. (2021), which also studies transfer learning in lowrank MDPs with nonlinear function approximations, they need to make the following assumptions:

Original: shared representation (identical to our Assumption 2.1).
Revised: shared representation (identical to our Assumption 3.1).

Original: task diversity (similar to our Assumption 2.4).
Revised: task diversity (similar to our Assumption 3.4).

Original: generative model access to both the source and the target tasks.
Revised: generative model access to both the source and the target tasks.

Original: In contrast, we only require generative model access to the source tasks and allow online learning in the target task.
Revised: In contrast, we only require generative model access to the source tasks and allow online learning in the target task.

Original: a somewhat strong coverage assumption saying that the data covariance matrix (under the generative data distribution) between arbitrary pairs of features ϕ, ϕ ′ ∈ Φ must be full rank.
Revised: a somewhat strong coverage assumption saying that the data covariance matrix (under the generative data distribution) between arbitrary pairs of features ϕ, ϕ ′ ∈ Φ must be full rank.

Original: In contrast, our analysis only requires coverage in the true feature ϕ
Revised: In contrast, our analysis only requires coverage in the true feature ϕ

Original: ⋆ in the source tasks.
Revised: ⋆ in the source tasks.

Original: the existence of an ideal distribution q on which the learned representation can extrapolate.
Revised: the existence of an ideal distribution q on which the learned representation can extrapolate.

Original: We do not require an assumption of a similar nature.
Revised: We do not require an assumption of a similar nature.

Original: Instead, we show that the data collected from our strategic reward-free exploration phase suffices for successful transfer.
Revised: Instead, we show that the data collected from our strategic reward-free exploration phase suffices for successful transfer.

Original: the uniqueness for each ϕ in the sense of linear-transform equivalence.
Revised: the uniqueness for each ϕ in the sense of linear-transform equivalence.

Original: Two representation functions ϕ and ϕ ′ can yield similar estimation result if and only if they differ by just an invertible linear transformation.
Revised: Two representation functions ϕ and ϕ ′ can yield similar estimation result if and only if they differ by just an invertible linear transformation.

Original: In contrast, we do not make any additional structural assumptions on the function class Φ beyond realizability.
Revised: In contrast, we do not make any additional structural assumptions on the function class Φ beyond realizability.

Original: In summary, our work present a theoretical framework that permits successful representation transfer based on significantly weaker assumptions.
Revised: In summary, our work present a theoretical framework that permits successful representation transfer based on significantly weaker assumptions.

Original: We believe that this is a solid step towards understanding transfer learning in RL.
Revised: We believe that this is a solid step towards understanding transfer learning in RL.

Original: Cheng et al.
Revised: Cheng et al.

Original: A concurrent work of Cheng et al.
Revised: A concurrent work of Cheng et al.

Original: (2022) also studies the exact same problem as ours.
Revised: (2022) also studies the exact same problem as ours.

Original: Both works study the setting where the agent performs reward-free exploration in the source tasks for representation learning and used the learned representation for the target task.
Revised: Both works study the setting where the agent performs reward-free exploration in the source tasks for representation learning and used the learned representation for the target task.

Original: Both works achieves similar sample complexity in source and target tasks, albeit using very different algorithms.
Revised: Both works achieves similar sample complexity in source and target tasks, albeit using very different algorithms.

Original: However, in addition to the assumptions that we made 2 , Cheng et al.
Revised: However, in addition to the assumptions that we made 4 , Cheng et al.

Original: (2022) has to make an additional assumption.
Revised: (2022) has to make an additional assumption.

Original: Assumption [-C.
-] For any two different models in the model class Φ × Ψ , say P 1 ( s ′ | s, a ) = ⟨ ϕ 1 ( s, a ) , µ 1 ( s ′ ) ⟩ and P 2 ( s ′ | s, a ) = ⟨ ϕ [-
-]
Revised: Assumption [-C.
-]B.1 For any two different models in the model class Φ × Ψ , say P 1 ( s ′ | s, a ) = ⟨ ϕ 1 ( s, a ) , µ 1 ( s ′ ) ⟩ and P 2 ( s ′ | s, a ) = ⟨ ϕ [-
-]2 ( s, a ) ,

Original: ( s, a ) , µ 2 ( s ′ ) ⟩ , there exists a constant C R such that for all ( s, a ) ∈ S × A and h ∈ [ H ] ,
Revised: µ 2 ( s ′ ) ⟩ , there exists a constant C R such that for all ( s, a ) ∈ S × A and h ∈ [ H ] ,

Original: [Equation]
Revised: [Equation]

Original: This assumption ensures that the point-wise TV error is bounded, as long as the population-level
Revised: This assumption ensures that the point-wise TV error is bounded, as long as the population-level

Original: TV error is bounded.
Revised: TV error is bounded.

Original: This assumption is used to transfer the MLE error from the source tasks to the target task.
Revised: This assumption is used to transfer the MLE error from the source tasks to the target task.

Original: This type of assumption is strong in the sense that we typically expect C R to scale with |S| .
Revised: This type of assumption is strong in the sense that we typically expect C R to scale with |S| .

Original: In contrast, our analysis (Lemma G.1) shows that this assumption is in fact not necessary assuming online access only to source tasks.
Revised: In contrast, our analysis (Lemma F.1) shows that this assumption is in fact not necessary assuming online access only to source tasks.

Original: The generative access to source task studied here, which enables transfer under weaker reachability assumptions is not studied in their work.
Revised: The generative access to source task studied here, which enables transfer under weaker reachability assumptions is not studied in their work.

Original: It is worth noting that Cheng et al.
Revised: It is worth noting that Cheng et al.

Original: (2022) also study offline RL in the target task which we do not cover, while we mainly focus on the setting of generative models in the source tasks and demonstrating a more complete picture by proving generative model access in source tasks is needed without additional assumptions.
Revised: (2022) also study offline RL in the target task which we do not cover, while we mainly focus on the setting of generative models in the source tasks and demonstrating a more complete picture by proving generative model access in source tasks is needed without additional assumptions.

Original: Comparing to (Cheng et al., 2022), we also further implement and perform experimental evaluations of our algorithm.
Revised: Comparing to (Cheng et al., 2022), we also further implement and perform experimental evaluations of our algorithm.

Original: C. R EWARD - FREE EXPLORATION IN THE SOURCE ENVIRONMENTS .

Original: So far we have assumed that we have a reward-free exploration black box algorithm that can provide an exploratory policy π k for each source task k .

Original: In this section, we give a detailed algorithm that achieves this goal.

Original: Recall that our transfer learning algorithm, for source task P

Original: ⋆k , relies on an exploratory policy π k such that the empirical covariance with respect to ϕ ⋆h is lower bounded.

Original: This ensures good exploration in the underlying ground truth feature space in P ⋆K .

Original: R EWARD F REE Algorithm 2 achieves this goal by first invoking the R EWARD F REE R EP -UCB algorithm (Algorithm 4) to learn an estimated linear

Original: MDP model (cid:98) P k for P ⋆k .

Original: Subsequently, we perform reward-free exploration (e.g., using L SVI -U CB with zero reward) within each (cid:98) P k which involves no further environment interactions.

Original: Lemma C.1 (Reward-free Exploration) Fix any source task k ∈

Original: [ K − 1] .

Original: Suppose Assumptions 2.2 and 2.3.

Original: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EWARD F REE (Algorithm 2) with

Original: N L SVI -U CB

Original: = (cid:101)

Original: Θ (cid:0)

Original: A 3 d 6 H 8 ψ − 2 (cid:1) and N R EWARD F REE = (cid:101) O (cid:16) A 3 d 4 H 6 log (cid:16) | Φ || Υ | δ (cid:17) N 2 L SVI -U CB (cid:17) returns a λ min -exploratory policy π k where λ min = (cid:101) Ω (cid:0)

Original: A − 3 d − 5 H − 7 ψ 2 (cid:1) .

Original: The sample complexity here is

Original: N R EWARD F REE episodes in the source task.

Original: To the best of our knowledge, Lemma C.1 is the first result that finds a full-rank policy cover in the low-rank MDP setting, and might be of independent interest.

Original: Wagenmaker et al.

Original: (2022) recently obtained a related guarantee in the linear MDP setting with known features ϕ ⋆ .

Original: Incorporating

Original: Lemma C.1 and Theorem 3.1, we get our final sample complexity bound.

Original: Theorem C.1 (Regret for R EP T RANSFER with R EWARD F REE subroutine) Suppose Assumptions 2.1-2.4 and 3.1 hold, and let δ ∈ (0 , 1) .

Original: [-Let { π k } K − 1 k =
-]
Revised: [-Let { π k } K − 1 k =
-]

Original: be exploratory policies learned from running R EWARD F REE on each source task with N L SVI -U CB and N R EWARD F REE set as in

Original: Lemma C.1.

Original: Then, w.p. 1 − δ , running R EP T RANSFER with { π k } K − 1 k =1 has regret in the target task of (cid:101) O (cid:16) ¯ αH 2 d 1 .

Original: 5 (cid:112) T log(1 /δ )

Original: (cid:17) , with at most (cid:101) O (cid:0) A 4 α 3max d 5 H 7 K 2 Tψ − 2 (log( | Φ | /δ )

Original: + K log | Υ | ) (cid:1) generative accesses per source task.

Original: D I MPOSSIBILITY R ESULTS
Revised: C Impossibility Results

Original: Here, we present an interesting result showing that the above assumptions we make are so weak that they do not even permit efficient transfer in supervised learning:
Revised: Here, we present an interesting result showing that the above assumptions we make are so weak that they do not even permit efficient transfer in supervised learning:

Original: Theorem D.1 (Counter-example in supervised learning) Assume that we want to perform conditional density estimation, where P ⋆k ( y | x ) =
Revised: Theorem C.1 (Counter-example in supervised learning) Assume that we want to perform conditional density estimation, where P ⋆k ( y | x ) =

Original: ϕ
Revised: ϕ

Original: ⋆ ( x ) ⊤ µ ⋆k ( y ) .
Revised: ⋆ ( x ) ⊤ µ ⋆k ( y ) .

Original: Under Assumption 2.1 (shared representation) and Assumption 2.4 (linear span), and assume that in each source task, one have access to a data generating distribution ρ k ( x ) such that λ min E ρ k
Revised: Under Assumption 3.1 (shared representation) and Assumption 3.4 (linear span), and assume that in each source task, one have access to a data generating distribution ρ k ( x ) such that λ min E ρ k

Original: [ ϕ ⋆ ( x ) ϕ
Revised: [ ϕ ⋆ ( x ) ϕ

Original: ⋆ ( x ) ⊤ ] ≥ ψ (reachability).
Revised: ⋆ ( x ) ⊤ ] ≥ ψ (reachability).

Original: No algorithm can consistently achieve E ρ K [ ∥ ˆ P ⋆K ( y | x ) − P ⋆K ( y | x ) ∥ TV ] ≤ 1 / 2 on the target task using the feature learned from the source tasks with probability more than 1 / 2 .
Revised: No algorithm can consistently achieve E ρ K [ ∥ ˆ P ⋆K ( y | x ) − P ⋆K ( y | x ) ∥ TV ] ≤ 1 / 2 on the target task using the feature learned from the source tasks with probability more than 1 / 2 .

Original: Proof of Theorem D.1.
Revised: Proof of Theorem C.1.

Original: Consider the following example.
Revised: Consider the following example.

Original: X
Revised: X

Original: =
Revised: =

Original: R 

Revised: R

Original: and we have the following 3 sets.
Revised: and we have the following 3 sets.

Original: [Equation]
Revised: [Equation]

Original: where B a (( x, y )) stands for the ball with radius a centered at ( x, y ) .
Revised: where B a (( x, y )) stands for the ball with radius a centered at ( x, y ) .

Original: These will be the support oftasks: task 1 and 2 are two source tasks, task 3 is the target task.
Revised: These will be the support oftasks: task 1 and 2 are two source tasks, task 3 is the target task.

Original: Let’s assume that P ⋆k ( x ) are uniform distribution on S k .
Revised: Let’s assume that P ⋆k ( x ) are uniform distribution on S k .

Original: Suppose that the feature class Φ only contains two functions:
Revised: Suppose that the feature class Φ only contains two functions:

Original: ϕ 1 : { x 1 ≤ 0 & x 2 ≥ x 1 } → (1 , 0) , and (0 , 1) otherwise ϕ 2 : { x 2 ≤ 0 & x 1 ≥ x 2 } → (0 , 1) , and (1 , 0) otherwise
Revised: ϕ 1 : { x 1 ≤ 0 & x 2 ≥ x 1 } → (1 , 0) , and (0 , 1) otherwise ϕ 2 : { x 2 ≤ 0 & x 1 ≥ x 2 } → (0 , 1) , and (1 , 0) otherwise

Original: That is, the feature maps from R 2 to the set of binary encoding of dimension 2, i.e. { (1 , 0) , (0 , 1) } .
Revised: That is, the feature maps from R 2 to the set of binary encoding of dimension 2, i.e. { (1 , 0) , (0 , 1) } .

Original: We further assume that µ ⋆ k
Revised: We further assume that µ ⋆k =

Original: = ( p 1 ( y ) , p 2 ( y )) for some distributions p 1 , p 2 , which is identical for all task k , where ∥ p 1 , p 2 ∥ TV = 1 .
Revised: ( p 1 ( y ) , p 2 ( y )) for some distributions p 1 , p 2 , which is identical for all task k , where ∥ p 1 , p 2 ∥ TV = 1 .

Original: We also assume that µ ⋆k is known to the learner
Revised: We also assume that µ ⋆k is known to the learner

Original: a prior, i.e.
Revised: a prior, i.e.

Original: Υ k = { µ ⋆k } for all k ∈ [ K ] , so all the learner needs to do is to pick the correct ϕ out of two candidates.
Revised: Υ k = { µ ⋆k } for all k ∈ [ K ] , so all the learner needs to do is to pick the correct ϕ out of two candidates.

Original: Given the above setup, it’s easy to verify that both Assumption 2.1 and Assumption 2.4 are satisfied, because the decision boundary of both ϕ 1 and ϕ 2 passes through the support of the source tasks, and all µ ⋆k ’s are identical.
Revised: Given the above setup, it’s easy to verify that both Assumption 3.1 and Assumption 3.4 are satisfied, because the decision boundary of both ϕ 1 and ϕ 2 passes through the support of the source tasks, and all µ ⋆k ’s are identical.

Original: However, ϕ 1 and ϕ 2 are equivalent in S 1 and S 2 in terms of their representation power, therefore no algorithm can always pick the correct feature function with probability more than 1 / 2 , regardless of the number of samples.
Revised: However, ϕ 1 and ϕ 2 are equivalent in S 1 and S 2 in terms of their representation power, therefore no algorithm can always pick the correct feature function with probability more than 1 / 2 , regardless of the number of samples.

Original: Suppose ϕ 1 is the true feature and the algorithm incorrectly chooses ϕ 2 .
Revised: Suppose ϕ 1 is the true feature and the algorithm incorrectly chooses ϕ 2 .

Original: Then, for x ∈ S 3 (cid:84) { x 1 ≥ 0 } which has probability mass 1 / 2 , ˆ P 3 ( y | x ) = p 2 whereas
Revised: Then, for x ∈ S 3 (cid:84) { x 1 ≥ 0 } which has probability mass 1 / 2 , ˆ P 3 ( y | x ) = p 2 whereas

Original: P ⋆ 3 ( y | x ) = p 1 .
Revised: P ⋆ 3 ( y | x ) = p 1 .

Original: Thus, the expected total variation distance between ˆ P K and P ⋆K is 1 / 2 .
Revised: Thus, the expected total variation distance between ˆ P K and P ⋆K is 1 / 2 .

Original: The above construction shows that our assumption are not sufficient to permit reliable representation transfer, even in the supervised learning setting.
Revised: The above construction shows that our assumption are not sufficient to permit reliable representation transfer, even in the supervised learning setting.

Original: Yet, surprisingly, these assumptions are sufficient in the RL setting, implying somehow that transfer learning in RL in easier than transfer learning in SL.
Revised: Yet, surprisingly, these assumptions are sufficient in the RL setting, implying somehow that transfer learning in RL in easier than transfer learning in SL.

Original: To understand this phenomenon, observe that in RL, the marginal distribution on ( s, a ) is not independent from the conditional density P ( s ′ | s, a ) we desire to estimate.
Revised: To understand this phenomenon, observe that in RL, the marginal distribution on ( s, a ) is not independent from the conditional density P ( s ′ | s, a ) we desire to estimate.

Original: In particular, if one collects data in the source tasks in an online fashion via running a policy, ρ ( s, a ) is structurally restricted to be an occupancy distribution generated by the ground-truth transition P ⋆ ( s ′ | s, a ) .
Revised: In particular, if one collects data in the source tasks in an online fashion via running a policy, ρ ( s, a ) is structurally restricted to be an occupancy distribution generated by the ground-truth transition P ⋆ ( s ′ | s, a ) .

Original: Such a connection can only exist in Markov chains, and our analysis elegantly utilizes this additional structure to establish the soundness of the learned representation.
Revised: Such a connection can only exist in Markov chains, and our analysis elegantly utilizes this additional structure to establish the soundness of the learned representation.

Original: Also note, crucially, that we never learn a representation to capture d 0 , which would suffer from similar issues as the supervised learning setting, but is not necessary for sample-efficient RL.
Revised: Also note, crucially, that we never learn a representation to capture d 0 , which would suffer from similar issues as the supervised learning setting, but is not necessary for sample-efficient RL.

Original: Next, we prove the impossibility result in Theorem 3.2, restated below as Theorem D.3.
Revised: Next, we prove the impossibility result in Theorem 5.1, restated below as Theorem C.3.

Original: This result shows that one can not achieve online learning in the source tasks without significantly stronger assumptions such as Assumption ?? .
Revised: This result shows that one can not achieve online learning in the source tasks without significantly stronger assumptions such as Assumption 5.1.

Original: Before that, we provide a preliminary version, showing that the learned ˆ ϕ is not sufficient to fit the transition model in the target task, which motivates the construction in Theorem D.3.
Revised: Before that, we provide a preliminary version, showing that the learned ˆ ϕ is not sufficient to fit the transition model in the target task, which motivates the construction in Theorem C.3.

Original: Theorem D.2 (Impossibility Result: Model Learning) Let M K be the set of K-task
Revised: Theorem C.2 (Impossibility Result: Model Learning) Let M K be the set of K-task

Original: multi-set that satisfies
Revised: multi-set that satisfies

Original: all tasks are Block MDPs; 2.
Revised: all tasks are Block MDPs; 2.

Original: all tasks satisfy Assumption 2.3 and Assumption 2.4; 3.
Revised: all tasks satisfy Assumption 3.3 and Assumption 3.4; 3.

Original: the latent dynamics are exactly the same for all source and target tasks.
Revised: the latent dynamics are exactly the same for all source and target tasks.

Original: For any pre-training algorithm A , there exists { P ⋆k } k =1: K ∈ M K and an occupancy distribution ρ K on the target task, such that with probability at least 1 / 2 , A will output a feature ˆ ϕ and for any µ
Revised: For any pre-training algorithm A , there exists { P ⋆k } k =1: K ∈ M K and an occupancy distribution ρ K on the target task, such that with probability at least 1 / 2 , A will output a feature ˆ ϕ and for any µ

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem D.2.
Revised: Proof of Theorem C.2.

Original: Consider a tabular MDP with 2 latent states z 1 , z 2 and an observation state space S
Revised: Consider a tabular MDP with 2 latent states z 1 , z 2 and an observation state space S

Original: = R 1 (cid:83) R 2 (cid:83) B 1 (cid:83) B 2 , where in task 1 one can only observe R 1 (cid:83) R 2 and in task 2 one can only observe B 1 (cid:83) B 2 .
Revised: = R 1 (cid:83) R 2 (cid:83) B 1 (cid:83) B 2 , where in task 1 one can only observe R 1 (cid:83) R 2 and in task 2 one can only observe B 1 (cid:83) B 2 .

Original: Correspondingly, o 1 ( s | z ) is only supported on R 1 (cid:83) R 2 (i.e., o 1 ( R i |
Revised: Correspondingly, o 1 ( s | z ) is only supported on R 1 (cid:83) R 2 (i.e., o 1 ( R i |

Original: z i ) = 1 ) and similar for task 2.
Revised: z i ) = 1 ) and similar for task 2.

Original: Let the latent state transition be such that P ( z 1 | z 1 , a ) = 1 and P ( z 2 | z 2 , a ) = 1 , i.e. only self-transition regardless of the actions.
Revised: Let the latent state transition be such that P ( z 1 | z 1 , a ) = 1 and P ( z 2 | z 2 , a ) = 1 , i.e. only self-transition regardless of the actions.

Original: Now, consider a 2-element feature class Ψ
Revised: Now, consider a 2-element feature class Ψ

Original: = { ψ 1 , ψ 2 } such that
Revised: = { ψ 1 , ψ 2 } such that

Original: [Equation]
Revised: [Equation]

Original: Denote ϕ i ( s, a )
Revised: Denote ϕ i ( s, a )

Original: =
Revised: =

Original: e ( ψ i ( s ) ,a ) for i ∈ [1 , 2] .
Revised: e ( ψ i ( s ) ,a ) for i ∈ [1 , 2] .

Original: Consider for each task k , a 2-element Υ k class in the form of Υ k = {
Revised: Consider for each task k , a 2-element Υ k class in the form of Υ k = {

Original: ( o k ( s | z 1 ) , o k
Revised: ( o k ( s | z 1 ) , o k

Original: ( s | z 2 )) , ( o k ( s | z 2 ) , o k
Revised: ( s | z 2 )) , ( o k ( s | z 2 ) , o k

Original: ( s | z 1 )) } .
Revised: ( s | z 1 )) } .

Original: Notice that ϕ 1 and ϕ 2 are merely permutations of one another and so given any single task data, the two hypothesis will not be distinguishable by any means.
Revised: Notice that ϕ 1 and ϕ 2 are merely permutations of one another and so given any single task data, the two hypothesis will not be distinguishable by any means.

Original: Therefore, for any algorithm, there is at least probability 1 / 2 that it will choose the wrong hypothesis if the ground truth ϕ
Revised: Therefore, for any algorithm, there is at least probability 1 / 2 that it will choose the wrong hypothesis if the ground truth ϕ

Original: ⋆ is sampled between ϕ 1 and ϕ 2 uniformly at random.
Revised: ⋆ is sampled between ϕ 1 and ϕ 2 uniformly at random.

Original: Suppose ϕ 1 is the correct hypothesis and ϕ 2 is the one that the algorithm picks (i.e., ˆ ϕ =
Revised: Suppose ϕ 1 is the correct hypothesis and ϕ 2 is the one that the algorithm picks (i.e., ˆ ϕ =

Original: ϕ 2 ).
Revised: ϕ 2 ).

Original: Let task 3 be such that any state emits to R 1 (cid:83) R 2 and
Revised: Let task 3 be such that any state emits to R 1 (cid:83) R 2 and

Original: B 1 (cid:83) B 2 each with probability 1 / 2 (i.e., o 3 ( R i | z i )
Revised: B 1 (cid:83) B 2 each with probability 1 / 2 (i.e., o 3 ( R i | z i )

Original: =
Revised: =

Original: o 3 ( B i | z i ) = 0 . 5 ).
Revised: o 3 ( B i | z i ) = 0 . 5 ).

Original: This construction satisfies
Revised: This construction satisfies

Original: Assumption 2.3 and Assumption 2.4.
Revised: Assumption 3.3 and Assumption 3.4.

Original: Then, within task 3, one would encounter observations from both R 1 and B 2 which should be mapped to latent state z 1 and z 2 respectively by the true decoder ϕ 1 , but are instead both mapped to latent state z 1 by the learned decoder ϕ 2 , and thus z 1 and z 2 become indistinguishable.
Revised: Then, within task 3, one would encounter observations from both R 1 and B 2 which should be mapped to latent state z 1 and z 2 respectively by the true decoder ϕ 1 , but are instead both mapped to latent state z 1 by the learned decoder ϕ 2 , and thus z 1 and z 2 become indistinguishable.

Original: Suppose ρ K ( z 1 ) =
Revised: Suppose ρ K ( z 1 ) =

Original: ρ K ( z 2 ) = 1 / 2 , then
Revised: ρ K ( z 2 ) = 1 / 2 , then

Original: [Equation]
Revised: [Equation]

Original: where the last second inequality uses triangle inequality, and the last equality comes from the fact that o 3 ( ·| z 1 ) and o 3 ( ·| z 2 ) have disjoint support which implies that ∥ p ⋆ 1 − p ⋆ 2 ∥ TV = 1 .
Revised: where the last second inequality uses triangle inequality, and the last equality comes from the fact that o 3 ( ·| z 1 ) and o 3 ( ·| z 2 ) have disjoint support which implies that ∥ p ⋆ 1 − p ⋆ 2 ∥ TV = 1 .

Original: Now, we are ready to restate and prove Theorem 3.2.
Revised: Now, we are ready to restate and prove Theorem 5.1.

Original: Theorem D.3 (Impossibility Result: Optimal Policy Identification) Let M K be the set of K-task
Revised: Theorem C.3 (Impossibility Result: Optimal Policy Identification) Let M K be the set of K-task

Original: multi-set that satisfies
Revised: multi-set that satisfies

Original: all tasks are Block MDPs; 2.
Revised: all tasks are Block MDPs; 2.

Original: all tasks satisfy Assumption 2.3 and Assumption 2.4; 3.
Revised: all tasks satisfy Assumption 3.3 and Assumption 3.4; 3.

Original: the latent dynamics are exactly the same for all source and target tasks.
Revised: the latent dynamics are exactly the same for all source and target tasks.

Original: For any pre-training algorithm A , there exists { P ⋆k } k =1: K ∈ M K , such that with probability at least 1 / 2 , A will output a feature ˆ ϕ , such that for any policy taking the functional form of π ( s ) = f (cid:16) { ˆ ϕ ( s, a ) } a ∈A , { r ( s, a ) } a ∈A (cid:17) , we have
Revised: For any pre-training algorithm A , there exists { P ⋆k } k =1: K ∈ M K , such that with probability at least 1 / 2 , A will output a feature ˆ ϕ , such that for any policy taking the functional form of π ( s ) = f (cid:16) { ˆ ϕ ( s, a ) } a ∈A , { r ( s, a ) } a ∈A (cid:17) , we have

Original: V ⋆ − V π ≥ 1 / 2 .
Revised: [Equation]

Original: Proof of Theorem D.3.
Revised: Proof of Theorem C.3.

Original: Consider a tabular MDP with H = 2 , two latent states z 1 , z 2 for h = 1 and two latent states z 3 , z 4 for h = 2 .
Revised: Consider a tabular MDP with H = 2 , two latent states z 1 , z 2 for h = 1 and two latent states z 3 , z 4 for h = 2 .

Original: • For h = 1 , let there be two actions a 1 , a 2 .
Revised: • For h = 1 , let there be two actions a 1 , a 2 .

Original: Let the observation state space be S
Revised: Let the observation state space be S

Original: = R 1 (cid:83) R 2 (cid:83) B 1 (cid:83) B 2 , where in task 1 one can only observe R 1 (cid:83) R 2 and in task 2 one can only observe B 1 (cid:83) B 2 .
Revised: = R 1 (cid:83) R 2 (cid:83) B 1 (cid:83) B 2 , where in task 1 one can only observe R 1 (cid:83) R 2 and in task 2 one can only observe B 1 (cid:83) B 2 .

Original: Correspondingly, o 1 ( s | z ) is only supported on R 1 (cid:83) R(i
Revised: Correspondingly, o 1 ( s | z ) is only supported on R 1 (cid:83) R(i.e.

Original: e., o 1 ( R i | z i ) = 1 ) and similar for task 2.
Revised: , o 1 ( R i | z i ) = 1 ) and similar for task 2.

Original: Let the latent state transition be such that P ( z 3 |
Revised: Let the latent state transition be such that P ( z 3 |

Original: z 1 , a 1 )
Revised: z 1 , a 1 )

Original: =
Revised: =

Original: P ( z 3 | z 2 , a 2 ) = 1 , and P ( z 4 | z 1 , a 2 ) =
Revised: P ( z 3 | z 2 , a 2 ) = 1 , and P ( z 4 | z 1 , a 2 ) =

Original: P ( z 4 | z 2 , a 1 ) = 1 .
Revised: P ( z 4 | z 2 , a 1 ) = 1 .

Original: All rewards arefor h = 1 .
Revised: All rewards arefor h = 1 .

Original: • For h = 2 , in state z 3 , all actions have reward 1 , and in state z 4 all actions have reward 0 .
Revised: • For h = 2 , in state z 3 , all actions have reward 1 , and in state z 4 all actions have reward 0 .

Original: • The initial state distribution is d 0 ( z 1 ) =
Revised: • The initial state distribution is d 0 ( z 1 ) =

Original: d 0 ( z 2 ) = 1 / 2 .
Revised: d 0 ( z 2 ) = 1 / 2 .

Original: Now, consider a 2-element feature class Ψ
Revised: Now, consider a 2-element feature class Ψ

Original: = { ψ 1 , ψ 2 } for h = 1 , such that
Revised: = { ψ 1 , ψ 2 } for h = 1 , such that

Original: [Equation]
Revised: [Equation]

Original: Denote ϕ i ( s, a )
Revised: Denote ϕ i ( s, a )

Original: =
Revised: =

Original: e ( ψ i ( s ) ,a ) for i ∈ [1 , 2] .
Revised: e ( ψ i ( s ) ,a ) for i ∈ [1 , 2] .

Original: In addition, define Υ = { µ 1 , µ 2 } where
Revised: In addition, define Υ = { µ 1 , µ 2 } where

Original: [Equation]
Revised: [Equation]

Original: Notice that ϕ 1 and ϕ 2 are merely permutations of one another and so given any single task data, the two hypothesis will not be distinguishable by any means.
Revised: Notice that ϕ 1 and ϕ 2 are merely permutations of one another and so given any single task data, the two hypothesis will not be distinguishable by any means.

Original: Therefore, for any algorithm, there is at least probability 1 / 2 that it will choose the wrong hypothesis.
Revised: Therefore, for any algorithm, there is at least probability 1 / 2 that it will choose the wrong hypothesis.

Original: Suppose ϕ 1 is the correct hypothesis and ϕis the one that the algorithm picks (i.e., ˆ ϕ = ϕ 2 ).
Revised: Suppose ϕ 1 is the correct hypothesis and ϕis the one that the algorithm picks (i.e., ˆ ϕ = ϕ 2 ).

Original: Let task 3 be such that any state emits to R 1 (cid:83) Rand B 1 (cid:83)
Revised: Let task 3 be such that any state emits to R 1 (cid:83) Rand B 1 (cid:83)

Original: B 2 each with probability 1 / 2 (i.e., o 3 ( R i |
Revised: B 2 each with probability 1 / 2 (i.e., o 3 ( R i |

Original: z i ) =
Revised: z i ) =

Original: o 3 ( B i | z i ) = 0 . 5 ).
Revised: o 3 ( B i | z i ) = 0 . 5 ).

Original: This construction satisfies Assumption 2.3 and Assumption 2.4.
Revised: This construction satisfies Assumption 3.3 and Assumption 3.4.

Original: Then, for any policy that only make decision based on ˆ ϕ
Revised: Then, for any policy that only make decision based on ˆ ϕ

Original: ( s, a ) and r ( s, a ) , π would output the same action for observations in R 1 and B 2 , or for B 1 and R 2 .
Revised: ( s, a ) and r ( s, a ) , π would output the same action for observations in R 1 and B 2 , or for B 1 and R 2 .

Original: However, notice that the optimal policy, which would try to go to z 3 from either z 1 or z 2 , will pick a 1 at R 1 and B 1 while picking a 2 at Rand B 2 , which means that the optimal policy will not agree on R 1 and B 2 , and it also will not agree on R 2 and B 1 .
Revised: However, notice that the optimal policy, which would try to go to z 3 from either z 1 or z 2 , will pick a 1 at R 1 and B 1 while picking a 2 at Rand B 2 , which means that the optimal policy will not agree on R 1 and B 2 , and it also will not agree on R 2 and B 1 .

Original: Thus clearly, no such policy as defined above is capable of capturing the optimal policy.
Revised: Thus clearly, no such policy as defined above is capable of capturing the optimal policy.

Original: From the reward perspective, notice that d π ( z 1 ) =
Revised: From the reward perspective, notice that d π ( z 1 ) =

Original: d π ( z 2 ) = 1 / 2 and d π ( R 1 ) =
Revised: d π ( z 2 ) = 1 / 2 and d π ( R 1 ) =

Original: d π ( R 2 ) =
Revised: d π ( R 2 ) =

Original: d π ( B 1 ) =
Revised: d π ( B 1 ) =

Original: d π ( B 1 ) = 1 / 4 .
Revised: d π ( B 1 ) = 1 / 4 .

Original: Since π ( R 1 )
Revised: Since π ( R 1 )

Original: =
Revised: =

Original: π ( B 2 ) , the agent will only be able to collect reward at one of the R 1 and B 2 (but not at both).
Revised: π ( B 2 ) , the agent will only be able to collect reward at one of the R 1 and B 2 (but not at both).

Original: Similarly, since π ( R 2 ) =
Revised: Similarly, since π ( R 2 ) =

Original: π ( B 1 ) , the agent will only be able to collect reward at one of the R 2 and B 1 by reaching z 3 (but not at both).
Revised: π ( B 1 ) , the agent will only be able to collect reward at one of the R 2 and B 1 by reaching z 3 (but not at both).

Original: This means that π will have average reward 1 / 2 .
Revised: This means that π will have average reward 1 / 2 .

Original: Since the optimal policy will be able to collect reward at all R 1 , R 2 , B 1 , B 2 , it will have average reward 1 .
Revised: Since the optimal policy will be able to collect reward at all R 1 , R 2 , B 1 , B 2 , it will have average reward 1 .

Original: This concludes the proof.
Revised: This concludes the proof.

Revised: (cid:101) s ∼ d (cid:98) π n − 1 h − 1 , (cid:101) a ∼ Unif( A ) , (cid:101) s ′ ∼ P ⋆h − 1 ( (cid:101) s, (cid:101) a ) , (cid:101) a ′ ∼ Unif( A ) , (cid:101) s

Revised: ′′ ∼ P ⋆h ( (cid:101) s ′ , (cid:101) a ′ );

Original: Theorem D.2 and Theorem D.3 show that it’s impossible to allow online learning in the source tasks without much stronger assumptions.
Revised: D h,n = D h,n − 1 ∪ { ( s, a, s ′ ) } , D ′ h,n = D ′ h,n − 1 ∪ { ( (cid:101) s ′ , (cid:101) a ′ , (cid:101) s ′′ ) } . D (cid:98) [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Theorem C.2 and Theorem C.3 show that it’s impossible to allow online learning in the source tasks without much stronger assumptions.

Original: In our paper, we show that our Assumption ?? , which ensures reachability in the raw states, is sufficient to establish an end-to-ending online transfer learning result.
Revised: In our paper, we show that our Assumption 5.1, which ensures reachability in the raw states, is sufficient to establish an end-to-ending online transfer learning result.

Original: However, it is unclear if Assumption ?? is necessary for online learning.
Revised: However, it is unclear if Assumption 5.1 is necessary for online learning.

Original: We leave this as an important direction of future work.
Revised: We leave this as an important direction of future work.

Original: E R EWARD - FREE R EP -UCB
Revised: D Reward-free Rep-UCB

Original: In this section, we adapt the Rep-UCB algorithm (Uehara et al., 2021) for reward-free exploration in a single task.
Revised: In this section, we adapt the Rep-UCB algorithm (Uehara et al., 2021) for reward-free exploration in a single task.

Original: We drop all task subscripts as this section is for a single task only, i.e. think about the task as being each source task.
Revised: We drop all task subscripts as this section is for a single task only, i.e. think about the task as being each source task.

Original: The original Rep-UCB algorithm was for infinite-horizon discounted
Revised: The original Rep-UCB algorithm was for infinite-horizon discounted

Original: MDPs, so we modify it to work for our undiscounted and finite-horizon setting.
Revised: MDPs, so we modify it to work for our undiscounted and finite-horizon setting.

Original: Our goal is to prove that Rep-UCB can learn a model that satisfies strong TV guarantees, i.e. Theorem E.1 and (5).
Revised: Our goal is to prove that Rep-UCB can learn a model that satisfies strong TV guarantees, i.e. Theorem D.1 and (5).

Original: Note that FLAMBE (Agarwal et al., 2020, Theorem 2) can be used for this directly, but at a worse (polynomial) sample complexity.
Revised: Note that FLAMBE (Agarwal et al., 2020, Theorem 2) can be used for this directly, but at a worse (polynomial) sample complexity.

Original: Thus, we do a bit more work to derive a new model-learning algorithm for low-rank MDPs, based on Rep-UCB, that is more sample efficient in the source tasks.
Revised: Thus, we do a bit more work to derive a new model-learning algorithm for low-rank MDPs, based on Rep-UCB, that is more sample efficient in the source tasks.

Original: A finite-horizon analysis of Rep-UCB was done in BRIEE (Zhang et al., 2022), so here we just need to replace BRIEE’s
Revised: A finite-horizon analysis of Rep-UCB was done in BRIEE (Zhang et al., 2022), so here we just need to replace BRIEE’s

Original: RepLearn ζ n with that of the MLE, which is how we learn (cid:98) ϕ
Revised: RepLearn ζ n with that of the MLE, which is how we learn (cid:98) ϕ

Original: and (cid:98) µ , as in
Revised: and (cid:98) µ , as in

Original: Rep-UCB.
Revised: Rep-UCB.

Original: Recall the notation of (Zhang et al., 2022),
Revised: Recall the notation of (Zhang et al., 2022),

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: (cid:13) (cid:98) ϕ

Original: h

Original: [Equation]
Revised: [Equation]

Original: n ( s, a ) (cid:13) (cid:13) (cid:13) (cid:98)

Original: [Equation]

Original: n (cid:98)

Original: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: (cid:88) [Equation] (cid:98)
Revised: =

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]

Original: Learn policy (cid:98) π n = argmax π V π (cid:98) P n , (cid:98) b n and let (cid:98) V n be its value.

Original: 8: Let (cid:98) n = argmin n ≥ N/ 2 (cid:98) V n .

Original: 9: Output: (cid:98) n, (cid:98) P (cid:98)

Original: [Equation]
Revised: [Equation]

Original: By using MLE (Uehara et al., 2021, Lemma 18) to learn models, with probability at least 1 − δ , for any n = 1 , 2 , ..., N and h = 0 , 1 , ..., H − 1 , we have
Revised: By using MLE (Uehara et al., 2021, Lemma 18) to learn models, with probability at least 1 − δ , for any n = 1 , 2 , ..., N and h = 0 , 1 , ..., H − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: whereζ
Revised: where

Original: [Equation]

Original: = O

Original: [Equation]

Original: n

Original: [Equation]
Revised: [Equation]

Original: and |M| =
Revised: and |M| =

Original: max h ∈ [ H ] | Φ h
Revised: max h ∈ [ H ] | Φ h

Original: ||
Revised: ||

Original: Υ h | .
Revised: Υ h | .

Original: We also adopt the same choice of α n , λ n parameters as BRIEE, which we assume from now on.
Revised: We also adopt the same choice of α n , λ n parameters as BRIEE, which we assume from now on.

Original: [Equation]
Revised: [Equation]

Original: As in Rep-UCB, we posit standard assumptions about realizability and normalization, on the (source) task of interest.
Revised: As in Rep-UCB, we posit standard assumptions about realizability and normalization, on the (source) task of interest.

Original: Assumption E.1 For any h = 0 , 1 , ..., H − 1 , we have ϕ ⋆h ∈
Revised: Assumption D.1 For any h = 0 , 1 , ..., H − 1 , we have ϕ ⋆h ∈

Original: Φ h and µ ⋆h ∈ Υ h .
Revised: Φ h and µ ⋆h ∈ Υ h .

Original: For any ϕ ∈ Φ h , ∥ ϕ
Revised: For any ϕ ∈ Φ h , ∥ ϕ

Original: ( s, a ) ∥ 2 ≤ 1 .
Revised: ( s, a ) ∥ 2 ≤ 1 .

Original: For all µ ∈ Υ h and any function g : S → R , we have ∥ (cid:82) s g ( s )d µ ( s ) ∥ 2 ≤ ∥ g ∥ ∞ √ d .
Revised: For all µ ∈ Υ h and any function g : S → R , we have ∥ (cid:82) s g ( s )d µ ( s ) ∥ 2 ≤ ∥ g ∥ ∞ √ d .

Original: Lemma E.1 Let r be any reward function.
Revised: Lemma D.1 Let r be any reward function.

Original: Suppose we ran Algorithm 4 with line 7 having reward r + (cid:98) b n instead of just (cid:98) b n .
Revised: Suppose we ran Algorithm 4 with line 7 having reward r + (cid:98) b n instead of just (cid:98) b n .

Original: Then, for any δ ∈ (0 , 1) , w.p. at least 1 − δ , we have
Revised: Then, for any δ ∈ (0 , 1) , w.p. at least 1 − δ , we have

Revised: N − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Start from the third equation of Zhang et al.
Revised: Start from the third equation of Zhang et al.

Original: (2022, Theorem A.4).
Revised: (2022, Theorem A.4).

Original: Following their proof until the last page of their proof, we arrive at the following: for any n = 1 , 2 , ..., N ,
Revised: Following their proof until the last page of their proof, we arrive at the following: for any n = 1 , 2 , ..., N ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By elliptical potential arguments
Revised: By elliptical potential arguments

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Thus, summing over n
Revised: Thus, summing over n

Original: [Equation]
Revised: [Equation]

Original: noting that nζ n , α n , λ n are increasing in n
Revised: noting that nζ n , α n , λ n are increasing in n

Original: [Equation]
Revised: [Equation]

Original: we can combine the above to get,
Revised: we can combine the above to get,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This gives the following useful corollary for reward free exploration.
Revised: This gives the following useful corollary for reward free exploration.

Original: Lemma E.2 For any δ ∈ (0 , 1) w.p.
Revised: Lemma D.2 For any δ ∈ (0 , 1) w [Equation]

Original:  at least 1 − δ we have
Revised: p. at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By definition of (cid:98) V (cid:98) n , we have
Revised: By definition of (cid:98) V (cid:98) n , we have

Original: [Equation]
Revised: [Equation]

Original: which is bounded by the previous lemma and the fact that V (cid:98) π n P ⋆ ,r =

Revised: which is bounded by the previous lemma and the fact that V (cid:98) π n P ⋆ ,r =

Original: = 0 , since in Algorithm 4, the reward function is zero.
Revised: = 0 , since in Algorithm 4, the reward function is zero.

Original: Conditioning on this, we now show that the environment (cid:98) P (cid:98) n has low TV error for any policy-induced distribution.
Revised: Conditioning on this, we now show that the environment (cid:98) P (cid:98) n has low TV error for any policy-induced distribution.

Original: Theorem E.1 For any policy π , we have
Revised: Theorem D.1 For any policy π , we have

Original: [Equation]
Revised: [Equation]

Original: P ⋆h ( s, a ) − (cid:98) P h, (cid:98) n ( s, a ) (cid:13)(cid:13)(cid:13) TV ≤ O (cid:32)

Original: H 3 d 2 |A| 1 .(cid:114) log( |M| NH/δ )

Original: N (cid:33) :=

Original: ε TV .

Original: Proof.
Revised: Proof.

Original: In this proof, let (cid:98) P = (cid:98) P (cid:98) n , which is the returned environment from the algorithm.
Revised: In this proof, let (cid:98) P = (cid:98) P (cid:98) n , which is the returned environment from the algorithm.

Original: Let r ( s, a ) =
Revised: Let r ( s, a ) =

Original: (cid:13)(cid:13)(cid:13) P ⋆h ( s, a ) − (cid:98)P h
Revised: (cid:13)(cid:13) (cid:13) P ⋆ h ( s, a ) − (cid:98)

Original:  ( s, a ) (cid:13)(cid:13)(cid:13) TV ∈ [0 , 2] .
Revised: P h ( s, a ) (cid:13)(cid:13) (cid:13) TV ∈ [0 , 2] .

Original: Then,
Revised: Then,

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: Simulation lemma)
Revised: P h

Original: [Equation]
Revised: [Equation]

Revised: s, a ) (cid:13)(cid:13)(cid:13) TV .

Original: Thus,
Revised: Thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: N
Revised: (by (Zhang et al., 2022, Lemma A.1))

Original: [Equation]
Revised: [Equation]

Original: ≲

Original: [Equation]
Revised: [Equation]

Original: N [Equation] Lemma E.2) ∈
Revised: ∈

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: N [Equation] This also gives us a guarantee on the TV distance between the visitation distributions induced by P ⋆ vs. by (cid:98) P .
Revised: This also gives us a guarantee on the TV distance between the visitation distributions induced by P ⋆ vs. by (cid:98) P .

Original: Lemma E.3 Suppose (cid:98) P satisfies the following for all h = 0 , 1 , ..., H − 1 ,
Revised: Lemma D.3 Suppose (cid:98) P satisfies the following for all h = 0 , 1 , ..., H − 1 ,

Original: [Equation]
Revised: [Equation]

Original: Then, for any h = 0 , 1 , ..., H − 1 , we have
Revised: Then, for any h = 0 , 1 , ..., H − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Note, for h = 0 , the sum is empty so the right hand side is 0 .
Revised: Note, for h = 0 , the sum is empty so the right hand side is 0 .

Original: Proof.
Revised: Proof.

Original: We proceed by induction for h = 0 , 1 , ..., H − 1 .
Revised: We proceed by induction for h = 0 , 1 , ..., H − 1 .

Original: For the base case of h = 0 , no transition has been taken, so that d π (cid:98) P, 0 =
Revised: For the base case of h = 0 , no transition has been taken, so that d π (cid:98) P, 0 =

Original: d πP ⋆ , 0 .
Revised: d πP ⋆ , 0 .

Original: Now let h ∈ { 0 , 1 , ..., H − 2 } be arbitrary, and suppose that the claim is true for h (IH).
Revised: Now let h ∈ { 0 , 1 , ..., H − 2 } be arbitrary, and suppose that the claim is true for h (IH).

Original: We want to show the claim holds for h + 1 .
Revised: We want to show the claim holds for h + 1 .

Original: One key fact we’ll use is that, for any measure µ , we have ∥ µ ∥ TV = sup ∥ f ∥ ∞ ≤ 1 | E µ [ f ] | .
Revised: One key fact we’ll use is that, for any measure µ , we have ∥ µ ∥ TV = sup ∥ f ∥ ∞ ≤ 1 | E µ [ f ] | .

Original: Below we use the notation that f ( s, π ) =
Revised: Below we use the notation that f [Equation] s, [Equation]

Original:  E a ∼ π ( s ) f ( s, a ) .
Revised: = E a ∼ [Equation] ( s ) f ( s, a ) .

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: (cid:35) (by IH and Jensen) ≤

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: t

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original:  as desired [Equation] Thus [Equation]
Revised: (by (4) and ∥ f ( s, π h +1 ) ∥ ≤ 1 ) as desired.

Original:  when combined with Theorem E. [Equation] we have for h = 0 , 1 , ..., H − 1 and any policy π 
Revised: Thus, when combined with Theorem D.1, we have for h = 0 , 1 , ..., H − 1 and any policy π ,

Original: [Equation]
Revised: [Equation]

Original: In other words, the sample complexity needed for a model-error of ε TV is
Revised: In other words, the sample complexity needed for a model-error of ε TV is

Original: [Equation]
Revised: [Equation]

Original: Note this is much better than FLAMBE’s guarantee (Agarwal et al., 2020, Theorem 2) which requires,
Revised: Note this is much better than FLAMBE’s guarantee (Agarwal et al., 2020, Theorem 2) which requires,

Original: [Equation]
Revised: [Equation]

Original: F R EWARD - FREE E XPLORATION
Revised: E Reward-free Exploration

Original: In this section, we show that the mixture policy returned by Algorithm 2 has good coverage.
Revised: In this section, we show that the mixture policy returned by Algorithm 2 has good coverage.

Original: Recall that Algorithm 2 contains two main steps:
Revised: Recall that Algorithm 2 contains two main steps:

Original: Step1 Learn a model (cid:98)
Revised: Step Learn a model (cid:98)

Original: P .
Revised: P .

Original: This was the focus of the previous section, where our modified R EP -UCB method obtained a strong TV guarantee ) by requiring number 
Revised: This was the focus of the previous section, where our modified R EP -UCB method obtained a strong TV guarantee ((5)) by requiring number of episodes at most,

Original: of

Original: episodes at

Original: most,

Original: [Equation]
Revised: [Equation]

Original: Step2 Run L SVI -U CB
Revised: Step Run L SVI -U CB

Original: (Algorithm 6) in the learned model (cid:98) P with reward at the e -th episode being b h,e
Revised: (Algorithm 6) in the learned model (cid:98) P with reward at the e -th episode being b h,e

Original: and U NIFORM A CTIONS =
Revised: and U NIFORM A CTIONS =

Original: T RUE .
Revised: T RUE .

Original: The optimistic bonus pushes the algorithm to explore directions that are not well-covered yet by the mixture policy up to this point.
Revised: The optimistic bonus pushes the algorithm to explore directions that are not well-covered yet by the mixture policy up to this point.

Original: With elliptical potential, we can establish that this process will terminate in polynomial number of steps.
Revised: With elliptical potential, we can establish that this process will terminate in polynomial number of steps.

Original: We now focus on Step 2.
Revised: We now focus on Step 2.

Original: Let
Revised: Let

Original: π
Revised: π

Original: +1 h denote rolling-in π for h steps and taking uniform actions on the h + 1 step, thus inducing a distribution over s h
Revised: +1 h denote rolling-in π for h steps and taking uniform actions on the h + 1 step, thus inducing a distribution over s h

Original: +1 , a h
Revised: +1 , a h

Original: +1 .
Revised: +1 .

Original: We abuse the notation a little and use π +1 −for a policy that just takes one uniform action from the initial distribution d 0 .
Revised: We abuse the notation a little and use π +1 −for a policy that just takes one uniform action from the initial distribution d 0 .

Original: Lemma F.1 Let δ ∈ (0 , 1) and run R EWARD F REE (Algorithm 2).
Revised: Lemma E.1 Let δ ∈ (0 , 1) and run R EWARD F REE (Algorithm 2).

Original: Let Λ h,N be the empirical covariance at the N -th iteration of L SVI -U CB
Revised: Let Λ h,N be the empirical covariance at the N -th iteration of L SVI -U CB

Original: (Algorithm 6).
Revised: (Algorithm 6).

Original: Then, w.p. at least 1 − δ we have,
Revised: Then, w.p. at least 1 − δ we have,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In this proof, we’ll treat the empirical MDP as P ⋆ , as that is the environment we’re running in.
Revised: In this proof, we’ll treat the empirical MDP as P ⋆ , as that is the environment we’re running in.

Original: Thus, we abuse notation and (cid:98) P h,e is the model-based perpsective of the linear MDP, i.e. (cid:98)
Revised: Thus, we abuse notation and (cid:98) P h,e is the model-based perpsective of the linear MDP, i.e. (cid:98)

Original: ϕ h (cid:98) µ h,e where (cid:98) µ h,e is Λ − 1 h,e (cid:80) e − 1 k =1 (cid:98) ϕ
Revised: ϕ h (cid:98) µ h,e where (cid:98) µ h,e is Λ − 1 h,e (cid:80) e − 1 k =1 (cid:98) ϕ

Original: h ( s kh , a kh ) δ ( s kh +1 ) .
Revised: h ( s kh , a kh ) δ ( s kh +1 ) .

Original: Also, in Algorithm 2, we set reward to be zero, but for the purpose of this analysis, suppose the reward function is precisely the (unscaled) bonus in
Revised: Also, in Algorithm 2, we set reward to be zero, but for the purpose of this analysis, suppose the reward function is precisely the (unscaled) bonus in

Original: L SVI -U CB , i.e. r h,e ( s h , a h ) =
Revised: L SVI -U CB , i.e. r h,e ( s h , a h ) =

Original: b h,e ( s h , a h ) .
Revised: b h,e ( s h , a h ) .

Original: This does not change the algorithm at all since the β -scaling of the bonus dominates this reward in the definition of (cid:98) Q h,e , but thinking about the reward in this way will make our analysis simpler.
Revised: This does not change the algorithm at all since the β -scaling of the bonus dominates this reward in the definition of (cid:98) Q h,e , but thinking about the reward in this way will make our analysis simpler.

Original: Recall the high-level proof structure of reward free guarantee of linear MDP (with known features (cid:98) ϕ )
Revised: Recall the high-level proof structure of reward free guarantee of linear MDP (with known features (cid:98) ϕ )

Original: (Wang et al., 2020, Lemma 3.2).
Revised: (Wang et al., 2020, Lemma 3.2).

Original: Step 1 Show that (cid:98) V h,e ∈ V h and w.p. 1 − δ, for all h, e ,
Revised: Step 1 Show that (cid:98) V h,e ∈ V h and w.p 1 − δ for all h, e ,

Original: [Equation]
Revised: [Equation]

Original: This step only uses self-normalized martingale bounds.
Revised: This step only uses self-normalized martingale bounds.

Original: So, line 9 can use any martingale sequence of states and actions, and this claim still holds, with bonus b h,e using the appropriate covariance under the data.
Revised: So, line 9 can use any martingale sequence of states and actions, and this claim still holds, with bonus b h,e using the appropriate covariance under the data.

Original: Step 2 Show optimism conditioned on Step 1.
Revised: Step 2 Show optimism conditioned on Step 1.

Original: Specifically, for all e = 1 , 2 , ..., N , we have E d 0 (cid:104) V [Equation]
Revised: Specifically, for all e = 1 , 2 , ..., N , we have E d 0 (cid:104) V ⋆ 0 ( s 0 , r e ) − (cid:98) V 0 ,e ( s 0 ) (cid:105) ≤ 0 .

Original: To show this, we need that (cid:98) V h,e
Revised: To show this, we need that (cid:98) V h,e

Original: ( s h ) = (cid:98) Q h,e ( s h , π eh ( s h )) ≥ (cid:98) Q h,e ( s h , π ⋆h ( s h ))
Revised: ( s h ) = (cid:98) Q h,e ( s h , π eh ( s h )) ≥ (cid:98) Q h,e ( s h , π ⋆h ( s h ))

Original: (this is for the unclipped case of V -optimism), which we have satisfied in the algorithm, i.e. π eh is greedy w.r.t.
Revised: (this is for the unclipped case of V -optimism), which we have satisfied in the algorithm, i.e. π eh is greedy w.r.t.

Original: (cid:98) Q h,e .
Revised: (cid:98) Q h,e .

Original: Step 

Revised: Step

Original: Bound the sum (cid:80) e (cid:98) V h,e , where we decompose it as a sum of expected bonuses with the expectation is under π e .
Revised: Bound the sum (cid:80) e (cid:98) V h,e , where we decompose it as a sum of expected bonuses with the expectation is under π e .

Original: Step 3 is the only place where we use the fact that s kh , a kh are data sampled from rolling out (cid:98) π e .
Revised: Step 3 is the only place where we use the fact that s kh , a kh are data sampled from rolling out (cid:98) π e .

Original: For
Revised: For

Original: Step 1 and 2, please refer to existing proofs in (Agarwal et al., 2019; Jin et al., 2020b; Wang et al., 2020).
Revised: Step 1 and 2, please refer to existing proofs in (Agarwal et al., 2019; Jin et al., 2020b; Wang et al., 2020).

Original: Now we show Step 3 for our modified algorithm with uniform actions.
Revised: Now we show Step 3 for our modified algorithm with uniform actions.

Original: First, let us show a simulation lemma.
Revised: First, let us show a simulation lemma.

Original: For any episode e = 1 , 2 , ..., N , for any s , recalling definition of reward being b h,e , we have
Revised: For any episode e = 1 , 2 , ..., N , for any s , recalling definition of reward being b h,e , we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the thresholding on (cid:98) V h,e ’s and the second inequality is due to Step 1.
Revised: where the first inequality is due to the thresholding on (cid:98) V h,e ’s and the second inequality is due to Step 1.

Original: Continuing in this fashion, we have
Revised: Continuing in this fashion, we have

Original: [Equation]
Revised: [Equation]

Original: Summing over e = 1 , 2 , ..., N , we have
Revised: Summing over e = 1 , 2 , ..., N , we have

Original: [Equation]
Revised: [Equation]

Original: For each h = 0 , 1 , ..., H − 1 , apply Azuma’s inequality to the martingale difference sequence ∆ e 
Revised: For each h = 0 , 1 , ..., H − 1 , apply Azuma’s inequality to the martingale difference sequence ∆ e = E

Original: = E ( π eh
Revised: [Equation] π eh

Original: [Equation]
Revised: [Equation]

Original: 1 ) +1 [ b h,e ( s h , a h )]
Revised: 1 ) +1 [ b h,e ( s h , a h )]

Original: − b h,e ( s eh , a eh ) .
Revised: − b h,e ( s eh , a eh ) .

Original: The envelope is at most 2 .
Revised: The envelope is at most 2 .

Original: So, w.p. 1 − δ ,
Revised: So, w.p. 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Now apply a self-normalized elliptical potential bound to the first term, giving that
Revised: Now apply a self-normalized elliptical potential bound to the first term, giving that

Revised: H

Original: [Equation]
Revised: [Equation]

Original: Thus, we finally have
Revised: Thus, we finally have

Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Consider any episode e = 1 , 2 , ..., N .
Revised: Consider any episode e = 1 , 2 , ..., N .

Original: By definition, Λ h,N ⪰ Λ h,e , so for all s, a we have pointwise that b h,N ( s, a ) ≤ b h,e ( s, a ) .
Revised: By definition, Λ h,N ⪰ Λ h,e , so for all s, a we have pointwise that b h,N ( s, a ) ≤ b h,e ( s, a ) .

Original: Hence, for all s , we have V ⋆ 0 ( s ; r N ) ≤ V ⋆ 0 ( s ; r e ) , and further using optimism, we have
Revised: Hence, for all s , we have V ⋆ 0 ( s ; r N ) ≤ V ⋆ 0 ( s ; r e ) , and further using optimism, we have

Original: [Equation]
Revised: [Equation]

Original: Now consider any h and policy π , and consider rolling it out for h − 1 steps and taking a random action.
Revised: Now consider any h and policy π , and consider rolling it out for h − 1 steps and taking a random action.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Summing over h incurs an extra H factor on the right.
Revised: Summing over h incurs an extra H factor on the right.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: Lemma F.2 (One-step back for Linear MDP)
Revised: Lemma E.2 (One-step back for Linear MDP)

Original: Suppose P h
Revised: Suppose P h

Original: = ( ϕ h , µ h ) is a linear MDP.
Revised: = ( ϕ h , µ h ) is a linear MDP.

Original: Suppose ρ is any mixture of n policies, and let Σ h :
Revised: Suppose ρ is any mixture of n policies, and let Σ h :

Original: = n E ρ (cid:2)
Revised: = n E ρ (cid:2)

Original: ϕ h ( s h , a h ) ϕ
Revised: ϕ h ( s h , a h ) ϕ

Original: h ( s h , a h ) ⊤ (cid:3) + λI denote the unnormalized covariance.
Revised: h ( s h , a h ) ⊤ (cid:3) + λI denote the unnormalized covariance.

Original: For any g : S ×
Revised: For any g : S ×

Original: A → R , policy π , and h = 0 , 1 , ..., H − 2 , we have
Revised: A → R , policy π , and h = 0 , 1 , ..., H − 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: h [Equation] Under reachability, we can show that small (squared) bonuses and spectral coverage, in the sense of having lower bounded eigenvalues, are somewhat equivalent.
Revised: Under reachability, we can show that small (squared) bonuses and spectral coverage, in the sense of having lower bounded eigenvalues, are somewhat equivalent.

Original: Lemma F.3 Let Σ be a symmetric positive definite matrix and define the bonus
Revised: Lemma E.3 Let Σ be a symmetric positive definite matrix and define the bonus

Original: b h ( s, a ) =
Revised: b h ( s, a ) =

Original: ∥
Revised: ∥

Original: ϕ ⋆h ( s, a ) ∥ Σ − 1 .
Revised: ϕ ⋆h ( s, a ) ∥ Σ − 1 .

Original: Then we have
Revised: Then we have

Original: For any policy π , E d πh (cid:2) b 2 h ( s, a ) (cid:3) ≤ 1 λ min (Σ) .
Revised: For any policy π , E d πh (cid:2) b 2 h ( s, a ) (cid:3) ≤ 1 λ min (Σ) .

Original: That is, coverage implies small squared bonus.
Revised: That is, coverage implies small squared bonus.

Original: Suppose reachability under ϕ
Revised: Suppose reachability under ϕ

Original: ⋆ (Assumption 2.3), then we have the converse: there exists (cid:98) π , for any policy π , E d (cid:101) πh (cid:2)
Revised: ⋆ (Assumption 3.3), then we have the converse: there exists (cid:98) π , for any policy π , E d (cid:101) πh (cid:2)

Original: b 2 h ( s, a ) (cid:3) ≥ ψ λ min (Σ) .
Revised: b 2 h ( s, a ) (cid:3) ≥ ψ λ min (Σ) .

Original: That is, small squared bonus implies coverage.
Revised: That is, small squared bonus implies coverage.

Original: Proof.
Revised: Proof.

Original: The first claim follows directly from Cauchy-Schwartz.
Revised: The first claim follows directly from Cauchy-Schwartz.

Original: Indeed, for any policy π , we have
Revised: Indeed, for any policy π , we have

Original: [Equation]
Revised: [Equation]

Original: For the second claim, Assumption 2.3 implies that there exist a policy (cid:101)
Revised: For the second claim, Assumption 3.3 implies that there exist a policy (cid:101)

Original: π such that for all vectors v ∈ R d with ∥ v ∥ 2 = 1 , we have E d (cid:101) πh (cid:2) ( ϕ ⋆h ( s, a ) ⊤ v ) 2 (cid:3) ≥ ψ .
Revised: π such that for all vectors v ∈ R d with ∥ v ∥ 2 = 1 , we have E d (cid:101) πh (cid:2) ( ϕ ⋆ h ( s, a ) ⊤ v ) 2 (cid:3) ≥ ψ .

Original: Now decompose Σ
Revised: Now decompose Σ

Original: = (cid:80) di{+d i =
+}
Revised: = (cid:80) {+d i =
+}

Original: =1 λ i v i v ⊤ i , where λ
Revised: λ i v i v ⊤ i , where λ

Original: i , v i are eigenvalue/vector pairs with ∥ v i ∥ 2 = 1 and λ 1 ≥ λ 2 ≥ ... ≥ λ d .
Revised: i , v i are eigenvalue/vector pairs with ∥ v i ∥ 2 = 1 and λ 1 ≥ λ 2 ≥ ... ≥ λ d .

Original: Then substituting this into the definition of the bonus, we have
Revised: Then substituting this into the definition of the bonus, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We now prove our main lemma for reward-free exploration, Lemma C.1.
Revised: We now prove our main lemma for reward-free exploration, Lemma 4.2.

Original: Lemma C.1 (Reward-free Exploration) Fix any source task k ∈
Revised: Lemma 4.2 (Reward-free Exploration) Fix any source task k ∈

Original: [ K − 1] .
Revised: [ K − 1] .

Original: Suppose Assumptions 2.2 and 2.3.
Revised: Suppose Assumptions 3.2 and 3.3.

Original: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EWARD F REE (Algorithm 2) with
Revised: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EWARD F REE (Algorithm 2) with

Original: N L SVI -U CB
Revised: N L SVI -U CB

Original: = (cid:101)
Revised: = (cid:101)

Original: Θ (cid:0)
Revised: Θ (cid:0)

Original: A 3 d 6 H 8 ψ − 2 (cid:1) and N R EWARD F REE = (cid:101) O (cid:16) A 3 d 4 H 6 log (cid:16) | Φ || Υ | δ (cid:17) N 2 L SVI -U CB (cid:17) returns a λ min -exploratory policy π k where λ min = (cid:101) Ω (cid:0)
Revised: A 3 d 6 H 8 ψ − 2 (cid:1) and N R EWARD F REE = (cid:101) O (cid:16) A 3 d 4 H 6 log (cid:16) | Φ || Υ | δ (cid:17) N 2 L SVI [Equation] U CB (cid:17) returns a λ min -exploratory policy π k where λ min = (cid:101) Ω (cid:0)

Original: A − 3 d − 5 H − 7 ψ 2 (cid:1) .
Revised: A − 3 d − 5 H − 7 ψ 2 (cid:1) .

Original: The sample complexity here is
Revised: The sample complexity here is

Original: N R EWARD F REE episodes in the source task.
Revised: N R EWARD F REE episodes in the source task.

Original: Proof of Lemma C.1.
Revised: Proof of Lemma 4.2.

Original: In this proof, let
Revised: In this proof, let

Original: [Equation]
Revised: [Equation]

Original: where λ = dH log( N L SVI -U CB /δ ) ≥ 1 .
Revised: where λ = dH log( N L SVI -U CB /δ ) ≥ 1 .

Original: This setting of λ satisfies the precondition for the Concentration of Inverse Covariances Zanette et al.
Revised: This setting of λ satisfies the precondition for the Concentration of Inverse Covariances Zanette et al.

Original: (2021, Lemma 39), which implies w.p. at least 1 − δ that
Revised: (2021, Lemma 39), which implies w.p. at least 1 − δ that

Original: [Equation]
Revised: [Equation]

Original: where we’ve also used the fact that λ ≥ 1 , so ( A + λI ) − 1 ⪯
Revised: where we’ve also used the fact that λ ≥ 1 , so ( A + λI ) − 1 ⪯

Original: ( A + I ) − 1 .
Revised: ( A + I ) − 1 .

Original: Under this event, for any π , we have,
Revised: Under this event, for any π , we have,

Original: [Equation]
Revised: [Equation]

Original: Now let h = 0 , 1 , ..., H − 2 be arbitrary.
Revised: Now let h = 0 , 1 , ..., H − 2 be arbitrary.

Original: By Assumption 2.3 (there exists some policy (cid:101) π with coverage) such that,
Revised: By Assumption 3.3 (there exists some policy (cid:101) π with coverage) such that,

Original: [Equation]
Revised: [Equation]

Original: (by Lemma F.3)
Revised: (by Lemma E.3)

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13) (

Original: [Equation] (by λ ≥ 1 ) ≤
Revised: (by Corollary E.1) ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:98) P [Equation] (by Corollary F.1) ≤ [Equation]
Revised: CB (by ε TV = 1 /N

Original: [Equation] 1) +
Revised: L SVI -U CB ) ≲ A 1 .

Original: [Equation]
Revised: [Equation]

Original: /N
Revised: d

Original: [Equation]
Revised: [Equation]

Original: (by ε TV = 1 /N
Revised: H

Original: L SVI -U CB ) ≲ A 1 . 5 d 2 H 3 (cid:112) log( dHN L SVI -U CB /δ ) /N
Revised: [Equation] log( dHN L SVI [Equation] U CB /δ ) /N

Revised: L SVI

Revised: [Equation]

Original: L SVI -U CB + 1 /N 
Revised: U CB + 1 /N L SVI

Revised: [Equation]

Original: L SVI -U CB (by (6) and Lemma F.1)
Revised: U CB (by (6) and Lemma E.1) ≲ A 1 . [Equation] d

Original: ≲ A 1 . 5 d 2 H 3 (cid:112) log( dHN L SVI -U CB /δ ) /N
Revised: [Equation] H [Equation] log( dHN L SVI -U CB /δ ) /N

Original: L SVI -U CB .
Revised: L SVI -U CB .

Original: Recall that λ = dH log( N L SVI -U CB /δ ) , we have, {+λ min (cid:16) E ρ +1 h (cid:2) ϕ ⋆h +
 ( s, a ) ϕ ⋆h +
+}
Revised: Recall that λ = dH log( N L SVI -U CB /δ ) , we have, {+λ min (cid:16) E ρ +1 h (cid:2) ϕ ⋆h +
 ( s, a ) ϕ ⋆h +
+}

Original:  [Equation]
Revised: ( s, a ) T (cid:3)(cid:17) = λ min (cid:0) Λ ⋆h +1 (cid:1) − λ N L SVI -U CB [Equation]

Original: where we’ve omitted the log terms for simplicity in the ≳ .
Revised: where we’ve omitted the log terms for simplicity in the ≳ .

Original: Now we optimize N L SVI -U CB to maximize this bound.
Revised: Now we optimize N L SVI -U CB to maximize this bound.

Original: For a, b > 0 , to maximize a function of the form f ( x ) = a √ x − bx , it’s best to set x ⋆ such that √ x ⋆ = 2 ba , resulting in value f
Revised: For a, b > 0 , to maximize a function of the form f ( x ) = a √ x − bx , it’s best to set x ⋆ such that √ x ⋆ = 2 ba , resulting in value f

Original: ( x ⋆ )
Revised: ( x ⋆ )

Original: = a 2 4 b .
Revised: = a 2 4 b .

Original: Setting,
Revised: Setting,

Original: [Equation]
Revised: [Equation]

Original: Hence, we need to set
Revised: Hence, we need to set

Original: [Equation]
Revised: [Equation]

Original: which results in a λ min lower bound of
Revised: which results in a λ min lower bound of

Original: [Equation]
Revised: [Equation]

Original: ψ

Original: A 3 d 5 H(cid:19) .

Original: Finally, we used the fact that ε TV = 1 /N
Revised: Finally, we used the fact that ε TV = 1 /N

Original: L SVI -U CB , which is set by the choice of N R EWARD F REE in the lemma statement to satisfy (5).
Revised: L SVI -U CB , which is set by the choice of N R EWARD F REE in the lemma statement to satisfy (5).

Original: The above proves coverage of ρ +1 h for h = 0 , 1 , ..., H − 2 .
Revised: The above proves coverage of ρ +1 h for h = 0 , 1 , ..., H − 2 .

Original: Finally to argue for ρ +1 − 1 , which is simply taking a random action at time h , we can simply invoke Assumption 2.3 for h = 0 to get a policy (cid:101) π that
Revised: Finally to argue for ρ +1 − 1 , which is simply taking a random action at time h , we can simply invoke Assumption 3.3 for h = 0 to get a policy (cid:101) π that

Original: [Equation]
Revised: [Equation]

Original: Corollary F.1 Let λ, Λ ⋆h , (cid:98) Λ h be defined as in the proof of Lemma C.1.
Revised: Corollary E.1 Let λ, Λ ⋆h , (cid:98) Λ h be defined as in the proof of Lemma 4.2.

Original: For any h = 0 , 1 , ..., H −and any policy π , we have
Revised: For any h = 0 , 1 , ..., H −and any policy π , we have

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this means that coverage in the learned features implies coverage in the true features.
Revised: Intuitively, this means that coverage in the learned features implies coverage in the true features.

Original: Proof.
Revised: Proof.

Original: For shorthand, let N = N L SVI -U CB .
Revised: For shorthand, let N = N L SVI -U CB .

Original: Apply Lemma F.2 (one-step back) to the learned model (cid:98) P and the function ( s, a ) (cid:55)→ ∥
Revised: Apply Lemma E.2 (one-step back) to the learned model (cid:98) P and the function ( s, a ) (cid:55)→ ∥

Original: ϕ ⋆h +

Revised: ϕ ⋆h +

Original: ( s, a ) ∥
Revised: ( s, a ) ∥

Original: ( Λ ⋆h +1 ) − 1 , which is bounded by λ − 1 / 2 ≤ 1 .
Revised: ( Λ ⋆h +1 ) − 1 , which is bounded by λ − 1 / 2 ≤ 1 .

Original: We have,
Revised: We have,

Original: E

Original: [Table] [Table] P [Table]
Revised: [Equation]

Original: +1 ( s h +1 , a h +1 ) (cid:13)(cid:13) ( Λ ⋆h +1 )
Revised: where we used the fact that

Original: [Table] [Table] E π, [Table]
Revised: [Equation]

Original: P

Original: [Table]

Original: where we used the fact that

Original: [Equation]
Revised: [Equation]

Original: where M is a positive definite matrix.
Revised: where M is a positive definite matrix.

Original: Thus, doing an initial change from d πh +1 to d π (cid:98) P,h +1 concludes the proof.
Revised: Thus, doing an initial change from d πh +1 to d π (cid:98) P,h +1 concludes the proof.

Original: G R EPRESENTATION T RANSFER
Revised: F Representation Transfer

Original: First, we prove Lemma 3.1, restated below.
Revised: First, we prove Lemma 4.1, restated below.

Original: Lemma 3.1 Suppose Assumption 2.4 and that for all source tasks k ∈
Revised: Lemma 4.1 Suppose Assumption 3.4 and that for all source tasks k ∈

Original: [ K − 1] we have a λ minexploratory policy π k .
Revised: [ K − 1] we have a λ minexploratory policy π k .

Original: Then, for any δ ∈ (0 , 1) , learning features (cid:98) ϕ using the cross-sampling procedure of Algorithm 1 satisfies (2) w.p. 1 − δ .
Revised: Then, for any δ ∈ (0 , 1) , learning features (cid:98) ϕ using the cross-sampling procedure of Algorithm 1 satisfies (2) w.p. 1 − δ .

Original: Furthermore, for any h = 0 , 1 , ..., H − 1 , there exists (cid:101) µ h : S → R d such that for any function g : S → [0 , 1] , ∥ (cid:82) g ( s )d (cid:101)
Revised: Furthermore, for any h = 0 , 1 , ..., H − 1 , there exists (cid:101) µ h : S → R d such that for any function g : S → [0 , 1] , ∥ (cid:82) g ( s )d (cid:101)

Original: µ h ( s ) ∥ 2 ≤ ¯ α √ d andsup
Revised: µ h ( s ) ∥ 2 ≤ ¯ α √ d and

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3.1.
Revised: Proof of Lemma 4.1.

Original: Fix an arbitrary π .
Revised: Fix an arbitrary π .

Original: Denote µ h ( s ′ )
Revised: Denote µ h ( s ′ )

Original: = (cid:80) K − 1 k =

Revised: = (cid:80) K − 1 k =

Original: α k ; h ( s ′ )
Revised: α k ; h ( s ′ )

Original: (cid:98) µ k ; h ( s ′ ) .
Revised: (cid:98) µ k ; h ( s ′ ) .

Original: First, notethat max g : S→ [0 , 1] (cid:13)(cid:13)(cid:13)(cid:13)(cid:90) µ h ( s ) g ( s ) d ( s ) (cid:13)(cid:13)(cid:13)(cid:13)≤ max g : S→ [0 , 1] K − 1 (cid:88) k = (cid:13)(cid:13)(cid:13)(cid:13)(cid:90) (cid:98) µ k ; h ( s ) α k ; h ( s ) g ( s ) d ( s ) (cid:13)(cid:13)(cid:13)(cid:13) ≤ K − 1 (cid:88) k
Revised: First, note

Original: = max s α k ; h ( s ) √ d (Since (cid:82) (cid:98) µ k ; h ( s ) g ( s ) d ( s ) ≤ √ d by 2.2) = ¯ α √ d
Revised: [Table]

Original: For any h = 0 , 1 , ..., H − 1 , we haveE π,P ⋆K (cid:13)(cid:13)(cid:13)(cid:98) ϕ h ( s h , a h ) ⊤ µ h ( · ) − ϕ ⋆h
Revised: we have

Original: ( s h , a h ) ⊤ µ ⋆K ; h ( · ) (cid:13)(cid:13)(cid:13) TV = E π,P ⋆K  (cid:88) s h + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) K − 1 (cid:88) k = α k ; h ( s h +1 )
Revised: [Table]

Original: (cid:16)(cid:98) ϕ h ( s h , a h ) ⊤ (cid:98) µ k ; h ( s h +1 ) − ϕ ⋆h ( s h , a h ) ⊤ µ ⋆k
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation] ≤ 
Revised:  ≤ α max

Original: [Equation]
Revised: [Equation]

Original: First consider the case when h = 0 .
Revised: First consider the case when h = 0 .

Original: At h = 0 , the distribution under P ⋆ K is the same as ν k,h , and so, we directly get that the above quantity is at most α max ζ 1 / 2 n ≤ ε , which proves the h = 0 case.
Revised: At h = 0 , the distribution under P ⋆K is the same as ν k,h , and so, we directly get that the above quantity is at most α max ζ 1 / 2 n ≤ ε , which proves the h = 0 case.

Original: Now consider any h
Revised: Now consider any h

Original: = 1 , 2 , ..., H − 1 .
Revised: = 1 , 2 , ..., H − 1 .

Original: To simplify notation, let us denote
Revised: To simplify notation, let us denote

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that λ min (Σ k, [Equation] ) ≥ λ min by assumption.
Revised: Note that λ min (Σ k,h ) ≥ λ min by assumption.

Original: Now continuing from where we left off, we take a one-step back as follows,
Revised: Now continuing from where we left off, we take a one-step back as follows,

Original: [Equation]
Revised: [Equation]

Original: By λ min guarantee of Σ k,h , and Jensen’s inequality to push the square inside, ≤ α max √ 
Revised: By λ min guarantee of Σ k,h , and Jensen’s inequality to push the square inside, ≤ α max √ λ min

Original: [Equation]
Revised: [Equation]

Original: min [Equation] By Assumption 2.4, the expectation over P ⋆K ; h − 1 is a linear combination of expectations over P ⋆j ; h − 1 , ≤ A 
Revised: By Assumption 3.4, the expectation over P ⋆K ; h − 1 is a linear combination of expectations over P ⋆j ; h − 1 , ≤ A 1 / 2 α 3 / 2 max √ λ min

Original: [Equation]
Revised: [Equation]

Original: 2 α 3 / 2 max √
Revised: k

Original: [Equation]
Revised: [Equation]

Original: √ λ
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Table]

Original: , where we used the MLE guarantee (2) in the last step. P
Revised: where we used the MLE guarantee (2) in the last step

Original: [Table]
Revised: [Table]

Original:  RAINING 
Revised: Algorithm 5 Transfer learning with online access P RE -T RAINING P HASE

Original: [Table]
Revised: [Table]

Original:  Input: num. L SVI -U CB episodes N L SVI -U CB , num. model-learning episodes N R EWARD F REE , size of cross-sampled datasets n , failure probability δ .
Revised: P HASE Additional Input: number of deployment episodes T .

Original: 1: for source task k = 1 , ..., K − 1 do 2: Find policy cover π k = R EWARD F REE ( P ⋆k , N L SVI -U CB , N R EWARD F REE , δ ) . (Algorithm 2) 3: for source task k
Revised: 1: Set β

Original: =1 , ..., K − 1 do 4: For each h = 0 , 1 , ..., H − 1 , sample D k as n i.i.d. ( s h , a h , s h +1 ) tuples from π k D EPLOYMENT P HASE
Revised: =

Original: Additional Input: number deployment episodes d  αdH 
Revised: H √ d + ¯ αdH (cid:112) log( dHT/δ ) .

Original: log
Revised: 2: Run L SVI -U CB (cid:16) { (cid:98)

Original: dHT δ ) . H − 1 h 
Revised: ϕ h } H − 1 h =

Revised: 0 , r = r K , T, β (cid:17) in the target task

Original: r K P 
Revised: P ⋆K

Original:  Next we state an analogous lemma for when we don’t need generative access to the source task, but instead assume Assumption ?? , and Assumption ?? .
Revised: Algorithm 6). Next we state an analogous lemma for when we don’t need generative access to the source task, but instead assume Assumption 5.1, and Assumption 5.2.

Original: Lemma G.1 Suppose Assumption ?? , and Assumption ?? .
Revised: Lemma F.1 Suppose Assumption 5.1, and Assumption 5.2.

Original: Now take the setup of Lemma 3.1 with the only difference being that (cid:98) ϕ is learned as in Algorithm 5.
Revised: Now take the setup of Lemma 4.1 with the only difference being that (cid:98) ϕ is learned as in Algorithm 5.

Original: Then, the same guarantee of Lemma 3. holds with a slightly different right hand side for the bound on the TV-error,
Revised: Then, the same guarantee of Lemma 4. holds with a slightly different right hand side for the bound on the TV-error,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma G.1.
Revised: Proof of Lemma F.1.

Original: Fix an arbitrary π .
Revised: Fix an arbitrary π .

Original: Denote µ h ( s ′ )
Revised: Denote µ h ( s ′ )

Original: = (cid:80) K − 1 k =

Revised: = (cid:80) K − 1 k =

Original: α k ; h ( s ′ )
Revised: α k ; h ( s ′ )

Original: ˆ µ k ; h ( s ′ ) .
Revised: ˆ µ k ; h ( s ′ ) .

Original: Then, some algebra with importance sampling gives us the bound,
Revised: Then, some algebra with importance sampling gives us the bound,

Original: [Equation]

Original: ⋆K

Original: [Equation]

Original: ϕ

Original: [Equation]

Original: ⊤

Original: [Equation]
Revised: [Equation]

Original: µ

Original: [Equation]

Original: TV ≤

Original: [Equation]

Original: ) −

Original: [Equation]

Original: [Equation]

Original: TV

Original: By
Revised: By

Original: Assumption ?? ,
Revised: Assumption 5.1,

Original: Assumption ?? , for any s, a , we have
Revised: Assumption 5.2, for any s, a , we have

Original: [Equation]
Revised: [Equation]

Original: ≤ 
Revised: ≤ 1 ψ raw

Original: [Equation]
Revised: [Equation]

Original: λ min , where we used the coverage-under- π k assumption in the last inequality.
Revised: min , where we used the coverage-under- π k assumption in the last inequality.

Original: In other words, for each k = 1 , 2 , ..., K − 1 , we have
Revised: In other words, for each k = 1 , 2 , ..., K − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: ∞ ≤ 1 ψ raw λ min , hence we can importance sample,
Revised: min , hence we can importance sample,

Original: [Equation]
Revised: [Equation]

Original: H P ROOFS FOR D EPLOYMENT P HASE H. O NLINE RL L EMMAS
Revised: G Proofs for Deployment Phase G.1 Online RL Lemmas

Original: Lemma H.1 (Self-normalized Martingale) Consider filtrations { F i } i =1 , 2 ,... , so that E [ ε i | F i − 1 ]
Revised: Lemma G.1 (Self-normalized Martingale) Consider filtrations { F i } i =1 , 2 ,... , so that E [ ε i | F i − 1 ]

Original: =and { ε i | F i − 1 } i =1 , 2 ,... are sub-Gaussian with parameter σ 2 .
Revised: =and { ε i | F i − 1 } i =1 , 2 ,... are sub-Gaussian with parameter σ 2 .

Original: Let { X i } i =1 , 2 ,... be random variables in a hilbert space H .
Revised: Let { X i } i =1 , 2 ,... be random variables in a hilbert space H .

Original: Suppose a linear operator Σ 0 : H → H is positive definite.
Revised: Suppose a linear operator Σ 0 : H → H is positive definite.

Original: For any t , define Σ t = Σ 

Revised: For any t , define Σ t = Σ

Original: + (cid:80) ti =
Revised: + (cid:80) ti =

Original: 1 X i X
Revised: 1 X i X

Original: Ti .
Revised: Ti .

Original: Then w.p. at least 1 − δ , we have,
Revised: Then w.p. at least 1 − δ , we have,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Lemma A.8 of (Agarwal et al., 2019).
Revised: Lemma A.8 of (Agarwal et al., 2019).

Original: Lemma H.2 Let Λ t = λI + (cid:80) ti=
Revised: Lemma G.2 Let Λ t = λI + (cid:80) ti

Original:  1 x i x Ti for x i ∈ R d and λ > 0 .
Revised: = 1 x i x Ti for x i ∈ R d and λ > 0 .

Original: Then (cid:80) ti =
Revised: Then (cid:80) ti =

Original: 1 x Ti (Λ t ) − 1 x i ≤ d .
Revised: 1 x Ti (Λ t ) − 1 x i ≤ d .

Original: Proof.
Revised: Proof.

Original: Lemma D.1 of (Jin et al., 2020b).
Revised: Lemma D.1 of (Jin et al., 2020b).

Original: H. P ROOF OF MAIN RESULTS
Revised: G.2 Proof of main results

Original: Let ( x ) ≤ y refer to the clamping operator, i.e. ( x ) ≤ y
Revised: Let ( x ) ≤ y refer to the clamping operator, i.e. ( x ) ≤ y

Original: = min { x, y } .
Revised: = min { x, y } .

Original: Let M V be the maximum possible value in the MDP with the given reward function.
Revised: Let M V be the maximum possible value in the MDP with the given reward function.

Original: Algorithm
Revised: Algorithm

Original: 6 L
Revised: 1: Input: Features { (cid:98)

Original: SVI -U CB
Revised: ϕ h } h =0 , 1 ,...,H − 1 , reward { r h } h =0 , 1 ,...,H − 1 , number of episodes N , bonus scaling parameter β , U NIFORM A CTIONS =

Original: 1: Input
Revised: F ALSE .

Original: 2: for episode e = 1 , 2 , ..., N do 3:
Revised: 2: for episode e = 1 , 2 , ..., N do 3:

Original: Initialize (cid:98) V H,e ( s ) = 0 , ∀ s 4: for step h
Revised: Initialize (cid:98) V H,e ( s ) = 0 , ∀ s 4: for step h

Original: = H − 1 , H − 2 ..., 0 do 5:
Revised: = H − 1 , H − 2 ..., 0 do 5:

Original: Learn best predictor for (cid:98) V eh +1 , Λ h,e = e − 1 (cid:88)
Revised: Learn best predictor for (cid:98) V eh +1 , Λ h,e = e − 1 (cid:88)

Original: k
Revised: k

Original: = (cid:98) ϕ
Revised: = (cid:98) ϕ

Original: h ( s kh , a kh ) (cid:98) ϕ
Revised: h ( s kh , a kh ) (cid:98) ϕ

Original: h ( s kh , a kh ) ⊤
Revised: h ( s kh , a kh ) ⊤

Original: + I, (cid:98) w h,e = Λ − 1 h,e e − 1 (cid:88) k
Revised: + I, (cid:98) w h,e = Λ − 1 h,e e − 1 (cid:88) k

Original: = (cid:98) ϕ
Revised: = (cid:98) ϕ

Original: h ( s kh , a kh ) (cid:98) V h
Revised: h ( s kh , a kh ) (cid:98) V h

Original: +1 ,e ( s kh +1 ) .
Revised: +1 ,e ( s kh +1 ) .

Original: Set bonus and value functions,
Revised: Set bonus and value functions,

Original: [Equation]
Revised: [Equation]

Original:  (cid:98) 
Revised: ϕ [Equation] [Equation] , (cid:98) V h,e ( s )

Original: [Equation]
Revised: [Equation]

Original:  max a (cid:110) (cid:98) Q h,e ( s, a ) (cid:111)(cid:17)≤ M V .
Revised: (cid:16) max a (cid:110) (cid:98) Q h,e ( s, a ) (cid:111)(cid:17)

Original: Set π eh ( s ) = argmax a (cid:98) Q h,e ( s, a ) .
Revised: [Table]

Original: Execute π e to collect a trajectory ( s eh , a eh ) H − 1 h =0 . If U NIFORM A CTIONS = T RUE , discard a eh and draw freshly sampled uniform actions independently for all h , i.e. a eh ∼ Unif( A ) .
Revised: [Table]

Original: 10: Return: uniform mixture ρ = Uniform ( { π e } Ne =1 ) .
Revised: [Table] Execute discard a sampled uniform actions - dently 10: Return

Original: Previously, Jin et al.
Revised: Previously, Jin et al.

Original: (2020b) analyzed L SVI -U CB
Revised: (2020b) analyzed L SVI -U CB

Original: under point-wise model-misspecification.
Revised: under point-wise model-misspecification.

Original: Here, we show that similar guarantees hold under a more general policy-distribution model-misspecification ε ms , captured by Assumption H.1.
Revised: Here, we show that similar guarantees hold under a more general policy-distribution model-misspecification ε ms , captured by Assumption G.1.

Original: Assumption H.1 Suppose for every h = 0 , 1 , ..., H − 1 , there exist (cid:101) µ h such that for any policy π ,
Revised: Assumption G.1 Suppose for every h = 0 , 1 , ..., H − 1 , there exist (cid:101) µ h such that for any policy π ,

Original: [Equation]
Revised: [Equation]

Original: We further assume that sup s,a,h (cid:13)(cid:13)(cid:13)(cid:101) µ h ( · ) T (cid:98) ϕ
Revised: We further assume that sup s,a,h (cid:13)(cid:13)(cid:13)(cid:101) µ h ( · ) T (cid:98) ϕ

Original: ( s, a )
Revised: ( s, a )

Original: (cid:13)(cid:13)(cid:13) TV ≤ M µ and ∥ f T (cid:101) µ h ∥ 2 ≤ M µ √ d ∥ f ∥ ∞ ∀ f :
Revised: (cid:13)(cid:13)(cid:13) TV ≤ M µ and ∥ f T (cid:101) µ h ∥ 2 ≤ M µ √ d ∥ f ∥ ∞ ∀ f :

Original: S → R , for some positive constant M µ .
Revised: S → R , for some positive constant M µ .

Original: In other words, we only need the model to be accurate on average under the occupancy distributions realizable by policies.
Revised: In other words, we only need the model to be accurate on average under the occupancy distributions realizable by policies.

Original: We also make a slight generalization on the regularization constant M µ , which is set to 1 in the original linear MDP definition (Jin et al., 2020b).
Revised: We also make a slight generalization on the regularization constant M µ , which is set to 1 in the original linear MDP definition (Jin et al., 2020b).

Original: Later, we will later instantiate the above assumption with our transferred (cid:101) µ h ( s ′ )
Revised: Later, we will later instantiate the above assumption with our transferred (cid:101) µ h ( s ′ )

Original: = (cid:80) K − 1 k =1 α k ; h ( s ′ )
Revised: = (cid:80) K − 1 k =1 α k ; h ( s ′ )

Original: (cid:98) µ k ; h ( s ′ ) , then for any s, a , we have 
Revised: (cid:98) µ k ; h ( s ′ ) , then for any s, a , we have ∥ (cid:101) µ h (cid:98)

Original: [Equation]
Revised: ϕ h ( s, a ) ∥ TV = (cid:88) s ′ (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) K − 1 (cid:88) k = α k ; h ( s ′ ) (cid:98) µ k ; h ( s ′ )

Original:  ≤ 
Revised: T (cid:98) ϕ h ( s, a ) (cid:12) (cid:12) (cid:12)(cid:12)(cid:12) ≤ (cid:88) s ′ K − 1 (cid:88) k

Original: [Equation] max [Equation]
Revised: = | α k ; h ( s ′ )

Original: by (cid:13)(cid:13)(cid:13)(cid:98) µ 
Revised: || (cid:98) µ k ; h ( s ′ )

Original: [Equation]

Original:  (cid:98) [Equation]
Revised: T (cid:98) ϕ

Original:  ( s [Equation] a 
Revised: h ( s, a ) |

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) TV ≤1 )
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: So we will set M µ = ¯ α .
Revised: So we will set M µ = ¯ α .

Original: Note that we only need the existence of (cid:101) µ h here, and (cid:101) µ h ( · ) T (cid:98)
Revised: Note that we only need the existence of (cid:101) µ h here, and (cid:101) µ h ( · ) T (cid:98)

Original: ϕ h ( s, a ) need not be a valid probability kernel.
Revised: ϕ h ( s, a ) need not be a valid probability kernel.

Original: In fact, it may even be negative valued.
Revised: In fact, it may even be negative valued.

Original: In this section, we make a model-based analysis of LSVI.
Revised: In this section, we make a model-based analysis of LSVI.

Original: Similar approaches have been used in prior works, e.g. Lykouris et al.
Revised: Similar approaches have been used in prior works, e.g. Lykouris et al.

Original: (2021); Agarwal et al.
Revised: (2021); Agarwal et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: For simplicity, we suppose that S is finite, but may be exponentially large, as we suffer no dependence on |S| .
Revised: For simplicity, we suppose that S is finite, but may be exponentially large, as we suffer no dependence on |S| .

Original: The proof can be easily extended to infinite state spaces by replacing inner products with P by integrals.
Revised: The proof can be easily extended to infinite state spaces by replacing inner products with P by integrals.

Original: Consider the following quantity,
Revised: Consider the following quantity,

Original: [Equation]
Revised: [Equation]

Original: where δ ( s ) is a one-hot encoding of the state s .
Revised: where δ ( s ) is a one-hot encoding of the state s .

Original: In words, this is the best choice for linearly (in (cid:98) ϕ
Revised: In words, this is the best choice for linearly (in (cid:98) ϕ

Original: h ( s, a ) ) predicting E s ′ ∼ P ⋆h
Revised: h ( s, a ) ) predicting E s ′ ∼ P ⋆h

Original: ( s,a )
Revised: ( s,a )

Original: [ δ ( s ′ )]
Revised: [ δ ( s ′ )]

Original: = P ⋆h ( s ′ | s, a ) .
Revised: = P ⋆h ( s ′ | s, a ) .

Original: We highlight that this is just a quantity for analysis and not computed in the algorithm.
Revised: We highlight that this is just a quantity for analysis and not computed in the algorithm.

Original: Finally, denote
Revised: Finally, denote

Original: [Equation]
Revised: [Equation]

Original: We will also sometimes use the shorthand Pf ( s, a ) for E s ′ ∼ P ( ·| s,a ) [ f ( s ′ )] .
Revised: We will also sometimes use the shorthand Pf ( s, a ) for E s ′ ∼ P ( ·| s,a ) [ f ( s ′ )] .

Original: For each h = 0 , 1 , ..., H − 1 , let V h denote the class of functions 
Revised: For each h = 0 , 1 , ..., H − 1 , let V h denote the class of functions (cid:26)

Original: [Equation]
Revised: [Equation]

Original: The motivation behind this construction is that V h satisfies the key property that all of the learned value functions (cid:98) V h,e during Algorithm 6 are captured in this class.
Revised: The motivation behind this construction is that V h satisfies the key property that all of the learned value functions (cid:98) V h,e during Algorithm 6 are captured in this class.

Original: Lemma [-H.
-] For any h = 0 , 1 , ..., H − 1 ,
Revised: Lemma [-H.
-]G.3 For any h = 0 , 1 , ..., H − 1 ,

Original: sup s (cid:12)(cid:12)(cid:12)(cid:98) V h,e ( s ) (cid:12)(cid:12)(cid:12) [Equation] M V .
Revised: sup s (cid:12)(cid:12)(cid:12)(cid:98) V h,e ( s ) (cid:12)(cid:12)(cid:12) ≤ M V .

Original: For any e = 1 , 2 , ..., N , we have (cid:98) V h,e ∈ V h . 3.
Revised: For any e = 1 , 2 , ..., N , we have (cid:98) V h,e ∈ V h . 3.

Original: ∀ f ∈ V h , we have sup s | f ( s ) | ≤ M V .
Revised: ∀ f ∈ V h , we have sup s | f ( s ) | ≤ M V .

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) w h,
Revised: where (cid:98) w h,e = Λ − 1 h,e e − 1 (cid:88) k = (cid:98)

Original: [Equation]
Revised: [Equation]

Original: k
Revised: h ( s kh

Original: [Equation]
Revised: [Equation]

Original: (cid:98) ϕ h (
Revised: a kh

Original: [Equation]
Revised: [Equation]

Original: (cid:98)
Revised: (cid:98)

Original: [Equation]

Original: +

Original: [Equation]

Original: e ( s kh +

Original: [Equation]
Revised: [Equation]

Original: From the thresholding, we have
Revised: From the thresholding, we have

Original: [Equation]
Revised: [Equation]

Original: We can bound the norm of (cid:98) w h,e as follows,
Revised: We can bound the norm of (cid:98) w h,e as follows,

Original: [Equation]
Revised: [Equation]

Original: We also required β ≤ B , and we regularized the covariance with I , so λ min is at least 1.
Revised: We also required β ≤ B , and we regularized the covariance with I , so λ min is at least 1.

Original: Hence (cid:98) V h,e satisfies all the conditions to be in V h .
Revised: Hence (cid:98) V h,e satisfies all the conditions to be in V h .

Original: Now we control the metric entropy of V h in ℓ ∞ , i.e. d ( f 1 , f 2 ) =
Revised: Now we control the metric entropy of V h in ℓ ∞ , i.e. d ( f 1 , f 2 ) =

Original: sup s | f 1 ( s ) − f 2 ( s ) | for f i ∈ V h .
Revised: sup s | f 1 ( s ) − f 2 ( s ) | for f i ∈ V h .

Original: Lemma H.4 Let ε > 0 be arbitrary and let N ε be the smallest ε -net with ℓ ∞ of V h .
Revised: Lemma G.4 Let ε > 0 be arbitrary and let N ε be the smallest ε -net with ℓ ∞ of V h .

Original: Then, [Equation]
Revised: Then, log |N ε |

Original:  d [Equation] [Equation]
Revised: ≤ d log(1 + 6 L/ε ) + log(1 + 6 B/ε ) + d 2 log(1 + 18 B 2 √ d/ε 2 ) .

Original: Proof.
Revised: Proof.

Original: Let f 1 , f 2 ∈ V h .
Revised: Let f 1 , f 2 ∈ V h .

Original: Then, 
Revised: Then, | f

Original: [Equation]
Revised: [Equation]

Revised: |

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: where we used for any a, b ≥ 0 , we have (cid:12)(cid:12)(cid:12) √ a − √ b (cid:12)(cid:12)(cid:12) = √ | a − b | √ a + √ b (cid:112) | a − b | ≤ (cid:112) | a − b | .

Original: Now proceeding like the Lemma 8.6 in the RL Theory Monograph (Agarwal et al., 2019), we have the result.
Revised: Now proceeding like the Lemma 8.6 in the RL Theory Monograph (Agarwal et al., 2019), we have the result.

Original: In this section, we’ll use the following bonus scaling parameter,
Revised: In this section, we’ll use the following bonus scaling parameter,

Original: [Equation]
Revised: [Equation]

Original: The following high probability event ( E model ) is a key step in our proof.
Revised: The following high probability event ( E model ) is a key step in our proof.

Original: Essentially, Theorem H. guarantees that, for all functions in V h , the model we learn is an accurate predictor of the expectation, up to a bonus and some vanishing terms.
Revised: Essentially, Theorem G. guarantees that, for all functions in V h , the model we learn is an accurate predictor of the expectation, up to a bonus and some vanishing terms.

Original: For all the following lemmas and theorems, suppose Assumption H.1 and the bonus scaling β is set as in (7).
Revised: For all the following lemmas and theorems, suppose Assumption G.1 and the bonus scaling β is set as in (7).

Original: Throughout the section, ζ h ( τ h ) refers to indicator functions of the trajectory τ h , where τ h =
Revised: Throughout the section, ζ h ( τ h ) refers to indicator functions of the trajectory τ h , where τ h =

Original: ( s 0 , s 1 , ..., s h ) .
Revised: ( s 0 , s 1 , ..., s h ) .

Original: As before, the expectations E π [ g ( τ h )] are with respect to the distribution of trajectories when π is executed in the environment P ⋆ .
Revised: As before, the expectations E π [ g ( τ h )] are with respect to the distribution of trajectories when π is executed in the environment P ⋆ .

Original: Theorem H.1 Let δ ∈ (0 , 1) .
Revised: Theorem G.1 Let δ ∈ (0 , 1) .

Original: Then, w.p. 1 − δ , for any time h , episode e , indicator functions ζ 1 , . .
Revised: Then, w.p. 1 − δ , for any time h , episode e , indicator functions ζ 1 , . .

Original: . , ζ H , and policy π , we have
Revised: . , ζ H , and policy π , we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Proof.
Revised: Proof.

Original: Condition on the outcome of Lemma H.5, which implies that w.p. 1 − δ , for any h, e, π, ζ h , we have
Revised: Condition on the outcome of Lemma G.5, which implies that w.p. 1 − δ , for any h, e, π, ζ h , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Also, for any h, e, π, ζ h , by Assumption H.1, we have (w.p. 1) that
Revised: Also, for any h, e, π, ζ h , by Assumption G.

Original: [Equation]
Revised: [Equation]

Revised: we have (w.p. 1) that

Original: [Equation]
Revised: [Equation]

Original: Combining these two yields the result, as
Revised: Combining these two yields the result, as

Original: [Equation]
Revised: [Equation]

Original:  Lemma H.5 Suppose Assumption H.1 and the bonus scaling β is set as in (7) .
Revised: [Table] Lemma G.5 Suppose Assumption G.1 and the bonus scaling β is set as in (7) .

Original: For any δ ∈ (0 , 1) , w.p. at least 1 − δ , we have for any time h , episode e , and policy π , 
Revised: For any δ ∈ (0 , 1) , w.p. at least 1 − δ , we have for any time h , episode e , and policy π , [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: Proof.
Revised: Proof.

Original: Consider any h, e, π .
Revised: Consider any h, e, π .

Original: Define ε kh := − δ ( s kh +1 )
Revised: Define ε kh := − δ ( s kh +1 )

Original: + P ⋆h
Revised: + P ⋆h

Original: ( s kh +1 | s kh , a kh ) , so that E [ ε kh | H k − 1 ] = 0 , where H k − 1 contains the states and actions before episode k .
Revised: ( s kh +1 | s kh , a kh ) , so that E [ ε kh | H k − 1 ] = 0 , where H k − 1 contains the states and actions before episode k .

Original: In what follows, we slightly abuse notation, as P ( s, a ) (cid:98) ϕ T ( s, a ) will denote the outer product, and hence a R S ×
Revised: In what follows, we slightly abuse notation, as P ( s, a ) (cid:98) ϕ T ( s, a ) will denote the outer product [Equation] and hence a R S ×

Original: d quantity.
Revised: d quantity

Original: [Equation]
Revised: [Equation]

Original: Rearranging, we have
Revised: Rearranging, we have

Original: [Equation]
Revised: [Equation]

Original: Now let f ∈ V h be arbitrary.
Revised: Now let f ∈ V h be arbitrary.

Original: For any s h , a h , multiply the above with (cid:98) ϕ
Revised: For any s h , a h , multiply the above with (cid:98) ϕ

Original: h ( s h , a h ) and multiply with f , we have
Revised: h ( s h , a h ) and multiply with f , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can deterministically bound Term (b) as follows,
Revised: We can deterministically bound Term (b) as follows,

Original: [Equation]
Revised: [Equation]

Original: (by Assumption H.1)
Revised: [Equation] Assumption G.1)

Original: This term will be lower order compared to the other two.
Revised: This term will be lower order compared to the other two.

Original: We now derive the bound for Term (c) for any fixed f ∈ V h .
Revised: We now derive the bound for Term (c) for any fixed f ∈ V h .

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Λ − 1 h

Revised: [Equation]

Revised: b h

Revised: [Equation]

Revised: e ( s

Revised: [Equation]

Revised: a

Revised: [Equation]

Original: Now we argue w.p. 1 − δ , for any e, h we have
Revised: Now we argue w.p. 1 − δ , for any e, h we have

Original: [Equation]
Revised: [Equation]

Original: which implies the claim about all s h , a h .
Revised: which implies the claim about all s h , a h .

Original: Indeed, we can apply Lemma H.1.
Revised: Indeed, we can apply Lemma G.1.

Original: Checking the preconditions, E P ⋆h ( s h ,a h ) (cid:2)
Revised: Checking the preconditions, E P ⋆h ( s h ,a h ) (cid:2)

Original: f T ε kh
Revised: f T ε kh

Original: | H k − 1 (cid:3) = 0 , σ ≤ | f T ε kh | ≤ ∥ f ∥ ∞ ∥
Revised: | H k − 1 (cid:3) = 0 , σ ≤ | f T ε kh | ≤ ∥ f ∥ ∞ ∥

Original: ε kh ∥ 1 ≤ 2 ∥V h ∥ ∞ , det(Σ 0 )
Revised: ε kh ∥ 1 ≤ 2 ∥V h ∥ ∞ , det(Σ 0 )

Original: = det I = 1 , and det(Σ t ) =
Revised: = det I = 1 , and det(Σ t ) =

Original: det(Λ h,e ) ≤ ( e + 1) d since the largest eigenvalue is e + 1 .
Revised: det(Λ h,e ) ≤ ( e + 1) d since the largest eigenvalue is e + 1 .

Original: So, w.p. at least 1 − δ , for all e , we have the above inequality.
Revised: So, w.p. at least 1 − δ , for all e , we have the above inequality.

Original: Thus, for any fixed f ∈ V h , w.p. 1 − δ , for all e, h we have,
Revised: Thus, for any fixed f ∈ V h , w.p. 1 − δ , for all e, h we have,

Original: [Equation]
Revised: [Equation]

Original: ≤ Term(
Revised: ≤ Term

Original: [Equation]
Revised: [Equation]

Revised: a

Revised: [Equation]

Original:  Term
Revised: + Term

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original:  Term ( c 
Revised: + Term ( c ) ≤ (cid:16) 4 ∥V h ∥ ∞ (1 + M µ ) (cid:112) log(1 /δ ) + d log( N ) + √ dN ∥V h ∥ ∞ ε ms (cid:17) b h,e ( s h , a h ) + (cid:16) ∥V h ∥ ∞ M µ √ d (cid:17) b h,e ( s h , a h ) + (cid:16) 4 ∥V h ∥ ∞ (cid:112) log(1 /δ ) + d log( N ) (cid:17) b h,e ( s h , a h ) ≲ (cid:16) √

Original: [Equation]
Revised: dN ∥V h ∥ ∞ ε ms + ∥V h ∥ ∞ M µ (cid:112) log(1 /δ ) + d log( N ) (cid:17) b h,e ( s h , a h ) .

Original: Now we apply a covering argument.
Revised: Now we apply a covering argument.

Original: Namely, union bound the above argument to every element in an ε net -net of V h .
Revised: Namely, union bound the above argument to every element in an ε net -net of V h .

Original: For any f ∈ V h , let (cid:101) f be its neighbor in the net s.t.
Revised: For any f ∈ V h , let (cid:101) f be its neighbor in the net s.t.

Original: ∥ (cid:101) f − f ∥ ∞ ≤ ε net , so we have (cid:12)(cid:12)(cid:12)(cid:16) 
Revised: ∥ (cid:101) f − f ∥ ∞ ≤ ε net , so we have (cid:12)(cid:12)(cid:12)(cid:16) (cid:98) P h,e ( s h , a h ) − (cid:101) P h ( s h , a h ) (cid:17) f

Original: [Equation]
Revised: [Equation]

Original:  a 
Revised: ≤ (cid:12)(cid:12)(cid:12)(cid:16) (cid:98) P h,e ( s h , a h ) − (cid:101) P h ( s h , a h ) (cid:17) (cid:101) f (cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:16) (cid:98) P

Original: [Equation]
Revised: [Equation]

Original: −
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: a h ) − (cid:101) P

Original: [Equation]
Revised: [Equation]

Original: + (cid:12)(cid:12)(cid:12)(cid:16) (cid:98) P h,e ( s h , a h ) − (cid:101) P h ( s h , a h ) (cid:17) ( (cid:101) f − f ) (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12)(cid:16) 
Revised: ( s h [Equation] a h ) (cid:17) ( (cid:101) f − f ) (cid:12)(cid:12)(cid:12) and (cid:12)(cid:12)(cid:12)(cid:16) (cid:98)

Revised: [Equation]

Revised: ( s

Revised: [Equation]

Revised: , a h ) − (cid:101)

Original: [Equation]
Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: ) (cid:17) (

Revised: [Equation]

Revised: [Equation]

Revised: net N.

Original: Setting ε net =
Revised: Setting ε net =

Original: N , the metric entropy is of the order d log( N ( M V + B ))
Revised: N , the metric entropy is of the order d log( N ( M V + B ))

Original: +log( BN )+
Revised: +log( BN )+

Original: d 2 log( BdN ) .
Revised: d 2 log( BdN ) .

Original: The error incurred with this epsilon net is a constant, which is lower order.
Revised: The error incurred with this epsilon net is a constant, which is lower order.

Original: Thus, we have ∀ 
Revised: Thus, we have ∀ s h ,

Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: sup f

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:16) (cid:98)

Original: [Equation]
Revised: [Equation]

Original: Note that β scales as √ log B , so one can find a valid B by solving β ≤ B for B .
Revised: Note that β scales as √ log B , so one can find a valid B by solving β ≤ B for B .

Original: Lemma H.6 Let f ∈ V h .
Revised: Lemma G.6 Let f ∈ V h .

Original: For any δ ∈ (0 , 1) , w.p. at least 1 − δ , for any time h , episode e , we have 
Revised: For any δ ∈ (0 , 1) , w.p. at least 1 − δ , for any time h , episode e , we have ∀

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First observe that
Revised: First observe that

Original: [Equation]
Revised: [Equation]

Original: e [Equation] (cid:88) k
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:98) [Equation] ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) e −
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:88) k = (cid:98) ϕ [Equation]
Revised: b

Original: [Equation]
Revised: [Equation]

Original: kh

Original: [Equation]
Revised: [Equation]

Original: a kh ) (cid:101)
Revised: where (cid:101)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Λ [Equation] 1 h,
Revised: ε k = (cid:16) P ⋆h ( s kh , a kh ) − (cid:101)

Original: [Equation]

Original: b h[Equation] ( s [Equation]
Revised: P h ( s kh

Original: [Equation]
Revised: [Equation]

Original: ) , where (cid:101) ε k = (cid:16) P ⋆h ( s kh , a kh ) − (cid:101) P h ( s kh , a kh ) (cid:17) f .
Revised: kh ) (cid:17) f .

Original: Now we will argue that w.p. 1 − δ , for all e, h , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: Now we will argue that w.p. 1 − δ , for all e, h , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: e − 1 (cid:88) k
Revised: e − 1 (cid:88) k

Original: = (cid:98) ϕ
Revised: = [Equation]

Original: h ( s kh , a kh ) (cid:101) ε k (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: N

Original: [Equation]
Revised: [Equation]

Original:  which will imply the claim for all s h , a h .
Revised: , which will imply the claim for all s h , a h .

Original: Apply self-normalized martingale concentration (Lemma H.1) to X i = (cid:98)
Revised: Apply self-normalized martingale concentration (Lemma G.1) to X i = (cid:98)

Original: ϕ h ( s ih , a ih ) and ε i = (cid:101)
Revised: ϕ h ( s ih , a ih ) and ε i = (cid:101)

Original: ε i − E
Revised: ε i − E

Original: [ (cid:101)
Revised: [ (cid:101)

Original: ε i
Revised: ε i

Original: | H i − 1 ] , where the expectation is over ( s ih , a ih ) in the definition of (cid:101) ε i .
Revised: | H i − 1 ] , where the expectation is over ( s ih , a ih ) in the definition of (cid:101) ε i .

Original: To see sub-Gaussianity, bound the envelope, | (cid:101)
Revised: To see sub-Gaussianity, bound the envelope, | (cid:101)

Original: ε k | ≤ ∥ f ∥ ∞ ∥
Revised: ε k | ≤ ∥ f ∥ ∞ ∥

Original: P ⋆ h ( s k h , a k h ) − (cid:101) µ h (cid:98) ϕ
Revised: P ⋆h ( s kh , a kh ) − (cid:101) µ h (cid:98) ϕ

Original: h ( s k h , a k h )
Revised: h ( s kh , a kh )

Original: ∥ TV ≤ ∥V h ∥ ∞ (1+ M µ ) , and thus σ ≤ | ˜ ε k | ≤ 2 ∥V h ∥ ∞ (1 + M µ ) .
Revised: ∥ TV ≤ ∥V h ∥ ∞ (1+ M µ ) , and thus σ ≤ | ˜ ε k | ≤ 2 ∥V h ∥ ∞ (1 + M µ ) .

Original: Now compute the determinants:
Revised: Now compute the determinants:

Original: det(Λ h, 0 ) = 1 and since λ max (Λ h,e )
Revised: det(Λ h, 0 ) = 1 and since λ max (Λ h,e )

Original: ≤ e + 1 , we have that log det(Λ h,e ) ≤ d log( e + 1) .
Revised: ≤ e + 1 , we have that log det(Λ h,e ) ≤ d log( e + 1) .

Original: Hence, w.p. at least 1 − δ , we have ∀
Revised: Hence, w.p. at least 1 − δ , we have ∀

Original: [Equation]
Revised: [Equation]

Original: e [Equation] By Assumption H.1 applied to π k (the data-generating policy for episode k ), we have | E [ (cid:101)
Revised: By Assumption G.1 applied to π k (the data-generating policy for episode k ), we have | E [ (cid:101)

Original: ε k | H k − 1 ] | ≤ ∥V h ∥ ∞ ε ms . Recall for any scalars c i and vectors x i , we have ∥ (cid:80)
Revised: ε k | H k − 1 ] | ≤ ∥V h ∥ ∞ ε ms . Recall for any scalars c i and vectors x i , we have ∥ (cid:80)

Original: i c i x i ∥ ≤ (cid:80) i | c i |∥ x i ∥ ≤(cid:112)(cid:80) i c
Revised: i c i x i ∥ ≤ (cid:80) i | c i |∥ x i ∥ ≤

Revised: [Equation]

Original:  2 i (cid:112)(cid:80) i ∥ x i ∥ 2 .
Revised: i c 2 i (cid:112)(cid:80) i ∥ x i ∥ 2 .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: by Lemma G.2)

Original: Combining these two bounds concludes the proof.
Revised: Combining these two bounds concludes the proof.

Original: Lemma H.7 (Optimism) Suppose ( E model ) holds.
Revised: Lemma G.7 (Optimism) Suppose ( E model ) holds.

Original: Let ι = ∥V h ∥ ∞ ε ms .
Revised: Let ι = ∥V h ∥ ∞ ε ms .

Original: Then, for any episode e = 1 , 2 , ..., N , we have
Revised: Then, for any episode e = 1 , 2 , ..., N , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Abusing notation, ζ − 1 ( · ) is the constant function 1.
Revised: Abusing notation, ζ − 1 ( · ) is the constant function 1.

Original: In particular, we have that
Revised: In particular, we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix any episode e .
Revised: Fix any episode e .

Original: We prove both claims via induction on h = H, H − 1 , H − 2 ..., 1 , 0 .
Revised: We prove both claims via induction on h = H, H − 1 , H − 2 ..., 1 , 0 .

Original: The base case holds trivially since (cid:98) V H,e and V ⋆H are zero at every state by definition.
Revised: The base case holds trivially since (cid:98) V H,e and V ⋆H are zero at every state by definition.

Original: Indeed, we have that for any π , including π ⋆ , that
Revised: Indeed, we have that for any π , including π ⋆ , that

Original: [Equation]
Revised: [Equation]

Original: Now let’s show the inductive step.
Revised: Now let’s show the inductive step.

Original: Let h ∈
Revised: Let h ∈

Original: { H − 1 , H − 2 , ..., 1 , 0 } be arbitrary and suppose the inductive hypothesis.
Revised: { H − 1 , H − 2 , ..., 1 , 0 } be arbitrary and suppose the inductive hypothesis.

Original: So suppose that V -optimism holds at h + 1 (we don’t even need Q -optimism in the future), i.e.
Revised: So suppose that V -optimism holds at h + 1 (we don’t even need Q -optimism in the future), i.e.

Original: [Equation]
Revised: [Equation]

Original: IH [Equation]
Revised: ≤ ( H − h − 1) ι (IH)

Original: Recalling that (cid:98) Q h,e ( s h , a h ) =
Revised: Recalling that (cid:98) Q h,e ( s h , a h ) =

Original: r h ( s h , a h ) + (cid:98) P h,e ( s h , a h ) (cid:98) V h
Revised: r h ( s h , a h ) + (cid:98) P h,e ( s h , a h ) (cid:98) V h

Original: +1 ,e + βb h,e
Revised: +1 ,e + βb h,e

Original: ( s h , a h ) , we have
Revised: ( s h , a h ) , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , a h ) (cid:17) (cid:98) V h +1 ,e − βb h,e ( s h , a h ) (cid:17) ζ h ( τ h ) (cid:105) + ( H − h −1) ι (by (IH)) ≤ (cid:12)(cid:12)(cid:12) E π ⋆ (cid:104)(cid:16) (cid:98) P h,e ( s h , a h )
Revised: −

Original: − P ⋆h ( s h , a h ) (cid:17) (cid:98) V h +1 ,e ζ h ( τ h ) (cid:105)(cid:12)(cid:12)(cid:12) − E π ⋆ [ βb h,e ( s h , a h ) ζ h
Revised: [Equation]

Original: ( τ h )] + ( H − h − 1) ι ≤ ι + ( H − h − 1) ι = ( H − h )
Revised: [Equation] = ( H − h )

Original: ι, (by ( E model ) and (cid:98) V h
Revised: ι, (by ( E model ) and (cid:98) V h

Original: +1 ,
Revised: + [Equation]

Original: e ∈ V h (Lemma H.3)) which proves the Q -optimism claim.
Revised: ∈ V [Equation] (Lemma G.3)) which proves the Q -optimism claim.

Original: Now let’s prove V -optimism.
Revised: Now let’s prove V -optimism

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: by Q -optimism.
Revised: by Q -optimism.

Original: Remark H.1{+G.
+} We did not require (cid:98) P h,e to be a valid transition!
Revised: Remark {+G.
+} We did not require (cid:98) P h,e to be a valid transition!

Original: It is in general unbounded and can even have negative entries!
Revised: It is in general unbounded and can even have negative entries!

Original: Lemma H.8 (Simulation) For any episode e = 1 , 2 , ..., N , we have
Revised: Lemma G.8 (Simulation) For any episode e = 1 , 2 , ..., N , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We progressively unravel the left hand side.
Revised: We progressively unravel the left hand side.

Original: For any s 0 ,
Revised: For any s 0 ,

Original: [Equation]
Revised: [Equation]

Original: where the inequality is due to the thresholding on the value function.
Revised: where the inequality is due to the thresholding on the value function.

Original: Now, perform this recursively on the P ⋆ 0 ( s 0 , π e 0 ( s 0 ))
Revised: Now, perform this recursively on the P ⋆ 0 ( s 0 , π e 0 ( s 0 ))

Original: (cid:16)(cid:98) V 1 ,e − V π e 1 (cid:17) term.
Revised: (cid:16)(cid:98) V 1 ,e − V π e 1 (cid:17) term.

Original: Doing this unravelling h times gives the result.
Revised: Doing this unravelling h times gives the result.

Original: Theorem H.2 Suppose Assumption H.1 and let β and bonus be defined as in (7) .
Revised: Theorem G.2 Suppose Assumption G.1 and let β and bonus be defined as in (7) .

Original: Let δ ∈ (0 , 1) .
Revised: Let δ ∈ (0 , 1) .

Original: Then w.p. at least 1 − δ , we have that the regret of LSVI is sublinear,
Revised: Then w.p. at least 1 − δ , we have that the regret of LSVI is sublinear,

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) O hides log dependence.
Revised: where (cid:101) O hides log dependence.

Original: Proof.
Revised: Proof.

Original: We first condition on the high-probability event ( E model ), which occurs w.p. at least 1 − δ .
Revised: We first condition on the high-probability event ( E model ), which occurs w.p. at least 1 − δ .

Original: Fix any arbitrary episode e .
Revised: Fix any arbitrary episode e .

Original: By optimism Lemma H.7 and the simulation lemma Lemma H.8, E
Revised: By optimism Lemma G.7 and the simulation lemma Lemma G.8,

Original: [Equation]
Revised: [Equation]

Original: Applying ( E model ) with no indicators, i.e. ζ h ( τ h )
Revised: Applying ( E model ) with no indicators, i.e. ζ h ( τ h )

Original: = 1 always, gives,
Revised: = 1 always, gives,

Original: [Equation]
Revised: [Equation]

Original: Now, summing over e = 1 , 2 , ..., N , we have
Revised: Now, summing over e = 1 , 2 , ..., N , we have

Original: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By Azuma’s inequality applied to the martingale difference ∆ e 
Revised: By Azuma’s inequality applied to the martingale difference ∆ e = E (cid:98) [Equation] b h [Equation]

Revised: ( s h , a h )] − b h

Original: = E (cid:98) π e [ b h,e ( s h , a h )] − b h,e ( s eh , a eh ) , which has envelope bounded by 2 , implies w.p. 1 − δ , ≤ 2 HNι 
Revised: [Equation] ( s eh [Equation] a eh ) , which has envelope bounded by 2 , implies w.p. 1 − δ , ≤ 2 HNι + 2 β

Original: [Equation]
Revised: [Equation]

Original: β
Revised: (cid:88) h [Equation] N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: It remains to bound the sum of expected bonuses.
Revised: It remains to bound the sum of expected bonuses.

Original: By Lemma H.2, we know that almost surely,
Revised: By Lemma G.2, we know that almost surely,

Original: [Equation]
Revised: [Equation]

Original: So, putting everything together,
Revised: So, putting everything together,

Original: [Equation]
Revised: [Equation]

Original: Note that Hι = H ∥V∥ ∞ ε ms = HM V ε ms is of lower order (with respect to N ), we can simply drop it.
Revised: Note that Hι = H ∥V∥ ∞ ε ms = HM V ε ms is of lower order (with respect to N ), we can simply drop it.

Original: This concludes the proof.
Revised: This concludes the proof.

Original: Corollary H.1{+G.
+} By setting δ = 1 /N , we have that expected regret also has the same rate as above.
Revised: Corollary {+G.
+} By setting δ = 1 /N , we have that expected regret also has the same rate as above.

Original: Proof.
Revised: Proof.

Original: The expected regret by law of total probability, since regret is at most NH ,
Revised: The expected regret by law of total probability, since regret is at most NH ,

Original: [Equation]
Revised: [Equation]

Original: P

Original: [Equation]

Original: [Equation]

Original: Since H is lower-order, we have the same rate.
Revised: Since H is lower-order, we have the same rate.

Original: I P ROOF OF M AIN T HEOREMS
Revised: H Proof of Main Theorems

Original: First we prove Theorem 3.1 and Theorem C.1.
Revised: First we prove Theorem 4.1 and Theorem 4.2.

Original: Theorem 3.1 (Regret under generative source access)
Revised: Theorem 4.1 (Regret under generative source access)

Original: Suppose Assumptions 2.1-2.4 and 3.1, and suppose the input policies π k are λ min -exploratory.
Revised: Suppose Assumptions 3.1-3.4 and 4.1, and suppose the input policies π k are λ min -exploratory.

Original: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EP T RANSFER when deployed in the target task has regret at most (cid:101) O (cid:16) ¯ αH 2 d 1 .
Revised: Then, for any δ ∈ (0 , 1) , w.p. 1 − δ , R EP T RANSFER when deployed in the target task has regret at most (cid:101) O (cid:16) ¯ αH 2 d 1 .

Original: 5 (cid:112) T log(1 /δ )
Revised: 5 (cid:112) T log(1 /δ )

Original: (cid:17) , with at most Kn generative accesses per source task, with n = O (cid:16) λ − 1 min
Revised: (cid:17) , with at most Kn generative accesses per source task, with n = O (cid:16) λ − 1min

Original: Aα 3max KT (cid:16) log | Φ | δ + K log | Υ | (cid:17)(cid:17) .
Revised: Aα 3max KT (cid:16) log | Φ | δ + K log | Υ | (cid:17)(cid:17) .

Original: Theorem C.1 (Regret for R EP T RANSFER with R EWARD F REE subroutine) Suppose Assumptions 2.1-2.4 and 3.1 hold, and let δ ∈ (0 , 1) .
Revised: Theorem 4.2 (Regret for R EP T RANSFER with R EWARD F REE subroutine) Suppose Assumptions 3.1-3.4 and 4.1 hold, and let δ ∈ (0 , 1) .

Original: Let { π k } K − 1 k =

Revised: Let { π k } K − 1 k =

Original: be exploratory policies learned from running R EWARD F REE on each source task with N L SVI -U CB and N R EWARD F REE set as in
Revised: be exploratory policies learned from running R EWARD F REE on each source task with N L SVI -U CB and N R EWARD F REE set as in

Original: Lemma C.1.
Revised: Lemma 4.2.

Original: Then, w.p. 1 − δ , running R EP T RANSFER with { π k } K − 1 k =1 has regret in the target task of (cid:101) O (cid:16) ¯ αH 2 d 1 .
Revised: Then, w.p. 1 − δ , running R EP T RANSFER with { π k } K − 1 k =1 has regret in the target task of (cid:101) O (cid:16) ¯ αH 2 d 1 .

Original: 5 (cid:112) T log(1 /δ )
Revised: 5 (cid:112) T log(1 /δ )

Original: (cid:17) , with at most (cid:101) O (cid:0) A 4 α 3max d 5 H 7 K 2 Tψ − 2 (log( | Φ | /δ )
Revised: (cid:17) , with at most (cid:101) O (cid:0) A 4 α 3max d 5 H 7 K 2 Tψ − 2 (log( | Φ | /δ )

Original: + K log | Υ | ) (cid:1) generative accesses per source task.
Revised: + K log | Υ | ) (cid:1) generative accesses per source task.

Original: Proof of Theorem 3.1 and Theorem C.1.
Revised: Proof of Theorem 4.1 and Theorem 4.2.

Original: For the regret bound, set M V = H and M µ = ¯ α and apply
Revised: For the regret bound, set M V = H and M µ = ¯ α and apply

Original: Theorem H.2.
Revised: Theorem G.2.

Original: This choice of M µ is valid by the argument following Assumption H.1.
Revised: This choice of M µ is valid by the argument following Assumption G.1.

Original: This gives us a regret bound of
Revised: This gives us a regret bound of

Original: [Equation]
Revised: [Equation]

Original: where ε ms can be made smaller than 1 / √ T , in which the second term dominates.
Revised: where ε ms can be made smaller than 1 / √ T , in which the second term dominates.

Original: Now, we calculate the pre-training phase sample complexity in a source task.
Revised: Now, we calculate the pre-training phase sample complexity in a source task.

Original: [Figure]

Original: solves it can First, let’s calculate the reward-free model learning sample complexity, i e. this is the number of samples required for learning (cid:98) P k .
Revised: First, let’s calculate the reward-free model learning sample complexity, i.e. this is the number of samples required for learning (cid:98) P k .

Original: Recall that we need this to be sufficiently large such that ε TV = 1 /N
Revised: Recall that we need this to be sufficiently large such that ε TV = 1 /N

Original: L SVI -U CB .
Revised: L SVI -U CB .

Original: As required by Lemma C.1, we need,
Revised: As required by Lemma 4.2, we need,

Original: [Equation]
Revised: [Equation]

Original: Second [Equation] we calculate the cross-sampling sample complexity.
Revised: Second, we calculate the cross-sampling sample complexity.

Original: Recall that n is the number of samples in each pairwise dataset.
Revised: Recall that n is the number of samples in each pairwise dataset.

Original: In order to reduce ϵ ms to 1 / √ T , by Lemma 3.1, we need
Revised: In order to reduce ϵ ms to 1 / √ T , by Lemma 4.1, we need

Original: [Equation]
Revised: [Equation]

Original: by (2)) which implies that we need
Revised: by [Equation] ) which implies that we need

Original: [Equation]
Revised: [Equation]

Original: Incorporating the coverage result from Lemma C.1 gives,
Revised: Incorporating the coverage result from Lemma 4.2 gives,

Original: [Equation]
Revised: [Equation]

Original: Since each task is in at most K − 2 pairwise datasets, each of size n , the total pre-training sample complexity per task is at most,
Revised: Since each task is in at most K − 2 pairwise datasets, each of size n , the total pre-training sample complexity per task is at most,

Original: [Equation]
Revised: [Equation]

Original: [Table] J E XPERIMENTS For the empirical study, we investigate the benefit of transfer learning under the Block MDP setting, on the challenging Rich Observation Combination Lock (comblock) benchmark. We choose this environment because one must recover the correct feature from rich observations and perform strategic exploration or otherwise pay for exponential sample complexity.
Revised: Now we prove Theorem 5.2, restated below.

Original: We include a visual overview in Fig. 1(a). This environment is uniquely challenging since it requires strategic exploration and latent state discovery at the same time, which results the failures of common deep RL methods (Misra et al., 2020), and theoretical RL approaches based on linear function approximation (and reproducing kernel Hilbert space) (Zhang et al., 2022).
Revised: Theorem 5.2 (Regret with online access)

Original: In particular, in this section we study the following questions: i) what are the benefits of representational transfer using multiple source tasks, and ii) whether generative access to source tasks is needed. We design two sets of experiments with various source and target environment configurations. We defer the details of the experiments (the design of vanilla comblock, hyperparameters, etc.) in Appendix K and include the essential design information in the following two sections.
Revised: Suppose Assumptions 3.1-3.4,5.1,5.2 hold.

Original: Baselines. We denote Source as the smallest sample complexity of L SVI -U CB using learned features from any of the source tasks; O-R EP T RANSFER as R EP T RANSFER with only online access to the source tasks; G-R EP T RANSFER as R EP T RANSFER with generative access to the source tasks; Oracle as learning in the target task with ground truth features; and Target as running B RIEE (Zhang et al., 2022) — the SOTA Block MDP algorithm, in the target task with no pretraining. J. C OMBLOCK WITHOUT PARTITIONED OBSERVATIONS
Revised: For any δ ∈ (0 , 1) , w.p.

Original: In this section we start off with an easier setting: we use 5source tasks, each with horizon H = 25 ,latent states per level and 10 actions. The latent transition dynamics are different for each source task, but notice that for comblock, the reachability assumption (c.r. Assumption 2.3) is always satisfied.
Revised: 1 − δ , Algorithm 5 with appropriately set parameters achieves a regret in the target task at most (cid:101) O (cid:16) ¯ αd 1 .

Original: The emission distribution of all the source and target tasks is identical, so Assumption ?? holds here. For the construction of the target task, for each timestep h , we choose the latent transition dynamics from one of the sources uniformly at random (thus Assumption 2.4 is satisfied). We record the number of episodes in the target environment that each method takes to solve the target environment in Table. 1(b).
Revised: 5 H 2 (cid:112) T log(1 /δ )

Original: We first observe that R EP T RANSFER with either online or generative access can solve the target task (since. Assumption ?? holds). Second, we observe that directly applying the learned feature from any single source task does not suffice to solve the target environment. This is because therepresentation learned from a single source task may collapse two latent states into a single one during encoding (e.g., if two latent states at the same time step have exactly identical latent transitions).
Revised: (cid:17) , with at most poly (cid:0) A, α max , d, H, K, T, ψ − 1 , ψ − 1 raw , log( | Φ || Υ | /δ ) (cid:1) online queries in the source tasks.

Original: See the visualization of comparisons between the learned decoders from a single source task and R EP T RANSFER in Fig. 2(a). Third, the result shows that R EP T RANSFER saves order of magnitude of target samples compared with training in the target environment from scratch using the SOTA
Revised: Proof of Theorem 5.2.

Original: Block MDPs algorithm BRIEE. This set of results verifies the empirical benefits of representation learning from multiple tasks, i.e., resolves ambiguity and speeds up downstream task learning . J. C OMBLOCK WITH PARTITIONED OBSERVATION SPACE (C OMBLOCK -PO) In this section, following the intuition of our lower bound (Theorem 3.2), we construct a setting where the supports of the emission distributions from each task are completely disjoint, while the emission distribution in the target task is a mixture of all source emissions and the latent dynamics are identical across tasks.
Revised: We follows the same format as the proof of Theorem 4.1.

Original: Hence Assumption 2.4 holds while Assumption ?? fails. So we expect that an algorithm without generative access to source tasks will fail based on Theorem 3.2. Specifically, under disjoint emission supports, a representation can decode the latent state correctly for each source, but permute latent state labels across sources, causing decoding errors on the target.
Revised: The regret bound is identical.

Original: We record the number of target episodes for each method to solve the target task in Table. 1(b). We observe that indeed the online version fails while the generative version still succeeds.
Revised: Now let’s compute the pre-training sample complexity. The regret bound requires us to set ε ms ≤ 1 / √ T . Here, our ε ms comes from Lemma F.1, so

Original: We show the visualizations of O-R EP T RANSFER and G-R EP T RANSFER in Fig. 2(b), and we note that the empirical results exactly match our theoretical results.
Revised: [Equation] which implies we need [Equation] Plugging in the coverage of Lemma 4.2,

Original: This ablation verifies that source generative model access is needed , unless one has
Revised: [Equation] Here, we only collect one dataset, so the total pre-training sample complexity is

Original: additional [Figure]
Revised: [Equation] [Table]

Original: K E XPERIMENT D ETAILS K. C ONSTRUCTION OF C OMBLOCK
Revised: I Experiment Details I.1 Construction of Comblock

Original: In this section we first introduce the vanilla Combination lock (Comblock) environment that is widely used as the benchmark for algorithms for Block MDPs.
Revised: In this section we first introduce the vanilla Combination lock (Comblock) environment that is widely used as the benchmark for algorithms for Block MDPs.

Original: We provide a visualization of the comblock environment in Fig. 1(a).
Revised: We provide a visualization of the comblock environment in Fig. 1(a).

Original: Concretely, the environment has a horizon H  and 3 latent states z i ; h , i ∈ { 0 , 1 , 2 } for each timestep h and 10 actions.
Revised: Concretely, the environment has a horizon H , and 3 latent states z i ; h , i ∈ { 0 , 1 , 2 } for each timestep h and 10 actions.

Original: Among the three latent states, we denote z 0 , zas the good states which leads to the final reward and z 2 as the bad states.
Revised: Among the three latent states, we denote z 0 , zas the good states which leads to the final reward and z 2 as the bad states.

Original: At the beginning of the task, the environment will uniformly and independent sample 1 out of the 10 actions for each good state z 0; h and z 1; h for each timestep h , and we denote these actions a 0; h , a 1; h as the optimal actions (corresponding to each latent state).
Revised: At the beginning of the task, the environment will uniformly and independent sample 1 out of the 10 actions for each good state z 0; h and z 1; h for each timestep h , and we denote these actions a 0; h , a 1; h as the optimal actions (corresponding to each latent state).

Original: These optimal actions, along with the task itself, determines the dynamics of the environment.
Revised: These optimal actions, along with the task itself, determines the dynamics of the environment.

Original: At each good latent state s 0; h or s 1; h , if the agent takes the correct action, the environment transits to the either good state at the next timestep (i.e., s 0; h +1 , s 1; h +1 ) with equal probability.
Revised: At each good latent state s 0; h or s 1; h , if the agent takes the correct action, the environment transits to the either good state at the next timestep (i.e., s 0; h +1 , s 1; h +1 ) with equal probability.

Original: Otherwise, if the agent takes any 9 of the bad actions, the environment will transition to the bad state s 2; h +1 deterministicly, and the bad states transit to only bad states at the next timestep deterministicly.
Revised: Otherwise, if the agent takes any 9 of the bad actions, the environment will transition to the bad state s 2; h +1 deterministicly, and the bad states transit to only bad states at the next timestep deterministicly.