Original: A F REE L UNCH FROM THE N OISE : P ROVABLE AND P RACTICAL E XPLORATION FOR R EPRESENTATION L EARNING
Revised: A F REE L UNCH FROM THE N OISE : P ROVABLE AND P RACTICAL E XPLORATION FOR R EPRESENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Representation learning lies at the heart of the empirical success of deep learning for dealing with the curse of dimensionality
Revised: Representation learning lies at the heart of the empirical success of deep learning for dealing with the curse of dimensionality.

Original: However, the power of representation learning has not been fully exploited yet in reinforcement learning RL) due to i) , the trade off between expressiveness and tractability; and ii) , the coupling between exploration and representation learning
Revised: However, the power of representation learning has not been fully exploited yet in reinforcement learning (RL), due to i) , the trade-off between expressiveness and tractability; and ii) , the coupling between exploration and representation learning.

Original: In this paper , we ﬁrst reveal the fact that under some noise assumption in the stochastic control model, we can ob tain the linear spectral feature of its corresponding Markov transition operator in closed- form for free .
Revised: In this paper, we ﬁrst reveal the fact that under some noise assumption in the stochastic control model, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free .

Original: Based on this observation, we propose Spectral Dynamics
Revised: Based on this observation, we propose Spectral Dynamics

Original: Embedding ( SPEDE ) , which breaks the trade off and completes optimistic explo- ration for representation learning by exploiting the structure of the noise.
Revised: Embedding (SPEDE) , which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.

Original: We provide rigorous theoretical analysis of SPEDE , and demonstrate the practical supe- rior performance over the existing state-of-the-art empirical algorithms on several benchmarks
Revised: We provide rigorous theoretical analysis of SPEDE, and demonstrate the practical superior performance over the existing state-of-the-art empirical algorithms on several benchmarks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) dedicates to solve the sequential decision making problem, where an agent is interacting with an unknown environment to ﬁnd the best policy that maximizes the expected cumulative rewards (Sutton & Barto, 2018).
Revised: Reinforcement learning (RL) dedicates to solve the sequential decision making problem, where an agent is interacting with an unknown environment to ﬁnd the best policy that maximizes the expected cumulative rewards (Sutton & Barto, 2018).

Original: It is known that the tabular algorithms direct controlling over the original state and action achieve the minimax-optimal regret depending on the cardinality of the state and action space (Jaksch et al., 2010; Osband & Van Roy, 2016; Azar et al., 2017; Jin et al., 2018).
Revised: It is known that the tabular algorithms direct controlling over the original state and action achieve the minimax-optimal regret depending on the cardinality of the state and action space (Jaksch et al., 2010; Osband & Van Roy, 2016; Azar et al., 2017; Jin et al., 2018).

Original: However, these algorithms become intractable for the real-world problems with an enormous number of states, due to the curse of dimensionality.
Revised: However, these algorithms become intractable for the real-world problems with an enormous number of states, due to the curse of dimensionality.

Original: Learning with function approximation upon good representation is a natural idea to tackle the curse and serving as the key for the success of deep learning (Bengio et al., 2013).
Revised: Learning with function approximation upon good representation is a natural idea to tackle the curse and serving as the key for the success of deep learning (Bengio et al., 2013).

Original: In fact, representation learning lies at the heart of the empirical successes of deep RL in video games (Mnih et al., 2013), robotics (Levine et al., 2016), Go (Silver et al., 2017), dialogue systems (Jiang et al., 2021) to name a few.
Revised: In fact, representation learning lies at the heart of the empirical successes of deep RL in video games (Mnih et al., 2013), robotics (Levine et al., 2016), Go (Silver et al., 2017), dialogue systems (Jiang et al., 2021) to name a few.

Original: Meanwhile, the importance and beneﬁts of the representation in RL is rigorously justiﬁed (Jin et al., 2020; Yang & Wang, 2020), which quantiﬁes the regret in terms of the dimension of the known representation based on a subclass in Markov decision processes (MDPs) (Puterman, 2014).
Revised: Meanwhile, the importance and beneﬁts of the representation in RL is rigorously justiﬁed (Jin et al., 2020; Yang & Wang, 2020), which quantiﬁes the regret in terms of the dimension of the known representation based on a subclass in Markov decision processes (MDPs) (Puterman, 2014).

Original: A natural question raises:
Revised: A natural question raises:

Original: How to design provably efﬁcient and practical algorithm for representation learning in RL?
Revised: How to design provably efﬁcient and practical algorithm for representation learning in RL?

Original: Here, by “provably efﬁcient” we mean the sample complexity of the algorithm can be rigorously characterized only in terms of the complexity of representation class, without explicit dependency on the number of states and actions, while by “practical” we mean the algorithm can be implemented and deployed for the real-world applications.
Revised: Here, by “provably efﬁcient” we mean the sample complexity of the algorithm can be rigorously characterized only in terms of the complexity of representation class, without explicit dependency on the number of states and actions, while by “practical” we mean the algorithm can be implemented and deployed for the real-world applications.

Original: Therefore, we not only require the representation learned is expressive enough for handling complex practical environments, but also require the operations in the algorithm tractable and computation/memory efﬁcient.
Revised: Therefore, we not only require the representation learned is expressive enough for handling complex practical environments, but also require the operations in the algorithm tractable and computation/memory efﬁcient.

Original: The major difﬁculty of this question lies in two-fold:
Revised: The major difﬁculty of this question lies in two-fold:

Original: i)
Revised: i)

Original: The trade-off between the expressiveness and the tractability in the design of the representations; ii)
Revised: The trade-off between the expressiveness and the tractability in the design of the representations; ii)

Original: The learning of representation is intimately coupled with exploration.
Revised: The learning of representation is intimately coupled with exploration.

Original: Speciﬁcally, a desired representation should be sufﬁciently expressive 1 to capture the dynamic system, while still tractable in practice.
Revised: Speciﬁcally, a desired representation should be sufﬁciently expressive 1 to capture the dynamic system, while still tractable in practice.

Original: However, in general, expressive representation leads to complicated optimization in learning.
Revised: However, in general, expressive representation leads to complicated optimization in learning.

Original: For example, the representation in the linear MDP is exponential  stronger than the latent variable MDPs in terms of expressiveness (Agarwal et al., 2020a).
Revised: For example, the representation in the linear MDP is exponential 1 For a formal deﬁnition of expressiveness, see (Agarwal stronger than the latent variable MDPs in terms of expressiveness (Agarwal et al., 2020a).

Original: However, its representation learning is either depending on a MLE oracle that is computational intractable due to the constraint on the regularity of conditional density (Agarwal et al., 2020a), or a complicated constrained min - max - min - max optimization (Modi et al., 2021).
Revised: However, its representation learning is either depending on a MLE oracle that is computational intractable due to the constraint on the regularity of conditional density (Agarwal et al., 2020a), or a complicated constrained min - max - min - max optimization (Modi et al., 2021).

Original: On the other hand, Misra et al.
Revised: On the other hand, Misra et al.

Original: (2020) considers the representation introduced by an encoder in block MDP (Du et al., 2019), in which the learning problem can be completed by a regression, but with the payoff that the representations in block MDP is even weaker than latent variable MDP
Revised: (2020) considers the representation introduced by an encoder in block MDP (Du et al., 2019), in which the learning problem can be completed by a regression, but with the payoff that the representations in block MDP is even weaker than latent variable MDP

Original: (Agarwal et al., 2020a).
Revised: (Agarwal et al., 2020a).

Original: Meanwhile, the coupling of the learning of representation and exploration induces the difﬁculty in practical algorithm design and analysis.
Revised: Meanwhile, the coupling of the learning of representation and exploration induces the difﬁculty in practical algorithm design and analysis.

Original: One cannot learn a precise representation without enough experiences from a comprehensive exploration, while the exploration depends on an reliable estimation of the representation.
Revised: One cannot learn a precise representation without enough experiences from a comprehensive exploration, while the exploration depends on an reliable estimation of the representation.

Original: Most of the known results depends on a policy-cover-based exploration (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a;
Revised: Most of the known results depends on a policy-cover-based exploration (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a;

Original: Modi et al., 2021), which maintains and samples a set of policies during training for systematic exploration, that signiﬁcant increases the computation and memory cost in implementation.
Revised: Modi et al., 2021), which maintains and samples a set of policies during training for systematic exploration, that signiﬁcant increases the computation and memory cost in implementation.

Original: In this work, we propose Spectral Dynamics Embedding (SPEDE), bypassing the aforementioned difﬁculties and answering the question afﬁrmatively.
Revised: In this work, we propose Spectral Dynamics Embedding (SPEDE), bypassing the aforementioned difﬁculties and answering the question afﬁrmatively.

Original: SPEDE is established on an observation that connects the stochastic control dynamics (Osband & Van Roy, 2014; Kakade et al., 2020) with linear MDPs in Section 3.
Revised: SPEDE is established on an observation that connects the stochastic control dynamics (Osband & Van Roy, 2014; Kakade et al., 2020) with linear MDPs in Section 3.

Original: Speciﬁcally, by exploiting the property of the noise in the stochastic control dynamics, we can recover the factorization of its corresponding Markov transition operator in closed-form without extra computation .
Revised: Speciﬁcally, by exploiting the property of the noise in the stochastic control dynamics, we can recover the factorization of its corresponding Markov transition operator in closed-form without extra computation .

Original: This equivalency immediately overcome the computational intractability in the model estimation in Agarwal et al.
Revised: This equivalency immediately overcome the computational intractability in the model estimation in Agarwal et al.

Original: (2020a), which breaks the trade-off between expressiveness and tractability.
Revised: (2020a), which breaks the trade-off between expressiveness and tractability.

Original: More importantly, the connection uniﬁes two highly-related but commonly-known different models, i.e. , stochastic nonlinear control models (Kakade et al., 2020) and linear MDPs (Jin et al., 2020), therefore, provides the opportunity to share beneﬁts from both sides: i) , it sheds the light on exploiting optimistic control for exploratory representation learning, instead of expensive policycover-based exploration; and ii) , it introduces the linear sufﬁcient feature from the spectral space of
Revised: More importantly, the connection uniﬁes two highly-related but commonly-known different models, i.e. , stochastic nonlinear control models (Kakade et al., 2020) and linear MDPs (Jin et al., 2020), therefore, provides the opportunity to share beneﬁts from both sides: i) , it sheds the light on exploiting optimistic control for exploratory representation learning, instead of expensive policycover-based exploration; and ii) , it introduces the linear sufﬁcient feature from the spectral space of

Original: Markov operator, in which the planning can be completed efﬁciently.
Revised: Markov operator, in which the planning can be completed efﬁciently.

Original: We rigorously characterize the statistical property of SPEDE in terms of regret w.r.t.
Revised: We rigorously characterize the statistical property of SPEDE in terms of regret w.r.t.

Original: the complexity of representation class in Section 4, without explicit dependence on the raw feature of state and action.
Revised: the complexity of representation class in Section 4, without explicit dependence on the raw feature of state and action.

Original: With the established uniﬁed view, our results generalize online control (Kakade et al., 2020) and linear MDP (Jin et al., 2020) beyond known features.
Revised: With the established uniﬁed view, our results generalize online control (Kakade et al., 2020) and linear MDP (Jin et al., 2020) beyond known features.

Original: We ﬁnally demonstrate the superior of SPEDE on the MuJoCo benchmarks in Section 5.
Revised: We ﬁnally demonstrate the superior of SPEDE on the MuJoCo benchmarks in Section 5.

Original: It signiﬁcantly outperforms the empirical stateof-the-art RL algorithms.
Revised: It signiﬁcantly outperforms the empirical stateof-the-art RL algorithms.

Original: To our knowledge, SPEDE is the ﬁrst representation learning algorithm achieving statistical, computational, and memory efﬁciency with sufﬁcient expressiveness.
Revised: To our knowledge, SPEDE is the ﬁrst representation learning algorithm achieving statistical, computational, and memory efﬁciency with sufﬁcient expressiveness.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: There have been many great attempts to learn a variety of algorithmic representation learning in
Revised: There have been many great attempts to learn a variety of algorithmic representation learning in

Original: RL designed for different purposes, e.g. , bisimulation (Ferns et al., 2004; Gelada et al., 2019; Castro, 2020), reconstruction (Watter et al., 2015; Hafner et al., 2019).
Revised: RL designed for different purposes, e.g. , bisimulation (Ferns et al., 2004; Gelada et al., 2019; Castro, 2020), reconstruction (Watter et al., 2015; Hafner et al., 2019).

Original: Recently, there are also several works considering the spectral features based on decomposing different variants of the transition operator, including successor features (Dayan, 1993;
Revised: Recently, there are also several works considering the spectral features based on decomposing different variants of the transition operator, including successor features (Dayan, 1993;

Original: Kulkarni et al., 2016), proto-value functions (Mahadevan & Maggioni, 2007; Wu et al., 2018), spectral state-aggregation (Duan et al., 2018; Zhang & Wang, 2019), and contrastive fourier features (Nachum & Yang, 2021).
Revised: Kulkarni et al., 2016), proto-value functions (Mahadevan & Maggioni, 2007; Wu et al., 2018), spectral state-aggregation (Duan et al., 2018; Zhang & Wang, 2019), and contrastive fourier features (Nachum & Yang, 2021).

Original: These works are highly-related to the proposed SPEDE.
Revised: These works are highly-related to the proposed SPEDE.

Original: Besides these features focus on state-only representation, the major difference between SPEDE and these spectral features lies in i) , the target operators in existing spectral features are state-state transition, which cancel the effect of action; ii) , the target operators are estimated based on empirical data from a ﬁxed behavior policy under the implicit assumption that the estimated operator is uniformly accurate , ignoring the major difﬁculty in exploration, while SPEDE carefully designed the systematic exploration with theoretical guarantee; iii) , most of the existing spectral features rely on explicitly decomposion of the operators, while SPEDE obtains the spectral for free .
Revised: Besides these features focus on state-only representation, the major difference between SPEDE and these spectral features lies in i) , the target operators in existing spectral features are state-state transition, which cancel the effect of action; ii) , the target operators are estimated based on empirical data from a ﬁxed behavior policy under the implicit assumption that the estimated operator is uniformly accurate , ignoring the major difﬁculty in exploration, while SPEDE carefully designed the systematic exploration with theoretical guarantee; iii) , most of the existing spectral features rely on explicitly decomposion of the operators, while SPEDE obtains the spectral for free .

Original: Turning to the rigorously-justiﬁed representation learning , a large body of effort focuses on policy-cover-based methods in an “explore-then-commit” strategy (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a; Modi et al., 2021).
Revised: Turning to the rigorously-justiﬁed representation learning , a large body of effort focuses on policy-cover-based methods in an “explore-then-commit” strategy (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a; Modi et al., 2021).

Original: These algorithms learn a uniformly accurate model through a reward-free exploration, upon which decouple the learning from the exploration.
Revised: These algorithms learn a uniformly accurate model/representation through a reward-free exploration, upon which decouple the learning from the exploration.

Original: Then, the representation from the learned model can be used for optimal policy seeking for the particular reward.
Revised: Then, the representation from the learned model can be used for optimal policy seeking for the particular reward.

Original: The major difﬁculty impedes their practical application is the computation and memory cost: the policy-cover-based exploration requires a set of exploratory polices to be maintained and sampled from during training, which can be extremely expensive.
Revised: The major difﬁculty impedes their practical application is the computa- tion and memory cost: the policy-cover-based exploration requires a set of exploratory polices to be maintained and sampled from during training, which can be extremely expensive.

Original: Another two related lines of research are model-based RL and online control , which are commonly known overlapped but separate communities considering different formulations of the dynamics.
Revised: Another two related lines of research are model-based RL and online control , which are commonly known overlapped but separate communities considering different formulations of the dynamics.

Original: Our ﬁnding establishes the equivalency among the model, so that bridges these communities.
Revised: Our ﬁnding establishes the equivalency among the model, so that bridges these communities.

Original: Osband & Van Roy (2014) and Kakade et al.
Revised: Osband & Van Roy (2014) and Kakade et al.

Original: (2020) are the most related to our work in each community.
Revised: (2020) are the most related to our work in each community.

Original: These models generalize their corresponded linear models, i.e. , Jin et al.
Revised: These models generalize their corresponded linear models, i.e. , Jin et al.

Original: and Cohen et al.
Revised: and Cohen et al.

Original: (2019), with general nonlinear model and kernel function within a known RKHS, respectively.
Revised: (2019), with general nonlinear model and kernel function within a known RKHS, respectively.

Original: The regret of the optimstic (pessimistic) algorithm has been carefully characterized for these models.
Revised: The regret of the optimstic (pessimistic) algorithm has been carefully characterized for these models.

Original: However, both of the proposed algorithms in Osband & Van Roy (2014) and Kakade et al.
Revised: However, both of the proposed algorithms in Osband & Van Roy (2014) and Kakade et al.

Original: (2020) require an planning oracle to seek the optimal policy, which might be computational intractable.
Revised: (2020) require an planning oracle to seek the optimal policy, which might be computational intractable.

Original: In SPEDE, this is easily handled in the equivalent linear MDP.
Revised: In SPEDE, this is easily handled in the equivalent linear MDP.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Markov Decision Process (MDP) is one of the most standard models studied in the reinforcement learning that can be denoted by the tuple M =
Revised: Markov Decision Process (MDP) is one of the most standard models studied in the reinforcement learning that can be denoted by the tuple M =

Original: ( S , A , r, T, µ, H ) , where S is the state space,
Revised: ( S , A , r, T, µ, H ) , where S is the state space,

Original: A is the action space, r : S ×
Revised: A is the action space, r : S ×

Original: A → R + is the reward function 2 (where
Revised: A → R + is the reward function 2 (where

Original: R + denotes the set of nonnegative real numbers), T : S ×
Revised: R + denotes the set of nonnegative real numbers), T : S ×

Original: A → ∆( S ) is the transition and µ is an initial state distribution and H is the horizon 3 (i.e. the length of each episode).
Revised: A → ∆( S ) is the transition and µ is an initial state distribution and H is the horizon 3 (i.e. the length of each episode).

Original: A (potentially non-stationary) policy π can be deﬁned as { π h } h ∈ [ H ] where π h : S → A , ∀ h ∈
Revised: A (potentially non-stationary) policy π can be deﬁned as { π h } h ∈ [ H ] where π h : S → A , ∀ h ∈

Original: [ H ] .
Revised: [ H ] .

Original: Following the standard notation, we deﬁne the value function V πh ( s h ) :
Revised: Following the standard notation, we deﬁne the value function V πh ( s h ) :

Original: = E T,π (cid:104)(cid:80) H − 1 t = h r ( s t , a t ) | s h
Revised: = E T,π (cid:104)(cid:80) H − 1 t = h r ( s t , a t ) | s h

Original: = s (cid:105) and the action-value function (i.e. the Q function) Q πh ( s h , a h ) =
Revised: = s (cid:105) and the action-value function (i.e. the Q function) Q πh ( s h , a h ) =

Original: E T,π (cid:104)(cid:80) H − 1 t = h r ( s t , a t ) | s h
Revised: E T,π (cid:104)(cid:80) H − 1 t = h r ( s t , a t ) | s h

Original: = s, a h = a (cid:105) , which are the expected cumulative rewards under transition T when executing policy π starting from s h and ( s h , a h ) .
Revised: = s, a h = a (cid:105) , which are the expected cumulative rewards under transition T when executing policy π starting from s h and ( s h , a h ) .

Original: With these two deﬁnitions at hand, it is straightforward to show the following Bellman equation:
Revised: With these two deﬁnitions at hand, it is straightforward to show the following Bellman equation:

Original: [Equation]
Revised: [Equation]

Original: Reinforcement learning aims at ﬁnding the optimal policy π ∗ =
Revised: Reinforcement learning aims at ﬁnding the optimal policy π ∗ =

Original: arg max π E s ∼ µ V π 0 ( s ) .
Revised: arg max π E s ∼ µ V π 0 ( s ) .

Original: It is well known that in the tabular setting when the state space and action space are ﬁnite, we can provably identify the optimal policy with both sample-efﬁcient and computational-efﬁcient optimism-based methods ( e.g. Azar et al., 2017; Jin et al., 2018; Zhang et al., 2021) with the complexity proportion to |S||A| .
Revised: It is well known that in the tabular setting when the state space and action space are ﬁnite, we can provably identify the optimal policy with both sample-efﬁcient and computational-efﬁcient optimism-based methods ( e.g. Azar et al., 2017; Jin et al., 2018; Zhang et al., 2021) with the complexity proportion to |S||A| .

Original: However, in practice, the cardinality of state and action space can be large or even inﬁnite.
Revised: However, in practice, the cardinality of state and action space can be large or even inﬁnite.

Original: Hence, we need to incorporate function approximation into the learning algorithm when we deal with such cases.
Revised: Hence, we need to incorporate function approximation into the learning algorithm when we deal with such cases.

Original: The linear MDP (Jin et al., 2020) or low-rank MDP (Agarwal et al., 2020a; Modi et al., 2021) is the most well-known reinforcement learning model that can incorporate linear function approximation with theoretical guarantee, thanks to the following assumption on the transition:
Revised: The linear MDP (Jin et al., 2020) or low-rank MDP (Agarwal et al., 2020a; Modi et al., 2021) is the most well-known reinforcement learning model that can incorporate linear function approximation with theoretical guarantee, thanks to the following assumption on the transition:

Original: [Equation]
Revised: [Equation]

Original: where φ : S ×
Revised: where φ : S ×

Original: A → H , µ : S → H are two feature maps and H is a Hilbert space.
Revised: A → H , µ : S → H are two feature maps and H is a Hilbert space.

Original: The most essential observation for them is that, Q πh ( s, a ) for any policy π is linear w.r.t φ
Revised: The most essential observation for them is that, Q πh ( s, a ) for any policy π is linear w.r.t φ

Original: ( s h , a h ) , due to the following observation 
Revised: ( s h , a h ) , due to the following observation (Jin et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: Therefore, φ serves as a sufﬁcient representation for the estimation of Q πh , that can provide uncertainty estimation with standard linear model analysis and eventually lead to sample-efﬁcient learning when φ is ﬁxed and known to the agent (see Theorem 3.1 in Jin et al., 2020).
Revised: Therefore, φ serves as a sufﬁcient representation for the estimation of Q πh , that can provide uncertainty estimation with standard linear model analysis and eventually lead to sample-efﬁcient learning when φ is ﬁxed and known to the agent (see Theorem 3.1 in Jin et al., 2020).

Original: However, we in general do not have such representation in advance 

Revised: However, we in general do not have such representation in advance

Original: and we need to learn the representation from the data, which constraints the applicability of the algorithms derived with ﬁxed and known representation.
Revised: and we need to learn the representation from the data, which constraints the applicability of the algorithms derived with ﬁxed and known representation.

Original:  S PECTRAL D YNAMICS E MBEDDING
Revised:  S PECTRAL D YNAMICS E MBEDDING

Original: It is naturally to consider how to perform sample-efﬁcient representation learning (and hence sample-efﬁcient reinforcement learning) that satisﬁes (1) in an online manner.
Revised: It is naturally to consider how to perform sample-efﬁcient representation learning (and hence sample-efﬁcient reinforcement learning) that satisﬁes (1) in an online manner.

Original: The most straight-
Revised: The most straight-

Original: One  MDP, where exclusive
Revised: One exception is the tabular MDP, where we can choose φ : S

Original:  2 |  correspondingly  hold.
Revised: A → R |S| 2 | | that has exclusive |S| non-zero element and µ : S → R |S| 2 |A| correspondingly deﬁned to make (1) hold.

Original: forward idea is performing the maximum likelihood estimation (MLE) in the representation space ( e.g. , Agarwal et al., 2020a).
Revised: forward idea is performing the maximum likelihood estimation (MLE) in the representation space ( e.g. , Agarwal et al., 2020a).

Original: Unfortunately, for general cases, such MLE is intractable, due to the constraints on the regularity of marginal distribution ( i.e. , (cid:104) φ ( s, a ) , (cid:82) s (cid:48) µ ( s (cid:48) )
Revised: Unfortunately, for general cases, such MLE is intractable, due to the constraints on the regularity of marginal distribution ( i.e. , (cid:104) φ ( s, a ) , (cid:82) s (cid:48) µ ( s (cid:48) )

Original: d s (cid:48) (cid:105) = 1 ) for all ( s, a ) ∈ S × A .
Revised: d s (cid:48) (cid:105) = 1 ) for all ( s, a ) ∈ S × A .

Original: Moreover, even we can perform MLE for certain cases (for example, the block
Revised: Moreover, even we can perform MLE for certain cases (for example, the block

Original: MDP), as the representation is estimated from the data, which can be inaccurate, all of the existing work apply the policy cover technique (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a;
Revised: MDP), as the representation is estimated from the data, which can be inaccurate, all of the existing work apply the policy cover technique (Du et al., 2019; Misra et al., 2020; Agarwal et al., 2020a;

Original: Modi et al., 2021) to enforce exploration.
Revised: Modi et al., 2021) to enforce exploration.

Original: However, such procedure can be both computational and memory expansive when we need amounts of exploratory policy to guarantee the coverage of whole state space, which makes it not a good practical choice.
Revised: However, such procedure can be both computational and memory expansive when we need amounts of exploratory policy to guarantee the coverage of whole state space, which makes it not a good practical choice.

Original: To overcome these issues, we introduce Spectral Dynamics Embedding (SPEDE), which leverages the noise structure to provide a simple but provable efﬁcient and practical algorithm for representation learning in RL.
Revised: To overcome these issues, we introduce Spectral Dynamics Embedding (SPEDE), which leverages the noise structure to provide a simple but provable efﬁcient and practical algorithm for representation learning in RL.

Original: We ﬁrst introduce our key observation, which provides us a practical way to implement the optimism in the face of uncertainty (OFU) principle.
Revised: We ﬁrst introduce our key observation, which provides us a practical way to implement the optimism in the face of uncertainty (OFU) principle.

Original: 3. K EY O BSERVATION
Revised: 3. K EY O BSERVATION

Original: Our most important observation is that, the density of isotropic Gaussian distribution can be expressed as the inner product of two feature maps, thanks to the reproducing property and the random
Revised: Our most important observation is that, the density of isotropic Gaussian distribution can be expressed as the inner product of two feature maps, thanks to the reproducing property and the random

Original: Fourier transform of the Gaussian kernel 5 (Rahimi & Recht, 2007):
Revised: Fourier transform of the Gaussian kernel 5 (Rahimi & Recht, 2007):

Original: [Equation]
Revised: [Equation]

Original: where k ( · , · ) is the Gaussian kernel with bandwidth σ : k ( x, y )
Revised: where k ( · , · ) is the Gaussian kernel with bandwidth σ : k ( x, y )

Original: =
Revised: =

Original: exp (cid:16) − (cid:107) x − y (cid:107) 

Revised: exp (cid:16) − (cid:107) x − y (cid:107)

Original: σ 2 (cid:17)
Revised: σ 2 (cid:17)

Original: , H is the
Revised: , H is the

Original: Reproducing Kernel Hilbert Space (RKHS) associated with k , ϕ ( x, ω, b ) = √ 2 cos( ω (cid:62) x + b ) , (cid:104) f, g (cid:105) p
Revised: Reproducing Kernel Hilbert Space (RKHS) associated with k , ϕ ( x, ω, b ) = √ 2 cos( ω (cid:62) x + b ) , (cid:104) f, g (cid:105) p

Original: = E p ( x ) [ f ( x ) g ( x )] and p ( ω, b ) =
Revised: = E p ( x ) [ f ( x ) g ( x )] and p ( ω, b ) =

Original: N ( ω ; 0 , 1 /σ 2 I ) · U ( b ; [0 , 2 π ]) with N and U denoting
Revised: N ( ω ; 0 , 1 /σ 2 I ) · U ( b ; [0 , 2 π ]) with N and U denoting

Original: Gaussian and Uniform distribution, respectively.
Revised: Gaussian and Uniform distribution, respectively.

Original: Consider the general transition dynamics,
Revised: Consider the general transition dynamics,

Original: [Equation]
Revised: [Equation]

Original: equivalently
Revised: equivalently

Original: [Equation]
Revised: [Equation]

Original: which is the basic setup in the empirical model-based reinforcement learning ( e.g. , Chua et al., 2018;
Revised: which is the basic setup in the empirical model-based reinforcement learning ( e.g. , Chua et al., 2018;

Original: Kurutach et al., 2018; Clavera et al., 2018;
Revised: Kurutach et al., 2018; Clavera et al., 2018;

Original: Wang et al., 2019), and the online (non)-linear control ( e.g. , Abbasi-Yadkori & Szepesv ´ ari, 2011; Mania et al., 2019; 2020; Simchowitz & Foster, 2020;
Revised: Wang et al., 2019), and the online (non)-linear control ( e.g. , Abbasi-Yadkori & Szepesv ´ ari, 2011; Mania et al., 2019; 2020; Simchowitz & Foster, 2020;

Original: Kakade et al., 2020).
Revised: Kakade et al., 2020).

Original: Here s ∈ R d , a ∈ A that can be continuous and f ∗ is a dynamic function.
Revised: Here s ∈ R d , a ∈ A that can be continuous and f ∗ is a dynamic function.

Original: By applying the reproducing property (3) or random Fourier transform (4) for the transition dynamics (5), we can obtain the feature φ and µ satisﬁes (1) for free .
Revised: By applying the reproducing property (3) or random Fourier transform (4) for the transition dynamics (5), we can obtain the feature φ and µ satisﬁes (1) for free .

Original: Speciﬁcally, taking the reproducing property as an example, we have that
Revised: Speciﬁcally, taking the reproducing property as an example, we have that

Original: [Equation]
Revised: [Equation]

Original: which means the problem (5) is indeed a linear MDP with φ ( s, a )
Revised: which means the problem (5) is indeed a linear MDP with φ ( s, a )

Original: =
Revised: =

Original: k ( f ∗ ( s, a ) , · ) and µ ( s (cid:48) )
Revised: k ( f ∗ ( s, a ) , · ) and µ ( s (cid:48) )

Original: = (2 πσ 2 )  d/ 2 k ( s (cid:48) , · ) .
Revised: = (2 πσ 2 ) − d/ 2 k ( s (cid:48) , · ) .

Original: Following (2), we know Q ( s, a ) is in the linear span of the φ ( s, a ) that is transformed from f ∗ ( s, a ) .
Revised: Following (2), we know Q ( s, a ) is in the linear span of the φ ( s, a ) that is transformed from f ∗ ( s, a ) .

Original: Therefore, ﬁnding a good representation of ( s, a ) is equivalent to ﬁnding a good estimation of f ∗ .
Revised: Therefore, ﬁnding a good representation of ( s, a ) is equivalent to ﬁnding a good estimation of f ∗ .

Original: In the next section, we will show that, with the well-known optimism in the face of uncertainty (OFU) principle, we can estimate f ∗ in an online manner with a both sample-efﬁcient and computational-efﬁcient algorithm.
Revised: In the next section, we will show that, with the well-known optimism in the face of uncertainty (OFU) principle, we can estimate f ∗ in an online manner with a both sampleefﬁcient and computational-efﬁcient algorithm.

Original: Remark (Computation-free Factorizable Noise Model):
Revised: Remark (Computation-free Factorizable Noise Model):

Original: We want to remark that, similar observations also hold for large amounts of distribution, e.g. , the Laplace and Cauchy distribution.
Revised: We want to remark that, similar observations also hold for large amounts of distribution, e.g. , the Laplace and Cauchy distribution.

Original: We refer the interested reader to Table 1 in Dai et al.
Revised: We refer the interested reader to Table 1 in Dai et al.

Original: (2014) for the known transformation of kernels and features.
Revised: (2014) for the known transformation of kernels and features.

Original: Here we focus on the Gaussian noise, as Gaussian distribution is the most natural continuous distribution we should work on.
Revised: Here we focus on the Gaussian noise, as Gaussian distribution is the most natural continuous distribution we should work on.

Original: 3. P RACTICAL A LGORITHM D ESCRIPTION
Revised: 3. P RACTICAL A LGORITHM D ESCRIPTION

Original: Here, we introduce a generic Thompson Sampling (TS) type algorithm in Algorithm 1 based on the
Revised: Here, we introduce a generic Thompson Sampling (TS) type algorithm in Algorithm 1 based on the

Original: OFU principle that leverage our observation at the previous section.
Revised: OFU principle that leverage our observation at the previous section.

Original: At the beginning, we provide a prior distribution P ( f ) that reﬂects our prior knowledge on f ∗ .
Revised: At the beginning, we provide a prior distribution P ( f ) that reﬂects our prior knowledge on f ∗ .

Original: Then for each episode, we draw a f from the posterior, ﬁnd the optimal policy with f using the planning algorithm, execute this policy and eventually inference the posterior with the new observation.
Revised: Then for each episode, we draw a f from the posterior, ﬁnd the optimal policy with f using the planning algorithm, execute this policy and eventually inference the posterior with the new observation.

Original: Notice that, we choose the policy optimistically with an sampled f , which enforces the exploration following the principle of OFU.
Revised: Notice that, we choose the policy optimistically with an sampled f , which enforces the exploration following the principle of OFU.

Original: Meanwhile, we only learn the representation with posterior inference, which decouples theexploration and representation learning.
Revised: Meanwhile, we only learn the dynamic with posterior inference and directly obtain the representation with Equation (3) or (4), which avoids additional error from the representation learning step.

Original: As all of our data is collected with f ∗ , our posterior will shrink to a point mass of f ∗ , which guarantees we can identify good representation and good policy with sufﬁcient number of data.
Revised: As all of our data is collected with f ∗ , our posterior will shrink to a point mass of f ∗ , which guarantees we can identify good representation and good policy with sufﬁcient number of data.

Original: One signiﬁcant part of SPEDE is the computational-efﬁcient planning with f k , thanks to the linear
Revised: One signiﬁcant part of SPEDE is the computational-efﬁcient planning with f k , thanks to the linear

Original: MDP formulation (6).
Revised: MDP formulation (6).

Original: Prior work assumes an oracle ( e.g. , Kakade et al., 2020) for such planning problem, but little is known on how to provably perform such planning efﬁciently.
Revised: Prior work assumes an oracle ( e.g. , Kakade et al., 2020) for such planning problem, but little is known on how to provably perform such planning efﬁciently.

Original: Notce that, with the feature φ ( s, a ) deﬁned via (3) and (4), we know that Q πh ( s, a ) is exactly linear in φ ( s, a ) , ∀ h, π .
Revised: Notce that, with the feature φ ( s, a ) deﬁned via (3) and (4), we know that Q πh ( s, a ) is exactly linear in φ ( s, a ) , ∀ h, π .

Original: Hence, we can perform a dynamic programming style algorithm that calculate Q h ( s, a ) with the given feature φ ( s, a ) , and then greedily select the action at each level h , which is simple yet efﬁcient.
Revised: Hence, we can perform a dynamic programming style algorithm that calculate Q h ( s, a ) with the given feature φ ( s, a ) , and then greedily select the action at each level h , which is simple yet efﬁcient.

Original: It is straightforward to show that the policy obtained with this dynamic programming algorithm is optimal with the mathematical induction.
Revised: It is straightforward to show that the policy obtained with this dynamic programming algorithm is optimal with the mathematical induction.

Original: We illustrate the detailed algorithm in Algorithm 2.
Revised: We illustrate the detailed algorithm in Algorithm 2.

Original: In such planning algorithm, we need to calculate the term (cid:82) V
Revised: In such planning algorithm, we need to calculate the term (cid:82) V

Original: h +1 ( s (cid:48) )
Revised: h +1 ( s (cid:48) )

Original: µ ( s (cid:48) )
Revised: µ ( s (cid:48) )

Original: d s (cid:48) and take the maximum of Q h ( s, a ) over a , which can be problematic when the number of states and actions can be large or even inﬁnite.
Revised: d s (cid:48) and take the maximum of Q h ( s, a ) over a , which can be problematic when the number of states and actions can be large or even inﬁnite.

Original: We will provide more discussion on this issue later.
Revised: We will provide more discussion on this issue later.

Original: We then address several practical issues in implementing the proposed SPEDE:
Revised: We then address several practical issues in implementing the proposed SPEDE:

Original: Posterior Sampling The exact posterior inference can be hard if f ∗ doesn’t lie in simple function class ( e.g. , linear function class) or has some derived property ( e.g. , conjugacy), so in practice we apply the existing mature approximate inference methods like Markov Chain Monte Carlo (MCMC) ( e.g. , Neal et al., 2011) and variational inference ( e.g. , Blei et al., 2017; Kingma & Welling, 2013).
Revised: The exact posterior inference can be hard if f ∗ doesn’t lie in simple function class ( e.g. , linear function class) or has some derived property ( e.g. , conjugacy), so in practice we apply the existing mature approximate inference methods like Markov Chain Monte Carlo (MCMC) ( e.g. , Neal et al., 2011) and variational inference ( e.g. , Blei et al., 2017; Kingma & Welling, 2013).

Original: In our implementation, we used Stochastic Gradient Langevin Dynamics (Welling & Teh, 2011;
Revised: In our implementation, we used Stochastic Gradient Langevin Dynamics (Welling & Teh, 2011;

Original: Cheng & Bartlett, 2018) to train an ensemble of models for achieving the purpose.
Revised: Cheng & Bartlett, 2018) to train an ensemble of models for achieving the purpose.

Original: Large State and Action Space
Revised: Large State and Action Space

Original: In general, we need to handle the case when the number of states and actions can be large, or even inﬁnite.
Revised: In general, we need to handle the case when the number of states and actions can be large, or even inﬁnite.

Original: Notice that, when the state space is large, we can estimate the term (cid:82) V
Revised: Notice that, when the state space is large, we can estimate the term (cid:82) V

Original: h +1 ( s (cid:48) )
Revised: h +1 ( s (cid:48) )

Original: µ ( s (cid:48) )
Revised: µ ( s (cid:48) )

Original: d s (cid:48) with regression based method using the samples from f (Ernst et al., 2005; Antos et al., 2008).
Revised: d s (cid:48) with regression based method using the samples from f (Ernst et al., 2005; Antos et al., 2008).

Original: For the continuous action space, we can apply principled policy optimization methods ( e.g. , Agarwal et al., 2020b) with an energy-based model (EBM) parametrized policy (Nachum et al., 2017; Dai et al., 2018), treat the linear Q π ( s, a ) as the gradient and perform mirror descent and eventually obtain the optimal policy.
Revised: For the continuous action space, we can apply principled policy optimization methods ( e.g. , Agarwal et al., 2020b) with an energy-based model (EBM) parametrized policy (Nachum et al., 2017; Dai et al., 2018), treat the linear Q π ( s, a ) as the gradient and perform mirror descent and eventually obtain the optimal policy.

Original: However, this is at the cost of an additional sampling step from the EBM policy.
Revised: However, this is at the cost of an additional sampling step from the EBM policy.

Original: In practice, we introduce a Gaussian policy and perform soft actor-critic (Haarnoja et al., 2018) policy update, which already provides good empirical perfor- mance.
Revised: In practice, we introduce a Gaussian policy and perform soft actor-critic (Haarnoja et al., 2018) policy update, which already provides good empirical perfor- mance.

Original: To sum up, for large state and action cases, we learn the critic in the learned representation space by regression, and obtain the Gaussian parametrized actor with SAC policy update step, in
Revised: To sum up, for large state and action cases, we learn the critic in the learned representation space by regression, and obtain the Gaussian parametrized actor with SAC policy update step, in

Original: Line 3 and 4 in Algorithm 2, respectively.
Revised: Line 3 and 4 in Algorithm 2, respectively.

Original: Inﬁnite Horizon Case Our algorithm can be provably extended to the inﬁnite horizon case with speciﬁc termination condition for each episode ( e.g. , see Jaksch et al., 2010).
Revised: Inﬁnite Horizon Case Our algorithm can be provably extended to the inﬁnite horizon case with speciﬁc termination condition for each episode ( e.g. , see Jaksch et al., 2010).

Original: And in practice, for the planning part we can solve the linear ﬁxed-point equation with the feature φ ( s, a ) using the popular algorithms like Fitted Q -iteration (FQI)
Revised: And in practice, for the planning part we can solve the linear ﬁxed-point equation with the feature φ ( s, a ) using the popular algorithms like Fitted Q -iteration (FQI)

Original: (Ernst et al., 2005; Antos et al., 2008).
Revised: (Ernst et al., 2005; Antos et al., 2008).

Original: that still guarantees to ﬁnd the optimal policy.
Revised: that still guarantees to ﬁnd the optimal policy.

Original:  T HEORETICAL G UARANTEES
Revised:  T HEORETICAL G UARANTEES

Original: In this section, we provide theoretical results for SPEDE, showing that SPEDE can identify informative representation and as a result, near-optimal policy in a sample-efﬁcient way.
Revised: In this section, we provide theoretical results for SPEDE, showing that SPEDE can identify informative representation and as a result, near-optimal policy in a sample-efﬁcient way.

Original: We ﬁrst deﬁne the notation of regret.
Revised: We ﬁrst deﬁne the notation of regret.

Original: Assume at episode k , the learner chooses the policy π k and observes a sequence { ( s kh , a kh ) } H − 1 h =0 .
Revised: Assume at episode k , the learner chooses the policy π k and observes a sequence { ( s kh , a kh ) } H − 1 h =0 .

Original: We deﬁne the regret of the ﬁrst K episodes (and deﬁne T :=
Revised: We deﬁne the regret of the ﬁrst K episodes (and deﬁne T :=

Original: KH )
Revised: KH )

Original: as:
Revised: as:

Original: [Equation]
Revised: [Equation]

Original: We want to provide a regret upper bound that is sublinear in T , as when T increases, we collect more data that can help us build a much more accurate estimation on the representation, which should decrease the per-step regret and make the overall regret scale sublinear in T .
Revised: We want to provide a regret upper bound that is sublinear in T , as when T increases, we collect more data that can help us build a much more accurate estimation on the representation, which should decrease the per-step regret and make the overall regret scale sublinear in T .

Original: As we consider the Thompson Sampling algorithm, we would like to study the expected regret E P ( f ) [Regret( K )] , which takes the prior P ( f ) into account.
Revised: As we consider the Thompson Sampling algorithm, we would like to study the expected regret E P ( f ) [Regret( K )] , which takes the prior P ( f ) into account.

Original: 4. A SSUMPTIONS
Revised: 4. A SSUMPTIONS

Original: Before we start, we ﬁrst state the assumptions we use to derive our theoretical results.
Revised: Before we start, we ﬁrst state the assumptions we use to derive our theoretical results.

Original: Assumptions on the environment We make the following assumption on the environment, which is common in the literature ( e.g. Azar et al., 2017; Jin et al., 2018; 2020).
Revised: Assumptions on the environment We make the following assumption on the environment, which is common in the literature ( e.g. Azar et al., 2017; Jin et al., 2018; 2020).

Original: Assumption 1 (Bounded Reward) .
Revised: Assumption 1 (Bounded Reward) .

Original: r ( s, a ) ∈ [0 , 1] , ∀ ( s, a ) ∈ S × A .
Revised: r ( s, a ) ∈ [0 , 1] , ∀ ( s, a ) ∈ S × A .

Original: Assumptions on the function class F
Revised: Assumptions on the function class F

Original: In practice, we generally approximate f ∗ with some complicated function approximators, so we focus on the setting where we want to ﬁnd f ∗ from a general function class F that need not to be linear with certain feature map.
Revised: In practice, we generally approximate f ∗ with some complicated function approximators, so we focus on the setting where we want to ﬁnd f ∗ from a general function class F that need not to be linear with certain feature map.

Revised: This can be helpful for MuJoCo benchmarks that have angle, angular velocity and torque of the agent in the raw state, which we don’t know how to construct the feature map to make f linear.

Original: We ﬁrst state some necessary deﬁnitions and assumptions on F .
Revised: We ﬁrst state some necessary deﬁnitions and assumptions on F .

Original: Deﬁnition 1 ( (cid:96) 2 -norm of functions) .
Revised: Deﬁnition 1 ( (cid:96) 2 -norm of functions) .

Original: Deﬁne (cid:107) f
Revised: Deﬁne (cid:107) f

Original: (cid:107) 2 :=
Revised: (cid:107) 2 :=

Original: max ( s,a ) ∈S×A (cid:107) f
Revised: max ( s,a ) ∈S×A (cid:107) f

Original: ( s, a ) (cid:107) 2 .
Revised: ( s, a ) (cid:107) 2 .

Original: Notice that it is not the commonly used (cid:96) 2 norm for the function, but it suits our purpose well.
Revised: Notice that it is not the commonly used (cid:96) 2 norm for the function, but it suits our purpose well.

Original: Assumption 2 (Bounded Output) .
Revised: Assumption 2 (Bounded Output) .

Original: We assume that (cid:107) f (cid:107) 2 ≤ C , ∀ f ∈ F .
Revised: We assume that (cid:107) f (cid:107) 2 ≤ C , ∀ f ∈ F .

Original: Assumption 3 (Realizability) .
Revised: Assumption 3 (Realizability) .

Original: We assume the ground truth dynamic function f ∗ ∈ F .
Revised: We assume the ground truth dynamic function f ∗ ∈ F .

Original: We then deﬁne the notion of covering number, which will be helpful in our algorithm derivation.
Revised: We then deﬁne the notion of covering number, which will be helpful in our algorithm derivation.

Original: Deﬁnition 2 (Covering Number (Wainwright, 2019)) .
Revised: Deﬁnition 2 (Covering Number (Wainwright, 2019)) .

Original: An (cid:15) -cover of F with respect to a metric ρ is a set
Revised: An (cid:15) -cover of F with respect to a metric ρ is a set

Original: { f i } i ∈ [ n ] ⊆ F , such that ∀ f ∈ F , there exists i ∈ [ n ] , ρ ( f, f i ) ≤ (cid:15) .
Revised: { f i } i ∈ [ n ] ⊆ F , such that ∀ f ∈ F , there exists i ∈ [ n ] , ρ ( f, f i ) ≤ (cid:15) .

Original: The (cid:15) -covering number is the cardinality of the smallest (cid:15) -cover, denoted as N ( F , (cid:15), ρ ) .
Revised: The (cid:15) -covering number is the cardinality of the smallest (cid:15) -cover, denoted as N ( F , (cid:15), ρ ) .

Original: Assumption 4 (Bounded Covering Number) .
Revised: Assumption 4 (Bounded Covering Number) .

Original: We assume that N ( F , (cid:15), (cid:107) · (cid:107) 2 ) < ∞ , ∀ (cid:15) > 0 .
Revised: We assume that N ( F , (cid:15), (cid:107) · (cid:107) 2 ) < ∞ , ∀ (cid:15) > 0 .

Original: Remark Basically, Assumption 2 means the the transition dynamic never pushes the state far from the origin, which holds widely in practice.
Revised: Remark Basically, Assumption 2 means the the transition dynamic never pushes the state far from the origin, which holds widely in practice.

Original: Assumption 3 guarantees that we can ﬁnd the exact f ∗ in
Revised: Assumption 3 guarantees that we can ﬁnd the exact f ∗ in

Original: F , or we will always suffer from the error induced by model mismatch.
Revised: F , or we will always suffer from the error induced by model mismatch.

Original: Assumption 4 ensures that we can estimate f ∗ with small error when we have sufﬁcient number of observations.
Revised: Assumption 4 ensures that we can estimate f ∗ with small error when we have sufﬁcient number of observations.

Original: Besides the bounded covering number, we also need an additional assumption on bounded eluder dimension, which is deﬁned in the following:
Revised: Besides the bounded covering number, we also need an additional assumption on bounded eluder dimension, which is deﬁned in the following:

Original: Deﬁnition 3 ( (cid:15) -dependency
Revised: Deﬁnition 3 ( (cid:15) -dependency

Original: (Osband & Van Roy, 2014)) .
Revised: (Osband & Van Roy, 2014)) .

Original: A state-action pair ( s, a ) ∈ S ×
Revised: A state-action pair ( s, a ) ∈ S ×

Original: A is (cid:15) -dependent on { ( s i , a i ) } i ∈
Revised: A is (cid:15) -dependent on { ( s i , a i ) } i ∈

Original: [ n ] ⊆ S × A with respect to F , if ∀ f, ˜ f ∈ F satisfying (cid:113)(cid:80) i ∈
Revised: [ n ] ⊆ S × A with respect to F , if ∀ f, ˜ f ∈ F satisfying (cid:113)(cid:80) i ∈

Original: [ n ] (cid:107) f
Revised: [ n ] (cid:107) f

Original: ( s i , a i ) − ˜ f
Revised: ( s i , a i ) − ˜ f

Original: ( s i , a i ) (cid:107) 22 ≤ (cid:15) satisﬁes that (cid:107) f
Revised: ( s i , a i ) (cid:107) 22 ≤ (cid:15) satisﬁes that (cid:107) f

Original: ( s, a ) − ˜ f
Revised: ( s, a ) − ˜ f

Original: ( s, a ) (cid:107) 2 ≤ (cid:15) .
Revised: ( s, a ) (cid:107) 2 ≤ (cid:15) .

Original: Furthermore, ( s, a ) is said to be (cid:15) -independent of { ( s i , a i ) } i ∈ [ n ] with respect to F if it is not (cid:15) -dependent on { ( s i , a i ) } i ∈ [ n ] .
Revised: Furthermore, ( s, a ) is said to be (cid:15) -independent of { ( s i , a i ) } i ∈ [ n ] with respect to F if it is not (cid:15) -dependent on { ( s i , a i ) } i ∈ [ n ] .

Original: Deﬁnition 4 (Eluder Dimension (Osband & Van Roy, 2014)) .
Revised: Deﬁnition 4 (Eluder Dimension (Osband & Van Roy, 2014)) .

Original: We deﬁne the eluder dimension dim E ( F , (cid:15) ) as the length d of the longest sequence of elements in S × A , such that ∃ (cid:15) (cid:48) ≥ (cid:15) , every element is (cid:15) (cid:48) -independent of its predecessors.
Revised: We deﬁne the eluder dimension dim E ( F , (cid:15) ) as the length d of the longest sequence of elements in S × A , such that ∃ (cid:15) (cid:48) ≥ (cid:15) , every element is (cid:15) (cid:48) -independent of its predecessors.

Original: Remark Intuitively, eluder dimension illustrates the number of samples we need to make our prediction on unseen data accurate.
Revised: Remark Intuitively, eluder dimension illustrates the number of samples we need to make our prediction on unseen data accurate.

Original: If the eluder dimension is unbounded, then we cannot make any meaningful prediction on unseen data, even we have large amounts of samples.
Revised: If the eluder dimension is unbounded, then we cannot make any meaningful prediction on unseen data, even we have large amounts of samples.

Original: Hence, to make the learning possible, we need the following bounded eluder dimension assumption.
Revised: Hence, to make the learning possible, we need the following bounded eluder dimension assumption.

Original: Assumption 5 (Bounded Eluder Dimension) .
Revised: Assumption 5 (Bounded Eluder Dimension) .

Original: We assume dim E ( F , (cid:15) ) < ∞ , ∀ (cid:15) > 0 .
Revised: We assume dim E ( F , (cid:15) ) < ∞ , ∀ (cid:15) > 0 .

Original: 4. M AIN R ESULT
Revised: 4. M AIN R ESULT

Original: Theorem 5 (Regret Bound) .
Revised: Theorem 5 (Regret Bound) .

Original: Assume Assumption 2 to 5 holds.
Revised: Assume Assumption 2 to 5 holds.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: where ˜ O represents the order up to logarithm factors.
Revised: where ˜ O represents the order up to logarithm factors.

Original: In general, log N ( F ,  1 /T, (cid:107) · (cid:107) 2 ) and dim E ( F,  1 /T )) should be scaled like polylog( T ) , hence our upper bound is sublinear in T .
Revised: For ﬁnite dimensional function class, log N ( F , T − 1 / 2 , (cid:107) · (cid:107) 2 ) and dim E ( F, T − 1 / 2 )) should be scaled like polylog( T ) , hence our upper bound is sublinear in T .

Original: The proof is in Appendix C.
Revised: The proof is in Appendix C.

Original: Here we brieﬂy sketch the proof idea.
Revised: Here we brieﬂy sketch the proof idea.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We ﬁrst construct an equivalent UCB algorithm (see Appendix B) and bound
Revised: We ﬁrst construct an equivalent UCB algorithm (see Appendix B) and bound

Original: Regret( K ) for it.
Revised: Regret( K ) for it.

Original: Then by the conclusion from Russo & Van Roy (2013; 2014); Osband & Van Roy (2014), we can directly translate the upper bound on Regret( K ) from UCB algorithm to an upper bound on E P ( f )
Revised: Then by the conclusion from Russo & Van Roy (2013; 2014); Osband & Van Roy (2014), we can directly translate the upper bound on Regret( K ) from UCB algorithm to an upper bound on E P ( f )

Original: [Regret( K )] of TS algorithm.
Revised: [Regret( K )] of TS algorithm.

Original: With the optimism, we know for episode k , V ∗ 0 ( s k 0 ) ≤ ˜ V π k 0 ,k
Revised: With the optimism, we know for episode k , V ∗ 0 ( s k 0 ) ≤ ˜ V π k 0 ,k

Original: ( s k 0 ) , where ˜ V π k h,k is the value function of policy π k under the model
Revised: ( s k 0 ) , where ˜ V π k h,k is the value function of policy π k under the model

Original: ˜ f k introduced in the UCB algorithm.
Revised: ˜ f k introduced in the UCB algorithm.

Original: Hence, the regret at episode k can be bounded by ˜
Revised: Hence, the regret at episode k can be bounded by ˜

Original: V π k 0 ,k ( s k 0 )
Revised: V π k 0 ,k ( s k 0 )

Original: − V π k 0 ( s k 0 ) , which is the value difference of the policy π k under the two models ˜ f k and f ∗ , that can be bounded by
Revised: − V π k 0 ( s k 0 ) , which is the value difference of the policy π k under the two models ˜ f k and f ∗ , that can be bounded by

Original: [Equation]
Revised: [Equation]

Original: Lemma 13 for the details), which means when the estimated model ˆ f is close to the real model f ∗ , the policy obtained by planning on ˆ f will only suffer from a small regret.
Revised: 14 for the details), which means when the estimated model ˆ f is close to the real model f ∗ , the policy obtained by planning on ˆ f will only suffer from a small regret.

Original: With Cauchy-Schwartz inequality, we only need to bound E (cid:104)(cid:80)
Revised: With Cauchy-Schwartz inequality, we only need to bound E (cid:104)(cid:80)

Original: k ∈
Revised: k ∈

Original: [ K ] (cid:80) H − 1 h =0 (cid:107) f ∗ ( s kh , a kh ) − ˜ f
Revised: [ K ] (cid:80) H − 1 h =0 (cid:107) f ∗ ( s kh , a kh ) − ˜ f

Original: k ( s kh , a kh ) (cid:107) 22 (cid:105) .
Revised: k ( s kh , a kh ) (cid:107) 22 (cid:105) .

Original: This term can be handled via Lemma 17.
Revised: This term can be handled via Lemma 18.

Original: With some additional technical steps, we can obtain the upper bound on Regret( K ) for the UCB algorithm, and hence the upper bound on E P ( f ) [Regret( K )] for the TS algorithm.
Revised: With some additional technical steps, we can obtain the upper bound on Regret( K ) for the UCB algorithm, and hence the upper bound on E P ( f ) [Regret( K )] for the TS algorithm.

Original: Kernelized Non-linear Regulator Notice that, for the linear function class F = { θ (cid:62) ϕ
Revised: Kernelized Non-linear Regulator Notice that, for the linear function class F = { θ (cid:62) ϕ

Original: ( s, a ) : θ ∈ R d
Revised: ( s, a ) : θ ∈ R d

Original: ϕ × d } where ϕ : S ×
Revised: ϕ × d } where ϕ : S ×

Original: A → R d ϕ is a ﬁxed and known feature map of certain RKHS 6 , when the feature and the parameters are bounded, the logarithm covering number can be bounded by log N ( F , (cid:15), (cid:107) · (cid:107) 2 ) (cid:46)
Revised: A → R d ϕ is a ﬁxed and known feature map of certain RKHS 6 , when the feature and the parameters are bounded, the logarithm covering number can be bounded by log N ( F , (cid:15), (cid:107) · (cid:107) 2 ) (cid:46)

Original: d ϕ log(1 /(cid:15) ) , and the eluder dimension can be bounded by dim E ( F , (cid:15) )
Revised: d ϕ log(1 /(cid:15) ) , and the eluder dimension can be bounded by dim E ( F , (cid:15) )

Original: (cid:46)
Revised: (cid:46)

Original: d ϕ log(1 /(cid:15) )
Revised: d ϕ log(1 /(cid:15) )

Original: (see Appendix D for the detail, notice that we provide a tighter bound of the eluder dimension compared with the one derived in Osband & Van Roy (2014)).
Revised: (see Appendix D for the detail, notice that we provide a tighter bound of the eluder dimension compared with the one derived in Osband & Van Roy (2014)).

Original: Hence, for linear function class, Theorem 5 can be translated into a regret upper bound of ˜ O
Revised: Hence, for linear function class, Theorem 5 can be translated into a regret upper bound of ˜ O

Original: ( Hd ϕ T 1 / 2 ) for sufﬁcient large
Revised: ( Hd ϕ T 1 / 2 ) for sufﬁcient large

Original: T , that matches the results of Kakade et al.
Revised: T , that matches the results of Kakade et al.

Original: Moreover, for the case of linear bandits when
Revised: Moreover, for the case of linear bandits when

Original: H = 1 , our bound can be translated into a regret upper bound of ˜ O
Revised: H = 1 , our bound can be translated into a regret upper bound of ˜ O

Original: ( d ϕ T 1 / 2 ) , that matches the lower bound (Dani et al., 2008) up to logarithmic terms.
Revised: ( d ϕ T 1 / 2 ) , that matches the lower bound (Dani et al., 2008) up to logarithmic terms.

Original: Compared with Kakade et al.
Revised: Compared with Kakade et al.

Original: and Osband & Van Roy (2014)
Revised: and Osband & Van Roy (2014)

Original: Our results have some connections with the results from Kakade et al.
Revised: Our results have some connections with the results from Kakade et al.

Original: and Osband & Van Roy (2014).
Revised: and Osband & Van Roy (2014).

Original: However, in Kakade et al.
Revised: However, in Kakade et al.

Original: , the authors only considers the case when F only contains linear functions w.r.t some known feature map, which constrains its application in practice.
Revised: , the authors only considers the case when F only contains linear functions w.r.t some known feature map, which constrains its application in practice.

Original: We instead, consider the general function approximation, which makes our algorithm applicable for more complicated models like deep neural networks.
Revised: We instead, consider the general function approximation, which makes our algorithm applicable for more complicated models like deep neural networks.

Original: Meanwhile, the regret bound from Osband & Van Roy (2014) depends on a global Lipschitz constant for the value function, which can be hard to quantify with either theoretical or empirical method.
Revised: Meanwhile, the regret bound from Osband & Van Roy (2014) depends on a global Lipschitz constant for the value function, which can be hard to quantify with either theoretical or empirical method.

Original: Instead, our regret bound gets rid of such dependency on the
Revised: Instead, our regret bound gets rid of such dependency on the

Original: Lipschitz constant with the simulation lemma that carefully exploit the noise structure.
Revised: Lipschitz constant with the simulation lemma that carefully exploit the noise structure.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we study the empirical performance of SPEDE in the OpenAI MuJoCo control suite (Brockman et al., 2016).
Revised: In this section, we study the empirical performance of SPEDE in the OpenAI MuJoCo control suite (Brockman et al., 2016).

Original: We use the environments from MBBL (Wang et al., 2019), which varies slightly from the original environments in terms of modifying the reward function so its gradient w.r.t.
Revised: We use the environments from MBBL (Wang et al., 2019), which varies slightly from the original environments in terms of modifying the reward function so its gradient w.r.t.

Original: the states exists and introducing early termination (ET).
Revised: the states exists and introducing early termination (ET).

Original: Note that the set of environments contains various control and manipulation tasks, which are commonly used for benchmarking both model-free and model-based RL algorithms (Yu et al., 2020; Kakade et al., 2020;
Revised: Note that the set of environments contains various control and manipulation tasks, which are commonly used for benchmarking both model-free and model-based RL algorithms (Yu et al., 2020; Kakade et al., 2020;

Original: Luo et al., 2018;
Revised: Luo et al., 2018;

Original: Haarnoja et al., 2018).
Revised: Haarnoja et al., 2018).

Original: As aforementioned, for practical implementation, our critic network consists of a representation network φ ( · ) and a linear layer on the top.
Revised: As aforementioned, for practical implementation, our critic network consists of a representation network φ ( · ) and a linear layer on the top.

Original: We follow the same procedure of
Revised: We follow the same procedure of

Original: Algorithm 1: (1) For ﬁnding the optimal policy, we run an actor-critic algorithm (SAC).
Revised: Algorithm 1: (1) For ﬁnding the optimal policy, we run an actor-critic algorithm (SAC).

Original: (2) We update the representation network of the critic function φ ( · ) with a momentum from the model dynamics network f ( · ) .
Revised: (2) We update the representation network of the critic function φ ( · ) with a momentum from the model dynamics network f ( · ) .

Original: We found this prevent updating the representation of too fast.
Revised: We found this prevent updating the representation of too fast.

Original: We provide the full set of experiments in Appendix E.2, an ablation on the momentum in Appendix E.3 and the hyperparameter we use in Appendix E.5.
Revised: We provide the full set of experiments in Appendix E.2, an ablation on the momentum in Appendix E.3 and the hyperparameter we use in Appendix E.6.

Original: Baselines We compare our method with various model-based RL baselines: PETS (Chua et al., 2018) with random shooting (RS) optimizer, PETS with cross entropy method (CEM) optimizer and
Revised: Baselines We compare our method with various model-based RL baselines: PETS (Chua et al., 2018) with random shooting (RS) optimizer, PETS with cross entropy method (CEM) optimizer and

Original: ME-TRPO (Kurutach et al., 2018).
Revised: ME-TRPO (Kurutach et al., 2018).

Original: Note that these are strong empirical baselines with many handtuned hyperparameters and engineering features ( e.g. , ensemble of models).
Revised: Note that these are strong empirical baselines with many handtuned hyperparameters and engineering features ( e.g. , ensemble of models).

Original: It is usually hard for any theoretically guaranteed model-based RL algorithm to match or surpass their performance (Kakade et al., 2020).
Revised: It is usually hard for any theoretically guaranteed model-based RL algorithm to match or surpass their performance (Kakade et al., 2020).

Original: Another natural baseline is the successor feature (Dayan, 1993), which is one of the representative spectral features.
Revised: Another natural baseline is the successor feature (Dayan, 1993), which is one of the representative spectral features.

Original: We compare with the deep successor feature (DeepSF)
Revised: We compare with the deep successor feature (DeepSF)

Original: (Kulkarni et al., 2016), and for a fair comparison, we only swap the representation objective of SPEDE with
Revised: (Kulkarni et al., 2016), and for a fair comparison, we only swap the representation objective of SPEDE with

Original: DeepSF and keep the other parts of the algorithm exactly the same.
Revised: DeepSF and keep the other parts of the algorithm exactly the same.

Original: SPEDE:
Revised: SPEDE:

Original: Performance with the Learned Representation Following Algorithm 1, we are interested in how SPEDE performs when we conduct planning on top of the representation induced by the dynamics model in each episode.
Revised: Performance with the Learned Representation Following Algorithm 1, we are interested in how SPEDE performs when we conduct planning on top of the representation induced by the dynamics model in each episode.

Original: As most of the rigorously-justiﬁed representation learning algorithms are computationally intractable/inefﬁcient, to demonstrate the effectiveness of representation used in SPEDE, we compare SPEDE with the deep successor features, which is one representative empirical representation learning algorithm.
Revised: As most of the rigorously-justiﬁed representation learning algorithms are computationally intractable/inefﬁcient, to demonstrate the effectiveness of representation used in SPEDE, we compare SPEDE with the deep successor features, which is one representative empirical representation learning algorithm.

Original: Moreover, as our method essentially models transition dynamics, we compare our methods with the state-of-the-art model-based RL algorithms.
Revised: Moreover, as our method essentially models transition dynamics, we compare our methods with the state-of-the-art model-based RL algorithms.

Original: We summarize the results of our method in Table 2.
Revised: We summarize the results of our method in Table 2.

Original: We see that our method achieves impressive performance comparing to model-based RL methods.
Revised: We see that our method achieves impressive performance comparing to model-based RL methods.

Original: Even in some hard environments that baselines fail to reach positive reward ( e.g. , MountainCar, Walker-ET), SPEDE manage to achieve a reward of 52.6 and 501.6 respectively.
Revised: Even in some hard environments that baselines fail to reach positive reward ( e.g. , MountainCar, Walker-ET), SPEDE manage to achieve a reward of 52.6 and 501.6 respectively.

Original: We also evaluate our representation by comparing SPEDE to the usage of deep successor feature (DeepSF).
Revised: We also evaluate our representation by comparing SPEDE to the usage of deep successor feature (DeepSF).

Original: Results show that on hard tasks like Humanoid and Walker,
Revised: Results show that on hard tasks like Humanoid and Walker,

Original: SPEDE manages to achieve 452.6 and 336.0 higher reward respectively.
Revised: SPEDE manages to achieve 452.6 and 336.0 higher reward respectively.

Original: SPEDE-REG: Policy Optimization with SPEDE Representation Regularizer
Revised: SPEDE-REG: Policy Optimization with SPEDE Representation Regularizer

Original: In order to evaluate whether our assumption on linear MDP is valid in empirical settings and study whether such assumption can help improve the performance, we add our model dynamics representation objective as a regularizer in addition to the original SAC algorithm for learning the Q -function.
Revised: In order to evaluate whether our assumption on linear MDP is valid in empirical settings and study whether such assumption can help improve the performance, we add our model dynamics representation objective as a regularizer in addition to the original SAC algorithm for learning the Q -function.

Original: Speciﬁcally, the algorithm SPEDE-REG consists of vanilla SAC objective with an additional loss putting constraints on the representation learned by the critic function, due to the intuition that the representation should satisfy the linear MDP dynamics.
Revised: Speciﬁcally, the algorithm SPEDE-REG consists of vanilla SAC objective with an additional loss putting constraints on the representation learned by the critic function, due to the intuition that the representation should satisfy the linear MDP dynamics.

Original: We compare its performance with the vanilla SAC algorithm to show the beneﬁts of dynamic representation.
Revised: We compare its performance with the vanilla SAC algorithm to show the beneﬁts of dynamic representation.

Original: Results in Table 2 show that adding such constraint signiﬁcantly improve the performance of SAC: on hard tasks like Hopper-ET, S-Humanoid-ET and
Revised: Results in Table 2 show that adding such constraint signiﬁcantly improve the performance of SAC: on hard tasks like Hopper-ET, S-Humanoid-ET and

Original: Humanoid-ET, SPEDE-REG improves the performance of SAC by 694.8, 1875.7 and 2000.4.
Revised: Humanoid-ET, SPEDE-REG improves the performance of SAC by 694.8, 1875.7 and 2000.4.

Original: Discussion of the Results We observe that in the environments with relatively simple dynamics (top row of Table 2), the proposed SPEDE achieves the SoTA, even comparing with SAC.
Revised: Discussion of the Results We observe that in the environments with relatively simple dynamics (top row of Table 2), the proposed SPEDE achieves the SoTA, even comparing with SAC.

Original: However, when the model dynamics of the environment become harder (bottom row of Table 2), the difference of the performance between the two approaches begin to enlarge.
Revised: However, when the model dynamics of the environment become harder (bottom row of Table 2), the difference of the performance between the two approaches begin to enlarge.

Original: Interestingly, our SPEDE achieves strong results comparing to model-based approaches, while the joint learning SPEDE-REG outperforms model-free algorithm by a huge margin.
Revised: Interestingly, our SPEDE achieves strong results comparing to model-based approaches, while the joint learning SPEDE-REG outperforms model-free algorithm by a huge margin.

Original: The performance promotion of SPEDE indicates the importance on learning a good representation based on model dynamics and again shows the effectiveness of our approach in both settings.
Revised: The performance promotion of SPEDE indicates the importance on learning a good representation based on model dynamics and again shows the effectiveness of our approach in both settings.

Original: In fact, the differences in the SoTA usage of SPEDE in easy environments and difﬁcult environments also reveals the important direction for our future work.
Revised: In fact, the differences in the SoTA usage of SPEDE in easy environments and difﬁcult environments also reveals the important direction for our future work.

Original: The current rigorous representation learning
Revised: The current rigorous representation learning

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: methods, e.g. , Du et al.
Revised: methods, e.g. , Du et al.

Original: ; Misra et al.
Revised: ; Misra et al.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: (2020a)
Revised: (2020a)

Original: and the proposed SPEDE, all rely on some model assumption.
Revised: and the proposed SPEDE, all rely on some model assumption.

Original: When the assumptions are satisﬁed, e
Revised: When the assumptions are satisﬁed, e

Original: [Figure]
Revised: [Figure]

Original: g. , Pendulum, Reacher, and others, our theoretically derived SPEDE variant works extremely well, even better than current
Revised: g. , Pendulum, Reacher, and others, our theoretically derived SPEDE variant works extremely well, even better than current

Original: SoTA.
Revised: SoTA.

Original: However, when the assumption is not fully satisﬁed, although the decoupled SPEDE achieves best performance among existing model-based RL and representation learning under fair comparison, the joint learned variant of SPEDE is more robust and promote the current SoTA with signiﬁcant margin.
Revised: However, when the assumption is not fully satisﬁed, although the decoupled SPEDE achieves best performance among existing model-based RL and representation learning under fair comparison, the joint learned variant of SPEDE is more robust and promote the current SoTA with signiﬁcant margin.

Original: An interesting question is whether we can rigorously justify the regularized SPEDE, which we leave as our future work
Revised: An interesting question is whether we can rigorously justify the regularized SPEDE, which we leave as our future work

Original: [Figure]
Revised: [Figure]

Original: Performance Curves To better understand how the sample complexity of our algorithm comparing to the prior model-based RL baselines, we plot the return versus environment steps in Figure 1.
Revised: Performance Curves To better understand how the sample complexity of our algorithm comparing to the prior model-based RL baselines, we plot the return versus environment steps in Figure 1.

Original: We see that comparing to prior model-based baselines, SPEDE enjoys great sample efﬁciency in these tasks.
Revised: We see that comparing to prior model-based baselines, SPEDE enjoys great sample efﬁciency in these tasks.

Original: We want to emphasize that from MBBL (Wang et al., 2019), model-based methods already show signiﬁcantly better sample efﬁciency compared to model-free methods ( e.g. PPO/TRPO).
Revised: We want to emphasize that from MBBL (Wang et al., 2019), model-based methods already show signiﬁcantly better sample efﬁciency compared to model-free methods ( e.g. PPO/TRPO).

Original: We provide additional results in Appendix E.2.
Revised: We provide additional results in Appendix E.2.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce SPEDE, which, to the best of our knowledge, is the ﬁrst provable and efﬁcient representation learning algorithm for RL, by exploiting the beneﬁts from noise.
Revised: In this paper, we introduce SPEDE, which, to the best of our knowledge, is the ﬁrst provable and efﬁcient representation learning algorithm for RL, by exploiting the beneﬁts from noise.

Original: We provide thorough theoretical analysis for SPEDE and strong empirical results demonstrate the effectiveness of our algorithm.
Revised: We provide thorough theoretical analysis for SPEDE and strong empirical results demonstrate the effectiveness of our algorithm.

Original: There are still several open problems, e.g. , how to rigorously justify the regularized
Revised: There are still several open problems, e.g. , how to rigorously justify the regularized

Original: SPEDE.
Revised: SPEDE.

Original: We leave these problems as interesting future work.
Revised: We leave these problems as interesting future work.

Original: Ethnics Statement Our paper focus on one of the core questions in RL, i e. , developing provable and practical algorithms for representation learning in online scenarios under general settings.
Revised: Ethnics Statement Our paper focus on one of the core questions in RL, i e. , developing provable and practical algorithms for representation learning in online scenarios under general settings.

Original: Although there can be several potential application of our work in the future, currently we cannot foresee any positive/negative social impact of our work other than the original positive/negative social impact of RL.
Revised: Although there can be several potential application of our work in the future, currently we cannot foresee any positive/negative social impact of our work other than the original positive/negative social impact of RL.

Original: As we conduct our experiments on standard MuJoCo benchmark, our experiments should not suffer from any discrimination/bias/fairness concerns.
Revised: As we conduct our experiments on standard MuJoCo benchmark, our experiments should not suffer from any discrimination/bias/fairness concerns.

Original: Reproducibility Statement Our code is directly adapted from the open-sourced code base from (Wang et al., 2019) and we have introduced our implementation details in Section 5 and
Revised: Reproducibility Statement Our code is directly adapted from the open-sourced code base from (Wang et al., 2019) and we have introduced our implementation details in Section 5 and

Original: Appendix E. We believe our results can be easily reproduced, and our code is available at https://sites.google.com/view/spede.
Revised: Appendix E. We believe our results can be easily reproduced, and our code is available at https://sites.google.com/view/spede.

Original: A B ACKGROUNDS ON R EPRODUCING K ERNEL H ILBERT S PACE
Revised: A B ACKGROUNDS ON R EPRODUCING K ERNEL H ILBERT S PACE

Original: We brieﬂy introduce the basic concepts of the Reproducing Kernel Hilbert Space, which is helpful on understanding our paper.
Revised: We brieﬂy introduce the basic concepts of the Reproducing Kernel Hilbert Space, which is helpful on understanding our paper.

Original: To start with, we ﬁrst deﬁne the inner product (Stein & Shakarchi, 2009).
Revised: To start with, we ﬁrst deﬁne the inner product (Stein & Shakarchi, 2009).

Original: Deﬁnition 6 (Inner Product) .
Revised: Deﬁnition 6 (Inner Product) .

Original: A function (cid:104)· , ·(cid:105) H : H × H → R is said to be an inner product on H if it satisﬁes the following conditions:
Revised: A function (cid:104)· , ·(cid:105) H : H × H → R is said to be an inner product on H if it satisﬁes the following conditions:

Original: Positive Deﬁniteness: ∀ u ∈ H , (cid:104) u, u (cid:105) ≥ 0 , and (cid:104) u, u (cid:105) = 

Revised: Positive Deﬁniteness: ∀ u ∈ H , (cid:104) u, u (cid:105) ≥ 0 , and (cid:104) u, u (cid:105) =

Original: ⇐⇒ u
Revised: ⇐⇒ u

Original: = 0 .
Revised: = 0 .

Original: Symmetry: ∀ u, v ∈ H , (cid:104) u, v (cid:105) ∈ (cid:104) v, u (cid:105) .
Revised: Symmetry: ∀ u, v ∈ H , (cid:104) u, v (cid:105) ∈ (cid:104) v, u (cid:105) .

Original: Bilinearity: ∀ α, β ∈ R , u, v, w ∈ H , (cid:104) αu + βv, w (cid:105) =
Revised: Bilinearity: ∀ α, β ∈ R , u, v, w ∈ H , (cid:104) αu + βv, w (cid:105) =

Original: α (cid:104) u, w (cid:105)
Revised: α (cid:104) u, w (cid:105)

Original: + β (cid:104) v, w (cid:105) .
Revised: + β (cid:104) v, w (cid:105) .

Original: Additionally, we can deﬁne a norm with the inner product: (cid:107) u (cid:107) =
Revised: Additionally, we can deﬁne a norm with the inner product: (cid:107) u (cid:107) =

Original: (cid:112) (cid:104) u, u (cid:105) .
Revised: (cid:112) (cid:104) u, u (cid:105) .

Original: A Hilbert space is a space equipped with an inner product and satisﬁes an additional technical condition of completeness.
Revised: A Hilbert space is a space equipped with an inner product and satisﬁes an additional technical condition of completeness.

Original: The ﬁnite-dimension vector space with the canonical inner product is an example of the Hilbert space.
Revised: The ﬁnite-dimension vector space with the canonical inner product is an example of the Hilbert space.

Original: We remark that H can also be a function space, for example, the space contains all square integrable functions (i.e. (cid:82) R f ( x ) 2 d x < ∞ , generally denoted as L 2 ) is also a
Revised: We remark that H can also be a function space, for example, the space contains all square integrable functions (i.e. (cid:82) R f ( x ) 2 d x < ∞ , generally denoted as L 2 ) is also a

Original: Hilbert space with inner product (cid:104) f, g (cid:105) =
Revised: Hilbert space with inner product (cid:104) f, g (cid:105) =

Original: (cid:82) R f ( x ) g ( x ) d x .
Revised: (cid:82) R f ( x ) g ( x ) d x .

Original: We then deﬁne the kernel, and introduce the notion of positive-deﬁnite kernel ( ´Alvarez et al., 2012).
Revised: We then deﬁne the kernel, and introduce the notion of positive-deﬁnite kernel ( ´Alvarez et al., 2012).

Original: Deﬁnition 7 ((Positive-Deﬁnite) Kernel) .
Revised: Deﬁnition 7 ((Positive-Deﬁnite) Kernel) .

Original: A function k : X → X → R is said to be a kernel on nonempty set X if there exists a Hilbert space H and a feature map φ : X → H
Revised: A function k : X → X → R is said to be a kernel on nonempty set X if there exists a Hilbert space H and a feature map φ : X → H

Original: such that ∀ x, x (cid:48) ∈ X , we have
Revised: such that ∀ x, x (cid:48) ∈ X , we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, the kernel is said to be positive deﬁnite if ∀ n ≥ 1 , ∀{ a i } i ∈ [ n ] ⊂ R and mutually distinct set { x i } i ∈
Revised: Moreover, the kernel is said to be positive deﬁnite if ∀ n ≥ 1 , ∀{ a i } i ∈ [ n ] ⊂ R and mutually distinct set { x i } i ∈

Original: [ n ] ⊂ X , we have that
Revised: [ n ] ⊂ X , we have that

Original: [Equation]
Revised: [Equation]

Original: Some well-known kernels include:
Revised: Some well-known kernels include:

Original: • Linear Kernel: k ( x, y )
Revised: • Linear Kernel: k ( x, y )

Original: = (cid:104) x, y (cid:105) , with the canonical feature map φ ( x ) = x . • Polynomial Kernel: k
Revised: = (cid:104) x, y (cid:105) , with the canonical feature map φ ( x ) = x . • Polynomial Kernel: k

Original: ( x, y )
Revised: ( x, y )

Original: =
Revised: =

Original: ( (cid:104) x, y (cid:105) + c ) m , where m ∈ N + and c ∈ R + .
Revised: ( (cid:104) x, y (cid:105) + c ) m , where m ∈ N + and c ∈ R + .

Original: • Gaussian (a.k.a radial basis function, RBF) Kernel: k ( x, y ) =
Revised: • Gaussian (a.k.a radial basis function, RBF) Kernel: k ( x, y ) =

Original: exp (cid:16) (cid:107) x − y (cid:107) 

Revised: exp (cid:16) (cid:107) x − y (cid:107)

Original: σ 2 (cid:17) .
Revised: σ 2 (cid:17) .

Original: It’s known that such kernel is positive deﬁnite.
Revised: It’s known that such kernel is positive deﬁnite.

Original: Now we can deﬁne the Reproducing Kernel Hilbert space (RKHS) (Aronszajn, 1950).
Revised: Now we can deﬁne the Reproducing Kernel Hilbert space (RKHS) (Aronszajn, 1950).

Original: Deﬁnition 8 (Reproducing Kernel Hilbert Space (RKHS)) .
Revised: Deﬁnition 8 (Reproducing Kernel Hilbert Space (RKHS)) .

Original: The Hilbert space H of R -valued function deﬁned on a non-emptry set X is said to be a reproducing kernel Hilbert space (RKHS) is there is a kernel k :
Revised: The Hilbert space H of R -valued function deﬁned on a non-emptry set X is said to be a reproducing kernel Hilbert space (RKHS) is there is a kernel k :

Original: X × X → R , such that
Revised: X × X → R , such that

Original: ∀ x ∈ X , k
Revised: ∀ x ∈ X , k

Original: ( x, · ) ∈ H .
Revised: ( x, · ) ∈ H .

Original: ∀ x ∈ X , f ∈ H , (cid:104) f, k ( x, · ) (cid:105) H = f
Revised: ∀ x ∈ X , f ∈ H , (cid:104) f, k ( x, · ) (cid:105) H = f

Original: ( x ) (a.k.a the reproducing property), which also implies that (cid:104) k
Revised: ( x ) (a.k.a the reproducing property), which also implies that (cid:104) k

Original: ( x, · ) , k ( y, · ) (cid:105) =
Revised: ( x, · ) , k ( y, · ) (cid:105) =

Original: k ( x, y ) .
Revised: k ( x, y ) .

Original: Here k is called a reproducing kernel of H .
Revised: Here k is called a reproducing kernel of H .

Original: We provide an intuitive interpretation on the deﬁnition of RKHS when H is the space of linear function.
Revised: We provide an intuitive interpretation on the deﬁnition of RKHS when H is the space of linear function.

Original: Consider X = R d
Revised: Consider X = R d

Original: and k ( x, y ) =
Revised: and k ( x, y ) =

Original: (cid:104) x, y (cid:105) .
Revised: (cid:104) x, y (cid:105) .

Original: With the deﬁnition of the kernel k , we can see that k ( x, · ) :
Revised: With the deﬁnition of the kernel k , we can see that k ( x, · ) :

Original: X → R is a linear function, and thus lies in H .
Revised: X → R is a linear function, and thus lies in H .

Original: Meanwhile, ∀ f ∈ H , there exists θ f such that f ( x ) = θ (cid:62) f x .
Revised: Meanwhile, ∀ f ∈ H , there exists θ f such that f ( x ) = θ (cid:62) f x .

Original: We deﬁne the inner product on H via (cid:104) f, g (cid:105) H = (cid:104) θ f , θ g (cid:105) , and thus (cid:104) f
Revised: We deﬁne the inner product on H via (cid:104) f, g (cid:105) H = (cid:104) θ f , θ g (cid:105) , and thus (cid:104) f

Original: ( , k ( x, · )) (cid:105) H
Revised: ( , k ( x, · )) (cid:105) H

Original: = θ (cid:62) x
Revised: = θ (cid:62) x

Original: = f ( x ) , which demonstrates the reproducing property, and shows that the space of linear function on any ﬁnite-dimensional vector space is an RKHS with linear kernel as the corresponding reproducing kernel.
Revised: = f ( x ) , which demonstrates the reproducing property, and shows that the space of linear function on any ﬁnite-dimensional vector space is an RKHS with linear kernel as the corresponding reproducing kernel.

Original: We state the following theorems without the proof.
Revised: We state the following theorems without the proof.

Original: Theorem 9 (Moore-Aronszajn (Aronszajn, 1950)) .
Revised: Theorem 9 (Moore-Aronszajn (Aronszajn, 1950)) .

Original: Every positive deﬁnite kernel k is associated with a unique RKHS H .
Revised: Every positive deﬁnite kernel k is associated with a unique RKHS H .

Original: Notice that, Moore-Aronszajn theorem guarantees that all of the positive kernel can be represented as the inner product in certain Hilbert space, hence we can have a linear representation of the Gaussian distribution induced by the reproducing property of Gaussian kernel, as we illustrated in the main text.
Revised: Notice that, Moore-Aronszajn theorem guarantees that all of the positive kernel can be represented as the inner product in certain Hilbert space, hence we can have a linear representation of the Gaussian distribution induced by the reproducing property of Gaussian kernel, as we illustrated in the main text.

Original: Theorem 10 (Bochner (Rudin, 2017)) .
Revised: Theorem 10 (Bochner (Rudin, 2017)) .

Original: A continuous, shift-invariant kernel (i.e. k ( x, y ) =
Revised: A continuous, shift-invariant kernel (i.e. k ( x, y ) =

Original: k ( x − y ) ) is positive deﬁnite if and only if k ( x − y ) is the Fourier transform of a non-negative measure ω , i.e.
Revised: k ( x − y ) ) is positive deﬁnite if and only if k ( x − y ) is the Fourier transform of a non-negative measure ω , i.e.

Original: [Equation]
Revised: [Equation]

Original: where P ( b ) is a uniform distribution on [0 , 2 π ] .
Revised: where P ( b ) is a uniform distribution on [0 , 2 π ] .

Original: Bochner’s theorem shows that any continuous positive deﬁnite shift-invariant kernel (e.g. Gaussian kernel, Laplacian kernel) can be represented as the inner product of random Fourier feature, which provides an additional way to provide a representation for certain distribution (see Rahimi & Recht, 2007;
Revised: Bochner’s theorem shows that any continuous positive deﬁnite shift-invariant kernel (e.g. Gaussian kernel, Laplacian kernel) can be represented as the inner product of random Fourier feature, which provides an additional way to provide a representation for certain distribution (see Rahimi & Recht, 2007;

Original: Dai et al., 2014).
Revised: Dai et al., 2014).

Original: B A N E QUIVALENT U PPER C ONFIDENCE B OUND A LGORITHM
Revised: B A N E QUIVALENT U PPER C ONFIDENCE B OUND A LGORITHM

Original: In this section, we provide a generic Upper Conﬁdence Bound (UCB) algorithm with the OFU principle, and show the connections and differences between the UCB algorithm and the TS algorithm.
Revised: In this section, we provide a generic Upper Conﬁdence Bound (UCB) algorithm with the OFU principle, and show the connections and differences between the UCB algorithm and the TS algorithm.

Original: The prototype for our UCB algorithm is illustrated in Algorithm 3.
Revised: The prototype for our UCB algorithm is illustrated in Algorithm 3.

Original: Algorithm 3 Upper Conﬁdence Bound (UCB) Algorithm
Revised: Algorithm 3 Upper Conﬁdence Bound (UCB) Algorithm

Original: Require: Number of Episodes K , Failure Probability δ ∈ (0 , 1) , Reward Function r
Revised: Require: Number of Episodes K , Failure Probability δ ∈ (0 , 1) , Reward Function r

Original: 1: Initialize the
Revised: 1: Initialize the

Original: 2: for do 3:
Revised: 2: for do 3:

Original: Compute π k via (cid:46)
Revised: Compute π k via (cid:46)

Original: Optimistic
Revised: Optimistic

Original: [Equation]
Revised: [Equation]

Original: where F k is deﬁned in (9).
Revised: where F k is deﬁned in (9).

Original: 4: for
Revised: 4: for

Original: 1 do (cid:46) Execute π k .
Revised: 1 do (cid:46) Execute π k .

Original: Execute a kh ∼ π hk ( s kh
Revised: Execute a kh ∼ π hk ( s kh

Original: Observe s h
Revised: Observe s h

Original: 7: end for 8:
Revised: 7: end for 8:

Original: Set H k
Revised: Set H k

Original: = H k − 1 ∪ { ( s kh , a kh , s kh +1 ) } H − 1 h =0 .
Revised: = H k − 1 ∪ { ( s kh , a kh , s kh +1 ) } H − 1 h =0 .

Original: 9: end for
Revised: 9: end for

Original: Notice that, the only difference between UCB algorithm and TS algorithm is the mechanism of ﬁnding f we use to plan for each episode (highlighted in blue).
Revised: Notice that, the only difference between UCB algorithm and TS algorithm is the mechanism of ﬁnding f we use to plan for each episode (highlighted in blue).

Original: For UCB algorithm, we perform an optimistic planning, which ﬁnds the ˜ f
Revised: For UCB algorithm, we perform an optimistic planning, which ﬁnds the ˜ f

Original: k that potentially has the largest cumulative reward.
Revised: k that potentially has the largest cumulative reward.

Original: However, such constrained optimization problem is NP-hard even for the simplest linear bandits (Dani et al., 2008).
Revised: However, such constrained optimization problem is NP-hard even for the simplest linear bandits (Dani et al., 2008).

Original: Instead, for TS algorithm, we only sample the f k from the posterior distribution, which gets rid of the complicated constraint optimization.
Revised: Instead, for TS algorithm, we only sample the f k from the posterior distribution, which gets rid of the complicated constraint optimization.

Original: We are interested in the UCB algorithm, as the worst case regret bound of the UCB algorithm can be directly translated to the expected regret bound of the TS algorithm without the need of explicit manipulation of the prior and the posterior(Russo &
Revised: We are interested in the UCB algorithm, as the worst case regret bound of the UCB algorithm can be directly translated to the expected regret bound of the TS algorithm without the need of explicit manipulation of the prior and the posterior(Russo &

Original: Van Roy, 2013; 2014; Osband & Van Roy, 2014).
Revised: Van Roy, 2013; 2014; Osband & Van Roy, 2014).

Original: Conﬁdence Set Construction Perhaps the most important part in OFU-style algorithm is the construction of conﬁdence set F k .
Revised: Conﬁdence Set Construction Perhaps the most important part in OFU-style algorithm is the construction of conﬁdence set F k .

Original: To enable sample-efﬁcient learning, the conﬁdence set should 1.
Revised: To enable sample-efﬁcient learning, the conﬁdence set should 1.

Original: contain f ∗ with high probability, so that we can identify f ∗ eventually; 2.
Revised: contain f ∗ with high probability, so that we can identify f ∗ eventually; 2.

Original: shrink as fast as possible, so that we can identify f ∗ efﬁciently.
Revised: shrink as fast as possible, so that we can identify f ∗ efﬁciently.

Original: In the tabular setting, F k is constructed via the concentration of sub-Gaussian/sub-Gamma random variable (e.g. Azar et al., 2017; Jin et al., 2018; Zhang et al., 2021), and in the linear MDP setting, F k is constructed via the concentration on the linear parameters.
Revised: In the tabular setting, F k is constructed via the concentration of sub-Gaussian/sub-Gamma random variable (e.g. Azar et al., 2017; Jin et al., 2018; Zhang et al., 2021), and in the linear MDP setting, F k is constructed via the concentration on the linear parameters.

Original: As we don’t assume any speciﬁc structures, we instead constructed F k via the concentration on the (cid:96) 2 error, following the idea of (Russo & Van Roy, 2013; Osband & Van Roy, 2014).
Revised: As we don’t assume any speciﬁc structures, we instead constructed F k via the concentration on the (cid:96) 2 error, following the idea of (Russo & Van Roy, 2013; Osband & Van Roy, 2014).

Original: Speciﬁcally, consider the least-square estimates deﬁned by
Revised: Speciﬁcally, consider the least-square estimates deﬁned by

Original: [Equation]
Revised: [Equation]

Original: As s kh +

Revised: As s kh +

Original: = f ∗ ( s kh , a kh )
Revised: = f ∗ ( s kh , a kh )

Original: + (cid:15) kh
Revised: + (cid:15) kh

Original: where (cid:15) kh is the Gaussian noise added to the step h at the k -th episode, we know ˆ f K will not deviate from f ∗ a lot.
Revised: where (cid:15) kh is the Gaussian noise added to the step h at the k -th episode, we know ˆ f K will not deviate from f ∗ a lot.

Original: Meanwhile, as K increases, the estimation ˆ f K should become closer to f ∗ .
Revised: Meanwhile, as K increases, the estimation ˆ f K should become closer to f ∗ .

Original: Speciﬁcally, deﬁne the empirical 2 -norm (cid:107) · (cid:107) 2 ,E t as
Revised: Speciﬁcally, deﬁne the empirical 2 -norm (cid:107) · (cid:107) 2 ,E t as

Original: [Equation]
Revised: [Equation]

Original: We can construct the conﬁdence set based on the following lemma:
Revised: We can construct the conﬁdence set based on the following lemma:

Original: Lemma 11 (Conﬁdence Set Construction (Russo & Van Roy, 2013; Osband & Van Roy, 2014)) .
Revised: Lemma 11 (Conﬁdence Set Construction (Russo & Van Roy, 2013; Osband & Van Roy, 2014)) .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix C.1.
Revised: The proof can be found in Appendix C.1.

Original: Notice that, the empirical 2 -norm (cid:107) f
Revised: Notice that, the empirical 2 -norm (cid:107) f

Original: − ˆ f K (cid:107) 2 ,E K scales linearly with K , and β ∗ K ( F , δ, α ) only scales as log K , so the conﬁdence set shrinks.
Revised: − ˆ f K (cid:107) 2 ,E K scales linearly with K , and β ∗ K ( F , δ, α ) only scales as log K , so the conﬁdence set shrinks.

Original: Meanwhile,
Revised: Meanwhile,

Original: Equation 10 guarantees that f ∗ ∈ F k , ∀ k with high probability.
Revised: Equation 10 guarantees that f ∗ ∈ F k , ∀ k with high probability.

Original: Hence, it satisﬁes our requirement for the conﬁdence set.
Revised: Hence, it satisﬁes our requirement for the conﬁdence set.

Revised: Regret Upper Bound We have the following upper bound of the regret for the UCB algorithm:

Revised: Theorem 12 (Regret Bound) .

Revised: Assume Assumption 2 to 5 holds.

Original:  C T ECHNICAL P ROOF C. P ROOF FOR L EMMA
Revised: We have that [Equation] where ˜ O represents the order up to logarithm factors. C T ECHNICAL P ROOF C. P ROOF FOR L EMMA

Original: Proof.
Revised: Proof.

Original: We ﬁrst show the following concentration on the (cid:96) 2 error:
Revised: We ﬁrst show the following concentration on the (cid:96) 2 error:

Original: Lemma 12 (Concentration of (cid:96) 2 error (Russo & Van Roy, 2013; Osband & Van Roy, 2014; Wang et al., 2020)) .
Revised: Lemma 13 (Concentration of (cid:96) 2 error (Russo & Van Roy, 2013; Osband & Van Roy, 2014; Wang et al., 2020)) .

Original: ∀ δ > 0 , f : S ×
Revised: ∀ δ > 0 , f : S ×

Original: A → R , we haveP
Revised: A → R , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne the ﬁltration H k,h = { ( s ih , a ih ) } i ∈
Revised: Deﬁne the ﬁltration H k,h = { ( s ih , a ih ) } i ∈

Original: [ k − 1] ,h =0 , ··· ,H − 1 ∪ { ( s ki , a ki ) } h − 1 h =0 , and the random variable Z k,h adapted to the ﬁltration H k,h via:
Revised: [ k − 1] ,h =0 , ··· ,H − 1 ∪ { ( s ki , a ki ) } h − 1 h =0 , and the random variable Z k,h adapted to the ﬁltration H k,h via:

Original: [Equation]
Revised: [Equation]

Original: 22 − (cid:107) f ( s kh , a kh ) − f ∗ ( s kh , a kh )

Original: [-+ f ∗ ( s kh , a kh ) − s kh +
-]
Revised: [-+ f ∗ ( s kh , a kh ) − s kh +
-]

Original: [-(cid:107)= − (cid:107) f ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 
 + 2 (cid:104) f ( s kh , a kh ) − f ∗ ( s kh , a kh ) , (cid:15) kh (cid:105) ,-] where (cid:15) kh =
Revised: [-(cid:107)= − (cid:107) f ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 
 + 2 (cid:104) f ( s kh , a kh ) − f ∗ ( s kh , a kh ) , (cid:15) kh (cid:105) ,-] where (cid:15) kh =

Original: s kh
Revised: s kh

Original: +1 − f ∗ ( s kh , a kh ) .
Revised: +1 − f ∗ ( s kh , a kh ) .

Original: Thus, E ( Z hk |H
Revised: Thus, E ( Z hk |H

Original: k,h ) = −(cid:107)
Revised: k,h ) = −(cid:107)

Original: f ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 , and Z hk + (cid:107) f
Revised: f ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 , and Z hk + (cid:107) f

Original: ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 is a martingale w.r.t H k,h .
Revised: ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 is a martingale w.r.t H k,h .

Original: Notice that we assume (cid:15) is an isotropic
Revised: Notice that we assume (cid:15) is an isotropic

Original: Gaussian noise with variance σ 2 on each of the dimension, thus the conditional moment generating function of Z hk + (cid:107) f
Revised: Gaussian noise with variance σ 2 on each of the dimension, thus the conditional moment generating function of Z hk + (cid:107) f

Original: ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 satisﬁes:
Revised: ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 satisﬁes:

Original: M k,h ( λ ) = log E [exp( λ ( Z hk + (cid:107) f ( s kh , a kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 )) |H k,h ] = log E [exp( (cid:104) 2 λf ( s kh , a kh )
Revised: [Equation]

Original: − f ∗ ( s kh , a kh ) , (cid:15) kh (cid:105) ) |H k,h ] ≤ 2 σ 2 λ 2 (cid:107) f
Revised: [Equation]

Original: ( s kh ) − f ∗ ( s kh , a kh ) (cid:107) 22 .

Original: Applying Lemma 4 in (Russo & Van Roy, 2013), we have that, ∀ x, λ ≥ 0 ,
Revised: Applying Lemma 4 in (Russo & Van Roy, 2013) [Equation] we have that, ∀ x, λ ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Take λ = 14 σ 2 , x = log 1 /δ , and notice that (cid:80) k ∈
Revised: Take λ = 14 σ 2 , x = log 1 /δ [Equation] and notice that (cid:80) k ∈

Original: [ K ] (cid:80) H − 1 h =

Revised: [ K ] (cid:80) H − 1 h =

Original: Z k,h
Revised: Z k,h

Original: = L 2 ,K ( f ∗ ) − L 2 ,K ( f ) , we have the desired result.
Revised: = L 2 ,K ( f ∗ ) − L 2 ,K ( f ) , we have the desired result.

Original: We construct an α -cover F α in F with respect to (cid:107) · (cid:107) 2 .
Revised: We construct an α -cover F α in F with respect to (cid:107) · (cid:107) 2 .

Original: With a standard union bound, we know that condition on f ∗ , with probability at least 1 − δ , we have that
Revised: With a standard union bound, we know that condition on f ∗ , with probability at least 1 − δ , we have that

Original: [Equation]
Revised: [Equation]

Original: Thus, we have that
Revised: Thus, we have that

Original: [Equation]
Revised: [Equation]

Original: DiscretizationError .
Revised: Discretization

Revised: [Equation]

Original: We then deal with the discretization error.
Revised: We then deal with the discretization error.

Original: Assume α ≤ 2 C (or otherwise we only have a trivial cover) and (cid:107)
Revised: Assume α ≤ 2 C (or otherwise we only have a trivial cover) and (cid:107)

Original: f α ( s, a ) − f ( s, a ) (cid:107) 2 ≤ α , we have that
Revised: f α ( s, a ) − f ( s [Equation] a ) (cid:107) 2 ≤ α , we have that

Original: [Equation]
Revised: [Equation]

Original: where the inequality is by Cauchy-Schwartz inequality and α ≤ 2 C .
Revised: where the inequality is by Cauchy-Schwartz inequality and α ≤ 2 C .

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Revised: s, a ) (cid:105) + (cid:107) f

Revised: ( s, a ) (cid:107) 22 − (cid:107) f

Revised: α ( s, a ) (cid:107)≤ 2 (cid:104) (cid:15), f α ( s, a ) − f ( s, a ) (cid:105) + 2 (cid:104) f ∗ ( s, a ) , f α ( s, a ) − f ( s, a ) (cid:105) + 2 Cα + α≤ 2 (cid:107) (cid:15) (cid:107) 2 α + 6 Cα.

Original: We now consider the concentration property of (cid:107) (cid:15) (cid:107) 2 .
Revised: We now consider the concentration property of (cid:107) (cid:15) (cid:107) 2 .

Original: Here we simply follow (Jin et al., 2019) and notice that (cid:15) is √ dσ -norm-sub-Gaussian, we have that
Revised: Here we simply follow (Jin et al., 2019) and notice that (cid:15) is √ dσ -norm-sub-Gaussian, we have that

Original: [Equation]
Revised: [Equation]

Original: By a union bound, we have that
Revised: By a union bound, we have that

Original: [Equation]
Revised: [Equation]

Original: Sum all these up, we can see with probability 1 − δ , ∀ K ∈ N , the discretization error is upper bounded by:
Revised: Sum all these up, we can see with probability 1 − δ , ∀ K ∈ N , the discretization error is upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: As we consider the least square estimate ˆ f K , we have that L 2 ,K
Revised: As we consider the least square estimate ˆ f K , we have that L 2 ,K

Original: ( ˆ f K ) − L 2 ,K ( f ∗ ) ≤ 0 .
Revised: ( ˆ f K ) − L 2 ,K ( f ∗ ) ≤ 0 .

Original: Substitute back, we have the desired results.
Revised: Substitute back, we have the desired results.

Original: C. S IMULATION L EMMA
Revised: C. S IMULATION L EMMA

Original: Lemma 13 (Simulation Lemma (adapted from Lemma 3.9 in (Kakade et al., 2020))) .
Revised: Lemma 14 (Simulation Lemma (adapted from Lemma 3.9 in (Kakade et al., 2020))) .

Original: Given ˆ f , ∀ s ∈ S , the value function ˆ V π
Revised: Given ˆ f , ∀ s ∈ S , the value function ˆ V π

Original: and V π corresponding to the model ˆ f and f ∗ satisﬁes
Revised: and V π corresponding to the model ˆ f and f ∗ satisﬁes

Original: [Equation]
Revised: [Table]

Original: σ,(cid:41)(cid:35) .

Original: Proof.
Revised: Proof.

Original: We ﬁrst show the following difference lemma:
Revised: We ﬁrst show the following difference lemma:

Original: Lemma 14 (Difference Lemma) .
Revised: Lemma 15 (Difference Lemma) .

Original: Assume the trajectory { ( s h , a h ) } H − 1 h =0 is generated via policy π and ground truth f ∗ , deﬁneV [Equation] =
Revised: Assume the trajectory { ( s h , a h ) } H − 1 h =0 is generated via policy π and ground truth f ∗ , deﬁne

Original: [Equation]
Revised: [Equation]

Original: then ∀ τ ∈ { 1 , · · · , H − 1 } , we have:
Revised: then ∀ τ ∈ { 1 , · · · , H − 1 } , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: When τ = 1 , we can obtain the result with a 

Revised: When τ = 1 , we can obtain the result with a

Original: =
Revised: =

Original: π
Revised: π

Original: ( s 0 ) and
Revised: ( s 0 ) and

Original: [Equation]
Revised: [Equation]

Original: We only need to show the case when τ = 2 , and the case when τ > 2 can be derived via recursion.
Revised: We only need to show the case when τ = 2 , and the case when τ > 2 can be derived via recursion.

Original: Notice that ˆ
Revised: Notice that ˆ

Original: [Equation]
Revised: [Equation]

Original: where the last equality is due to the fact that a 1 = π ( s 1 ) .
Revised: where the last equality is due to the fact that a 1 = π ( s 1 ) .

Original: We then follow the idea of “optional stopping” used in (Kakade et al., 2020) and show the following “optional stopping” simulation lemma.
Revised: We then follow the idea of “optional stopping” used in (Kakade et al., 2020) and show the following “optional stopping” simulation lemma.

Original: Lemma 15 (“Optional Stopping” Simulation Lemma) .
Revised: Lemma 16 (“Optional Stopping” Simulation Lemma) .

Original: Consider the stochastic process over the trajectories { ( s h , a h ) } H − 1 h =0 generated via policy π and ground truth f ∗ , where the randomness is from the Gaussian noise in the dynamics.
Revised: Consider the stochastic process over the trajectories { ( s h , a h ) } H − 1 h =0 generated via policy π and ground truth f ∗ , where the randomness is from the Gaussian noise in the dynamics.

Original: Deﬁne a stopping time τ w.r.t this stochastic process and a given model ˆ f via:
Revised: Deﬁne a stopping time τ w.r.t this stochastic process and a given model ˆ f via:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, deﬁne a random variable: ˜
Revised: Furthermore, deﬁne a random variable: ˜

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is w.r.t the stochastic process over the trajectories.
Revised: where the expectation is w.r.t the stochastic process over the trajectories.

Original: Proof.
Revised: Proof.

Original: Deﬁne the ﬁltration F

Original: [Equation] [Equation] [Equation] where (cid:15) i is the noise that add to the dynamics at step i .
Revised: Deﬁne the ﬁltration F h := { (cid:15) i } h − 1 i =0 , where (cid:15) i is the noise that add to the dynamics at step i .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: which is a Doob martingale with respect to F i (Grimmett & Stirzaker, 2020).
Revised: which is a Doob martingale with respect to F i (Grimmett & Stirzaker, 2020).

Original: As τ ≤ H , by Doob’s optional stopping theorem, we have that
Revised: As τ ≤ H , by Doob’s optional stopping theorem, we have that

Original: [Equation]
Revised: [Equation]

Original: We then provide a bound for M τ .
Revised: We then provide a bound for M τ .

Original: By Lemma 14, we have that
Revised: By Lemma 15, we have that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the third inequality follows the deﬁnition of τ (and thus V πτ ( s τ )
Revised: where the third inequality follows the deﬁnition of τ (and thus V πτ ( s τ )

Original: =
Revised: =

Original: ˜ V πτ ( s τ ) and ˆ V πh ( s h ) =
Revised: ˜ V πτ ( s τ ) and ˆ V πh ( s h ) =

Original: ˜ V πh ( s h ) for h < τ .)
Revised: ˜ V πh ( s h ) for h < τ .)

Original: The proof is then concluded via the following observation:
Revised: The proof is then concluded via the following observation:

Original: [Equation]
Revised: [Equation]

Original: where the third equality is due to the fact that 1 h − 1 <τ is measurable under F h − 1 .
Revised: where the third equality is due to the fact that 1 h − 1 <τ is measurable under F h − 1 .

Original: Before we ﬁnally provide the proof of Lemma 13, we state the following lemma that bound the expectation under two isotropic Gaussian distribution with different mean:
Revised: Before we ﬁnally provide the proof of Lemma 14, we state the following lemma that bound the expectation under two isotropic Gaussian distribution with different mean:

Original: Lemma 16 (Difference of Expectation under Different Mean Isotropic Gaussian) .
Revised: Lemma 17 (Difference of Expectation under Different Mean Isotropic Gaussian) .

Original: ∀ (approximately measurable) positive function g , we have that
Revised: ∀ (approximately measurable) positive function g , we have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: We then calculate
Revised: We then calculate

Original: [Equation]
Revised: [Equation]

Revised: σ

Revised: [Equation]

Revised: (cid:19) .

Original: Also notice that, as g is positive, a simple bound is that 
Revised: Also notice that, as g is positive, a simple bound is that E

Original: [Equation]
Revised: [Equation]

Revised: µ

Revised: [Equation]

Revised: I

Revised: [Equation]

Revised: [Equation]

Revised: 2 I )

Revised: [Equation]

Revised: z )]

Revised: [Equation]

Revised: µ

Revised: [Equation]

Revised: [ g

Revised: [Equation]

Revised: [Equation]

Original: Thus,
Revised: Thus,

Revised: E

Original: [Equation]
Revised: [Equation]

Original: Notice that, if (cid:107) µ 2 − µ 1 (cid:107) ≥ σ , then exp (cid:16) (cid:107) µ 2 − µ1 (cid:107) 22 σ(cid:17) − 1 ≥ 1 .
Revised: (cid:115) min (cid:26) exp (cid:18) (cid:107) µ [Equation] − µ

Original: Meanwhile, when x ∈ [0 , 1] , exp( x ) ≤ 1 + 2 x .
Revised: [Equation]

Original: Thus, (cid:115) min (cid:26) exp (cid:18) (cid:107)µ 2 − µ 1 (cid:107)σ(cid:19) − 1 ,(cid:27) ≤ (cid:115) min (cid:26) 1 + 2 (cid:107) µ
Revised: (cid:107)

Original: [Table]
Revised: [Equation]

Original: µ 1 (cid:107)σ− 1 ,(cid:27) = min (cid:26) 2 (cid:107) µ 2 − µ 1 (cid:107)σ,(cid:27) , which ﬁnishes the proof. With Lemma 15, we have that ˆ
Revised: (cid:19) − 1 ,(cid:27) . Notice that, if (cid:107) µ 2 − µ 1 (cid:107) ≥ σ , then exp (cid:16) (cid:107) µ 2 − µ 1 (cid:107) 22 σ(cid:17) − 1 ≥ 1 .

Original: [Equation] [Table] [Equation] [Table]
Revised: Meanwhile, when x ∈ [0 , 1] , exp( x ) ≤ 1 + 2 x .

Original: [Equation] [Table] [Equation] [Table]
Revised: Thus, (cid:115) min (cid:26) exp (cid:18) (cid:107) µ 2 − µ 1 (cid:107)σ(cid:19) − 1 ,(cid:27) ≤ (cid:115) min (cid:26) 1 + 2 (cid:107) µ

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: µ 1 (cid:107)σ− 1 ,(cid:27) =

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: min (cid:26) 2 (cid:107) µ 2 − µ 1 (cid:107)σ,(cid:27) , which ﬁnishes the proof.

Original: [Table] [Equation] [Table] [Equation]
Revised: With Lemma 16, we have that ˆ V

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: (cid:40)

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: where the second inequality is due to Lemma
Revised: where the second inequality is due to Lemma

Original: [Table]
Revised: [Table]

Original: and the last inequality is due to the fact that ˜
Revised: and the last inequality is due to the fact that ˜

Revised: V πh ( s (cid:48) h +1 ) ≤ H , ∀ h

Original: [Equation]
Revised: [Equation]

Original: ) ≤ H , ∀ h .

Original: C
Revised: C. S UM OF W IDTH S QUARE

Original: S UM OF W IDTH S QUARE

Original: Lemma 17 (Bound on the Sum of Width Square) .
Revised: Lemma 18 (Bound on the Sum of Width Square) .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: If { β ∗ k } k ∈ [ K ] is a non-decreasing sequence, and (cid:107) f (cid:107) 2 < C, ∀ f ∈ F , then: (cid:88)
Revised: If { β ∗ k } k ∈ [ K ] is a non-decreasing sequence, and (cid:107) f (cid:107) 2 < C, ∀ f ∈ F , then: (cid:88)

Original: k ∈
Revised: k ∈

Original: [ K ]
Revised: [ K ]

Original: H − 1 (cid:88)
Revised: H − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst show the following lemma, which will be helpful in our proof.
Revised: We ﬁrst show the following lemma, which will be helpful in our proof.

Original: Lemma 18 (Lemma 1 in (Osband & Van Roy, 2014)) .
Revised: Lemma 19 (Lemma 1 in (Osband & Van Roy, 2014)) .

Original: If { β k } k ∈ [ K ] is a non-decreasing sequence, we have
Revised: If { β k } k ∈ [ K ] is a non-decreasing sequence, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst consider when w F k ( s kh , a kh ) > (cid:15) and is (cid:15) -dependent on n disjoint sub-sequences of { ( s ih , a ih ) } i ∈
Revised: We ﬁrst consider when w F k ( s kh , a kh ) > (cid:15) and is (cid:15) -dependent on n disjoint sub-sequences of { ( s ih , a ih ) } i ∈

Original: [ k − 1] .
Revised: [ k − 1] .

Original: By the deﬁnition of (cid:15) -dependent, we know (cid:107) ¯ f − f (cid:107) 2 ,E k > n(cid:15) 2 .
Revised: By the deﬁnition of (cid:15) -dependent, we know (cid:107) ¯ f − f (cid:107) 2 ,E k > n(cid:15) 2 .

Original: On the other hand, by triangle inequality, we know (cid:107) ¯ f − f (cid:107) 2 ,E k ≤ 2 √ β k ≤ 2 √ β K , thus n < 4 β K (cid:15) 2 .
Revised: On the other hand, by triangle inequality, we know (cid:107) ¯ f − f (cid:107) 2 ,E k ≤ 2 √ β [Equation] ≤ 2 √ β K , thus n < 4 β K (cid:15) 2 .

Original: Hence we know when w F k ( s k h , a k h ) > (cid:15) , then ( s h , a h ) is at most (cid:15) -dependent on 4 β K (cid:15) 2 disjoint sub-sequences of { ( s ih , a ih ) }
Revised: Hence we know when w F k ( s kh , a kh ) > (cid:15) , then ( s h , a h ) is at most (cid:15) -dependent on 4 β K (cid:15) 2 disjoint sub-sequences of { ( s ih , a ih ) }

Original: i ∈
Revised: i ∈

Original: [ k − 1] .
Revised: [ k − 1] .

Original: We then show that, for any sequence { ( s i , a i ) } i ∈
Revised: We then show that, for any sequence { ( s i , a i ) } i ∈

Original: [ N ] , there is some element ( s j , a j ) that is (cid:15) - dependent on at least n dim E ( F ,(cid:15) )
Revised: [ N ] , there is some element ( s j , a j ) that is (cid:15) - dependent on at least n dim E ( F ,(cid:15) )

Original: − H disjoint sub
Revised: − H disjoint sub

Original: -sequences of { ( s i , a i ) }
Revised: -sequences of { ( s i , a i ) }

Original: i ∈
Revised: i ∈

Original: [ j − 1] .
Revised: [ j − 1] .

Original: Let n satisﬁes
Revised: Let n satisﬁes

Original: that n dim E ( F , (cid:15) ) + 1 ≤ N ≤ ( n + 1)dim E ( F , (cid:15) ) , and we will construct n disjoint sub-sequences { B i } i ∈ [ n ] .
Revised: that n dim E ( F , (cid:15) ) + 1 ≤ N ≤ ( n + 1)dim E ( F , (cid:15) ) , and we will construct n disjoint sub-sequences { B i } i ∈ [ n ] .

Original: We ﬁrst let B i
Revised: We ﬁrst let B i

Original: =
Revised: =

Original: ( s i , a i ) } , ∀ i ∈ [ n ] .
Revised: ( s i , a i ) } , ∀ i ∈ [ n ] .

Original: If ( s k +1 , a k +1 ) is (cid:15) -dependent on each B
Revised: If ( s k +1 , a k +1 ) is (cid:15) -dependent on each B

Original: i , i ∈ [ n ] , we have the desired results.
Revised: i , i ∈ [ n ] , we have the desired results.

Original: Otherwise, we append ( s k +1 , a k +1 ) to the sub-sequence that it is (cid:15) - independent with.
Revised: Otherwise, we append ( s k +1 , a k +1 ) to the sub-sequence that it is (cid:15) - independent with.

Original: Repeat this process until some j > n + 1 is (cid:15) -dependent on each sub-sequence or we have reached N .
Revised: Repeat this process until some j > n + 1 is (cid:15) -dependent on each sub-sequence or we have reached N .

Original: In the latter case we have (cid:80) i ∈ [ n ] | B i | ≥ n dim E ( F , (cid:15) )
Revised: In the latter case we have (cid:80) i ∈ [ n ] | B i | ≥ n dim E ( F , (cid:15) )

Original: (here we can add at most H − 1 data to avoid the case we need a new episode of data), and since each element of a sub-sequence is (cid:15) -independent with its predecessors, | B i | ≤ dim E ( F , (cid:15) ) , ∀ i by the deﬁnition of eluder dimension.
Revised: (here we can add at most H − 1 data to avoid the case we need a new episode of data), and since each element of a sub-sequence is (cid:15) -independent with its predecessors, | B i | ≤ dim E ( F , (cid:15) ) , ∀ i by the deﬁnition of eluder dimension.

Original: Thus | B i | =
Revised: Thus | B i | =

Original: dim E ( F , (cid:15) ) , ∀ i .
Revised: dim E ( F , (cid:15) ) , ∀ i .

Original: And in this case, ( s N , a N ) must be (cid:15) -dependent on each sub-sequence by the deﬁnition of eluder dimension.
Revised: And in this case, ( s N , a N ) must be (cid:15) -dependent on each sub-sequence by the deﬁnition of eluder dimension.

Original: Notice that, as our data is collected in an episodic pattern, there are at most H − 1 sub-sequences that contains ”imaginary” ﬁnal episode data introduced to the construction.
Revised: Notice that, as our data is collected in an episodic pattern, there are at most H − 1 sub-sequences that contains ”imaginary” ﬁnal episode data introduced to the construction.

Original: In this case, we know that there are at least
Revised: In this case, we know that there are at least

Original: [Table]
Revised: [Table]

Original: sub-sequences that ( s N , a N ) is (cid:15) -dependent, which ﬁnishes our claim.
Revised: sub-sequences that ( s N , a N ) is (cid:15) -dependent, which ﬁnishes our claim.

Original: We ﬁnally consider the sub-sequence B = { ( s kh , a kh ) } with w F k ( s kh , a kh ) > (cid:15) .
Revised: We ﬁnally consider the sub-sequence B = { ( s kh , a kh ) } with w F k ( s kh , a kh ) > (cid:15) .

Original: We know that each element in B is (cid:15) -dependent on at most 4 β K (cid:15) 2 disjoint sub-sequence of B , but at least (cid:15) -dependent on | B | dim E ( F ,(cid:15) )
Revised: We know that each element in B is (cid:15) -dependent on at most 4 β K (cid:15) 2 disjoint sub-sequence of B , but at least (cid:15) -dependent on | B | dim E ( F ,(cid:15) )

Original: − H sub-
Revised: − H sub-

Original: sequence of B .
Revised: sequence of B .

Original: Thus we know | B | ≤ (cid:16) 4 β K (cid:15) 

Revised: Thus we know | B | ≤ (cid:16) [Equation] (cid:15)

Original: + H (cid:17) dim E ( F , (cid:15) ) , which concludes the proof.
Revised: + H (cid:17) dim E ( F , (cid:15) ) , which concludes the proof.

Original: For notation simplicity, we deﬁne w t,h :=
Revised: For notation simplicity, we deﬁne w t,h :=

Original: w F t ( s th , a th ) .
Revised: w F t ( s th , a th ) .

Original: We ﬁrst reorder the sequence { w t,h } k ∈
Revised: We ﬁrst reorder the sequence { w t,h } k ∈

Original: [ K ] , 0 ≤ h ≤ H − 1 → { w i } i ∈
Revised: [ K ] , 0 ≤ h ≤ H − 1 → { w i } i ∈

Original: [ KH ] , such that w 1 ≥ · · · w TH .
Revised: [ KH ] , such that w 1 ≥ · · · w TH .

Original: Then we have (cid:88)
Revised: Then we have (cid:88)

Original: [Equation]
Revised: [Equation]

Original: As we order the sequence, w k ≥ (cid:15) means 
Revised: As we order the sequence, w j ≥ (cid:15) means (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Hence we know(cid:15) ≤
Revised: Hence we know

Original: [Equation]
Revised: [Equation]

Original: which means if w i ≥ T − 1 / 2 , then w i < min (cid:110) 2 C, (cid:113) 4 β K dimE
Revised: which means if w i ≥ T − 1 / 2 , then w i < min (cid:110) 2 C, (cid:113) [Equation] dim

Original: [Equation]
Revised: [Equation]

Original: F ,T − 1 / 2 ) k − Hdim E
Revised: F [Equation] [Equation] H

Original: [Equation]
Revised: [Equation]

Original:  F ,T − 1 / 2 ) (cid:111) .
Revised: E ( F [Equation] / 2 ) (cid:111) .

Original: Hence, (cid:88) i ∈
Revised: Hence, (cid:88) i ∈

Original: [ KH ]
Revised: [ KH ]

Original: [Equation]
Revised: [Equation]

Original:  1 / 2 ) k − H dim E ( F , T − 1 / 2 ) ≤ 4 C 2 H dim E (cid:16) F , T − 1 / 2 (cid:17)
Revised: ≥ T − 1 / 2 ≤ 4 C 2 H dim E (cid:16) F , T − 1 / 2 (cid:17) + T (cid:88) j = H dim E ( F ,T − 1 / 2 )+ 4 β K dim E ( F , T − 1 / 2 )

Revised: j − H dim E ( F , T − 1 / 2 ) ≤ 4 C 2 H dim E (cid:16) F , T − 1 / 2 (cid:17)

Original: + 4 β K dim E (cid:16) F , T − 1 / 2 (cid:17) (1 + log T ) , which ﬁnishes the proof.
Revised: + 4 β K dim E (cid:16) F , T − 1 / 2 (cid:17) (1 + log T ) , which ﬁnishes the proof.

Original: C. P ROOF FOR T HEOREM 
Revised: C. P ROOF FOR T HEOREM 5 AND T HEOREM

Original: Proof.
Revised: Proof.

Original: Deﬁne E k
Revised: Deﬁne E k

Original: =
Revised: =

Original: P f ∗ ( f ∗ ∈ F k ) .
Revised: P f ∗ ( f ∗ ∈ F k ) .

Original: When constructing the conﬁdence set, take α = T − 1 / 2 and δ = 0 .
Revised: When constructing the conﬁdence set, take α = T − 1 / 2 and δ = 0 .

Original: 25 in Lemma 11, which leads to 
Revised: 25 in Lemma 11, which leads to β ∗ k := 8 σ

Original: [Equation]
Revised: [Equation]

Revised: log(4 N ( F , T − 1 / 2 , (cid:107) · (cid:107) 2 ))

Revised: + HT − 1 / 2 (12 C + (cid:112) 8 dσ 2 log(16 k 2 H )) .

Original: With our conﬁdence set construction, we know that (cid:80) k ∈ [ K ] P ( ¯ E k ) ≤ 0 .
Revised: With our conﬁdence set construction, we know that (cid:80) k ∈ [ K ] P ( ¯ E k ) ≤ 0 .

Original: Notice that
Revised: Notice that

Original: Regret( K ) =
Revised: Regret( K ) =

Original: (cid:88) k ∈
Revised: (cid:88) k ∈

Original: [ K ] [Equation]{+(cid:2) V ∗ 
 ( s k 0 )+}
Revised: [ K ] {+(cid:2) V ∗ 
 ( s k 0 )+}

Original:  ≤ E  
Revised: − V π k 0 ( s k 0 ) (cid:3) ≤ E   (cid:88) k ∈ [ K ] E (cid:2)

Original: [Equation]
Revised: [Equation]

Original: k
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ≤ E   (cid:88) k ∈

Original: [ K ]

Original: E
Revised: V

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: ≤ H
Revised: s

Original: [Equation]
Revised: [Equation]

Original: k ∈ K (cid:118)(cid:117)(cid:117)(cid:116) E (cid:34)
Revised: [Equation] ] (cid:3) [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: (cid:88) h = min (cid:40)
Revised: k ∈ [ K ]

Original: [Equation]
Revised: [Equation]

Original: (cid:107) [Equation]
Revised: k ) ≤ E   (cid:88)

Original: f [Equation]
Revised: k ∈

Original: [Equation]
Revised: [ K ]

Original: , a
Revised: E (cid:104) ˜

Original: [Equation]
Revised: [Equation]

Original: ) − f

Original: [Equation]
Revised: [Equation]

Original: ( s kh , a
Revised: (cid:105)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: σ,(cid:41)(cid:35) + 0 .

Original: 5 H
Revised: 5 H

Original: [Equation]
Revised: [Table]

Original: where the ﬁrst equality is due to the fact that the total reward for each episode is bounded in [0 , H ] , the second inequality is due to the optimism and our conﬁdence set construction, the third inequality is due to Lemma 13, the fourth inequality is due to Cauchy-Schwartz inequality and the ﬁnal inequality is due to Lemma 17.
Revised: where the ﬁrst equality is due to the fact that the total reward for each episode is bounded in [0 , H ] , the second inequality is due to the optimism and our conﬁdence set construction, the third inequality is due to Lemma 14, the fourth inequality is due to Cauchy-Schwartz inequality and the ﬁnal inequality is due to Lemma 18 , which concludes the proof of Theorem 12.

Original: D B OUNDS ON THE C OMPLEXITY T ERM UNDER L INEAR R EALIZABILITY
Revised: Following the idea of (Russo & Van Roy, 2013; 2014; Osband & Van Roy, 2014), we can translate the worst-case regret bound for UCB algorithm into the expected regret bound for TS algorithm, that conclude the proof of Theorem 5. Remark It can be undesirable that our regret bound scale with σ − 1 , which means our algorithm can perform pretty bad when the noise level is extremely low. It is also more or less counter-intuitive. We want to remark that, such phenomenon is only an artifact introduced by our proof strategy.

Revised: The simulation lemma (Lemma 14) works well when f ( s, a ) − ˜

Revised: f ( s, a ) is small.

Revised: However, we need to tolerate some bad episodes to collect sufﬁcient samples, that can eventually make the error small.

Revised: Fortunately, the regret of such bad episode is at most H .

Revised: Hence, we can use the following strategy to get rid of the dependency on σ − 1 .

Revised: Deﬁnition 20 (Bad and Good Episodes) .

Revised: Deﬁne episode k as a bad episode, if ∃ h ∈ { 0 , 1 , · · · , H − 1 } , such that w k,h := w F k ( s kh , a kh ) is the largest H dim E ( F , σ 2 T 1 / 2 ) elements in the set { w k,h } k ∈

Revised: [ K ] , 0 ≤ h ≤ H − 1 .

Revised: Deﬁne episode k as a good episode, if it is not a bad episode.

Revised: By the deﬁnition, we know there are at most H dim E ( F , σ 2 T − 1 / 2 ) bad episodes.

Revised: We then show the following lemma, that can be directly generalized from Lemma 18, by setting (cid:15)

Revised: =

Revised: σ 2 T − 1 / 2 and remove the terms from bad episodes.

Revised: Lemma 21.

Revised: If { β ∗ k } k ∈ [ K ] is a non-decreasing sequence, and (cid:107) f (cid:107) 2 < C, ∀ f ∈ F , then:

Revised: [Equation]

Revised: Eventually, we can obtain the following regret bound, by setting the regret of bad episodes as H , and bounding the regret of good episodes with Lemma 21.

Revised: Theorem 22 (Improved Regret Bound) .

Revised: Assume Assumption 2 to 5 holds.

Revised: Take α = σ 2 T − 1 / 2 and δ = 0 .

Original: {+25 in Lemma 11, which leads to β ∗ k := 
+}
Revised: {+25 in Lemma 11, which leads to β ∗ k := 
+}

Revised: σ 2 log(4 N ( F , σ 2 T − 1 / 2 , (cid:107) · (cid:107) 2 ))

Revised: + Hσ 2 T − 1 / 2 (12 C + (cid:112) 8 dσ 2 log(16 k 2 H )) .

Revised: We have that

Revised: Regret( K ) ≤ (cid:113) H

Revised: [Equation]

Revised: T (

Revised: [Equation]

Revised: + 1)dim

Revised: [Equation]

Revised: [Equation]

Revised: dim

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: We would like to remark, that the deﬁnition of bad and good episodes is only used for the proof.

Revised: We don’t need to make any modiﬁcation on the algorithm.

Revised: Notice that, as β ∗ k ∝ σ 2 , our upper bound in 22 can only scale with σ − 1 through the logarithm covering number log(4 N ( F , σ 2 T − 1 / 2 and eluder dimension dim E ( F , σ 2 T − 1 / 2 ) .

Revised: When F is a linear function class, both term should scale with polylog( σ ) , that matches the result from (Kakade et al., 2020).

Revised: D B OUNDS ON THE C OMPLEXITY T ERM UNDER L INEAR R EALIZABILITY

Original: We provide the upper bound on the covering number and the eluder dimension of F when F := { θ (cid:62) ϕ : θ ∈ R d ϕ × d , (cid:107) θ (cid:107) 2 ≤ W } where ϕ : S ×
Revised: We provide the upper bound on the covering number and the eluder dimension of F when F := { θ (cid:62) ϕ : θ ∈ R d ϕ × d , (cid:107) θ (cid:107) 2 ≤ W } where ϕ : S ×

Original: A → R d ϕ is some known feature map.
Revised: A → R d ϕ is some known feature map.

Original: We ﬁrst make the following standard assumption:
Revised: We ﬁrst make the following standard assumption:

Original: Assumption 6 (Bounded Feature) .
Revised: Assumption 6 (Bounded Feature) .

Original: [Equation]
Revised: [Equation]

Original:  C OVERING N UMBER
Revised: D. C OVERING N UMBER

Original: Theorem 19 (Covering Number Bound) .
Revised: Theorem 23 (Covering Number Bound) .

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Notice that, by Cauchy-Schwartz inequality, we have that
Revised: Notice that, by Cauchy-Schwartz inequality, we have that

Original: [Equation]
Revised: [Equation]

Original: Thus, denote ε =
Revised: Thus, denote ε =

Original: [ ε i ] i ∈ [ d ] , we have that
Revised: [ ε i ] i ∈ [ d ] , we have that

Original: [Equation]
Revised: [Equation]

Original: Hence, to ﬁnd an (cid:15) -cover for F , we just need to ﬁnd an (cid:15)/B -cover of { θ : θ ∈ R d ϕ × d , (cid:107) θ (cid:107) 2 ≤ W } .
Revised: Hence, to ﬁnd an (cid:15) -cover for F , we just need to ﬁnd an (cid:15)/B -cover of { θ : θ ∈ R d ϕ × d , (cid:107) θ (cid:107) 2 ≤ W } .

Original: By standard argument on the covering number of Euclidean space (e.g. Lemma 5.7 in (Wainwright, 2019)), we can conclude the desired result.
Revised: By standard argument on the covering number of Euclidean space (e.g. Lemma 5.7 in (Wainwright, 2019)), we can conclude the desired result.

Original: D. E LUDER D IMENSION
Revised: D. E LUDER D IMENSION

Original: Theorem 20 (Eluder Dimension Bound) .
Revised: Theorem 24 (Eluder Dimension Bound) .

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Our proof follows the idea in (Russo & Van Roy, 2013).
Revised: Our proof follows the idea in (Russo & Van Roy, 2013).

Original: Deﬁne w k := sup
Revised: Deﬁne w k := sup

Original: [Equation]
Revised: [Equation]

Original: (cid:48) , θ 1 , θ 2 ∈ R d ϕ × d , (cid:107) θ 1 (cid:107) ≤ W, (cid:107) θ 2 (cid:107) ≤ W   .
Revised:   [Equation]

Original: For notation simplicity, deﬁne ϕ
Revised: For notation simplicity, deﬁne ϕ

Original: k :=
Revised: k :=

Original: ϕ
Revised: ϕ

Original: ( s i , a i ) , θ := θ 1 − θ 2 , and Φ k := (cid:80) i ∈ [ k − 1]
Revised: ( s i , a i ) , θ := θ 1 − θ 2 , and Φ k := (cid:80) i ∈ [ k − 1]

Original: ϕ i ϕ (cid:62) i .
Revised: ϕ i ϕ (cid:62) i .

Original: Obviously, we have that (cid:107) θ (cid:107) ≤ 2 W .
Revised: Obviously, we have that (cid:107) θ (cid:107) ≤ 2 W .

Original: Moreover, by straightforward calculation, we know (cid:88)i ∈ [ k − 1] (cid:0) ( [Equation] (cid:62)
Revised: Moreover, by straightforward calculation, we know (cid:88)

Original: [Equation]

Original: i ( s i , a i ) (cid:1)

Original: [Equation]

Original: = Trace(

Original: [Equation]

Original: (cid:62)

Original: [Equation]
Revised: [Equation]

Original: Deﬁne V k :=
Revised: Deﬁne V k :=

Original: Φ k + ( (cid:15) (cid:48) ) 2 4 W 

Revised: Φ k + ( (cid:15) (cid:48) ) 2 4 W

Original: I , we start from considering the problem
Revised: I , we start from considering the problem

Original: [Equation]
Revised: [Equation]

Original: subject to Trace( θ(cid:62)
Revised: subject to Trace( θ

Original: [Equation]
Revised: [Equation]

Original: k θ
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ≤ 2 (cid:15) 2 .

Original: The Lagrangian can be formed as
Revised: The Lagrangian can be formed as

Original: [Equation]
Revised: [Equation]

Original: The optimality condition of θ is
Revised: The optimality condition of θ is

Original: [Equation]
Revised: [Equation]

Original: As V k is of full rank, λV k − ϕ k ϕ (cid:62) k has rank at least d ϕ − 1 (as ϕ k ϕ (cid:62) k is of rank 1 ).
Revised: As V k is of full rank, λV k − ϕ k ϕ (cid:62) k has rank at least d ϕ − 1 (as ϕ k ϕ (cid:62) k is of rank 1 ).

Original: So the equation
Revised: So the equation

Original: [Equation]
Revised: [Equation]

Original: only has one non-zero solution.
Revised: only has one non-zero solution.

Original: Substitute back, we know that (deﬁne (cid:107) x 
Revised: Substitute back, we know that (deﬁne (cid:107) x (cid:107) A

Original: [Equation]
Revised: [Equation]

Original: With the conclusion above
Revised: x (cid:62) Ax ):

Original: [Equation]
Revised: [Equation]

Original:  we have that
Revised: With the conclusion above, we have that

Original: [Equation]
Revised: [Equation]

Original: Hence, if w k ≥ (cid:15) (cid:48) , then ϕ k V − 1 k ϕ k ≥ 0 . 5 .
Revised: Hence, if w k ≥ (cid:15) (cid:48) , then ϕ k V − 1 k ϕ k ≥ 0 . 5 .

Original: Moreover, with Matrix Determinant Lemma, if w i ≥ (cid:15) (cid:48) , ∀ i <
Revised: Moreover, with Matrix Determinant Lemma, if w i ≥ (cid:15) (cid:48) , ∀ i <

Original: k , we have
Revised: k , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: Hence, we know
Revised: Hence, we know

Original: [Equation]
Revised: [Equation]

Original: Now we only need to ﬁnd the largest k that can make this inequality hold.
Revised: Now we only need to ﬁnd the largest k that can make this inequality hold.

Original: For notation simplicity, deﬁne α := 
Revised: For notation simplicity, deﬁne α := 4 W 2 B 2 ( (cid:15) (cid:48) ) 2 , n = k − 1 d . As log(1 + x ) ≥ x 1+ x and log x ≤ x/e , we have n

Original: [Equation]
Revised: [Equation]

Original: W
Revised: log

Original: [Equation]
Revised: [Equation]

Original: B 2 (

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ) 2 , n = k − 1 d . As log(1 + x ) ≥ x 1+ x and log x ≤ x/e , we have
Revised: log(

Original: [Equation]
Revised: [Equation]

Revised: 1) + log n ≤ log( α

Revised: [Equation]

Revised: 1) + log 3 + log( n

Revised: [Equation]

Revised: 3) ≤ log(

Revised: [Equation]

Revised: + 1) + log

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: e.

Original: Substitute back, we can obtain the desired result.
Revised: Substitute back, we can obtain the desired result.

Original: E E XPERIMENTAL D ETAILS E. A LGORITHM S UMMARY
Revised: E E XPERIMENTAL D ETAILS E. A LGORITHM S UMMARY

Original: Our algorithm is easily built on SAC.
Revised: Our algorithm is easily built on SAC.

Original: The only difference we make is we decouple the critic network into a representation network φ ( · ) and a linear layer l ( · ) on top of the representation.
Revised: The only difference we make is we decouple the critic network into a representation network φ ( · ) and a linear layer l ( · ) on top of the representation.

Original: The representation network is governed by the model dynamics loss in SPEDE, and we train a linear layer to predict the Q -value as it lies in the linear space of the representation guaranteed by our analysis.
Revised: The representation network is governed by the model dynamics loss in SPEDE, and we train a linear layer to predict the Q -value as it lies in the linear space of the representation guaranteed by our analysis.

Original: We update the representation by a momentum factor and keep the policy update the same procedure as
Revised: We update the representation by a momentum factor and keep the policy update the same procedure as

Original: SAC.
Revised: SAC.

Original: E. F ULL E XPERIMENTS
Revised: E. F ULL E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: E.
Revised: E.

Original: [Table]
Revised: [Table]

Original: A BLATIONS
Revised: A BLATIONS

Original: Our ablation experiments are trying to study an important design choice of the practical algorithm: the momentum used to update the critic function.
Revised: Our ablation experiments are trying to study an important design choice of the practical algorithm: the momentum used to update the critic function.

Original: We summarize the results in Table 3.
Revised: We summarize the results in Table 3.

Original: We can see that using a small large momentum factor such as 0.999 shows better performance.
Revised: We can see that using a small large momentum factor such as 0.999 shows better performance.

Original: This is intuitively understandable: large momentum factor slows down the update speed of the representation of the critic function and thus stabilize the training.
Revised: This is intuitively understandable: large momentum factor slows down the update speed of the representation of the critic function and thus stabilize the training.

Original: Such phenomenon illustrates the importance of slowly update the representation.
Revised: Such phenomenon illustrates the importance of slowly update the representation.

Original: [Table]
Revised: [Table]

Revised: E. C OMPARISON TO LC

Revised: We provide a comparison of empirical results with LC3 (Kakade et al., 2020), which is also an algorithm with rigorous theoretical guarantees.

Revised: Despite the major difference that we are learning the representation while LC3 assumes a given feature, the performance of SPEDE is much better than

Revised: LC3 in tasks like Mountain Car and Hopper.

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: E. P ERFORMANCE C URVES
Revised: . P ERFORMANCE C URVES

Original: We provide an additional performance curve including ME-TRPO in Figure 2 for a reference.
Revised: We provide an additional performance curve including ME-TRPO in Figure 2 for a reference.

Original: [Figure]
Revised: [Figure]

Original: E. H YPERPARAMETERS
Revised: E. H YPERPARAMETERS

Original: We conclude the hyperparameter we use in our experiments in the following.
Revised: We conclude the hyperparameter we use in our experiments in the following.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]