Original: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning
Revised: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.
Revised: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.

Original: An agent failing to explore systematically will inevitably fail to learn efficiently.
Revised: An agent failing to explore systematically will inevitably fail to learn efficiently.

Original: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.
Revised: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.

Original: An agent following the OFU principle explores actively and efficiently.
Revised: An agent following the OFU principle explores actively and efficiently.

Original: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.
Revised: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.

Original: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.
Revised: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.

Original: BOO is a general algorithm shown to be sample-efficient for finite spectrum RKHS.
Revised: BOO is a general algorithm shown to be sample-efficient for various model classes.

Original: We also developed effective optimization techniques based on natural gradients and entropy regularization.
Revised: We also developed effective optimization techniques based on natural gradients and entropy regularization.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives.
Revised: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives.

Original: In this paper, we consider the RL in Markov decision processes (MDPs), where the agent observes the state of the environment at each timestep and makes decisions accordingly.
Revised: In this paper, we consider the RL in Markov decision processes (MDPs), where the agent observes the state of the environment at each timestep and makes decisions accordingly.

Original: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.
Revised: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.

Original: Exploitation is to make the best-rewarding decision based on the information the agent possesses, and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.
Revised: Exploitation is to make the best-rewarding decision based on the information the agent possesses and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.

Original: An algorithm cannot be sample-efficient without balancing them properly.
Revised: An algorithm cannot be sample-efficient without balancing them properly.

Original: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [1].
Revised: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [1].

Original: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.
Revised: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.

Original: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [2, 3].
Revised: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [2, 3].

Original: Therefore, all practical algorithms resolve this dilemma by achieving a delicate balance between seeking reward and gathering information.
Revised: Therefore, all practical algorithms resolve this dilemma by achieving a delicate balance between seeking reward and gathering information.

Original: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [4–6].
Revised: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [4–6].

Original: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.
Revised: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.

Original: If it ever stands a chance that a policy is highly profitable, the agent will try it out.
Revised: If it ever stands a chance that a policy is highly profitable, the agent will try it out.

Original: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.
Revised: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.

Original: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.
Revised: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.

Original: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [7, 8].
Revised: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [7, 8].

Original: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [8].
Revised: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [8].

Original: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization practical.
Revised: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization practical.

Original: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.
Revised: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.

Original: Our contribution is three-fold:
Revised: Our contribution is three-fold:

Original: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.
Revised: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.

Original: BOO is then proposed for enforcing such requirement.
Revised: BOO is then proposed for enforcing such requirement.

Original: • We prove that BOO is efficient in terms of the frequentist regret for finite spectrum RKHS.
Revised: • We prove that BOO is efficient in terms of the frequentist regret for various model classes.

Original: • We derive an optimization method for BOO based on the natural gradient and entropy regularization.
Revised: • We derive an optimization method for BOO based on the natural gradient and entropy regularization.

Original:  Preliminaries
Revised:  Preliminaries

Original: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.
Revised: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.

Original: 2.1 Reinforcement Learning in Markov Decision Processes
Revised: 2.1 Reinforcement Learning in Markov Decision Processes

Original: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon
Revised: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon

Original: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.
Revised: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.

Original: The algorithm adopted by the agent is denoted as A .
Revised: The algorithm adopted by the agent is denoted as A .

Original: At each episode, the agent is spawned at the initial state s 1 ∈ S .
Revised: At each episode, the agent is spawned at the initial state s 1 ∈ S .

Original: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R
Revised: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R

Original: ( s h , a h ) .
Revised: ( s h , a h ) .

Original: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .
Revised: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .

Original: This process repeats until the episode ends at the H -th period, and another episode begins.
Revised: This process repeats until the episode ends at the H -th period, and another episode begins.

Original: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )
Revised: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )

Original: =
Revised: =

Original: ρ 

Revised: ρ

Original: ( s ) holds for any state s and action a .
Revised: ( s ) holds for any state s and action a .

Original: A finite MDP is an MDP with finite state and action spaces.
Revised: A finite MDP is an MDP with finite state and action spaces.

Original: A policy π is a function mapping a state and a period to an action distribution.
Revised: A policy π is a function mapping a state and a period to an action distribution.

Original: The value function for the MDP M , policy π , and period h is defined recursively as
Revised: The value function for the MDP M , policy π , and period h is defined recursively as

Original: [Equation]
Revised: [Equation]

Original: where ¯ R M ( s, a ) =
Revised: where ¯ R M ( s, a ) =

Original: E r ∼ R M
Revised: E r ∼ R M

Original: ( s,a )
Revised: ( s,a )

Original: [ r ] , and Q π,MH ( s H , a H ) =
Revised: [ r ] , and Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: We denote by t =
Revised: We denote by t =

Original: ( k − 1) H + h
Revised: ( k − 1) H + h

Original: the number of the total time steps at the h -th period of the k -th episode.
Revised: the number of the total time steps at the h -th period of the k -th episode.

Original: The history prior to t is defined as H t =
Revised: The history prior to t is defined as H t =

Original: ( s 1 , a 1 , r 1 , s 2 , . .
Revised: ( s 1 , a 1 , r 1 , s 2 , . .

Original: , s t − 1 , a t − 1 , r t − 1 ) .
Revised: , s t − 1 , a t − 1 , r t − 1 ) .

Original: For notational convenience, we also use X kh to represent X t , where t =
Revised: For notational convenience, we also use X kh to represent X t , where t =

Original: ( k − 1) H + h .
Revised: ( k − 1) H + h .

Original: The agent is assumed to be capable of memorizing the entire history.
Revised: The agent is assumed to be capable of memorizing the entire history.

Original: It produces a policy π k and a model M k per episode in light of the history H k 1 , where k 1 stands for ( k − 1) H +1 .
Revised: It produces a policy π k and a model M k per episode in light of the history H k 1 , where k 1 stands for ( k − 1) H +1 .

Original: In this paper, we only consider cases where the policy π k is an optimal policy of the MDP M k , i.e., π k ∈ arg max π V π,M k
Revised: In this paper, we only consider cases where the policy π k is an optimal policy of the MDP M k , i.e., π k ∈ arg max π V π,M k

Original: ( s 1 ) .
Revised: ( s 1 ) .

Original: For any history H , A ( H ) defines a distribution over models and policies.
Revised: For any history H , A ( H ) defines a distribution over models and policies.

Original: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .
Revised: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .

Original: In the frequentist viewpoint, there exists an unknown true MDP M ∗ , and the performance metric, regret, is defined in terms of this true MDP:
Revised: In the frequentist viewpoint, there exists an unknown true MDP M ∗ , and the performance metric, regret, is defined in terms of this true MDP:

Original: [Equation]
Revised: [Equation]

Original: where V ∗ h
Revised: where V ∗ h

Original: = V π ∗ ,M ∗ h , π ∗ is an optimal policy of the true MDP M ∗ , K is the total number of episodes,
Revised: = V π ∗ ,M ∗ h , π ∗ is an optimal policy of the true MDP M ∗ , K is the total number of episodes,

Original: T =
Revised: T =

Original: KH is the number of the total time steps, and
Revised: KH is the number of the total time steps, and

Original: ∆ k = E [ V ∗ 1 ( s 1 ) − V π k ,M ∗( s 1 )] .
Revised: ∆ k = E [ V ∗ 1 ( s 1 ) − V π k ,M ∗( s 1 )] .

Original: In the Bayesian viewpoint, the unknown MDP is treated as a random variable and assigned a prior ρ M .
Revised: In the Bayesian viewpoint, the unknown MDP is treated as a random variable and assigned a prior ρ M .

Original: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .
Revised: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .

Original: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T ,
Revised: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T ,

Original: BayesRegret( T, A , ρ M ) =
Revised: BayesRegret( T, A , ρ M ) =

Original: E M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .
Revised: E M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .

Original: 2.2 Optimism in the Face of Uncertainty
Revised: 2.2 Optimism in the Face of Uncertainty

Original: Optimism in the face of uncertainty is a strategy for information gathering.
Revised: Optimism in the face of uncertainty is a strategy for information gathering.

Original: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.
Revised: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.

Original: If we make an atrocious guess, we effectively rule that out and pick another next time.
Revised: If we make an atrocious guess, we effectively rule that out and pick another next time.

Original: Otherwise, we end up finding a competitive solution that incurs little regrets.
Revised: Otherwise, we end up finding a competitive solution that incurs little regrets.

Original: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.
Revised: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.

Original: M k ∈ M k , where M k is a confidence set constructed using empirical data H k 1 such that M ∗ sin M k with high probability.
Revised: M k ∈ M k , where M k is a confidence set constructed using empirical data H k 1 such that M ∗ sin M k with high probability.

Original: The pseudocode of the OFU algorithm is shown in Algorithm 1.
Revised: The pseudocode of the OFU algorithm is shown in Algorithm 1.

Original: Algorithm
Revised: Algorithm

Original: RL 1: for episode k = 1 , 2 , . . .
Revised: RL 1: for episode k = 1 , 2 , . . .

Original: do 2:
Revised: do 2:

Original: Construct a confidence set M k with H k3:
Revised: Construct a confidence set M k with H k3:

Original: Compute π k ∈ arg max π max M k V π,M k( s 1 ) s.t.
Revised: Compute π k ∈ arg max π max M k V π,M k( s 1 ) s.t.

Original: M k ∈ M k 4:
Revised: M k ∈ M k 4:

Original: Execute π k for an episode
Revised: Execute π k for an episode

Original:  Related Work
Revised:  Related Work

Original: Our algorithm is designed following a Bayesian perspective.
Revised: Our algorithm is designed following a Bayesian perspective.

Original: It extends frequentist OFU algorithms to Bayesian settings by using a Bayesian credible region.
Revised: It extends frequentist OFU algorithms to Bayesian settings by using a Bayesian credible region.

Original: This idea is first proposed by [9] in bandit problems.
Revised: This idea is first proposed by [9] in bandit problems.

Original: Their method replaces frequentist reward confidence interval with posterior reward quantile and is provably efficient in terms of the frequentist measure of performance.
Revised: Their method replaces frequentist reward confidence interval with posterior reward quantile and is provably efficient in terms of the frequentist measure of performance.

Original: [10] extends their approach to RL by modeling the transition distribution as set-valued Bernoulli distribution.
Revised: [10] extends their approach to RL by modeling the transition distribution as set-valued Bernoulli distribution.

Original: We further generalize this idea to use any Bayesian credible region in place of the frequentist confidence set.
Revised: We further generalize this idea to use any Bayesian credible region in place of the frequentist confidence set.

Original: BOO balances exploration and exploitation via optimizing both the value and the log-posterior density.
Revised: BOO balances exploration and exploitation via optimizing both the value and the log-posterior density.

Original: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [11] and is named as reward-biased maximum likelihood estimation (RBMLE).
Revised: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [11] and is named as reward-biased maximum likelihood estimation (RBMLE).

Original: They have applied this approach to multi-armed bandits [12], contextual bandits [13], and RL where the model belongs to a known finite set [11, 14].
Revised: They have applied this approach to multi-armed bandits [12], contextual bandits [13], and RL where the model belongs to a known finite set [11, 14].

Original: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [15].
Revised: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [15].

Original: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and RKHS.
Revised: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and general model class.

Original: In this regard, BOO could also be referred to as reward-biased maximum a posteriori (RBMAP).
Revised: In this regard, BOO could also be referred to as reward-biased maximum a posteriori (RBMAP).

Original:  Bayesian Optimistic Optimization
Revised:  Bayesian Optimistic Optimization

Original: In this section, we derive the learning objective of BOO as a Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.
Revised: In this section, we derive the learning objective of BOO as a Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.

Original: 4.1 Constrained Bayesian Optimistic Optimization
Revised: 4.1 Constrained Bayesian Optimistic Optimization

Original: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.
Revised: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.

Original: The constrained Bayesian optimistic optimization (CBOO) is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the
Revised: The constrained Bayesian optimistic optimization (CBOO) is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the

Original: Bayesian credible region.
Revised: Bayesian credible region.

Original: A credible region with a 1 − α k level of confidence is a set M k such that Pr( M k |H k 1 ) ≥ 1 − α k , where Pr( ·|H k 1 ) is the posterior distribution given history H k 1 , andPr( M k |H [Equation] =
Revised: A credible region with a 1 − α k level of confidence is a set M k such that Pr( M k |H k 1 ) ≥ 1 − α k , where Pr( ·|H k 1 ) is the posterior distribution given history H k 1 , and

Original: [Equation] M k 
Revised: Pr( M k |H k

Original: [Equation]
Revised: [Equation]

Original:  k Pr [Equation] M k |H 
Revised: = (cid:82) M k ∈M k Pr( M k |H k

Original: [Equation]
Revised: [Equation]

Original: d M k .
Revised: d M k .

Original: By the construction of the credible region, we have M ∗ ∈
Revised: By the construction of the credible region, we have M ∗ ∈

Original: M k holds with probability 1 − α k given any history
Revised: M k holds with probability 1 − α k given any history

Original: H k 1 .
Revised: H k 1 .

Original: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,
Revised: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,

Original: [Equation]
Revised: [Equation]

Original: where V kh =
Revised: where V kh =

Original: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.
Revised: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.

Original: We define a distance metric d ( M 1 , M 2 )
Revised: We define a distance metric d ( M 1 , M 2 )

Original: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .
Revised: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .

Original: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .
Revised: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .

Original: However, designing a value concentration credible region could be intractable for generic model classes.
Revised: However, designing a value concentration credible region could be intractable for generic model classes.

Original: As a reasonable alternative, we propose the highest density region (HDR) [16] or, in the Bayesian context, the highest posterior density (HPD) region [17], i.e.,
Revised: As a reasonable alternative, we propose the highest density region (HDR) [16] or, in the Bayesian context, the highest posterior density (HPD) region [17], i.e.,

Original: M k = { M k | Pr( M k |H k 1 ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k 1 ) ≥ 1 − α k .
Revised: M k = { M k | Pr( M k |H k 1 ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k 1 ) ≥ 1 − α k .

Original: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.
Revised: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.

Original: There are two problems preventing the CBOO from being a practical algorithm.
Revised: There are two problems preventing the CBOO from being a practical algorithm.

Original: The first is that the ϵ k in the definition of the HPD region is unknown.
Revised: The first is that the ϵ k in the definition of the HPD region is unknown.

Original: Although we may approximate it with the
Revised: Although we may approximate it with the

Original: Monte Carlo approximation [18], an estimator of high/low quantiles has high variance rendering the approximation difficult.
Revised: Monte Carlo approximation [18], an estimator of high/low quantiles has high variance rendering the approximation difficult.

Original: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [7].
Revised: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [7].

Original: Algorithm 2 CBOO
Revised: Algorithm 2 CBOO

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 4.2 Lagrangian Relaxation of CBOO
Revised: 4.2 Lagrangian Relaxation of CBOO

Original: By introducing a Lagrange multiplier λ k , we transform the CBOO with HPD regions into an unconstrained optimization problem,
Revised: By introducing a Lagrange multiplier λ k , we transform the CBOO with HPD regions into an unconstrained optimization problem,

Original: [Equation]
Revised: [Equation]

Original: Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k 1 ) are irrelevant, and the optimization is equivalent to max π,M 
Revised: Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k 1 ) are irrelevant, and the optimization is equivalent to max π,M (cid:16)

Original: [Equation]
Revised: [Equation]

Original: This gives rise to the BOO algorithm as shown in Algorithm 3.
Revised: This gives rise to the BOO algorithm as shown in Algorithm 3.

Original: Algorithm
Revised: Algorithm

Original: 1: for episode k =1 , 2 , . . . do 2:
Revised: 1: for episode k = do 2:

Original: [Equation]
Revised: [Equation]

Revised: episode

Original: However, the problem is how to determine the value of λ k .
Revised: However, the problem is how to determine the value of λ k .

Original: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.
Revised: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.

Original: A strict discussion is presented in
Revised: A strict discussion is presented in

Original: Section 5.
Revised: Section 5.

Original: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).
Revised: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).

Original: When only one sample is taken, Algorithmreduces to the well-known posterior sampling for reinforcement learning (PSRL) [19].
Revised: When only one sample is taken, Algorithmreduces to the well-known posterior sampling for reinforcement learning (PSRL) [19].

Original: As the number of samples d goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.
Revised: As the number of samples d goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.

Original: Algorithmvia
Revised: Algorithm

Original: Sampling 1: for episode k
Revised: 1: for episode k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: do 2:
Revised: do 2:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Execute π  an episode
Revised: Execute π k an episode

Original: Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k log Pr( M
Revised: Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k log Pr

Original: [Equation]
Revised: [Equation]

Original: ( s 1 ) .
Revised: ( s 1 ) .

Original: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.
Revised: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.

Original: The thing that matters is their variation.
Revised: The thing that matters is their variation.

Original: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k 1 ) .
Revised: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k 1 ) .

Original: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.
Revised: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.

Original: If dominated by the probability, it shows too strong a preference for high probability models and hesitates to explore.
Revised: If dominated by the probability, it shows too strong a preference for high probability models and hesitates to explore.

Original: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in
Revised: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in

Original: Appendix B).
Revised: Appendix B).

Original: Theorem 4.1 (Variation of the log-posterior density) .
Revised: Theorem 4.1 (Variation of the log-posterior density) .

Original: Suppose X n =
Revised: Suppose X n =

Original: ( X 1 , X 2 , . . .
Revised: ( X 1 , X 2 , . . .

Original: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .
Revised: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .

Original: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.
Revised: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.

Original: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .
Revised: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .

Original: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distribution is
Revised: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distribution is

Original: Pr( X n | θ, A ) is a distribution depending on θ .
Revised: Pr( X n | θ, A ) is a distribution depending on θ .

Original: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log
Revised: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log

Original: Pr( θ | X n ) is at least m/ 2 when n is large enough.
Revised: Pr( θ | X n ) is at least m/ 2 when n is large enough.

Original: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [20].
Revised: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [20].

Original: It is also possible to strengthen this result and derive a convergence of the variance of log-posterior density.
Revised: It is also possible to strengthen this result and derive a convergence of the variance of log-posterior density.

Original: Indeed, it is empirically observed that this quantity converges rapidly to a constant.
Revised: Indeed, it is empirically observed that this quantity converges rapidly to a constant.

Original: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.
Revised: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.

Original: k .
Revised: k .

Original: Otherwise, the algorithm fails to find out the optimal value and policy.
Revised: Otherwise, the algorithm fails to find out the optimal value and policy.

Original: Suppose that the standard deviation of V π ik ,M ikis˜ O ( k − 1 /α ) .
Revised: Suppose that the standard deviation of V π ik ,M ikis O ( k − 1 /α ) .

Original: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k 1 ) matches that of the value term
Revised: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k 1 ) matches that of the value term

Original: V π ik ,M ik .
Revised: V π ik ,M ik .

Original: Given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .
Revised: Indeed, given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .

Original: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.
Revised: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.

Original: That is to say, the optimal policy of some MDPs could be inherently easier to determine than the others.
Revised: That is to say, the optimal policy of some MDPs could be inherently easier to determine than the others.

Original: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.
Revised: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.

Original: For example, consider a ReLU bandit problem, where B = { a ∈ R d |∥ a ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a Gaussian reward of mean max( θ ⊤ a, 0) .
Revised: For example, consider a ReLU bandit problem, where B = { p ∈ R d |∥ p ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a Gaussian reward of mean max( θ ⊤ a, 0) .

Original: If it happens that the agent selects an action a in the inactive region I = { a ∈ B | θ ⊤ a ≤ 0} , the resulting observation reveals little information about the optimal action.
Revised: If it happens that the agent selects an action a in the inactive region where θ ⊤ a ≤ 0 , the resulting observation reveals little information about the optimal action.

Original: Suppose the parameter θ is selected such that the positive regionB − I is of a maximum width ϵ > 0 .
Revised: Suppose the parameter θ is selected such that the positive-reward action region is of a maximum width ϵ > 0 .

Original: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive region, but the uncertainty  starts diminishing rapidly whenever the positive  region is identified.
Revised: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive reward region. However, the uncertainty about the optimal action starts diminishing rapidly whenever the positive reward region is identified.

Original: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .
Revised: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .

Original: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.
Revised: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.

Original: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.
Revised: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.

Original:  Regret Analysis
Revised:  Regret Analysis

Original: This section analyzes the regret of BOO for finite spectrum RKHS.
Revised: This section analyzes the regret of BOO for general model classes.

Original: To begin with, we introduce complexity measurement for function classes.
Revised: To begin with, we introduce complexity measurement for function classes.

Original: 5.1 Complexity Measurement
Revised: 5.1 Complexity Measurement

Original: Our regret analysis uses the notions of the covering number and the eluder dimension [21, 22].
Revised: Our regret analysis uses the notions of the covering number and the eluder dimension [21, 22].

Original: They measure different aspects of function classes.
Revised: They measure different aspects of function classes.

Original: The covering number is introduced for measuring the size of the function class.
Revised: The covering number is introduced for measuring the size of the function class.

Original: Definition 1 (Covering number) .
Revised: Definition 1 (Covering number) .

Original: Let ( G, ∥ · ∥ ) be a metric space and F ⊆ G .
Revised: Let ( M, ∥ · ∥ ) be a metric space and F ⊆ M .

Original: The covering number
Revised: The covering number

Original: N ( F , α, ∥ · ∥ ) for α > 0 is the minimum cardinality of C ⊆ F such that for all f ∈ F , there exists c ∈ C satisfying ∥
Revised: N ( F , α, ∥ · ∥ ) for α > 0 is the minimum cardinality of C ⊆ F such that for all f ∈ F , there exists c ∈ C satisfying ∥

Original: f − c ∥ ≤ α .
Revised: f − c ∥ ≤ α .

Original: The eluder dimension is a complexity measure for the exploration difficulty.
Revised: The eluder dimension is a complexity measure for the exploration difficulty.

Original: Similar to [23], we generalize the original eluder dimension [21] to the distributional eluder dimension by measuring the  difference on a distribution.
Revised: Similar to [23], we generalize the original eluder dimension [21] to the distributional eluder dimension by measuring the ℓ 2 difference on a distribution.

Original: Definition 2 ( ϵ -dependence between distributions [22, 23]) .
Revised: Definition 2 ( ϵ -dependence between distributions [22, 23]) .

Original: Let F be a function class defined ona set
Revised: Let F be a function class defined on

Original: X , and ν, µ 1 , . .
Revised: X , and ν, µ 1 , . .

Original: . , µ n be probability measures over X .
Revised: . , µ n be probability measures over X .

Original: We say ν is ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } w.r.t.
Revised: We say ν is ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } w.r.t.

Original: F if any f 1 , f 2 ∈ F satisfying (cid:112)(cid:80) ni
Revised: F if any f 1 , f 2 ∈ F satisfying (cid:112)(cid:80) ni

Original: =

Revised: =

Original: µ i ( ∥ f 1 − f 2 ∥ 22 )  ≤ ϵ also satisfies ν ( ∥ f 1 − f 2 ∥ 22 ) ≤ ϵ2 .
Revised: µ i ( ∥ f 1 − f 2 ∥ 2 ) 2 ≤ ϵ also satisfies ν ( ∥ f 1 − f 2 ∥ 2 ) ≤ ϵ .

Original: Here, µ ( f ) =
Revised: Here, µ ( f ) =

Original: (cid:82) f d µ .
Revised: (cid:82) f d µ .

Original: If ν is not ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } , then it is said to be ϵ -independent of { µ 1 , µ 2 , . . .
Revised: If ν is not ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } , then it is said to be ϵ -independent of { µ 1 , µ 2 , . . .

Original: , µ n } w.r.t.
Revised: , µ n } w.r.t.

Original: F .
Revised: F .

Original: Let f ∗ be the real function where observations are generated.
Revised: Let f ∗ be the real function where observations are generated.

Original: Intuitively, a distribution ν is ϵindependent of { µ 1 , µ 2 , . . .
Revised: Intuitively, a distribution ν is ϵindependent of { µ 1 , µ 2 , . . .

Original: , µ n } means that a function f indistinguishable from the real function f ∗ on historical distributions could still be significantly different on the distribution ν .
Revised: , µ n } means that a function f indistinguishable from the real function f ∗ on historical distributions could still be significantly different on the distribution ν .

Original: Definition 3 (Distributional eluder (DE) dimension [22, 23]) .
Revised: Definition 3 (Distributional eluder (DE) dimension [22, 23]) .

Original: Let F be a function class defined ona set X , and Π be a class of probability measures over X .
Revised: Let F be a function class defined on X , and Π be a class of probability measures over X .

Original: The distributional eluder dimension DEdim( F , Π , ϵ ) is the length of the longest sequence in Π such that every element in the sequence is ϵ ′ -independent of its predecessors for some ϵ ′ ≥ ϵ .
Revised: The distributional eluder dimension DEdim( F , Π , ϵ ) is the length of the longest sequence in Π such that every element in the sequence is ϵ ′ -independent of its predecessors for some ϵ ′ ≥ ϵ .

Original: The distributional eluder dimension states that the bad event that a historically indistinguishable function f fails to match the real function on future data cannot happen indefinitely.
Revised: The distributional eluder dimension states that the bad event that a historically indistinguishable function f fails to match the real function on future data cannot happen indefinitely.

Original: In the worst case, after DEdim( F , Π , ϵ ) such events, all historically indistinguishable function matches the real function with ϵ -precision.Let ∆ X =
Revised: In the worst case, after DEdim( F , Π , ϵ ) such events, all historically indistinguishable function matches the real function with ϵ -precision.

Original: { δ x | δ x ( y ) = δ ( x − y ) , x, y ∈ X} be the set of Dirac delta distribution centered around each x ∈ X . The distributional eluder dimension equals the original eluder dimension when Π = ∆ X .
Revised: The distributional eluder dimension equals the original eluder dimension when Π is the set of Dirac delta distribution centered around each point in the function’s domain.

Original: In RL, the distribution set we are concerned with is the distributions induced by the models.
Revised: In RL, the distribution set we are concerned with is the distributions induced by the models.

Original: Definition 4 (Probability measures induced by models) .
Revised: Definition 4 (Probability measures induced by models) .

Original: Let M be a set of models and M ∗ be the real
Revised: Let M be a set of models and M ∗ be the real

Original: MDP.
Revised: MDP.

Original: The probability measures induced by M and M ∗ is Π M ∗ M = { ρ M,M ∗ = 1 H (cid:80)
Revised: The probability measures induced by M and M ∗ is Π M ∗ M = { ρ π,M = 1 H (cid:80)

Original: Hh =1 ρ M,M ∗ h | M ∈
Revised: Hh =1 ρ π,Mh | M ∈

Original: M} , where ρ M,M ∗ h denotes the state-action distribution at period h induced by executing an optimal policy π of M in the MDP M ∗ .
Revised: M , π ∈ arg max V π,M( s 1 ) } , where ρ π,Mh denotes the state-action distribution at period h induced by executing an optimal policy π of M in the MDP M ∗ .

Original: We consider the reproducing kernel Hilbert space (RKHS) ball, F = { f  ∈ H|∥
Revised: We consider the following function classes: Linear models: F = { f θ | θ ∈ Θ , f θ ( x ) = θϕ ( x ) , ∀ x ∈ X} , where Θ ⊂ R m × d and ∥ θ ∥ 2 ≤ r θ for all θ ∈ Θ , ϕ ( x ) ∈ R d , and ∥ ϕ ( x ) ∥ 2 ≤ r ϕ for all x ∈ X . 2. Generalized linear model: F = { f θ | θ ∈ Θ , f θ ( x ) =

Original: f ∥ H ≤ r H } , where
Revised: g ( θϕ ( x )) , ∀ x ∈ X} , where Θ ⊂ R m × d , ∥ θ ∥ 2 ≤ r θ for all θ ∈ Θ , ϕ ( x ) ∈ R d , and ∥ ϕ ( x ) ∥ 2 ≤ r ϕ for all x ∈ X . Assume there exist some h and h such that the gradient of g satisfies 0 < h ≤ g ′ ( x ) ≤ h for all x ∈ R d satisfying ∥ x ∥ 2 ≤ r θ r ϕ . 3. Reproducing kernel Hilbert space (RKHS) ball:

Original:  H is vector-valued RKHS on a set X  with a separable kernel K :X × X
Revised: F = { f ∈ H|∥ f ∥ H ≤ r H } , where H is vector-valued RKHS on a Lebesgue space ( X , Σ , µ ) with a separable kernel K :

Original:  → R m × m constructed by K ( x, y ) ij =
Revised: X × X → R m × m constructed by K ( x, y ) ij =

Original: K X ( x, y ) I m ( i, j ) .
Revised: K X ( x, y ) K m ( i, j ) .

Original: Here, K X is a kernel for scalar-valued RKHS on X, and I m ∈ R m × m is an identity matrix.
Revised: Here, K X and K m are kernels for scalar-valued RKHS on X and [ m ] , respectively.

Original: Let F X = { f ∈ H X |∥ f ∥ H X ≤ r H } . The RKHS norm ∥ · ∥ Hof the vector-valued RKHS is ∥ f∥ H = ( (cid:80) mi =1 ∥ f i ∥ 2 H X ) 1 / 2 [24]. For a continuous symmetric positive definite kernel K on X , according to Mercer’s theorem,
Revised: The RKHS norm ∥ · ∥ H is ∥ f

Original: K X ( x, y ) =
Revised: ∥ H =

Original: (cid:80) i ∈
Revised: ( (cid:82) ∥

Original: N λ i e i ( x ) e i ( y ) , where x, y ∈ X , { e i } i are eigenfunctions, and the corresponding eigenvalues { λ i } i are nonnegative.
Revised: f ( x ) ∥ 22 d µ ) 1 / 2 . Table 1 lists the complexity of different function classes measured by eluder dimension and covering number. Here, we extend previous covering number bound for scalar-valued functions [22, 24] to vector-valued functions.

Original: [-For a d -finite spectrum RKHS, the eigenvalues { λ i } i satisfies λ i = 
-] for all i > d .
Revised: [-For a d -finite spectrum RKHS, the eigenvalues { λ i } i satisfies λ i = 
-]The upper bounds for distributional eluder dimension is first established. The complexity of RKHS relies on the eigenvalue decay property of the kernel.

Original: In Appendix A, we prove that the log covering number and the distributional eluder dimension for d -finite spectrum RKHS are O ( md log( √ mr H /α )) and O ( d log( r H /ϵ )) , respectively. Here, the upper bound of the distributional eluder dimension is established for any set of probability measures on X .
Revised: The finite dimensional linear kernel, the squared exponential kernel, and the Martérn kernel are examples of the finite spectrum, exponential decay, and polynomial decay kernels, respectively. The detailed explanation and derivation are in Appendix A.

Original: 5.2 BOO Regret
Revised: 5.2 BOO Regret

Original: This section analyze the regret of BOO.
Revised: This section analyze the regret of BOO.

Original: Instead of general MDPs, we now work with a specific type of MDPs consisting of a model function and an additive sub-Gaussian noise. 
Revised: Instead of general MDPs, we now work with a specific type of MDPs consisting of a model function and an additive sub-Gaussian noise. [Table]

Original: Assumption 1 (Models with additive sub-Gaussian noises) .
Revised: Assumption 1 (Models with additive sub-Gaussian noises) .

Original: Let the state space S be a subset of R m .
Revised: Let the state space S be a subset of R d .

Original: Denote the state-action space S ⊗A as X and state-reward space S ⊗ R as Y .
Revised: Denote the state-action space S ⊗A as X and state-reward space S ⊗ R as Y .

Original: Let F ⊂ Y X be a class of model functions.
Revised: Let F ⊂ Y X be a class of model functions.

Original: For each y ∈ Y , ϵ y is a σ -sub-Gaussian noise ϵ y , i.e., log E [ e v ⊤ ϵ y ] ≤ ∥ v ∥ 22 σ 2 /for
Revised: For each y ∈ Y , ϵ y is a σ -sub-Gaussian noise ϵ y , i.e., log E [ e v ⊤ ϵ y ] ≤ ∥ v ∥ 22 σ 2 /for

Original: all v ∈ R m .
Revised: all v ∈ R d .

Original: There exists a function f ∗ :
Revised: There exists a function f ∗ :

Original: X → Y in F such that for any x ∈ X , f ∗ ( x ) + ϵ f ∗ ( x ) is distributed according to the joint distribution of P M ∗ ( x ) and R M ∗ ( x ) .
Revised: X → Y in F such that for any x ∈ X , f ∗ ( x ) + ϵ f ∗ ( x ) is distributed according to the joint distribution of P M ∗ ( x ) and R M ∗ ( x ) .

Original: Assumption 1 requires the output distribution to be determined by its mean.
Revised: Assumption 1 requires the output distribution to be determined by its mean.

Original: This requirement is satisfied when the output distribution is modeled as, for example, Gaussian with known variance.
Revised: This requirement is satisfied when the output distribution is modeled as, for example, Gaussian with known variance.

Original: The categorical distribution also satisfies this requirement by representing each category as a one-hot vector, in which case the mean vector equates the probability vector of the  distribution.
Revised: The categorical distribution also satisfies this requirement by representing each category as a one-hot vector, in which case the mean is just the probability vector specifying the entire distribution.

Original: Since the noise is assumed sub-Gaussian, we adopt the independent Gaussian likelihood as a surrogate for the actual likelihood function [13], i.e.,
Revised: Since the noise is assumed sub-Gaussian, we adopt the independent Gaussian likelihood as a surrogate for the actual likelihood function [13], i.e.,

Original: [Equation]
Revised: [Equation]

Original: In this way, we avoid restricting our discussion on specific types of distributions.
Revised: In this way, we avoid restricting our discussion on specific types of distributions.

Original: The second assumption is the Lipschitz continuity of the one-step value function, which connects the model difference to the value difference.
Revised: The second assumption is the Lipschitz continuity of the one-step value function, which is needed for connecting the model difference to the value difference.

Revised: Roughly speaking, this requires that when the outputs of model functions are close, the corresponding transition distributions are close.

Original: Definition 5 (One-step Value Lipschitz Continuity) .
Revised: Assumption 2 (One-step Value Lipschitz Continuity) .

Original: Let U x,V ( M ) be the one-step value function with the state-action pair x  and the value function V  , i.e., U x,V ( M ) 
Revised: Let U x,V ( M ) be the one-step value function with the state-action pair x ∈ X and the value function V : S → R , i.e., U x,V ( M ) = ¯ R M ( x )

Original: =
Revised: +

Original: ¯ R M ( x ) + P Mx ( V ) , where P x ( V ) =(cid:82) V ( s [Equation]
Revised: P Mx ( V ) , where P x ( V ) =

Original:  P [Equation] s ′ | x ) d s ′ .
Revised: (cid:82) V ( s ′ ) P ( s ′ | x ) d s ′ .

Original: We define the Lipschitz constant with respect to the 2 -norm as
Revised: We define the Lipschitz constant w.r.t. the 2 -norm as

Original: [Equation]
Revised: [Equation]

Original: where f M is the model function of M .
Revised: where f M is the model function of M .

Original: Let M be a set of MDPs.
Revised: Let M be a set of MDPs.

Original: The global Lipschitz constant w.r.t.
Revised: The global Lipschitz constant w.r.t.

Original: M is L = max x,π,M ∈M L x,V π,M .
Revised: M is L = max x,π,M ∈M L x,V π,M .

Original: Finally, we derive the asymptotic regret of the BOO algorithm in Theorem 5.1.
Revised: Finally, we derive the asymptotic regret of the BOO algorithm in Theorem 5.1.

Original: Notice that the regret of BOO relies on both thedecay rate of the scaling parameter and the complexity of model classes measured by the distributional eluder dimension and the log covering number.
Revised: Notice that the regret of BOO relies on both the complexity of model classes measured by the distributional eluder dimension and the log covering number and the decay rate of the scaling parameter, which conforms to our previous discussion.

Original: Theorem 5.1 (Bayesian optimistic optimization regret) .
Revised: Theorem 5.1 (Bayesian optimistic optimization regret) .

Original: Let the set of model functions F = { f ∈ H|∥
Revised: Assume that ∥ f ( x ) ∥ 2 ≤ r F for all x ∈ X and f ∈ F , and the log-prior probability uniformly bounded by Z .

Original: f ∥ H ≤ r H } be a d -finite spectrum RKHS.
Revised: The distributional eluder dimension is assumed to satisfy DEdim( F , Π M ∗ M , ϵ ) ≤ E (1 /ϵ )

Original: Assume that ∥ f ( x ) ∥ 2≤ r F for all x ∈ X and f ∈ F , and the log-prior probability uniformly bounded by some constant. Setting λ
Revised: e 1 (log(1 /ϵ )) e 2 for e 1 , e 2 ≥ 0 , and the log covering number satisfies log N ( F , α, ∥ · ∥ ∞ ) ≤ C (1 /α )

Original: k
Revised: c

Revised: 1 (log(1 /α ))

Original:  = ck − v for c > 0 and 0 < v 1 < 1 or v = 0 , the frequentist regret of BOO is
Revised: c 2 for c 1 , c 2 ≥ 0 . Then, setting λ k = ck − v for c > 0 , e > 0 , the frequentist regret of BOO is

Original: [Equation]
Revised: [Equation]

Original: where m is the dimension of the state space, v ∗ = 12 , and c r = √ r +1 r
Revised: 6) if v <

Original: [Equation]
Revised: [Equation]

Original: 2 L r F σ 3 √ m +1 for r > 0 .
Revised: 7) if otherwise.

Original: As suggested by Theorem 5.1, the best performance of BOO is achieved by setting v = v ∗ = 1 / 2 .
Revised: As suggested by Theorem 5.1, the best performance of BOO is achieved by setting the decay rate to

Original: Let λ ∗ k = c r / √ k be the optimal scaling parameter. We can further interpret it as λ ∗ k = ξ Vk /ξ Mk , where ξ Mk stands for the variation of the log-posterior density for the model M k , and ξ Vk represents the value uncertainty. In contrast to the discussion in Section 4, where we consider samples from the posterior distribution and the variation of log-posterior density remains constant, the width of the
Revised: [Equation]

Original: HPD region measured by the variation of log-posterior density is, in fact, growing at a rate depending on the log covering number.

Original: Accroding to Lemma C.6 in Appendix C.1, ξ Mk = O (log k ) .

Original: Therefore, we have ξ Vk = λ ∗ k ξ Mk = O (cid:16) log k √ k (cid:17) .

Original: Section 6 will highlight the need of manipulating with ξ Vk and ξ Mk in optimization.

Original:  Optimization
Revised:  Optimization

Original: This section discusses the optimization of the BOO algorithm.
Revised: This section discusses the optimization of the BOO algorithm.

Original: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.
Revised: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.

Original: As discussed previously, the optimal scaling parameter λ k for thisalgorithm isproportional to the value uncertainty, i.e., λ k =
Revised: For infinitedimensional RKHS, this is the only choice.

Original: ξ Vk = λ ∗ k ξ Mk .
Revised: Since PSRL, the single-sample version of Algorithm 4, already achieves the state-of-the-art performance, combining multiple samples may seem meaningless.

Original: The potential advantage of Algorithm 4over PSRL is that, unlike PSRL, it does not require posterior samples to be independent.
Revised: We argue that the potential advantage of Algorithm 4 is that, unlike PSRL, it does not require posterior samples to be independent.

Original: Even the requirement that samples are from the posterior can be relaxed.
Revised: Even the requirement that samples are from the posterior can be relaxed.

Original: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.
Revised: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.

Original: We can also perform the optimization via gradient-based methods.
Revised: For finite-dimensional models, we can optimize via gradient-based methods.

Original: The objective function of BOO consists of two parts.
Revised: The objective function of BOO consists of two parts.

Original: The gradient of the log-posterior density part w.r.t.
Revised: The gradient of the log-posterior density part w.r.t.

Original: the model is just the gradient of the log-prior and the log-likelihood.
Revised: the model is just the gradient of the log-prior and the log-likelihood.

Original: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [25, 26] (proof in Appendix D.1).
Revised: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [25, 26] (proof in Appendix D.1).

Original: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.
Revised: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.

Original: Theorem 6.1 (Value model gradient) .
Revised: Theorem 6.1 (Value model gradient) .

Original: Suppose that the transition function P M and reward function
Revised: Suppose that the transition function P M and reward function

Original: R M of model M is parameterized by θ , the gradient of the value V π,M [Equation] s 1 ) w.r.t.
Revised: R M of model M is parameterized by θ , the gradient of the value V π,M( s 1 ) w.r.t.

Original: the model is 
Revised: the model is ∇

Original: [Equation]
Revised: [Equation]

Original: where τ = ( s 1 , a 1 , . .
Revised: where τ = ( s 1 , a 1 , . .

Original: , s H , a H ) is a trajectory, and
Revised: , s H , a H ) is a trajectory, and

Original: P M ( s h
Revised: P M ( s h

Original: +1{++
+} | s h , a h ) is the probability of s h + under distribution P M
Revised: {++
+} | s h , a h ) is the probability of s h + under distribution P M

Original: ( s h , a h ).
Revised: ( s h , a h )

Original: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [27] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.
Revised: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [27] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.

Original: The natural policy gradient proposed by [28] improves the efficiency of the policy gradient [29] by preconditioning the gradient with the inverse Fisher information matrix.
Revised: The natural policy gradient proposed by [28] improves the efficiency of the policy gradient [29] by preconditioning the gradient with the inverse Fisher information matrix.

Original: We formalize the natural model gradient in a similar fashion.
Revised: We formalize the natural model gradient in a similar fashion.

Original: Let D Rk be normalized empirical state-action distribution for rewards D Rk ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80)
Revised: Let D Rk be normalized empirical state-action distribution for rewards D Rk ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80)

Original: Hh =1 I ( s ih = s, a ih = a ) .
Revised: Hh =1 I ( s ih = s, a ih = a ) .

Original: The normalized empirical state-action distribution for transitions is defined slightly differently as D P k ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80)
Revised: The normalized empirical state-action distribution for transitions is defined slightly differently as D Pk ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80)

Original: H − 1 h
Revised: H − 1 h

Original: [-=
-] I ( s ih = s, a ih = a ) since the transition pair in the H -th period is not observed by definition.
Revised: [-=
-]=1 I ( s ih = s, a ih = a ) since the transition pair in the H -th period is not observed by definition.

Original: Denoting the BOO objective in the k -th episode as J k = λ
Revised: Denoting the BOO objective in the k -th episode as J k = λ

Original: k log Pr( M |H k 1 ) + V π,M( s 1 ) , the natural model gradient is ¯ ∇ θ J k = F − 1 k ∇ θ J k , where F k = E τ ∼ π,M [ 1 H (cid:80) 
Revised: k log Pr( M |H k 1 ) + V π,M( s 1 ) , the natural model gradient is ¯ ∇ θ J k = F − 1 k ∇ θ J k , where F k = E τ ∼ π,M [ 1 H (cid:80) H h

Original: Hh =1 G ¯ Rs h ,a h + 1 H − 1 (cid:80) H − 1 h
Revised: =1 G ¯ Rs h ,a h + 1 H − 1 (cid:80) H − 1 h

Original: =1 F Ps h ,a h ] +
Revised: =1 F Ps h ,a h ] +

Original: E ( s,a ) ∼ D Rk [ F Rs,a ] + E ( s,a ) ∼ D
Revised: E ( s,a ) ∼ D Rk [ F Rs,a ] + E ( s,a ) ∼ D

Original: Pk [ F Ps,a ] .
Revised: Pk [ F Ps,a ] .

Original: Here, F Ps,a = E s ′ ∼ P M
Revised: Here, F Ps,a = E s ′ ∼ P M

Original: ( s h ,a h ) [( ∇ θ log P M
Revised: ( s h ,a h ) [( ∇ θ log P M

Original: ( s ′ | s h , a h )) 2 ] and
Revised: ( s ′ | s h , a h )) 2 ] and

Original: F Rs,a = E r ∼ R M ( s h ,a h ) [( ∇ θ log R M ( r | s h , a h )) 2 ] are the Fisher information matrix of transition and reward distributions.
Revised: F Rs,a = E r ∼ R M ( s h ,a h ) [( ∇ θ log R M ( r | s h , a h )) 2 ] are the Fisher information matrix of transition and reward distributions.

Original: The matrix G ¯ Rs,a = ∇ θ z ⊤ R ∇ 2 z R ¯ R M ( s, a ) ∇ θ z R is the generalized Gauss-Newton matrix [30], where the reward distribution is parameterized by z R ⊆
Revised: The matrix G ¯ Rs,a = ∇ θ z ⊤ R ∇ 2 z R ¯ R M ( s, a ) ∇ θ z R is the generalized Gauss-Newton matrix [30], where the reward distribution is parameterized by z R ⊆

Original: z , and z = f θ ( s, a ) is the output of the model function f θ .
Revised: z , and z = f θ ( s, a ) is the output of the model function f θ .

Original: Note that we have G ¯ Rs,a = 0 when the mean reward ¯ R M ( s, a ) is linear in z .
Revised: Note that we have G ¯ Rs,a = 0 when the mean reward ¯ R M ( s, a ) is linear in z .

Original: This is satisfied for most model classes, for example, those specified by Assumption 1.
Revised: This is satisfied for most model classes, for example, those specified by Assumption 1.

Original: Therefore, commonly, F k contains only the Fisher information matrix, and the natural gradient can be interpreted geometrically as the steepest descent direction in the information geometry induced by F k instead of in the Euclidean geometry.
Revised: Therefore, commonly, F k contains only the Fisher information matrix, and the natural gradient can be interpreted geometrically as the steepest descent direction in the information geometry induced by F k instead of in the Euclidean geometry.

Original: Loosely speaking, the distance in the information geometry is measured by the square root of KL divergence between distributions rather than the ℓ 2 distance of parameters [31].
Revised: Loosely speaking, the distance in the information geometry is measured by the square root of KL divergence between distributions rather than the ℓ 2 distance of parameters [31].

Original: The natural gradient characterizes some global characteristics of the loss surface and therefore facilitates global convergence.
Revised: The natural gradient characterizes some global characteristics of the loss surface and therefore facilitates global convergence.

Original: We refer interested readers to [31] for a thorough discussion on the interpretation and implementation of the natural gradient.
Revised: We refer interested readers to [31] for a thorough discussion on the interpretation and implementation of the natural gradient.

Original: Another intricacy of this problem is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.
Revised: Another intricacy of this problem is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.

Original: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.
Revised: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.

Original: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.
Revised: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.

Original: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.
Revised: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.

Original: The smoothing effect is also observed previously in [32].
Revised: The smoothing effect is also observed previously in [32].

Original: The entropy-regularized learning objective mimics the maximum entropy RL [33]:
Revised: The entropy-regularized learning objective mimics the maximum entropy RL [33]:

Original: [Equation]
Revised: [Equation]

Original: where KL( p ∥ q ) stands for the relative entropy, ˆ π h is a prior policy ensuring π h is absolutely continuous w.r.t.
Revised: where KL( p ∥ q ) stands for the relative entropy, ˆ π h is a prior policy ensuring π h is absolutely continuous w.r.t.

Original: ˆ π h , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.
Revised: ˆ π h , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.

Original: Here, the entropy term is downscaledin proportion to the value uncertainty such that the influence of regularization diminishes with time.
Revised: Here, the entropy term is downscaled such that the influence of regularization diminishes with time.

Original: Consider the case that there exists a ϵ -optimal policy π ϵ for any fixed model M such that, for some b > 0 , E τ ∼ π ϵ ,M 
Revised: Consider the case that there exists a ϵ -optimal policy π ϵ for any fixed model M such that, for some b > 0 , E τ ∼ π ϵ ,M (cid:104)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: Hh
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( π ϵh
Revised: KL

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: )) (cid:105) ≤ b and sup π ∗ V π ∗ ,M( s 1 ) − V π ϵ ,M( s 1 ) ≤ ϵ . Then, the value of the maximum entropy policy
Revised: s [Equation]

Original: ˜ π
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: = arg max
Revised: and sup

Original: [Equation]
Revised: [Equation]

Original: ( s 1 ) satisfies that
Revised: ( s 1 ) ≤ ϵ .

Revised: Then, the value of the maximum entropy policy ˜ π ∗ = arg max ˜ π V ˜ π,M( s 1 ) satisfies that

Original: [Equation]
Revised: [Equation]

Original: If ϵ is sufficiently small, then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty, which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance.
Revised: If ϵ is sufficiently small [Equation] then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty, which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance [Equation]

Original: The entropy-regularized value is equivalently defined by the following Bellman backup,
Revised: The entropy-regularized value is equivalently defined by the following Bellman backup,

Original: [Equation]
Revised: [Equation]

Original: where ˜ Q π,MH ( s H , a H ) =
Revised: where ˜ Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [33] or the soft actor-critic algorithm [34].
Revised: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [33] or the soft actor-critic algorithm [34].

Original: In finite MDPs, the optimal value is given by soft value iteration,
Revised: In finite MDPs, the optimal value is given by soft value iteration,

Original: [Equation]
Revised: [Equation]

Original: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as
Revised: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as

Original: [Equation]
Revised: [Equation]

Original: The gradient of the entropy-regularized value w.r.t.
Revised: The gradient of the entropy-regularized value w.r.t.

Original: the model is similar to that specified in Theorem 6.1, with the value replaced by the entropy-regularized value.
Revised: the model is similar to that specified in Theorem 6.1, with the value replaced by the entropy-regularized value.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes BOO as a generic model-based reinforcement learning algorithm.
Revised: This paper proposes BOO as a generic model-based reinforcement learning algorithm.

Original: It enjoys ˜ O ( √ K ) regret for finite spectrum RKHS.
Revised: It enjoys sublinear regret for a variety of model classes.

Original: To optimize the BOO objective, we develop a natural model gradient that facilitates learning by optimizing the model in the space of distributions.
Revised: To optimize the BOO objective, we develop a natural model gradient that facilitates learning by optimizing the model in the space of distributions.

Original: We also propose to use an entropy-regularized objective that smooths the loss surface and makes BOO amenable to incremental implementation.
Revised: We also propose to use an entropy-regularized objective that smooths the loss surface and makes BOO amenable to incremental implementation.

Original: However, to apply BOO in real-world reinforcement learning problems, it is essential to adapt the scaling parameter of BOO on a per-instance basis, which is an appealing direction for future work. 
Revised: However, to apply BOO in real-world reinforcement learning problems, it is essential to adapt the scaling parameter of BOO on a per-instance basis, which is an appealing direction for future work. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Revised: [No] No relevant potential negative impacts is identified.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [Yes] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]