Original: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning
Revised: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.
Revised: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.

Original: An agent failing to explore systematically will inevitably fail to learn efficiently.
Revised: An agent failing to explore systematically will inevitably fail to learn efficiently.

Original: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.
Revised: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.

Original: An agent following the OFU principle explores actively and efficiently.
Revised: An agent following the OFU principle explores actively and efficiently.

Original: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.
Revised: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.

Original: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.
Revised: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.

Original: BOO is a general algorithm shown to be sample-efficient for various model classes.
Revised: BOO is a general algorithm proved to be sample-efficient for models in a finite-dimensional reproducing kernel

Original:  We also developed effective optimizationtechniques based on natural gradients and entropy regularization.
Revised: Hilbert space. We also develop techniques for effective optimization and show through some simulation experiments that BOO is competitive with the existing algorithms.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives.
Revised: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives [1, 2].

Original: In this paper, we consider the RL in Markov decision processes (MDPs), where the agent observes the state of the environment at each timestep and makes decisions accordingly.
Revised: In this paper, we consider the RL in Markov decision processes (MDPs) where the agent observes the state of the environment at each timestep and makes decisions accordingly.

Original: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.
Revised: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.

Original: Exploitation is to make the best-rewarding decision based on the  informationthe agent possesses and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.
Revised: Exploitation is to make the best-rewarding decision based on the agent’s information and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.

Original: An algorithm cannot be sample-efficient without balancing them properly.
Revised: An algorithm cannot be sample-efficient without balancing them properly.

Original: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [1].
Revised: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [3].

Original: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.
Revised: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.

Original: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [2, 3].
Revised: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [4–6].

Original: Therefore, all practical algorithms  resolve this dilemma by achieving a delicate balance between seeking reward and gathering information.
Revised: Therefore, all practical algorithms [7–10] resolve this dilemma by achieving a delicate balance between seeking rewards and gathering information.

Original: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [4–6].
Revised: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [11–13].

Original: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.
Revised: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.

Original: If it ever stands a chance that a policy is highly profitable, the agent will try it out.
Revised: If it ever stands a chance that a policy is highly profitable, the agent will try it out.

Original: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.
Revised: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.

Original: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.
Revised: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.

Original: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [7, 8].
Revised: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [14, 15].

Original: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [8].
Revised: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [15].

Original: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization  practical.
Revised: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization more practical.

Original: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.
Revised: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.

Original: Our contribution is three-fold:
Revised: Our contribution is three-fold:

Original: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.
Revised: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.

Original: BOO is then proposed for enforcing such requirement.
Revised: BOO is then proposed as the Lagrangian relaxation of this constrained optimization.

Original: • We prove that BOO is efficient in terms of the frequentist regret for various model classes.
Revised: • We prove that BOO is efficient in terms of the frequentist regret for finite-dimensional reproducing kernel Hilbert space (RKHS).

Original: • We derivean optimization method for BOObased on the natural gradient and entropy regularization.
Revised: • We derive optimization methods for BOO and demonstrate empirical evidence that BOO is competitive with UCRL2 [16] and PSRL [17].

Original:  Preliminaries
Revised:  Preliminaries

Original: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.
Revised: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.

Original: 2.1 Reinforcement Learning in Markov Decision Processes
Revised: 2.1 Reinforcement Learning in Markov Decision Processes

Original: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon
Revised: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon

Original: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.
Revised: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.

Original: The algorithm adopted by the agent is denoted as A .
Revised: The algorithm adopted by the agent is denoted as A .

Original: At each episode, the agent is spawned at the initial state s 1 ∈ S .
Revised: At each episode, the agent is spawned at the initial state s 1 ∈ S .

Original: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R
Revised: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R

Original: ( s h , a h ) .
Revised: ( s h , a h ) .

Original: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .
Revised: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .

Original: This process repeats until the episode ends at the H -th period, and another episode begins.
Revised: This process repeats until the episode ends at the H -th period, and another episode begins.

Original: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )
Revised: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )

Original: =
Revised: =

Original: ρ 

Revised: ρ

Original: ( s ) holds for any state s and action a .
Revised: ( s ) holds for any state s and action a .

Original: A finite MDP is an MDP with finite state and action spaces.
Revised: A finite MDP is an MDP with finite state and action spaces.

Original: A policy π is a function mapping a state and a period to an action distribution.
Revised: A policy π is a function mapping a state and a period to an action distribution.

Original: The value function for the MDP M , policy π , and period h is defined recursively as
Revised: The value function for the MDP M , policy π , and period h is defined recursively as

Original: [Equation]
Revised: [Equation]

Original: where ¯ R M ( s, a ) =
Revised: where ¯ R M ( s, a ) =

Original: E r ∼ R M
Revised: E r ∼ R M

Original: ( s,a )
Revised: ( s,a )

Original: [ r ] , and Q π,MH ( s H , a H ) =
Revised: [ r ] , and Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: We denote by t = ( k − 1) H + h the number of the total time steps at the h -th period of the k -th episode.
Revised: We use X k,h to represent the variable X at the h -th period of the k -th episode.

Revised: For notational convenience, X k,h is sometimes abbreviated as X kh .

Original: The history prior to t is defined as H t =
Revised: The history prior to k -th episode is defined as H k =

Original: ( s 1 ,  a 1 ,r 1 , s 2 , . . , st − 1 , a t − 1 , r t − 1 ) .
Revised: ( s 1 , 1 , a 1 , 1 , r 1 , 1 , s 1 , 2 , . . .

Original: For notational convenience, we also use X kh to represent X t , where t = ( k − 1) H + h .
Revised: , s k − 1 ,H , a k − 1 ,H , r k − 1 ,H ) .

Original: The agent is assumed to be capable of memorizing the entire history.
Revised: The agent is assumed to be capable of memorizing the entire history.

Original: It produces a policy π k and a model M k per episode in light of the history H k 1 , where k 1 stands for ( k − 1) H +1 . In this paper, we only consider cases where the policy π k is an optimal policyof the MDP M k , i.e., π k ∈ arg max π V π,M k
Revised: In this paper, we consider model-based reinforcement learning algorithms which produce a model M k per episode in light of the history H k and derive its corresponding optimal policy π k ∈ arg max π V π,M k

Original: ( s 1 ) .
Revised: ( s 1 ) for execution.

Revised: We use the terms model and

Revised: MDP interchangeably.

Original: For any history H , A ( H ) defines a distribution over models and policies.
Revised: For any history H , A ( H ) defines a distribution over models and policies.

Original: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .
Revised: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .

Original: In the frequentist viewpoint, there exists an unknown true MDP M ∗  , and the  performance metric, regret, is defined in terms of this true MDP:
Revised: In the frequentist viewpoint, there exists an unknown true MDP M ∗ . We abbreviate V π ∗ ,M ∗ h as V ∗ h , where π ∗ is an optimal policy of the true MDP M ∗ . The frequentist performance metric, regret, is defined in terms of the true MDP:

Original: [Equation]
Revised: [Equation]

Original: where  V ∗ h = V π ∗ ,M ∗ h , π ∗ is an optimal policy of the true MDP M ∗ , K is the total number of episodes,
Revised: where ∆ k is defined as E π k ∼ A ( H k ) (cid:104) V ∗ 1 ( s 1 ) − V π k ,M ∗( s 1 ) (cid:105) , K is the total number of episodes,

Original: T = KH is the number of the total time steps, and ∆ k = E [ V ∗1 ( s 1 ) − V π k ,M ∗( s 1 )] .
Revised: T = KH is the number of the total time steps, and H K +1 ∼ A , M ∗ means that the history is sampled by the interaction of the algorithm A and the real MDP M ∗ .

Original: In the Bayesian viewpoint, the unknown MDP  is treated as a random variable and assigned a prior ρ M .
Revised: In the Bayesian viewpoint, the unknown MDP M ∗ is treated as a random variable and assigned a prior ρ M .

Original: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .
Revised: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .

Original: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T , BayesRegret( T, A , ρ M ) =
Revised: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T , BayesRegret( T, A , ρ M ) =

Original: E M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .
Revised: E M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .

Original: 2.2 Optimism in the Face of Uncertainty
Revised: 2.2 Optimism in the Face of Uncertainty

Original: Optimism in the face of uncertainty is a strategy for information gathering.
Revised: Optimism in the face of uncertainty is a strategy for information gathering.

Original: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.
Revised: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.

Original: If we make an atrocious guess, we effectively rule that out and pick another next time.
Revised: If we make an atrocious guess, we effectively rule that out and pick another next time.

Original: Otherwise, we end up finding a competitive solution that incurs little regrets.
Revised: Otherwise, we end up finding a competitive solution that incurs little regrets.

Original: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.
Revised: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.

Original: M k ∈ M k , where M k is a confidence set constructed using empirical data H k1 such that M ∗ sin M k with high probability.
Revised: M k ∈ M k , where M k is a confidence set constructed using empirical data H k such that M ∗ ∈ M k with high probability.

Original: The pseudocode of the OFU algorithm is shown in Algorithm 1.Algorithm
Revised: The pseudocode of the OFU algorithm is shown in Algorithm 1.

Original: RL 1: for episode k = 1 , 2 , . . .
Revised: 1: for episode k = 1 , 2 , . . .

Original: do 2:
Revised: do 2:

Original: Construct a confidence set M k with H k3:
Revised: Construct a confidence set M k with H k 3:

Original: Compute π k ∈ arg max π max M k V π,M k( s 1 ) s.t.
Revised: Compute π [Equation] ∈ [Equation]

Original: M k ∈ M k 4:
Revised: [Equation]

Original: Execute π k for an episode
Revised: Execute π k for an episode

Original:  Related Work
Revised:  Related Work

Original: Our algorithm is designed following a Bayesian perspective.
Revised: One line of research resolves the exploration and exploitation dilemma by formalizing the reinforcement learning problem as a planning problem in Bayes-Adaptive MDPs (BAMDPs) [3], which treats the unknown MDP parameter as an additional hidden variable and maintains a belief distribution of the parameter. This line of work shares the scalability problem, which is caused by the exponential increase of possible histories w.r.t. the planning horizon. That is, the planning in BAMDPs is

Original: It extends frequentist OFU algorithms to Bayesian settings by using a Bayesian credible region.
Revised: PSPACE-complete [4] and requires exponential time to solve. Intuitively, this is because methods based on BAMDPs deal with the belief distributions of MDPs (or histories) rather than a single MDP. Efficient planning algorithms in BAMDPs do exist. Exploiting the root sampling technique, we can implement an algorithm that gets rid of the posterior distribution and only requires posterior sampling [18].

Revised: Nonetheless, the underlying scalability issue remains prominent.

Original: This idea is first proposed by [9] in bandit problems.
Revised: Methods following the OFU principle construct an optimistic estimate of the unknown MDP and execute its optimal policy. Since these methods plan on a single MDP estimate, they are computationally preferable compared to methods based on BAMDPs. The UCRL2 [16] is one such method. However, unlike tabular and linear MDPs for which constructing an optimistic estimate is analytically tractable [11, 13], constructing an optimistic estimate for general MDPs involves a constrained joint optimization of model and policy and is computationally prohibitive.

Original: Their method replaces frequentist reward confidence interval with posterior reward quantile and  is provably efficient in terms of the frequentist measure of performance.
Revised: Hence, previous model-based OFU algorithms [19, 20] for general model classes cannot be implemented and rely on posterior sampling for exploration. The posterior sampling for reinforcement learning (PSRL) works by selecting a random MDP from the posterior distribution and executing its optimal policy [21]. This strategy ensures that a policy is selected according to the probability that it is the optimal policy of the real model.

Original: [10] extends their approach to RL by modeling the transition distribution as set-valued Bernoulli distribution.
Revised: It is shown that PSRL is at least as good as any frequentist OFU algorithm in terms of Bayesian regret [22, 15]. PSRL is argued to be better than optimism since it is more computationally tractable [15]. Nevertheless, as opposed to OFU algorithms, incremental implementation of PSRL is challenging because it requires replanning after each sampling.

Original: We further generalize this idea to use any Bayesian credible region in place of the frequentist confidence set.
Revised: Practical implementation instead tries to directly sample from the posterior distribution of the optimal value function [23–25]. However, the theoretical guarantee for these methods is only established for tabular MDPs [25]. H-UCRL was devoted to resolving the intractability of model-based optimistic exploration for general models [26]. It proposes to convert the joint optimization of model and policy into a hallucinated control problem.

Original: BOO balances exploration and exploitation via optimizing both the value and the log-posterior density. 
Revised: This approach ignores the correlation between state-action pairs and treats them separately, causing inefficiency as reflected by the extra dependency on the cumulative posterior variance in their regret bound. Our method, BOO, also tries to attack the intractability of optimism, which builds an optimistic model by optimizing both the value and the log-posterior density. It avoids the defect of H-UCRL since maximizing the log-posterior density naturally enforces the correlation between state-action pairs.

Original: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [11] and is named as reward-biased maximum likelihood estimation (RBMLE).
Revised: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [27] and is named as reward-biased maximum likelihood estimation (RBMLE).

Original: They have applied this approach to multi-armed bandits [12], contextual bandits [13], and RL where the model belongs to a known finite set [11, 14].
Revised: They have applied this approach to multi-armed bandits [28], contextual bandits [29], and RL where the model belongs to a known finite set [27, 30].

Original: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [15].
Revised: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [31].

Original: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and general model class.
Revised: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and finite dimensional RKHS.

Original: In this regard, BOO could also be referred to as reward-biased maximum a posteriori (RBMAP).
Revised: In this regard, BOO could also be referred to as reward-biased maximum a posteriori.

Revised: Concerning the regret analysis, previous regret analysis for general model-based reinforcement learning [19, 20] relies on the fact that the constructed model is the most optimistic one in the confidence set.

Revised: The regret analysis of BOO differs with them significantly since the model constructed by BOO may not belong to a confidence set or a credible region.

Revised: This difference entails a distinct analysis, where we show that neither the large deviation from the real model nor the possibly pessimistic estimation of the model causes a large regret.

Original:  Bayesian Optimistic Optimization
Revised:  Bayesian Optimistic Optimization

Original: In this section, we derive the learning objective of BOO as a Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.
Revised: In this section, we derive the learning objective of Bayesian optimistic optimization (BOO) as a Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.

Revised: Assuming the model class resides in a finite-dimensional RKHS, we show that BOO enjoys ˜ O ( √ K ) regret.

Original: 4.1 Constrained Bayesian Optimistic Optimization
Revised: 4.1 Constrained BOO

Original: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.
Revised: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.

Original: The constrained Bayesian optimistic optimization (CBOO) is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the Bayesian credible region.
Revised: The constrained BOO is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the Bayesian credible region.

Original: A credible region with a 1 − α k level of confidence is a set M k such that Pr( M k |H k1 ) ≥ 1 − α k , where Pr( ·|H k1 ) is the posterior distribution given history H k1 , andPr( M k |H k
Revised: A credible region with a 1 − α k level of confidence is a set M k such that Pr( M k |H k ) ≥ 1 − α k , where Pr( ·|H k ) is the posterior distribution given history H k , and

Original: [Equation]

Original:  = (cid:82) M k ∈M k Pr( M k |H k [Equation] d M k .
Revised: Pr( M k |H k ) = (cid:82) M k ∈M k Pr( M k |H k ) d M k .

Original: By the construction of the credible region, we have M ∗ ∈
Revised: By the construction of the credible region, we have M ∗ ∈

Original: M k holds with probability 1 − α k given any history H k1 .
Revised: M k holds with probability 1 − α k given any history H k .

Original: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,
Revised: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,

Original: [Equation]
Revised: [Equation]

Original: where V kh =
Revised: where V kh =

Original: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.
Revised: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.

Original: We define a distance metric d ( M 1 , M 2 )
Revised: We define a distance metric d ( M 1 , M 2 )

Original: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .
Revised: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .

Original: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .
Revised: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .

Original: However, designing a value concentration credible region could be intractable for generic model classes.
Revised: However, designing a value concentration credible region could be intractable for generic model classes.

Original: As a reasonable alternative, we propose the highest density region (HDR) [16] or, in the Bayesian context, the highest posterior density (HPD) region [17], i.e.,
Revised: As a reasonable alternative, we propose the highest density region (HDR) [32] or, in the Bayesian context, the highest posterior density (HPD) region [33], i.e.,

Original: M k = { M k | Pr( M k |H k1 ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k1 ) ≥ 1 − α k .
Revised: M k = { M k | Pr( M k |H k ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k ) ≥ 1 − α k .

Original: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.
Revised: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.

Original: There are two problems preventing the CBOO from being a practical algorithm.
Revised: There are two problems preventing the constrained BOO from being a practical algorithm.

Original: The first is that the ϵ k in the definition of the HPD region is unknown.
Revised: The first is that the ϵ k in the definition of the HPD region is unknown.

Original: Although we may approximate it with the Monte Carlo approximation [18], an estimator of high/low quantiles has high variance rendering the approximation difficult.
Revised: Although we may approximate it with the Monte Carlo approximation [34], an estimator of high/low quantiles has high variance rendering the approximation difficult.

Original: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [7].
Revised: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [14].

Original: Algorithm 2 CBOO
Revised: Algorithm 2 Constrained BOO t

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation]
Revised: M

Original: [Equation]
Revised: [Equation]

Revised: ∈

Original: 4.2  Lagrangian Relaxation of CBOO
Revised: 4.2 BOO as Lagrangian Relaxation of Constrained BOO

Original: By introducing a Lagrange multiplier λ k , we transform the CBOO with HPD regions into an unconstrained optimization problem, 
Revised: By introducing a Lagrange multiplier λ k , we transform the constrained BOO with HPD regions into an unconstrained optimization problem, max π,M (cid:16) V π,M

Original: [Equation]
Revised: [Equation]

Original:  Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k1 ) are irrelevant, and the optimization is equivalent to max π,M (cid:16) 
Revised: where Pr( M |H k ) = Pr( H k | M )Pr( M ) Pr( H k ) . Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k ) are irrelevant, and the optimization is equivalent to max π,M (cid:16) V π,M

Original: [Equation]
Revised: [Equation]

Revised: + λ k (log Pr( H k | M ) + log Pr( M )) (cid:17) .

Original: This gives rise to the BOO algorithm as shown in Algorithm 3.
Revised: This gives rise to the BOO algorithm as shown in Algorithm 3.

Original: Algorithm 1: for episode k = do 2:
Revised: Algorithm 3 BOO

Original: [Equation]
Revised: [Equation]

Original: episode

Original: However, the problem is how to determine the value of λ k .
Revised: However, the problem is how to determine the value of λ k .

Original: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.
Revised: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.

Original: A strict discussion is presented in
Revised: A strict discussion is presented in

Original: Section 5.
Revised: Appendix B.

Original: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).
Revised: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).

Original: When only one sample is taken, Algorithmreduces to the well-known posterior sampling for reinforcement learning (PSRL) [19].
Revised: When only one sample is taken, Algorithm 4 reduces to the well-known PSRL [21].

Original: As the number of samples d goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.
Revised: As the number of samples j goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.

Original: Algorithm 1: for episode k [Equation] [Equation]
Revised: Algorithm 4 BOO via Posterior Sampling

Original: do 2:
Revised: arg

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Execute π k

Original: an episode Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k log Pr
Revised: Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k [Equation] Pr

Original: [Equation]
Revised: [Equation]

Original: ( s 1 ) .
Revised: ( s 1 ) .

Original: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.
Revised: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.

Original: The thing that matters is their variation.
Revised: The thing that matters is their variation.

Original: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k1 ) .
Revised: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k ) .

Original: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.
Revised: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.

Original: If  dominated by the probability, it showstoo strong a  preference for high probability models and hesitates to explore.
Revised: If the variation is dominated by the probability, it shows a strong preference for high-probability models and hesitates to explore.

Original: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in
Revised: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in

Original: Appendix B).
Revised: Appendix A).

Original: Theorem 4.1 (Variation of the log-posterior density) .
Revised: Theorem 4.1 (Variation of the log-posterior density) .

Original: Suppose X n =
Revised: Suppose X n =

Original: ( X 1 , X 2 , . . .
Revised: ( X 1 , X 2 , . . .

Original: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .
Revised: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .

Original: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.
Revised: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.

Original: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .
Revised: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .

Original: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distributionis
Revised: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distribution

Original: Pr( X n | θ, A ) is a distribution depending on θ .
Revised: Pr( X n | θ, A ) is a distribution depending on θ .

Original: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log
Revised: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log

Original: Pr( θ | X n ) is at least m/ 2 when n is large enough.
Revised: Pr( θ | X n ) is at least m/ 2 when n is large enough.

Original: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [20].
Revised: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [35].

Revised: The theorem only certifies a lower bound on the variance rather than establishing its convergence.

Revised: Arguably, this is mostly a technical issue arising from the unboundedness of the log-posterior density.

Original: It isalso possible to strengthen this result and derive a convergence of the variance of log-posterior density.
Revised: It is possible to strengthen this result and derive the convergence of the variance.

Original: Indeed, it is empirically observed that this quantity converges rapidly to a constant.
Revised: Indeed, it is empirically observed that this quantity converges rapidly to a constant.

Original: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.
Revised: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.

Original: k .
Revised: k .

Original: Otherwise, the algorithm fails to find out the optimal value and policy.
Revised: Otherwise, the algorithm fails to find out the optimal value and policy.

Original: Suppose that the standard deviation of V π ik ,M ikis  O ( k − 1 /α ) .
Revised: Suppose that the standard deviation of V π ik ,M ikis ˜ O ( k − 1 /α ) , where ˜ O is a variant of the big O notation that ignores logarithmic factors.

Original: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k1 ) matches that of the value term V π ik ,M ik .
Revised: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k ) matches that of the value term V π ik ,M ik .

Original: Indeed, given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .
Revised: Given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .

Original: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.
Revised: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.

Original: That is to say, the optimal policy of some MDPs could be inherently easier to determine than the others.
Revised: That is to say, the optimal policy of some MDPs could be inherently easier to determine than the others.

Original: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.
Revised: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.

Original: For example, consider a ReLU bandit problem, where B = { p ∈ R d |∥ p ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a Gaussian reward of mean max( θ ⊤ a, 0) .
Revised: For example, consider a ReLU bandit problem, where B = { a ∈ R d |∥ a ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a Gaussian reward of mean max( θ ⊤ a, 0) .

Original: If it happens that the agent selects an action a in the inactive region where θ ⊤ a ≤ 0  , the resulting observation reveals little information about the optimal action.
Revised: If it happens that the agent selects an action a in the inactive region I = { a ∈ B | θ ⊤ a ≤ 0 } , the resulting observation reveals little information about the optimal action.

Original: Suppose the parameter θ is selected such that the positive-reward action region  is of a maximum width ϵ > 0 .
Revised: Suppose the parameter θ is selected such that the positive region B \ I is of a maximum width ϵ > 0 .

Original: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive reward region. However, the uncertaintyabout the optimal action starts diminishing rapidly whenever the positivereward region is identified.
Revised: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive region, but the uncertainty starts diminishing rapidly whenever the positive region is identified.

Original: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .
Revised: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .

Original: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.
Revised: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.

Original: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.
Revised: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.

Original:  RegretAnalysis
Revised: 4.3 BOO Regret

Original: This section analyzes the regret of BOO for general model classes. To begin with, we introduce complexity measurement for function classes. 5.1 Complexity Measurement

Original: Our regret analysis uses the notions of the covering number and the eluder dimension [21, 22]. They measure different aspects of function classes. The covering number is introduced for measuring the size of the function class. Definition 1 (Covering number) .
Revised: In this section, we introduce the regret of BOO.

Original: Let ( M, ∥ · ∥ ) be a metric space and F ⊆ M .

Original: The covering number

Original: N ( F , α, ∥ · ∥ ) for α > 0 is the minimum cardinality of C ⊆ F such that for all f ∈ F , there exists c ∈ C satisfying ∥

Original: f − c ∥ ≤ α . The eluder dimension is a complexity measure for the exploration difficulty. Similar to [23], we generalize the original eluder dimension [21] to the distributional eluder dimension by measuring the ℓ 2 difference on a distribution. Definition 2 ( ϵ -dependence between distributions [22, 23]) .
Revised: The regret of BOO relies on both the decay rate of the scaling parameter and the complexity of model class.

Original: Let F be a  function  class defined on X , and ν, µ 1 , . . . , µ n be probability measures over X . We say ν is ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } w.r.t.
Revised: We assume that the model class resides in a d -dimensional RKHS, which roughly means any function in the model class can be represented as a linear function of a potentially unknown finite-dimensional feature map.

Original: F if any f 1 , f 2 ∈ F satisfying (cid:112)(cid:80) ni

Original: [-=
-]
Revised: [-=
-]

Original: µ i ( ∥ f 1 − f 2 ∥ 2 ) 2 ≤ ϵ also satisfies ν ( ∥ f 1 − f 2 ∥ 2 ) ≤ ϵ .

Original: Here, µ ( f ) =

Original: (cid:82) f d µ .

Original: If ν is not ϵ -dependent on { µ 1 , µ 2 , . . . , µ n } , then it is said to be ϵ -independent of { µ 1 , µ 2 , . . .

Original: , µ n } w.r.t.

Original: F .

Original: Let f ∗ be the real function where observations are generated.

Original: Intuitively, a distribution ν is ϵindependent of { µ 1 , µ 2 , . . .

Original: , µ n } means that a function f indistinguishable from the real function f ∗ on historical distributions could still be significantly different on the distribution ν .

Original: Definition 3 (Distributional eluder (DE) dimension [22, 23]) .

Original: Let F be a function class defined on X , and Π be a class of probability measures over X .

Original: The distributional eluder dimension DEdim( F , Π , ϵ ) is the length of the longest sequence in Π such that every element in the sequence is ϵ ′ -independent of its predecessors for some ϵ ′ ≥ ϵ .

Original: The distributional eluder dimension states that the bad event that a historically indistinguishable function f fails to match the real function on future data cannot happen indefinitely.

Original: In the worst case, after DEdim( F , Π , ϵ ) such events, all historically indistinguishable function matches the real function with ϵ -precision.

Original: The distributional eluder dimension equals the original eluder dimension when Π is the set of Dirac delta distribution centered around each point in the function’s domain.

Original: In

Original: RL, the distribution set we are concerned with is  the distributions induced by the models. Definition 4 (Probability measures induced by models) . Let M be a set ofmodels and M ∗ be the real MDP.
Revised: This assumption is not very restrictive because it places no restriction on the choice of feature map except for the finitude of the dimension.

Original: The probability measures induced by M and M ∗ is Π M ∗ M = { ρ π,M = 1 H (cid:80)

Original: Hh =1 ρ π,Mh | M ∈

Original: M , π ∈ arg max V π,M( s 1 ) } , where ρ π,Mh denotes the state-action distribution at period h induced by executing an optimal policy π of M in the MDP M ∗ .

Original: We consider the following function classes:

Original: Linear models: F = { f θ | θ ∈ Θ , f θ ( x ) =

Original: θϕ ( x ) , ∀ x ∈ X} , where Θ ⊂ R m × d and ∥ θ ∥ 2 ≤ r θ for all θ ∈ Θ , ϕ ( x ) ∈ R d , and ∥ ϕ ( x ) ∥ 2 ≤ r ϕ for all x ∈ X . 2.

Original: Generalized linear model: F = { f θ | θ ∈ Θ , f θ ( x ) =

Original: g ( θϕ ( x )) , ∀ x ∈ X} , where

Original: Θ ⊂ R m × d , ∥ θ ∥ 2 ≤ r θ for all θ ∈ Θ , ϕ ( x ) ∈ R d , and ∥ ϕ ( x ) ∥ 2 ≤ r ϕ for all x ∈ X .

Original: Assume there exist some h and h such that the gradient of g satisfies 0 < h ≤ g ′ ( x ) ≤ h for all x ∈ R d satisfying ∥ x ∥ 2 ≤ r θ r ϕ . 3.

Original: Reproducing kernel Hilbert space (RKHS) ball:

Original: F = { f ∈ H|∥

Original: f

Original: ∥ H ≤ r H } , where H is vector-valued RKHS on a Lebesgue space ( X , Σ , µ ) with a separable kernel K :

Original: X × X → R m × m constructed by K ( x, y ) ij =

Original: K X ( x, y ) K m ( i, j ) .

Original: Here, K X and K m are kernels for scalar-valued RKHS on X and [ m ] , respectively.

Original: The RKHS norm ∥ · ∥ H is ∥ f

Original: ∥ H =

Original: ( (cid:82) ∥

Original: f ( x ) ∥ 22 d µ ) 1 / 2 .

Original: Table 1 lists the complexity of different function classes measured by eluder dimension and covering number.

Original: Here, we extend previous covering number bound for scalar-valued functions [22, 24] to vector-valued functions.

Original: The upper bounds for distributional eluder dimension is first established.

Original: The complexity of RKHS relies on the eigenvalue decay property of the kernel.

Original: The finite dimensional linear kernel, the squared exponential kernel, and the Martérn kernel are examples of the finite spectrum, exponential decay, and polynomial decay kernels, respectively.

Original: The detailed explanation and derivation are in Appendix A. 5.2 BOO Regret This section analyze the  regret of BOO.
Revised: In Appendix B, we derive the asymptotic regret of the BOO algorithm in Theorem B.1.

Original: Instead of general MDPs, we now work with a specific type of MDPs consisting of a model function and an additive sub-Gaussian noise.

Original: [Table]

Original: Assumption 1 (Models with additive sub-Gaussian noises) .

Original: Let the state space S be a subset of R d .

Original: Denote the state-action space S ⊗A as X and state-reward space S ⊗ R as Y .

Original: Let F ⊂ Y X be a class of model functions.

Original: For each y ∈ Y , ϵ y is a σ -sub-Gaussian noise ϵ y , i.e., log E [ e v ⊤ ϵ y ] ≤ ∥ v ∥ 22 σ 2 /for all v ∈ R d . There exists a function f ∗ : X → Y in F such that for any x ∈ X , f ∗ ( x ) + ϵ f ∗ ( x ) isdistributed according to the joint distribution of P M ∗ ( x ) and R M ∗ ( x ) .
Revised: As suggested by Theorem B.1, the optimal asymptotic regret of BOO is O (cid:16) √ K log K (cid:17) , i.e., ˜ O (cid:16) √ K (cid:17) , achieved by setting λ ∗ k = c/ √ k , where λ ∗ k is the optimal scaling parameter and c is a constant associated with the model class.

Original: Assumption 1 requires the output distribution to be determined by its mean. This requirement is satisfied when the output distribution is modeled as, for example, Gaussian with known variance. The categorical distribution also satisfies this requirement by representing each category as a one-hot vector, in which case the mean is just the probability vector specifying the entire distribution. Since the noise is assumed sub-Gaussian, we adopt the independent Gaussian likelihood as a surrogate for the actual likelihood function [13], i.e.,
Revised: More detailed derivations and conclusions can be found in Appendix B.

Original: [Equation]

Original: In this way, we avoid restricting our discussion on specific types of distributions.

Original: The second assumption is the Lipschitz continuity of the one-step value function, which is needed for connecting the model difference to the value difference.

Original: Roughly speaking, this requires that when the outputs of model functions are close, the corresponding transition distributions are close.

Original: Assumption 2 (One-step Value Lipschitz Continuity) .

Original: Let U x,V ( M ) be the one-step value function with the state-action pair x ∈ X and the value function V : S → R , i.e., U x,V ( M ) = ¯ R M ( x ) + P Mx ( V ) , where P x ( V ) =
Revised: We can further interpret λ ∗ k = c/ √ k as λ ∗ k = ξ Vk /ξ Mk , where ξ Mk stands for the variation of the log-posterior density for the model M k , and ξ Vk represents the value uncertainty.

Original: (cid:82) V ( s ′ ) P ( s ′ | x ) d s ′ .

Original: We define the Lipschitz constant w.r.t.

Original: the 2 -norm as

Original: [Equation]

Original: where f M is the model function of M .

Original: Let M be a set of MDPs.

Original: The global Lipschitz constant w.r.t. M is L = max x,π,M ∈M L x,V π,M . Finally, we derive the asymptotic regret of the BOO algorithm in Theorem 5.1. Notice that the regret of BOO relies on both the complexity of model classes measured by the distributional eluder dimension and the log covering number and the decay rate of the scaling parameter, which conforms to our previous discussion.
Revised: In contrast to the discussion in Section 4, where we consider samples from the posterior distribution and the variation of log-posterior density remains constant, the width of the HPD region measured by the variation of log-posterior density is, in fact, growing at a rate depending on the log covering number.

Original: Theorem 5.1 (Bayesian optimistic optimization regret) .

Original: Assume that ∥ f ( x ) ∥ 2 ≤ r F for all x ∈ X and f ∈ F , and the log-prior probability uniformly bounded by Z . The distributional eluder dimension is assumed to satisfy DEdim( F , Π M ∗ M , ϵ ) ≤ E (1 /ϵ ) e 1 (log(1 /ϵ )) e 2 for e 1 , e 2 ≥ 0 , and the log covering number satisfies log N ( F , α, ∥ · ∥ ∞ ) ≤ C (1 /α ) 
Revised: Accroding to Lemma D.7 in Appendix D.1, ξ Mk = O (log k ) .

Original: c

Original: 1 (log(1 /α )) c 2 for c 1 , c 2 ≥ 0 . Then, setting λ  k  = ck − v for c > 0 , e > 0 , the frequentist regret of BOO is [Equation]
Revised: Therefore, we have ξ Vk = λ ∗ k ξ Mk = O (cid:16) log k √ k (cid:17) .

Original: 6) if v < [Equation] 7) if otherwise. As suggested by Theorem 5.1, the best performance of BOO is achieved by setting the decay rate to
Revised: Section 5 will highlight the need of manipulating with ξ Vk and ξ Mk in optimization.

Original: [Equation]

Original:  Optimization
Revised:  Optimization

Original: This section discusses the optimization of the BOO algorithm.
Revised: In this section, we introduce optimization methods for BOO based on posterior sampling and gradient descent, respectively, and discuss in detail the problems of gradient-based optimization and our proposed solutions. 5.1 Optimization via Posterior Sampling

Original: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.
Revised: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.

Original: For infinitedimensional RKHS, this  is  the only choice.
Revised: As discussed previously, the optimal scaling parameter λ k for this algorithm is proportional to the value uncertainty, i.e., λ k =

Original: Since PSRL, the single-sample version of Algorithm 4, already achieves the state-of-the-art performance, combining multiple samples may seem meaningless.
Revised: ξ Vk = λ ∗ k ξ Mk .

Original: We argue that the potential advantage of Algorithm 4  is that, unlike PSRL, it does not require posterior samples to be independent.
Revised: The potential advantage of Algorithm 4 over PSRL is that, unlike PSRL, it does not require posterior samples to be independent.

Original: Even the requirement that samples are from the posterior can be relaxed.
Revised: Even the requirement that samples are from the posterior can be relaxed.

Original: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.
Revised: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.

Revised: 5.2 Optimization via Gradient-Based Methods

Original: For finite-dimensional models, we can optimize via gradient-based methods.
Revised: We can also perform the optimization via gradient-based methods.

Original: The objective function of BOO consists of two parts. The gradient of the log-posterior density part w.r.t.
Revised: The objective function of BOO consists of two parts, V π,M ik( s 1 ) and λ k (log Pr( H k | M ik ) + log Pr( M ik )) .

Original: the model is just the gradient of the log-prior and the log-likelihood.
Revised: The gradients of the loglikelihood and the log-prior are easily obtained.

Original: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [25, 26] (proof in Appendix D.1).
Revised: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [36, 37] (proof in Appendix E.1).

Original: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.
Revised: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.

Original: Theorem 6.1 (Value model gradient) .
Revised: Theorem 5.1 (Value model gradient) .

Original: Suppose that the transition function P M  and reward function
Revised: Suppose that the transition function P M θ and reward function

Original: R M  of model Mis parameterized by θ , the gradient of the value V π,M( s 1 ) w.r.t.
Revised: R M θ of model M θ , the gradient of the value V π,M θ( s 1 ) w.r.t.

Original: the model is ∇ 
Revised: the model is ∇ θ V π,M θ

Original: [Equation]
Revised: [Equation]

Original: where τ = ( s 1 , a 1 , . .
Revised: where τ = ( s 1 , a 1 , . .

Original: , s H , a H ) is a trajectory,  and P M  ( s h
Revised: , s H , a H ) is a trajectory, τ ∼ π, M θ means that the trajectory is formed by the interaction of the policy π and the model M θ , and P M θ ( s h

Original: +

Revised: +

Original: | s h , a h ) is the probability of s h + under distribution P M  ( s h , a h ) 
Revised: | s h , a h ) is the probability of s h + under distribution P M θ ( s h , a h ) .

Original: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [27] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.
Revised: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [38] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.

Original: The natural policy gradient proposed by [28] improves the  efficiency of  the policy gradient [29] by preconditioning the gradient with the inverse Fisher information matrix. We formalize the natural model gradient in a similar fashion.
Revised: We propose some techniques to improve the optimization efficiency of BOO, and conduct ablation experiments to verify the effectiveness of our proposed methods in Section G. Two of the most effective techniques are described below, and the rest of the techniques are detailed in Appendix F.

Original: Let D Rk be normalized empirical state-action distribution for rewards D Rk ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80) Hh =1 I ( s ih = s, a ih = a ) .
Revised: 5.2.1 Mean Reward Bonus

Original: The normalized empirical state-action distribution for transitions is defined slightly differently as D Pk ( s, a ) ∝ (cid:80) k − 1 i =1 (cid:80)
Revised: In the gradient-based optimization, the model becomes optimistic on state-action pairs it visits frequently, which in turn makes these state-action pairs more appealing.

Original: H − 1 h =1 I ( s ih = s, a ih = a ) since the transition pair in the H -th period is not observed by definition. Denoting the BOO objective in the k -th episode as J k = λ k log Pr( M |H k 1 ) + V π,M( s 1 ) , the natural model gradient is ¯ ∇ θ J k = F − 1 k ∇ θ J k , where F k = E τ ∼ π,M [ 1 H (cid:80) H h
Revised: This mutual strengthening phenomenon makes optimization easily stuck at local optima.

Original: =1 G ¯ Rs h ,a h + 1 H − 1 (cid:80) H − 1 h =1 F Ps h ,a h ] + E ( s,a ) ∼ D Rk [ F Rs,a ] + E ( s,a ) ∼ D Pk [ F Ps,a ] .
Revised: One way to solve this problem is to increase the rewards of all state-action pairs, which raises the attractiveness of less visited state-action pairs.

Original: Here, F Ps,a = Es ′ ∼ P M ( s h ,a h ) [( ∇ θ log P M
Revised: This can be achieved by adding a bonus term ξ Vk H E ( s,a ) ∼ U S×A

Original:  ( s ′ | s h , a h )) 2 ] and
Revised: [ R ( s, a )] to the BOO objective.

Original: F Rs,a = E r ∼ R M ( s h ,a h ) [( ∇ θ log R M ( r | s h , a h )) 2 ] are the Fisher information matrix of transition and reward distributions. The matrix G ¯ Rs,a = ∇ θ z ⊤ R ∇ 2 z R ¯ R M ( s, a ) ∇ θ z R is the generalized Gauss-Newton matrix [30], where the reward distribution is parameterized by z R ⊆
Revised: Here, U S×A is the uniform distribution over the state-action space. The coefficient ξ Vk ensures that the bonus decays with the value uncertainty. The unvisited state-action pairs will eventually be visited by the policy because they have sufficiently high rewards.

Original: z , and z = f θ ( s, a ) is the output of the model function f θ . Note that we have G ¯ Rs,a = 0 when the mean reward ¯ R M ( s, a ) is linear in z . This is satisfied for most model classes, for example, those specified by Assumption 1.
Revised: Our experiments will show that, in tabular setting, this method is very effective. However, a concern is that it might fail to scale to high dimensional state-action space.

Original: Therefore, commonly, F k contains only the Fisher information matrix, and the natural gradient can be interpreted geometrically as the steepest descent direction in the information geometry induced by F k instead of in the Euclidean geometry. Loosely speaking, the distance in the information geometry is measured by the square root of KL divergence between distributions rather than the ℓ 2 distance of parameters [31]. The natural gradient characterizes some global characteristics of the loss surface and therefore facilitates global convergence. We refer interested readers to [31] for a thorough discussion on the interpretation and implementation of the natural gradient.
Revised: 5.2.2 Entropy Regularization

Original: Another intricacy of this problem is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.
Revised: Another intricacy of the optimization is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.

Original: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.
Revised: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.

Original: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.
Revised: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.

Original: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.
Revised: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.

Original: The smoothing effect  is also observed previously in [32].
Revised: The smoothing effect of entropy regularization is also discussed previously in [39].

Original: The entropy-regularized learning objective mimics the maximum entropy RL [33]:
Revised: The entropy-regularized learning objective mimics the maximum entropy RL [40]:

Original: [Equation]
Revised: [Equation]

Original: where KL( p ∥ q ) stands for the relative entropy, ˆ πh is a prior policy ensuring πh is absolutely continuous w.r.t.
Revised: where KL( p ∥ q ) stands for the relative entropy, ˆ π is a prior policy ensuring π is absolutely continuous w.r.t.

Original: ˆ πh , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.
Revised: ˆ π , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.

Original: Here, the entropy term is downscaled  such that the influence of regularization diminishes with time.
Revised: The entropy term is downscaled in proportion to the value uncertainty such that the influence of regularization diminishes with time.

Original: Consider the case that there exists a ϵ -optimal policy π ϵ  for anyfixed model M such that, for some b > 0 , E τ ∼ π ϵ ,M (cid:104)(cid:80)
Revised: We denote by M θ the set of models parameterized by θ . Let b ϵ be the smallest number ensuring that, for any M θ ∈ M θ , there exists

Original: [Equation] =
Revised: π ϵ ∈ Π ϵ =

Original: [Equation] KL
Revised: { π | KL( π ( s, h )

Original: [Equation] π 
Revised: ∥ ˆ π ( s, h ))

Original: [Equation] s [Equation] ≤ 
Revised: ≤ b ϵ , ∀ s ∈ S , h ∈

Original: [Equation] and sup 
Revised: [ H ] } such that sup π ∗ V π

Original: [Equation]
Revised: [Equation]

Original: ( s 1 ) ≤ ϵ . Then, the value of the maximum entropy policy ˜ π ∗ = arg max ˜ π V ˜ π,M( s 1 ) satisfies that
Revised: Then, the value of the maximum entropy policy ˜ π ∗ = arg max ˜ π V ˜ π,M( s 1 ) satisfies that

Original: [Equation]
Revised: [Equation]

Original: If ϵ is sufficiently small
Revised: If ϵ is sufficiently small

Original: [Equation]
Revised: [Equation]

Original: then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty, which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance
Revised: then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty [Equation] which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance

Original: [Equation]
Revised: [Equation]

Original: The entropy-regularized value is equivalently defined by the following Bellman backup,
Revised: The entropy-regularized value is equivalently defined by the following Bellman backup,

Original: [Equation]
Revised: [Equation]

Original: where ˜ Q π,MH ( s H , a H ) =
Revised: where ˜ Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [33] or the soft actor-critic algorithm [34].
Revised: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [40] or the soft actor-critic algorithm [41].

Original: In finite MDPs, the optimal value is given by soft value iteration,
Revised: In finite MDPs, the optimal value is given by soft value iteration,

Original: [Equation]
Revised: [Equation]

Revised: ) where π ( a h | s h , h ) is the probability of a h under the distribution π ( s h , h ) .

Original: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as
Revised: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as

Original: [Equation]
Revised: [Equation]

Original: The gradient of the entropy-regularized value w.r.t.
Revised: The gradient of the entropy-regularized value w.r.t.

Original: the model is similar to that specified in Theorem 6.1, with the value replaced by the entropy-regularized value.
Revised: the model is similar to that specified in Theorem 5.1, with the value replaced by the entropy-regularized value.

Original:  Conclusion
Revised:  Experiments

Revised: This section compares the performance of BOO with PSRL and UCRL2 in RiverSwim, Chain, and

Revised: Random MDPs.

Revised: RiverSwim and Chain are hard-exploration MDPs requiring the agent to explore efficiently, while Random MDPs test the average performance.

Revised: Two implementation of BOO are presented, namely, FiniteBOO and BPS.

Revised: FiniteBOO is the BOO with entropy regularization and mean reward bonus mentioned in Section 5.2, which is empirically found to be the best variant in tabular setting according to the ablation study.

Revised: BPS is an implementation of Algorithm 4 (BOO via posterior sampling).

Revised: [Figure]

Revised: 6.1 RiverSwim

Revised: The RiverSwim is an MDP where states are organized in chains, and the agent can move left or right, as shown in Figure 1(a).

Revised: Although the rightmost state has a huge reward, the action of moving right fails with a high probability.

Revised: Only a policy moves right at each time period has a small chance of success.

Revised: We start with the experiment on the RiverSwim to demonstrate the performance of our algorithm in the face of high transition uncertainty.

Revised: We perform experiments for ten seeds on RiverSwim with |S| =

Revised: H = 5 , |A| = 2 and record the cumulative regret over 100,000 time steps.

Revised: As shown in Figure 2(a), our algorithm is competitive to PSRL and outperforms UCRL2 significantly.

Revised: 6.2 Chain

Revised: The chain MDP is a variant of the RiverSwim, which has Gaussian rewards and relatively deterministic transitions, as shown in Figure 1(b).

Revised: Although transitions are relatively certain, the stochastic rewards make the problem difficult to explore.

Revised: The horizon H and the number of states |S| are equal to the length of the chain.

Revised: We evaluate our algorithms in Chain of a length L ∈ { 10 , 20 , 40 } for 100 , 000 episodes and ten random trials.

Revised: Figure 2(d ∼ f) illustrates that our algorithm compares favorably with PSRL and

Revised: UCRL2, which certifies the effectiveness of BOO in problems requiring long-term planning.

Revised: 6.3 Random MDPs

Revised: Random MDPs are tabular MDP models randomly generated from a prior distribution and used to test the general performance of the algorithm.

Revised: We randomly generate 100 stochastic MDPs with |S| = |A| =

Revised: H = 5 and |S| = H = 20 , |A| =

Revised: from the prior and measure the performance of algorithms over 10 , 000 timesteps.

Revised: As shown in

Revised:       (SLVRGHV      

Revised:  & X P X O D WL Y H  U H J U H W D5LYHU6ZLP       (SLVRGHV 

Revised: 

Revised:        E5DQGRP0'3VZLWKVL]H 

Revised:      (SLVRGHV      F5DQGRP0'3VZLWKVL]H %3 )LQLWH%2 365/ 8&5/       (SLVRGHV  

Revised:      & X P X O D WL Y H  U H J U H W G&KDLQZLWKOHQJWK       (SLVRGHV        H&KDLQZLWKOHQJWK       (SLVRGHV 

Revised: 

Revised:    I&KDLQZLWKOHQJWK

Revised: Figure 2(b), the performance of BOO and BPS is close to PSRL, and both algorithms outperform

Revised: UCRL2 significantly.

Revised: However, the performance gap between FiniteBOO and PSRL in Figure 2(c) indicates that there remains a challenge in the optimization of large-scale MDPs.

Revised:  Conclusion

Revised: This paper proposes BOO as a generic model-based reinforcement learning algorithm.

Revised: It is provably sample-efficient and enjoys ˜ O ( √ K ) regret for models in a finite-dimensional RKHS, where K is the number of episodes.

Revised: To optimize the BOO objective, we propose the value model gradient and optimization techniques, such as entropy regularization, to improve its efficiency.

Original: This paper proposes BOO as a generic model-based reinforcement learning algorithm.
Revised: Through our experiments, we have shown that BOO is competitive with PSRL and outperforms UCRL greatly. However, to apply BOO in real-world reinforcement learning problems, there remains lots of work to be done. Importantly, we need to develop methods that further facilitate the gradient-based optimization of BOO in large-scale problems. It is also an appealing direction for future work to adapt the scaling parameter of BOO on an instance-dependent basis.

Original: It enjoys sublinear regret for a variety of model classes. To optimize the BOO objective, we develop a natural model gradient that facilitates learning by optimizing the model in the space of distributions.
Revised: This work is supported by National Key Research and Development Program of China (2020AAA 0107200), the National Science Foundation of China (61921006, 61876119, 62276126), the Natural

Original: We also propose to use an entropy-regularized objective that smooths the loss surface and makes BOO amenable to incremental implementation.
Revised: Science Foundation of Jiangsu (BK20221442), and the Fundamental Research Funds for the Central

Original: However, to apply BOO in real-world reinforcement learning problems, it is essential to adapt the scaling parameter of BOO on a per-instance basis, which is an appealing direction for future work.
Revised: Universities (022114380010).