Original: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning
Revised: Bayesian Optimistic Optimization: Optimistic Exploration for Model-based Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.
Revised: Reinforcement learning (RL) is a general framework for modeling sequential decision making problems, at the core of which lies the dilemma of exploitation and exploration.

Original: An agent failing to explore systematically will inevitably fail to learn efficiently.
Revised: An agent failing to explore systematically will inevitably fail to learn efficiently.

Original: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.
Revised: Optimism in the face of uncertainty (OFU) is a conventionally successful strategy for efficient exploration.

Original: An agent following the OFU principle explores actively and efficiently.
Revised: An agent following the OFU principle explores actively and efficiently.

Original: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.
Revised: However, when applied to model-based RL, it involves specifying a confidence set of the underlying model and solving a series of nonlinear constrained optimization, which can be computationally intractable.

Original: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.
Revised: This paper proposes an algorithm, Bayesian optimistic optimization (BOO), which adopts a dynamic weighting technique for enforcing the constraint rather than explicitly solving a constrained optimization problem.

Original: BOO is a general algorithm proved to be sample-efficient for models in a finite-dimensional reproducing kernel
Revised: BOO is a general algorithm proved to be sample-efficient for models in a finite-dimensional reproducing kernel

Original: Hilbert space.
Revised: Hilbert space.

Original: We also develop techniques for effective optimization and show through some simulation experiments that BOO is competitive with the existing algorithms.
Revised: We also develop techniques for effective optimization and show through some simulation experiments that BOO is competitive with the existing algorithms.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives [1, 2].
Revised: Reinforcement learning (RL) is a sequential decision-making problem in which an agent acts in an unknown environment while maximizing the cumulative rewards it receives [1, 2].

Original: In this paper, we consider the RL in Markov decision processes (MDPs) where the agent observes the state of the environment at each timestep and makes decisions accordingly.
Revised: In this paper, we consider the RL in Markov decision processes (MDPs) where the agent observes the state of the environment at each timestep and makes decisions accordingly.

Original: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.
Revised: Since the environment is unknown, maximizing the cumulative rewards naturally involves a trade-off between exploration and exploitation.

Original: Exploitation is to make the best-rewarding decision based on the agent’s information and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.
Revised: Exploitation is to make the best-rewarding decision based on the agent’s information and exploration means actively gathering information about the environment so that the agent understands better about the environment and thus makes better decisions in the future.

Original: An algorithm cannot be sample-efficient without balancing them properly.
Revised: An algorithm cannot be sample-efficient without balancing them properly.

Original: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [3].
Revised: Theoretically, the exploration and exploitation dilemma admits a Bayesian optimal solution [3].

Original: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.
Revised: That is to consider the RL problem a so-called Bayes-Adaptive MDP (BAMDP), a special case of partially observable Markov decision processes (POMDPs), where the parameter of the dynamics is unobservable.

Original: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [4–6].
Revised: Although this formulation provides useful insights, the POMDP formulation is computationally intractable [4–6].

Original: Therefore, all practical algorithms [7–10] resolve this dilemma by achieving a delicate balance between seeking rewards and gathering information.
Revised: Therefore, all practical algorithms [7–10] resolve this dilemma by achieving a delicate balance between seeking rewards and gathering information.

Original: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [11–13].
Revised: Optimism in the face of uncertainty (OFU) is one of the conventionally successful approaches for this balance and is established as an efficient learning principle in various cases [11–13].

Original: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.
Revised: The OFU principle makes the agent an optimist who is always optimistic about the uncertainty of the environment.

Original: If it ever stands a chance that a policy is highly profitable, the agent will try it out.
Revised: If it ever stands a chance that a policy is highly profitable, the agent will try it out.

Original: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.
Revised: By executing an optimistic policy, the agent either receives high rewards or gains information after observing unexpected outcomes.

Original: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.
Revised: OFU involves constructing a confidence set of possible MDPs and solving for the most optimistic one within the confidence set.

Original: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [14, 15].
Revised: Unfortunately, this is typically a constrained nonlinear non-concave optimization and is impractical to solve [14, 15].

Original: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [15].
Revised: Therefore, a model-based OFU algorithm is previously deemed an inferior choice [15].

Original: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization more practical.
Revised: In this paper, we propose a model-based OFU algorithm, Bayesian optimistic optimization (BOO), which enforces the constraint via a dynamic weighting technique and therefore makes the optimization more practical.

Original: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.
Revised: We show that BOO is a general-purpose model-based OFU RL algorithm that is both provably sample-efficient and comparatively computationally tractable.

Original: Our contribution is three-fold:
Revised: Our contribution is three-fold:

Original: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.
Revised: • We extend the OFU algorithms to the Bayesian setting by requiring the model to reside in a Bayesian credible region.

Original: BOO is then proposed as the Lagrangian relaxation of this constrained optimization.
Revised: BOO is then proposed as the Lagrangian relaxation of this constrained optimization.

Original: • We prove that BOO is efficient in terms of the frequentist regret for  finite-dimensional reproducing kernel Hilbert space (RKHS).
Revised: • We prove that BOO is efficient in terms of the frequentist regret for a finite-dimensional reproducing kernel Hilbert space (RKHS).

Original: • We derive optimization methods for BOO and demonstrate empirical evidence that BOO is competitive with UCRL2 [16] and PSRL [17].
Revised: • We derive optimization methods for BOO and demonstrate empirical evidence that BOO is competitive with UCRL2 [16] and PSRL [17].

Original:  Preliminaries
Revised:  Preliminaries

Original: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.
Revised: This section introduces the preliminaries, including RL in MDPs and the principle of optimism in the face of uncertainty.

Original: 2.1 Reinforcement Learning in Markov Decision Processes
Revised: 2.1 Reinforcement Learning in Markov Decision Processes

Original: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon
Revised: We consider the problem where a reward-seeking agent repeatedly interacts with a finite-horizon

Original: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.
Revised: MDPs M = ( S , A , P, R, H, s 1 ) , where S and A are the state and action spaces, respectively.

Original: The algorithm adopted by the agent is denoted as A .
Revised: The algorithm adopted by the agent is denoted as A .

Original: At each episode, the agent is spawned at the initial state s 1 ∈ S .
Revised: At each episode, the agent is spawned at the initial state s 1 ∈ S .

Original: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R
Revised: It takes an action a h ∈ A at each period h within an episode and receives a random reward, r h ∼ R

Original: ( s h , a h ) .
Revised: ( s h , a h ) .

Original: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .
Revised: Then, the environmental state transitions to the next state, s h +1 ∼ P ( s h , a h ) .

Original: This process repeats until the episode ends at the H -th period, and another episode begins.
Revised: This process repeats until the episode ends at the H -th period, and another episode begins.

Original: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )
Revised: Note that the fixed initial state is not a stringent condition since starting from an initial distribution ρ 0 is equivalent to starting from a special state s 1 such that P ( s | s 1 , a )

Original: =
Revised: =

Original: ρ 

Revised: ρ

Original: ( s ) holds for any state s and action a .
Revised: ( s ) holds for any state s and action a .

Original: A finite MDP is an MDP with finite state and action spaces.
Revised: A finite MDP is an MDP with finite state and action spaces.

Original: A policy π is a function mapping a state and a period to an action distribution.
Revised: A policy π is a function mapping a state and a period to an action distribution.

Original: The value function for the MDP M , policy π , and period h is defined recursively as
Revised: The value function for the MDP M , policy π , and period h is defined recursively as

Original: [Equation]
Revised: [Equation]

Original: where ¯ R M ( s, a ) =
Revised: where ¯ R M ( s, a ) =

Original: E r ∼ R M
Revised: E r ∼ R M

Original: ( s,a )
Revised: ( s,a )

Original: [ r ] , and Q π,MH ( s H , a H ) =
Revised: [ r ] , and Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: We use X k,h to represent the variable X at the h -th period of the k -th episode.
Revised: We use X k,h to represent the variable X at the h -th period of the k -th episode.

Original: For notational convenience, X k,h is sometimes abbreviated as X kh .
Revised: For notational convenience, X k,h is sometimes abbreviated as X kh .

Original: The history prior to k -th episode is defined as H k =
Revised: The history prior to k -th episode is defined as H k =

Original: ( s 1 , 1 , a 1 , 1 , r 1 , 1 , s 1 , 2 , . . .
Revised: ( s 1 , 1 , a 1 , 1 , r 1 , 1 , s 1 , 2 , . . .

Original: , s k − 1 ,H , a k − 1 ,H , r k − 1 ,H ) .
Revised: , s k − 1 ,H , a k − 1 ,H , r k − 1 ,H ) .

Original: The agent is assumed to be capable of memorizing the entire history.
Revised: The agent is assumed to be capable of memorizing the entire history.

Original: In this paper, we consider model-based reinforcement learning algorithms which produce a model M k per episode in light of the history H k and derive its corresponding optimal policy π k ∈ arg max π V π,M k
Revised: In this paper, we consider model-based RL algorithms which produce a model M k per episode in light of the history H k and derive its corresponding optimal policy π k ∈ arg max π V π,M k

Original: ( s 1 ) for execution.
Revised: ( s 1 ) for execution.

Original: We use the terms model and MDP interchangeably.
Revised: We use the terms model and MDP interchangeably.

Original: For any history H , A ( H ) defines a distribution over models and policies.
Revised: For any history H , A ( H ) defines a distribution over models and policies.

Original: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .
Revised: Within the k -th episode, the agent samples an action from π k ( s kh , h ) at each period h .

Original: In the frequentist viewpoint, there exists an unknown true MDP M ∗ .
Revised: In the frequentist viewpoint, there exists an unknown true MDP M ∗ .

Original: We abbreviate V π ∗ ,M ∗ h as V ∗ h , where π ∗ is an optimal policy of the true MDP M ∗ .
Revised: We abbreviate V π ∗ ,M ∗ h as V ∗ h , where π ∗ is an optimal policy of the true MDP M ∗ .

Original: The frequentist performance metric, regret, is defined in terms of the true MDP:
Revised: The frequentist performance metric, regret, is defined in terms of the true MDP:

Original: [Equation]
Revised: [Equation]

Original: where ∆ k is defined as E π k ∼ A ( H k ) (cid:104) V ∗ 1 ( s 1 ) − V π k ,M ∗( s 1 ) (cid:105) , K is the total number of episodes,
Revised: where ∆ k is defined as E π k ∼ A ( H k ) (cid:104) V ∗ 1 ( s 1 ) − V π k ,M ∗( s 1 ) (cid:105) , K is the total number of episodes,

Original: T = KH is the number of the total time steps, and H K +1 ∼ A , M ∗ means that the history is sampled by the interaction of the algorithm A and the real MDP M ∗ .
Revised: T = KH is the number of the total time steps, and H K +1 ∼ A , M ∗ means that the history is sampled by the interaction of the algorithm A and the real MDP M ∗ .

Original: In the Bayesian viewpoint, the unknown MDP M ∗ is treated as a random variable and assigned a prior ρ M .
Revised: In the Bayesian viewpoint, the unknown MDP M ∗ is treated as a random variable and assigned a prior ρ M .

Original: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .
Revised: All MDPs in the support of ρ M differ only in the transition function P and the reward function R .

Original: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T , BayesRegret( T, A , ρ M ) =
Revised: The Bayesian objective of the agent is to minimize the Bayesian regret up to time T , BayesRegret( T, A , ρ M ) =

Original: E
Revised: E

Original: M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .
Revised: M ∗ ∼ ρ M [Regret( T, A , M ∗ )] .

Original: 2.2 Optimism in the Face of Uncertainty
Revised: 2.2 Optimism in the Face of Uncertainty

Original: Optimism in the face of uncertainty is a strategy for information gathering.
Revised: Optimism in the face of uncertainty is a strategy for information gathering.

Original: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.
Revised: When the optimal action is not clear given the current information, it is preferable to hazard an optimistic guess.

Original: If we make an atrocious guess, we effectively rule that out and pick another next time.
Revised: If we make an atrocious guess, we effectively rule that out and pick another next time.

Original: Otherwise, we end up finding a competitive solution that incurs little regrets.
Revised: Otherwise, we end up finding a competitive solution that incurs little regrets.

Original: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.
Revised: This idea is mathematically realized as a constrained optimistic optimization, max π k ,M k V k 1 ( s 1 ) s.t.

Original: M k ∈ M k , where M k is a confidence set constructed using empirical data H k such that M ∗ ∈ M k with high probability.
Revised: M k ∈ M k , where M k is a confidence set constructed using empirical data H k such that M ∗ ∈ M k with high probability.

Original: The pseudocode of the OFU algorithm is shown in Algorithm 1.
Revised: The pseudocode of the OFU algorithm is shown in Algorithm 1.

Original: 1: for episode k = 1 , 2 , . . .
Revised: 1: for episode k = 1 , 2 , . . .

Original: do 2:
Revised: do 2:

Original: Construct a confidence set M k with H k 3:
Revised: Construct a confidence set M k with H k 3:

Original: Compute π
Revised: Compute π

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Execute π k for an episode
Revised: Execute π k for an episode

Original:  Related Work
Revised:  Related Work

Original: One line of research resolves the exploration and exploitation dilemma by formalizing the reinforcement learning problem as a planning problem in Bayes-Adaptive MDPs (BAMDPs) [3], which treats the unknown MDP parameter as an additional hidden variable and maintains a belief distribution of the parameter.
Revised: One line of research resolves the exploration and exploitation dilemma by formalizing the RL problem as a planning problem in Bayes-Adaptive MDPs (BAMDPs) [3], which treats the unknown MDP parameter as an additional hidden variable and maintains a belief distribution of the parameter.

Original: This line of work shares the scalability problem, which is caused by the exponential increase of possible histories w.r.t.
Revised: This line of work shares the scalability problem, which is caused by the exponential increase of possible histories w.r.t.

Original: the planning horizon.
Revised: the planning horizon.

Original: That is, the planning in BAMDPs is PSPACE-complete [4] and requires exponential time to solve.
Revised: That is, the planning in BAMDPs is PSPACE-complete [4] and requires exponential time to solve.

Original: Intuitively, this is because methods based on BAMDPs deal with the belief distributions of MDPs (or histories) rather than a single MDP.
Revised: Intuitively, this is because methods based on BAMDPs deal with the belief distributions of MDPs (or histories) rather than a single MDP.

Original: Efficient planning algorithms in BAMDPs do exist.
Revised: Efficient planning algorithms in BAMDPs do exist.

Original: Exploiting the root sampling technique, we can implement an algorithm that gets rid of the posterior distribution and only requires posterior sampling [18].
Revised: Exploiting the root sampling technique, we can implement an algorithm that gets rid of the posterior distribution and only requires posterior sampling [18].

Original: Nonetheless, the underlying scalability issue remains prominent.
Revised: Nonetheless, the underlying scalability issue remains prominent.

Original: Methods following the OFU principle construct an optimistic estimate of the unknown MDP and execute its optimal policy.
Revised: Methods following the OFU principle construct an optimistic estimate of the unknown MDP and execute its optimal policy.

Original: Since these methods plan on a single MDP estimate, they are computationally preferable compared to methods based on BAMDPs.
Revised: Since these methods plan on a single MDP estimate, they are computationally preferable compared to methods based on BAMDPs.

Original: The UCRL2 [16] is one such method.
Revised: The UCRL2 [16] is one such method.

Original: However, unlike tabular and linear MDPs for which constructing an optimistic estimate is analytically tractable [11, 13], constructing an optimistic estimate for general MDPs involves a constrained joint optimization of model and policy and is computationally prohibitive.
Revised: However, unlike tabular and linear MDPs for which constructing an optimistic estimate is analytically tractable [11, 13], constructing an optimistic estimate for general MDPs involves a constrained joint optimization of model and policy and is computationally prohibitive.

Original: Hence, previous model-based
Revised: Hence, previous model-based

Original: OFU algorithms [19, 20] for general model classes cannot be implemented and rely on posterior sampling for exploration.
Revised: OFU algorithms [19, 20] for general model classes cannot be implemented and rely on posterior sampling for exploration.

Original: The posterior sampling for reinforcement learning (PSRL) works by selecting a random MDP from the posterior distribution and executing its optimal policy [21].
Revised: The posterior sampling for reinforcement learning (PSRL) works by selecting a random MDP from the posterior distribution and executing its optimal policy [21].

Original: This strategy ensures that a policy is selected according to the probability that it is the optimal policy of the real model.
Revised: This strategy ensures that a policy is selected according to the probability that it is the optimal policy of the real model.

Original: It is shown that
Revised: It is shown that

Original: PSRL is at least as good as any frequentist OFU algorithm in terms of Bayesian regret [22, 15].
Revised: PSRL is at least as good as any frequentist OFU algorithm in terms of Bayesian regret [22, 15].

Original: PSRL is argued to be better than optimism since it is more computationally tractable [15].
Revised: PSRL is argued to be better than optimism since it is more computationally tractable [15].

Original: Nevertheless, as opposed to OFU algorithms, incremental implementation of PSRL is challenging because it requires replanning after each sampling.
Revised: Nevertheless, as opposed to OFU algorithms, incremental implementation of PSRL is challenging because it requires replanning after each sampling.

Original: Practical implementation instead tries to directly sample from the posterior distribution of the optimal value function [23–25].
Revised: Practical implementation instead tries to directly sample from the posterior distribution of the optimal value function [23–25].

Original: However, the theoretical guarantee for these methods is only established for tabular MDPs [25].
Revised: However, the theoretical guarantee for these methods is only established for tabular MDPs [25].

Original: H-UCRL was devoted to resolving the intractability of model-based optimistic exploration for general models [26].
Revised: H-UCRL was devoted to resolving the intractability of model-based optimistic exploration for general models [26].

Original: It proposes to convert the joint optimization of model and policy into a hallucinated control problem.
Revised: It proposes to convert the joint optimization of model and policy into a hallucinated control problem.

Original: This approach ignores the correlation between state-action pairs and treats them separately, causing inefficiency as reflected by the extra dependency on the cumulative posterior variance in their regret bound.
Revised: This approach ignores the correlation between state-action pairs and treats them separately, causing inefficiency as reflected by the extra dependency on the cumulative posterior variance in their regret bound.

Original: Our method, BOO, also tries to attack the intractability of optimism, which builds an optimistic model by optimizing both the value and the log-posterior density.
Revised: Our method, BOO, also tries to attack the intractability of optimism, which builds an optimistic model by optimizing both the value and the log-posterior density.

Original: It avoids the defect of H-UCRL since maximizing the log-posterior density naturally enforces the correlation between state-action pairs.
Revised: It avoids the defect of H-UCRL since maximizing the log-posterior density naturally enforces the correlation between state-action pairs.

Original: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [27] and is named as reward-biased maximum likelihood estimation (RBMLE).
Revised: We note that, when the prior is uniform, this idea is equivalent to balance value versus log-likelihood, which is first explored in [27] and is named as reward-biased maximum likelihood estimation (RBMLE).

Original: They have applied this approach to multi-armed bandits [28], contextual bandits [29], and
Revised: They have applied this approach to multi-armed bandits [28], contextual bandits [29], and

Original: RL where the model belongs to a known finite set [27, 30].
Revised: RL where the model belongs to a known finite set [27, 30].

Original: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [31].
Revised: A constrained version of RBMLE is also successfully applied to RL of linear quadratic control systems [31].

Original: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and finite dimensional RKHS.
Revised: Our algorithm can be considered generalizing RBMLE to a Bayesian perspective and finite dimensional RKHS.

Original: In this regard, BOO could also be referred to as reward-biased maximum a posteriori.
Revised: In this regard, BOO could also be referred to as reward-biased maximum a posteriori.

Original: Concerning the regret analysis, previous regret analysis for general model-based reinforcement learning [19, 20] relies on the fact that the constructed model is the most optimistic one in the confidence set.
Revised: Concerning the regret analysis, previous regret analysis for general model-based RL [19, 20] relies on the fact that the constructed model is the most optimistic one in the confidence set.

Original: The regret analysis of BOO differs with them significantly since the model constructed by BOO may not belong to a confidence set or a credible region.
Revised: The regret analysis of BOO differs with them significantly since the model constructed by BOO may not belong to a confidence set or a credible region.

Original: This difference entails a distinct analysis, where we show that neither the large deviation from the real model nor the possibly pessimistic estimation of the model causes a large regret.
Revised: This difference entails a distinct analysis, where we show that neither the large deviation from the real model nor the possibly pessimistic estimation of the model causes a large regret.

Original:  Bayesian Optimistic Optimization
Revised:  Bayesian Optimistic Optimization

Original: In this section, we derive the learning objective of Bayesian optimistic optimization (BOO) as a
Revised: In this section, we derive the learning objective of Bayesian optimistic optimization (BOO) as a

Original: Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.
Revised: Lagrangian relaxation of a constrained optimization problem and give an intuitive interpretation of the resulting objective.

Original: Assuming the model class resides in a finite-dimensional RKHS, we show that BOO enjoys ˜ O ( √ K ) regret.
Revised: Assuming the model class resides in a finite-dimensional RKHS, we show that BOO enjoys ˜ O ( √ K ) regret.

Original: 4.1 Constrained BOO
Revised: 4.1 Constrained BOO

Original: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.
Revised: The conventional OFU algorithm, as demonstrated in Algorithm 1, contains a constrained optimistic optimization, where we look for an optimistic MDP M k ∈ M k and its corresponding optimal policy such that the value is maximized.

Original: The constrained BOO is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the Bayesian credible region.
Revised: The constrained BOO is almost the same (see Algorithm 2), except the frequentist confidence set is now replaced with the Bayesian credible region.

Original: A credible region with a 1 − α k level of confidence is a set M k such
Revised: A credible region with a 1 − α k level of confidence is a set M k such

Original: that Pr( M k |H k ) ≥ 1 − α k , where Pr( ·|H k ) is the posterior distribution given history H k , and
Revised: that Pr( M k |H k ) ≥ 1 − α k , where Pr( ·|H k ) is the posterior distribution given history H k , and

Original: Pr( M k |H k ) = (cid:82)
Revised: Pr( M k |H k ) = (cid:82)

Original: M k ∈M k Pr( M k |H k ) d M k .
Revised: M k ∈M k Pr( M k |H k ) d M k .

Original: By the construction of the credible region, we have M ∗ ∈
Revised: By the construction of the credible region, we have M ∗ ∈

Original: M k holds with probability 1 − α k given any history H k .
Revised: M k holds with probability 1 − α k given any history H k .

Original: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,
Revised: Therefore, the per-episode Bayesian regret is bounded with probability 1 − α k ,

Original: [Equation]
Revised: [Equation]

Original: where V kh =
Revised: where V kh =

Original: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.
Revised: V π k ,M k h , and the optimism term ˜∆ opt k is less than or equal to 0 by construction.

Original: We define a distance metric d ( M 1 , M 2 )
Revised: We define a distance metric d ( M 1 , M 2 )

Original: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .
Revised: = max π | V π,M 1( s 1 ) − V π,M 2( s 1 ) | .

Original: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .
Revised: It is preferable to have a credible region M such that the set width max M 1 ,M 2 ∈M d ( M 1 , M 2 ) is minimized since the set width certifies an upper bound on ˜∆ conc k .

Original: However, designing a value concentration credible region could be intractable for generic model classes.
Revised: However, designing a value concentration credible region could be intractable for generic model classes.

Original: As a reasonable alternative, we propose the highest density region (HDR) [32] or, in the Bayesian context, the highest posterior density (HPD) region [33], i.e.,
Revised: As a reasonable alternative, we propose the highest density region (HDR) [32] or, in the Bayesian context, the highest posterior density (HPD) region [33], i.e.,

Original: M k = { M k | Pr( M k |H k ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k ) ≥ 1 − α k .
Revised: M k = { M k | Pr( M k |H k ) ≥ ϵ k } , where ϵ k is the largest constant such that Pr( M k |H k ) ≥ 1 − α k .

Original: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.
Revised: This kind of region features a desirable property that it occupies the smallest volume in the sample space among all credible regions of the same confidence level and has a potentially small set width.

Original: There are two problems preventing the constrained BOO from being a practical algorithm.
Revised: There are two problems preventing the constrained BOO from being a practical algorithm.

Original: The first is that the ϵ k in the definition of the HPD region is unknown.
Revised: The first is that the ϵ k in the definition of the HPD region is unknown.

Original: Although we may approximate it with the Monte Carlo approximation [34], an estimator of high/low quantiles has high variance rendering the approximation difficult.
Revised: Although we may approximate it with the Monte Carlo approximation [34], an estimator of high/low quantiles has high variance rendering the approximation difficult.

Original: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [14].
Revised: The other difficulty is that the constrained joint optimization of model and policy is an NP-hard problem even in bandits with linear reward and quadratic constraints [14].

Original: Algorithm 2 Constrained BOO t
Revised: Algorithm 2 Constrained BOO t

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: 4.2 BOO as Lagrangian Relaxation of Constrained BOO
Revised: 4.2 BOO as Lagrangian Relaxation of Constrained BOO

Original: By introducing a Lagrange multiplier λ k , we transform the constrained BOO with HPD regions into an unconstrained optimization problem, max π,M (cid:16) V π,M
Revised: By introducing a Lagrange multiplier λ k , we transform the constrained BOO with HPD regions into an unconstrained optimization problem, max π,M (cid:16) V π,M

Original: [Equation]
Revised: [Equation]

Original: where Pr( M |H k ) =
Revised: where Pr( M |H k ) =

Original: Pr( H k | M )Pr( M ) Pr( H k ) .
Revised: Pr( H k | M )Pr( M ) Pr( H k ) .

Original: Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k ) are irrelevant, and the optimization is equivalent to max π,M (cid:16) V π,M
Revised: Once the Lagrange multiplier is determined, the constant log ϵ k and the marginal likelihood log Pr( H k ) are irrelevant, and the optimization is equivalent to max π,M (cid:16) V π,M

Original: [Equation]
Revised: [Equation]

Original: + λ k (log Pr( H k | M ) + log Pr( M )) (cid:17) .
Revised: + λ k (log Pr( H k | M ) + log Pr( M )) (cid:17) .

Original: This gives rise to the BOO algorithm as shown in Algorithm 3.
Revised: This gives rise to the BOO algorithm as shown in Algorithm 3.

Original: Algorithm 3 BOO
Revised: Algorithm 3 BOO

Original: [Equation]
Revised: [Equation]

Original: However, the problem is how to determine the value of λ k .
Revised: However, the problem is how to determine the value of λ k .

Original: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.
Revised: We note that the following argument by posterior sampling provides an inexact yet insightful viewpoint.

Original: A strict discussion is presented in
Revised: A strict discussion is presented in

Original: Appendix B.
Revised: Appendix B.

Original: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).
Revised: The maximization of the BOO objective can be considered the problem of selecting the best one among multiple posterior samples (see Algorithm 4).

Original: When only one sample is taken,
Revised: When only one sample is taken,

Original: Algorithm 4 reduces to the well-known PSRL [21].
Revised: Algorithm 4 reduces to the well-known PSRL [21].

Original: As the number of samples j goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.
Revised: As the number of samples j goes to infinity, maximizing among all posterior samples gives approximately the solution of the above optimization.

Original: Algorithm 4 BOO via Posterior Sampling
Revised: Algorithm 4 BOO via Posterior Sampling

Original: arg
Revised: arg

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k
Revised: Notice that we can regard Algorithm 4 as maximizing over several identically distributed random variables, λ k

Original: [Equation]
Revised: [Equation]

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: ( s 1 ) .
Revised: ( s 1 ) .

Original: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.
Revised: The magnitude of their mean does not matter since subtracting a constant will not change the optimum.

Original: The thing that matters is their variation.
Revised: The thing that matters is their variation.

Original: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k ) .
Revised: We need to make sure that the variation is not dominated by either V π ik ,M ik( s 1 ) or λ k log Pr( M ik |H k ) .

Original: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.
Revised: If the value dominates the variation, the algorithm will select an unreliable model, which causes inefficiency.

Original: If the variation is dominated by the probability, it shows a strong preference for high-probability models and hesitates to explore.
Revised: If the variation is dominated by the probability, it shows a strong preference for high-probability models and hesitates to explore.

Original: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in
Revised: Theorem 4.1 shows that the standard deviation of log probability is at least a constant (proof in

Original: Appendix A).
Revised: Appendix A).

Original: Theorem 4.1 (Variation of the log-posterior density) .
Revised: Theorem 4.1 (Variation of the log-posterior density) .

Original: Suppose X n =
Revised: Suppose X n =

Original: ( X 1 , X 2 , . . .
Revised: ( X 1 , X 2 , . . .

Original: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .
Revised: , X n ) are observations from a stochastic process whose distribution P θ depends on θ ∈ Θ , an open subset of R m .

Original: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.
Revised: Assume that the posterior is asymptotic normal, and the log-posterior density is continuous.

Original: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .
Revised: The variance of the log-posterior density satisfies lim inf n →∞ Var θ [log Pr( θ | X n )] ≥ m/ 2 .

Original: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distribution
Revised: The process where an algorithm A interacts with a random MDP parameterized by θ is a stochastic process where the observation X t is the state-action pair ( s t , a t ) , and the observation distribution

Original: Pr( X n | θ, A ) is a distribution depending on θ .
Revised: Pr( X n | θ, A ) is a distribution depending on θ .

Original: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log
Revised: Thus, by Theorem 4.1, we know that if the posterior distribution is asymptotic normal, then, for a random sample θ from the posterior distribution, the variance of the random variable log

Original: Pr( θ | X n ) is at least m/ 2 when n is large enough.
Revised: Pr( θ | X n ) is at least m/ 2 when n is large enough.

Original: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [35].
Revised: Theorem 4.1 relies on the asymptotic normality of the posterior distribution, which is satisfied under some regularity conditions specified in [35].

Original: The theorem only certifies a lower bound on the variance rather than establishing its convergence.
Revised: The theorem only certifies a lower bound on the variance rather than establishing its convergence.

Original: Arguably, this is mostly a technical issue arising from the unboundedness of the log-posterior density.
Revised: Arguably, this is mostly a technical issue arising from the unboundedness of the log-posterior density.

Original: It is possible to strengthen this result and derive the convergence of the variance.
Revised: It is possible to strengthen this result and derive the convergence of the variance.

Original: Indeed, it is empirically observed that this quantity converges rapidly to a constant.
Revised: Indeed, it is empirically observed that this quantity converges rapidly to a constant.

Original: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.
Revised: For a proper RL algorithm, it is clear that the standard deviation of V π ik ,M ik( s 1 ) should shrink w.r.t.

Original: k .
Revised: k .

Original: Otherwise, the algorithm fails to find out the optimal value and policy.
Revised: Otherwise, the algorithm fails to find out the optimal value and policy.

Original: Suppose that the standard deviation of V π ik ,M ikis ˜ O ( k − 1 /α ) , where ˜ O is a variant of the big O notation that ignores logarithmic factors.
Revised: Suppose that the standard deviation of V π ik ,M ikis ˜ O ( k − 1 /α ) , where ˜ O is a variant of the big O notation that ignores logarithmic factors.

Original: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k ) matches that of the value term V π ik ,M ik .
Revised: We need to set the scaling parameter λ k proportionally such that the variation of the log-posterior density term log Pr( M ik |H k ) matches that of the value term V π ik ,M ik .

Original: Given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .
Revised: Given a specific model class and noise type, it is possible to derive a worst-case rate for the shrinkage of the value uncertainty and thus determine the proper value of λ k .

Original: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.
Revised: Nonetheless, the rate of information revealing could appear in an instance-dependent manner.

Original: That is to say, the optimal policy of some
Revised: That is to say, the optimal policy of some

Original: MDPs could be inherently easier to determine than the others.
Revised: MDPs could be inherently easier to determine than the others.

Original: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.
Revised: Even in a fixed MDP, the rate of uncertainty shrinkage could also change abruptly.

Original: For example, consider a ReLU bandit problem, where B = { a ∈ R d |∥ a ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a
Revised: For example, consider a ReLU bandit problem, where B = { a ∈ R d |∥ a ∥ 2 ≤ 1 } , and the agent at each episode selects an action a ∈ B and receives a

Original: Gaussian reward of mean max( θ ⊤ a, 0) .
Revised: Gaussian reward of mean max( θ ⊤ a, 0) .

Original: If it happens that the agent selects an action a in the inactive region I = { a ∈ B | θ ⊤ a ≤ 0 } , the resulting observation reveals little information about the optimal action.
Revised: If it happens that the agent selects an action a in the inactive region I = { a ∈ B | θ ⊤ a ≤ 0 } , the resulting observation reveals little information about the optimal action.

Original: Suppose the parameter θ is selected such that the positive region B \ I is of a maximum width ϵ > 0 .
Revised: Suppose the parameter θ is selected such that the positive region B \ I is of a maximum width ϵ > 0 .

Original: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive region, but the uncertainty starts diminishing rapidly whenever the positive region is identified.
Revised: Then, in the worst case, the agent needs to explore Ω(1 /ϵ d − 1 ) actions in order to find the positive region, but the uncertainty starts diminishing rapidly whenever the positive region is identified.

Original: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .
Revised: The view of matching the log probability variation with the value uncertainty provides rationales for the dynamic adjustment of λ k .

Original: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.
Revised: Meanwhile, it points out the potential limitation of decaying λ k with a fixed rate.

Original: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.
Revised: In this paper, we focus on methods that scale λ k with a fixed rate, but as mentioned above, it would be fascinating to adjust it in an instance-dependent manner.

Original: 4.3 BOO Regret
Revised: 4.3 BOO Regret

Original: In this section, we introduce the regret of BOO.
Revised: In this section, we introduce the regret of BOO.

Original: The regret of BOO relies on both the decay rate of the scaling parameter and the complexity of model class.
Revised: The regret of BOO relies on both the decay rate of the scaling parameter and the complexity of model class.

Original: We assume that the model class resides in a d -dimensional RKHS, which roughly means any function in the model class can be represented as a linear function of a potentially unknown finite-dimensional feature map.
Revised: We assume that the model class resides in a d -dimensional RKHS, which roughly means any function in the model class can be represented as a linear function of a potentially unknown finite-dimensional feature map.

Original: This assumption is not very restrictive because it places no restriction on the choice of feature map except for the finitude of the dimension.
Revised: This assumption is not very restrictive because it places no restriction on the choice of feature map except for the finitude of the dimension.

Original: In Appendix B, we derive the asymptotic regret of the BOO algorithm in Theorem B.1.
Revised: In Appendix B, we derive the asymptotic regret of the BOO algorithm in Theorem B.1.

Original: As suggested by Theorem B.1, the optimal asymptotic regret of BOO is O (cid:16) √ K log K (cid:17) , i.e., ˜ O (cid:16) √ K (cid:17) , achieved by setting λ ∗ k = c/ √ k , where λ ∗ k is the optimal scaling parameter and c is a constant associated with the model class.
Revised: As suggested by Theorem B.1, the optimal asymptotic regret of BOO is O (cid:16) √ K log K (cid:17) , i.e., ˜ O (cid:16) √ K (cid:17) , achieved by setting λ ∗ k = c/ √ k , where λ ∗ k is the optimal scaling parameter and c is a constant associated with the model class.

Original: More detailed derivations and conclusions can be found in Appendix B.
Revised: More detailed derivations and conclusions can be found in Appendix B.

Original: We can further interpret λ ∗ k = c/ √ k as λ ∗ k = ξ Vk /ξ Mk , where ξ Mk stands for the variation of the log-posterior density for the model M k , and ξ Vk represents the value uncertainty.
Revised: We can further interpret λ ∗ k = c/ √ k as λ ∗ k = ξ Vk /ξ Mk , where ξ Mk stands for the variation of the log-posterior density for the model M k , and ξ Vk represents the value uncertainty.

Original: In contrast to the discussion in Section 4, where we consider samples from the posterior distribution and the variation of log-posterior density remains constant, the width of the HPD region measured by the variation of log-posterior density is, in fact, growing at a rate depending on the log covering number.
Revised: In contrast to the discussion in Section 4, where we consider samples from the posterior distribution and the variation of log-posterior density remains constant, the width of the HPD region measured by the variation of log-posterior density is, in fact, growing at a rate depending on the log covering number.

Original: Accroding to Lemma D.7 in Appendix D.1, ξ Mk = O (log k ) .
Revised: Accroding to Lemma D.7 in Appendix D.1, ξ Mk = O (log k ) .

Original: Therefore, we have ξ Vk = λ ∗ k ξ Mk = O (cid:16) log k √ k (cid:17) .
Revised: Therefore, we have ξ Vk = λ ∗ k ξ Mk = O (cid:16) log k √ k (cid:17) .

Original: Section 5 will highlight the need of manipulating with ξ Vk and ξ Mk in optimization.
Revised: Section 5 will highlight the need of manipulating with ξ Vk and ξ Mk in optimization.

Original:  Optimization
Revised:  Optimization

Original: In this section, we introduce optimization methods for BOO based on posterior sampling and gradient descent, respectively, and discuss in detail the problems of gradient-based optimization and our proposed solutions.
Revised: In this section, we introduce optimization methods for BOO based on posterior sampling and gradient descent, respectively, and discuss in detail the problems of gradient-based optimization and our proposed solutions.

Original: 5.1 Optimization via Posterior Sampling
Revised: 5.1 Optimization via Posterior Sampling

Original: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.
Revised: Algorithm 4 provides a method of optimizing the BOO objective via posterior sampling.

Original: As discussed previously, the optimal scaling parameter λ k for this algorithm is proportional to the value uncertainty, i.e., λ k =
Revised: As discussed previously, the optimal scaling parameter λ k for this algorithm is proportional to the value uncertainty, i.e., λ k =

Original: ξ Vk = λ ∗ k ξ Mk .
Revised: ξ Vk = λ ∗ k ξ Mk .

Original: The potential advantage of Algorithm 4 over PSRL is that, unlike PSRL, it does not require posterior samples to be independent.
Revised: The potential advantage of Algorithm 4 over PSRL is that, unlike PSRL, it does not require posterior samples to be independent.

Original: Even the requirement that samples are from the posterior can be relaxed.
Revised: Even the requirement that samples are from the posterior can be relaxed.

Original: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.
Revised: These features are essential either when the posterior distribution is approximated or when the posterior samples are produced by Markov chain Monte Carlo methods and are correlated.

Original: 5.2 Optimization via Gradient-Based Methods
Revised: 5.2 Optimization via Gradient-Based Methods

Original: We can also perform the optimization via gradient-based methods.
Revised: We can also perform the optimization via gradient-based methods.

Original: The objective function of BOO consists of two parts, V π,M ik( s 1 ) and λ k (log Pr( H k | M ik ) + log Pr( M ik )) .
Revised: The objective function of BOO consists of two parts, V π,M ik( s 1 ) and λ k (log Pr( H k | M ik ) + log Pr( M ik )) .

Original: The gradients of the loglikelihood and the log-prior are easily obtained.
Revised: The gradients of the loglikelihood and the log-prior are easily obtained.

Original: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [36, 37] (proof in Appendix E.1).
Revised: For the value part, we provide a value model gradient that admits a similar form as the well-known policy gradient [36, 37] (proof in Appendix E.1).

Original: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.
Revised: The value model gradient is amenable to Monte Carlo approximation and can be computed exactly for finite MDPs.

Original: Theorem 5.1 (Value model gradient) .
Revised: Theorem 5.1 (Value model gradient) .

Original: Suppose that the transition function P M θ and reward function
Revised: Suppose that the transition function P M θ and reward function

Original: R M θ of model M θ , the gradient of the value V π,M θ( s 1 ) w.r.t.
Revised: R M θ of model M θ , the gradient of the value V π,M θ( s 1 ) w.r.t.

Original: the model is ∇ θ V π,M θ
Revised: the model is ∇ θ V π,M θ

Original: [Equation]
Revised: [Equation]

Original: where τ = ( s 1 , a 1 , . .
Revised: where τ = ( s 1 , a 1 , . .

Original: , s H , a H ) is a trajectory, τ ∼ π, M θ means that the trajectory is formed by the interaction of the policy π and the model M θ , and P M θ ( s h
Revised: , s H , a H ) is a trajectory, τ ∼ π, M θ means that the trajectory is formed by the interaction of the policy π and the model M θ , and P M θ ( s h

Original: +

Revised: +

Original: | s h , a h ) is the probability of s h + under distribution P M θ ( s h , a h ) .
Revised: | s h , a h ) is the probability of s h + under distribution P M θ ( s h , a h ) .

Original: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [38] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.
Revised: Nevertheless, the value model gradient suffers the same inefficiency just as the policy gradient [38] since the model gradient for a particular state-action pair is 0 whenever it is not visited under the current model and policy.

Original: We propose some techniques to improve the optimization efficiency of BOO, and conduct ablation experiments to verify the effectiveness of our proposed methods in Section G. Two of the most effective techniques are described below, and the rest of the techniques are detailed in Appendix F.
Revised: We propose some techniques to improve the optimization efficiency of BOO, and conduct ablation experiments to verify the effectiveness of our proposed methods in Section G. Two of the most effective techniques are described below, and the rest of the techniques are detailed in Appendix F.

Original: 5.2.1 Mean Reward Bonus
Revised: 5.2.1 Mean Reward Bonus

Original: In the gradient-based optimization, the model becomes optimistic on state-action pairs it visits frequently, which in turn makes these state-action pairs more appealing.
Revised: In the gradient-based optimization, the model becomes optimistic on state-action pairs it visits frequently, which in turn makes these state-action pairs more appealing.

Original: This mutual strengthening phenomenon makes optimization easily stuck at local optima.
Revised: This mutual strengthening phenomenon makes optimization easily stuck at local optima.

Original: One way to solve this problem is to increase the rewards of all state-action pairs, which raises the attractiveness of less visited state-action pairs.
Revised: One way to solve this problem is to increase the rewards of all state-action pairs, which raises the attractiveness of less visited state-action pairs.

Original: This can be achieved by adding a bonus term ξ Vk H E ( s,a ) ∼ U S×A
Revised: This can be achieved by adding a bonus term ξ Vk H E ( s,a ) ∼ U S×A

Original: [ R ( s, a )] to the BOO objective.
Revised: [ R ( s, a )] to the BOO objective.

Original: Here, U S×A is the uniform distribution over the state-action space.
Revised: Here, U S×A is the uniform distribution over the state-action space.

Original: The coefficient ξ Vk ensures that the bonus decays with the value uncertainty.
Revised: The coefficient ξ Vk ensures that the bonus decays with the value uncertainty.

Original: The unvisited state-action pairs will eventually be visited by the policy because they have sufficiently high rewards.
Revised: The unvisited state-action pairs will eventually be visited by the policy because they have sufficiently high rewards.

Original: Our experiments will show that, in tabular setting, this method is very effective.
Revised: Our experiments will show that, in tabular setting, this method is very effective.

Original: However, a concern is that it might fail to scale to high dimensional state-action space.
Revised: However, a concern is that it might fail to scale to high dimensional state-action space.

Original: 5.2.2 Entropy Regularization
Revised: 5.2.2 Entropy Regularization

Original: Another intricacy of the optimization is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.
Revised: Another intricacy of the optimization is that the optimal solution of the BOO objective could change dramatically across the parameter space from episode to episode, which renders the optimization extremely hard.

Original: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.
Revised: A revealing fact is that a small change in the model could change the optimal policy dramatically, making the loss landscape unsmooth.

Original: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.
Revised: Hence, we introduce an entropy-regularized optimization procedure, which starts with a high initial entropy regularization and gradually annealing.

Original: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.
Revised: The entropy plays a role in smoothing the policy’s loss landscape such that the optimal policy will not change drastically when the model changes.

Original: The smoothing effect of entropy regularization is also discussed previously in [39].
Revised: The smoothing effect of entropy regularization is also discussed previously in [39].

Original: The entropy-regularized learning objective mimics the maximum entropy RL [40]:
Revised: The entropy-regularized learning objective mimics the maximum entropy RL [40]:

Original: [Equation]
Revised: [Equation]

Original: where KL( p ∥ q ) stands for the relative entropy, ˆ π is a prior policy ensuring π is absolutely continuous w.r.t.
Revised: where KL( p ∥ q ) stands for the relative entropy, ˆ π is a prior policy ensuring π is absolutely continuous w.r.t.

Original: ˆ π , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.
Revised: ˆ π , the hyperparameter ζ controls the amount of entropy, and ˜ V is the entropy-regularized value.

Original: The entropy term is downscaled in proportion to the value uncertainty such that the influence of regularization diminishes with time.
Revised: The entropy term is downscaled in proportion to the value uncertainty such that the influence of regularization diminishes with time.

Original: We denote by M θ the set of models parameterized by θ .
Revised: We denote by M θ the set of models parameterized by θ .

Original: Let b ϵ be the smallest number ensuring that, for any M θ ∈ M θ , there exists
Revised: Let b ϵ be the smallest number ensuring that, for any M θ ∈ M θ , there exists

Original: π ϵ ∈ Π ϵ =
Revised: π ϵ ∈ Π ϵ =

Original: { π | KL( π ( s, h )
Revised: { π | KL( π ( s, h )

Original: ∥
Revised: ∥

Original: ˆ π ( s, h ))
Revised: ˆ π ( s, h ))

Original: ≤ b ϵ , ∀ s ∈ S , h ∈
Revised: ≤ b ϵ , ∀ s ∈ S , h ∈

Original: [ H ] } such that sup π ∗ V π
Revised: [ H ] } such that sup π ∗ V π

Original: [Equation]
Revised: [Equation]

Original: Then, the value of the maximum entropy policy ˜ π ∗ = arg max ˜ π V ˜ π,M( s 1 ) satisfies that
Revised: Then, the value of the maximum entropy policy ˜ π ∗ = arg max ˜ π V ˜ π,M( s 1 ) satisfies that

Original: [Equation]
Revised: [Equation]

Original: If ϵ is sufficiently small
Revised: If ϵ is sufficiently small

Original: [Equation]
Revised: [Equation]

Original: then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty
Revised: then the sub-optimality gap caused by entropy regularization will decrease at the same rate as the decay of value uncertainty

Original: [Equation]
Revised: [Equation]

Original: which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance
Revised: which ensures that the resulting policy covers multiple uncertain actions without detriment to the performance

Original: [Equation]
Revised: [Equation]

Original: The entropy-regularized value is equivalently defined by the following Bellman backup,
Revised: The entropy-regularized value is equivalently defined by the following Bellman backup,

Original: [Equation]
Revised: [Equation]

Original: where ˜ Q π,MH ( s H , a H ) =
Revised: where ˜ Q π,MH ( s H , a H ) =

Original: ¯ R M
Revised: ¯ R M

Original: ( s H , a H ) .
Revised: ( s H , a H ) .

Original: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [40] or the soft actor-critic algorithm [41].
Revised: The optimization of the policy can be carried out by, for example, the maximum entropy actor-critic algorithm [40] or the soft actor-critic algorithm [41].

Original: In finite MDPs, the optimal value is given by soft value iteration,
Revised: In finite MDPs, the optimal value is given by soft value iteration,

Original: [Equation]
Revised: [Equation]

Original: ) where π ( a h | s h , h ) is the probability of a h under the distribution π ( s h , h ) .
Revised: ) where π ( a h | s h , h ) is the probability of a h under the distribution π ( s h , h ) .

Original: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as
Revised: The maximum entropy policy ˜ π ∗ is then derived from the optimal value as

Original: [Equation]
Revised: [Equation]

Original: The gradient of the entropy-regularized value w.r.t.
Revised: The gradient of the entropy-regularized value w.r.t.

Original: the model is similar to that specified in Theorem 5.1, with the value replaced by the entropy-regularized value.
Revised: the model is similar to that specified in Theorem 5.1, with the value replaced by the entropy-regularized value.

Original:  Experiments
Revised:  Experiments

Original: This section compares the performance of BOO with PSRL and UCRL2 in RiverSwim, Chain, and
Revised: This section compares the performance of BOO with PSRL and UCRL2 in RiverSwim, Chain, and

Original: Random MDPs.
Revised: Random MDPs.

Original: RiverSwim and Chain are hard-exploration MDPs requiring the agent to explore efficiently, while Random MDPs test the average performance.
Revised: RiverSwim and Chain are hard-exploration MDPs requiring the agent to explore efficiently, while Random MDPs test the average performance.

Original: Two implementation of BOO are presented, namely, FiniteBOO and BPS.
Revised: Two implementations of BOO are presented, namely, FiniteBOO and BPS.

Original: FiniteBOO is the BOO with entropy regularization and mean reward bonus mentioned in Section 5.2, which is empirically found to be the best variant in tabular setting according to the ablation study.
Revised: FiniteBOO is the BOO with entropy regularization and mean reward bonus mentioned in Section 5.2, which is empirically found to be the best variant in tabular setting according to the ablation study.

Original: BPS is an implementation of Algorithm 4 (BOO via posterior sampling).
Revised: BPS is an implementation of Algorithm 4 (BOO via posterior sampling).

Original: [Figure]
Revised: [Figure]

Original: 6.1 RiverSwim
Revised: 6.1 RiverSwim

Original: The RiverSwim is an MDP where states are organized in chains, and the agent can move left or right, as shown in Figure 1(a).
Revised: The RiverSwim is an MDP where states are organized in chains, and the agent can move left or right, as shown in Figure 1(a).

Original: Although the rightmost state has a huge reward, the action of moving right fails with a high probability.
Revised: Although the rightmost state has a huge reward, the action of moving right fails with a high probability.

Original: Only a policy moves right at each time period has a small chance of success.
Revised: Only a policy moves right at each time period has a small chance of success.

Original: We start with the experiment on the RiverSwim to demonstrate the performance of our algorithm in the face of high transition uncertainty.
Revised: We start with the experiment on the RiverSwim to demonstrate the performance of our algorithm in the face of high transition uncertainty.

Original: We perform experiments for ten seeds on RiverSwim with |S| =
Revised: We perform experiments for ten seeds on RiverSwim with |S| =

Original: H = 5 , |A| = 2 and record the cumulative regret over 100,000 time steps.
Revised: H = 5 , |A| = 2 and record the cumulative regret over 100,000 time steps.

Original: As shown in Figure 2(a), our algorithm is competitive to PSRL and outperforms UCRL2 significantly.
Revised: As shown in Figure 2(a), our algorithm is competitive to PSRL and outperforms UCRL2 significantly.

Original: 6.2 Chain
Revised: 6.2 Chain

Original: The chain MDP is a variant of the RiverSwim, which has Gaussian rewards and relatively deterministic transitions, as shown in Figure 1(b).
Revised: The chain MDP is a variant of the RiverSwim, which has Gaussian rewards and relatively deterministic transitions, as shown in Figure 1(b).

Original: Although transitions are relatively certain, the stochastic rewards make the problem difficult to explore.
Revised: Although transitions are relatively certain, the stochastic rewards make the problem difficult to explore.

Original: The horizon H and the number of states |S| are equal to the length of the chain.
Revised: The horizon H and the number of states |S| are equal to the length of the chain.

Original: We evaluate our algorithms in Chain of a length L ∈ { 10 , 20 , 40 } for 100 , 000 episodes and ten random trials.
Revised: We evaluate our algorithms in Chain of a length L ∈ { 10 , 20 , 40 } for 100 , 000 episodes and ten random trials.

Original: Figure 2(d ∼ f) illustrates that our algorithm compares favorably with PSRL and
Revised: Figure 2(d ∼ f) illustrates that our algorithm compares favorably with PSRL and

Original: UCRL2, which certifies the effectiveness of BOO in problems requiring long-term planning.
Revised: UCRL2, which certifies the effectiveness of BOO in problems requiring long-term planning.

Original: 6.3 Random MDPs
Revised: 6.3 Random MDPs

Original: Random MDPs are tabular MDP models randomly generated from a prior distribution and used to test the general performance of the algorithm.
Revised: Random MDPs are tabular MDP models randomly generated from a prior distribution and used to test the general performance of the algorithm.

Original: We randomly generate 100 stochastic MDPs with |S| = |A| =
Revised: We randomly generate 100 stochastic MDPs with |S| = |A| =

Original: H = 5 and |S| = H = 20 , |A| =
Revised: H = 5 and |S| = H = 20 , |A| =

Original: from the prior and measure the performance of algorithms over 10 , 000 timesteps.
Revised: from the prior and measure the performance of algorithms over 10 , 000 timesteps.

Original: As shown in
Revised: As shown in

Original:       (SLVRGHV      
Revised:       (SLVRGHV      

Original:  & X P X O D WL Y H  U H J U H W D5LYHU6ZLP       (SLVRGHV 
Revised:  & X P X O D WL Y H  U H J U H W D5LYHU6ZLP       (SLVRGHV 

Original: 
Revised: 

Original:        E5DQGRP0'3VZLWKVL]H 
Revised:        E5DQGRP0'3VZLWKVL]H 

Original:      (SLVRGHV      F5DQGRP0'3VZLWKVL]H %3 )LQLWH%2 365/ 8&5/       (SLVRGHV  
Revised:      (SLVRGHV      F5DQGRP0'3VZLWKVL]H %3 )LQLWH%2 365/ 8&5/       (SLVRGHV  

Original:      & X P X O D WL Y H  U H J U H W G&KDLQZLWKOHQJWK       (SLVRGHV        H&KDLQZLWKOHQJWK       (SLVRGHV 
Revised:      & X P X O D WL Y H  U H J U H W G&KDLQZLWKOHQJWK       (SLVRGHV        H&KDLQZLWKOHQJWK       (SLVRGHV 

Original: 
Revised: 

Original:    I&KDLQZLWKOHQJWK
Revised:    I&KDLQZLWKOHQJWK

Original: Figure 2(b), the performance of BOO and BPS is close to PSRL, and both algorithms outperform
Revised: Figure 2(b), the performance of BOO and BPS is close to PSRL, and both algorithms outperform

Original: UCRL2 significantly.
Revised: UCRL2 significantly.

Original: However, the performance gap between FiniteBOO and PSRL in Figure 2(c) indicates that there remains a challenge in the optimization of large-scale MDPs.
Revised: However, the performance gap between FiniteBOO and PSRL in Figure 2(c) indicates that there remains a challenge in the optimization of large-scale MDPs.

Original:  Conclusion
Revised:  Conclusion

Original: This paper proposes BOO as a generic model-based reinforcement learning algorithm.
Revised: This paper proposes BOO as a generic model-based RL algorithm.

Original: It is provably sample-efficient and enjoys ˜ O ( √ K ) regret for models in a finite-dimensional RKHS, where K is the number of episodes.
Revised: It is provably sample-efficient and enjoys ˜ O ( √ K ) regret for models in a finite-dimensional RKHS, where K is the number of episodes.

Original: To optimize the BOO objective, we propose the value model gradient and optimization techniques, such as entropy regularization, to improve its efficiency.
Revised: To optimize the BOO objective, we propose the value model gradient and optimization techniques, such as entropy regularization, to improve its efficiency.

Original: Through our experiments, we have shown that BOO is competitive with PSRL and outperforms UCRL greatly.
Revised: Through our experiments, we have shown that BOO is competitive with PSRL and outperforms UCRL2 greatly.

Original: However, to apply BOO in real-world reinforcement learning problems, there remains lots of work to be done.
Revised: However, to apply BOO in real-world RL problems, there remains lots of work to be done.

Original: Importantly, we need to develop methods that further facilitate the gradient-based optimization of BOO in large-scale problems.
Revised: Importantly, we need to develop methods that further facilitate the gradient-based optimization of BOO in large-scale problems.

Original: It is also an appealing direction for future work to adapt the scaling parameter of BOO on an instance-dependent basis.
Revised: It is also an appealing direction for future work to adapt the scaling parameter of BOO on an instance-dependent basis.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work is supported by National Key Research and Development Program of China (2020AAA 0107200), the National Science Foundation of China (61921006, 61876119, 62276126), the Natural
Revised: This work is supported by National Key Research and Development Program of China (2020AAA 0107200), the National Science Foundation of China (61921006, 61876119, 62276126), the Natural

Original: Science Foundation of Jiangsu (BK20221442), and the Fundamental Research Funds for the Central
Revised: Science Foundation of Jiangsu (BK20221442), and the Fundamental Research Funds for the Central

Original: Universities (022114380010).
Revised: Universities (022114380010).