Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees
Anonymous Author(s)
AffiliationAddress
Abstract
We consider reinforcement learning in an environment modeled by an episodic,finite, stage-dependent Markov decision process of horizon H with S states, and Aactions.
The performance of an agent is measured by the regret after interactingwith the environment for T episodes.
We propose an optimistic posterior sampling algorithm for reinforcement learning ( OPSRL ), a simple variant of posteriorsampling that only needs a number of posterior samples logarithmic in H , S , A ,and T per state-action pair.
For OPSRL we guarantee a high-probability regretbound of order at most (cid:101) O ( √ H 3 SAT ) ignoring poly log( HSAT ) terms.
The keynovel technical ingredient is a new sharp anti-concentration inequality for linearforms which may be of independent interest.
Specifically, we extend the normalapproximation-based lower bound for Beta distributions by Alfers and Dinges[1984] to Dirichlet distributions.
Our bound matches the lower bound of orderΩ( √ H 3 SAT ) , thereby answering the open problems raised by Agrawal and Jia[2017b] for the episodic setting.
Introduction
In reinforcement learning an agent interacts with an environment, whose underlying mechanism isunknown, by sequentially taking actions, receiving rewards, and transitioning to the next state [Suttonand Barto, 1998].
With the goal of maximizing the expected sum of the collected rewards the agentmust carefully balance between exploring in order to gather more information about the environmentand exploiting the current knowledge to collect the rewards.
In this paper, we are interested in solvingthis exploration-exploitation dilemma by injecting noise into the agent’s decision-making process.
We model the environment as an episodic, finite, unknown Markov decision process (MDP) of horizon
H, with S states and A actions.
In particular, we consider the stage-dependent setting where therewards and the transition probability distributions can vary within an episode.
After T episodes, theperformance of an agent is measured through regret which is the difference between the cumulativereward the agent could have obtained by acting optimally and what the agent really obtained.
Jin et al.
[2018] and Domingues et al.
[2020] provide a problem-independent lower bound of orderΩ( √ H 3 SAT ) for this setting; see also Azar et al.
[2017] for a lower bound when the transitions arestage-independent.
One generic solution to the exploration-exploitation dilemma is the principle of optimism in the faceof uncertainty .
A simple way to implement this principle consists in building upper confidence bound(UCB) on the optimal Q-value function through the addition of bonuses to the rewards.
This is doneby either model-based algorithms [Azar et al., 2017, Dann et al., 2017, Zanette and Brunskill, 2019]or model-free algorithms [Jin et al., 2018, Zhang et al., 2020, Menard et al., 2021]; see also [Jakschet al., 2010, Fruit et al., 2018, Talebi and Maillard, 2018] for the non-episodic setting.
Notably, among others, both the upper confidence bound value iteration ( UCBVI ) of Azar et al.
[2017] and the
UCB-Advantage algorithm of Zhang et al.
[2020] enjoys a problem-independent regret bound 1 oforder 2 (cid:101) O ( √ H 3 SAT ) that matches the aforementioned lower bound for T large enough and up toterms poly-logarithmic in H, S, A, T .
Another way is to implement the optimism by injecting noise .
A typical example is the randomleast-square value iteration ( RLSVI , Osband et al., 2016b, Russo, 2019) algorithm which at eachepisode computes new Q-values by noisy value iteration from an estimated model and then actsgreedily with respect to them.
In particular, a Gaussian noise is added to the reward before applyingthe Bellman operator to encourage exploration.
Indeed, when the variance of the noise is carefullychosen, it allows to obtain optimistic Q-values with at least a fixed probability.
Russo [2019] firstproved a regret bound of order (cid:101)
O
( H 2 S 3 / 2 √ AT ) for RLSVI .
Later, Xiong et al.
[2021] obtained anoptimal regret bound of order (cid:101) O ( √ H 3 SAT ) for a modified version of RLSVI where the varianceof the injected Gaussian noise is scaled by a term similar to the Bernstein bonuses used in UCBVI .
Added: Note that the RLSVI was also successfully extended beyond the tabular case to settings withfunction approximation, e.g. see Ishfaq et al., 2021, Zanette et al., 2020.
Recently, Pacchiano et al.
[2021] analyzed a version of RLSVI where the Gaussian noise is replacedby a bootstrap sample of the past rewards and added pseudo rewards in the same fashion as Kvetonet al.
The algorithm proposed by Pacchiano et al.
[2021], comes with a regret bound of order(cid:101)
O ( H 2 S √ AT ) .
By generalizing the Thompson sampling algorithm
[Thompson, 1933] originally given for stochasticmulti-armed bandit, Osband et al.
[2013] propose a posterior sampling for reinforcement learning( PSRL ).
PSRL algorithm also relies on noise to drive exploration.
The general idea behind it is tomaintain a surrogate Bayesian model on the MDP, for instance, a Dirichlet posterior on the transitionprobability distribution if the rewards are known.
At each episode, a new MDP is sampled (i.e., atransition probability for each state-action pair) according to the posterior distribution of the Bayesianmodel.
Then, the agent acts optimally in this sampled MDP.
As the posterior is not well concentratedin the unexplored region of the MDP, the probability that the Q-value of the sampled MDP is optimisticin this region is high.
Therefore, the agent will be incentivized to explore.
Although the original Thompson sampling is well-studied in the frequentist setting Added: [Agrawal and Goyal, 2012,
Kaufmann et al., 2012, Agrawal and Goyal, 2013, Zhang, 2022] and the Bayesian setting [Thompson,1933, Russo and Roy, 2016, Russo and Van Roy, 2014], most of the analysis of PSRL only provide
Bayesian regret bounds [Osband et al., 2013, Abbasi-Yadkori and Szepesvári, 2015, Osband et al. ,2016b, Ouyang et al., 2017, Osband and Van Roy, 2017], i.e., when the true MDP is effectivelysampled according to the prior of the surrogate Bayesian model.
Despite this lack of guarantees, PSRLdemonstrates competitive empirical performance in comparison to bonus-based algorithms [Osbandet al., 2013, Osband and Van Roy, 2017].
Additionally, the exploration mechanism used by PSRL(and RLSVI ) was successfully extended outside the tabular setting and used in deep RL environments[Osband et al., 2016a, 2018, 2019].
One exception to the above is the work of Agrawal and Jia [2017b] that studies PSRL from a frequentistperspective in the infinite-horizon, non-episodic average reward setting.
In particular, they provide aregret bound 3 of order 4 (cid:101)
O
( H 2 S √ AT ) for an optimistic version of PSRL that we call SOS-OPS-RLsince it switches between two types of sampling of the transitions: (1) simple optimistic sampling ,when the number of observed transitions at a given state-action pair is too small.
In this case,the sampled transition is a random mixture between the uniform distribution over the states andan empirical estimate of the true transition biased by some bonus-like terms; or if the number ofobserved transitions at a given state-action pair is large enough (2) optimistic posterior sampling,where (cid:101) O ( S ) samples from an inflated Dirichlet posterior are used instead of one sample used in PSRL .
As acknowledged by the authors, there was a mistake in the initial submission of their work where the previously announced bound was claimed to be √ better, see Agrawal and Jia [2017a], et al.
Then, from these (cid:101) O ( S ) sampled transition probabilities we select the most optimistic one i.e., theone leading to the largest optimal Q-value.
The key idea underpinning the analysis of SOS-OPS-RL , and PSRL -like algorithms in general, isto control the deviations of the Dirichlet posterior on the transition probability distributions.
Inparticular, we need to show that the posterior spreads enough to ensure optimism .
To this end,
Agrawal and Jia [2017b] derive an anti-concentration bound for any fixed projection of a Dirichletrandom vector.
The latter result in turn relies upon an equivalent representation of a Dirichlet vectorin terms of independent Beta random variables and an anti-concentration bound for the corresponding
Beta distribution.
However, this anti-concentration inequality is not uniformly tight, in particular itspolynomial dependence on the number of states S is suboptimal.
Agrawal and Jia [2017b] conclude with two open problems.
The first question is whether one canreduce the number of posterior samples required per state-action pair from (cid:101)
O ( S ) to constant orlogarithmic in S .
The second asks if it is possible to obtain a near-optimal regret bound and inparticular to improve the dependence on S .
In this paper, we answer both of them in the affirmativein the episodic setting.
Indeed, we propose optimistic posterior sampling algorithm for reinforcementlearning ( OPSRL ) that only requires (cid:101) O (1) samples from an inflated posterior while enjoying a nearoptimal problem independent regret bound of order (cid:101) O ( √ H 3 SAT ) .
OPSRL is a simple optimisticvariant of PSRL which, in particular, does not rely at all on ”simple” (bonus-based) optimisticsampling.
The essential ingredient for OPSRL ’s analysis is our novel anti-concentration bound for the projectionsof a Dirichlet random vector (Theorem 3.3).
We base it on a tight Gaussian approximation forlinear forms of a Dirichlet random vector.
This latter approximation can be seen as a substantialgeneralization to Dirichlet distributions of the result obtained by Alfers and Dinges [1984] for the case of Beta distributions.
We obtain this approximation through a refined non-asymptotic analysisof the integral representation for the density of a linear form of a Dirichlet random vector, which wasfirst derived 5 by Tiapkin et al.
We believe that the new anti-concentration inequality presentedin this work could be of independent interest, e.g., to tighten or simplify analysis of non-parametricThompson sampling like algorithms [Riou and Honda, 2020, Baudry et al., 2021a,b] for stochastic multi-armed bandits.
• We propose the OPSRL algorithm for tabular, stage-dependent, episodic RL.
It is a simple optimistic variant of the PSRL algorithm that only needs (cid:101) O
(1) posterior samples per state-action pair.
For OPSRL , we provide a regret bound of order (cid:101) O ( √ H 3 SAT ) matching the problem independentlower bound up to poly- log terms.
In particular we answer positively to two open questions by
Agrawal and Jia [2017b] in the episodic setting.
• We derive a new anti-concentration inequality for a linear form of a Dirichlet random vector
(Theorem 3.3) which is essential for the analysis of OPSRL .
This result is a generalization to theDirichlet case of the one provided by Alfers and Dinges [1984] for Beta distributions.
Setting
We consider a finite episodic MDP (cid:0) S , A , H, { p h } h ∈ [ H ] , { r h } h ∈
[ H ] (cid:1) , where S is the set of states,
A is the set of actions, H is the number of steps in one episode, p h ( s ′ | s, a ) is the probabilitytransition from state s to state s ′ by taking the action a at step h, and r h ( s, a ) ∈ [0 , 1] is the boundeddeterministic 6 reward received after taking the action a in state s at step h .
Note that we consider thegeneral case of rewards and transition functions that are possibly non-stationary, i.e., that are allowedto depend on the decision step h in the episode.
We denote by S and A the number of states andactions, respectively.
Policy & value functions A deterministic policy π is a collection of functions π h : S
→ A for all 128 h ∈
[ H ] , where every π h maps each state to a single action.
The value functions of π , denoted by
V πh , as well as the optimal value functions, denoted by V ⋆h are given by the Bellman and the optimal 130 Bellman equations,
[Equation]
where by definition, V ⋆H +1 ≜ V πH +1 ≜ 0 .
Furthermore, p h f ( s, a ) ≜ E s ′ ∼ p h ( ·| s,a ) [ f ( s ′ )] denotes theexpectation operator with respect to the transition probabilities p h and π h g ( s ) ≜
g ( s, π h ( s )) denotesthe composition with the policy π at step h .
Learning problem The agent, to which the transitions are unknown (the rewards are assumedto be known for simplicity), interacts with the environment during T episodes of length H , witha fixed initial state s 1 .
Before each episode t the agent selects a policy π t based only on the pastobserved transitions up to episode t − 1 .
At each step h ∈
[ H ] in episode t , the agent observes a states th ∈ S , takes an action π th ( s th )
=
a th ∈ A and makes a transition to a new state s th +1 according tothe probability distribution p h ( s th , a th ) and receives a deterministic reward r h ( s th , a th ) .
Regret The quality of an agent is measured through its regret, that is the difference between what itcould obtain (in expectation) by acting optimally and what it really gets,
[Equation]
Counts The number of times the state action-pair ( s, a ) was visited in step h in the first tepisodes is denoted as n th ( s, a ) ≜ (cid:80) ti =1 1 (cid:8) ( s ih , a ih ) =
( s, a ) (cid:9) .
Next, we define n th ( s ′ | s, a ) ≜(cid:80) ti =1 1 (cid:8) ( s ih , a ih , s ih +1 ) =
( s, a, s ′ )
(cid:9) the number of transitions from s to s ′ at step h .
Improper Dirichlet distribution For m ∈ N ∗ , the probability simplex of dimension m is denotedby ∆ m .
For α ∈ ( R ++ ) m +1 , we denote by D ir( α ) the Dirichlet distribution on ∆ m with parameter α .
We also extend this distribution to improper parameter α ∈ ( R + ) m +1 such that (cid:80) mi =
α i > 0 by injecting D ir(( α i ) i : α i > 0 ) into ∆ m .
Precisely, we say that p ∼ D ir( α ) if ( p i ) i : α i > 0 ∼ D ir(( α i )
i : α i > 0 )and all other coordinates are zero.
Additional notation For N ∈ N ++ , we define the set [ N ] ≜ { 1 , . . .
, N } .
We denote theuniform distribution over this set by U
nif[ N ] .
The vector of dimension N with all entries oneis 1 N ≜ (1 , . .
. , 1) T .The
empirical probability distribution (cid:98) p th ( s, a ) is defined as (cid:98)
p th
( s ′ | s, a ) =n th ( s ′ | s, a ) /n
th ( s, a ) if n th ( s, a ) > 0 and (cid:98) p 0 h ( s ′ | s, a ) =
/S
otherwise.
Appendix A references allthe notation used.
Algorithm
In this section we describe the OPSRL algorithm.
In spirit, OPSRL proceeds similarly as PSRL exceptthat it uses several posterior samples instead and acts optimistically with respect to them, explainingthe name Optimistic Posterior Sampling for Reinforcement Learning ( OPSRL ).
Optimistic pseudo-state
In order to define the prior used by OPSRL , we extend the state space S byan absorbing pseudo-state s 0 with reward r h ( s 0 , a ) ≜ r 0 > 1 for all h, a and transition probabilitydistribution p h ( s ′ | s 0 , a ) ≜ 1 { s ′ =
s 0 } .
A similar pseudo-state was already introduce in previousworks, see for example Brafman and Tennenholtz [2002], Szita and L˝orincz [2008].
We denote by
S ′ =
S ∪ { s 0 } the augmented states space and by ∆ S ′ the set of probability distributions over S ′ .
Pseudo-counts We define the pseudo-counts, n th ( s, a ) ≜ n th ( s, a ) + n 0 , as the counts shifted by aninitial value n 0 .
This shift corresponds to prior transitions to the pseudo-state, that is n th ( s ′ | s,
a ) ≜n th
( s ′ | s, a ) + n 0 1 { s ′ =
s 0 } .
Similar to the empirical transitions, we define a pseudo-empiricaltransition probability distribution as p th ( s, a ) =
n th ( s ′ | s, a ) /n
th ( s, a ) .
Inflated Bayesian model Like PSRL , we define a Bayesian model on the transition probabilitydistributions, except that the prior/posterior is inflated.
The practice of inflating the posterior iscommon in the analysis of Thompson sampling like algorithm, see Agrawal and Jia [2017b], Abeilleand Lazaric [2017].
Precisely, the inflated prior is a Dirichlet distribution D ir (cid:16)(cid:0) n 0 h ( s ′ | s, a ) /κ (cid:1)
s ′
∈S ′ (cid:17)parameterized by the initial pseudo-counts, and some constant κ > 0 controlling the inflation.
Thusthe prior is a Dirac distribution at a deterministic transition leading to the artificial state s 0 .
Thenthe inflated posterior is also a Dirichlet distribution D ir (cid:16)(cid:0) n th ( s ′ | s, a ) /κ
(cid:1) s ′ ∈S ′ (cid:17) .
Note that the prioris a proper prior (i.e., a valid probability distribution), but it will be updated in an improper way,i.e., probability transitions with no mass under the prior could get mass in the posterior, as they getpositive counts.
Optimistic posterior sampling After episode t, for each state-action pair ( s, a ) and steph ∈
[ H ] we sample J independent transition probability distributions (cid:101) p t,jh ( s, a ) ∼
D ir (cid:16)(cid:0) n th ( s ′ | s, a ) /κ (cid:1) s ′ ∈S ′ (cid:17) from the inflated posterior.
Then, the Q-values are obtained by optimistic backward induction with these transitions.
Precisely the value after the last step is zero
V tH +1 ( s ) ≜ 0 and the optimal Bellman equations become
[Equation]
The next policy is greedy with the Q-values π t +1 h ( s ) ∈ arg max a ∈A Q th ( s, a ) .
The complete procedure of OPSRL is described in Algorithm 1 for a general family of distributions parameterized by thepseudo-counts over the transitions instead of the inflated Dirichlet prior/posterior.
[Equation]
h =
π t h ( s t h ) .
Observe s th +1 ∼ p
h ( s th , a th ) . Increment the pseudo-count n th ( s th +1 | s th , a th ) . 9: end for 10: end for
3.1 Analysis
We fix δ ∈ (0 , 1) and the number of samples
[Equation]
where c J = 1 / log(2 / (1+Φ(1))) and Φ( · ) is the cumulative distribution function (CDF) of a normaldistribution.
Note that J has a logarithmic dependence on S, A, H, T, and 1 /δ .
We now state the regret bound of OPSRL with a full proof in Appendix B. and a sketch in Section 3.2.
Theorem 3.1.
Consider a parameter δ ∈ (0 , 1) .
Let κ ≜ 2(log(12 SAH/δ )
+ 3 log(e π (2 T +1)))
n 0 ≜ ⌈ κ ( c 0 + log 17 / 16 ( T )) ⌉ , r 0 ≜ 2 , where c 0 is an absolute constant defined in (4) ; see
Appendix B.2.
Then for OPSRL , with probability at least 1 − δ ,
[Equation]
where L ≜ O (log( HSAT/δ )) .
Computational complexity OPSRL is a model-based algorithm, and thus gets the O ( HS 2 A ) spacecomplexity as PSRL .
Since we need (cid:101) O (1) posterior samples per state-action pair the time complexityof OPSRL is of order (cid:101)
O
( HS 2 A ) per episode, the same as PSRL up to poly-logarithmic terms.
Buildingon the idea of Efroni et al.
[2019], in Appendix F we propose the Lazy-OPSRL algorithm a moretime-efficient version of OPSRL .
Instead of recomputing the Q-value by backward induction beforeeach episode, Lazy-OPSRL only performs one step of optimistic incremental planning at the visitedstates.
It enjoys a regret bound of the same order (cid:101) O ( √ H 3 SAT ) as OPSRL but with an improvedtime-complexity per episode of O ( HSA ) , see Theorem F.1 in Appendix F.
Comparison with SOS-OPS-RL and PSRL
One structural difference between OPSRL and
SOS-OPS-RL of Agrawal and Jia [2017a] is that OPSRL only relies on optimistic posterior sampling while SOS-OPS-RL also uses simple optimistic sampling: a mixture of the uniform distributionover the states and an empirical estimate of the true transition kernel biased by some bonus-liketerms.
In particular, OPSRL does not use bonus-like quantities which could lead to poor empiricalperformance [Osband and Van Roy, 2017].
Another important issue is the number of posteriorsamples.
SOS-OPS-RL needs (cid:101) O ( S ) posterior samples in order to obtain a regret bound of order(cid:101)
O ( H 2 S √ AT ) whereas OPSRL needs only (cid:101) O
(1) samples and obtains a better regret bound.
Notethat if we choose the number of posterior samples as J = 1 in OPSRL we recover PSRL up to twotechnical differences: First, the posterior is inflated in order to increase its variance.
This technicaltrick was already used by Agrawal and Jia [2017a] and allows to guarantee optimism with a smallnumber of posterior samples, see Section 3.2.
Second, OPSRL uses a particular prior which is a Diracdistribution at a deterministic transition towards an optimistic pseudo-state.
This prior is needed tocontrol the deviations of the (inflated) posterior, see Theorem D.2.
Comparison with RLSVI
Both OPSRL and RLSVI build on the same mechanism for exploration.
RLSVI just adds an Gaussian noise to the Q-values whereas OPSRL injects the noise naturally viaa random transition sampled from a Dirichlet distribution.
As controlling the deviation of the
Q-value obtained with additive Gaussian noise is not difficult, the analysis of RLSVI is relativelystraightforward Added: [Russo, 2019, Ishfaq et al., 2021].
On the contrary the analysis of OPSRL ismuch more involved, see Section 3.2.
However, the benefit of optimistic posterior sampling in OPSRLis that it adapts automatically to the variance of the estimates of the transitions which is central fora regret bound with an optimal dependence on the horizon H [Azar et al., 2017].
Adapting to thevariance with RLSVI is much more involved and artificial, see Xiong et al.
This is probablyone reason why RLSVI performs empirically worse than PSRL
[Osband et al., 2016a].
3.2 Proof sketch
The proof of Theorem 3.1 consists of three important steps.
The first step is devoted to the approximation for tails of weighted sums of Dirichlet distribution and embodies the main technical contributionof the paper.
Step 1.
Exponential and Gaussian approximation for Dirichlet distribution
The first resultgeneralizes Riou and Honda [2020] to Dirichlet distributions with real parameters.
Let us first recallthe definition of the minimum Kullback-Leibler divergence for p ∈ ∆ m where m ∈ N + , a functionf : { 0 , . . .
, m } → [0 , b ] for some b ∈ R + and u ∈ R ,
[Equation]
where we recall that pf ≜ E X ∼ p
f ( X ) .
This quantity appears already in the analysis of nonparametric bounded multi-arm stochastic bandits, see Honda and Takemura [2010], Cappé et al.[2013].
As the Kullback-Leibler divergence, the minimum Kullback-Leibler divergence admits avariational formula by Lemma 18 of Garivier et al.
[2018] up to rescaling for any u ∈ (0 , b ) ,
[Equation]
Theorem 3.2 (Exponential upper bound, see Theorem D.1) .
For any α = ( α 0 , α 1 , . . .
, α m ) ∈
R m +1++define
p ∈ ∆ m such that p ( ℓ ) = α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Then for anyf : { 0 , . . .
, m } → [0 , b ] and 0 < µ < b, we have
[Equation]
The second result is devoted to a tight Gaussian lower bound for the distribution of a linear functionof Dirichlet random vector.
Here we follow the ideas of Alfers and Dinges [1984] and use the exactexpression for the density of a linear form of Dirichlet random vector derived by Tiapkin et al.
Theorem 3.3 (Gaussian lower bound, see Theorem D.2) .
For any α = ( α 0 + 1 , α 1 , . . .
, α m ) ∈
R m +1++ , define p ∈ ∆ m such that p ( ℓ )
=
α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Fix ε ∈ (0 , 1)and assume that α 0 ≥ c ( ε ) + log 17 / 16 ( α ) for c ( ε ) defined in (11) , Appendix D, and α ≥ 2 α 0 .
Then for any f : { 0 , . . .
, m } → [0 , b 0 ] such that f (0) =
b 0 , f ( j ) ≤ b < b 0 / 2 , j ∈ { 1 , . . .
, m } andµ ∈ ( pf, b 0 ) ,
[Equation]
We emphasize that increasing the parameter α 0 corresponding to the largest value of f by 1 is crucial.
The same technique was used by Alfers and Dinges [1984] to derive a lower bound on the tails of the
Beta distribution.
Remark 3.4 .
We stress that the anti-concentration inequality of Tiapkin et al.
[2022, Theorem D.2]is not sufficient for our purposes; their additional factor α − 3 / 2 in front of the exponent makes itunusable for the analysis of OPSRL .
Indeed, this inequality would imply (cid:101)
O
( T 3 / 2 ) samples from theinflated posterior in order to get optimism with high-probability, whereas with our refined bound(Theorem 3.3) we only need (cid:101) O
(1) posterior samples.
Proof sketch of Theorem 3.3.
We start from the integral representation for the density by Tiapkinet al.
[2022, Proposition D.3].
Define Z ≜ wf for w ∼ D
ir
( α 0 + 1 , α 1 , . .
α m ) , then for anyu ∈ (0 , b 0 ) ,
[Equation]
One additional term (1 + i( b 0 − u ) s ) − 1 comes from increasing the parameter α 0 by 1 correspondingto the value f (0) =
b 0 .
In the same spirit as it was done by Tiapkin et al.
[2022], we apply the method of saddle point (see
Fedoryuk, 1977, Olver, 1997) to the complex integral above.
Informally, for α 0 , α, b 0 large enoughthe following approximation holds
[Equation]
where λ ⋆ is the unique solution to the problem (2) and σ
= E X ∼ p (cid:2) ( f ( X ) −
[Equation]
⋆ ( f ( X ) − u ) ) 2 (cid:3) .
The formalstatement can be found in Lemma D.5 of Appendix D.
Next we perform a change of variables t 2 / 2 = K inf ( p, u, f ) in the above expression to get
[Equation]
where ϕ ( x | µ, σ 2 ) is a density of N ( µ, σ 2 ) and D ( u ) is a weight function bounded from below by 1(see Lemma D.6 of Appendix D).
This lower bound on D ( u ) concludes the proof.
Comparison with anti-concentration bound by Agrawal and Jia [2017b]
We emphasise that ourtechnique of deriving a Gaussian-like lower bound is substantially different from the methodologyused by Agrawal and Jia [2017b].
The latter one was based on reduction of a weighted sum of Dirichlet random vector to a weighted sum of independent Beta distributed random variables and asubsequent application of the Berry-Esseen inequality, whereas our approach relies on the integralrepresentation for the density of the corresponding linear projection of Dirichlet random vector.
Added: In particular, the Berry-Esseen inequality is likely to be very coarse since it uses only thefirst three moments of the distribution and therefore generates an additional S -factor.
At the sametime, our analysis is much better fitted to the Dirichlet distribution and provides a very tight lowerbound.
The tightness of our bounds can be checked by comparing it to a similar result for the betadistribution derived in Alfers and Dinges [1984].
Step 2.
Optimism Next, we apply Theorem 3.3 to prove that the estimate of Q-function Q th isoptimistic with high probability for our choice of inflation parameter κ and a number of posteriorsamples J :
Q th ( s, a ) ≥ Q ⋆h ( s, a ) for any ( s, a, h, t ) ∈ S ×
A ×
[ H ] × [ T ] .
We show that the inequalities max j ∈ [ J ] { (cid:101) p t,
jh V ⋆h +
( s, a ) } ≥ p
h V ⋆h +
( s, a ) hold for all ( s, a, h, t ) ∈
S ×
A ×
[ H ] ×
[ T ] with high probability.
First, we notice that (cid:101) p t,jh ( s, a ) ∼ D ir
( α 0 + 1 , α 1 , . .
. , α S )for α 0 = n 0 /κ − 1 , α i = n th ( s i | s, a ) /κ and α = ( n th ( s, a ) − κ ) /κ .
Additionally, define a probabilitydistribution q ∈ ∆ S such that q ( i ) =
α i /α .
This distribution slightly differs from p th ( s, a ) becauseof an additional +1 in the parameters of the Dirichlet distribution.
Next, we may apply Theorem 3.3with ε = 1 / 2 and a proper choice of n 0 = n 0 ( ε ) ,
[Equation]
, where ζ ≜ ( n t h − κ ) K inf ( q, p h V[-⋆h +
-]
⋆ h +1 ( s, a ) , V ⋆ h +1 ) and Φ( · ) is a cumulative distribution function (CDF)of a standard normal distribution.
By a concentration argument we have
[Equation]
with high probability for an appropriate choice of κ = (cid:101) O (1) .
For this step of the proofthe presence of the inflation parameter κ is crucial: this parameter increases the variance of(cid:101) p t,jh ( s, a ) to ensure that the above inequality holds with a constant probability.
Next, by taking the maximum over J =
O
(log( SATH/δ ))
samples and applying union bound, we guarantee that the inequality max j ∈ [ J ] { (cid:101) p t,
jh V ⋆h
+
( s, a ) } ≥ p
h V ⋆h +
( s, a ) holds simultaneously for all( s, a, h, t ) ∈ S
×
A ×
[ H ] ×
[ T ] with probability at least 1 − δ/ 2 .
The formal statement and theproof could be found in Proposition B.4 of Appendix B.2.
Finally, the standard backward induction over h ∈ [ H ] concludes optimism.
Indeed, the base ofinduction h
= H + 1 is trivial.
Next, by the Bellman equations for Q th and Q ⋆h we have
[Equation]
The induction hypothesis implies V th +1 ( s ′ ) ≥ Q th +1 ( s ′ ,
π ⋆ ( s ′ ))
≥ Q ⋆h +1 ( s ′ , π ⋆ ( s ′ ))
=
V ⋆h +
( s ′ )
for any s ′ ∈ S .
Hence,
[Equation]
with probability at least 1 − δ/ 2 .
Step 3.
Regret bound The rest of proof directly follows Azar et al.
[2017], where UCBVI algorithmwith Bernstein bonuses was analyzed.
By the optimism, we have
[Equation]
where δ th ≜ V t − 1 h ( s th ) − V π t h ( s th ) .
The quantity δ th can be decomposed as follows using the Bellmanequation for V π t and Q t − 1 h ,
[Equation]
The terms ( C ) , ( D ) , and ξ th are standard in the analysis of the optimistic algorithms.
The term ( B )could be upper-bounded by r 0 · n 0 ·
H n t − 1 h ( s th ,a th ) and turns out to be one of second-order terms.
The analysisof ( A ) is novel and requires application of the Bernstein inequality for Dirichlet distributions thatfollows from Theorem 3.2 and is spelled out in the following lemma.
Lemma 3.5 (see Lemma C.6 in Appendix C) .
For any α = ( α 0 , α 1 , . . .
, α m ) ∈
R m +1++ definep ∈ ∆ m such that p ( ℓ ) =
α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Then for any f : { 0 , . .
, m } →[0 , b ] such that f (0) =
b and δ ∈ (0 , 1) ,
[Equation]
≤ δ.
As opposed to Lemma C.8 of Tiapkin et al.
[2022], the last result applies to Dirichlet distributions withnon-integer parameters as in our case (due to the presence of the inflation parameter κ ).
Therefore, wesee that the term ( A ) can be upper bounded by a quantity which has the same role as in the analysisof UCBVI .
After using the Bernstein bound, the rest of the proof follows from the analysis of UCBVIwith the Bernstein bonuses and Bayes-UCBVI ; see Azar et al. [2017] and Tiapkin et al.
Conclusion
In this work, we presented the OPSRL algorithm which can be viewed as a simple optimistic variantof the PSRL algorithm.
Notably, OPSRL only needs (cid:101) O
(1) posterior samples per state-action.
Weproved that the regret of OPSRL is upper-bounded with high probability by (cid:101) O ( √ H 3 SAT ) , matchingthe problem-independent lower-bound of order Ω( √ H 3 SAT ) for T large enough and up to termspoly-logarithmic in H, S, A, and T .
While our work addresses the open questions raised by Agrawaland Jia [2017b] in the episodic setting, obtaining the same results in the infinite-horizon averagereward setting remains an open issue.
We believe that it is possible to adapt our analysis to thisother setting up to some technical adjustments.
Ultimately, another open question, is to obtain a high-probability regret bound for PSRL , that is, when using only a single posterior sample and notinflating the posterior.
As a further future research direction we believe it could be interesting toobtain a model-free algorithm that relies on the same mechanism as OPSRL for exploration.
Indeed,such an algorithm could avoid the use of complicated bonuses adopted by the current model-freealgorithms while reducing the memory complexity of OPSRL .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]