Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees
Abstract
We consider reinforcement learning in an environment modeled by an episodic, finite, stage-dependent Markov decision process of horizon H with S states, and A actions.
The performance of an agent is measured by the regret after interacting with the environment for T episodes.
We propose an optimistic posterior sampling algorithm for reinforcement learning ( OPSRL ), a simple variant of posterior sampling that only needs a number of posterior samples logarithmic in H , S , A , and T per state-action pair.
For OPSRL we guarantee a high-probability regret bound of order at most (cid:101) O ( √ H 3 SAT ) ignoring poly log( HSAT ) terms.
The key novel technical ingredient is a new sharp anti-concentration inequality for linear forms which may be of independent interest.
Specifically, we extend the normal approximation-based lower bound for Beta distributions by Alfers and Dinges [1984] to Dirichlet distributions.
Our bound matches the lower bound of order Ω( √ H 3 SAT ) , thereby answering the open problems raised by Agrawal and Jia [2017b] for the episodic setting.
Introduction
In reinforcement learning an agent interacts with an environment, whose underlying mechanism is unknown, by sequentially taking actions, receiving rewards, and transitioning to the next state [Sutton and Barto, 1998].
With the goal of maximizing the expected sum of the collected rewards, the agent must carefully balance between exploring in order to gather more information about the environment and exploiting the current knowledge to collect the rewards.
In this paper, we are interested in solving this exploration-exploitation dilemma by injecting noise into the agent’s decision-making process.
We model the environment as an episodic, finite, unknown Markov decision process (MDP) of horizon
H, with S states and A actions.
In particular, we consider the stage-dependent setting where the rewards and the transition probability distributions can vary within an episode.
After T episodes, the performance of an agent is measured through regret which is the difference between the cumulative reward the agent could have obtained by acting optimally and what the agent really obtained.
Jin et al.
[2018] and Domingues et al.
[2020] provide a problem-independent lower bound of order Ω( √ H 3 SAT ) for this setting; see also Azar et al.
[2017] for a lower bound when the transitions are stage-independent.
One generic solution to the exploration-exploitation dilemma is the principle of optimism in the face of uncertainty .
A simple way to implement this principle consists in building upper confidence bound (UCB) on the optimal Q-value function through the addition of bonuses to the rewards.
This is done by either model-based algorithms [Azar et al., 2017, Dann et al., 2017, Zanette and Brunskill, 2019] or model-free algorithms [Jin et al., 2018, Zhang et al., 2020, Menard et al., 2021]; see also [Jaksch et al., 2010, Fruit et al., 2018, Talebi and Maillard, 2018] for the non-episodic setting.
Notably, among others, both the upper confidence bound value iteration ( UCBVI ) of Azar et al.
[2017] and the
UCB-Advantage algorithm of Zhang et al.
[2020] enjoys a problem-independent regret bound 1 of order 2 (cid:101) O ( √ H 3 SAT ) that matches the aforementioned lower bound for T large enough and up to terms poly-logarithmic in H, S, A, T .
Another way is to implement the optimism by injecting noise .
A typical example is the random least-square value iteration ( RLSVI , Osband et al., 2016b, Russo, 2019) algorithm which at each episode computes new Q-values by noisy value iteration from an estimated model and then acts greedily with respect to them.
In particular, a Gaussian noise is added to the reward before applying the Bellman operator to encourage exploration.
Indeed, when the variance of the noise is carefully chosen, it allows to obtain optimistic Q-values with at least a fixed probability.
Russo [2019] first proved a regret bound of order (cid:101)
O
( H 2 S 3 / 2 √ AT ) for RLSVI .
Later, Xiong et al.
[2021] obtained an optimal regret bound of order (cid:101) O ( √ H 3 SAT ) for a modified version of RLSVI where the variance of the injected Gaussian noise is scaled by a term similar to the Bernstein bonuses used in UCBVI .
Note that the RLSVI was also successfully extended beyond the tabular case to settings with function approximation, e.g. see Ishfaq et al., 2021, Zanette et al., 2020.
Recently, Pacchiano et al.
[2021] analyzed a version of RLSVI where the Gaussian noise is replaced by a bootstrap sample of the past rewards and added pseudo rewards in the same fashion as Kveton et al.
The algorithm proposed by Pacchiano et al.
[2021], comes with a regret bound of order (cid:101)
O ( H 2 S √ AT ) .
By generalizing the Thompson sampling algorithm
[Thompson, 1933] originally given for stochastic multi-armed bandit, Osband et al.
[2013] propose a posterior sampling for reinforcement learning ( PSRL ).
PSRL algorithm also relies on noise to drive exploration.
The general idea behind it is to maintain a surrogate Bayesian model on the MDP, for instance, a Dirichlet posterior on the transition probability distribution if the rewards are known.
At each episode, a new MDP is sampled (i.e., a transition probability for each state-action pair) according to the posterior distribution of the Bayesian model.
Then, the agent acts optimally in this sampled MDP.
As the posterior is not well concentrated in the unexplored region of the MDP, the probability that the Q-value of the sampled MDP is optimistic in this region is high.
Therefore, the agent will be incentivized to explore.
Although the original Thompson sampling is well-studied in the frequentist setting [Agrawal and Goyal, 2012,
Kaufmann et al., 2012, Agrawal and Goyal, 2013, Zhang, 2022] and the Bayesian setting [Thompson, 1933, Russo and Roy, 2016, Russo and Van Roy, 2014], most of the analysis of PSRL only provide
Bayesian regret bounds [Osband et al., 2013, Abbasi-Yadkori and Szepesvári, 2015, Osband et al., 2016b, Ouyang et al., 2017, Osband and Van Roy, 2017], i.e., when the true MDP is effectively sampled according to the prior of the surrogate Bayesian model.
Despite this lack of guarantees, PSRL demonstrates competitive empirical performance in comparison to bonus-based algorithms [Osband et al., 2013, Osband and Van Roy, 2017].
Additionally, the exploration mechanism used by PSRL (and RLSVI ) was successfully extended outside the tabular setting and used in deep RL environments [Osband et al., 2016a, 2018, 2019].
One exception to the above is the work of Agrawal and Jia [2017b] that studies PSRL from a frequentist perspective in the infinite-horizon, non-episodic average reward setting.
In particular, they provide a regret bound 3 of order 4 (cid:101) O
( H 2 S √ AT ) for an optimistic version of PSRL that we call SOS-OPS-RL since it switches between two types of sampling of the transitions: (1) simple optimistic sampling , when the number of observed transitions at a given state-action pair is too small.
In this case, the sampled transition is a random mixture between the uniform distribution over the states and an empirical estimate of the true transition biased by some bonus-like terms; or if the number of observed transitions at a given state-action pair is large enough (2) optimistic posterior sampling, where (cid:101) O ( S ) samples from an inflated Dirichlet posterior are used instead of one sample used in PSRL .
Then, from these (cid:101) O ( S ) sampled transition probabilities we select the most optimistic one i.e., the one leading to the largest optimal Q-value.
The key idea underpinning the analysis of SOS-OPS-RL , and PSRL -like algorithms in general, is to control the deviations of the Dirichlet posterior on the transition probability distributions.
In particular, we need to show that the posterior spreads enough to ensure optimism .
To this end,
Agrawal and Jia [2017b] derive an anti-concentration bound for any fixed projection of a Dirichlet random vector.
The latter result in turn relies upon an equivalent representation of a Dirichlet vector in terms of independent Beta random variables and an anti-concentration bound for the corresponding
Beta distribution.
However, this anti-concentration inequality is not uniformly tight, in particular its polynomial dependence on the number of states S is suboptimal.
Agrawal and Jia [2017b] conclude with two open problems.
The first question is whether one can reduce the number of posterior samples required per state-action pair from (cid:101)
O ( S ) to constant or logarithmic in S .
The second asks if it is possible to obtain a near-optimal regret bound and in particular to improve the dependence on S .
In this paper, we answer both of them in the affirmative in the episodic setting.
Indeed, we propose optimistic posterior sampling algorithm for reinforcement learning ( OPSRL ) that only requires (cid:101) O (1) samples from an inflated posterior while enjoying a nearoptimal problem independent regret bound of order (cid:101) O ( √ H 3 SAT ) .
OPSRL is a simple optimistic variant of PSRL which, in particular, does not rely at all on ”simple” (bonus-based) optimistic sampling.
The essential ingredient for OPSRL ’s analysis is our novel anti-concentration bound for the projections of a Dirichlet random vector (Theorem 3.3).
We base it on a tight Gaussian approximation for linear forms of a Dirichlet random vector.
This latter approximation can be seen as a substantial generalization to Dirichlet distributions of the result obtained by Alfers and Dinges [1984] for the case of Beta distributions.
We obtain this approximation through a refined non-asymptotic analysis of the integral representation for the density of a linear form of a Dirichlet random vector, which was first derived 5 by Tiapkin et al.
We believe that the new anti-concentration inequality presented in this work could be of independent interest, e.g., to tighten or simplify analysis of non-parametric
Thompson sampling like algorithms [Riou and Honda, 2020, Baudry et al., 2021a,b] for stochastic
multi-armed bandits.
• We propose the OPSRL algorithm for tabular, stage-dependent, episodic RL.
It is a simple optimistic variant of the PSRL algorithm that only needs (cid:101) O
(1) posterior samples per state-action pair.
For OPSRL , we provide a regret bound of order (cid:101) O ( √ H 3 SAT ) matching the problem independent lower bound up to poly- log terms.
In particular we answer positively to two open questions by Agrawal and Jia [2017b] in the episodic setting.
• We derive a new anti-concentration inequality for a linear form of a Dirichlet random vector (Theorem 3.3) which is essential for the analysis of OPSRL .
This result is a generalization to the Dirichlet case of the one provided by Alfers and Dinges [1984] for Beta distributions
Setting
We consider a finite episodic MDP (cid:0) S , A , H, { p h } h ∈ [ H ] , { r h } h ∈
[ H ] (cid:1) , where S is the set of states,
A is the set of actions, H is the number of steps in one episode, p h ( s ′ | s, a ) is the probability transition from state s to state s ′ by taking the action a at step h, and r h ( s, a ) ∈ [0 , 1] is the bounded deterministic 6 reward received after taking the action a in state s at step h .
Note that we consider the general case of rewards and transition functions that are possibly non-stationary, i.e., that are allowed to depend on the decision step h in the episode.
We denote by S and A the number of states and actions, respectively.
Policy & value functions A deterministic policy π is a collection of functions π h : S
→ A for all h ∈
[ H ] , where every π h maps each state to a single action.
The value functions of π , denoted by
V πh , as well as the optimal value functions, denoted by V ⋆h are given by the Bellman and the optimal Bellman equations,
[Equation]
where by definition, V ⋆H +1 ≜ V πH +1 ≜ 0 .
Furthermore, p h f ( s, a ) ≜ E s ′ ∼ p h ( ·| s,a ) [ f ( s ′ )] denotes the expectation operator with respect to the transition probabilities p h and π h g ( s ) ≜
g ( s, π h ( s )) denotes the composition with the policy π at step h .
Learning problem The agent, to which the transitions are unknown (the rewards are assumed to be known for simplicity), interacts with the environment during T episodes of length H , with a fixed initial state s 1 .
Before each episode t the agent selects a policy π t based only on the past observed transitions up to episode t − 1 .
At each step h ∈
[ H ] in episode t , the agent observes a state s th ∈ S , takes an action π th ( s th )
=
a th ∈ A and makes a transition to a new state s th +1 according to the probability distribution p h ( s th , a th ) and receives a deterministic reward r h ( s th , a th ) .
Regret The quality of an agent is measured through its regret, that is the difference between what it could obtain (in expectation) by acting optimally and what it really gets,
[Equation]
Counts The number of times the state action-pair ( s, a ) was visited in step h in the first t episodes is denoted as n th ( s, a ) ≜ (cid:80) ti =1 1 (cid:8) ( s ih , a ih ) =
( s, a ) (cid:9) .
Next, we define n th ( s ′ | s, a ) ≜ (cid:80) ti =1 1 (cid:8) ( s ih , a ih , s ih +1 ) =
( s, a, s ′ )
(cid:9) the number of transitions from s to s ′ at step h .
Improper Dirichlet distribution For m ∈ N ∗ , the probability simplex of dimension m is denoted by ∆ m .
For α ∈ ( R ++ ) m +1 , we denote by D ir( α ) the Dirichlet distribution on ∆ m with parameter α .
We also extend this distribution to improper parameter α ∈ ( R + ) m +1 such that (cid:80) mi =
α i > 0 by injecting D ir(( α i ) i : α i > 0 ) into ∆ m .
Precisely, we say that p ∼ D ir( α ) if ( p i ) i : α i > 0 ∼ D ir(( α i )
i : α i > 0 ) and all other coordinates are zero.
Additional notation For N ∈ N ++ , we define the set [ N ] ≜ { 1 , . . .
, N } .
We denote the uniform distribution over this set by U nif[ N ] .
The vector of dimension N with all entries one is 1 N ≜ (1 , . .
. , 1) T .
The empirical probability distribution (cid:98) p th ( s, a ) is defined as (cid:98)
p th ( s ′ | s, a ) = n th ( s ′ | s, a ) /n
th ( s, a ) if n th ( s, a ) > 0 and (cid:98) p 0 h ( s ′ | s, a ) =
/S
otherwise.
Appendix A references all the notation used.
Algorithm
In this section we describe the OPSRL algorithm.
In spirit, OPSRL proceeds similarly as PSRL except that it uses several posterior samples instead and acts optimistically with respect to them, explaining the name Optimistic Posterior Sampling for Reinforcement Learning ( OPSRL ).
Optimistic pseudo-state
In order to define the prior used by OPSRL , we extend the state space S by an absorbing pseudo-state s 0 with reward r h ( s 0 , a ) ≜ r 0 > 1 for all h, a and transition probability distribution p h ( s ′ | s 0 , a ) ≜ 1 { s ′ =
s 0 } .
A similar pseudo-state was already introduce in previous works, see for example Brafman and Tennenholtz [2002], Szita and L˝orincz [2008].
We denote by
S ′ =
S ∪ { s 0 } the augmented states space and by ∆ S ′ the set of probability distributions over S ′ .
Pseudo-counts We define the pseudo-counts, n th ( s, a ) ≜ n th ( s, a ) + n 0 , as the counts shifted by an initial value n 0 .
This shift corresponds to prior transitions to the pseudo-state, that is n th ( s ′ | s, a ) ≜ n th ( s ′ | s, a ) + n 0 1 { s ′ =
s 0 } .
Similar to the empirical transitions, we define a pseudo-empirical transition probability distribution as p th ( s, a ) =
n th ( s ′ | s, a ) /n
th ( s, a ) .
Inflated Bayesian model Like PSRL , we define a Bayesian model on the transition probability distributions, except that the prior/posterior is inflated.
The practice of inflating the posterior is common in the analysis of Thompson sampling like algorithm, see Agrawal and Jia [2017b], Abeille and Lazaric [2017].
Precisely, the inflated prior is a Dirichlet distribution D ir (cid:16)(cid:0) n 0 h ( s ′ | s, a ) /κ (cid:1)
s ′
∈S ′ (cid:17) parameterized by the initial pseudo-counts, and some constant κ > 0 controlling the inflation.
Thus the prior is a Dirac distribution at a deterministic transition leading to the artificial state s 0 .
Then the inflated posterior is also a Dirichlet distribution D ir (cid:16)(cid:0) n th ( s ′ | s, a ) /κ
(cid:1) s ′ ∈S ′ (cid:17) .
Note that the prior is a proper prior (i.e., a valid probability distribution), but it will be updated in an improper way, i.e., probability transitions with no mass under the prior could get mass in the posterior, as they get positive counts.
Optimistic posterior sampling After episode t, for each state-action pair ( s, a ) and step h ∈
[ H ] we sample J independent transition probability distributions (cid:101) p t,jh ( s, a ) ∼
D ir (cid:16)(cid:0) n th ( s ′ | s, a ) /κ (cid:1) s ′ ∈S ′ (cid:17) from the inflated posterior.
Then, the Q-values are obtained by optimistic backward induction with these transitions.
Precisely the value after the last step is zero
V tH +1 ( s ) ≜ 0 and the optimal Bellman equations become
[Equation]
The next policy is greedy with the Q-values π t +1 h ( s ) ∈ arg max a ∈A Q t h ( s, a ) .
The complete procedure of OPSRL is described in Algorithm 1 for a general family of distributions parameterized by the pseudo-counts over the transitions instead of the inflated Dirichlet prior/posterior.
3.1 Analysis
We fix δ ∈ (0 , 1) and the number of samples
[Equation]
where c J = 1 / log(2 / (1+Φ(1))) and Φ( · ) is the cumulative distribution function (CDF) of a normal distribution.
Note that J has a logarithmic dependence on S, A, H, T, and 1 /δ .
We now state the regret bound of OPSRL with a full proof in Appendix B. and a sketch in Section 3.2.
Theorem 3.1.
Consider a parameter δ ∈ (0 , 1) .
Let κ ≜ 2(log(12 SAH/δ )
+ 3 log(e π (2 T + 1)))
n 0 ≜ ⌈ κ ( c 0 + log 17 / 16 ( T )) ⌉ , r 0 ≜ 2 , where c 0 is an absolute constant defined in (4) ; see
Appendix B.2.
Then for OPSRL , with probability at least 1 − δ ,
[Equation]
1: Input: Family of probability distributions ρ : N S +1+ → ∆ S ′ over transitions, initial pseudocount n 0 h , number of posterior samples J .
2: for t ∈ [ T ] do 3:
For all ( s, a, h ) ∈ S × A ×
[ H ] , sample J independent transitions (cid:101) p t − 1 ,j h ( s, a ) ∼ ρ (cid:0) n t − 1 h ( s ′ | s, a )
s ′ ∈S ′ (cid:1) , j ∈ [ J ] .
{+Optimistic backward induction: set V t − 1 H +
+}
( s )
= 0 and recursively for h ∈
[ H ] , compute
Q t − 1 h ( s, a ) =
r h ( s, a ) + max j ∈ [ J ] (cid:8)(cid:101) p t − 1 ,j h
V t − 1 h
{++
+}
( s, a ) (cid:9) ,
V t − 1 h ( s ) = max a ∈A Q t − 1 h ( s, a ) , π th ( s ) ∈ arg max a ∈A
Q t − 1 h ( s, a ) .
5: for h ∈ [ H ] do 6:
Play a th = π th
( s th ) .
Observe s th +1 ∼ p
h ( s th , a th ) . Increment the pseudo-count n th ( s th +1 | s th , a th ) . 9: end for 10: end for where L ≜ O (log( HSAT/δ )) .
Computational complexity OPSRL is a model-based algorithm, and thus gets the O ( HS 2 A ) space complexity as PSRL .
Since we need (cid:101) O (1) posterior samples per state-action pair the time complexity of OPSRL is of order (cid:101)
O ( HS 2 A ) per episode, the same as PSRL up to poly-logarithmic terms.
Building on the idea of Efroni et al.
[2019], in Appendix F we propose the Lazy-OPSRL algorithm a more time-efficient version of OPSRL .
Instead of recomputing the Q-value by backward induction before each episode, Lazy-OPSRL only performs one step of optimistic incremental planning at the visited states.
It enjoys a regret bound of the same order (cid:101) O ( √ H 3 SAT ) as OPSRL but with an improved time-complexity per episode of O ( HSA ) , see Theorem F.1 in Appendix F.
Comparison with SOS-OPS-RL and PSRL
One structural difference between OPSRL and
SOS-OPS-RL of Agrawal and Jia [2017a] is that OPSRL only relies on optimistic posterior sampling while SOS-OPS-RL also uses simple optimistic sampling: a mixture of the uniform distribution over the states and an empirical estimate of the true transition kernel biased by some bonus-like terms.
In particular, OPSRL does not use bonus-like quantities which could lead to poor empirical performance [Osband and Van Roy, 2017].
Another important issue is the number of posterior samples.
SOS-OPS-RL needs (cid:101) O ( S ) posterior samples in order to obtain a regret bound of order (cid:101) O
( H 2 S √ AT ) whereas OPSRL needs only (cid:101) O (1) samples and obtains a better regret bound.
Note that if we choose the number of posterior samples as J = 1 in OPSRL we recover PSRL up to two technical differences: First, the posterior is inflated in order to increase its variance.
This technical trick was already used by Agrawal and Jia [2017a] and allows to guarantee optimism with a small number of posterior samples, see Section 3.2.
Second, OPSRL uses a particular prior which is a Dirac distribution at a deterministic transition towards an optimistic pseudo-state.
This prior is needed to control the deviations of the (inflated) posterior, see Theorem D.2.
Comparison with RLSVI
Both OPSRL and RLSVI build on the same mechanism for exploration.
RLSVI just adds an Gaussian noise to the Q-values whereas OPSRL injects the noise naturally via a random transition sampled from a Dirichlet distribution.
As controlling the deviation of the
Q-value obtained with additive Gaussian noise is not difficult, the analysis of RLSVI is relatively straightforward [Russo, 2019, Ishfaq et al., 2021].
On the contrary the analysis of OPSRL is much more involved, see Section 3.2.
However, the benefit of optimistic posterior sampling in OPSRL is that it adapts automatically to the variance of the estimates of the transitions which is central for a regret bound with an optimal dependence on the horizon H [Azar et al., 2017].
Adapting to the variance with RLSVI is much more involved and artificial, see Xiong et al.
This is probably one reason why RLSVI performs empirically worse than PSRL
[Osband et al., 2016a].
3.2 Proof sketch
The proof of Theorem 3.1 consists of three important steps.
The first step is devoted to the approximation for tails of weighted sums of Dirichlet distribution and embodies the main technical contribution of the paper.
Step 1.
Exponential and Gaussian approximation for Dirichlet distribution The first result generalizes Riou and Honda [2020] to Dirichlet distributions with real parameters.
Let us first recall the definition of the minimum Kullback-Leibler divergence for p ∈ ∆ m where m ∈ N + , a function f : { 0 , . . .
, m } → [0 , b ] for some b ∈ R + and u ∈ R ,
[Equation]
where we recall that pf ≜ E X ∼ p
f ( X ) .
This quantity appears already in the analysis of nonparametric bounded multi-arm stochastic bandits, see Honda and Takemura [2010], Cappé et al.
As the Kullback-Leibler divergence, the minimum Kullback-Leibler divergence admits a variational formula by Lemma 18 of Garivier et al.
[2018] up to rescaling for any u ∈ (0 , b ) ,
[Equation]
Theorem 3.2 (Exponential upper bound, see Theorem D.1) .
For any α = ( α 0 , α 1 , . . .
, α m ) ∈
R m +1++ define p ∈ ∆ m such that p ( ℓ ) = α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Then for any f : { 0 , . .
, m } → [0 , b ] and 0 < µ < b, we have
[Equation]
The second result is devoted to a tight Gaussian lower bound for the distribution of a linear function of Dirichlet random vector.
Here we follow the ideas of Alfers and Dinges [1984] and use the exact expression for the density of a linear form of Dirichlet random vector derived by Tiapkin et al.
Theorem 3.3 (Gaussian lower bound, see Theorem D.2) .
For any α = ( α 0 + 1 , α 1 , . . .
, α m ) ∈
R m +1++ , define p ∈ ∆ m such that p ( ℓ )
=
α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Fix ε ∈ (0 , 1) and assume that α 0 ≥ c ( ε ) + log 17 / 16 ( α ) for c ( ε ) defined in (11) , Appendix D, and α ≥ 2 α 0 .
Then for any f : { 0 , . . .
, m } → [0 , b 0 ] such that f (0) =
b 0 , f ( j ) ≤ b < b 0 / 2 , j ∈ { 1 , . . .
, m } and µ ∈ ( pf, b 0 ) ,
[Equation]
We emphasize that increasing the parameter α 0 corresponding to the largest value of f by 1 is crucial.
The same technique was used by Alfers and Dinges [1984] to derive a lower bound on the tails of the
Beta distribution.
Remark 3.4 .
We stress that the anti-concentration inequality of Tiapkin et al.
[2022, Theorem D.2] is not sufficient for our purposes; their additional factor α − 3 / 2 in front of the exponent makes it unusable for the analysis of OPSRL .
Indeed, this inequality would imply (cid:101)
O
( T 3 / 2 ) samples from the inflated posterior in order to get optimism with high-probability, whereas with our refined bound (Theorem 3.3) we only need (cid:101) O
(1) posterior samples.
Proof sketch of Theorem 3.3.
We start from the integral representation for the density by Tiapkin et al.
[2022, Proposition D.3].
Define Z ≜ wf for w ∼ D
ir
( α 0 + 1 , α 1 , . .
α m ) , then for any u ∈ (0 , b 0 ) ,
[Equation]
One additional term (1 + i( b 0 − u ) s ) − 1 comes from increasing the parameter α 0 by 1 corresponding to the value f (0) =
b 0 .
In the same spirit as it was done by Tiapkin et al.
[2022], we apply the method of saddle point (see
Fedoryuk, 1977, Olver, 1997) to the complex integral above.
Informally, for α 0 , α, b 0 large enough the following approximation holds
[Equation]
where λ ⋆ is the unique solution to the problem (2) and σ
= E X ∼ p (cid:2) (
[Equation]
− u ) ) 2 (cid:3) .
The formal statement can be found in Lemma D.5 of Appendix D.
Next we perform a change of variables t 2 / 2 = K inf ( p, u, f ) in the above expression to get
[Equation]
where ϕ ( x | µ, σ 2 ) is a density of N ( µ, σ 2 ) and D ( u ) is a weight function bounded from below by(see Lemma D.6 of Appendix D).
This lower bound on D ( u ) concludes the proof.
Comparison with anti-concentration bound by Agrawal and Jia [2017b]
We emphasise that our technique of deriving a Gaussian-like lower bound is substantially different from the methodology used by Agrawal and Jia [2017b].
The latter one was based on reduction of a weighted sum of
Dirichlet random vector to a weighted sum of independent Beta distributed random variables and a subsequent application of the Berry-Esseen inequality, whereas our approach relies on the integral representation for the density of the corresponding linear projection of Dirichlet random vector.
In particular, the Berry-Esseen inequality is likely to be very coarse since it uses only the first three moments of the distribution and therefore generates an additional S -factor.
At the same time, our analysis is much better fitted to the Dirichlet distribution and provides a very tight lower bound.
The tightness of our bounds can be checked by comparing it to a similar result for the beta distribution derived in Alfers and Dinges [1984].
Step 2.
Optimism Next, we apply Theorem 3.3 to prove that the estimate of Q-function Q th is optimistic with high probability for our choice of inflation parameter κ and a number of posterior samples J : Q th ( s, a ) ≥ Q ⋆h ( s, a ) for any ( s, a, h, t ) ∈ S ×
A ×
[ H ] × [ T ] .
We show that the inequalities max j ∈ [ J ] { (cid:101) p t,
jh V ⋆h +
( s, a ) } ≥ p
h V ⋆h +
( s, a ) hold for all ( s, a, h, t ) ∈
S ×
A ×
[ H ] ×
[ T ] with high probability.
First, we notice that (cid:101) p t,jh ( s, a ) ∼ D ir
( α 0 + 1 , α 1 , . .
. , α S ) for α 0 = n 0 /κ − 1 , α i = n th ( s i | s, a ) /κ and α = ( n t h ( s, a ) − κ ) /κ .
Additionally, define a probability distribution q ∈ ∆ S such that q ( i ) =
α i /α .
This distribution slightly differs from p th ( s, a ) because of an additional +1 in the parameters of the Dirichlet distribution.
Next, we may apply Theorem 3. with ε = 1 / 2 and a proper choice of n 0 = n 0 ( ε ) ,
[Equation]
p t [Equation] jh [Equation]
⋆h
[Equation]
1 ( s, a ) ≥ p [Equation] V ⋆h
[Equation]
1 ( s [Equation] a [Equation]
(cid:32)
[Equation]
(cid:32)(cid:114)
[Equation]
{+ζ κ (cid:33)(cid:33) , where ζ ≜ ( n th − κ ) K inf ( q, p h V ⋆h +
+}
( s, a ) , V ⋆h +1 ) and Φ( · ) is a cumulative distribution function (CDF) of a standard normal distribution.
By a concentration argument we have
[Equation]
with high probability for an appropriate choice of κ = (cid:101) O (1) .
For this step of the proof the presence of the inflation parameter κ is crucial: this parameter increases the variance of (cid:101) p t,jh ( s, a ) to ensure that the above inequality holds with a constant probability.
Next, by taking the maximum over J =
O
(log( SATH/δ ))
samples and applying union bound, we guarantee that the inequality max j ∈ [ J ] { (cid:101) p t,
jh V ⋆h
+
( s, a ) } ≥ p
h V ⋆h +
( s, a ) holds simultaneously for all ( s, a, h, t ) ∈ S ×
A ×
[ H ] ×
[ T ] with probability at least 1 − δ/ 2 .
The formal statement and the proof could be found in Proposition B.4 of Appendix B.2.
Finally, the standard backward induction over h ∈ [ H ] concludes optimism.
Indeed, the base of induction h
= H + 1 is trivial.
Next, by the Bellman equations for Q th and Q ⋆h we have
[Equation]
The induction hypothesis implies V th +1 ( s ′ ) ≥ Q th [Equation] [Equation]
s [Equation] , π
[Equation] s ′ [Equation] ) ≥
[-= V ⋆h +
-][Equation]
) for any s ′ ∈ S .
Hence,
[Equation]
with probability at least 1 − δ/ 2 .
Step 3.
Regret bound The rest of proof directly follows Azar et al.
[2017], where UCBVI algorithm with Bernstein bonuses was analyzed.
By the optimism, we have
[Equation]
where δ th ≜ V t − 1 h ( s th ) − V π t h ( s th ) .
The quantity δ th can be decomposed as follows using the Bellman equation for V π t and Q t − 1 h ,
[Equation]
The terms ( C ) , ( D ) , and ξ th are standard in the analysis of the optimistic algorithms.
The term ( B ) could be upper-bounded by [Equation]
n t − 1 h ( s th ,a th ) and turns out to be one of second-order terms.
The analysis of ( A ) is novel and requires application of the Bernstein inequality for Dirichlet distributions that follows from Theorem 3.2 and is spelled out in the following lemma.
Lemma 3.5 (see Lemma C.6 in Appendix C) .
For any α = ( α 0 , α 1 , . . .
, α m ) ∈
R m +1++ define p ∈ ∆ m such that p ( ℓ ) =
α ℓ /α, ℓ = 0 , . . .
, m , where α = (cid:80) mj
=
α j .
Then for any f : { 0 , . .
, m } → [0 , b ] such that f (0) =
b and δ ∈ (0 , 1) ,
[Equation]
As opposed to Lemma C.8 of Tiapkin et al.
[2022], the last result applies to Dirichlet distributions with non-integer parameters as in our case (due to the presence of the inflation parameter κ ).
Therefore, we see that the term ( A ) can be upper bounded by a quantity which has the same role as in the analysis of UCBVI .
After using the Bernstein bound, the rest of the proof follows from the analysis of UCBVI with the Bernstein bonuses and Bayes-UCBVI ; see Azar et al.
[2017] and Tiapkin et al.
Experiments
In this section we provide experiment to compare OPSRL with some baselines on simple tabular environment; see details in Appendix G.
In particular, we illustrate that OPSRL is competitive with the original PSRL algorithm and outperforms bonus-based algorithms such as UCBVI . Baselines We compare OPSRL with the following baselines: UCBVI (with Hoeffding-type bonuses) and UCBVI-B (with Bernstein-type bonuses) Azar et al. [2017], PSRL Osband et al.
[2013], and RLSVI
Osband et al.
[2016b].
See Appendix G for full details on parameters for OPSRL and baselines.
[Figure]
Results In Figure 1, we plot the regret of the various baselines and OPSRL in the grid world environment.
In this experiment, we observe that OPSRL achieves competitive results with respect to
PSRL .
It is not completely surprising since they share the same Bayesian model on the transitions up to the prior.
We shall elaborate more on the influence of the prior in Appendix G. We also note that
OPSRL outperforms UCBVI and RLSVI .
This difference may be explained by the fact that OPSRL ’s optimism implies (in the worst case) KL bonuses as in Filippi et al.
The KL bonuses are stronger than Bernstein bonuses, see Lemma E.1, because they somehow rely on all moments of the empirical distribution rather than the first two moments as in the case of Bernstein bonuses or first moments for Hoeffding bonuses or for the variance of the Gaussian noise in RLSVI .
Note also that in
OPSRL , we do not have to solve the complex convex program to compute the KL bonuses Filippi et al.
[2010], which could be computationally intensive.
Conclusion
In this work, we presented the OPSRL algorithm which can be viewed as a simple optimistic variant of the PSRL algorithm.
Notably, OPSRL only needs (cid:101) O
(1) posterior samples per state-action.
We proved that the regret of OPSRL is upper-bounded with high probability by (cid:101) O ( √ H 3 SAT ) , matching the problem-independent lower-bound of order Ω( √ H 3 SAT ) for T large enough and up to terms poly-logarithmic in H, S, A, and T .
While our work addresses the open questions raised by Agrawal and Jia [2017b] in the episodic setting, obtaining the same results in the infinite-horizon average reward setting remains an open issue.
We believe that it is possible to adapt our analysis to this other setting up to some technical adjustments.
Ultimately, another open question, is to obtain a high-probability regret bound for PSRL , that is, when using only a single posterior sample and not inflating the posterior.
As a further future research direction we believe it could be interesting to obtain a model-free algorithm that relies on the same mechanism as OPSRL for exploration.
Indeed, such an algorithm could avoid the use of complicated bonuses adopted by the current model-free algorithms while reducing the memory complexity of OPSRL .
Acknowledgments and Disclosure of Funding
D. Belomestny acknowledges the financial support from Deutsche Forschungsgemeinschaft (DFG),
Grant Nr.
The work of D. Tiapkin, D. Belomestny and A. Naumov was prepared within the framework of the HSE University Basic Research Program.
Pierre Ménard acknowledges the support of the Chaire SeqALO (ANR-20-CHIA-0020-01).