Near-Optimal Randomized Exploration for Tabular Markov Decision Processes
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study algorithms using randomized value functions for exploration in reinforcement learning.
This type of algorithms enjoys appealing empirical performance.
We show that when we use 1) a single random seed in each episode, and 2) a
Bernstein-type magnitude of noise, we obtain a worst-case (cid:101) O (cid:16) H √ SAT (cid:17) regretbound for episodic time-inhomogeneous Markov Decision Process where S is thesize of state space, A is the size of action space, H is the planning horizon and
T is the number of interactions.
This bound polynomially improves all existingbounds for algorithms based on randomized value functions, and for the ﬁrst time,matches the Ω (cid:16) H √ SAT (cid:17) lower bound up to logarithmic factors.
Our resulthighlights that randomized exploration can be near-optimal, which was previouslyachieved only by optimistic algorithms.
To achieve the desired result, we develop1)
a new clipping operation to ensure both the probability of being optimistic andthe probability of being pessimistic are lower bounded by a constant, and 2) a newrecursive formula for the absolute value of estimation errors to analyze the regret.
Introduction
This paper concerns learning in tabular Markov Decision Processes (MDP), arguably the mostfundamental model for reinforcement learning (RL).
Existing algorithms that achieve the nearoptimal minimax (cid:101)
O (cid:16) H √ SAT (cid:17) regret bound are based on the principle of Optimism in the faceof Uncertainty (OFU), such as upper conﬁdence bound (UCB) [7, 52, 16, 56, 54].
1 Here S is thenumber of states, A is the number of actions, H is the planning horizon and T is the total number ofinteractions between the agent and the environment.
Another broad category is algorithms with randomized exploration such as Thompson Sampling
These algorithms inject (carefully tuned) random noise to value function to encourageexploration.
UCB-type algorithms enjoy well-established theoretical guarantees but suffer fromdifﬁcult implementation since an upper conﬁdence bound is usually infeasible for many practicalmodels like neural networks.
Instead, practitioners prefer randomized exploration such as noisynetworks in [19], and algorithms with randomized exploration have been widely used in practice[37, 13, 11, 35].
However, how to design randomized exploration algorithms in a principled way andperform randomized exploration optimally is far from clear.
While randomized exploration can havegreat performance in practice, theoretically, the best known worst-case regret bound for algorithmswith randomized exploration is (cid:101) O (cid:16) H 2 S √ AT (cid:17) [2], which is far worse than that of the UCB-typealgorithms.
In this paper, we introduce a new randomized exploration algorithm and show it enjoys a near-optimal (cid:101) O (cid:16) H √ SAT (cid:17) worst-case regret bound, thus closing the gap.
Our work sheds newlight on randomized exploration on both the algorithmic side and the theoretical side.
Our Contributions.
Our contributions are summarized below:
• We propose a new algorithm, Single Seed Randomization ( SSR ), which incorporates a crucialalgorithmic idea: using a single random seed for the entire episode, in contrast to previous methodsof randomized exploration which use one seed for each time step.
SSR is able to explore moreefﬁciently than previous methods by avoiding having noise at different time steps canceling with eachother.
Theoretically, we show, thanks to this new idea, if one uses a Hoeffding-type magnitude ofnoise, SSR achieves an (cid:101) O (cid:
) H 1 . 5 √ SAT (cid:17) regret bound, improving upon the best existing result onrandomized exploration algorithm [2].
• We further design a new Bernstein-type magnitude of noise for our algorithm, and achieve an(cid:101) O (cid:16) H √ SAT (cid:17) regret bound, resolving an open problem raised in [2].
To our knowledge, this is theﬁrst time that a Bernstein-type bound is used in randomized exploration.
More importantly, our upperbound matches the Ω (cid:16) H √ SAT (cid:17) minimax lower bound up to logarithmic factors.
We note that our goal is not to show randomized exploration is better than optimistic algorithms [7] inthe tabular setting.
Instead, we aim to provide a solid theoretical understanding of a practically relevantalgorithm.
Indeed, understanding randomized exploration itself is an important theoretical researchdirection and has attracted much interest in the community [3, 4, 6, 41, 51, 47, 2, 36, 39, 37, 35].
Main Challenge and Technical Overview.
Besides the aforementioned algorithmic ideas (singlerandom seed and Bernstein-type magnitude of noise), we also need additional ideas in analysis toprove the desired regret bound.
The main challenge is that unlike UCB-type algorithms, the estimatedvalue in algorithm with randomized exploration, is not an upper bound of the true optimal value.
Thisleads to the failure of directly utilizing their analysis, which only need to analyze the one-sided errorin estimation.
We instead work on the absolute value of the estimation error, whose analysis is morecomplicated than that for the one-sided error in UCB-type algorithms.
Working with absolute valueforces us to ensure that both the probability that the estimated value is optimistic and the probabilitythat the estimated value is pessimistic are lower bounded.
However, the clipping strategy in existingalgorithm cannot maintain pessimism.
To tackle with this issue, we develop a new clipping method.
Below we list our technical contributions.1.
First, we propose a new clipping strategy to constrain the estimated value function (cf.
Eqn.
Previous clipping strategies in [51, 2] are based on uncertainty and can only maintain optimism.
Ourclipping strategy directly works on the value function, which is similar to those used in UCB-typealgorithms [7, 22, 56].
Our clipping strategy can maintain both the optimism and pessimism.
Inaddition, the number of times that the clipping is used can still be bounded.2.
Second, we prove that the single seed randomization ensures that the estimated value function canboth be optimistic or pessimistic with constant probability at all states and timesteps.
This is strongerthan previous randomized exploration algorithms that are only shown to be optimistic at the initialstate with constant probability.
With this property, we can then bound the difference between theoptimal value function and estimated value function from both above and below, which results in abound on its absolute value.
See Section 5.1, Appendix C and Appendix D.3.
Third, we prove a novel recursion argument on the absolute value of the policy estimation error.
As mentioned in [2], the recursion in UCB-type algorithms can not be directly utilized because ourestimated value function is not a high-probability upper bound of the true optimal value function.
With the bound of absolute value, we are able to prove new recursion formulas and together we cancontrol the policy estimation error.
See Section 5.2 and Appendix E.4.
At last, we bound the sum of variance in a novel manner.
In [7], the UCB-type estimationguarantees that the policy estimation error is always positive so the difference of the variance can bedirectly bounded.
We generalize the argument to the absolute value of the estimation error to boundthe sum of variance.
See Section 5.3.1 and
Appendix G.
Related Work
In this section we review existing provably efﬁcient algorithms for tabular MDP.
There is a long list ofsample complexity guarantees for tabular MDP [27, 9, 25, 43, 44, 28, 8, 21, 45, 30, 36, 14, 7, 15, 38,5, 22, 20, 46, 16, 18, 42, 41, 53, 12, 55, 50, 40, 34, 54, 48, 2, 41, 5, 17, 33, 31].
The state-of-the-artmethods are based on upper conﬁdence bound (UCB) [7, 52, 16, 56, 54, 33, 31].
For the settingconsidered in this paper where the transition is time-inhomogeneous and the reward is bounded by 1 ,one can achieve an (cid:101) O (cid:16) H √ SAT (cid:17) in the regime where T is sufﬁciently large.
Algorithms with randomized exploration have been proved to enjoy favorable regret bounds in banditproblems [29, 3, 26, 10, 4].
In certain settings, randomized exploration can match the worst-caseregret bound of UCB-based approaches and achieve nearly minimax optimal regret bounds [24, 4].
However, for RL, existing theory for randomized exploration are far from optimal [2, 41, 5, 49, 51].
For the setting considered in this paper, the sharpest existing regret bound among algorithms withrandomized exploration is (cid:101) O (cid:16) H 2 S √ AT (cid:17) proved in [2].
Our paper closes this gap and thus deepensour understanding about randomized exploration.
Preliminaries
We consider time-inhomogeneous ﬁnite-horizon MDP M
=
( H, S , A , P, R, s 1 ) , where |S| = S and|A|
= A .
Here, S = { 1 , ..., S } is the ﬁnite state space.
A = { 1 , ..., A } is the ﬁnite action space.
H is the length of an episode.
For convenience, we take s 1 to be the ﬁxed initial state, although amore general initial distribution will not change the conclusion.
P : S ×
A ×
[ H ] → (cid:52) ( S ) is thetransition function, where if the agent stays at state s and takes action a at time h , it transits to states (cid:48) with probability P h,s,a ( s (cid:48) )
∈ [0 , 1] .
R : S ×
A ×
[ H ] → [0 , 1] is the reward function, whereif the agent stays at s and takes action a at time h , it will receive reward r h,s,a ∈ [0 , 1] such that 103 E [ r h,s,a ] =
R h,s,a .
A deterministic policy for such a MDP is deﬁned as a tuple π
=
( π 1 , . . .
, π H ) , where π h : S (cid:55)→ A .
The associated value function at state s ∈ S and level h ∈ { 1 , . . .
, H } is recursively deﬁned as
[Equation]
For convenience, we set V πH +1 = 0 ∈ R S .
The corresponding optimal value function is V ∗ h ( s ) =max π ∈ Π V πh ( s ) , where Π is the set of all possible deterministic policies.
For a particular algorithm
Alg , let π k denote the policy that Alg employs during episode k .
Then, the regret of running Alg on
MDP M for K episodes is deﬁned as
[Equation]
Note that the regret, Reg ( M, K, Alg ) , is a random variable due to randomness in state transition andthe algorithm, Alg .
In this paper, we show the regret of our proposed algorithm can be upper boundedwith high probability, and the upper bound matches the known lower bound up to logarithmic factors.
To facilitate our later analysis, we introduce some notations for empirical estimation.
At episode k ,we collect a trajectory ( s k 1 , a k 1 , r k 1 , · · · , s kH , a kH , r kH ) as speciﬁed in Algorithm 1.
Let n k ( h, s, a ) = 113 (cid:80)
k −l =1 1 { ( s lh , a lh ) =
( s, a ) } be the number of times action a is taken at state s and time h beforeepisode k , where 1 {·} is the indicator function.
We deﬁne
[Equation]
Then, deﬁne empirical MDP based on our observation and estimation before episode k as the tupleˆ M k =
( H, S , A , ˆ P k , ˆ R k , s 1 ) .
Since ˆ P kh,s,a is not a valid distribution over S , for being rigorous, we
Input: H, S, A , perturbation type ty ∈
Ho , Be } 1 for episodes k = 1 , 2 , . .
K do
Sample ˆ z k ∼ N (0 , 1)
Deﬁne terminal value function Q H +1 ,k = 0 ∈ R SA and V H
+1 ,k = 0 ∈ R S for time periods h = H, . .
. , 1 do
Q h,k ( s, a ) ← ˆ R kh,s,a + (cid:68) ˆ P kh,s,a , V h
+1 ,k (cid:69) + σ k ty ( h, s, a ) ˆ z k ; // σ k ty h, s, a is defined
Deﬁne V = clip [-
-]max a end
Agent takes this episode
Observe
10 end can imagine there is an additional virtual absorbing state that every state will transit to with remainingprobability.
In addition to the above notations, let (cid:101) O ( · ) , (cid:101) Θ ( · ) and (cid:101) Ω ( · ) be asymptotic notations ignoring all polylogarithmic terms.
For distribution D ∈ ∆ S and value function V ∈ R S , let V ( D, V ) denote the variance of V under distribution D , which is deﬁned as V ( D, V )
= (cid:80) s ∈S D ( s ) ( V ( s ) − (cid:104) D, V (cid:105) )
Forconstant a > 0 , we deﬁne the corresponding clipping function as clip a ( · ) = max {− a, min { a, ·}} .
Immediately we have | clip a ( x ) | ≤ a for any a > 0 .
We introduce the deﬁnitions of other notationswhen used.
In appendix, we summarize the notations and deﬁnitions used in this paper.
Main Results
4.1 Algorithm
The main contribution of this paper is that we show algorithm with randomized value functionscan achieve regret that matches the known lower bound Ω (cid:16) H √ SAT (cid:17) [21, 17] up to logarithmicfactors in the tabular setting.
To facilitate exploration, this type of algorithms uses random valueperturbation instead of deterministic bonus.
The algorithm we consider is summarized in Algorithm1.
In our algorithm, SSR , the random perturbation ensures that optimism/pessimism can be obtainedwith constant probability in each episode.
Moreover, randomized value function has its origin fromposterior sampling for reinforcement learning (Thompson sampling).
The randomized perturbationcan be interpreted as approximate sampling from the posterior distribution of the value function onrandomized training data [41].
We ﬁrst give an overview of SSR .
In Algorithm 1, the policy used at episode k is computed using theempirical MDP, ˆ M k =
( H, S , A , ˆ P k , ˆ R k , s 1 ) , which is based on observation and estimation beforeepisode k .
However, instead of directly choosing optimal policy for ˆ M k , we add a small randomperturbation when computing the value of each state and action pair.
To be more precise, at eachepisode k , we ﬁrst estimate the reward and transition function for each state s and action a based on(2) and (3).
Then, we compute the value function for state s and action a ,
[Equation]
Here, ˆ z k ∼ N (0 , 1) is a standard Gaussian random variable sampled once every episode.
Themagnitude of the perturbation, σ k ty depends on how many samples n k ( h, s, a ) we have observed andhow conﬁdent we are on the estimations ˆ R kh,s,a and ˆ P kh,s,a .
We will discuss more about the choice ofthe magnitude later in this section.
In order to prevent estimated value function from behaving badly, we add a clipping to the valuefunction:
[Equation]
As our analysis will show, this kind of clipping can bound the value function, maintain optimismand pessimism and also guarantee that clipping will not happen for a lot of times.
The constant 2(instead of 1 ) plays a crucial role because it means the value function grows at an additive rate of2 from h
= H to h = 1 .
If we do not consider the added noise, then the value function should atmost grow 1 at each timestep because the reward is at most 1 .
For our clipping technique, if a clipis triggered, there exists a timestep such that the added noise is more than 1 , which is equivalentto a small number of visits (cf.
Deﬁnition 23 and Lemma 8).
As our later analysis will show, theclipping only affects the lower-order term and will not compromise the long-term performance of thealgorithm.
Finally, after computing the value function and clipping, SSR chooses the action a kh thatmaximizes Q h,k ( s kh , a ) at each time step, h = 1 , ..., H , throughout the episode.
Note that from a Bayesian perspective, when there is no clipping, in Algorithm 1, Q h,k followsdistribution
[Equation]
This resembles posterior sampling because when estimating some parameter θ ∗ ∼ N (cid:0) 0 , β 2 (cid:1) basedon noisy observations θ 1 , ..., θ n ∼ N ( θ, β 2 ) , the posterior distribution of θ ∗ given { θ i } ni =1 isθ ∗ | { θ i } ni =1 ∼ N (cid:16) 1 n +1 (cid:80) ni
=
θ i , β 2 n +1 (cid:17) .
Although exact posterior sampling may not be possible incomplex reinforcement learning settings, in SSR , σ k ty ( h, s, a ) is chosen at scale (cid:101) Θ (cid:16) 1 / (cid:112) n k ( h, s, a ) (cid:17)and therefore can be interpreted as doing approximate posterior sampling.
Moreover, SSR can beviewed as a variant of Randomized Least Square Value Iteration (RLSVI).
The major differences areat the clipping function and a single random seed used in each episode instead of different randomseeds at different tuples ( h, s, a ) .
We will discuss more about the choice of the random seed laterin this section.
We refer to [37] and [41] for a more detailed discussion on the relationship among
RLSVI, posterior sampling and randomized value function.
In the following paragraphs, we discuss in more details about the three major algorithmic innovations:
Single Random Seed in Each Episode.
SSR is similar to the algorithms analyzed in [41] and [2].
The major difference is that in the algorithm we propose, we use a single random seed ˆ z k to generatethe perturbations for all time steps h
= 1 , ..., H in an episode k .
When using different random seeds in an episode, the algorithm can be optimistic in some time stepwhile being pessimistic in others.
Then, the effects of the perturbations at different time steps willcancel with each other.
As a result, to ensure sufﬁcient exploration, the magnitude of the perturbationhas to large.
This issue was also pointed out in [2, 1].
A large perturbation magnitude can increase the instability of the algorithm and worsen the algorithm’sperformance.
When a single random seed is used, a small perturbation magnitude is enough toguarantee that the algorithm is optimistic with constant probability in any episode.
We are able toshow that using a single random seed can signiﬁcantly increase the stability of the algorithm andtherefore enjoy much smaller regret.
Coincidentally, [47] also uses a similar single randomizationin bandit problems to build a near-optimal randomized exploration algorithm and our work can betreated as its natural extension to RL problems.
Clipping.
To obtain a tight regret bound, the estimated value function needs to be well bounded.
In [41], no clipping is used and the estimated value function is at the order of (cid:101)
O
( H 5 / 2 S ) , whichresults in a suboptimal regret bound.
Generally there are two types of clipping methods.
The ﬁrst oneis uncertainty-based, i.e. the value is clipped to H − h + 1 at timestep h
whenever the uncertaintyis large [51, 2].
However this type of clipping cannot maintain pessimism which is critical in ouranalysis.
The other kind of clipping is value-based, mostly in UCB-type algorithms [23].
Thesealgorithms truncate estimated value greater than a certain threshold, i.e. H − h
+ 1 at time step h .
The problem here is that the number of clippings cannot be bounded because if the true value functionis close to H − h + 1 at timestep h , the clipping will happen with some constant probability.
Our clipping method leverages both type of clipping methods in the existing literature.
Thoughour clipping is based on the value function, we show that whenever the clip is triggered, the estimation error must be large, which implies that the uncertainty at that state is large.
This clippingmethod inherits the desired properties from both uncertainty-based and value-based clipping, i.e. theoptimism/pessimism is maintained and the number of clippings can be bounded.
Magnitude of Perturbation.
A large magnitude of perturbation can encourage exploration, but atthe same time increase instability.
In our algorithm, the magnitudes are chosen as the smallest valuesso that the algorithm can be optimistic with constant probability.
Since the value function can roughlybe bounded by O ( H ) , a naive choice of the perturbation magnitude can be Θ (cid:16) H/ (cid:112) n k ( h, s, a ) (cid:17) .
Inthis way, by Hoeffding’s inequality, as long as the random Gaussian variable sampled
ˆ z k is bigger thana constant, which happens with constant probability, the estimated value function will be optimistic.
By similar reasoning, we can see that the estimated value function will also be pessimistic withconstant probability.
To make the magnitude even smaller, inspired by [7] who showed one can use an (empirical)
Bernstein’s inequality to derive a sharp exploration bonus for UCB-based algorithms, we proposea new choice of perturbation magnitude based on Bernstein’s inequality.
The Bernstein-basedperturbation uses the empirical variance of the value function, which makes it smaller than the
Hoefﬁding-based one mostly, but still maintains optimism with constant probability.
In our paper, we study both types of magnitudes.
In particular, we show that the regret of SSR basedon Bernstein’s inequality matches the known lower bound Ω (cid:16) H √ SAT (cid:17) .
Following are the twochoices:
[Equation]
[Equation]
where subscript “Ho” represents that the perturbation is based on Hoeffding’s inequality and “Be”represents Bernstein’s inequality, correspondingly.
Here, for proof convenience, ˜ P kh,s,a is deﬁned byreplacing the denominator in ˆ P kh,s,a by max { n k ( h, s, a ) , 1 } . To clarify, when subscript “ty” is used,which stands for “type” as a placeholder for “Ho” or “Be”, it means that there is no need to write twocopies of expressions for Hoeffding-based and Bernstein-based noises separately.
Practical Considerations.
Here, we explain why randomized exploration is widely used in practiceand why our algorithmic formulation practically has advantage over UCB-type algorithms.
In randomized exploration, there are usually two important components: (1) the algorithm (e.g., Algorithm1) and (2) the noise magnitude ( σ ty ).
In practice, the main advantage of randomized exploration liesin the algorithm component.
The generalization from the tabular setting to the function approximationsetting is straightforward: one can just add a random regularization term in the value estimation step,whose details can be found in [35].
On the other hand, the generalization of optimistic algorithmsfrom the tabular setting to the function approximation setting is more non-trivial because it oftenrequires an explicit construction of the conﬁdence set.
For the second component, although generalizing our strategy of tuning noise magnitude to the real-world function approximation setting is indeednot straightforward, it is often set as a hyper-parameter in practice.
4.2 Regret Analysis
We analyze the regret, deﬁned in (1), of our algorithm SSR using both types of perturbations.
Ourmain theorems are presented in Theorem 1 and 2.
In particular, Theorem 2 shows SSR with Bernsteinbased perturbation can achieve the regret that matches the known lower bound Ω (cid:16) H √ SAT (cid:17) up tologarithmic factors.
We sketch the proof of Theorem 1 and Theorem 2 in Section 5.
Theorem 1.
If the Hoeffding-type noise (5) is used, then for any MDP M =
( H, S , A , P, R, s 1 ) , with probability at least 1 − δ , Algorithm 1 satisﬁes
[Equation]
In particular, when T ≥ (cid:101) Ω (cid:0) H 5 S 3 A (cid:1) , it holds that Reg( M, K, SSR Ho ) ≤ (cid:101) O (cid:16) H 1 . 5 √ SAT (cid:17) .
Theorem 2.
If the Bernstein-type noise (6) is used, then when T ≥ (cid:101) Ω (cid:0) H 5 S 2 A (cid:1) , for any MDP
M =
( H, S , A , P, R, s 1 ) , with probability at least 1 − δ , Algorithm 1 satisﬁes
[Equation]
In particular, if we further have T ≥ (cid:101) Ω (cid:0) H 6 S 3 A (cid:1) , it then holds that Reg( M, K, SSR Be ) ≤(cid:101) O (cid:16) H √ SAT (cid:17) .
We give a brief comparison between SSR and other related works.
[41] shows that RLSVI, analgorithm similar to SSR , can achieve ˜ O (cid:16) H 2 .
5 S 1 . 5 √ AT (cid:17) regret in expectation over the randomnessof MDP and the algorithm.
In [2], an improved high probability regret bound (cid:101) O (cid:16) H 2 S √ AT (cid:17) isproposed, which is the sharpest bound for randomized algorithms prior to this work.
Our paper closesthe gap between those previous bounds and the lower bound in tabular setting.
Proof Outline
In this section, we present an proof outline of Theorem 1 and 2.
Since their proofs follow the sameframework, we will present an uniﬁed outline and explain the individual steps particularly for eachcase when necessary.
The details of complete proof are deferred to the appendix.
Notation For the ease of exposure, we will use a simpliﬁed notations during this sketch.
Speciﬁcally,let x =
( h, s, a ) and x kh =
( h, s kh , a kh ) .
5.1 Concentration and Optimism/Pessimism
We start by introducing a set of MDPs M k ty as a conﬁdence set such that the empirical MDPˆ M k belongs to it with high probability, meaning that we have a good estimation of the true MDP.
Speciﬁcally, with M (cid:48)
=
( H, S , A , P (cid:48) , R (cid:48) , s 1 ) , we deﬁne
[Equation]
where (cid:113) e k Ho ( x ) =
σ k Ho ( x ) and (cid:113)
e k Be ( x ) ≈ σ k
Be ( x ) .
Deﬁne the event C k
ty :=
(cid:110)
ˆ M k ∈ M k ty (cid:111) .
Then, by applying Hoeffding’s inequality or Bernstein’s inequality, for both types of perturbation, it is possible to show that
[Equation]
Since the value function is bounded in [0 , H ] , this inequality tells us that the regret incurred by badestimation is at most (cid:101)
O ( H ) .
To be precise, it holds with high probability that
[Equation]
Then, to better control the estimated value function, we need it to be bounded, which requiresus to clip it.
Speciﬁcally, we will use two crucial properties of our clipping method.
First, if
Q h,k ( s, a ) ≥ Q ∗ h ( s, a ) , ∀ ( s, a ) ∈ S × A , then we have V h,k ( s ) ≥ V ∗ h ( s ) , ∀ s ∈ S .
Similarly if
Q h,k ( s, a ) ≤ Q ∗ h ( s, a ) , ∀ ( s, a ) ∈ S × A , then we have V h,k ( s ) ≤ V ∗ h ( s ) , ∀ s ∈ S .
In addition, we can prove that whenever a clip is triggered for s kh , we have n k ( h, s kh , a kh ) ≤ α k withα k
= (cid:101)
O ( H 2 ) .
As a result, it is possible to show that the total regret incurred by clipping is at most
(cid:101) O (cid:0) H 4 SA (cid:1) , which is a lower-order term when T is sufﬁciently large.
That is, let E cumH,k denote theevent that there is no clipping during episode k .
Then, it holds with high probability that
[Equation]
As claimed before, because of the randomness in Gaussian noise, our algorithm SSR will encourageexploration and it takes effect when there is no clipping and the estimation is not too bad.
In otherwords, it can be optimistic.
However, also because of this randomness, its optimism only holds in aprobabilistic sense.
In precise, it is possible to show that
[Equation]
where the value of constant C ty depends on the type of noise we choose.
Meanwhile, we can alsoprove a very similar probabilistic pessimism, which means to have V h,k ( s ) ≤ V ∗ h
( s ) , ∀ h ∈
[ H ] , s ∈
S with constant probability.
The property of optimism and pessimism will help us upper bound theabsolute value of V ∗ 1 ( s k 1 ) − V 1 ,k
( s k 1 ) , which will be discussed soon.
5.2 Regret Decomposition
Now, given equations (7) and (8), we can see that for each episode k , it only remains to bound
[Equation] [Equation]
( s
[Equation]
Technically, the further deﬁned the good event G k will helpmake V h,k better-behaved.
Its precise deﬁnition will be given in the appendix.
Therefore, it issufﬁcient to bound 1 {G k } (cid:16) V ∗ 1 − V π k 1 ,k (cid:17) ( s k 1 ) , which means to have
Reg
[Equation]
M, K,
[Equation]
ty
[Equation]
To proceed, we need to deﬁne two auxiliary value functions V h,k and V h,k , which are obtained byvirtually running policy π k on some deliberately perturbed MDPs.
In particular, they are designedsuch that V h,k ≤ V h,k ≤ V h,k holds under the good event G k .
Pessimism Term Here, as a technical novelty, we bound the pessimism term’s absolute value.
Meanwhile, different from [51] and [2], by applying both optimism and pessimism, we do not resortto an independent copy of the perturbed MDP to bound the pessimism term and give a conceptuallysimpler analysis.
In particulary, by deﬁning C 1 = 1 / min { C Ho , C Be } .
it is possible to show that
[Equation]
The full proof is given in Appendix under Lemma 15.
Estimation Error Term
The sum of pessimism term and estimation error term can be furtherbounded via the techniques of recursion used in [7].
However, we want to emphasize the difference that in their algorithm, the estimated value is optimistic with high probability, which makes
V h,k ( s kh ) − V ∗ h ( s kh ) always positive.
Instead, since our optimism only holds with constant probability,we use absolute value to keep the estimation error terms positive.
As a result, we show that
[Equation]
where L denotes some poly-logarithmic term and M h,k denotes some martingale difference sequenceterm at period h , episode k .
The full proof is given in Appendix under Lemma 19,
5.3 Combining Different Terms
By combining equations (10), (11) and (12) and applying concentration inequalities to MDP M h,k ,it is possible to show that
[Equation]
Then, a ﬁnal high-probability regret bound can be obtained by summing each individual terms overk, h separately.
It is well-known among literature that
[Equation]
Recall the deﬁnition of σ k Ho in equation (5).
By using these two inequalities, the bound in equation (13) can be made explicit if we use Hoeffding-type noise.
As a result, we have
[Equation]
5.3.1 Bound on Sum of Variance
Analyses become more involved when Bernstein-type noise is used.
Speciﬁcally, notice that inequalities in (14) cannot directly be used to bound (cid:80) k,h V (cid:16)
˜ P kx kh ,
V h
+1 ,k (cid:17) .
Here, we apply sometechniques developed in [7].
However, since the optimism only holds with constant probability, thedetails for speciﬁc terms are quite different.
For the ease of exposure, we will ignore all constants and deﬁne ˆ V ∗ h,k = V (cid:16)
˜ P kx kh , V ∗ h (cid:17) , ˆ V h,k =
V (cid:16) ˜ P kx kh , V h,k (cid:17) .
Then, by using Cauchy-Schwartz inequality and equation (14), we can get
U def = (cid:88) k,h 1 {G k
[Equation] [Equation]
Here, note that U ≈ (cid:80) k,h σ k Be ( x kh ) .
Then, after some steps of algebra, it is possible to show that
K (cid:88) k
=
H (cid:88) h
[Equation]
(When T ≥ (cid:101) Ω ( H 5 S 2 A ) )
=
⇒ U ≤ (cid:101) O
(cid:16)(cid:112) HSA ( HT + H 2 U ) (cid:17) ≤ (cid:101) O (cid:
) H √ SAT + H 1 . 5 √ U (cid:17) .
(By using equation (15))
Now, we can see that (cid:80) k,h σ k
Be ( x kh ) ≈ U ≤ (cid:101) O (cid:16) H √ SAT (cid:17) satisﬁes this inequality.
Finally, by plugging this result back into equation (13), we can have
[Equation]
2 A (cid:17) , which matches the known lower bound when T ≥ (cid:101) Ω ( H 6 S 3 A ) .
Conclusion
We gave a new algorithm with randomized exploration, SSR , for tabular MDP, which enjoys anear-optimal (cid:101) O (cid:16) H √ SAT (cid:17) regret bound in the time-homogeneous model.
Previously, near-optimalregret bounds can only be achieved by optimistic algorithms.
Our result also highlights the importanceof using a single random seed for the entire episode and using the variance information in tuning themagnitude of noise (cf.
Bernstein’s inequality).
One important open problem is whether randomized exploration can a achieve a horizon-free regretbound in the time-homogeneous model where the transition is the same at different levels [52, 48, 54].
Another possible future direction is to consider whether the sub-optimal lower order terms (cid:101)
O ( H 4 S 2 A )can be further improved to relax the current requirement T ≥ (cid:101) Ω ( H 6 S 3 A ) for being near-optimal.