Finite-Time Regret of Thompson Sampling Algorithms for Exponential Family Multi-Armed Bandits
Abstract
We study the regret of Thompson sampling (TS) algorithms for exponential familybandits, where the reward distribution is from a one-dimensional exponential family, which covers many common reward distributions including Bernoulli, Gaussian,
Gamma, Exponential, etc.
We propose a Thompson sampling algorithm, termed
ExpTS, which uses a novel sampling distribution to avoid the under-estimationof the optimal arm.
We provide a tight regret analysis for ExpTS, which simultaneously yields both the ﬁnite-time regret bound as well as the asymptotic regretbound.
In particular, for a K -armed bandit with exponential family rewards, ExpTS over a horizon T is sub-UCB (a strong criterion for the ﬁnite-time regret thatis problem-dependent), minimax optimal up to a factor √ log K , and asymptotically optimal, for exponential family rewards.
Moreover, we propose ExpTS + , byadding a greedy exploitation step in addition to the sampling distribution used in
ExpTS, to avoid the over-estimation of sub-optimal arms.
ExpTS + is an anytimebandit algorithm and achieves the minimax optimality and asymptotic optimalitysimultaneously for exponential family reward distributions.
Our proof techniquesare general and conceptually simple and can be easily applied to analyze standard
Thompson sampling with speciﬁc reward distributions.
Introduction
The Multi-Armed Bandit (MAB) problem is centered around a fundamental model for balancingthe exploration versus exploitation trade-off in many online decision problems.
In this problem,the agent is given an environment with a set of K arms [ K ] = { 1 , 2 , · · · , K } .
At each time stept , the agent pulls an arm
A t ∈ [ K ] based on observations of previous t − 1 time steps, and then areward r t is revealed at the end of the step.
In real-world applications, reward distributions oftenhave different forms such as Bernoulli, Gaussian, etc.
As suggested by Auer et al.
[8, 9], Agrawaland Goyal [5], Lattimore [29], Garivier et al.
[16], a good bandit strategy should be general enoughto cover a sufﬁciently rich family of reward distributions.
In this paper, we assume the reward r t isindependently generated from some canonical one-parameter exponential family of distributions witha mean value µ A t .
It is a rich family that covers many common distributions including Bernoulli,
Gaussian, Gamma, Exponential, and others.
The goal of a bandit strategy is usually to maximize the cumulative reward over T time steps, whichis equivalent to minimizing the regret, deﬁned as the expected cumulative difference between playingthe best arm and playing the arm according to the strategy: R µ ( T ) =
T · max i ∈
[ K ] µ i − E
[ (cid:80) Tt =1 r t ] .
We assume, without loss of generality, µ 1 = max i ∈
[ K ] µ i is the best arm throughout this paper.
For aﬁxed bandit instance (i.e., mean rewards µ 1 , · · · , µ K are ﬁxed), Lai and Robbins [26] shows that for distributions that are continuously parameterized by their means,
[Equation]
where kl ( µ i , µ 1 ) is the Kullback-Leibler divergence between two distributions with mean µ i and µ 1 .
A bandit strategy satisfying lim T →∞ R µ ( T ) / log T = (cid:80) i>
[Equation]
µ 1 − µ i kl ( µ i ,µ 1 ) is said to be asymptoticallyoptimal or achieve the asymptotic optimality in regret.
The asymptotic optimality is one of the mostimportant statistical properties in regret minimization, which shows that an algorithm is consistentlygood when it is played for inﬁnite steps and thus should be a basic theoretical requirement of anygood bandit strategy [8].
In practice, we can only run the bandit algorithm for a ﬁnite number T steps, which is the timehorizon of interest in real-world applications.
Therefore, the ﬁnite-time regret is the ultimate propertyof a practical bandit strategy in regret minimization problems.
A strong notion of ﬁnite-time regretbounds is called the sub-UCB criteria [29], where the regret satisﬁes
[Equation]
where ∆ i = µ 1 − µ i is the sub-optimal gap between arm 1 and arm i .
Note that the regret bound in(1.2) is a problem-dependent bound since it depends on the bandit instance and the sub-optimal gaps.
Sub-UCB is an important metric for ﬁnite-time regret bound and has been adopted by recent work of
Lattimore [29], Bian and Jun [10].
Another special type of ﬁnite-time bounds is called the worst-caseregret, which is deﬁned as the ﬁnite-time regret of an algorithm on any possible bandit instancewithin a bandit class.
Speciﬁcally, for a ﬁnite time horizon T , Auer et al.
[8] proves that any strategyhas at least worst-case regret Ω( √ KT ) for a K -armed bandit.
We say the strategy that achieves aworst-case regret O ( √ KT ) is minimax optimal or achieves the minimax optimality.
Different fromthe asymptotic optimality, the minimax optimality characterizes the worst-case performance of thebandit strategy in ﬁnite steps.
A vast body of literature in multi-armed bandits [6, 5, 22, 32, 16, 29] have been pursing the aforementioned theoretical properties of bandit algorithms: generality, asymptotic optimality, problemdependent ﬁnite-time regret, and minimax optimality.
However, most of them focus on one or twoproperties and sacriﬁce the others.
Moreover, many of existing theoretical analyses of bandit strategiesare for optimism-based algorithm.
The theoretical analysis of Thompson sampling (TS) is much lessunderstood until recently, which has been shown to exhibit superior practical performances comparedto the state-of-the-art methods [13, 34].
Speciﬁcally, its ﬁnite-time regret, asymptotic optimality, andnear minimax optimality have been studied by Agrawal and Goyal [3, 4, 5] for Bernoulli rewards.
Jinet al.
[20] proved the minimax optimality of TS for sub-Gaussian rewards.
For exponential familyreward distributions, the asymptotic optimality is shown by Korda et al.
[25], but no ﬁnite-time regretof TS is provided.
See Table 1 for a comprehensive comparison of these results.
In this paper, we study the regret of Thompson sampling for exponential family reward distributionsand address all the theoretical properties of TS.
We propose a variant of TS algorithm with a generalsampling distribution and a tight analysis for frequentist regret bounds.
Our analysis simultaneouslyyields both the ﬁnite-time regret bound and the asymptotic regret bound.
Speciﬁcally, the main contributions of this paper are summarized as follows:
• We propose ExpTS, a general variant of Thompson Sampling, that uses a novel sampling distribution with a tight anti-concentration bound to avoid the under-estimation of the optimal arm and atight concentration bound to avoid the over-estimation of sub-optimal arms.
For exponential familyof reward distributions, we prove that ExpTS is the ﬁrst Thompson sampling algorithm achievingthe sub-UCB criteria, which is a strong notion of problem-dependent ﬁnite-time regret bounds.
We
further show that ExpTS is also simultaneously minimax optimal up to a factor of √ log K , as wellas asymptotically optimal, where K is the number of arms.•
We also propose ExpTS + , which explores between the sample generated in ExpTS and the empirical mean reward for each arm, to get rid of the extra √ log K factor in the worst-case regret.
Thus ExpTS + is the ﬁrst Thompson sampling algorithm that is simultaneously minimax andasymptotically optimal for exponential family of reward
distributions.•
Our regret analysis of ExpTS can be easily extended to analyze Thompson sampling with commonreward distributions in the exponential family.
For Gaussian reward with known variance, we prove
[Table] [Table]
[Table] [Table]
that TS with Gaussian posteriors (Gaussian-TS) is asymptotically optimal and minimax optimal upto a factor of √ log K .
For Bernoulli reward, we prove that TS with Beta posteriors (Bernoulli-TS)is asymptotically optimal and minimax optimal up to a factor of √ log K .
Similar to the idea of
ExpTS + , we can add a greedy exploration step to the posterior distributions used in these variantsof TS, and then the algorithms are simultaneously minimax and asymptotically optimal.
Our techniques are novel and conceptually simple.
First, we introduce a lower conﬁdence boundin the regret decomposition to avoid the under-estimation of the optimal arm, which is importantin obtaining the ﬁnite-time regret bound.
Speciﬁcally, Jin et al.
[20] (Lemma 5 in their paper)shows that for Gaussian reward distributions, Gaussian-TS has a regret bound at least in the orderof Ω( √ KT log T ) if the standard regret decomposition in existing analysis of Thompson sampling[5, 30, 20] is adopted.
With our new regret decomposition that is conditioned on the lower conﬁdencebound introduced in this paper, we improve the worst-case regret of Gaussian-TS for Gaussian rewarddistributions to O ( √ KT log K ) .
Second, we do not require the closed form of the reward distribution, but only make use of thecorresponding concentration bounds.
This means our results can be readily extended to other rewarddistributions.
For example, we can extend ExpTS + to sub-Gaussian reward distributions and thealgorithm is simultaneously minimax and asymptotically optimal 1 , which improve the results of
MOTS proposed by Jin et al.
[20] (see Table 1).
Third, the idea of ExpTS + is simple and can be used to remove the extra √ log K factor in theworst-case regret.
We note that MOTS [20] can also achieve the minimax optimal via the clipped
Gaussian.
However, it is not clear how to generalize the clipping idea to the exponential familyof reward distribution.
Moreover, it uses the MOSS [6] index for clipping, which needs to knowthe horizon T in advance and thus cannot be extended to the anytime setting, while ExpTS + is ananytime bandit algorithm which does not need to know the horizon length in advance.
Notations.
We let T be the total number of time steps, K be the total number of arms, and[ K ] = { 1 , 2 , · · · , K } .
For simplicity, we assume arm 1 is the optimal throughout this paper,
i.e.,µ 1 = max i ∈
[ K ] µ i .
We denote log + ( x ) = max { 0 , log x } and ∆ i := µ 1 − µ i , i ∈
[ K ] \ { 1 } for thegap between arm 1 and arm i .
We let T i ( t ) :
= (cid:80) tj =1 1 { A t = i } be the number of pulls of arm i atthe time step t , (cid:98) µ i ( t ) :
= 1 /T
i ( t ) (cid:80) tj =1 (cid:2) r j · 1 { A t = i } (cid:3) be the average reward of arm i at the timestep t , and (cid:98) µ is be the average reward of arm i after its s -th pull.
Related Work
There are series of works pursuing the asymptotic regret bound and worst-case regret bound for MAB.
For asymptotic optimality, UCB algorithms [15, 31, 5, 29], Thompson sampling [23, 25, 5, 20],
Bayes-UCB [22], and other methods [21, 10] are all shown to be asymptotically optimal.
Amongthem, only a few [15, 12, 25] can be extended to exponential families of distributions.
One notableresult in Cappé et al.
[12] shows that for [0 , 1] bounded distribution, there exists an algorithm that hasregret (cid:80) i> 1 ∆ i log T/ kl ( µ i , µ 1 ) +
O ( (cid:80) i> 1 (log T ) 4 / 5 log log T · ∆ i ) 2 , which is better than (1.2).
It is an interesting problem whether we can achieve such regret for unbounded reward distributions.
For the worst-case regret, MOSS [6] is the ﬁrst algorithm proved to be minimax optimal.
Later,
KL-UCB ++ [5], AdaUCB [29], MOTS [20] also join the family.
The anytime version of MOSSis studied by Degenne and Perchet [14].
There are also some works that focus on the near optimalproblem-dependent regret bound [27, 28].
As far as we know, no algorithm has been proved toachieve the sub-UCB criteria, asymptotic optimality, and minimax optimality simultaneously forexponential family reward distributions.
For Thompson sampling, Russo and Van Roy [33] studied the Bayesian regret.
They show thatthe Bayesian regret of Thompson sampling is never worse than the regret of UCB.
Bubeck and
Liu [11] further showed the Bayesian regret of Thompson sampling is optimal using the regretanalysis of MOSS.
There are also a line of works focused on the frequentist regret of TS.
Agrawaland Goyal [3] proposed the ﬁrst ﬁnite time regret analysis for TS.
Kaufmann et al.
[23], Agrawaland Goyal [4] proved that TS with Beta posteriors is asymptotically optimal for Bernoulli rewarddistributions.
Korda et al.
[25] extended the asymptotic optimality to the exponential family of rewarddistributions.
Subsequently, for Bernoulli rewards, Agrawal and Goyal [5] proved that TS with Betaprior is asymptotically optimal and has worst-case regret O ( √ KT log T ) .
Besides, they showed that
TS with Gaussian posteriors can achieve a better worst-case regret bound O ( √ KT log K ) .
Theyalso proved that for Bernoulli rewards, TS with Gaussian posteriors has a worst-case regret at leastΩ( √ KT log K ) .
Very recently, Jin et al.
[20] proposed the MOTS algorithm that can achieve theminimax optimal regret O ( √ KT ) for multi-armed bandits with sub-Gaussian rewards but at thecost of losing the asymptotic optimality by a multiplicative factor of 1 /ρ , where 0 < ρ < 1 is anarbitrarily ﬁxed constant.
For bandits with Gaussian rewards, Jin et al.
[20] proved that MOTScombined with a Rayleigh distribution can achieve the minimax optimality and the asymptoticoptimality simultaneously.
We refer readers to Tables 1 and 2 for more details.
Preliminary on Exponential Family Distributions
A one-dimensional canonical exponential family [15, 17, 32] is a parametric set of probabilitydistributions with respect to some reference measure, with the density function given by
[Equation]
where θ is the model parameter, and c is a real function.
Denote the measure of p θ ( x ) as ν θ .
Then,the above deﬁnition can be rewritten as
[Equation]
for some measure ρ and b ( θ ) = log( (cid:82)
e xθ d ρ ( x )) .
We make the classic assumption used by Garivierand Cappé [15], Ménard and Garivier [32] that b ( θ ) is twice differentiable with a continuous secondderivative.
Then, we can verify that exponential families have the following properties:
[Equation]
Let µ = E [ ν θ ] .
The above equality means that the mapping between the mean value µ of ν ( θ ) andthe parameter θ is one-to-one.
Hence, exponential family of distributions can also be parameterizedby the mean value µ = b (cid:48)
( θ ) .
Note that b (cid:48)(cid:48)
( θ ) > 0 for all θ , which implies b (cid:48) ( · ) is invertible and itsinverse function b (cid:48)− 1 satisﬁes θ =
b (cid:48)− 1 ( µ ) .
In this paper, we will use the notion of Kullback-Leibler(KL) divergence.
The KL divergence between two exponential family distributions with parameter θand θ (cid:48) respectively is deﬁned as follows:
[Equation]
Recall that the mapping θ (cid:55)→ µ is one-to-one.
We can deﬁne an equivalent notion of the KL divergencebetween random variables ν θ and ν θ (cid:48) as a function of the mean values µ and µ (cid:48) respectively:
[Equation]
[Equation] [Equation]
(cid:48)
[Equation]
=
[Equation]
( ν
[Equation]
, ν
[Equation]
(cid:48) ) , where E [ ν θ ] = µ and E [ ν θ (cid:48) ]
= µ (cid:48) .
Similarly, we deﬁne V ( µ ) =
Var( ν b (cid:48)− 1 ( µ ) ) as the variance of anexponential family random variable ν θ with mean µ .
We assume the variances of exponential familydistributions used in this paper are bounded by a constant V > 0 :0 <
V ( µ ) ≤ V < + ∞ .
We have the following property of the KL divergence between exponential family distributions.
Proposition 3.1 (Harremoës [17]) .
Let µ and µ (cid:48) be the mean values of two exponential family distributions.
The Kullback-Leibler divergence between them can be calculated as follows:
[Equation]
It is worth noting that exponential families cover many of the most common distributions used inpractice such as Bernoulli, exponential, Gamma, and Gaussian distributions.
In particular, for two
Gaussian distributions with the same known variance σ 2 and different means µ and µ (cid:48) , we can choose
V ( · ) =
σ 2 , and recover the results in Proposition 3.1 as kl ( µ, µ (cid:48) )
= ( µ − µ (cid:48) ) 2 / (2 σ 2 ) .
For two
Bernoulli distributions with mean values µ and µ (cid:48) respectively, the variance upper bound is set as
V = 1 / 4 .
Thus we can recover the result in Proposition 3.1 as kl ( µ, µ (cid:48) )
= µ log( µ/µ (cid:48) )
+ (1 −µ ) log((
− µ ) / (1 − µ (cid:48) )) .
Based on Proposition 3.1, we can also verify the following properties.
Proposition 3.2 (Jin et al.
For all µ and µ (cid:48) , we have
[Equation]
In addition, for (cid:15) > 0 and µ ≤ µ (cid:48) − (cid:15) , we can obtain that
[Equation]
kl
[Equation]
Thompson Sampling for Exponential Family Reward Distributions
We present a general variant of Thompson sampling for exponential family rewards in Algorithm 1,which is termed ExpTS.
At round t , ExpTS maintains an estimate of a sampling distribution for eacharm, denoted as P .
The algorithm generates a sample parameter θ i ( t ) for each arm i independentlyfrom their sampling distribution and chooses the arm that attains the largest sample parameter.
Foreach arm i ∈
[ K ] , the sampling distribution P is usually deﬁned as a function of the total numberof pulls T
i ( t ) and the empirical average reward (cid:98) µ i ( t ) .
After pulling the chosen arm, the algorithmupdates T i ( t ) and (cid:98) µ i ( t ) for each arm based on the reward r t it receives and proceeds to the nextround.
It is worth noting that we study the frequentist regret bound of Algorithm 1 and thus ExpTS is notrestricted as a Bayesian method.
As pointed out by Abeille and Lazaric [1], Jin et al.
[20], Kim et al.
[24], Zhang [35], the sampling distribution does not have to be a posterior distribution derivedfrom a pre-deﬁned prior distribution.
Therefore, we call P the sampling distribution instead of theposterior distribution as in Bayesian regret analysis of Thompson sampling [33, 11].
To obtain theﬁnite-time regret bound of ExpTS for exponential family rewards, we will discuss the choice of ageneral sampling distribution and a new proof technique.
4.1 Challenges in Regret Analysis for Exponential Family Bandits
Before we choose a speciﬁc sampling distribution P for ExpTS, we ﬁrst discuss the main challengesin the ﬁnite-time regret analysis of Thompson sampling, which is the main motivation for our designof P in the next subsection.
Under-Estimation of the Optimal Arm.
Denote (cid:98) µ is as the average reward of arm i after its s -thpull, T i ( t ) as the number of pulls of arm i at time t , and P ( (cid:98) µ is , s ) as the sampling distribution of armi after its s -th pull.
The regret of the algorithm contributed by pulling arm i is ∆
i E [ T i ( T )] ,
where
T i ( T ) is the total number of pulls of arm i .
All existing analyses of ﬁnite-time regret bounds for
TS [3–5, 20] decompose this regret term as ∆ i E [ T i ( T )] ≤ D i
+ h i (∆ i , T, θ i (1) , . .
, θ i ( T )) , whereh i () is a quantity characterizing the over-estimation of arm i which can be easily dealt with by someconcentration properties of the sampling distribution (see Lemma A.3 for more details).
The term
D i characterizes the under-estimation of the optimal arm 1 , which is usually bounded as follows inexisting work:
[Equation]
where G 1 s ( (cid:15) )
= 1 − F 1 s ( µ 1 − (cid:15) ) , F 1 s is the CDF of the sampling distribution P ( (cid:98) µ
1 s , s ) , and(cid:15) = Θ(∆ i ) .
In other words, G 1 s ( (cid:15) )
=
P ( θ 1 ( t ) > µ 1 − (cid:15) ) is the probability that the best arm will notbe under-estimated from the mean reward by a margin (cid:15) .
Furthermore, we can interpret the quantityin (4.1) as the result of a union bound indicating how many samples TS requires to ensure that atleast one sample of the best arm { θ 1 ( t ) }
Tt =1 is larger than µ 1 − (cid:15) .
If G 1 s ( (cid:15) ) is too small, arm 1 couldbe signiﬁcantly under-estimated, and thus D i will be unbounded.
In fact, as shown in Lemma 5 by
Jin et al.
[20], for MAB with Gaussian rewards, TS using Gaussian posteriors will unavoidably sufferfrom the lower bound of K · D i = Ω( √ KT log T ) .
To address the above issue, we introduce a lower conﬁdence bound for measuring the under-estimationproblem.
We use a new decomposition of the regret that bounds D i with the following term
[Equation]
where Low s is a lower conﬁdence bound of (cid:98) µ 1 s .
Intuitively, due to the concentration of arm 1 ’srewards, the probability of (cid:98) µ 1 s ≤ Low s is very small.
Thus, even when G 1 s ( (cid:15) ) is small, the overallregret can be well controlled.
In the regret analysis of TS, we can bound (4.2) from two facets: (1) the lower conﬁdence bound canbe proved using the concentration property of the reward distribution; and (2) the term G 1 s ( (cid:15) ) =
221 P ( θ 1 ( t ) > µ 1 − (cid:15) ) can be upper bounded by the anti-concentration property for the samplingdistribution P .
To achieve an optimal regret, one needs to carefully balance the interplay betweenthese two bounds.
For a speciﬁc reward distribution (e.g., Gaussian, Bernoulli) as is studied by
Agrawal and Goyal [5], Jin et al.
[20], there are already tight anti-concentration inequalities for thereward distribution, and thus the lower conﬁdence bound is tight.
Therefore, by choosing Gaussian or
Bernoulli as the prior (which leads to a Gaussian or Beta sampling distribution P ), we can use existinganti-concentration bounds for Gaussian [2, Formula 7.1.13] or Beta [18, Prop.
A.4] distributions toobtain a tight bound of G 1 s ( (cid:15) ) .
In this paper, we study the general exponential family of reward distributions, which has no closedform.
Thus we cannot obtain a tight concentration bound for ˆ µ 1 s as in special cases such as Gaussianor Bernoulli rewards.
This increases the hardness of tightly bounding term (4.2) and it is imperativefor us to design a sampling distribution P with a tight anti-concentration bound that can carefullycontrol G 1 s ( (cid:15) ) without any knowledge of the closed form distribution of the average reward ˆ µ 1 s .
Dueto the generality of exponential family distributions, it is challenging and nontrivial to ﬁnd such asampling distribution to obtain a tight ﬁnite-time regret bound.
4.2 Sampling Distribution Design in Exponential Family Bandits
In this subsection, we show how to choose a sampling distribution P that has a tight anti-concentrationbound to overcome the under-estimation of the optimal arm and concentration bound to overcomethe over-estimation of the suboptimal arms.
For the simplicity of notation, we denote P ( µ, n ) as the sampling distribution, where µ and n aresome input parameters.
In particular, for ExpTS, we will choose µ = (cid:98) µ i ( t ) and n = T i ( t ) for armi ∈
[ K ] at round t .
We deﬁne P ( µ, n ) as a distribution with PDF
[Equation]
where ( kl ( µ, x )) (cid:48) denotes the derivative of kl ( µ, x ) with respect to x , and b n is a function of n andwill be chosen later.
We assume the reward is supported in [ R min , R max ] .
Note that R min = 0 , and R max = 1 for
Bernoulli rewards, and R min =
−∞ , and R max = ∞ for Gaussian rewards.
Let p ( x ) and q ( x ) bethe density functions of two exponential family distributions with mean values µ p and µ q respectively.
By the deﬁnition in Section 3, we have kl ( µ p , µ q ) =
KL ( p ( x ) , q ( x ))
= (cid:82) R max R min p ( x ) log p ( x ) q ( x ) d x .
Proposition 4.1.
If the mean reward of q ( x ) is equal to the maximum value in its support, i.e.,µ q = R max , we will have kl ( µ, R max )
=
∞ for any µ < R max .
Proof.
First consider the case that R max < ∞ .
Since the mean value concentrates on the maximumvalue, we must have q ( x ) =
for all x < R max , which immediately implies kl ( µ, R max )
= ∞
forany µ < R max .
For the case that R max = ∞ , from (3.3) and the assumption that V < ∞ , we alsohave kl ( µ, ∞ ) =
( ∞ − µ ) 2 /V = ∞ .
Similarly, we can also prove that kl ( µ, R min ) =
∞ for µ > R min .
Based on these properties, wecan easily verify that a sample from the proposed sampling distribution θ ∼ P has the following tailbounds: for z ∈ [ µ, R max ) , it holds that
[Equation]
and for z ∈ ( R min , µ ] , it holds that
[Equation]
Note that (cid:82) R max R min f
( x ; µ, n ) d x = (cid:82) µ R min f
( x ; µ, n
[Equation]
d x + (cid:82) R max µ f ( x ; µ, n ) d x = 1 , which indicates the
PDF of P is well-deﬁned.
Intuition for the Design of the Sampling Distribution.
The tail bounds in (4.4) and (4.5) provideproper anti-concentration and concentration bounds for the sampling distribution P as long as wehave corresponding lower and upper bounds of e − nb n · kl ( µ,z ) .
When n is large, we will choose b n tobe close to 1 , and thus (4.4) and (4.5) ensure that the sample of the corresponding arm concentratesin the interval ( µ − (cid:15), µ + (cid:15) ) with an exponentially small probability e − n kl ( µ − (cid:15),µ + (cid:15) ) , which is crucialfor achieving a tight ﬁnite-time regret.
How to Sample from P .
We show that sampling from P is tractable since the CDF of P is invertible.
And thus we do not need to use approximate sampling methods such as Monte Carlo Markov Chainand Hastings-Metropolis [25].
In particular, according to (4.4) and (4.5), the CDF of P ( µ, n ) is
[Equation]
To sample from P ( µ, n ) , we can ﬁrst pick y uniformly random from [0 , 1] .
Then, for y ≥ 1 / 2 ,we solve the equation y = 1 − 1 / 2 e − nb n · kl ( µ,x ) for x ( x ≥ µ ), which is equivalent to solvinglog(1 / (2(1 − y ))) / ( nb n ) =
kl ( µ, x ) .
For y ≤ 1 / 2 , we solve the equation y = 1 / 2 e − nb n · kl ( µ,x ) forx ( x ≤ µ ), which is equivalent to solving log(1 / (2 y )) / ( nb n ) =
kl ( µ, x ) .
In this way, x would be anexact sample from distribution P .
[Table]
4.3 Regret Analysis of ExpTS for Exponential Family Rewards
Now we present the regret bound of ExpTS for general exponential family bandits.
The samplingdistribution used in Algorithm 1 is deﬁned in (4.3)
[Table]
Theorem 4.2.
Let b n = ( n − 1) /n .
Let P be the sampling distribution deﬁned in Section 4.2.
Theregret of Algorithm 1 satisﬁes the following ﬁnite-time bounds:
[Equation]
where λ ≥ 16 (cid:112) V/T , and also satisﬁes the following asymptotic bound simultaneously:
[Equation]
Remark 4.3.
Similar to the argument by Auer and Ortner [7], we can see that the logarithm termin (4.6) is the main term for suitable λ .
For instance, if we choose λ = 16 (cid:112) V/T , we will havemax i ∈
[ K ] , ∆ i ≤ λ ∆ i T ≤ √ V T , which is in the order of O ( V/ ∆ i ) due to ∆ i ≤ λ .
Thus it is obviousto see that the regret in (4.6) satisﬁes the sub-UCB criteria.
It is worth highlighting that ExpTS is an anytime algorithm and simultaneously satisﬁes the sub-UCBcriteria in (1.2), the minimax optimal regret up to a factor √ log K , and the asymptotically optimalregret.
ExpTS is also the ﬁrst Thompson sampling algorithm that provides ﬁnite-time regret boundsfor exponential family of rewards.
Compared with state-of-the-art MAB algorithms listed in Table 2,
ExpTS is comparable to the best known UCB algorithms that work for exponential family of rewarddistributions and no algorithms can dominate ExpTS.
In particular, compared with MS [10] and
OCUCB [28], ExpTS is asymptotically optimal for exponential family of rewards, while MS is onlyasymptotically optimal for sub-Gaussian rewards and OCUCB is not asymptotically optimal.
We notethat Exponential Family does not cover the sub-Gaussian rewards.
However, since we only use thetail bound to approximate the reward distribution, ExpTS can also be extended to solve sub-Gaussianreward bandits, which we leave as a future open direction.
4.4 Gaussian and Bernoulli Reward Distributions
The choice of P in (4.3) seems complicated for a general exponential family reward distribution,even though we only need the sampling distribution to satisfy a nice tail bound derived from thisreward distribution.
Nevertheless, when the reward distribution has a closed form such as Gaussianand Bernoulli distributions, we can directly use the posterior in standard Thompson sampling andobtain the asymptotic and ﬁnite-time regrets in the previous section.
Theorem 4.4.
If the reward follows a Gaussian distribution with a known variance V , we can setthe sampling distribution in Algorithm 1 as N ( (cid:98) µ i ( t ) , V/T i ( t )) .
The resulting algorithm (denoted as
Gaussian-TS) enjoys the same regret bounds presented in Theorem 4.2.
Remark 4.5.
Jin et al.
[20] shows in Lemma 5 of their paper that for Gaussian rewards, GaussianTS has a regret bound at least Ω( √ V KT log T ) if the standard regret decomposition discussed in
Section 4.1 is adopted in the proof [5, 30, 20].
With our new regret decomposition and the lowerconﬁdence bound introduced in (4.2) , we improve the worst-case regret of Gaussian-TS for Gaussianrewards to O ( √ V KT log K ) .
Jin et al.
[20] also shows that their algorithms MOTS/MOTS-
J can overcome the under-estimationissue of (4.1) .
However, they are either at the cost of sacriﬁcing the asymptotic optimality or notgeneralizable to exponential family bandits.
In speciﬁc, (1) For Gaussian rewards, MOTS [20]enlarges the variance of Gaussian posterior by a factor of 1 /ρ , where ρ ∈ (0 , 1) , which losesthe asymptotic optimality by a factor of 1 /ρ resultantly.
For Gaussian rewards, MOTS- J[20] introduces the Rayleigh posterior to overcome the under-estimation while maintaining theasymptotic optimality.
However, it is not clear whether the idea can be generalized to exponentialfamily rewards.
Interestingly, their experimental results show that compared with Rayleigh posterior,
Gaussian posterior actually has a smaller regret empirically.
Therefore, to use a Gaussian samplingdistribution, the new regret decomposition and the novel lower conﬁdence bound in our paper is abetter way to overcome the under-estimation issue of Gaussian-TS.
Theorem 4.6.
If the reward distribution is Bernoulli, we can set the sampling distribution P in
Algorithm 1 as Beta posterior B ( S i ( t )+1 , T i ( t ) − S
i ( t )+1) , where S i ( t ) is the number of successesamong the T i ( t ) plays of arm i .
We denote the resulting algorithm as Bernoulli-TS, which enjoys thesame regret bounds as in Theorem 4.2.
Remark 4.7.
Agrawal and Goyal [5] proved that for Bernoulli rewards, Thompson sampling with
Beta posterior is asymptotically optimal and has a worst-case regret in the order of O ( √ KT log T ) .
Our regret analysis improve the worst-case regret to O ( √ KT log K ) .
Besides, Agrawal and Goyal[5] proved that Gaussian-TS applied to the Bernoulli reward setting has a regret O ( √ KT log K ) .
However, no asymptotic regret was guaranteed in this setting.
Minimax Optimal Thompson Sampling for Exponential Family Rewards
In this section, in order to remove the extra logarithm term in the worst-case regret of ExpTS, weintroduce a new sampling distribution that adds a greedy exploration step to the sampling distributionused in ExpTS.
Speciﬁcally, the new algorithm ExpTS
+ is the same as ExpTS but uses a newsampling distribution P + ( µ, n ) .
A sample θ is generated from P + ( µ, n ) in the following way: θ = µwith probability 1 − 1 /K and θ ∼ P ( µ, n ) with probability 1 /K .
Over-Estimation of Sub-Optimal Arms.
We ﬁrst elaborate the over-estimation issue of sub-optimalarms, which results in the extra √ log K term in the worst-case regret of Thompson sampling.
Toexplain, suppose that the sample of each arm i has a probability p
=
P ( θ i ( t ) ≥ θ 1 ( t )) to becomelarger than the sample of arm 1.
Note that when this event happens, the algorithm chooses the wrongarm and thus incurs a regret.
Intuitively, the probability of making a mistake will be K − 1 timeslarger due to the union bound over K − 1 sub-optimal arms, which leads to an additional √ log Kfactor in the worst-case regret.
To reduce the probability P ( θ i ( t ) ≥ θ 1 ( t )) , ExpTS + adds a greedystep that chooses the ExpTS sample with probability 1 /K and chooses the arm with the largestempirical average reward with probability 1 − 1 /K .
Then we can prove that for sufﬁciently larges , with high probability we have ˆ µ is < θ 1 ( t ) and in this case it holds that P ( θ i ( t ) ≥ θ 1 ( t ))
= p/K .
Thus the extra factor √ log K in regret is removed.
In speciﬁc, we have the following theorem showing that ExpTS + is asymptotically optimal andminimax optimal simultaneously.
Theorem 5.1.
Let b n = ( n − 1) /n .
The regret of ExpTS + satisﬁes
[Equation]
and
[Equation]
This is the ﬁrst time that the Thompson sampling algorithm achieves the minimax and asymptoticallyoptimal regret for exponential family of reward distributions.
Moreover, ExpTS + is also an anytimealgorithm since it does not need to know the horizon T in advance.
Remark 5.2 (Sub-Gaussian Rewards) .
In the proof of Theorem 5.1, we do not need the strict form ofthe PDF of the empirical mean reward (cid:98) µ is , but only need the maximal inequality (Lemma H.1).
Thismeans that the proof can be straightforwardly extended to sub-Gaussian reward distributions, wheresimilar maximal inequality holds [21].
It is worth noting that MOTS proposed by Jin et al.
[20] (Thompson sampling with a clipped Gaussianposterior) also achieves the minimax optimal regret for sub-Gaussian rewards, but it can not keepthe asymptotic optimality simultaneously with the same algorithm parameters.
In particular, toachieve the minimax optimality, MOTS will have an additional 1 /ρ factor in the asymptotic regretwith 0 < ρ < 1 .
Moreover, different from ExpTS + , MOTS is only designed for ﬁxed T setting andthus is not an anytime algorithm.
Remark 5.3 (Gaussian and Bernoulli Rewards) .
Following the idea of ExpTS + , we can easily obtainnew algorithms Gaussian-TS + and Bernoulli-TS + for Gaussian and Bernoulli rewards.
Using asimilar proof, we can show that they are simultaneously minimax and asymptotically optimal.
Conclusions
We studied Thompson sampling for exponential family of reward distributions.
We proposed the
ExpTS algorithm and proved it satisﬁes the sub-UCB criteria for problem-dependent ﬁnite-timeregret, as well as achieves the asymptotic optimality and the minimax optimality up to a factor of 370 √ log K for exponential family rewards.
Furthermore, we proposed a variant of ExpTS, dubbed
ExpTS + , that adds a greedy exploration step to balance between the sample generated in ExpTSand the empirical mean reward for each arm.
We proved that ExpTS + is simultaneously minimaxand asymptotically optimal.
We also extended our proof techniques to standard Thompson samplingwith common posterior distributions and improved existing results.
For Gaussian/Bernoulli rewards,we proved Gaussian-TS/Bernoulli-TS satisﬁes the sub-UCB criteria and enjoys the asymptoticallyoptimal regret and minimax optimal regret O ( √ KT log K ) .
Although ExpTS + is simultaneouslyminimax and asymptotically optimal, it does not satisﬁes the sub-UCB criteria anymore.
It wouldbe an interesting future direction to design a sampling distribution such that TS is asymptoticallyoptimal, minimax optimal and matches the sub-UCB criteria at the same time