A Simple and Optimal Policy Design for Online Learning with Safety against Heavy-tailed Risk
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the classical multi-armed bandit problem and design simple-toimplement new policies that simultaneously enjoy two properties: worst-caseoptimality for the expected regret, and safety against heavy-tailed risk for theregret distribution.
We start by showing that many widely used policies such asthe standard Upper Confidence Bound policy and the Thompson Sampling policyincur heavy-tailed risk; that is, the worst-case probability of incurring a linearregret slowly decays at a polynomial rate of 1 /T , where T is the time horizon.
Wefurther show that this heavy-tailed risk exists for all “instance-dependent consistent”policies.
Next, for the two-armed bandit setting, we provide a simple policy designthat (i) has the worst-case optimality for the expected regret at order ˜ O ( √ T ) and(ii) has the worst-case tail probability of incurring a linear regret decay at an exponential rate exp( − Ω( √ T )) .
We further prove that this exponential decaying rateof the tail probability is optimal across all policies that have worst-case optimalityfor the expected regret.
Finally, we generalize the policy design and analysis tothe general setting with an arbitrary K number of arms.
We provide detailedcharacterization of the tail probability bound for any regret threshold under ourpolicy design.
Numerical experiments are conducted to illustrate the theoreticalfindings.
Our results reveal insights on the incompatibility between consistencyand light-tailed risk, whereas indicate that worst-case optimality on expected regretand light-tailed risk are compatible.
Our policy design and proof techniques mayhave the potential to be adapted to other online learning tasks that prefer to havelight-tailed risk on regret distribution.
Introduction
The stochastic multi-armed bandit (MAB) problem is a widely studied problem in the domain ofsequential decision-making under uncertainty, with many applications such as online advertising,recommendation systems, clinical trials, financial portfolio design, etc.
It also has valuable theoreticalinsights exhibiting the exploration-exploitation trade-offs. For policy design and analysis, much ofthe MAB literature uses the metric of maximizing the expected cumulative reward, or equivalentlyminimizing the expected regret (where regret is defined as the difference between the cumulativereward obtained by always pulling the best arm and by executing a policy that does not a prioriknow the reward distributions).
The optimality of a policy is often characterized through its expectedregret’s rate (order of dependence) on the experiment horizon T .
In this paper, however, we show that renowned policies (such as the standard Upper Confidence
Bound (UCB) policy, the Successive Elimination (SE) policy and the Thompson Sampling (TS)policy) that are designed to enjoy optimality in terms of expected regret can incur a “heavy-tailed risk”.
That is, the distribution of the regret has a heavy tail — the probability of incurring a linear regret slowly decays at a polynomial rate Ω(poly(1 /T ))
as T tends to infinity.
In contrast, a “light-tailed”risk means that the probability of a policy incurring a linear regret decays at an exponential rateexp( − Ω( T β )) for some β > 0 .
The heavy-tailed risk can be undesired when an MAB policy is usedin applications that are sensitive to tail risks (e.g., finance, healthcare, supply chain).
Our work isprimarily motivated by trying to answer the following question.
Can we design a simple policy thaton one hand enjoys the optimality under the expected regret notion, whereas on the other handhas light-tailed risk?
The contributions of our work are summarized as follows.
Instant-dependent consistency and light-tailed risk are incompatible.
We show that a wide
range of policies, including the standard UCB and SE policy, and the TS policy, suffer from heavytailed risk.
More precisely, each of them incurs a linear regret with probability Ω(poly(1 /T )) = exp( − O (ln T )) .
We generalize the result and show that any instant-dependent consistent policy cannot be light-tailed: if an instant-dependent consistent policy has the probability of incurring alinear regret decay as exp( − f ( T )) , then f ( T ) must be o ( T β ) as T → + ∞ for any β > 0 .2.
Worst-case optimality and light-tailed risk can co-exist.
Starting from the two-armed bandit setting, we provide a simple policy design and prove that it enjoys both the worst-case optimality˜ O ( √ T ) for the expected regret and the light-tailed risk exp( − Ω( √ T )) for the regret distribution. We
also show that such exponential decaying rate of the tail probability is optimal within the class ofworst-case optimal policies.
Extensions and Experiments.
We extend the result from the two-armed bandit to the general
K -armed bandit and characterize the tail probability bound for any regret threshold in an explicit form.
We prove that under our policy design, the worst-case probability of incurring a regret larger
than x is bounded by exp( − Ω( x/K √ T )) .
We find that the associated proof techniques are new andmay benefit broader analysis on regret distribution and tail risk.
Our result also partially solves anopen problem raised in [12].
Experiments 1 are also conducted to illustrate our theoretical findings.
1.1 Related Work
Our work builds upon the vast literature of designing and analyzing policies for the multi-armed bandit(MAB) problem and its various extensions.
A comprehensive review can be found in [5, 15, 12].
Astandard paradigm for obtaining a near-optimal regret is to first fix some confidence parameter δ > 0 .
Then a “good event” is defined such that good properties are retained conditioned on the event (forexample, in the MAB problem, the good event is such that the mean of each arm always lies in theconfidence bound).
Then one can obtain both high-probability and worst-case expected regret boundsthrough careful analysis on the good event.
It is known that the MAB problem has the following regretbound: for any fixed δ ∈ (0 , 1) , the regret bound of UCB is bounded by O ( (cid:112) KT ln( T/δ )) withprobability at least 1 − δ .
Or equivalently speaking, the probability of incurring a Ω( (cid:112) KT ln( T/δ ))regret is bounded by δ .
However, the parameter δ must be an input parameter for the policy.
We willdiscuss this issue in more details in Section 3.
In Section 17.1 of [12], an open question is asked: Is itpossible to design a single policy such that the worst-case probability of incurring a Ω( √ KT ln(1 /δ ))regret is bounded by δ for any δ > 0 ?
We partially solve this question by designing a policy such thatfor any δ > 0 , the probability of incurring a Ω( K √ T ( √ ln T + ln(1 /δ ))) regret is bounded by δ .
There has been not much work on understanding the tail risk of bandit algorithms.
Two earlier worksare [4, 13] and they study the concentration properties of the regret around the instance-dependentmean O (ln T ) .
They show that in general the regret of the policies concentrate only at a polynomialrate.
That is, the probability of incurring a regret of c (ln T ) p (with c > 0 and p > 1 fixed) isapproximately polynomially decaying with T .
Different from our work, the concentration in theirwork is under an instance-dependent environment, and so such polynomial rate might be differentacross different instances.
Nevertheless, their results indicate that standard bandit algorithms generallyhave undesirable concentration properties.
Recently, [3] show that an online learning policy with thegoal of obtaining logarithmic regret can be fragile, in the sense that a mis-specified risk parameter(e.g.
, the parameter for subgaussian noises) in the policy can incur an instance-dependent expectedregret polynomially dependent on T .
They then focus on robust algorithm development to circumventthe issue.
Note that their goal is to handle mis-specification related with risk, but still the task is tominimize the expected regret.
Another recent work is [8].
They show that for an information-theoretic optimized bandit policy, the probability of incurring a linear regret is at least Ω(1 /T ) .
We generalizetheir result by stating that a much larger family of policies with consistency is heavy-tailed: it incursa linear regret with probability Ω(poly(1 /T )) .
Further, aside from this negative result, we give apositive result by showing that our proposed new policy design can lead to both worst-case light-tailedrisk (tail bound exponentially decaying with √ T ) and worst-case optimality (expected regret boundedby ˜ O ( √ T ) ).
The previous works do not have this positive result.
Very recently, there are some works trying to understand the behaviour of UCB and TS policies byconsidering the diffusion approximations (see, e.g., [2, 17, 7, 10]).
Although some distributionalcharacterizations of regret are obtained in these works, the asymptotic regime is typically set such thatthe gaps between arm means shrink with the total time horizon.
This makes their results incomparableto ours because we study how the tail probability decays with T under unchanged environments.
Another line of works closely related with ours involve solving risk-averse formulations of the MABproblem (see, e.g., [14, 9, 18, 16, 6, 11]).
Compared to standard MAB problems, the main differencein their works is that arm optimality is defined using formulations other than the expected value, suchas mean-variance criteria and (conditional) value-at-risk.
These formulations consider some singlemetric that is different compared to the expected regret.
From the formulation perspective, our workis different in the sense that we develop policies that on one hand maintain the classical worst-caseoptimal expected regret, whereas simultaneously achieve light-tailed risk bound.
The policy designand analysis in our work are therefore also different from the literature and might be of independentinterest.
Setup and Notation
The setting is described as follows.
Fix a time horizon of T and the number of arms as K .
In eachtime t ∈ [ T ] , the decision maker (DM) needs to decide which arm A t ∈
[ K ] should be pulled.
Tobe more precise, let H t
= { A 1 , X 1 , · · · , A t − 1 , X t − 1 } be the history prior to time t .
When t = 1 ,
H 1 = ∅ .
In time t , the DM adopt a policy π t : H t (cid:55)−→ A t that maps the history H t to an action
A t , where A t follows a discrete probability distribution on [ K ] determined by H t .
The environmentthen independently samples a reward r t,A t = θ
A t + ϵ t,A t and reveals it to the DM.
Here, θ A t is the 115 mean reward of arm A t , and ϵ t,A t is an independent zero-mean noise term.
We assume that ϵ t,A t is 116 σ -subgaussian.
That is, there exists a σ > 0 such that for any time t and arm k ,
[Equation]
Let θ =
( θ 1 , · · · , θ K ) be the mean vector.
Let θ ∗ = max { θ 1 , · · · , θ K } be the optimal mean rewardamong the K arms.
Note that DM does not know θ at the beginning, except that θ ∈ [0 , 1] K .
Theempirical regret of the policy π =
( π 1 , · · · , π T ) under the mean vector θ over a time horizon of T isdefined as
[Equation]
Let ∆ k
= θ ∗ − θ k be the gap between the optimal arm and the k th arm.
Let n t,k be the number oftimes arm k has been pulled up to time t .
That is, n t,k = (cid:80) ts =1 1 { A s = k } .
For simplicity, we willalso use n k = n T,k to denote the total number of times arm k is pulled throughout the whole timehorizon.
We define t k ( n ) as the time period that arm k is pulled for the n th time.
Define the pseudoregret as
[Equation]
and the genuine noise as
[Equation]
Then the empirical regret can also be written as ˆ R πθ ( T ) =
R πθ ( T ) − N π ( T ) .
The following simplelemma gives the mean and the tail probability of the genuine noise N π ( T ) .
Intuitively, it showswhen bounding the mean or the tail probability of the empirical regret, one only need to consider thepseudo regret term.
We will make it more precise when we discuss the proof of main theorems.
Lemma
We have E [ N π ( T )]
= 0 and
[Equation]
Before proceeding, we introduce some other notations.
Throughout the paper, we use O ( · ) ( ˜ O ( · ) )
andΩ( · ) ( ˜Ω( · ) ) to present upper and lower bounds on the growth rate up to constant (logarithmic) factors,and Θ( · ) ( ˜Θ( · ) ) to characterize the rate when the upper and lower bounds match up to constant(logarithmic) factors.
We use o ( · ) to present strictly dominating upper bounds.
In addition, for anya, b ∈ R , a ∧ b = min { a, b } and a ∨ b = max { a, b } .
For any a ∈ R , a + = max { a, 0 } .
2.1 Light-tailed Risk, Instance-dependent Consistency, Worst-case Optimality
Now we discuss several core concepts that we are interested in.
Light-tailed risk.
A policy is called light-tailed, if for any constant c > 0 , there exists some β > and constant C > such that
[Equation]
Note that here, we allow β and C to be dependent on c .
In brief, a policy has light-tailed risk if theprobability of incurring a linear regret can be bounded by an exponential term of polynomial T :
[Equation]
for some β > 0 .
If a policy is not light-tailed, we say it is heavy-tailed .
Instance-dependent consistency.
A policy is called consistent, if for any underlying true mean vector θ , the policy has that
[Equation]
holds for any β > 0 .
In brief, a policy is consistent if the expected regret can never be polynomiallygrowing in T for any fixed instance.
Worst -case optimality.
A policy is said to be worst case optimal, if for any β >
we
have
[Equation]
In brief, a policy is worst-case optimal if the worst-case expected regret can never be growing ina polynomial rate faster than T 1 / 2 .
Note that here we adopt a relaxed definition of optimality, inthe sense that we do not clarify how the regret scale with the number of arms K compared to thatin literature.
The notion of worst-case optimality in this work focuses on the dependence on T .
For example, a policy with worst-case regret O (poly( K ) √ T · poly(ln T )) is also optimal by ourdefinition.
It is well known that for the multi-armed bandit problem, one can design algorithms to achieve bothinstance-dependent consistency and worst-case optimality.
Among them, two types of policies are ofprominent interest: Successive Elimination (SE) and Upper Confidence Bound (UCB).
We list thealgorithm paradigms in Algorithm 1 and 2.
The bonus term rad ( n ) is typically set as
[Equation]
with η > 0 being some tuning parameter.
Algorithm 1: A = [ K ] .
t ← 0 . 2: while t < T do 3: Pull each arm in A once. t ← t + K .
Eliminate any k ∈ A from A if ∃ k ′ :
ˆ µ t,k ′ − rad ( n t,k ′ ) >
ˆ µ t,k + rad ( n t,k ) 5: end while Algorithm Bound 1: A = [ K ] .
t ← 0 .
2: while t < T do 3: t ← t + 1 .
Pull the arm with the highest UCB:
UCB t − 1 ,k
= ˆ µ t − 1 ,k + rad ( n t − 1 ,k ) .
5: end while
Main Results: Two-armed Bandit
We start from the simple two-armed bandit setting.
The general multi-armed setting is deferred to thenext section.
We first show that standard policies considered widely in the literature are heavy-tailed.
The result reveals a fundamental incompatibility between consistency and light-tailed risk.
Then weshow how to add a simple twist to standard confidence bound based policies to obtain light-tailedpolicies.
Moreover, we show that our design leads to an optimal tail decaying rate.
3.1 Instance-dependent Consistency Causes Heavy-tailed Risk
Theorem 1 If a policy is instance-dependent consistent, then it can never be light-tailed.
Moreover,if π = SE or UCB with
[Equation]
for some η > 0 , we have the following stronger argument.
For any c > 0 , there exists C π > 0 suchthat
[Equation]
Theorem 1 suggests that a consistent policy must have a risk tail heavier than an exponential one.
The proof of Theorem 1 relies on a change of measure argument appeared in [8].
We consider theenvironment where the noise ϵ is Gaussian with standard deviation σ .
Intuitively speaking, if we wanta policy to be adaptive enough to handle different instances, then the policy will be sensitive to risk.
Moreover, for the family of confidence bound related policies (SE and UCB), the standard bonusterm will lead to a tail polynomially dependent on T . This polynomial dependency stems solely fromthe fact that when π = SE or UCB, for any fixed θ , we have
[Equation]
We note that Theorem 1 is general enough to include other families of policies aside from SE and UCB.
One example is the Thompson Sampling (TS) policy.
It has been established that π = TS with Beta prior has the property (3) (see, e.g., Theorem 1 and 2 in [1]).
Our proof suggests that (2) alsoholds for π = TS.
We need to remark on the difference between Theorem 1 and high-probability bounds in previousliterature.
It has been well-established that UCB or SE with
[Equation]
n achieves ˜ O
( (cid:112) T · polylog( T/δ )) regret with probability at least 1 − δ .
Such design also leads to aconsistent policy.
However, the bound holds only for fixed δ .
In fact, one can see that the bonus designis dependent on the confidence parameter δ .
If δ = exp( − Ω( T β )) with β > 0 , then the scaling speedof the regret with respect to T can only be greater than 1 / 2 , which is sub-optimal.
As a comparison,in our problem, ideally we seek to find a single policy such that it achieves ˜ O
( (cid:112) T · polylog( T/δ ))regret for any δ > 0 .
Up till now, we make two core observations.
First, from standard MAB results, consistency andoptimality can hold simultaneously.
Second, from Theorem 1, consistency and light-tailed riskare always incompatible.
Then a natural question arises: Can we design a policy that enjoys bothoptimality and light-tailed risk?
If we can, then can we make the tail risk decay with T in an optimalrate?
We answer these two questions with an affirmative “yes” in the next section.
3.2 A New Policy Design Achieving Light-tailed Risk and Worst-case Optimality
In this section, we propose a new policy design that achieves both light-tailed risk and worst-caseoptimality.
The design is simple.
We still use the idea of confidence bounds, but instead of setting thebonus as (1), we set
[Equation]
with η > 0 being a tuning parameter.
Theorem 2 gives performance guarantees for the mean and thetail probability of the empirical regret when π = SE.
Theorem 2 For the two-armed bandit problem, the SE policy with η ≥ 4 and the bonus term being(4) satisfies the following two properties.
sup θ E [ ˆ R πθ ]
= O ( √ T ln T ) .
For any c > 0 and anyα ∈ (1 / 2 , 1] , we have
[Equation]
The first item in Theorem 2 means that with the modified bonus term, the worst case regret isstill bounded by O ( √ T ln T ) , which is the same as the regret bounds for SE and UCB with thestandard bonus term (1).
The second item shows that the tail probability of incurring a Ω( T α )regret ( α > 1 / 2 ) is exponentially decaying in Ω( T α − 1 / 2 ) , and thus the policy is light-tailed.
Thedetailed proof of Theorem 2 is provided in the supplementary material.
An illustrative road-mapof the proof is delegated to Section 4, where we provide the proof idea for a theorem that is a strictgeneralization of Theorem 2.
Here, we give some intuition on the new bonus design.
Our new bonusterm inflates the standard one by a factor of (cid:112) T/n .
This means our policy is more conservative thanthe traditional confidence bound methods, especially at the beginning.
In fact, one can observe thatfor the first Θ( √ T ) time periods, our policy consistently explores between arm 1 and 2 , regardless ofthe environment.
A natural corollary is that our policy is never “consistent”, and this is reasonablefollowing Theorem 1.
However, the bonus term (4) decays at a faster rate on the number of pullingtimes n compared to (1).
This means as the experiment goes on, the policy leans towards exploitation.
We need to stress that this is not the same as the explore-then commit policy, which is well-known toachieve a sub-optimal Θ( T 2 / 3 ) regret.
The following theorem shows that the risk tail in Theorem 2 is not improvable.
That is, if the policyπ is worst-case optimal, then for fixed α ∈ (1 / 2 , 1] , the exponent of α − 1 / 2 is tight.
Theorem 3 Let c ∈ (0 , 1 / 2) .
Consider the 2-armed bandit problem with gaussian noise.
Let π be aworst-case optimal policy.
That is, for any α > 1 / 2 ,
[Equation]
Then for any α ∈ (1 / 2 , 1] ,
[Equation]
holds for any β > α − 1 / 2 .
Theorem 3 also relies on the change of measure argument appeared in Theorem 1.
However, thereare two important differences: we only have worst-case optimality rather than consistency, and theregret threshold cT α is in general not linear in T .
Therefore, we need to take care of constructing thespecific “hard” instance when doing the change of measure.
The detailed proof is delegated to thesupplementary material.
Extensions: Multi-armed Bandit
In this section, we provide an extension of our previous tail probability bound in Theorem 2 from thefollowing three aspects: (a) a general K -armed bandit setting; (b) an analysis for UCB aside from
SE; (c) a detailed characterization of the tail bound for any fixed regret threshold.
Theorem 4 For the K -armed bandit problem, π = SE or π = UCB with
[Equation]
n satisfy the following two properties.
If η ≥ 4 , then sup θ E [ R πθ ] ≤ 4 K + 4 Kσ √ ηT ln T . 2.
Ifη > 0 , then for any x > 0 , we have
[Equation]
Proof Idea.
We provide a road-map of proving Theorem 4.
The expected regret bound is proved usingstandard techniques.
That is, we define the good event to be such that the mean of each arm alwayslies in the confidence bounds throughout the whole time horizon.
Conditioned on the good event, theregret of each arm is bounded by O ( √ T ln T ) , and thus the total expected regret is O ( K √ T ln T ) .
The proof of the tail bound requires more effort.
Without loss of generality, we assume arm 1 isoptimal.
We first illustrate the proof for π = SE.
Step 1.
We use
[Equation]
The term with the genuine noise can be easily bounded using Lemma 1.
We are left to bound the tailof the pseudo regret.
By a union bound, we observe that
[Equation]
Thus, we reduce bounding the sum of the regret incurred by different arms to bounding that by asingle sub-optimal arm.
Step 2.
For any k ̸ = 1 , we define
[Equation] arm k } .
With a slight abuse of notation, we let n 0 = ⌈ x/ (2 K ∆ k )
⌉ − 1 .
If both n k ∆ k ≥ x/ (2 K ) and S khappen, then arm 1 and k are not eliminated after each of them being pulled n 0 times.
This indicates
[Equation]
The probability of this event can be bounded using concentration of subgaussian variables, whichyields the second term in the tail bound in Theorem 4.
The choice of n 0 is important.
We note that atthis step, even if we replace our new bonus term by the standard one, the bound still holds.
Step 3.
Now consider the situtation when ¯ S k happens.
This means after some phase n , the optimal arm1 is eliminated by some arm k ′ , while arm k is not eliminated.
Note that k = k ′ does not necessarilyhold when K > 2 .
As a consequence, we have the following two events hold simultaneously:
[Equation]
The first event leads to
Mean of some noise terms ≥ 2 σ
[Equation]
The second event leads to
Mean of some noise terms ≥ ∆ k ≥ x 2 KT .
Now comes the trick to deal with an arbitrary n .
We bound the probabilities of the two eventsseparately and take the minimum of the two probabilities ( P ( A ∩ B ) ≤ min { P ( A ) , P ( B ) } ( ∀ A, B ) ).
Then such minimum can be further bounded using the basic inequality ( a + b ) / 2 ≥ √ ab ( ∀ a, b ≥ 0) .
This yields the last term in Theorem 4.
We note that at this step, the standard bonus term (1) does notsuffice to get an exponential bound.
The √ T/n design in our new bonus term plays a crucial role.
We next illustrate the proof for π = UCB.
The proof is in fact simpler.
We use the first step in theproof for π = SE.
For fixed k , we also take the same n 0 =
⌊ x/ (2 K ∆ k ) ⌋ − 1 .
The difference hereis that we do not need to define the event S k .
When arm k is pulled for the ( n 0 + 1) th time, by thedesign of the UCB policy, there exists some n such that
[Equation]
Now comes the trick.
The event is included by a union of two events described as follows:
[Equation]
and
[Equation]
The probability of each of the two events can be bounded using similar techniques adopted whenπ = SE.
In fact, it is implicitly shown in our proof that UCB can yield better constants than SE.
Westill need to emphasize that when bounding the second event, similar to the argument for π = SE, thechoice of our new bonus term is crucial.
Remarks.
For the tail bound, from our proof road-map, one can see that the tail bound in Theorem 4is also valid for the pseudo-regret sup θ P
( R πθ ( T ) ≥ x ) .
In fact, with some simple manipulations onthe inequality, if we lety
[Equation]
then for any x ≥ 0 , we can get a neat form (we illustrate this point in the supplementary material)
[Equation]
There are two observations:(a)
For any η > 0 , our policy always yields a O ( √ T ) expected regret (although with a constantlarger than that in the first result in Theorem 4).
In fact, notice that for any x >
[Equation]
If we let x = 2 K + CσK (cid:112) ( η ∨ 1 /η )
T ln T with C > 0 being moderately large, then
P ( R πθ ( T ) ≥ x ) · T = O (1) .
As a result, the worst-case regret becomes
[Equation]
This observation shows that the regret order of our policy design is not related to theparameter η , as opposed to the standard UCB policy with (1), where a very small η maypossibly make the UCB policy no longer enjoy a ˜ O ( √ T ) worst-case regret 2 .(b)
If we set η = 1 and
[Equation]
then one can see that for any δ > 0 , with probability at least 1 − δ , the regret of our policyis bounded by
[Equation]
This partially solve the open question in Section 17.1 of [12] (we’ve mentioned it in theintroduction) up to a logarithmic factor.
Numerical Experiments
In this section, we provide a brief two-armed numerical experiment result and defer the experimentdetails and other multi-armed experiment results to the supplementary material.
We consider a twoarmed-bandit problem with θ =
8) , σ = 1 , T = 500 and Gaussian noise.
We test four policies:
SE and UCB with the classical bonus design described in (1), and SE_new and UCB_new with theproposed new bonus design in (4).
The tuning parameter has 4 choices: η ∈ { 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 } .
Foreach policy and η , we run 5000 simulation paths and for each path we collect the cumulative reward.
We plot the empirical distribution (histogram) for a policy’s cumulative reward in Figure 1.
Indeed,one can observe that for both SE and UCB with (1), there is a significant part of distribution around100 , indicating a significant risk of incurring a linear regret, especially when η is small.
In contrast,with the new design (4), the reward is highly concentrated for every η > 0 with almost no tail risk ofgetting a low total reward.
Particularly, when η = 0 .
1 , UCB_new achieves both high empirical meanand light-tailed distribution.
Conclusion
[Figure] In this work, we consider the MAB problem with a joint goal of minimizing the worst-case expectedregret and obtaining light-tailed probability bound of the regret distribution.
We characterize theinterplay among three concepts: light-tailed risk, instance-dependent consistency, and worst-caseoptimality.
We demonstrate that light-tailed risk and instance-dependent consistency are incompatible,and show that light-tailed risk and worst-case optimality can co-exist through a simple new policydesign.
We discuss insights and generalizations of our results, and study the empirical performance.
An open question is whether we can improve the regret bound in Theorem 4 to Θ( √ KT ln T ) andget a probability bound of
[Equation]
One may be tempted to try a bonus term like σ √ ηT ln T √ Kn . However, this will make the standard regretbound analysis fail because the mean of an arm is no longer guaranteed to lie in the confidencebounds throughout the whole time horizon with high probability.
Moreover, it’s challenging to showthe log probability bound of the form − x/ √ KT rather than − x/K √ T in Theorem 4.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[N/A]
We actually collect rewards and draw the empirical distribution.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table] [Table] any
[Table] [Table] if applicable [Table]
[N/ [Table] [Table]
the estimated [Table] paid to participants and the total [Table]
on participant compensation?
[N/A
[Table]