A Simple and Optimal Policy Design for Online Learning with Safety against Heavy-tailed Risk
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the classical multi-armed bandit problem and design simple-toimplement new policies that simultaneously enjoy two properties: worst-caseoptimality for the expected regret, and safety against heavy-tailed risk for the regretdistribution.
Recently, [9] showed that information-theoretic optimized banditpolicies as well as general UCB policies suffer from some serious heavy-tailed risk;that is, the probability of incurring a linear regret slowly decays at a polynomial rateof 1 /T , as T (the time horizon) increases.
Inspired by their result, we further showthat any policy that incurs an instance-dependent O (ln T ) regret must incur a linearregret with probability Ω(poly(1 /T )) and that the heavy-tailed risk actually existsfor all “instance-dependent consistent" policies.
Next, for the two-armed banditsetting, we provide a simple policy design that (i) has the worst-case optimality forthe expected regret at order ˜ O ( √ T ) and (ii) has the worst-case tail probability ofincurring a linear regret decay at an exponential rate exp( − Ω( √ T )) .
We furtherprove that this exponential decaying rate of the tail probability is optimal acrossall policies that have worst-case optimality for the expected regret.
Finally, wegeneralize the policy design and analysis to the general setting with an arbitrary K number of arms.
We provide detailed characterization of the tail probabilitybound for any regret threshold under our policy design.
Numerical experimentsare conducted to illustrate the theoretical findings.
Our results reveal insights onthe incompatibility between consistency and light-tailed risk, whereas indicate thatworst-case optimality on expected regret and light-tailed risk are compatible.
Ourpolicy design and proof techniques may have the potential to be adapted to otheronline learning tasks that prefer to have light-tailed risk on regret distribution.
Introduction
The stochastic multi-armed bandit (MAB) problem is a widely studied problem in the domain ofsequential decision-making under uncertainty, with many applications such as online advertising,recommendation systems, clinical trials, financial portfolio design, etc.
It also has valuable theoreticalinsights exhibiting the exploration-exploitation trade-off For policy design and analysis, much ofthe MAB literature uses the metric of maximizing the expected cumulative reward, or equivalentlyminimizing the expected regret (where regret is defined as the difference between the cumulativereward obtained by always pulling the best arm and by executing a policy that does not a prioriknow the reward distributions).
The optimality of a policy is often characterized through its expectedregret’s rate (order of dependence) on the experiment horizon T .
In this paper, however, we show that renowned policies (such as the standard Upper Confidence
Bound (UCB) policy, the Successive Elimination (SE) policy and the Thompson Sampling (TS)policy) that are designed to enjoy optimality in terms of expected regret can incur a “heavy-tailed risk”.
That is, the distribution of the regret has a heavy tail — the probability of incurring a linear regretslowly decays at a polynomial rate Ω(poly(1 /T ))
as T tends to infinity.
In contrast, a “light-tailed”risk means that the probability of a policy incurring a linear regret decays at an exponential rateexp( − Ω( T β )) for some β > 0 .
The heavy-tailed risk can be undesired when an MAB policy is usedin applications that are sensitive to tail risks (e.g., finance, healthcare, supply chain).
Our work isprimarily motivated by trying to answer the following question.
Can we design a simple policy thaton one hand enjoys the optimality under the expected regret notion, whereas on the other handhas light-tailed risk?
The results of our work are summarized as follows.
Instant-dependent consistency and light-tailed risk are incompatible.
Inspired by the resultsin [8]
we adapt their change of measure argument to show that any policy that incurs an instance-
dependent O (ln T ) regret must incur a linear regret with probability Ω(poly(1 /T )) .
We additionallyshow that any instant-dependent consistent policy cannot be light-tailed: if an instant-dependentconsistent policy has the probability of incurring a linear regret decay as exp( − f ( T )) , then f ( T )must be o ( T β ) as T → + ∞ for any β > 0 .2.
Worst-case optimality and light-tailed risk can co-exist.
Starting from the two-armed banditsetting, we provide a simple policy design and prove that it enjoys both the worst-case optimality˜ O ( √ T ) for the expected regret and the light-tailed risk exp( − Ω( √ T )) for the regret distribution.
Wealso show that such exponential decaying rate of the tail probability is optimal within the class of worst-case optimal policies.
Extensions and Experiments.
We extend the result from the two-armed bandit to the general
K -armed bandit and characterize the tail probability bound for any regret threshold in an explicitform.
We prove that under our policy design, the worst-case probability
of incurring a regret largerthan x is bounded by exp( − Ω( x/K √ T )) .
We find that the associated proof techniques are new andmay benefit broader analysis on regret distribution and tail risk.
Our result also partially solves anopen problem raised in [13].
Experiments are also conducted to illustrate our theoretical findings.
1.1 Related Work
Our work builds upon the vast literature of designing and analyzing policies for the multi-armed bandit(MAB) problem and its various extensions.
A comprehensive review can be found in [5, 16, 13].
Astandard paradigm for obtaining a near-optimal regret is to first fix some confidence parameter δ > 0 .
Then a “good event” is defined such that good properties are retained conditioned on the event (forexample, in the MAB problem, the good event is such that the mean of each arm always lies in theconfidence bound).
Then one can obtain both high-probability and worst-case expected regret boundsthrough careful analysis on the good event.
It is known that the MAB problem has the following regretbound: for any fixed δ ∈ (0 , 1) , the regret bound of UCB is bounded by O ( (cid:112) KT ln( T/δ )) withprobability at least 1 − δ .
Or equivalently speaking, the probability of incurring a Ω( (cid:112) KT ln( T/δ ))regret is bounded by δ .
However, the parameter δ must be an input parameter for the policy.
We willdiscuss this issue in more details in Section 3.
In Section 17.1 of [13], an open question is asked: Is itpossible to design a single policy such that the worst-case probability of incurring a Ω( √ KT ln(1 /δ ))regret is bounded by δ for any δ > 0 ?
We partially solve this question by designing a policy such thatfor any δ > 0 , the probability of incurring a Ω( K √ T ( √ ln T + ln(1 /δ ))) regret is bounded by δ .
There has been not much work on understanding the tail risk of bandit algorithms.
Our work isinspired by [9].
They showed that optimized UCB bandit designs are fragile to mis-specificationsand they modified UCB algorithms to ensure a desired polynomial rate of tail risk, which makesthe algorithms more robust to mis-specifications.
Different from their work, we propose a simpleand new policy design that leads to both light-tailed risk (tail bound exponentially decaying with 81 √
T ) and worst-case optimality (expected regret bounded by ˜ O ( √ T ) ).
Besides [9], two earlier worksare [4, 14] and they study the concentration properties of the regret around the instance-dependentmean O (ln T ) .
They show that in general the regret of the policies concentrate only at a polynomialrate.
That is, the probability of incurring a regret of c (ln T ) p (with c > 0 and p > 1 fixed) isapproximately polynomially decaying with T .
Different from our work, the concentration in theirwork is under an instance-dependent environment, and so such polynomial rate might be differentacross different instances.
Nevertheless, their results indicate that standard bandit algorithms generallyhave undesirable concentration properties.
Recently, [3] show that an online learning policy with thegoal of obtaining logarithmic regret can be fragile, in the sense that a mis-specified risk parameter
(e.g., the parameter for subgaussian noises) in the policy can incur an instance-dependent expectedregret polynomially dependent on T .
They then focus on robust algorithm development to circumventthe issue.
Note that their goal is to handle mis-specification related with risk, but still the task is tominimize the expected regret.
Very recently, there are some works trying to understand the behaviour of UCB and TS policies byconsidering the diffusion approximations (see, e.g., [2, 18, 8, 11]).
Although some distributionalcharacterizations of regret are obtained in these works, the asymptotic regime is typically set such thatthe gaps between arm means shrink with the total time horizon.
This makes their results incomparableto ours because we study how the tail probability decays with T under unchanged environments.
Another line of works closely related with ours involve solving risk-averse formulations of the MABproblem (see, e.g., [15, 10, 19, 17, 7, 12]).
Compared to standard MAB problems, the main differencein their works is that arm optimality is defined using formulations other than the expected value, suchas mean-variance criteria and (conditional) value-at-risk.
These formulations consider some singlemetric that is different compared to the expected regret.
From the formulation perspective, our workis different in the sense that we develop policies that on one hand maintain the classical worst-caseoptimal expected regret, whereas simultaneously achieve light-tailed risk bound.
The policy designand analysis in our work are therefore also different from the literature.
Setup and Notation
The setting is described as follows.
Fix a time horizon of T and the number of arms as K .
In eachtime t ∈ [ T ] , the decision maker (DM) needs to decide which arm A t ∈
[ K ] should be pulled.
Tobe more precise, let H t
= { A 1 , X 1 , · · · , A t − 1 , X t − 1 } be the history prior to time t .
When t = 1 ,
H 1 = ∅ .
In time t , the DM adopt a policy π t : H t (cid:55)−→ A t that maps the history H t to an action
A t , where A t follows a discrete probability distribution on [ K ] determined by H t .
The environmentthen independently samples a reward r t,A t = θ
A t + ϵ t,A t and reveals it to the DM.
Here, θ A t is the 114 mean reward of arm A t , and ϵ t,A t is an independent zero-mean noise term.
We assume that ϵ t,A t is 115 σ -subgaussian.
That is, there exists a σ > 0 such that for any time t and arm k ,
[Equation]
Let θ =
( θ 1 , · · · , θ K ) be the mean vector.
Let θ ∗ = max { θ 1 , · · · , θ K } be the optimal mean rewardamong the K arms.
Note that DM does not know θ at the beginning, except that θ ∈ [0 , 1] K .
Theempirical regret of the policy π =
( π 1 , · · · , π T ) under the mean vector θ over a time horizon of T isdefined as
[Equation]
Let ∆ k
= θ ∗ − θ k be the gap between the optimal arm and the k th arm.
Let n t,k be the number oftimes arm k has been pulled up to time t .
That is, n t,k = (cid:80) ts =1 1 { A s = k } .
For simplicity, we willalso use n k = n T,k to denote the total number of times arm k is pulled throughout the whole timehorizon.
We define t k ( n ) as the time period that arm k is pulled for the n th time.
Define the pseudoregret as
[Equation]
and the genuine noise as
[Equation]
Then the empirical regret can also be written as ˆ R πθ ( T ) =
R πθ ( T ) − N π ( T ) .
The following simplelemma gives the mean and the tail probability of the genuine noise N π ( T ) .
Intuitively, it showswhen bounding the mean or the tail probability of the empirical regret, one only need to consider thepseudo regret term.
We will make it more precise when we discuss the proof of main theorems.
Lemma
We have E [ N π ( T )]
= 0 and
[Equation]
Before proceeding, we introduce some other notations.
Throughout the paper, we use O ( · ) ( ˜ O ( · ) )
andΩ( · ) ( ˜Ω( · ) ) to present upper and lower bounds on the growth rate up to constant (logarithmic) factors,and Θ( · ) ( ˜Θ( · ) ) to characterize the rate when the upper and lower bounds match up to constant(logarithmic) factors.
We use o ( · ) to present strictly dominating upper bounds.
In addition, for anya, b ∈ R , a ∧ b = min { a, b } and a ∨ b = max { a, b } .
For any a ∈ R , a + = max { a, 0 } .
2.1 Light-tailed Risk, Instance-dependent Consistency, Worst-case Optimality
Now we discuss several core concepts that we are interested in.1.
Light-tailed risk.
A policy is called light-tailed, if for any constant c > 0 , there exists some β > 0and constant C > 0 such that
[Equation]
Note that here, we allow β and C to be dependent on c .
In brief, a policy has light-tailed risk if theprobability of incurring a linear regret can be bounded by an exponential term of polynomial T :
[Equation]
for some β > 0 .
If a policy is not light-tailed, we say it is heavy-tailed .2.
Instance-dependent consistency.
A policy is called consistent, if for any underlying true meanvector θ , the policy has that
[Equation]
holds for any β > 0 .
We emphasize that here “instant-dependent" means the formula above holds forany fixed environment parameter θ .
In brief, a policy is consistent if the expected regret can never bepolynomially growing in T for any fixed instance.3.
Worst-case optimality.
A policy is said to be worst-case optimal, if for any β > 0 , we have
[Equation]
In brief, a policy is worst-case optimal if the worst-case expected regret can never be growing ina polynomial rate faster than T 1 / 2 .
Note that here we adopt a relaxed definition of optimality, inthe sense that we do not clarify how the regret scale with the number of arms K compared to thatin literature.
The notion of worst-case optimality in this work focuses on the dependence on T .
For example, a policy with worst-case regret O (poly( K ) √ T · poly(ln T )) is also optimal by ourdefinition.
It is well known that for the multi-armed bandit problem, one can design algorithms to achieve bothinstance-dependent consistency and worst-case optimality.
Among them, two types of policies are ofprominent interest: Successive Elimination (SE) and Upper Confidence Bound (UCB).
We list thealgorithm paradigms in Algorithm 1 and 2.
The bonus term rad ( n ) is typically set as
[Equation]
with η > 0 being some tuning parameter.
Algorithm 1 Successive Elimination
Algorithm 2 Upper Confidence Bound t ←
Main Results: Two-armed Bandit
We start from the simple two-armed bandit setting.
The general multi-armed setting is deferred to thenext section.
We first show that standard policies considered widely in the literature are heavy-tailed.
The result reveals a fundamental incompatibility between consistency and light-tailed risk.
Then weshow how to add a simple twist to standard confidence bound based policies to obtain light-tailedpolicies.
Moreover, we show that our design leads to an optimal tail decaying rate.
3.1 Instance-dependent Consistency Causes Heavy-tailed Risk
Theorem 1 If a policy π is instance-dependent consistent, then it can never be light-tailed.
Moreover,if π satisfies
[Equation]
then we have the following stronger argument.
For any c > 0 , there exists C π > 0 such that
[Equation]
Theorem 1 suggests that a consistent policy must have a risk tail heavier than an exponential one.
The proof of Theorem 1 relies on a change of measure argument appeared in [9].
We considerthe environment where the noise ϵ is Gaussian with standard deviation σ .
Intuitively speaking,if we want a policy to be adaptive enough to handle different instances, then the policy will besensitive to risk.
Moreover, if the policy achieves O (ln T ) regret for any fixed instance θ (theconstant is typically dependent on θ ), then the probability of incurring a linear regret becomesexp( − O (ln T ))
= Ω(poly(1 /T )) .
One special case is the family of confidence bound relatedpolicies (SE and UCB) where from Theorem 1 one can see that the standard bonus term (1) willalways lead to a tail polynomially dependent on T .
Another example is the Thompson Sampling (TS)policy.
It has been established that π = TS with Beta prior has the property (2) (see, e.g., Theorem 1and 2 in [1]).
Theorem 1 then suggests that (3) also holds for π = TS.
We need to remark on the difference between Theorem 1 and high-probability bounds in previousliterature.
It has been well-established that UCB or SE withrad
[Equation]
n
[Equation]
n achieves ˜ O
( (cid:112) T · polylog( T/δ )) regret with probability at least 1 − δ
(see, e.g., Section 1.3 in [16],
Section 7.1 in [13]).
Such design also leads to a consistent policy.
However, the bound holds onlyfor fixed δ .
In fact, one can see that the bonus design is dependent on the confidence parameter δ .
Ifδ = exp( − Ω( T β )) with β > 0 , then the scaling speed of the expected regret with respect to T canonly be greater than 1 / 2 , which is sub-optimal.
As a comparison, in our problem, ideally we seek tofind a single policy such that it achieves ˜ O
( (cid:112) T · polylog( T/δ )) regret for any δ > 0 .
Up till now, we make two core observations.
First, from standard MAB results, consistency andoptimality can hold simultaneously.
Second, from Theorem 1, consistency and light-tailed riskare always incompatible.
Then a natural question arises: Can we design a policy that enjoys bothoptimality and light-tailed risk?
If we can, then can we make the tail risk decay with T in an optimalrate?
We answer these two questions with an affirmative “yes” in the next section.
3.2 A New Policy Design Achieving Light-tailed Risk and Worst-case Optimality
In this section, we propose a new policy design that achieves both light-tailed risk and worst-caseoptimality.
The design is simple.
We still use the idea of confidence bounds, but instead of setting thebonus as (1), we set
[Equation]
with η > 0 being a tuning parameter.
Theorem 2 gives performance guarantees for the mean and thetail probability of the empirical regret when π = SE.
Theorem 2 For the two-armed bandit problem, the SE policy with η ≥ 4 and the bonus term being(4) satisfies the following two properties.
sup θ E [ ˆ R πθ ]
= O ( √ T ln T ) .
For any c > 0 and anyα ∈ (1 / 2 , 1] , we have
[Equation]
The first item in Theorem 2 means that with the modified bonus term, the worst case regret isstill bounded by O ( √ T ln T ) , which is the same as the regret bounds for SE and UCB with thestandard bonus term (1).
The second item shows that the tail probability of incurring a Ω( T α )regret ( α > 1 / 2 ) is exponentially decaying in Ω( T α − 1 / 2 ) , and thus the policy is light-tailed.
Thedetailed proof of Theorem 2 is provided in the supplementary material.
An illustrative road-mapof the proof is delegated to Section 4, where we provide the proof idea for a theorem that is a strictgeneralization of Theorem 2.
Here, we give some intuition on the new bonus design.
Our new bonusterm inflates the standard one by a factor of (cid:112) T/n .
This means our policy is more conservative thanthe traditional confidence bound methods, especially at the beginning.
In fact, one can observe thatfor the first Θ( √ T ) time periods, our policy consistently explores between arm 1 and 2 , regardless ofthe environment.
A natural corollary is that our policy is never “consistent”, and this is reasonablefollowing Theorem 1.
However, the bonus term (4) decays at a faster rate on the number of pullingtimes n compared to (1).
This means as the experiment goes on, the policy leans towards exploitation.
We need to stress that this is not the same as the explore-then commit policy, which is well-known toachieve a sub-optimal Θ( T 2 / 3 ) regret (see, e.g., Section 1.2 in [16], Section 6.1 in [13]).
The following theorem shows that the risk tail in Theorem 2 is not improvable.
That is, if the policyπ is worst-case optimal, then for fixed α ∈ (1 / 2 , 1] , the exponent of α − 1 / 2 is tight.
Theorem 3 Let c ∈ (0 , 1 / 2) .
Consider the 2-armed bandit problem with Gaussian noise.
Let π be aworst-case optimal policy.
That is, for any α > 1 / 2 ,
[Equation]
Then for any α ∈ (1 / 2 , 1] ,
[Equation]
holds for any β > α − 1 / 2 .
Theorem 3 also relies on the change of measure argument appeared in Theorem 1.
However, thereare two important differences: we only have worst-case optimality rather than consistency, and theregret threshold cT α is in general not linear in T .
Therefore, we need to take care of constructing thespecific “hard” instance when doing the change of measure.
The detailed proof is delegated to thesupplementary material.
Extensions: Multi-armed Bandit
In this section, we provide an extension of our previous tail probability bound in Theorem 2 from thefollowing three aspects: (a) a general K -armed bandit setting; (b) an analysis for UCB aside from
SE; (c) a detailed characterization of the tail bound for any fixed regret threshold.
Theorem 4 For the K -armed bandit problem, π = SE or π = UCB with [Equation] n )
[Equation]
n satisfy the following two properties.
If η ≥ 4 , then sup θ E [ R π θ ] ≤ 4 K + 4 Kσ √ ηT ln T . 2.
Ifη > 0 , then for any x > 0 , we have
[Equation]
≤
[Equation]
Proof Idea.
We provide a road-map of proving Theorem 4.
The expected regret bound is proved usingstandard techniques.
That is, we define the good event to be such that the mean of each arm alwayslies in the confidence bounds throughout the whole time horizon.
Conditioned on the good event, theregret of each arm is bounded by O ( √ T ln T ) , and thus the total expected regret is O ( K √ T ln T ) .
The proof of the tail bound requires more effort.
Without loss of generality, we assume arm 1 isoptimal.
We first illustrate the proof for π = SE.
Step 1.
We use
[Equation]
The term with the genuine noise can be easily bounded using Lemma 1.
We are left to bound the tailof the pseudo regret.
By a union bound, we observe that
[Equation]
Thus, we reduce bounding the sum of the regret incurred by different arms to bounding that by asingle sub-optimal arm.
Step 2.
For any k ̸ = 1 , we define
S k = { Arm 1 is not eliminated before arm k } .
With a slight abuse of notation, we let n 0 = ⌈ x/ (2 K ∆ k )
⌉ − 1 .
If both n k ∆ k ≥ x/ (2 K ) and S khappen, then arm 1 and k are not eliminated after each of them being pulled n 0 times.
This indicatesˆ µ
[Equation]
The probability of this event can be bounded using concentration of subgaussian variables, whichyields the second term in the tail bound in Theorem 4.
The choice of n 0 is important.
We note that atthis step, even if we replace our new bonus term by the standard one, the bound still holds.
Step 3.
Now consider the situtation when ¯ S k happens.
This means after some phase n , the optimal arm1 is eliminated by some arm k ′ , while arm k is not eliminated.
Note that k = k ′ does not necessarilyhold when K > 2 .
As a consequence, we have the following two events hold simultaneously:
[Equation]
The first event leads to
Mean of some noise terms
[Equation]
The second event leads to
Mean of some noise terms [Equation]
Now comes the trick to deal with an arbitrary n .
We bound the probabilities of the two eventsseparately and take the minimum of the two probabilities ( P ( A ∩ B ) ≤ min { P ( A ) , P ( B ) } ( ∀ A, B ) ).
Then such minimum can be further bounded using the basic inequality ( a + b ) / 2 ≥ √ ab ( ∀ a, b ≥ 0) .
This yields the last term in Theorem 4.
We note that at this step, the standard bonus term (1) does notsuffice to get an exponential bound.
The √ T/n design in our new bonus term plays a crucial role.
We next illustrate the proof for π = UCB.
The proof is in fact simpler.
We use the first step in theproof for π = SE.
For fixed k , we also take the same n 0 =
⌊ x/ (2 K ∆ k ) ⌋ − 1 .
The difference hereis that we do not need to define the event S k .
When arm k is pulled for the ( n 0 + 1) th time, by thedesign of the UCB policy, there exists some n such that
[Equation]
Now comes the trick.
The event is included by a union of two events described as follows:
[Equation]
and
[Equation]
The probability of each of the two events can be bounded using similar techniques adopted whenπ = SE.
In fact, it is implicitly shown in our proof that UCB can yield better constants than SE.
Westill need to emphasize that when bounding the second event, similar to the argument for π = SE, thechoice of our new bonus term is crucial.
Remarks.
For the regret bound, we note that compared to the optimal ˜Θ( √ KT ) bound, we have anadditional √ K term. We should point out that the additional √ K term is not surprising under thebonus term (4). An intuitive explanation is as follows. Compared to the bonus term (1), we widen thebonus term by a factor of (cid:112) T/n .
Among the K − 1 arms, there must exist an arm such that it ispulled no more than T/K times throughout the whole time horizon.
That is, the bonus term of thisarm is always inflated by a factor of at least √ K .
The standard regret bound analysis will, as a result,lead to an additional √ K factor compared to the optimal regret bound ˜Θ( √ KT ) .
For the tail bound, from our proof road-map, one can see that the tail bound in Theorem 4 is alsovalid for the pseudo-regret sup θ P
( R πθ ( T ) ≥ x ) .
In fact, with some simple manipulations on theinequality, if we let
[Equation]
then for any x ≥ 0 , we can get a neat form (we illustrate this point in the supplementary material)
[Equation]
There are two observations:(a)
For any η > 0 , our policy always yields a O ( √ T ) expected regret (although with a constantlarger than that in the first result in Theorem 4).
In fact, notice that for any x >
[Equation]
If we let x = 2 K + CσK (cid:112) ( η ∨ 1 /η )
T ln T with C > 0 being moderately large, then
P ( R πθ ( T ) ≥ x ) · T = O (1) .
As a result, the worst-case regret becomes
[Equation]
This observation shows that the regret order of our policy design is not related to theparameter η , as opposed to the standard UCB policy with (1), where a very small η maypossibly make the UCB policy no longer enjoy a ˜ O ( √ T ) worst-case regret 1 .(b)
If we set η = 1 and
[Equation]
then one can see that for any δ > 0 , with probability at least 1 − δ , the regret of our policyis bounded by
[Equation]
This partially solve the open question in Section 17.1 of [13] (we’ve mentioned it in the introduction) up to a logarithmic factor
Numerical Experiments
In this section, we provide a brief two-armed numerical experiment result and defer the experimentdetails and other multi-armed experiment results to the supplementary material.
We consider a twoarmed-bandit problem with θ =
8) , σ = 1 , T = 500 and Gaussian noise.
We test four policies:
SE and UCB with the classical bonus design described in (1), and SE_new and UCB_new with theproposed new bonus design in (4).
The tuning parameter has 4 choices: η ∈ { 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 } .
Foreach policy and η , we run 5000 simulation paths and for each path we collect the cumulative reward.
We plot the empirical distribution (histogram) for a policy’s cumulative reward in Figure 1.
Indeed,one can observe that for both SE and UCB with (1), there is a significant part of distribution around100 , indicating a significant risk of incurring a linear regret, especially when η is small.
In contrast,with the new design (4), the reward is highly concentrated for every η > 0 with almost no tail risk ofgetting a low total reward.
Particularly, when η = 0 .
1 , UCB_new achieves both high empirical meanand light-tailed distribution.
Conclusion
In this work, we consider the MAB problem with a joint goal of minimizing the worst-case expectedregret and obtaining light-tailed probability bound of the regret distribution.
We characterize theinterplay among three concepts: light-tailed risk, instance-dependent consistency, and worst-caseoptimality.
We demonstrate that light-tailed risk and instance-dependent consistency are incompatible,and show that light-tailed risk and worst-case optimality can co-exist through a simple new policydesign.
We discuss insights and generalizations of our results, and study the empirical performance.
An open question is whether we can improve the regret bound in Theorem 4 to Θ( √ KT ln T ) andget a probability bound of
[Equation]
Another interesting direction is to generalize our policy design to the setting where noises are notsub-Gaussian, but come from general distributions.
In fact, there is a separate line of literaturestudying bandit problems with non sub-Gaussian noise (see, e.g., [6]).
We hope our framework canbe integrated with this line of literature to handle certain types of non sub-Gaussian noise, providedwith controls on the tail behavior of the random noises.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A]
We actually collect rewards and draw the empiricaldistribution.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]