Best of Both Worlds Model Selection
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the problem of model selection in bandit scenarios in the presence ofnested policy classes, with the goal of obtaining simultaneous adversarial andstochastic (“best of both worlds") high-probability regret guarantees.
Our approachrequires that each base learner comes with a candidate regret bound that may ormay not hold, while our meta algorithm plays each base learner according to aschedule that keeps the base learner’s candidate regret bounds balanced until theyare detected to violate their guarantees.
We develop careful mis-speciﬁcation testsspeciﬁcally designed to blend the above model selection criterion with the abilityto leverage the (potentially benign) nature of the environment.
We recover themodel selection guarantees of the CORRAL [3] algorithm for adversarial environments, but with the additional beneﬁt of achieving high probability regret bounds,speciﬁcally in the case of nested adversarial linear bandits.
More importantly, ourmodel selection results also hold simultaneously in stochastic environments undergap assumptions.
These are the ﬁrst theoretical results that achieve best of bothworld (stochastic and adversarial) guarantees while performing model selection in(linear) bandit scenarios.
Introduction
A fundamental challenge in sequential decision-making is the ability of the learning agent to adapt tothe unknown properties of the environment they interact with.
While adversarial environments mayrequire some caution, we would also like to leverage situations where more benign scenarios may bedisclosed as a result of this interaction.
In the literature on multi-armed bandits, this adaptation capabilities has often taken two forms:
Best-of-both worlds guarantees which were pioneered by [10] and subsequently studied by a number of authors [e.g. 29, 5, 28, 30, 33, 21].
Here, the goal is to design algorithms achieving bothstochastic and adversarial environment guarantees simultaneously, without knowing the type ofenvironment in advance.
Similar in spirit is the stream of literature on stochastic rewards withadversarial corruptions [23, 17, 34, 21, 31], where an adversary is assumed to corrupt the stochasticrewards observed by the algorithm, and the regret guarantees are expected to degrade gracefully withthe total amount of corruption, without knowing this amount in advance.
Model selection guarantees which were initiated by [3], and subject since then to intense investigations [e.g. 15, 1, 27, 4, 16, 12, 8, 14, 22, 26, 13].
Here, we assume we have access to a pool of
M base bandit algorithms each operating, say, within a different class of models or under differentassumptions on the environment, and the aim is to design a bandit meta-algorithm that learns tosimulate the best base algorithm in hindsight, without knowing in advance which one will be best forthe environment at hand.
This approach has often been used in the literature to capture bandit modelselection problems.
In fact, a natural instantiation of this framework is when we have a sequence ofnested policy classes, and the goal is to single out the best policy within this nested family, by payingas price only the complexity of the policy class the optimal policy falls into.
In some sense, Item 2 is more general than Item 1, since one may attempt to achieve a best-of-both-world performance bypooling a stochastic bandit algorithm with an adversarial bandit one, and expect the meta-algorithm ontop of them to eventually learn to follow one of the two.
Similarly, in the setting of stochastic rewardswith adversarial corruptions, one can pool together stochastic algorithms operating with increasingguessed levels of corruptions, and let the meta-algorithm learn to single out the one corresponding tothe true corruption level.
In this paper, we combine the two items above into a bandit algorithm that exhibits both modelselection and best-of-both worlds regret guarantees simultaneously.
Our framework encompasses inparticular a well-known linear bandit model selection scenario, where the action set A is a (ﬁnite butlarge) subset of R d M , for some maximal dimension d M > 0 , and we deal with a hierarchy of possibledimensions d 1 < . . .
< d M .
At time t , the learner plays an action a t 2 A and receives as rewardeither r t = a > t !
t , where !
t 2 R d M is an adversarially-generated reward vector (adversarial setting),or r t = a > t !
+ white noise,
where !
2 R d M is a ﬁxed but unknown reward vector (stochastic setting).
Associated with dimensions d 1 < . . .
<
d M are a nested family of policy classes ⇧ 1 ⇢ . . .
⇢ ⇧ M ,and M base algorithms
A 1 , . . . ,
A M , where the i -th algorithm
A
i is a linear bandit algorithm thatworks under the assumption that only the ﬁrst d i components of !
t (or ! ) are nonzero.
Hence
A i operates with dimension d i in that all policies ⇡
⇧ i are probability distributions which areprojections over action set
A i of the set of probability distributions in ⇧ M .
Here, A i is the projectionof the full-dimensional action set A onto its ﬁrst d i dimensions.
Notice that this implies that thepolicy classes are nested: ⇧ 1 ✓ ⇧ 2 . . .
✓ ⇧ M .
If only the ﬁrst d i ?
dimensions ( i ? being unknownto the learner) of each !
t (adversarial setting) or !
(stochastic setting) are nonzero, we design analgorithm whose regret bound scales as poly ( d i ? )
p T in the adversarial case and, simultaneously,as poly ( d M )log T  if the environment happens to be stochastic.
Speciﬁc consideration is given tothe dependence on d i ? .
We show through a lower bounding argument that in the stochastic case aguarantee of the form d M log T  , that is, where d M replaces the more desirable factor d i ? , is inevitable ,if we want to insist on obtaining a log T  -like result.
Thus our bounds reﬂect the best achievable regretrates depending on d M in the stochastic case and on poly ( d i ? ) in the adversarial case.
For our results to hold, some technical conditions are required on the base algorithms, like a notionof (high probability) stability and a notion of action space extendability, formally deﬁned in Section2.
We show that these conditions are fulﬁlled by known algorithms, like an anytime variant of the
GeometricHedge.
P algorithm in [7], and a high-probability variant of the EXP4 algorithm in [6].
Related work.
Among the references we mentioned above, those which are most relevant toour work, as directly related to model selection and best-of-both-worlds guarantees are perhaps[4, 13, 21, 31].
In both [4] and [13], model selection regret guarantees for stochastic contextualbandits are given which take the form poly ( d i ? )
p T .
Yet, no best-of-both world model selectionresults are contained in those papers.
Closer in spirit are [21] and [31].
In [21], the authors considermodel selection problems on top of adversarially corrupted stochastic linear bandit problems, wherethe total level of corruption C is unknown in advance.
It is worth stressing that this is a modelselection problem which is substantially different (and actually easier, see Section 3) than ours, as theselection applies to C instead of the dimensions d i .
In fact, the model selection procedure in [21]looks very different from ours, and can be roughly seen as a robust version of a doubling trick appliedto C .
These results are extended in [31], where the authors also consider RL settings, and moregeneral forms of corruptions than [21].
Like ours, their model selection algorithm follows the idea ofregret balancing and elimination of [1, 13].
On the lower bound side, relevant papers include [32, 24],where the authors investigate the Pareto frontier of model selection for (contextual) stochastic bandits.
It is shown in particular that a regret upper bound of the form p
d i ?
T cannot be achieved.
However,these papers do not explicitly cover gap-dependent regret guarantees.
Problem Setting and Assumptions
We start off by deﬁning the adversarial scenario.
An adversarial contextual bandit problem is arepeated game between a learner A and an environment B .
We consider the general decision-makingscenario where the learner A has at its disposal a class of policies ⇧ A made up of functions ⇡ of the form ⇡ : X !
A , where  A denotes the set of probability distributions over A .
At each round t , 90 the interaction between
A and B is as follows:
Learner A selects a policy ⇡ t 2 ⇧ A2.
Simultaneously, the environment B selects context x t 2 X and reward function r t :A ⇥ X !
[0 , 1] , and reveals x t to A3.
Learner A takes an action a t ⇠ ⇡ t ( · | x t ) and observes reward o t =
r t ( a t , x t ) .
The regret Reg A ( T , ⇧ 0 ) of algorithm A in rounds T ✓ N against a policy class ⇧ 0 is the difference 96 between the learner’s accumulated reward in
T and the performance of the best ﬁxed policy from ⇧ 0 :
[Equation]
`
[Equation]
where E a ⇠ ⇡ [ r ` ( a, x ` )]
=
P a 2 A ⇡ ( a | x ` ) r ` ( a, x ` ) denotes the expectation over the action drawn 98 from the given policy ⇡ which may itself be a random quantity.
For ease of notation, we willomit the comparator policy class when ⇧ 0 =
⇧ A is the policy class of A and replace T by t when
T =
[ t ] := { 1 , . . .
, t } .
Hence, Reg A ( t ) is the regret of A against its own policy class up to round t .
The stochastic scenario we consider is similar to the above, except in the way contexts x t and rewardvalues r t ( a t , x t ) are generated.
Speciﬁcally, B generates contexts x t in an i.i.d.
fashion according toa ﬁxed (but arbitrary and unknown) distribution D over X , while the reward r t ( a t , x t ) is such thatfor all ﬁxed ( a, x ) 2 A ⇥ X , we have E [ r t ( a, x )]
=
r ( a, x ) , for some ﬁxed function r ( · , · ) in someknown class of reward functions.
In this case, we measure performance through pseudo-regret
PseudoReg A
[Equation]
where E x ⇠ D ,a ⇠ ⇡ is the expectation over contexts x and the action a ⇠ ⇡ ( ·| x ) drawn from a policy ⇡ 107 that may itself be a random quantity.
We say an environment B is stochastic with gap  > 0 if thereis a policy ⇡ ?
⇧ such that
[Equation]
A notable example of the above is the following linear bandit scenario.
In the adversarial case(“adversarial linear bandits”)
the action space A is a subset of R d , for some dimension d , the contextx t is irrelevant and, upon playing action a t 2 A , the environment generates a reward which is a linearfunction of the actions, r t ( a t , x t ) =
r t ( a t ) = a > t !
t , where !
t is chosen adversarially at every roundwithin some known class of d -dimensional vectors.
In the stochastic case (“stochastic linear bandits”),the only difference is that we simply have r t ( a t , x t ) =
r t ( a t ) = a > t ! + ✏ t , where ! is a ﬁxed andunknown vector in the class of d -dimensional vectors, and ✏ t is a sub-Gaussian noise.
Moreover, wehave gap  if there is ⇡ ?
⇧ such that E a ⇠ ⇡ ?
[Equation]
.
Many algorithms A in the literature exist that enjoy a regret bound against their policy class ⇧ A ,holding in all environments B satisfying certain conditions.
If this regret bound holds, we say that Ais adapted to this environment.
Here, we allow algorithms to also compete against policy classes ⇧ 0other than ⇧ A , and focus on algorithms that come with regret bounds of the following form.
Deﬁnition 1 (Adapted) .
We call A adapted to environment B and policy class ⇧ 0 if, with probabilityat least 1   , the following bound holds simultaneously for all t 2 N :
[Equation]
The term R ( ⇧ A ) is a (known) measure of complexity of the policy class ⇧ A used by A .
Examples of algorithms satisfying Deﬁnition 1 include a version of the Geometric Hedge algorithmin linear bandits [7] and the EXP4 algorithm [6] in contextual bandits with ﬁnite action sets.
Wewill discuss them in detail below.
Before introducing the model selection questions addressed inthis work, we present two stronger versions of the above deﬁnition, which we call high-probabilitystability and extendability .
These will be useful for model selection among multiple learners.
As we will show with examples later, these stricter conditions can be established for several commonsettings under the same conditions for which adaptivity from Deﬁnition 1 is guaranteed.
Additional conditions.
The ﬁrst condition, high-probability stability (or h-stability ) generalizesadaptivity to the case where A only observes a certain noisy version of the reward.
To state thecondition formally, consider a more general interaction protocol between A and the environment B ,where Step 3 above is replaced by3a.
Let b t ⇠ Bernoulli( ⇢ ) for a ﬁxed and known ⇢ 2 (0 , 1] .
Learner A takes an actiona t ⇠ ⇡ t ( · | x t ) and observes an importance-sampled version of the reward o t = b t r t ( a t ,x t ) ⇢ .
This encompasses the original protocol with ⇢ = 1 as a special case.
An algorithm is h-stable if itmaintains its regret guarantee up to a 1 / p ⇢ penalty in this more general interaction protocol:
Deﬁnition 2 (h-stability) .
An algorithm A is high probability stable (h-stable) in an environment Bagainst policy class ⇧ 0 if it satisﬁes for any constant ⇢ 2 (0 , 1] a regret bound
[Equation]
that holds with probability at least 1   simultaneously for all t 2 N .
Note that A is adapted to B and ⇧ 0 if it is h-stable for ⇢ = 1 .
Compared to the notion of stabilityproposed in [3], the one in Deﬁnition 2 is more demanding, in that it requires the importanceweighted regret bound to hold with high probability, rather than in expectation.
Since we aim forhigh-probability regret bounds in adversarial and stochastic settings, this stronger notion is natural.
The second condition, extendability , generalizes h-stability to environments with additional actions, asspeciﬁed next.
Let B be the original environment with action set A and let ¯ A k
=
A [ { a 0 1 , a 0 2 , . . .
, a 0 k }be the action set extended by k extra special actions a 0 1 , a 0 2 , . .
, a 0 k .
Further, let ¯ ⇧ = { ⇡ : X ! 149  ¯ A k } be an extended version of the original policy set ⇧ = { ⇡ : X !
A } that contains all policies 150 of the original policy set and the single-action policies{ a 0 i } that always choose a certain specialaction a 0 i , i.e., ¯ ⇧ ◆ ⇧ [ { 1 { a 0 1 } , 1 { a 0 2 } , . . .
, 1 { a 0 k }} .
3 We further allow the environment on theextended action space ¯ A k to choose any values for rewards r t ( a 0 i , x t ) , possibly depending on theentire history and all rewards r t ( a, x t ) assigned to other actions a 6 = a 0 i in that round.
We denote theset of all such extended environments by B k ( B , ¯ A k ) .
An algorithm A is now extendable if we canrun it with the extended policy set ¯ ⇧ k
and it competes well against ⇧ 0 and policies 1 { a 0 i } in all suchextended environments.
Deﬁnition 3 (Extendability) .
Consider an algorithm A with policy set ⇧ A ✓ X !
A in 158 environment B .
We call A k -extendable in B against ⇧ 0 if there is an extended policy set¯ ⇧ k ◆ ⇧ A [ { 1 { a 0 1 } , . . .
1 { a 0 k }} such that A equipped with the extended policy set ¯ ⇧ k is h-stableagainst ⇧ 0 [ { 1 { a 0 1 } , . . .
1 { a 0 k }} in all environments B 0 2 B
B , ¯ A k ) that extend B from action 161 space
A to action space ¯
A k
=
A [ { a 0 1 , . .
, a 0 k } .
One relevant example of h-stable and extendable algorithm working in the adversarial linear banditscenario is an anytime variant of the Geometric Hedge algorithm from [7] with exploration ruled by
John’s ellipsoid (e.g., [9]) – see
Appendix B.
Another example is a high-probability variant of the
EXP4 algorithm from [6], which operates with ﬁnite sets of policies.
2.1 Model selection and best-of-both-worlds regret
Our model selection for best-of-both worlds regret guarantees can be described as follows.
We have anested family of policy classes ⇧ 1 ✓ . . .
✓ ⇧ M , with (known) complexities R ( ⇧ 1 )  . . .
 R ( ⇧ M ) .
A meta-learning algorithm M has access to M base algorithms
A 1 , . . . ,
A M , algorithm A
i operatingwith policy class ⇧ i .
These algorithms we sometimes refer to as base learners .
Let i ?
be the smallestindex of the base learner that competes against the largest policy class ⇧ M in the following sense:i ?
=
min { i 2 [ M ]: A i is ( M  i ) -extendable and h-stable in B against ⇧ M } .
[Equation]
=
[Equation] [Equation]
: ^
[Equation]
A running Arbe
, i + 1 , t )
The goal of model selection is to devise a meta-algorithm M that has access to the base learners
A M , and which achieves with probability 1   a regret bound of the form
[Equation]
holding for all t , whenever A M is h-stable and the environment B is adversarial.
Simultaneously, if Bis stochastic with gap  , then we must have
[Equation]
Notice that the above requirement on the pseudo-regret in stochastic environments only requiresa dependence on R ( ⇧ M ) , instead of R ( ⇧ i ? ) .
This is motivated by the fact that in stochasticenvironments with gap  it is generally impossible to obtain model selection guarantees of the form
R ( ⇧ i ? )
log t  – see Appendix A for a proof of this claim.
Adversarial Model Selection Using Regret Balancing
We now introduce our algorithm for model selection in adversarial bandit problems with highprobability regret guarantees.
The algorithm is shown in Algorithm 1, and follows the regretbalancing principle.
This principle has been applied successfully to model selection in bandit and RLproblems with stochastic rewards [13, 26, 1] and with corrupted stochastic rewards [31].
Our work isthe ﬁrst to extend this approach to adversarial rewards.
Regret Balancing.
In each round, we choose the index of a base learner b t by sampling from acategorical distribution with probabilities ⇢ s , ⇢ s +1 , . .
⇢ M that remain ﬁxed throughout the epoch.
The policy of learner A b t is then used to sample the action a t that is passed to the environment.
Afterreceiving the reward r t ( a t , x t ) , Algorithm 1 updates each base learner
A i with r t ( a t , x t ) importanceweighted by the probability that the learner was selected, i.e., 1 { b t = i } r t ( a t ,x t ) ⇢ i .
Thus, we update allbase learners.
This is closer to the Corral algorithm [3] than to the regret balancing approaches forthe stochastic setting, which only update the selected learner A b t .
If the probabilities are set to ⇢ i / 1 R ( ⇧ i ) 2 then after t rounds, each learner A i is followed roughly ⇢ i ttimes.
To see why the regrets of all learners are balanced if they are h -stable against ⇧ M in this case, consider the following.
Denote by reg t ( a ) =
[ E a ⇠ ⇡ ?
[ r t ( a, x t )]  r t ( a, x t )] the regret in round t ofaction a where ⇡ ? 2 argmax ⇡ 2 ⇧ M
P
[Equation]
⇡ [
[Equation]
t ( a, x t )] is the best policy after all rounds T .
Theregret in rounds where A i was selected can be bounded using standard concentration arguments as
T
[Equation]
where the ﬁnal inequality holds because A i is h-stable.
From the deﬁnition of ⇢ i we have R (˜ ⇧ i ) p ⇢ i =
[Equation]
 R (˜ ⇧ s ) , which shows that the total regret in those rounds is bounded by˜ O ( R (˜ ⇧ s ) p T ) .
Thus, if base learners are all h-stable, then the regret incurred by each of them iscomparable to the regret incurred by A s , the learner with the smallest complexity R ( ⇧ s ) .
Eliminating Base Learners.
If a learner A i is not h-stable and may have linear regret, then theprobabilistic schedule above which plays this learner roughly ⇢ i t times yields linear regret.
Wetherefore monitor the performance of each learner and terminate the epoch whenever a learnerperforms signiﬁcantly worse than expected, and thus cannot be h-stable in the environment.
Toidentify such cases, we compare estimates of the rewards of all pairs of base learners as follows.
Foreach learner
A i , ^
CRew i ( t 0 , t ) in Equation 1 is an unbiased estimate of the learners reward sequence
CRew i ( t 0 , t ) =
P t ` = t 0 +1 r
` ( a i ` , x ` ) (see the appendix for details about how these estimates arecomputed).
Further, using conﬁdence bounds D i ( t  t 0 ) from Equation 1, we can constructs aconﬁdence interval for CRew i ( t 0 , t ) as
[Equation]
If the conﬁdence intervals for two learners with indices i < j are more than the h-stable regretbound of i apart, see Equation 2, then deem all learners with index up to i not h-stable and restart thealgorithm with a reduced set of base learners.
This kind of elimination condition has already been used in stochastic environments [27, 13, 31],but it requires substantially more care in an adversarial setting.
In settings with stochastic rewards(even with corruption, e.g., [31]), no learner can achieve rewards that are signiﬁcantly higher than theoptimal policy.
In contrast, in the adversarial setting, it is possible to have negative regret against anyﬁxed policy.
Thus, if we were to use the elimination condition in Equation 2 with our base learners asis, then we may eliminate an h-stable base learner i when another base learner j has negative regretagainst the best ﬁxed policy ⇡ ?
t .
This could in turn lead to undesirable regret in subsequent epochswhen only learners with R ( ⇧ i )  R
( ⇧ i ? ) are left.
Linking Base Learner Performances.
To address this issue, we link the performance of baselearners.
Instead of instantiating each learner
A i with its original policy set ⇧ i , we apply it to anextended problem with M  i additional actions ˜ a
i +1 , ˜
a i +2 , . .
˜ a M , and an extended policy set ˜ ⇧ ithat includes all original policies, along with policies that only choose one of the additional actions ˜ a i ,that is ˜ ⇧ i ◆ ⇧ i [ { 1 { ˜ a
i +1 } , . . .
1 { ˜ a M }} .
Whenever a base learner A i chooses one of the additional 227 actions ˜ a j , then the action proposed by A j is followed.
Essentially, running each base learner withsuch an extended action set allows it to choose to follow the actions proposed by any learner withhigher index in the hierarchy.
The beneﬁt of linking base learners this way is that each base learnernow not only competes against the best ﬁxed policy in their set, but also against all learners abovethem in the hierarchy.
Thus, if A i is h-stable and extendable (see Deﬁnition 3), then it satisﬁes aregret bound of the form
[Equation]
against any learner A j with j > i .
As a result, since the LHS of Equation 3 is approximately^ CRew j ( t 0 , t )  ^
CRew i ( t 0 , t ) ±
D i ( t 0 , t ) ±
D j ( t 0 , t ) , the test in Equation 2 cannot trigger for such
A i , and only base learners that are not h-stable or extendable can be eliminated.
In Appendix C, we show that our regret balancing algorithm achieves the following regret bound:
Theorem 4.
Consider a run of Algorithm 1 with Arbe (  , 1 , 0) and M base algorithms with 1 
R ( e ⇧ 1 )  · · ·  R ( e ⇧ M ) where
˜ ⇧ i is the extended version of policy class ⇧ i with ( M  i ) additional actions.
Then with probability at least 1  poly( M )  the regret Reg( t, ⇧ M ) for all rounds t  i ?
isbounded by
[Equation]
where i ? is the smallest index of the base algorithm that is h -stable.
In most settings, the complexity R ( e ⇧ i ? )
R ( e ⇧ 1 )  p M and the ﬁrst term dominates.
This regret recoversthe expected regret guarantees of Corral [2] when used with learning rates that do not requireknowledge of i ?
but as stronger high-probability bounds.
To the best of our knowledge, Theorem 4is the ﬁrst high-probability regret bound for adversarial model selection.
To illustrate our result,consider to common problem of model selection with nested model classes of dimensions d i =
i fori 2 [ M ] , discussed in the introduction.
We can use GeometricHedge.
P as base learners which areh-stable and extendable if adapted (see Appendix B) and the complexity of extended policy classes
R ( e ⇧ i ) =
R ( ⇧ i ) + M  i  d i + M is not much larger than those of the original policy classes.
Arbe with such base learners achieves a regret bound of order ˜
O
( d 2 i ?
p t ) up to factors of M andlog -factors which are typically small.
Adversarial Model Selection with Best of Both Worlds Guarantees
In this section, we present our algorithm for adversarial model selection that preserves a logarithmicregret guarantee in case it interacts with a stochastic environment with gap  > 0 .
In order to achievesuch a best-of-both-worlds guarantee, we will combine our adversarial regret balancing techniquewith the algorithmic strategy of Bubeck and Slivkins [11].
The main result for our algorithm is:
Theorem 5.
Consider a run of Algorithm 2 with inputs t 0 = 0 , arbitrary policy policy
[Equation]
258 and M base learners
A 1 , . . . ,
A M .
Then with probability at least 1  poly ( M )  , the following twoconditions hold for all t  M 2 simultaneously.
In any adversarial or stochastic environment B , theregret is bounded as
[Equation]
If B is stochastic and there is a unique policy with gap  > 0 , then the pseudo-regret is bounded as,
[Equation]
The algorithm has the same regret bound as Arbe up to a p ln t factor.
However, in addition, italso maintains poly-logarithmic pseudo-regret if the environment is stochastic and the best policyexhibits a positive gap.
Our pseudo-regret bound depends polynomially on R ( ⇧ M ) in contrast to the
R ( ⇧ i ? )
dependency for the adversarial regret rate.
This may seem undesirable but, as mentioned in
Section 2.1, model selection in stochastic environments with poly ( R ( ⇧ i ? ) , ln( R ( ⇧ M ))
ln( t )  regret isimpossible (see Appendix A), let alone while maintaining a best-of-both-worlds guarantee with p Tregret in adversarial environments.
Hence, our algorithm achieves the best kind of guarantee we canhope for, up to improvements in the order of polynomial dependencies.
Our algorithm proceeds in two distinct phases.
The ﬁrst, Arbe-Gap shown in Algorithm 2 is designedto identify a suitable candidate for the optimal policy and to estimate its gap.
If such a policy thatperforms signiﬁcantly better than any other policy emerges, the algorithm enters the second phase,
Arbe-GapExploit which hones in on this policy by playing it most of the time while monitoring itsregret in case the environment turns out to be adversarial after all (in which case we simply run Arbe ).
We now describe both phases.
4.1 First Phase: Candidate Policy Identiﬁcation and Gap Estimation
This goal of this phase is to always maintain the desired adversarial model selection regret guaranteeand simultaneously identify the gap between the best policy and the rest if one exists.
To achieve
1 Input: failure probability t 0 , focus policy b n , learners
i
Mi = s 2 Set ⇧ M +
=
⇧ M \{ b ⇡ } and A M +1 as a copy of A M with policy class ⇧ M + 3 Initialize base learners
A s , A s +1 , . . .
, A M +1 with extended policy classes ˜ ⇧ s , ˜ ⇧ s +1 , . . .
, ˜ ⇧ M + 4 Set sampling probabilities ⇢ i =
R (˜ ⇧ i )
P
M +1 j = s R
(˜ ⇧ j )  2 for all
i 2 { s, . . .
M + 1 } , and ⇢ i
=
for i < s 5 for round t = t 0 + 1 , t 0 + 2 , . . .
do
Sample base learner index t
Get t and compute a it ⇠ ⇡
·| x t
, the base learner A
i proposes for x t
Play t
} r t a t ,x t ) ⇢ i
if Equation 2 holds between A i , A j for s  i < j  M + 1 then restart algorithm by running Arbe-Gap (  , t, b ⇡ , n + 1 , ( A i +1 , · · · , A M )) //
Gap Test: A M better than A M +1 ?
Set W ( t 0 , t ) = ⇥ ✓q
[Equation]
Set b
[Figure]
[Equation]
[Figure]
+1 ( t 0 ,t ) t  t W ( t 0 , t ) if 2 W ( t 0 ,
t )
 b  t  R ( e ⇧ M ) 2 then
Run Arbe-GapExploit (Appendix D) with inputs  , t
=
t, A M , b ⇡ and b  =
b  t
Run Arbe with inputs t 0 , s,  // New Candidate Policy Test if a policy ⇡ 2 ⇧ M \ { b ⇡ } has been selected in more than 3 t 4 of all t  9 rounds then restart algorithm by running Arbe-Gap (  , t, ⇡ , n + 1 , ( A i )
Mi = s ) the ﬁrst goal, we employ the regret balancing and elimination technique of Arbe , see Lines 3–11 of
Algorithm 2 which are virtually identical to Algorithm 1.
For the second goal, determining the gap, Arbe-Gap maintains a candidate b ⇡ 2 ⇧ M for the optimal 282 policy and estimates its gap as follows.
The learner hierarchy A 1 , . . .
, A M is augmented at the topwith an additional learner
A M +1 operating on the policy class ⇧ M +1 =
⇧ M \ { b ⇡ } .
Since A M +1 284 is identical to A M except that it does not have access to b ⇡ , we can obtain a gap estimate for b ⇡ bymonitoring the difference in reward estimates ^
CRew M +
and ^ CRew M of A M +1 and A M .
In fact,b  t in Line 13 is a lower conﬁdence bound on the difference between the best policies in b ⇧ M andb ⇧ M \ { b ⇡ } and thus, the gap of b ⇡ .
We test at every round whether b  t exceeds its conﬁdence width2 W ( t 0 , t ) .
If this test triggers then b ⇡ must have a positive gap  of order W ( t 0 , t ) and further, b  t mustbe a multiplicative estimate of  , that is,
[Equation]
The latter holds, because b  t + 2 W ( t 0 , t )is an upper-conﬁdence bound on  and the test condition impliesb
t  b  t + 2 W
( t 0 , t )   .
Sincein this case, we have determined that b ⇡ is optimal with a gap of order b  t , we can move on to thesecond phase Arbe-GapExploit discussed later.
Assume the candidate policy b ⇡ is indeed optimal and exhibits a positive gap  .
Since b  t concentratesaround  at a rate of W ( t 0 , t ) ⇡ poly ( R M ( e ⇧ M ))
p t  t, the condition of the gap test in Line 14 musttrigger after at most t  t 0 .
poly ( R M ( e ⇧ M ))
rounds.
Finally, since Arbe-Gap always maintains thepoly ( R ( e ⇧ i ? )
p t  t 0 adversarial regret rate, the total pseudo-regret incurred until the test triggers isof order poly ( R M ( e ⇧ M ))
, leading to the second term in the bound in Equation 5.
With the techniques above, we can reliably detect a positive gap if the candidate policy b ⇡ exhibitsone.
It remains to identify a suitable candidate b ⇡ of the optimal policy ⇡ ?
in stochastic environments.
To do so, we use the following two observations: Arbe-Gap maintains a poly ( R ( e ⇧ i ? )
p t adversarialregret rate overall and each policy but ⇡ ? will incur on average at least a regret of  per round.
Hence, in order to maintain that regret, Arbe-Gap must select ⇡ ?
in the majority of all rounds whent = !
( poly ( R ( e ⇧ i ? )) ) .
Otherwise the regret grows as ⌦ ( t  ) = !
( poly ( R ( e ⇧ i ? ))
p t ) violating theadversarial rate.
To leverage this observation and identify a suitable candidate policy b ⇡ , Line 17 of
Algorithm 2 always checks whether there is a policy other than the current candidate policy thathas been selected in at least 3 / 4 of all rounds.
6 If so, the algorithm is restarted with this policy ascandidate b ⇡ .
One can show that there are at most O (ln t ) restarts due to candidate policy switches,only increasing the adversarial regret rate of Arbe-Gap by a factor of O ( p ln t ) compared to Arbe ’s.
Our candidate policy selection approach is similar to that by Wei et al.
[31] for the corrupted rewardsetting
but they require each adapted base learner to achieve a logarithmic regret rate in the ﬁrst place.
Instead, our approach only requires an adversarial regret rate from the base learner.
4.2 Second Phase: Exploitation
Since each base learner only needs to satisfy a p T regret rate even in stochastic environments, wegenerally cannot hope to recover logarithmic pseudo-regret by only selecting among them.
Forlogarithmic pseudo-regret, we need to ensure that the given policy b ⇡ with gap estimate
b  is playedsufﬁciently often.
However, we also need to monitor its regret against all other policies in case theenvironments turns out to be adversarial.
If at any point, b ⇡ fails to maintain a gap of order
b  , we canconclude that the environment is adversarial.
Then Arbe-GapExploit returns and we simply play aninstance of Arbe (Line 16 of Algorithm 2).
We will present a brief summary of the main intuition behind our exploitation phase approach hereand defer a longer discussion and the detailed pseudo-code to Appendix D. In each round, weplay policy b ⇡ with probability approximately 1  poly ( R ( ⇧ M ))
b  2 t , and with the remaining probabilitypoly ( R ( ⇧ M ))
b  2 t a version of base learner A M with policy class ⇧ M \ { b ⇡ } .
If the environment is indeed stochastic, then b ⇡ = ⇡ ? does not incur any pseudo-regret and the totalregret in other t 0 ⇡ t · poly ( R ( ⇧ M ))
b  2 t ⇡ poly ( R ( ⇧ M ))
b rounds can be bounded as t 0 ·  +Reg A M ( t 0 , ⇧ M \{ b ⇡ } ) ⇡ poly ( R ( ⇧ M ))  ln( t ) .
The ﬁrst term is the regret of the best policy in ⇧ M \ { b ⇡ } and the second 327 term is the regret of A M against that policy.
Since A M is h-stable on ⇧ \ { b ⇡ } , its regret against thatpolicy is at most poly ( R ( ⇧ M ))
p t
and we get the desired pseudo-regret.
To ensure good regret in the adversarial case, we need to detect quickly enough when b ⇡ does notexhibit a performance gap anymore and fall back to a fully adversarial algorithm.
Similar to b  t inthe ﬁrst phase, we use a lower conﬁdence bound on the average performance gap and continuouslytest whether it falls below b  2 .
This simple approach would give p t adversarial regret but may exhibita poly ( R ( ⇧ M )) dependency.
Fortunately, we can avoid this dependency and retain the desired modelselection regret rates by extending the intuition above to also test an upper bound on the gap.
Fordetails, see Appendix D.
Conclusions
We have described and analyzed a novel model selection scheme for bandit algorithms that beneﬁtsfrom best-of-both-worlds high probability regret guarantees.
Our machinery can be speciﬁcallyapplied to adversarial/stochastic linear bandit tasks, where model selection is performed on theunknown dimensionality of the linear reward function.
This has required extending the regretbalancing technique of model selection from stochastic to adversarial rewards and a very carefulhandling of the associated mis-speciﬁcation tests.
The base learners aggregated by our meta-algorithmhave to satisfy an anytime high probability regret guarantee in the adversarial case, along with regretstability and action space extendability properties which we have shown are satisﬁed by (variants of)known algorithm in the bandits literature.
Our best-of-both world model selection regret guarantees cannot in general be improved, speciﬁcallyin stochastic environments with gaps, where it is generally impossible to obtain log t -like modelselection bounds that only depend on the complexity of ⇧ i ? .
On the other hand, it would be nice tosee in Theorem 5 a better polynomial dependence on M and R ( ⇧ M ) . Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our workis of purely theoretical nature, and we are not aware of any potential negative societal
impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See in particular
the deﬁnitions and desiderata in Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] All proofs are provided
in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No](d) Did you discuss whether
and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]