Better Best of Both Worlds Bounds for Bandits with Switching Costs
Abstract
We study best-of-both-worlds algorithms for bandits with switching cost, recently addressed by Rouyer, Seldin, and Cesa-Bianchi [14].
We introduce a surprisingly simple and effective algorithm that simultaneously achieves minimax optimal regret bound (up to logarithmic factors) of O ( 𝑇 2 / 3 ) in the oblivious adversarial setting and a bound of O ( min { log ( 𝑇 )/ ∆ 2 ,𝑇 2 / 3 }) in the stochastically-constrained regime, both with (unit) switching costs, where ∆ is the gap between the arms.
In the stochastically constrained case, our bound improves over previous results due to [14], that achieved regret of O ( 𝑇 1 / 3 / ∆ ) .
We accompany our results with a lower bound showing that, in general, ˜ Ω ( min { 1 / ∆ 2 ,𝑇 2 / 3 }) switching cost regret is unavoidable in the stochastically-constrained case for algorithms with O ( 𝑇 2 / 3 ) worst-case switching cost regret.
Introduction
Multi Armed Bandit (MAB) is one of the most fundamental problems in online learning and sequential decision making.
This problem is often framed as a sequential game between a player and an environment played over 𝑇 rounds.
In each round, the player chooses an action from a finite set [ 𝐾 ] = { 1 , . . .
, 𝐾 } and incurs a loss in [ 0 , 1 ] for that action.
The environment then only reveals the loss of the chosen action—this is referred to as bandit feedback.
The goal of the player is to minimize the regret , which measures the difference between the cumulative loss of the player and that of the best arm in hindsight
common regimes often studied in the MAB literature are the adversarial setting [6] and the socalled stochastically-constrained setting [18] which is a generalization of the more classical stochastic setting.
In the former regime, losses are generated arbitrarily and possibly by an adversary; in the latter, losses are assumed to be generated in a way that one arm performs better (in expectation) than any other arm, by a margin of ∆ > 0.
Both regimes have witnessed a flurry of research [1, 3, 5, 6, 12, 17] leading to optimal regret bounds in each of the settings.
Recently, significant effort has been dedicated for designing best-of-both-worlds MAB algorithms, where one does not have a-priori knowledge on the underlying environment but still wishes to enjoy the optimal regret in both regimes simultaneously [4, 7, 15, 16, 18, 19].
Most notably, Zimmert and
Seldin [19] analyzed the Tsallis-INF algorithm and established that it achieves optimal regret bounds in both stochastically-constrained and adversarial environments, matching the corresponding lower bounds asymptotically.
Another well-studied variant of the MAB setting is that of Bandits with switching cost [2, 8–10], where the learner suffers not only regret but also a penalty for switching actions.
As shown by
Dekel et al.
[8] adding a unit switching cost to the regret incurs a lower bound of ˜ Ω (cid:0) 𝐾 1 / 3 𝑇 2 / 3 (cid:1) , in contrast to Ω (√ 𝐾𝑇 ) in the standard setting, highlighting the difficulty of this setup.
Recently Rouyer et al.
[14] asked the question of how best-of-both-worlds algorithms can be obtained for MAB where switching costs are considered.
Rouyer et al.
[14] managed to show a best-of-both-worlds type algorithm that, for constant 𝐾 and cost-per-switch, achieves optimal regret bound of O ( 𝑇 2 / 3 ) in the oblivious adversarial regime, whereas in the stochastically-constrained setup their upper bound is of order
O ( 𝑇 1 / 3 / ∆ ) , which was unknown to be optimal.
In this work we tighten the above gap.
we introduce a new algorithm that improves the bound of [14] and achieves O ( min { log 𝑇 / ∆ 2 ,𝑇 2 / 3 }) in the stochastically constrained case (while obtaining the same optimal regret bound in the worst-case).
Further, we provide a lower regret bound, and show that the above bound is tight, up to logarithmic factors, in the best-of-both-worlds setup.
For the more general case of 𝐾 > 2 arms, our algorithm still improves over [14], however in that case our lower bounds and upper bound do not fully coincide; we leave this as an open question for future study.
Setup and Background
In the classic Multi Armed Bandit problem with 𝐾 arms, a game is played consecutively over 𝑇 >
𝐾 rounds.
At each round 𝑡 ≤ 𝑇 , an adversary (also called the environment) generates a loss vector ℓ 𝑡 ∈ [ 0 , 1 ] 𝐾 .
The player (referred to as learner) selects an arm 𝐼 𝑡 ∈
[ 𝐾 ] according to some distribution 𝑝 𝑡 ∈ ∆ 𝐾 where ∆ 𝐾 (cid:66) { 𝑝 ∈ [ 0 , 1 ] 𝐾 : (cid:80) 𝑖 ∈
[ 𝐾 ] 𝑝 𝑖 = 1 } , and observes ℓ 𝑡,𝐼 𝑡 , which is also defined to be its loss at round 𝑡 .
Notice that the learner never has access to the entire loss vector ℓ 𝑡 ∈ [ 0 , 1 ] 𝐾 .
The performance of the learner is measured in terms of the regret .
The regret of the learner is defined as
[Equation]
Another common performance measure we care about is the pseudo-regret of the algorithm:
[Equation]
We next describe two common variants of the problem, which differ in the way the losses are generated.
Adversarial (oblivious) regime:
In the oblivious adversarial regime, at the beginning of the game the environment chooses the loss vectors ℓ 1 , . . . , ℓ 𝑇 , and they may be entirely arbitrary.
In general, the objective of the learner is to minimize its expected regret for the adversarial regime.
One can observe that the expected pseudo-regret coincides with the expected regret , in this setting.
More generally, it can be seen that the expected regret upper bounds the expected pseudo-regret.
Namely, 𝔼 [ R 𝑇 ] ≤ 𝔼 [ R 𝑇 ] .
Stochastically-constrained adversarial regime:
We also consider the stochastically-constrained adversarial regime [18].
In this setting we assume that the loss vectors are drawn from distributions such that there exists some 𝑖 ★ ∈
[ 𝐾 ] ,
[Equation]
independently of 𝑡 1 .
That is, the gap between arms remains constant throughout the game, while the losses { ℓ 𝑡,𝑖 } 𝑡 ∈[ 𝑇 ] of any arm 𝑖 are drawn from distributions that are allowed to change over time and may depend on the learner’s past actions 𝐼 1 , . . .
𝐼 𝑡 − 1 .
It is well known that the stochasticallyconstrained adversarial regime generalizes the well-studied stochastic regime that assumes the losses are generated in an i.i.d. manner.
We denote the best arm at round 𝑡 to be 𝑖 ★𝑡 = arg min 𝑖 ∈
[ 𝐾 ] 𝔼 [ ℓ 𝑡,𝑖 ] .
Note that since the gap between arms is constant we have that ∀ 𝑡 ∈
[ 𝑇 ] : 𝑖 ★𝑡 = 𝑖 ★ where 𝑖 ★ is the optimal arm.
We consider the case where there is a unique best arm.
Also, we denote the gap between arm 𝑖 and 𝑖 ★ to be ∆ 𝑖 (cid:66) 𝔼 [ ℓ 𝑡,𝑖 − ℓ 𝑡,𝑖 ★ ] and we let ∆ min =
min 𝑖 ≠ 𝑖 ★ ∆ 𝑖 .
We note that in the stochastically-constrained case, the pseudo-regret is often expressed by the sub-optimality gaps ∆ 𝑖 , and it is given by:
[Equation]
2.1 Multi-Armed Bandits with Switching Cost
In the problem described above, there is no limitation on the number of times the player is allowed to switch arms between consecutive rounds.
In this work, we consider a setup where the regret is accompanied by a switching cost, as suggested by Arora et al.
We then measure our performance by the switching cost regret, parameterized by the switching cost parameter λ ≥ 0 :
[Equation]
where S 𝑇 (cid:66) (cid:80) 𝑡 ∈
[ 𝑇 ] 1 { 𝐼 𝑡 ≠ 𝐼 𝑡 − 1 } .
Best-of-both-worlds with switching cost.
Rouyer et al.
[14] considered the setting of switching cost in the framework of best-of-both-worlds analysis.
They showed (Thm 1 therein): that there exists an algorithm, Tsallis-Switch, for which in the adversarial regime the pseudo-regret of Tsallis-Switch for λ ∈ Ω ( √︁ 𝐾 / 𝑇 ) :
[Equation]
and in the stochastically constrained setting:
[Equation]
Main results
Our main result improves over the work of Rouyer et al.
[14] and provides an improved best-of-bothworlds algorithm for the setting of switching cost
Theorem 1.
Provided that λ ≥ √︁ 𝐾 / 𝑇 , the expected pseudo-regret with switching cost of “Switch
Tsallis, Switch!”
(Algorithm 1) satisfies the following simultaneously:
• In the adversarial regime, 𝔼
[Equation]
• In the stochastically constrained adversarial regime
[Equation]
We next compare the bound of “Switch Tsallis, Switch!”
and Rouyer et al.
We first observe that for small switching cost, λ ≤ 𝑂 (cid:0)√︁ 𝐾 / 𝑇 (cid:1) , both algorithms basically ignore the switching cost and run standard Tsallis without any type of change hence the algorithms actually coincide, so we only care for the case λ ≥ √︃ 𝐾𝑇 .
Also, notice that in the adversarial regime Eq.
(2) and Eq.
(4) are equivalent and both algorithms obtain the minimax optimal regret (up to logarithmic factors).
In the stochastically constrained regime comparing Eqs.
(3) and (5), note that for ∆ min ≤ ( λ 𝐾 ) 1 / 3 𝑇 − 1 / 3 log 𝑇 , we have that
[Equation]
which is comparable to our bound up to logarithmic factors.
On the other hand, if ∆ min ≥ ( λ 𝐾 ) 1 /
[Equation] [Equation]
log
[Equation]
which is comparable to Eq.
It can also be observed that when ∆ min is large enough (larger than say 𝑇 − 1 / 3 log 𝑇 ) our bound improves over Eq.
(3) by a factor of ˜ O
( 𝑇 1 / 3 ∆ min ) .
Next, we describe a lower bound, which demonstrates that our bounds are tight for 𝐾
= 2 (up to logarithmic factors).
Theorem 2.
Let 𝐴 be a randomized player in a multi armed bandit game of 𝐾 arms played over 𝑇 rounds with a switching cost regret guarantee of O
( 𝐾 1 / 3 𝑇 2 / 3 ) in the adversarial regime.
Then, for every ∆ > 0 there exists a stochastically-constrained sequence of losses ℓ 1 , . . . , ℓ 𝑇 with minimal gap parameter ∆ , that 𝐴 incurs R 𝑇 + S 𝑇 =
˜ Ω (cid:0) min (cid:8) 1 / ∆ 2 , 𝐾 1 / 3 𝑇 2 / 3 (cid:9)(cid:1)
[Equation]
Algorithm
Our algorithm, “Switch Tsallis, Switch!”
(see Algorithm 1), is a simple modification of Tsallis-INF.
We start by playing the original Tsallis-INF algorithm introduced by [19], and after a certain amount of switches we switch to a second phase that plays a standard block no-regret algorithm.
[Equation]
The idea is motivated by our observation that under the stochastically constrained setting, there is a natural bound on the number of switches which is of order 𝑂 ( R 𝑇 / ∆ min ) , so as long as this number doesn’t exceed the worst case bound of the adversarial setting we have no reason to perform blocks.
In other words, we start by playing under the assumption that we are in the stochastically-constrained regime and if the number of switches is larger than expected, we break and move to an algorithm that handles only the oblivious adversarial case.
Best-of-both-worlds algorithms that start under stochasticity assumption and break are natural to consider.
Indeed, in the standard setting, without switching they were studied and suggested [4, 7].
However, while successful at the stochastic case, they suffer from a logarithmic factor in the adversarial regime.
Moreover, in the standard best-of-both-worlds setup (without switching cost), the optimal methods don’t attempt to identify the regime (stochastic or adversarial).
In contrast, what we observe here, is that once switching cost is involved, the criteria to shift between the regimes becomes quite straightforward which allows us to design such a simple algorithm.
Indeed, unlike regret which is hard to estimate, the cost of switching is apparent to the learner, hence we can verify when our switching loss exceeds what we expect in the adversarial regime and decide to switch to enjoy both worlds.
Proofs
Before delving into the proof, we provide a brief review of the Tsalis-INF algorithm guarantees , introduced by Zimmert and Seldin [19], which serves as the backbone of Algorithm 1.
5.1 Technical Preliminaries
Theorem 3 ([19, Thm 1]) .
The expected pseudo-regret of Tsallis-INF under the adversarial regime satisfies:
[Equation]
and in the stochastically constrained setting:
[Equation]
[Equation] [Equation]
(cid:41)
Arora et al.
[2] showed how, given a no-regret algorithm against an oblivious adversary, one can convert the algorithm to be played over mini-batches of size
𝐵 and obtain a regret of 𝐵 𝔼 (cid:2) R 𝑇 / 𝐵 (cid:3) .
In turn, the regret with switching cost is bounded by 𝔼 (cid:2) R λ 𝑇 (cid:3) ≤ 𝐵 𝔼 (cid:2) R 𝑇 / 𝐵 (cid:3) + λ 𝑇𝐵 .
Applying their approach to build a mini-batched version of Tsallis-INF leads to Algorithm 2 which we depict above, yielding a similar algorithm to a one suggested by Rouyer et al.
Using the above bound for the special case of Tsallis-INF, we obtain the following guarantee:
Corollary 4.
There exists an algorithm ,in particular - Algorithm 2 with constant blocks of size 𝐵
[Equation]
, for which the expected regret with switching cost, satisfies
[Equation]
Indeed, using the above observation when 𝑇 / 𝐵 may not be a natural number,
[Equation]
5.2 Proof of Theorem
“Switch Tsallis, Switch!” consists of two parts.
We will denote the regret and pseudo-regret attained after the first part by R ( 1 ) and R ( 1 ) respectively, while the regret and pseudo-regret achieved in
Line 9 are denoted by R ( 2 ) and R ( 2 ) .
Similarly, S ( 1 ) and S ( 2 ) express the number of switches of each segment.
The proof of the adversarial case, i
[Equation]
e. Eq.
(4) is straightforward and follows by explicitly bounding the regret in each of these phases:
[Equation]
Corollary
[Equation]
We thus continue to the stochastically constrained case.
For the proof we rely on the following two
Lemmas:
Lemma 5.
For every loss sequence, Algorithm 1 satisfies:
[Equation]
Proof.
Consider the switching cost regret of Line 9 (Algorithm 2) conditioned on S ( 1 ) ,
[Equation]
Corollary
[Equation]
Where we used that by the general regret definition we have that for any algorithm 𝔼 (cid:2) R 𝑇 (cid:3) ≤ 𝔼 (cid:2) R 𝑇 (cid:3) .
Taking expectation on both sides of Eq.
(8) concludes the proof.
■
Lemma 6.
Suppose we run Algorithm 1 against a stochastically constrained loss sequence with gap ∆ min , then:
[Equation]
Proof.
Consider some arbitrary arm 𝑖 ∈
[ 𝐾 ] .
When a switch occurs at round 𝑡 , either 𝐼 𝑡 − 1 or 𝐼 𝑡 are different from 𝑖 ★ .
Hence, using linearity of expectation one can bound the expected number of switches 𝔼 [ S ] , regardless to the environment regime (either adversarial or stochastically-constrained adversarial) as follows:
[Equation]
where we have used the fact that ℙ ( 𝐼 𝑡 = 𝑖 )
=
𝔼 [ 𝑝 𝑡,𝑖 ] .
Additionally, in the stochastically-constrained regime we also have that:
[Equation]
Utilizing the stopping criterion of “Switch Tsallis, Switch!,” we can bound the expected number of switches 𝔼 [ S ( 1 ) ] , regardless to the environment regime (either adversarial or stochasticallyconstrained adversarial) and obtain the desired result ■
The total expected pseudo-regret is upper bounded by the summation of the pseudo-regret attained for each part of the algorithm.
We will consider two cases.
[Equation] [Equation]
5.3 Proof of Theorem
Our lower bound builds upon the work of Dekel et al.
[8] and we adapt it to the stochasticallyconstrained adversarial regime.
Dekel et al.
[8] suggested the following process, depicted in Algorithm 3, to generate an adversarial loss sequence.
With correct choice of parameter ∆ = 𝑂 ( 𝑇 − 1 / 3 ) , the process ensures for any deterministic player, a regret of order
𝔼 [ R 𝑇 ]
=
˜ Ω
( 𝐾 1 / 3 𝑇 2 / 3 ) .
For our purposes we need to take care of two things: First we need to generalize the bound to arbitrary ∆ .
Second, one can see that the loss sequence generated by the adversary in Algorithm 3, is not stochastically constrained (as defined in Section 2).
To cope with this, we develop a more fine-grained analysis over a modified loss sequence that assures the stochastically-constrained assumption is met.
Towards proving Theorem 2, we present the following Lemmas.
Lemma 7.
Let (cid:8) ℓ 1 , . . . , ℓ 𝑇 (cid:9) be the stochastic sequence of loss functions defined in Algorithm 3 for 𝐾 = 2 .
Then for 𝑇 ≥ 4 and any deterministic player against this sequence it holds
[Equation]
Lemma 8.
Let (cid:8) ℓ 1 , . . . , ℓ 𝑇 (cid:9) be the stochastic sequence of loss functions defined in Algorithm 3 with ∆ ≤ 𝑎𝐾 1 / 3 𝑇 − 1 / 3 log − 9 / 2𝑇 .
Then for 𝑇 ≥ τ and any deterministic player against this sequence, with a switching cost regret guarantee of O
( 𝐾 1 / 3 𝑇 2 / 3 ) , against an arbitrary sequence, it holds
[Equation]
for some universal constants 𝑎, 𝑐, τ > 0 .
We deter the proofs of Lemmas 7 and 8 to the supplementary material, and we now proceed with proving our desired lower bound in Theorem 2.
Proof of Theorem 2.
It can be observed, that the process depicted in Algorithm 3 is almost stochastically constrained, in fact, if at 𝑡 and 𝑖 we do not perform clipping to [ 0 , 1 ] , i.e. ℓ 𝑡 ( 𝑖 ) =
ℓ ′ 𝑡 ( 𝑖 ) the sequence is indeed stochastically constrained.
Let us define then an event 𝐻 as follows,
[Equation]
If we restrict ∆ ≤ 16 , the event 𝐻 implies that ℓ 𝑡 = ℓ ′ 𝑡 for all 𝑡 ∈ [ 𝑇 ] .
By standard Gaussian arguments one can derive the following Lemma.
Lemma 9 ([8, Lemma 1]) .
For any δ ∈ ( 0 , 1 ) it holds
[Equation]
Setting δ = 1 / 𝑇 and σ = 1 /( 9 log 2 𝑇 )
we get,
[Equation] [Equation]
[Equation] [Equation]
If 0 ≤ ℓ ′ 𝑡 ( 𝑖 ) ≤ 1, set ℓ 𝑡 ( 𝑖 ) =
ℓ ′ 𝑡 ( 𝑖 ) , else perform clipping:
[Equation]
[Equation] [Equation]
This implies,
[Equation]
Taken this together with Lemma 7 we get that for any deterministic player,
[Equation]
Here we used the fact that for any ∆ > 16 there exists a trivial lower bound of 1.
Clearly, a simple derivation shows that the lower bound in Eq.
(9) holds for any 𝐾 ≥ 2, as we can always extend the loss sequence for 𝐾 > 2 by setting ℓ 𝑡,𝑖
= 1 for any 𝑖 > 2.
In addition, using Lemma 8, when ∆ ≤ O ( 𝐾 1 / 3 𝑇 − 1 / 3 log − 9 / 2𝑇 ) , we get that for any deterministic player with a guarantee of
O ( 𝐾 1 / 3 𝑇 2 / 3 ) switching regret,
[Equation]
Combining both lower bounds in Eqs.
(9) and (10) and observing that
∆ 𝑇 ≥ Ω ( 𝐾 1 / 3 𝑇 2 / 3 / log 9 / 2 2 𝑇 )
when ∆ ≥ Ω ( 𝐾 1 / 3 𝑇 − 1 / 3 log − 9 / 2𝑇 ) , we obtain that for any ∆ > 0,
[Equation]
In other words, if we let the loss sequence to be the conditional process given that 𝐻 is fulfilled, we obtain a stochastic process that generates a random sequence that satisfies the conditions of a stochastically-constrained loss sequence, i.e. Eq.
(1) is met.
Eq.
(11), in turn, bounds the expected regret given that our loss sequence is drawn from the above process.
Next, since any randomized player can be implemented by a random combination of deterministic players we conclude that Eq.
(11) holds for any randomized player.
This then immediately implies that there exists a deterministic loss sequence { ℓ 1 , . . . , ℓ 𝑇 } , that is stochastically-constrained, for which
R 𝑇 + S 𝑇 is lower bounded by the RHS of Eq.
Lastly, we argue that 𝔼 [ R 𝑇 ]
=
𝔼 [ R 𝑇 ] .
This is a direct implication of the loss sequence construction, as for all 𝑡 we have 𝑖 ★ = arg min 𝑖 ∈
[ 𝐾 ] ℓ 𝑡,𝑖 .
Therefore, the acclaimed bounds are achieved also with respect to the pseudo-regret, which concludes the proof.
■
Discussion
Best-of-both-worlds algorithm is an extremely challenging setting and in particular, the case of regret with switching cost poses interesting challenges.
We presented here an algorithm that achieves (up to logarithmic factors) optimal minimax rates for the case of two arms, i.e. 𝐾 = 2.
Surprisingly, the result is obtained using a very simple modification of the standard best-of-both-worlds TsallisINF algorithm.
We note that our analysis is agnostic to any best-of-both-worlds algorithm and that
Tsallis-INF serves only as a building block of our proposed method.
For example, employing the refined bound of Masoudian and Seldin [13] in our analysis follows naturally and will improve our bounds accordingly.
Additionally, it is important to mention, that while we assumed the time horizon 𝑇 is known to the algorithm in advanced, it is not a necessary assumption.
One can use a simple doubling trick while preserving the upper bound under the adversarial regime and suffer an extra multiplicative logarithmic factor of O (cid:0) log 𝑇 (cid:1) in the stochastically constrained regime so the bound at Eq.
(5) becomes,
[Equation] [Equation]
Several open problems though seem to arise from our work.
Open Question 1.
Given arbitrary 𝐾 , what is the optimal minimax regret rate, in the stochastically constrained setting, of any algorithm that achieves, in the adversarial regime, regret of
[Equation]
In particular, in our result, it is interesting to find out if the term O (cid:16) (cid:80) 𝑖 ≠ 𝑖 ★ λ log 𝑇 ∆ min ∆ 𝑖 (cid:17) can be replaced by O (cid:16) (cid:80)
λ log 𝑇 ∆ 2 𝑖 (cid:17) .
Note that this term is obtained by a worst-case analysis of the switching cost that assumes that we obtained the regret by only switching from the optimal arm to the consecutive second-to-best arm.
It seems more likely that any reasonable algorithm roughly switches to each arm 𝑖 , order of ˜ O
( 1 / ∆ 2 𝑖 ) times, leading to more optimistic rate.
Another natural open problem is to try and generalize the lower bound to the general case.
For simplicity we state the next problem for the case all arms have the same gap.
Open Question 2.
Suppose ∆ 1 = ∆ 2 = , . . . , =
∆ min , and ∆ min ≥ ( λ 𝐾 ) 1 / 3 𝑇 − 2 / 3 .
Is it possible to construct an algorithm that achives regret, in the adversarial regime of
[Equation]
and in the stochastically constrained case:
[Equation]
Finally, we would like to stress that our lower bound applies to a stochastically constrained setting, where in principle we often care to understand the stochastic case:
Open Question 3.
What is the optimal expected pseudo-regret with switching cost that can be achieved by an algorithm that achieves regret, in the adversarial regime of
[Equation]
against an i.i.d sequence ℓ 1 , . . . , ℓ 𝑇 that satisfies Eq.
Achieving a non-trivial lower bound for the above case seems like a very challenging task.
In particular, it is known that, if we don’t attempt to achieve best-of-both-worlds rate then an upper bound of 𝑂 ( (cid:80) 𝑖 ≠ 𝑖 ★ log 𝑇 ∆ 𝑖 ) is achievable [9, 10].
Interestingly, then, our lower bound at Theorempresents a separation between the stochastic and stochastically constrained case, leaving open the possibility that a best-of-both-worlds algorithm between adversarial and stochastically constrained case is possible but not necessarily against a stochastic player.
Proving the reverse may require new algorithmic techniques.
In particular, the current analysis of Tsallis-INF is valid for the stochasticallyconstrained case as much as to the stochastic case.
An improved upper bound for the pure stochastic case, though, cannot improve over the stochastically constrained case as demonstrated by Theorem 2.
Open Question 4.
Is the uniqueness of the best arm mandatory in the case of bandits with switching cost?
The case of multiple best arms introduces new challenges.
Whereas Ito [11] showed that Tsallis-INF can handle the case of multiple best arms, it is unclear whether one can use their results to obtain nontrivial bounds in the switching cost setting.
In their experiments, Rouyer et al.
[14], demonstrated this challenge, suggesting that the requirement of the uniqueness of the best arm is necessary in order to obtain improved bounds.
We leave the question of this necessity, in the case of bandits with switching cost and in particular in a best-of-both-worlds setting, to future research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]