Original: Trading Off Resource Budgets For Improved Regret BoundsAnonymous Author s)
Revised: Trading Off Resource Budgets For Improved Regret Bounds

Original: Abstract
Revised: Abstract

Original: We consider a variant of adversarial online learning where in each round one picks B  arms and incurs cost equal to the minimum of the costs of each arm chosen.Westudy the trade-off between the budget B and the regret R ∗ T relative to the singlebest ﬁxed arm in hindsight.
Revised: In this work we consider a variant of adversarial online learning where in each round one picks B out of N arms and incurs cost equal to the minimum of the costs of each arm chosen.

Revised: We propose an algorithm called Follow the Perturbed Multiple

Original: By adapting the techniques of Kalai and Vempala[2005], we show that in the full feedback setting Following the Top B perturbed Leaders (FTL) achieves expected regret O ( T 1 B
Revised: Leaders (FPML) for this problem, which we show (by adapting the techniques of Kalai and Vempala [2005]) achieves expected regret O ( T 1 B

Original: +1 ln( N ) BB +1 ) with N arms overtime horizon T , and allows one to trade off an arm budget for improved regretbounds.
Revised: +1 ln( N ) BB +1 ) over time horizon T relative to the single best arm in hindsight.

Revised: This introduces a trade-off between the budget B and the single-best-arm regret, and we proceed to investigate several applications of this trade-off.

Original: We observe that algorithms which use standard regret minimizers assubroutines can sometimes be adapted by replacing these subroutines with FTL. We use thisobservation to generalize existing algorithms for Online Submodular Function Maximization [Streeter and Golovin, 2008] in both the full feedbackandsemi-bandit feedback settings to allow trade-offs between resource budgets andregret bounds, and  we empirically evaluate these new algorithms on an onlinehyperparameter optimization problem.
Revised: First, we observe that algorithms which use standard regret minimizers as subroutines can sometimes be adapted by replacing these subroutines with FPML, and we use this to generalize existing algorithms for Online Submodular Function Maximization [Streeter and Golovin, 2008] in both the full feedback and semi-bandit feedback settings. Next, we empirically evaluate our new algorithms on an online black-box hyperparameter optimization problem.

Original: Likewise, we show how FTL can lead tonew algorithms for Linear Programming which require stronger oracles at thebeneﬁt of fewer oracle calls.
Revised: Finally, we show how FPML can lead to new algorithms for Linear Programming which require stronger oracles at the beneﬁt of fewer oracle calls.

Original:  Introduction
Revised:  Introduction

Original: Adversarial online learning is a well-studied framework for sequential decision making with numerousapplications.
Revised: Adversarial online learning is a well-studied framework for sequential decision making with numerous applications.

Original: In  round t = 1 , . . .
Revised: In each round t = 1 , . . .

Original: , T , an adversary chooses a hidden cost function c t : 
Revised: , T , an adversary chooses a hidden cost function c t : A !

Original: A → [0 , 1]from a set  A to costs in [0 , 1] .
Revised: [0 , 1] from a set of arms A to costs in [0 , 1] .

Original: An algorithm must then choose an arm a t ∈ A, and incurs cost c t ( a t ) .
Revised: An algorithm must then choose an arm a t 2 A and incurs cost c t ( a t ) .

Original: In the full feedback setting (Online Learning with Experts (OLwE)), the algorithm then observes thecost function c t .
Revised: In the full feedback setting (Online Learning with Experts (OLwE)), the algorithm then observes the cost function c t .

Original: In the partial feedback setting (Multi-Armed Bandits (MAB))
Revised: In the partial feedback setting (Multi-Armed Bandits (MAB))

Original: the algorithm onlyobserves the incurred cost c t ( a t ) .
Revised: the algorithm only observes the incurred cost c t ( a t ) .

Original: The objective is to give algorithms which minimize  the differencebetween the algorithm’s cumulative cost and the cumulative cost of the single best arm in hindsight.
Revised: The objective is to ﬁnd algorithms which minimize regret , deﬁned as the difference between the algorithm’s cumulative cost and the cumulative cost of the single best arm in hindsight.

Original: In this paper we consider a search-like variant of these problems where in each round one can pick asubset of arms S t ⊂ A with | S t | =
Revised: In this paper we consider a search-like variant of these problems where in each round one can pick a subset of arms S t ⇢ A with | S t | =

Original: B ≥ 1 , and keep the arm with the lowest cost.
Revised: B  1 , and keep the arm with the smallest cost .

Original: Three examplesinclude:
Revised: This variant appears naturally in many settings, including:

Original: Online algorithm portfolios [Gomes and Selman, 2001]: In each round t , one receives aproblem instance x t , and can pick a subset S t ⊂ A of algorithms to run in parallel to solvex t .
Revised: Online algorithm portfolios [Gomes and Selman, 2001]: In each round t , one receives a problem instance x t , and can pick a subset S t ⇢ A of algorithms to run in parallel to solve x t .

Original: For example, x t could be a boolean satisﬁability (SAT) problem, and A could be a collection of different SAT solving algorithms.
Revised: For example, x t could be a boolean satisﬁability (SAT) problem, and A could be a collection of different SAT solving algorithms.

Original: We let c t ( a ) =
Revised: We let c t ( a ) =

Original: if a solves x t and c t ( a ) = otherwise.
Revised: if a solves x t and c t ( a ) =otherwise.

Original: Then if any a ∈ S t ﬁnds a solution to x t we incur 0 cost in this round.
Revised: Then if any a 2 S t ﬁnds a solution to x t we incur 0 cost in this round.

Original: Another example is online hyperparameter optimization (see Section 4).
Revised: Another example is online hyperparameter optimization (see Section 4).

Original:  Online bidding [Chen et al., 2016a]: In each round t , an auctioneer sets up a ﬁrst-price auction for bidders S t ⊂ A .
Revised: [Table] Online bidding [Chen et al., 2016a]: In each round t , an auctioneer sets up a ﬁrst-price auction for bidders S t ⇢ A .

Original: Each bidder a ∈ A has a price 1 − c t ( a ) they are willing to pay, and the auctioneer receives max a ∈ S t 1 − c t ( a ) =
Revised: Each bidder a 2 A has a price 1  c t ( a ) they are willing to pay, and the auctioneer receives max [Equation]

Original: 1 − min a ∈ S t c t ( a ) , and so maximizingrevenue is equivalent to minimizing costs.
Revised: [Equation] t ( a ) , and so maximizing revenue is equivalent to minimizing costs.

Original: Adaptive network routing [Awerbuch and Kleinberg, 2008]: In each round t , a networkrouter receives a data packet x t and can pick a selection of network routes S t ⊂ A to send itto its destination via in parallel.
Revised: Adaptive network routing [Awerbuch and Kleinberg, 2008]: In each round t , a network router receives a data packet x t and can pick a selection of network routes S t ⇢ A to send it to its destination via in parallel.

Original: Let the cost c t ( a ) of a route a
Revised: Let the cost c t ( a ) of a route a

Original: be the total time taken for x tto reach its destination via a ; the router receives cost min a ∈ S t c t ( a ) equal to the smallestdelay.
Revised: be the total time taken for x t to reach its destination via a ; the router receives cost min a 2 S t c t ( a ) equal to the smallest delay.

Original: In manynatural applications the budget B is a restricted resource (e.g. compute time) we wouldlike to keep small. This paper studies how one can trade off budget resources for  guarantees onthe standard regret objective.
Revised: In many applications the budget B is a restricted resource (e.g. compute time or number of cores) we would like to keep small; this paper studies how one can trade off budget resources for better guarantees on the standard regret objective.

Original: Formally, for any randomized algorithm ALG which chooses subset 
Revised: Formally, for any randomized algorithm ALG which chooses subset S t ⇢ A in round t , and thus incurs cost c t ( S t ) : = min

Original:  S t⊂ A in round t , and where c t ( S t ) :
Revised: [Equation] 2 S t c t ( [Equation]

Original: = min a ∈ S t c t ( a ) , 
Revised: ) , deﬁne

Original: [Equation]
Revised: [Equation]

Original: is the worst-case expected regret of ALG relative to the single best arm in hindsight, where theexpectation is with respect to the randomness of ALG .
Revised: to be the worst-case expected regret of ALG relative to the single best arm in hindsight, where the expectation is with respect to the randomness of ALG .

Original: 1 What guarantees can we give on R ∗ T as afunction of our budget B ?
Revised: * What guarantees can we give on R ⇤ T as a function of our budget B ?

Original: In the full feedback setting when B = 1 , this is the standard OLwE problemwhere it is known that R ∗ T = Ω( √ T ) and there are algorithms which achieve R ∗ T ≤
Revised: In the full feedback setting when B = 1 , this is the standard OLwE problem where it is known that R ⇤ T = ⌦ ( p T ) and there are algorithms which achieve R ⇤ T 

Original: 2 (cid:112) T ln( N )
Revised: [Equation]

Original: [Lattimore and Szepesvári, 2020] where N = |A| .
Revised: [Lattimore and Szepesvári, 2020], where N = |A| .

Original: When B = N the algorithm which chooses S t =
Revised: When B = N the algorithm which chooses S t =

Original: A in each round achieves R ∗ T = 0 .
Revised: A in each round achieves R ⇤ T = 0 .

Original: But what bounds on R ∗ T can be achieved in the intermediateregime when 1 < B < N ?
Revised: But what bounds on R ⇤ T can be achieved in the intermediate regime when 1 < B < N ?

Original: To the best of our knowledge this question has not been directly answeredby any prior work.
Revised: To the best of our knowledge this question has not been directly answered by any prior work.

Original: Theoretical results: We present a new algorithm, Follow the Top perturbed Leaders ( FTL ), andshow that in the full feedback setting R ∗ T ( FTL ) ≤
Revised: Theoretical results: We present a new algorithm for this learning problem called Follow the Perturbed Multiple Leaders ( FPML ), and show that in the full feedback setting R ⇤ T ( FPML ) 

Original: O ( T 1 B +1 ln( N ) BB +1 ) .
Revised: O ( T 1 B +1 ln( N ) BB +1 ) .

Original: This allows for a directtrade-off between the budget B and  regret bounds, and recovers the familiar O ( (cid:112) T ln( N ))
Revised: This allows for a direct trade-off between the budget B and the regret bound (in particular, allowing resources B  ⌦ (ln( T )) leads to regret constant in T ) and recovers the familiar O ( p

Original: boundwhen B = 1 .
Revised: T ln( N )) bound when B = 1 .

Revised: We then show that in the semi-bandit feedback setting (where the algorithm ﬁnds out only the costs of the arms it chooses) this bound can be converted to

Original: We show this can be converted to R ∗ T ( FTL ) ≤ O ( T 1 B +1{++
+} ( K ln( N ))
Revised: R ⇤ T ( FPML )  O ( T 1 B {++
+} ( K ln( N ))

Original: BB +1 )in thesemi-bandit feedback setting if one has unbiased cost estimators bounded in [0 , K ] .
Revised: BB +1 ) if one has unbiased cost estimators bounded in [0 , K ] .

Original:  Online Submodular Function Maximization (OSFM) is a problem which has applications in algorithmportfolio design, database query processing and sensor placement.
Revised: We also consider the more general problem of Online Submodular Function Maximization (OSFM), for which prior work gives an online greedy algorithm OG [Streeter and Golovin, 2008].

Original: When given a budget of B perround, Streeter and Golovin [2008] give an algorithm OG which minimizes regret  with respect to thebest ﬁxed  schedule  of length B hindsight.
Revised: When given a budget of B per round, OG achieves regret O ( p TB ln( N )) with respect to (1  e  1 ) OPT( B ), where OPT( B ) is the performance of the best ﬁxed length- B schedule (see Section 3 for a formal deﬁnition of OSFM). Note that in this guarantee the regret benchmark is a function of the algorithm budget.

Original: We present OG hybrid ,  a generalization of OG  which uses{+are decoupled from the algorithm budget. This allows one to more easily quantify the trade-off of increasing the budget against a ﬁxed regret objective. As a special case, we are able to show that having a budget of B = B 
+}
Revised: By replacing a subroutine in OG with FPML , we generalize OG to a new algorithm OG hybrid . Unlike OG , OG hybrid is able to give regret bounds against benchmarks which {+are decoupled from the algorithm budget. This allows one to more easily quantify the trade-off of increasing the budget against a ﬁxed regret objective. As a special case, we are able to show that having a budget of B = B 
+}

Original: FTL as a subroutine and is capable of trading off a budget constraint B for regret bounds against thebest length B (cid:48) ≤ B schedule in hindsight (See Section 3 for details).
Revised: d ln( T ) 2 e per round allows one to achieve regret O ( B 0 ln( T ) ln ( N )) with respect to OPT( B 0 ).

Revised: One interpretation of this result is that if you are willing to increase your budget (e.g. runtime) by a factor of ln( T ) 2 , you are able to improve your performance guarantee benchmark from (1  e  1 ) OPT( B 0 ) to OPT( B 0 ).

Revised: Likewise, your regret growth rate in terms of the number of rounds changes from O ( p T ) to O (ln( T )) .

Revised: Finally, in Section 5 we show how to use FTML to generalize a technique for solving linear programs assuming access to an oracle which solves relaxed forms of the linear program.

Revised: To obtain an " - approximate solution to the linear program requires

Revised: [Equation]

Revised: oracle calls, where the parameters ( B, ⇢ ) are related to the power of the oracle and n is the number of linear constraints.

Revised: The case B = 1 coincides with known results.

Revised: Experimental results: We benchmark both FPML and OG hybrid on an online black-box hyperparameter optimization problem based on the 2020 NeurIPS BBO challenge [Turner et al., 2021].

Revised: We ﬁnd that both these new algorithms outperform OG for various compute budgets.

Revised: We are able to explain why this happens for this speciﬁc dataset, and discuss the scenarios under which each algorithm would perform better.

Original: Techniques: Minimizing R ∗ T is an important subroutine for a large variety of applications including
Revised: Techniques: Minimizing R ⇤ T is an important subroutine for a large variety of applications including

Original: Linear Programming, Boosting, and solving zero sum games [Arora et al., 2012].
Revised: Linear Programming, Boosting, and solving zero sum games [Arora et al., 2012].

Original: Traditionally anexperts algorithm such as Hedge [Littlestone and Warmuth, 1994], which pulls a single arm perround, will be used as a subroutine to minimize R ∗ T .
Revised: Traditionally an experts algorithm such as Hedge [Littlestone and Warmuth, 1994], which pulls a single arm per round, will be used as a subroutine to minimize R ⇤ T .

Original: We highlight how in the cases of OSFM and
Revised: We highlight how in the cases of OSFM and

Original: Linear Programming, one can simply replace a single arm R ∗ T -minimizing subroutine with FTL andget performance bounds with little or no alteration to the original proofs.
Revised: Linear Programming, one can simply replace a single arm R ⇤ T -minimizing subroutine with FPML and get performance bounds with little or no alteration to the original proofs.

Original: The resulting algorithmshave improved bounds (due to improved bounds on R ∗ T when B > 1 ) at the cost of qualitativelychanging the algorithm (e.g. requiring a larger budget or more powerful oracle).
Revised: The resulting algorithms have improved bounds (due to improved bounds on R ⇤ T when B > 1 ) at the cost of qualitatively changing the algorithm (e.g. requiring a larger budget or more powerful oracle).

Original: This is signiﬁcantbecause it highlights the potential of how bounds on R ∗ T when B > 1 can lead to new results inother application areas.
Revised: This is signiﬁcant because it highlights the potential of how bounds on R ⇤ T when B > 1 can lead to new results in other application areas.

Original: In Section 2.1 we  highlight how the proof techniques of Kalai and Vempala[2005] for bounding R ∗ T in the traditional experts setting can naturally be generalized to the casewhen B > 1 , which may be of independent interest.Experimental results: We benchmark both FTL and OG hybrid in a setting similar to the NeurIPS BBO challenge [Turner et al., 2021] for the problem of online black-box hyperparameter optimization.
Revised: In Section 2.1 we also highlight how the proof techniques of Kalai and Vempala [2005] for bounding R ⇤ T in the traditional experts setting can naturally be generalized to the case when B > 1 , which is of independent interest.

Original: We ﬁnd that both these algorithms outperform OG for various compute budgets.

Original: We are able toexplain

Original: why this happens for this speciﬁc dataset, and discuss the scenarios under which eachalgorithm would perform better.

Original: 1.1 Relation to prior work
Revised: 1.2 Relation to prior work

Original: One can alternatively formulate the problem we consider as receiving the maximum reward r t ( a ) =1 − c t ( a ) of each arm chosen instead of the minimum cost.
Revised: One can alternatively formulate more gewe consider as receiving the maximum reward r t ( a ) = 1  c t ( a ) of each arm chosen instead of the minimum cost.

Revised: In this maximum of rewards formulation, the problem ﬁts within the OSFM framework where (a) all actions are unit-time and (b) the submodular job function is always a maximum of rewards.

Original: This maximum of rewards formulationﬁts within the framework of Online Submodular Function Maximization (OSFM), and has alsobeen separately studied as the K-MAX problem (here K = B ) Chen et al.
Revised: The rewards formulation of the problem has also been separately studied as the K-MAX problem (here K = B ) Chen et al.

Original: [2016a].
Revised: [2016a].

Original: In the OSFMsetting, Streeter and Golovin [2008] give an online greedy approximation algorithm which guarantees
Revised: In the OSFM setting, Streeter and Golovin [2008] give an online greedy approximation algorithm which guarantees

Original: E [(1 − e − 1 ) OPT ( B ) − Reward T ] ≤ O ( (cid:112) TB ln( N )) in the full feedback adversarial setting, where
Revised: E [(1  e  1 ) OPT ( B )  Reward T ]  O ( p TB ln( N )) in the full feedback adversarial setting, where

Original: OPT ( B ) is the cumulative reward of the best ﬁxed subset ofsize B  in hindsight, and Reward T is thecumulative reward of the algorithm.
Revised: OPT ( B ) is the cumulative reward of the best ﬁxed subset of B arms in hindsight, and Reward T is the cumulative reward of the algorithm.

Original: A similar bound of O ( B (cid:112) TN ln
Revised: A similar bound of O ( B p

Original: ( N )) can be given in a semifeedback setting.
Revised: TN ln( N )) can be given in a semifeedback setting.

Original: Conversely in the full feedback setting, Streeter and Golovin [2007] shows that anyalgorithm has worst-case regret E [ OPT ( B ) − Reward T ] ≥ Ω( (cid:112) TB ln
Revised: Conversely in the full feedback setting, Streeter and Golovin [2007] shows that any algorithm has worst-case regret E [ OPT ( B )  Reward T ]  ⌦ ( p

Original: ( N/B ))
Revised: TB ln( N/B ))

Original: when one receivesthe maximum of rewards in each round.
Revised: when one receives the maximum of rewards in each round.

Original: Chen et al.
Revised: Chen et al.

Original: [2016a] study the K-MAX problem and othernon-linear reward functions in the stochastic combinatorial multi-armed bandit setting.
Revised: [2016a] study the K-MAX problem and other non-linear reward functions in the stochastic combinatorial multi-armed bandit setting.

Original: Assuming therewards satisfy certain distributional assumptions, they give an algorithm which achieves distributionindependent regret bounds of E [(1 − ε ) OPT ( B ) − Reward T ] ≤ O ( (cid:112)
Revised: Assuming the rewards satisfy certain distributional assumptions, they give an algorithm which achieves distributionindependent regret bounds of E [(1  " ) OPT ( B )  Reward T ]  O ( p

Original: TBN ln( T )) for ε > 0 withsemi-bandit feedback.
Revised: TBN ln( T )) for " > 0 with semi-bandit feedback.

Original: Note however that we consider the adversarial setting in this paper.
Revised: Note however that we consider the adversarial setting in this paper.

Original: More broadly, these problems fall within the combinatorial online learning setting where an algorithmmay pull a subset of arms in each round.
Revised: More broadly, these problems fall within the combinatorial online learning setting where an algorithm may pull a subset of arms in each round.

Original: Much prior work has focused on combinatorial banditswhere the reward is linear in the subset of arms chosen, which can model applications including onlineadvertising and online shortest paths [Cesa-Bianchi and Lugosi, 2012, Audibert et al., 2014, Combeset al., 2015].
Revised: Much prior work has focused on combinatorial bandits where the reward is linear in the subset of arms chosen, which can model applications including online advertising and online shortest paths [Cesa-Bianchi and Lugosi, 2012, Audibert et al., 2014, Combes et al., 2015].

Original: The case of non-linear reward is comparatively less studied, but having non-linearrewards (such as max) allows one to model a wider variety of problems including online expectedutility maximization
Revised: The case of non-linear reward is comparatively less studied, but having non-linear rewards (such as max) allows one to model a wider variety of problems including online expected utility maximization

Original: [Li and Deshpande, 2011, Chen et al., 2016a].
Revised: [Li and Deshpande, 2011, Chen et al., 2016a].

Original: As some examples of prior workin the stochastic setting, [Gopalan et al., 2014] uses Thompson Sampling to deal with non-linearrewards of functions of subsets of arms (including the max function), but requires the rewards tocome from a known parametric distribution.
Revised: As some examples of prior work in the stochastic setting, [Gopalan et al., 2014] uses Thompson Sampling to deal with non-linear rewards of functions of subsets of arms (including the max function), but requires the rewards to come from a known parametric distribution.

Original: Chen et al.
Revised: Chen et al.

Original: [2016b] considers a model where the subsetof arms pulled is randomized based on pulling a ‘super-arm’, and the reward is a non-linear functionof the values of the arms pulled.
Revised: [2016b] considers a model where the subset of arms pulled is randomized based on pulling a ‘super-arm’, and the reward is a non-linear function of the values of the arms pulled.

Original: In the adversarial setting, Han et al.
Revised: In the adversarial setting, Han et al.

Original: [2021] study the combinatorial
Revised: [2021] study the combinatorial

Original: MAB problem when rewards can be expressed as a d -degree polynomial.
Revised: MAB problem when rewards can be expressed as a d -degree polynomial.

Original: In contrast to prior work which focuses on giving algorithms which compete against benchmarkswhich have the same budget as the algorithm, this work is concerned with the trade-off betweenregret bounds and budget size.
Revised: In contrast to prior work which focuses on giving algorithms which compete against benchmarks which have the same budget as the algorithm, this work is concerned with the trade-off between regret bounds and budget size.

Original: We focus on giving regret bounds against OPT (1) , and we use thisresult in Section 3 to get regret bounds against OPT ( B (cid:48) ) for B (cid:48) < B in OSFM.
Revised: We focus on giving regret bounds against OPT (1) , and we use this result in Section 3 to get regret bounds against OPT ( B 0 ) for B 0 < B in OSFM.

Original: Decoupling the regretbenchmark OPT ( B (cid:48) ) from the algorithm budget B can be useful when one would like to controlthe strength of a regret bound against a speciﬁc target OPT ( B (cid:48) ) for theoretical or applied reasons.
Revised: Decoupling the regret benchmark OPT ( B 0 ) from the algorithm budget B can be useful when one would like to control the strength of a regret bound against a speciﬁc target OPT ( B 0 ) for theoretical or applied reasons.

Original: For example Arora et al.
Revised: For example Arora et al.

Original: [2012] survey a wide variety of applications which rely on bounding R ∗ T ,but bounds such as E
Revised: [2012] survey a wide variety of applications which rely on bounding R ⇤ T , but bounds such as E

Original: [(1 − e − 1 ) OPT ( B ) − Reward T ] ≤ O ( (cid:112) TB ln( N )) do not immediately implyuseful bounds on OPT (1) − Reward T .
Revised: [(1  e  1 ) OPT ( B )  Reward T ]  O ( p TB ln( N )) do not immediately imply useful bounds on OPT (1)  Reward T .

Original:  Follow theTop Perturbed  Leaders
Revised:  Follow the Perturbed Multiple Leaders

Original: We begin by considering the full feedback setting.
Revised: We begin by considering the full feedback setting.

Original: We ﬁrst check that allowing the algorithm tochoose B > 1 arms per round, while only competing against the best single ﬁxed arm in hindsight,does not make the problem trivial.
Revised: We ﬁrst check that allowing the algorithm to choose B > 1 arms per round, while only competing against the best single ﬁxed arm in hindsight, does not make the problem trivial.

Original: We do this by showing that any deterministic algorithm withbudget B < N still achieves linear regret in the number of rounds.
Revised: We do this by showing that any deterministic algorithm with budget B < N still achieves linear regret in the number of rounds.

Original: This is achieved by settingc t ( a ) = 1 if a ∈ S t , c t ( a ) = 0 otherwise.
Revised: This is achieved by setting c t ( a ) = 1 if a 2 S t , c t ( a ) = 0 otherwise.

Original: Proposition 1.
Revised: Proposition 1.

Original: In the full feedback setting, any deterministic algorithm with arm budget B ≤ N perround has worst-case regret R ∗ T ≥ (cid:0) 1 − BN (cid:1) T .
Revised: In the full feedback setting, any deterministic algorithm with arm budget B  N per round has worst-case regret R ⇤ T

Original:  Likewise, it can be shown that the algorithm which chooses a uniformly random subset of B arms ineach round has worst-case expected regret at least T (1 − BN ) B (achieved by having one arm have cost0 across all rounds and every other arm having cost 1 ).
Revised: [Equation] Likewise, it can be shown that the algorithm which chooses a uniformly random subset of B arms in each round has worst-case expected regret at least T (1  B N ) B (achieved by having one arm have cost 0 across all rounds and every other arm having cost 1 ).

Original: These two observations show that any solutionfor achieving sub-linear regret in T requires randomization which depends in some non-trivial wayon the prior observed costs even when B > 1 .
Revised: These two observations show that any solution for achieving sub-linear regret in T requires randomization which depends in some non-trivial way on the prior observed costs even when B > 1 .

Original: 2.1 Generalizing Follow the Perturbed Leader
Revised: 2.1 Generalizing Follow the Perturbed Leader

Original: Choosing the current lowest perturbed-cost arm in each round, Follow the Perturbed Leader ( FPL )
Revised: Choosing the current lowest perturbed-cost arm in each round, Follow the Perturbed Leader ( FPL )

Original: [Kalai and Vempala, 2005], is a well-known regret minimization technique which achieves optimalworst-case regret against adaptive adversaries in the OLwE setting.
Revised: [Kalai and Vempala, 2005], is a well-known regret minimization technique which achieves optimal worst-case regret against adaptive adversaries in the OLwE setting.

Original: In this section we generalizethe FPL algorithm to Follow theTop Perturbed  Leaders ( FTL ).
Revised: In this section we generalize the FPL algorithm to Follow the Perturbed Multiple Leaders ( FPML ).

Original: In each round, FTL perturbs thecumulative costs of each arm by adding noise, and then picks the B arms with lowest cumulativeperturbed cost.
Revised: In each round, FPML perturbs the cumulative costs of each arm by adding noise, and then picks the B arms with lowest cumulative perturbed cost.

Original: This is precisely FPL when B = 1 .
Revised: This is precisely FPL when B = 1 .

Original: We show how one can extend the prooftechniques of Kalai and Vempala [2005] in a natural way to prove that FTL achieves worst-caseregret R ∗ T ( FTL ) ≤ 2 T 1 B +1 (1 + ln( N ))
Revised: We show how one can extend the proof techniques of Kalai and Vempala [2005] in a natural way to prove that FPML achieves worst-case regret R ⇤ T ( FPML )  2 T 1 B +1 (1 + ln( N ))

Original: BB +1 .
Revised: BB +1 .

Original: Require: N ≥ B ≥ 1 , ε > 0 .
Revised: Require: N  B  1 , " > 0 .

Original: Initialize the cumulative cost C 

Revised: Initialize the cumulative cost C

Original: ( a )← 0 for each arm a ∈ A .
Revised: ( a ) 0 for each arm a 2 A .

Original: for round t = 1 , . . .
Revised: for round t = 1 , . . .

Original: , T do 
Revised: , T do 1.

Original: For each arm a ∈ A , draw a noise perturbation p t ( a ) ∼ 1 ε perturbed t −
Revised: For each arm a 2 A , draw a noise perturbation p t ( a ) ⇠ 1 " Exp.

Original:  t −
Revised: Calculate the perturbed cumulative costs for round t  1 , ˜ C t  1 ( a ) [Equation] Pull the B arms with the lowest perturbed cumulative costs according to ˜ C t  1 .

Original: the arbitrarily.
Revised: Break ties arbitrarily.

Revised: Update the cumulative costs

Original: end for
Revised: for

Original: Theorem 2. In the full feedback setting, where S t ⊂ A is the subset of arms chosen by FTL in roundt , we have:
Revised: each arm C t a )

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation] = [Equation] c
Revised:  a

Original: [Equation]
Revised: [Equation]

Original: ( a
Revised: end for

Original: [Equation]
Revised: Theorem 2.

Original: ) (cid:35)
Revised: In the full feedback setting, where S t ⇢ A is the subset of arms chosen by FPML in round t , we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, for ε =
Revised: In particular, for " =

Original: ((ln( N ) + 1) /T ) 1 / [Equation] B +1) , we have
Revised: ((ln( N ) + 1) /T ) 1 / ( B +1) , we have

Original: [Equation]
Revised: [Equation]

Original: The proof follows the same three high level steps which appear in Kalai and Vempala [2005] for FPL ,but we extend these ideas to the case where B > 1 .
Revised: The proof follows the same three high level steps which appear in Kalai and Vempala [2005] for FPL , but we extend these ideas to the case where B > 1 .

Original: We ﬁrst observe that the algorithm which picksthe B lowest cumulative cost arms in each round only incurs regret when the best arm in round t isnot one of thebest B arms in round t − 1 .
Revised: We ﬁrst observe that the algorithm which picks the B lowest cumulative cost arms in each round only incurs regret when the best arm in round t is not one of the

Original: Lemma 3.
Revised: Lemma 3.

Original: Consider a ﬁxed sequence of cost functions c 1 , . .
Revised: Consider a ﬁxed sequence of cost functions c 1 , . .

Original: . , c T .
Revised: . , c T .

Original: Let a ∗ ,jt be the j th lowestcumulative cost arm in hindsight after the ﬁrst t rounds, breaking ties arbitrarily.
Revised: Let a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds, breaking ties arbitrarily.

Original: Let S ∗ t :={ a ∗ ,jt − 1 |j ∈
Revised: Let S ⇤ t :=

Original:  [ B ] } be the set of the B lowest cost arms at the end of round t − 1 .
Revised: { a ⇤ ,jt  1 | j 2 [ B ] } be the set of the B lowest cost arms at the end of round t  1 .

Original: Then for each i ∈ [ T ] ,
Revised: Then for each i 2 [ T ] ,

Original: [Equation]
Revised: [Equation]

Original: and R i − R i − 1 ≤ 1 [ a ∗ , 1 i(cid:54)∈ S ∗ i ] .
Revised: and R i  R i  1  1 [ a ⇤ , 1 i

Original:  This is a generalization of the familiar result that when B = 1 , following the leader has regretbounded by the number of times the leader is overtaken [Kalai and Vempala, 2005].
Revised: [Equation] This is a generalization of the familiar result that when B = 1 , following the leader has regret bounded by the number of times the leader is overtaken [Kalai and Vempala, 2005].

Original: The second step is to argue that if the cumulative costs are perturbed slightly, it becomes unlikelythat the event { a ∗ , 1 t (cid:54)∈ S ∗ t } will occur.
Revised: The second step is to argue that if the cumulative costs are perturbed slightly, it becomes unlikely that the event { a ⇤ , 1 t 62 S ⇤ t } will occur.

Original: One way to see this is as follows: ﬁx a round t , and let C t − 1 ( a ) be the cumulative cost of a at the end of round t − 1 .
Revised: One way to see this is as follows: ﬁx a round t , and let C t  1 ( a ) be the cumulative cost of a at the end of round t  1 .

Original: Let M :=
Revised: Let M :=

Original: C t − 1 ( a ∗ ,B +1 ) .
Revised: C t  1 ( a ⇤ ,B +1 ) .

Original: Then everya ∈ S  t has C t − 1 ( a ) ≤ M .
Revised: Then every a 2 S ⇤ t has C t  1 ( a )  M .

Original: If it is also true that C t − 1 ( a ) < M − c t ( a ) for any a ∈ S  t then the event{ a ∗ , 1 t (cid:54)∈ S ∗ t } cannot occur.
Revised: If it is also true that C t  1 ( a ) < M  c t ( a ) for any a 2 S ⇤ t then the event { a ⇤ , 1 t 62 S ⇤ t } cannot occur.

Original: This is because C t ( a ) <
Revised: This is because C t ( a ) <

Original: ( M − c t ( a ))
Revised: ( M  c t ( a ))

Original: + c t ( a ) <
Revised: + c t ( a ) =

Original: M but any a (cid:48) ∈ A − S thas C t ( a (cid:48) ) ≥ M , so a (cid:48) (cid:54) = a ∗ , 1 t .
Revised: M but any a 0 2 A  S t has C t ( a 0 )  M , so a 0 6 = a ⇤ , 1 t .

Original: If we had initially perturbed each C t − 1 ( a ) by subtracting independentexponential noise p ( a ) ∼ 1 ε Exp, then conditional on M the event { C t − 1 ( a ) <
Revised: If we had initially perturbed each C t  1 ( a ) by subtracting independent exponential noise p ( a ) ⇠ 1 " Exp, then conditional on M the event { C t  1 ( a ) <

Original: M − c t ( a ) } is jointlyindependent for each a ∈ S  t .
Revised: M  c t ( a ) } is jointly independent for each a 2 S ⇤ t .

Original: Moreover the probability of this inequality not holding is equal to163 P [ p ( a ) < v + c t ( a )
Revised: Moreover the probability of this inequality not holding is equal to P [ p ( a ) < v + c t ( a )

Original: | p ( a ) ≥ v ] for v equal to the unperturbed cost of a at round t − 1 minus M ,which is bounded by εc t ( a ) (due to the memorylessness property of the exponential distribution).
Revised: | p ( a )  v ] for v equal to the unperturbed cost of a at round t  1 minus M , which is bounded by " c t ( a ) (due to the memorylessness property of the exponential distribution).

Original: Lemma 4.
Revised: Lemma 4.

Original: Fix a sequence of cost functions c 1 , . .
Revised: Fix a sequence of cost functions c 1 , . .

Original: . , c T .
Revised: . , c T .

Original: Let C i ( a ) 
Revised: Let C i ( a ) =

Original: = (cid:80) it =1 c t ( a ) and ˜ C i ( a ) =
Revised: P it =1 c t ( a ) and ˜ C i ( a ) =

Original: C i ( a ) − p ( a ) be the perturbed cumulative cost of arm a at the end of round i , where p ( a ) ∼ 1 ε Exp.
Revised: C i ( a )  p ( a ) be the perturbed cumulative cost of arm a at the end of round i , where p ( a ) ⇠ 1 " Exp.

Original: Let ˜ a ∗ , jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds using these perturbedcosts, and let ˜ S ∗ t = { ˜ a ∗ ,jt − 1 | j ∈ [ B ] } .
Revised: Let ˜ a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds using these perturbed costs, and let ˜ S ⇤ t = { ˜ a ⇤ ,jt  1 | j 2 [ B ] } .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Again, when B = 1 this argument and bound coincides with the argument given by Kalai and
Revised: Again, when B = 1 this argument and bound coincides with the argument given by Kalai and

Original: Vempala [2005].
Revised: Vempala [2005].

Original: The ﬁnal step is to combine Lemmas 3 and 4 to argue that FTL achieves expected regret at mostE (cid:104)
Revised: The ﬁnal step is to combine Lemmas 3 and 4 to argue that FPML achieves expected regret at most

Original: ε B (cid:80)
Revised: E h " B P

Original: Tt =1 c t ( ˜ S ∗ t ) (cid:105) with respect to the perturbed cumulative cost ˜ C T .
Revised: Tt =1 c t ( ˜ S ⇤ t ) i with respect to the perturbed cumulative cost ˜ C T .

Original: Since max a ∈A E [ p ( a )] ≤1+ln( N ) ε we can argue we also achieve low expected regret with respect to the unperturbed cost C T .
Revised: Since max a 2 A E [ p ( a )]  1+ln( N ) " we can argue we also achieve low expected regret with respect to the unperturbed cost C T .

Original: In the setting of this paper, drawing new random perturbations p t ( a ) in each round is not strictlynecessary (we can take p t ( a ) =
Revised: In the setting of this paper, drawing new random perturbations p t ( a ) in each round is not strictly necessary (we can take p t ( a ) =

Original: p 1 ( a ) for t > 1 ), but it is necessary to achieve regret bounds whencost functions can depend on prior arm choices of the algorithm (the adaptive adversarial setting).
Revised: p 1 ( a ) for t > 1 ), but it is necessary to achieve regret bounds when cost functions can depend on prior arm choices of the algorithm (the adaptive adversarial setting).

Original: Inthe setting of this paper where the costs are ﬁxed, the expected regret in either case is the same.
Revised: In the setting of this paper where the costs are ﬁxed, the expected regret in either case is the same.

Original: Probabilistic guarantees: One advantage of this proof technique is that the regret is boundedusing the positive random variable(cid:80)
Revised: Probabilistic guarantees: One advantage of this proof technique is that the regret is bounded using the positive random variable

Original: Tt =
Revised: P

Original: 1 1 (cid:104) ˜ a ∗ , 1 t (cid:54)∈ ˜ S ∗ t (cid:105) .
Revised: [Equation] i .

Original: This means that one can apply methods like
Revised: This means that one can apply methods like

Original: Markov inequality to give a probabilistic guarantee of small regret, which is substantially strongerthan saying the regret is small in expectation.
Revised: Markov inequality to give a probabilistic guarantee of small regret, which is substantially stronger than saying the regret is small in expectation.

Original: Comments on settings of parameters: When B = 1 we recover the standard O ( (cid:112) T ln( N ))regret bound for the OLwE problem.
Revised: Comments on settings of parameters: When B = 1 we recover the standard O ( p T ln( N )) regret bound for the OLwE problem.

Original: For B > 1 , the regret growth rate as a function of the numberof rounds is T 1 B +1 .
Revised: For B > 1 , the regret growth rate as a function of the number of rounds is T 1 B +1 .

Original: In particular, when B = Ω(ln( T )) grows slowly with the number of rounds,the expected regret becomes O (ln( N )) and does not grow with the number of rounds T .
Revised: In particular, when B = ⌦ (ln( T )) grows slowly with the number of rounds, the expected regret becomes O (ln( N )) and does not grow with the number of rounds T .

Original: If we usea tighter inequality in the proof of Lemma 4, it is possible to get constant expected regret when
Revised: If we use a tighter inequality in the proof of Lemma 4, it is possible to get constant expected regret when

Original: B = ln( T ) ln
Revised: B = ln( T ) ln

Original: ( A ) grows slowly with the number of arms and rounds.
Revised: ( A ) grows slowly with the number of arms and rounds.

Original: Lower bounds: A standard technique for constructing lower bounds in the online experts settingwith B = 1 is to consider costs which are i.i.d.
Revised: Lower bounds: A standard technique for constructing lower bounds in the online experts setting with B = 1 is to consider costs which are i.i.d.

Original: Bernoulli ( p ) [Lattimore and Szepesvári, 2020].
Revised: Bernoulli ( p ) [Lattimore and Szepesvári, 2020].

Original: Unfortunately this technique fails when B > 1 because the expected cost of the minimum of B i.i.d.
Revised: Unfortunately this technique fails when B > 1 because the expected cost of the minimum of B i.i.d.

Original: Bernoulli random variables is generally smaller than the expected cost of the best arm in hindsightunless p is very close to 1 .
Revised: Bernoulli random variables is generally smaller than the expected cost of the best arm in hindsight unless p is very close to 1 .

Original: We are able to show very weak lower bounds in the full feedback settingof R ∗ T =
Revised: We are able to show very weak lower bounds in the full feedback setting of R ⇤ T =

Original: Ω(ln( N )) for constant B and T = Ω(ln( N )) , but there is a substantial gap between thisand the upper bound.
Revised: ⌦ (ln( N )) for constant B and T = ⌦ (ln( N )) , but there is a substantial gap between this and the upper bound.

Original: We think constructing stronger lower bounds is an interesting problem forfuture work which may require new analysis techniques.
Revised: We think constructing stronger lower bounds is an interesting problem for future work which may require new analysis techniques.

Original: Partial feedback: The semi-bandit feedback setting is a form of partial feedback where the algorithm only observes the individual costs of the arms it pulls.
Revised: Partial feedback: The semi-bandit feedback setting is a form of partial feedback where the algorithm only observes the individual costs of the arms it pulls.

Original: It can be shown that passing unbiasedcost function estimates to FTL results in a similar regret bound in the semi-bandit feedback setting;the result is speciﬁc to FTL and using unbiased cost functions does not generally work for any
Revised: It can be shown that passing unbiased cost function estimates to FPML results in a similar regret bound in the semi-bandit feedback setting; the result is speciﬁc to FPML and using unbiased cost functions does not generally work for any

Original: R ∗ T -minimizing algorithm when B > 1 because of the non-linearity of the cost function.
Revised: R ⇤ T -minimizing algorithm when B > 1 because of the non-linearity of the cost function.

Original: In the caseof FTL this is not an issue because the same bounding technique using Lemma 3 holds in expectationwhen using unbiased cost estimators.
Revised: In the case of FPML this is not an issue because the same bounding technique using Lemma 3 holds in expectation when using unbiased cost estimators.

Original: A naïve way to generate unbiased cost estimates in this settingis to use an additional arm to uniformly sample costs; in Section 4 we explore geometric sampling[Neu and Bartók, 2013] for getting unbiased cost estimates which is effective in practice.
Revised: A naïve way to generate unbiased cost estimates in this setting is to use an additional arm to uniformly sample costs; in Section 4 we explore geometric sampling [Neu and Bartók, 2013] for getting unbiased cost estimates which is effective in practice.

Original: Proposition 5.
Revised: Proposition 5.

Original: Deﬁne the algorithm FTL-partial which simulates FTL , passing it unbiased costestimates ˆ c t ∈ [0 , K ] at round t ∈ [ T ] , 2 and copies the arm choices of FTL .
Revised: Deﬁne the algorithm FPML-partial which simulates FPML , passing it unbiased cost estimates ˆ c t 2 [0 , K ] at round t 2 [ T ] , * and copies the arm choices of FPML .

Original: Then we have
Revised: Then we have

Original: R ∗ T ( FTL-partial ) ≤ ln( N ) /ε
Revised: R ⇤ [Equation] For

Original: + T (1 − e − Kε ) B . For ε = (cid:16) ln( N ) TK B (cid:17) 1 B +1 , R ∗ T ( FTL-partial ) ≤
Revised: [Equation]

Original: O ( T 1 B +1 ( K ln( N )) BB +1 ) .
Revised: O ( [Equation] [Equation]

Original:  Generalized regret bounds for Online Submodular FunctionMaximization
Revised:  Generalized regret bounds for Online Submodular Function Maximization

Original: Streeter and Golovin [2008] considered the related problem of Online Submodular Function Maximization which captures a number of previously studied problems as special cases.
Revised: Streeter and Golovin [2008] considered the more general problem of Online Submodular Function Maximization which captures a number of previously studied problems as special cases.

Original: For the sake ofemphasizing the key ideas, we restrict attention to the full feedback setting where each action hasunit duration.
Revised: For the sake of emphasizing the key ideas, we restrict attention to the full feedback setting where each action has unit duration.

Revised: The OSFM problem in this setting is as follows:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Deﬁne a schedule to be a ﬁnite sequence of actions 3 a ∈ A , and let S be the set of allschedules; the length (cid:96) ( S ) of a schedule S ∈ S is the number of actions it contains.
Revised: Deﬁne a schedule to be a ﬁnite sequence of actions * a 2 A , and let S be the set of all schedules; the length ` ( S ) of a schedule S 2 S is the number of actions it contains.

Original: Deﬁne a job tobe a function f : S → [0 , 1] such that for any schedules S 1 , S 2 ∈ S and any action a ∈ A :1.
Revised: Deﬁne a job to be a function f : S ! [0 , 1] such that for any schedules S 1 , S 2 2 S and any action a 2 A :

Original: f ( S 1 ) ≤ f ( S 1 ⊕ S 2 ) and f ( S 2 )
Revised: f ( S 1 )  f ( S 1  S 2 ) and f ( S 2 )

Original: ≤ f ( S 1 ⊕ S 2 ) (monotonicity) ;2. f ( S ⊕ S
Revised:  f ( S 1  S 2 ) (monotonicity) ; 2.

Original: ⊕ (cid:104) a (cid:105) ) − f ( S ⊕ S 2 ) ≤ f ( S 1 ⊕ (cid:104) a (cid:105) )
Revised: f ( S 1  S 2  h a i )  f ( S 1  S 2 )  f ( S 1  h a i )

Original: − f ( S 1 ) (submodularity) .
Revised:  f ( S 1 ) (submodularity) .

Original: Deﬁnition 2 (Online Submodular Function Maximization) .
Revised: Deﬁnition 2 (Online Submodular Function Maximization) .

Original: The problem consists of a game with Trounds.
Revised: The problem consists of a game with T rounds.

Original: We are given some ﬁxed B > 0 and at each round t ∈ [ T ] we must choose a schedule S t ∈ Swith (cid:96) ( S t ) ≤ B to be evaluated by a job f t which is only revealed after our choice.
Revised: We are given some ﬁxed B > 0 and at each round t 2 [ T ] we must choose a schedule S t 2 S with ` ( S t )  B to be evaluated by a job f t which is only revealed after our choice.

Original: The goal is tomaximize the cumulative output Reward T := (cid:80) T t =1 f t ( S t ) .
Revised: The goal is to maximize the cumulative output Reward T := P Tt =1 f t ( S t ) .

Original: Streeter and Golovin [2008] propose an online greedy algorithm OG which achieves the guarantee(1 − e − 1 ) OPT ( B ) − Reward T ≤ O ( (cid:112) TB
Revised: Streeter and Golovin [2008] propose an online greedy algorithm OG which achieves the guarantee (1  e  1 ) OPT ( B )  Reward T  O ( p

Original:  ln( N )) in expectation, where OPT ( B ) is the cumulativereward of the best ﬁxed schedule of length B in hindsight.
Revised: TB ln( N )) in expectation, where OPT ( B ) is the cumulative reward of the best ﬁxed schedule of length B in hindsight.

Original: In this section we explain how to use FTLto extend their algorithm to allow a trade-off between budget resources and regret bounds.
Revised: In this section we explain how to use FPML to extend their algorithm to allow a trade-off between budget resources and regret bounds.

Original: The algorithm OG has two key ideas.
Revised: The algorithm OG has two key ideas.

Original: Suppose that we start with an empty schedule S 0 := ∅ .
Revised: Suppose that we start with an empty schedule S 0 : = ; .

Original: The ﬁrst idea is that if we knew f 1 , . . .
Revised: The ﬁrst idea is that if we knew f 1 , . . .

Original: , f T in advance, we could greedily construct S i := S i − 1 ⊕(cid:104) arg max a i ∈A (cid:80)
Revised: , f T in advance, we could greedily construct S i := S i  1  h arg max a i 2 A P

Original: Tt =1 f t ( S i − 1 ⊕ (cid:104)
Revised: Tt =1 f t ( S i  1  h

Original: a i (cid:105) ) 
Revised: a i i )  f t

Original: − f t ( S i − 1 ) (cid:105) for i = 1 , . . 
Revised: ( S i  1 ) i for i = 1 , . . .

Original: , B , where a i is the best greedy armin hindsight for greedy round i .
Revised: , B , where a i is the best greedy arm in hindsight for greedy round i .

Original: It can be shown that submodularity then implies (1 − e − 1 ) OPT ( B ) ≤ 232 (cid:80)
Revised: It can be shown that submodularity then implies (1  e  1 ) OPT ( B ) 

Original:  Tt =1 f ( S B ) .
Revised: P Tt =1 f ( S B ) .

Original: Since we don’t know f 1 , . .
Revised: Since we don’t know f 1 , . .

Original: , f T in advance, the second idea is to run B copies of a
Revised: , f T in advance, the second idea is to run B copies of a

Original: R ∗ T regret minimizer, where the i th copy tries to compete with achieving the same improvement ofcumulative reward as the best ﬁxed greedy action a i in hindsight.
Revised: R ⇤ T regret minimizer, where the i th copy tries to compete with achieving the same improvement of cumulative reward as the best ﬁxed greedy action a i in hindsight.

Original: The regret bound on the i th copywith respect to the best greedy arm in hindsight is O ( √ T ln N ) ; across the B copies one can showthe net regret compared to the ofﬂine greedy solution is bounded by O ( √ TB ln N ) , which is wherethe ﬁnal bound comes from.
Revised: The regret bound on the i th copy with respect to the best greedy arm in hindsight is O ( p T ln N ) ; across the B copies one can show the net regret compared to the ofﬂine greedy solution is bounded by O ( p TB ln N ) , which is where the ﬁnal bound comes from.

Original: In summary, OG works as follows: for each round t ∈ [ T ] , run B greedyrounds.
Revised: In summary, OG works as follows: for each round t 2 [ T ] , run B greedy rounds.

Original: In greedy round i = 1 , . . .
Revised: In greedy round i = 1 , . . .

Original: , B , pull the arm a ti proposed by the i th black box R ∗ T -minimizer,set S t,i :=
Revised: , B , pull the arm a ti proposed by the i th black box R ⇤ T -minimizer, set S t,i :=

Original: S t,i − 1 ⊕ (cid:104) a ti (cid:105) , and feed back the greedy rewards r t,i ( a ) =
Revised: S t,i  1  h a ti i , and feed back the greedy rewards r t,i ( a ) =

Original: f t ( S t,i − 1 ⊕ (cid:104) a (cid:105) ) − f t ( S t,i − 1 )to the i th black box.
Revised: f t ( S t,i  1  h a i )  f t ( S t,i  1 ) to the i th black box.

Original: We propose a hybrid version of OG , named OG hybrid , which  allows us to compete  against OPT( B (cid:48) )for B (cid:48)
Revised: We propose a hybrid version of OG , called OG hybrid , which for any budget B allows us to compete asymptotically well against OPT( B 0 ) for any chosen B 0  B/ ln( T ) . The algorithm OG hybrid is based on the following two changes to OG :

Original: ≤ B based on the following two changes: (1) Instead of having B(cid:48) greedy rounds, we have B (cid:48) ln( T ) greedy rounds.
Revised: Instead of having B greedy rounds, we have B 0 ln( T ) greedy rounds.

Original: One can show that the extra factor of ln( T ) allows one to drop the (1 − e − 1 )term in the regret bound.
Revised: One can show that the extra factor of ln( T ) allows one to drop the (1  e  1 ) term in the regret bound.

Original: (2) Instead of running a one-arm-pulling R ∗ T minimizer in each greedyround, we run FTL which pulls ˜ B{+b B/B 
 ln( T ) c+} arms.
Revised: Instead of running a one-arm-pulling R ⇤ T minimizer in each greedy round, we run FPML which pulls {+b B/B 
 ln( T ) c+} arms.

Original: This allows us to improve the regret bound for each of the B (cid:48)
Revised: This allows us to improve the regret bound for each of the B 0 ln( T )

Original: ln( T ) R ∗ T -minimizers and directly translates to a tighter overall regret bound.
Revised: R ⇤ T -minimizers and directly translates to a tighter overall regret bound.

Original: Besides these twochanges, the algorithm is identical to that in Streeter and Golovin [2008].
Revised: Besides these two changes, the algorithm is identical to that in Streeter and Golovin [2008].

Original: More generally,we let OG hybrid ( B, ˜ B ) denote the algorithm where each FTL box has a budget of ˜ B and there are B/ ˜ Bgreedy rounds, so that the total number of arms pulled in each round is B .
Revised: More generally, let OG hybrid ( B, ˜ B ) denote the algorithm where each FPML box has a budget of ˜ B and there are B/ ˜ B greedy rounds, so that the total number of arms pulled in each round is B .

Original: This algorithm is a hybridof OG and FTL in the sense that OG hybrid ( B, 1) is OG with Follow thePerturbed Leader as the
Revised: This algorithm is a hybrid of OG and FPML in the sense that OG hybrid ( B, 1) is OG with Follow the

Original:  R ∗ T -minimizing subroutine.
Revised: Perturbed Leader as the R ⇤ T -minimizing subroutine.

Original: On the other hand, OG hybrid ( B, B ) is FTL .
Revised: On the other hand, OG hybrid ( B, B ) is FPML .

Original: Varying ˜ B allows us tointerpolate between these two algorithms by varying the budget we give to the FTL subroutines.
Revised: Varying ˜ B allows us to interpolate between these two algorithms by varying the budget we give to the FPML subroutines.

Original: On a technical note, because we are pulling ˜ B ≥ 1 arms for each greedy choice, we require aslight strengthening on the monotonicity condition which is common to many practical applicationsincluding the experiments we consider in the next section:
Revised: On a technical note, because we are pulling ˜ B  1 arms for each greedy choice, we require a slight strengthening on the monotonicity condition which is common to many practical applications including the experiments we consider in the next section:

Original: Assumption 1.
Revised: Assumption 1.

Original: In addition to monotonicity and submodularity, each job f : S → [0 , 1] also satisﬁesf ( S 1 ⊕ S 2 ⊕ S 3 )≥ f
Revised: In addition to monotonicity and submodularity, each job f : S ! [0 , 1] also satisﬁes f ( S 1  S 2  S 3 )

Original:  ( S 1 ⊕ S 3 ) for any schedules S 1 , S 2 , S 3 ∈ S .
Revised:  f ( S 1  S 3 ) for any schedules S 1 , S 2 , S 3 2 S .

Original: We can then give the following bounds:
Revised: We can then give the following bounds:

Original: Theorem 6.
Revised: Theorem 6.

Original: Under Assumption 1 and for B = (cid:100) B (cid:48) (cid:101) B ln( T ) (cid:101) , algorithm OG hybrid experiencesexpected regret
Revised: For any choices of B 0 , ˜ B , under Assumption 1 and with budget B := d Be B ln( T ) e , algorithm OG hybrid (B, ˜ B ) experiences expected regret

Original: [Equation]
Revised: O ⇣ B 0 ln( T ) T 1 / ( e B +1) ln( N ) e

Original:  relative to the best-in-hindsight ﬁxed schedule of length B (cid:48) .
Revised: B/ ( e B +1) ⌘ relative to the best-in-hindsight ﬁxed schedule of length B 0 .

Original: In particular, if (cid:101) B =(cid:100)
Revised: In particular, if e B =

Original:  ln( T ) (cid:101) (and so 
Revised: d ln( T ) e (and so B =

Original: B = B (cid:48) (cid:100) ln( T ) (cid:101) 2 ) the regret is bounded by O ( B (cid:48) ln( T ) ln ( N )) .
Revised: Bd ln( T ) e 2 ) the regret is bounded by O ( B 0 ln( T ) ln( N )) .

Original: This bound allows us the ﬂexibility of trading off a schedule budget B for how tightly we would liketo compete with the best ﬁxed schedule of length B (cid:48) ≤ B in hindsight.
Revised: This bound allows us the ﬂexibility of trading off a schedule budget B for how tightly we would like to compete with the best ﬁxed schedule of length B 0  B in hindsight.

Original: Partial feedback: Streeter and Golovin [2008] extend their algorithm to handle partial feedbackby replacing the R ∗ T regret minimizers with the bandit algorithm [-Exp
-] [Auer et al., 2002] which onlyrequires feedback on the arms which are pulled.
Revised: Partial feedback: Streeter and Golovin [2008] extend their algorithm to handle partial feedback by replacing the R ⇤ T regret minimizers with the bandit algorithm [-Exp
-]Exp3 [Auer et al., 2002] which only requires feedback on the arms which are pulled.

Original: Likewise, one can replace FTL with FTL-partialin OG hybrid to get an algorithm which gives regret bounds in a semi-bandit feedback setting (wherewe can observe f t ( S ) for any schedule S consisting of actions which were pulled in round t ).
Revised: Likewise, one can replace FPML with FPMLpartial in OG hybrid to get an algorithm which gives regret bounds in a semi-bandit feedback setting (where we can observe f t ( S ) for any schedule S consisting of actions which were pulled in round t ).

Original: Weempirically compare the partial feedback versions of OG and OG hybrid in the next section.
Revised: We empirically compare the partial feedback versions of OG and OG hybrid in the next section.

Original:  Experiments: online hyperparameter optimization
Revised:  Experiments: online hyperparameter optimization

Original: The problem of black-box optimization —where a hidden function is to be minimized using as fewevaluations as possible—has recently generated increased interest in the context of hyperparameterselection in deep learning [Snoek et al., 2012, Liu et al., 2020b, Bouthillier and Varoquaux, 2020].
Revised: The problem of black-box optimization —where a hidden function is to be minimized using as few evaluations as possible—has recently generated increased interest in the context of hyperparameter selection in deep learning [Snoek et al., 2012, Liu et al., 2020b, Bouthillier and Varoquaux, 2020].

Original: As a result, the 2020 NeurIPS BBO Challenge [Turner et al., 2021] invited participants’ optimizersto compete to ﬁnd the best possible conﬁgurations of several ML models on a number of commondatasets, given a limited budget of training cycles for each.
Revised: As a result, the 2020 NeurIPS BBO Challenge [Turner et al., 2021] invited participants’ optimizers to compete to ﬁnd the best possible conﬁgurations of several ML models on a number of common datasets, given a limited budget of training cycles for each.

Original: One of the key ﬁndings was thatsophisticated new algorithms are normally outperformed on average by techniques that ensembleexisting methods [Liu et al., 2020a], i.e. optimizers tend to have varying strengths and weaknessesthat are suited to different task types.
Revised: One of the key ﬁndings was that sophisticated new algorithms are normally outperformed on average by techniques that ensemble [Table] [Table] existing methods [Liu et al., 2020a], i.e. optimizers tend to have varying strengths and weaknesses that are suited to different task types.

Original: In a scenario where many hyperparameter selection problemsare to be processed (e.g. in a data center) and limited computing resources are available, it maythus be desirable to learn over time how best to choose B optimizers to apply independentlytoeach problem (e.g. to run in parallel on B available CPU cores).
Revised: In a scenario where many hyperparameter selection problems are to be processed (e [Table] g. in a data center) and limited computing resources are available, it may thus be desirable to learn over time how best to choose B optimizers to apply independently to each problem (e [Table]

Revised: g. to run in parallel on B available CPU cores).

Original: This is a natural partial feedbackapplication of bandit algorithms that pull multiple arms per round and receive the best score foundacross the optimizers which are run.
Revised: This is a natural partial feedback application of bandit algorithms that pull multiple arms per round and receive the best score found across the optimizers which are run.

Original: 4.1 Experimental setup
Revised: 4.1 Experimental setup

Original: We follow a similar approach to the NeurIPS BBO Challenge; the optimization problem at eachround t ∈ [ T ] is to choose the hyperparameters of either a multi-layer perceptron (MLP) or a lassoclassiﬁer for one of 184 classiﬁcation tasks from the Pembroke Machine Learning Benchmark [Olsonet al., 2017]
Revised: We follow a similar approach to the NeurIPS BBO Challenge; the optimization problem at each round t 2 [ T ] is to choose the hyperparameters of either a multi-layer perceptron (MLP) or a lasso classiﬁer for one of 184 classiﬁcation tasks from the Pembroke Machine Learning Benchmark [Olson et al., 2017]

Original: (so T = 368 ).
Revised: (so T = 368 ).

Original: At each round the bandit algorithm must select B Bayesian black-boxoptimizers from a choice of 9 to run in parallel on the current problem; the received reward 4 foreach optimizer was calculated as a [0 , 1] -normalized measure of where the best training loss attainedsits between (a) the expected best loss from a random hyperparameter search and (b) an estimate ofthe best possible loss attainable (the same approach used by the Bayesmark package [Uber, 2020]).
Revised: At each round the bandit algorithm must select B Bayesian black-box optimizers from a choice of 9 to run in parallel on the current problem; the received reward * for each optimizer was calculated as a [0 , 1] -normalized measure of where the best training loss attained sits between (a) the expected best loss from a random hyperparameter search and (b) an estimate of the best possible loss attainable (the same approach used by the Bayesmark package [Uber, 2020]).

Original: Rewards are only observed for the optimizers which are run (semi-bandit feedback).
Revised: Rewards are only observed for the optimizers which are run (semi-bandit feedback).

Original: For each budgetlevel B = 1 , . . .
Revised: For each budget level B = 1 , . . .

Original: , 6 , we ran a benchmarking study comparing the performance of the following banditalgorithms in this setting: (1) FTL -partial ( B ) from Section 2.1, using geometric sampling [Neu and Bartók, 2013] to construct unbiased cost estimates (see appendix for details).
Revised: , 6 , we ran a benchmarking study comparing the performance of the following bandit algorithms in this setting: (1) FPML-partial ( B ) from Section 2.1, using geometric sampling [Neu and Bartók, 2013] to construct unbiased cost estimates (see appendix for details).

Original: (2) OG hybrid ( B, ˜ B )from Section 3, with FTL-partial as a subroutine.
Revised: (2) OG hybrid ( B, ˜ B ) from Section 3, with FPML-partial as a subroutine.

Original: We do this for varying values of ˜ B to see howperformance changes.
Revised: We do this for varying values of ˜ B to see how performance changes.

Original: (3) OG ( B ) , the partial feedback version of the original online greedy algorithmfrom Streeter and Golovin [2008].
Revised: (3) OG ( B ) , the partial feedback version of the original online greedy algorithm from Streeter and Golovin [2008].

Original: We benchmark performance against BIH ( B ) , the score of the Best ﬁxed subset of size B In Hindsight.
Revised: We benchmark performance against BIH ( B ) , the score of the Best ﬁxed subset of size B In Hindsight.

Original: In all cases the ε parameter for the bandit subroutines
Revised: In all cases the " parameter for the bandit subroutines

Original: FTL-partial and Exp3 was set to their theoretically optimal values given B , N and T without anyﬁne-tuning (for FTL-partial we use Proposition 5).
Revised: FPML-partial and Exp3 was set to their theoretically optimal values given B , N and T without any ﬁne-tuning (for FPML-partial we use Proposition 5).

Original: We run each algorithm setting 100 times toestimate the mean and standard deviation of the performance.
Revised: We run each algorithm setting 100 times to estimate the mean and standard deviation of the performance.

Original: [Table] 4.2 Results and discussion
Revised: 4.2 Results and discussion

Original: As expected, OG hybrid ( B, 1) has very similar performance to OG ( B ) in all cases because theyimplement essentially the same algorithm (the difference being due to different choices of one-armpulling R ∗ T minimizers, FTL-partial (1) and Exp3 ).
Revised: As expected, OG hybrid ( B, 1) has very similar performance to OG ( B ) in all cases because they implement essentially the same algorithm (the difference being due to different choices of one-arm pulling R ⇤ T minimizers, FPML-partial (1) and Exp3 ).

Original: However, we notice that in every instance,allocating more budget to the FTL-partial subroutine in OG hybrid (increasing ˜ B ) while keeping theoverall budget constant improved the average score performance.
Revised: However, we notice that in every instance, allocating more budget to the FPML-partial subroutine in OG hybrid (increasing ˜ B ) while keeping the overall budget constant improved the average score performance.

Original: This means that OG hybrid wasalways at least as good as OG for all parameter settings, and that FTL-partial outperformed both ofthese algorithms in all cases.
Revised: This means that OG hybrid was always at least as good as OG for all parameter settings, and that FPML-partial outperformed both of these algorithms in all cases.

Original: This is perhaps surprising, because OG and OG hybrid are designed toachieve low regret against the stronger benchmark of the best subset of B (cid:48) ≥ 1 arms in hindsight,while FTL-partial is only designed to achieve low regret with respect to the single best optimizer inhindsight.
Revised: This is perhaps surprising, because OG and OG hybrid are designed to achieve low regret against the stronger benchmark of the best subset of B 0  1 arms in hindsight, while FPML-partial is only designed to achieve low regret with respect to the single best optimizer in hindsight.

Original: Towards explaining this observation, we notice that for this particular dataset, the bestsubset of B arms in hindsight happens to be very similar to the set of the individual best performing
Revised: Towards explaining this observation, we notice that for this particular dataset, the best subset of B arms in hindsight happens to be very similar to the set of the individual best performing

Original: B arms in hindsight, and FTL achieves low regret with with respect to the latter (see Proposition 7below).
Revised: B arms in hindsight, and FPML achieves low regret with with respect to the latter (see Propositionbelow).

Original: This raises the interesting question of when certain regret objectives might be better thanothers in practice.
Revised: This raises the interesting question of when certain regret objectives might be better than others in practice.

Original: Proposition 7.
Revised: Proposition 7.

Original: In the full feedback setting, the expected regret of FTL with B arms relative to the setof the individual best performing B arms in hindsight is at most [1 − B − 1 + ln( |A| /B )] /ε
Revised: In the full feedback setting, the expected regret of FPML with B arms relative to the set of the individual best performing B arms in hindsight is at most [1  B  1 + ln( |A| /B )] / "

Original: + T (1 −exp( − εB )) .
Revised: + T (  exp(  " B )) .

Original: Synthetic tasks: We also evaluated these partial feedback algorithms in a number of syntheticenvironments, exploring examples where (a) the optimal subset of B arms, (b) the subset of B armschosen greedily, (c) the individual best performing B arms, perform in various ways relative to eachother; OG approximates (b) and FTL-partial approximates (c) , so the closeness of either of theseto (a) determines each algorithm’s performance.
Revised: Synthetic tasks: We also evaluated these partial feedback algorithms in a number of synthetic environments, exploring examples where (a) the optimal subset of B arms, (b) the subset of B arms chosen greedily, (c) the individual best performing B arms, perform in various ways relative to each other; OG approximates (b) and FPML-partial approximates (c) , so the closeness of either of these to (a) determines each algorithm’s performance.

Original: We ﬁnd that problem instances exist where OGoutperforms FTL-partial and vice versa.
Revised: We ﬁnd that problem instances exist where OG outperforms FPML-partial and vice versa.

Original: See the appendix for details.
Revised: See the appendix for details.

Original:  Connections to other problems
Revised:  Connections to other problems

Original: Arora et al.
Revised: Arora et al.

Original: [2012] surveys a wide variety of algorithmic problems and shows how they can all besolved using a R ∗ T minimizing subroutine in the standard OLwE setting with B = 1 .
Revised: [2012] surveys a wide variety of algorithmic problems and shows how they can all be solved using a R ⇤ T minimizing subroutine in the standard OLwE setting with B = 1 .

Original: The purposeof this section is to highlight the relative ease with which algorithms like FTL can sometimes beplugged in to existing algorithms which use an R ∗ T minimizer as black box subroutine, with little orno alteration to their proofs of correctness.
Revised: The purpose of this section is to highlight the relative ease with which algorithms like FPML can sometimes be plugged in to existing algorithms which use an R ⇤ T minimizer as black box subroutine, with little or no alteration to their proofs of correctness.

Original: We already saw an example of this in Section 3, wherethe regret minimizing subroutine was replaced with FTL .
Revised: We already saw an example of this in Section 3, where the regret minimizing subroutine was replaced with FPML .

Original: The resulting algorithm gave improvedregret guarantees at the cost of higher budget requirements, without changing the structure of theoriginal proof given in Streeter and Golovin [2008].
Revised: The resulting algorithm gave improved regret guarantees at the cost of higher budget requirements, without changing the structure of the original proof given in Streeter and Golovin [2008].

Original: In this section we take Linear Programming asan example, and illustrate what its plug-in algorithm looks like.
Revised: In this section we take Linear Programming as an example, and illustrate what its plug-in algorithm looks like.

Original: We are unsure whether the resultingalgorithm is necessarily useful because it requires a more powerful oracle than the one supposedin Arora et al.
Revised: We are unsure whether the resulting algorithm is necessarily useful because it requires a more powerful oracle than the one supposed in Arora et al.

Original: ; but we do think that exploring the algorithms which result from this plug-intechnique more broadly may be an area of interest for future work.
Revised: ; but we do think that exploring the algorithms which result from this plug-in technique more broadly may be an area of interest for future work.

Original: 5.1 Linear Programming
Revised: 5.1 Linear Programming

Original: We consider the Linear Programming (LP) problem from Arora et al.
Revised: We consider the Linear Programming (LP) problem from Arora et al.

Original: [2012]: Given a convex set
Revised: [2012]: Given a convex set

Original: P , an n × m matrix A with entries in R , and a vector b ∈ R n , the task is to ﬁnd an x ∈ P such that
Revised: P , an n ⇥ m matrix A with entries in R , and a vector b 2 R n , the task is to ﬁnd an x 2 P such that

Original: Ax ≥ b , or determine that no such x exists.
Revised: Ax  b , or determine that no such x exists.

Original: We assume that we have an oracle which solves thefollowing easier problem (where A i denotes the i th row of A ):
Revised: We assume that we have an oracle which solves the following easier problem (where A i denotes the i th row of A ):

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: A ( ρ, B ) -bounded oracle for ρ ≥ 0 is an algorithm, which when given a joint distribution d over [ n ] B , ﬁnds an x ∈ P such that
Revised: A ( ⇢ , B ) -bounded oracle for ⇢  0 is an algorithm, which when given a joint distribution d over [ n ] B , ﬁnds an x 2 P such that

Original: [Equation]
Revised: [Equation]

Original: or determines that no such x exists.
Revised: or determines that no such x exists.

Original: If an x is found, then ∀ i ∈ [ n ] , | A i x − b i | ≤ ρ .
Revised: If an x is found, then 8 i 2 [ n ] , | A i x  b i |  ⇢ .

Original: When B = 1 this is a simpliﬁed version of the oracle deﬁned in Arora et al.
Revised: When B = 1 this is a simpliﬁed version of the oracle deﬁned in Arora et al.

Original:  and it is equivalentto an oracle which can ﬁnd x ∈ P which satisﬁes a single linear constraint d (cid:62) Ax ≥ d (cid:62) b (whenviewing d as a vector in R n ).
Revised: [2012], and it is equivalent to an oracle which can ﬁnd x 2 P which satisﬁes a single linear constraint d > Ax  d > b (when viewing d as a vector in R n ).

Original: When B = n , solving this problem can be as hard as solving theoriginal LP problem by jointly choosing i j to have probability mass 1 on the j th linear constraint.1 <
Revised: When B = n , solving this problem can be as hard as solving the original LP problem by jointly choosing i j to have probability mass 1 on the j th linear constraint.

Original:  B < n represents an intermediate regime where the oracle needs to ﬁnd an x which ‘fools’ thejoint distribution d .
Revised: 1 < B < n represents an intermediate regime where the oracle needs to ﬁnd an x which ‘fools’ the joint distribution d .

Original: For simplicity, we assume that we have access to an oracle which takes as inputbounded cost functions c 1 , . . .
Revised: For simplicity, we assume that we have access to an oracle which takes as input bounded cost functions c 1 , . . .

Original:  c t − 1 , and outputs the joint distribution d t over arms of FTL in roundt after observing cost functions c 1 , . . . 
Revised: , c t  1 , and outputs the joint distribution d t over arms of FPML in round t after observing cost functions c 1 , . . . ,

Original: , c t − 1 .
Revised: c t  1 .

Original: In practice such an oracle could be achieved by e.g.sampling arm draws from FTL to approximate d t .
Revised: In practice such an oracle could be achieved by e.g. sampling arm draws from FPML to approximate d t .

Original: Proposition 8.
Revised: Proposition 8.

Original: Let ε > 0 .
Revised: Let " > 0 .

Original: Suppose there exists a ( ρ, B ) -bounded oracle for the feasibility problem? ∃ x ∈ P s.t.
Revised: Suppose there exists a ( ⇢ , B ) -bounded oracle for the feasibility problem 9 x 2 P s.t.

Original: Ax ≥ b .
Revised: Ax  b .

Original: Then there is an algorithm which either ﬁnds an x ∈ P s.t. ∀ i ∈{+
+}
Revised: Then there is an algorithm which either ﬁnds an x 2 P s.t. {+
+}

Original:  [ n ] , A i x ≥b i − ε , or correctly concludes that the problem is infeasible.
Revised: i 2 [ n ] , A i x  b i  " , or correctly concludes that the problem is infeasible.

Original: The algorithm makes at most T =(cid:0) 1 ε (cid:1) B +1 B (4 ρ ) B +1 B (1+ln( n )) calls to the ( ρ, B ) -bounded oracle and FTL oracle, with a total runtimeof O ( T ) .
Revised: The algorithm makes at most T =  1 "  B +1 B (4 ⇢ ) B +1 B (1 + ln( n )) calls to the ( ⇢ , B ) -bounded oracle and FPML oracle, with a total runtime of O ( T ) .

Original: The proof technique for general B ≥ 1 is essentially identical to the proof given in Arora et al.
Revised: The proof technique for general B  1 is essentially identical to the proof given in Arora et al.

Original: [2012]for B = 1 except that we are able to use a stronger bound on R ∗ T (see appendix for details).
Revised: [2012] for B = 1 except that we are able to use a stronger bound on R ⇤ T (see appendix for details).

Original: Comment on bound: When B = 1 , we require Ω(( 1 ε ) 2 ) steps in order to ﬁnd an x which is ε -closeto satisfying the constraint.
Revised: Comment on bound: When B = 1 , we require ⌦ (( 1 " ) 2 ) steps in order to ﬁnd an x which is " -close to satisfying the constraint.

Original: The ( 1 ε ) 2 term comes from the fact that when B = 1 , the average regretfor OLwE is Ω( √ T/T ) =
Revised: The ( 1 " ) 2 term comes from the fact that when B = 1 , the average regret for OLwE is ⌦ ( p T/T ) =

Original: T − 12 , so it takes T ≥ ( 1 ε ) 2 steps for the average regret to be ≤ ε .
Revised: T  12 , so it takes T  ( 1 " ) 2 steps for the average regret to be  " .

Original: Thequadratic dependence on an accuracy parameter is therefore common in many applications which use
Revised: The quadratic dependence on an accuracy parameter is therefore common in many applications which use

Original: OLwE with B = 1 as a subroutine (including Boosting [Schapire, 1990] and solving zero sum games[Freund and Schapire, 1999]).
Revised: OLwE with B = 1 as a subroutine (including Boosting [Schapire, 1990] and solving zero sum games [Freund and Schapire, 1999]).

Original: For general B ≥ 1 , we only require Ω(( 1 ε ) B +1 B ) steps  for the averageregret to be ≤ ε .
Revised: For general B  1 , we only require O (( 1 " ) B +1 B ) steps (suppressing terms related to n and ⇢ ) for the average regret to be  " .

Original: In the case of Linear Programming, this is at the expense of requiring a strongeroracle for the problem.
Revised: In the case of Linear Programming, this is at the expense of requiring a stronger oracle for the problem.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presented a new algorithm, Follow the Perturbed Top Leaders, which allows one to directlytrade off budget constraints for bounds on regret.

Original: We showed how FTL can be used as a subroutineto generate new algorithms for Online Submodular Function Optimization and Linear Programmingwhich trade off resources and oracle power for improved performance guarantees.

Original: We also highlight a number of areas for future work: (a) Lower bounds: Can we reduce the gapbetween the upper and lower bounds of this problem?

Original: Improving lower bounds may require newtechniques than are traditionally used in the OLwE setting.

Original: Plug-in algorithms: Are there othercases where using FTL in existing algorithms can lead to new theoretical results?

Original: And when arethese new algorithms practically useful

Original: Which regret benchmarks are useful in practice: Theexperiments of Section 4 showed that algorithms designed to minimize regret with respect to a singlearm can sometimes in practice outperform algorithms designed to minimize regret with respect tothe stronger benchmark of the best subset of arms.

Original: Are certain regret objectives better than others indifferent practical applications?

Original: A Section 2: Follow the the top leadersProof of Proposition 1

Original: Proof.

Original: Construct a cost sequence inductively for t = 1 , 2 , . .

Original: , T : Let S t ⊂

Original: A be the deterministicchoice of arms the bandit algorithm chooses for round t given the previously chosen cost functionsc 1 , . . .

Original: , c t − 1 .

Original: Now choose c t ( a ) = 1 if a ∈ S t , and c t ( a ) = 0 otherwise.

Original: Then the bandit algorithmachieves cost T .

Original: The total cost summed over all arms is (cid:80) Tt =

Original: 1 | S t | ≤ BT , so there must exist at leastone a (cid:48) ∈

Original: A such that (cid:80) Tt =1 c t ( a (cid:48) )

Original: ≤ BTN .

Original: Thus R ∗ T ≥ T − BTN =

Original: (1 − BN ) T .

Original: Proof of Lemma 3

Original: Proof.

Original: Let R i =

Original: (cid:80) it =1 c t ( S ∗ t )

Original: − min a ∗ ∈A (cid:80) it

Original: =1 c t ( a ∗ ) be the regret at the end of round i .

Original: Then theincrease in regret in round i is

Original: [Equation]

Original: (cid:54)∈

Original: [Equation]

Original: ] and the result follows by evaluating (cid:80) Tt =1 r t . Lemma 9. (Proof of independence for Lemma 4) Let X 1 , . . .
Revised: This paper presented a new algorithm, Follow the Perturbed Multiple

Original: , X K be jointly independent continuousrandom variables.

Original: Let i 1 , . .

Original: i k

Original: and v i 1 , . .

Original: , v i k be the indices and values of the largest k < Krandom variables, and let X :=

Original: { X i | i (cid:54)∈ { i 1 , . . .

Original: , i k } be the smallest K − k random variables.

Original: Then conditional on ( i j , v i j ) j ∈ [ k ] , the values of each X i ∈

Original: X are jointly independent.

Original: Moreover, themarginal distribution X i | ( i j , v i j ) j ∈ [ k ] for i (cid:54)∈ { i 1 , . . .

Original: , i k } is X i |

Original: X i ≤ min j ∈ [ k ] v i j .

Original: Proof.

Original: Let M :=

Original: min j ∈ [ k ] v i j .

Original: The conditional joint density function isf ( X 1 , . . .

Original: , X K | ( i j , v i j ) j ∈ [ k ]

Original: [Equation]

Original: ( X

Original: [Equation]

Original: ( i j

Original: [Equation]

Original: i j

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ∝

Original: [Equation]

Original: [-i.e. the joint density factorizes for each X j (which implies joint independence), and marginally thedensity for X j ∈ X is ∝ f ( X j ) 
-]
Revised: [-i.e. the joint density factorizes for each X j (which implies joint independence), and marginally thedensity for X j ∈ X is ∝ f ( X j ) 
-]

Original: [ X j ≤ M ] which gives the required result.

Original: Proof of Lemma

Original: Proof.

Original: Fix a round t .

Original: Consider the jointly independent random variables

Original: X a = ˜

Original: C t − 1 ( a ) for a ∈ A .

Original: Condition on the values and identities of the N − B largest of these random variables, i.e. condition on

Original: E = { ( X a ∗ ,jt − 1 , a ∗ ,jt − 1 ) } Nj = B +1 , and let M =

Original: X a ∗ ,B +1 t −be the minimum perturbed cost among these nonleading arms.

Original: Impose an ordering on A and let l 1 , . . .

Original: , l B ∈ A−{ a ∗ ,j } Nj =

Original: B +1 be the remaining arms(the top B leaders) ordered lexicographically (i.e. not necessarily in order of cumulative perturbed cost).

Original: Then it can be shown that the distribution of the random variables X l 1 , . . .

Original: , X l B conditionedon E is jointly independent, and the marginal distribution of X l j given E is X l j | ( X l j ≤ M ) (see

Original: Lemma 9).

Original: Now observe that if X l j < M − c t ( l j ) for any j ∈ [ B ] , then the event (˜ a ∗ , 1 t (cid:54)∈ ˜ S ∗ t ) isimpossible.

Original: This is because l j ∈ ˜ S ∗ t , but for any a (cid:54)∈ S ∗ t , ˜ C t ( a ) ≥ M but ˜ C t ( l j ) =

Original: X l j + c t ( l j ) < M(i.e.

Original: l j cannot be overtaken by any non-top- B -leader in round t ). Therefore we have [Equation] [Equation]
Revised: Leaders, which allows one to directly trade off budget constraints for bounds on regret.

Original: (2) follows by independence, (4) is due to the memorylessness property of the exponential distribution(with equality unless C t − 1 ( l j ) − M < 0 ), and (6) follows because 1 − e − x ≤ x for x ≥ 0 andc t ( S ∗ t ) ≥ (cid:81) a ∈ S ∗ t c t ( a )

Original: =

Original: (cid:81)

Original: Bj =1 c t ( l j ) .

Original: The ﬁnal claim follows by taking the expectation over theconditioned event E .

Original: Proof of Theorem

Original: Proof.

Original: Consider a modiﬁed version of FTL where p t ( a ) = p 1 ( a ) = p ( a ) for all t > 1 (i.e. we keepthe random perturbation ﬁxed across rounds). Then this version of FTL picks the set ˜ S ∗ t in round t ,and the regret can be bounded as 
Revised: We showed how FPML can be used as a subroutine to generate new algorithms for Online Submodular Function Optimization and Linear

Original: [Equation]

Original: The second term is ≤ 0 .

Original: The ﬁrst term can be interpreted as the regret of a modiﬁed version of MABwith a 0 th round with cost function − p , where we are only allowed to pull arms from round t = 1 .

Original: The regret increase incurred in the 0 th round is at most max a ∈A p ( a ) .

Original: For the remaining rounds, weuse Lemma 3 followed by Lemma 4 to get

Original: [Equation]

Original: Where the inequality on E [max a ∈A p (˜ a )] comes from Kalai and Vempala [2005].

Original: The ﬁnal step isto argue that the unmodiﬁed version of FTL which chooses independent noise p t ( a ) in each roundalso achieves this bound.

Original: This is immediate because both versions of the algorithm incur the sameexpected cost in each round, and E (cid:104)(cid:80) Tt =1 c t ( FTL ) (cid:105) =

Original: (cid:80) Tt =1 E

Original: [ c t ( FTL )] .

Original: Having new randomperturbations in each round is not necessary against oblivious adversaries, but is necessary to achievethe regret bound against adaptive adversaries.

Original: Lower bound:

Original: Proposition 10.

Original: (Lower bounds)

Original: In the full feedback setting, any randomized algorithm has R ∗ T ≥Ω (cid:16)(cid:0) 14 (cid:1) B (log 2 ( N ) − log 2 ( B )) (cid:17) for T ≥ Ω(log 2 ( N ) − log 2 ( B )) .

Original: Proof. First suppose N = 2 k for [-some k ∈ N . Let A 
 = A .-]
Revised: Programming which trade off resources and oracle power for [-some k ∈ N . Let A 
 = A .-]improved performance guarantees.

Original: In round t = 1 , . . .

Original: , T , we let A t be auniformally randomly chosen subset of A t − 1 of size max(2 k − t , 1) , and we let c t ( a ) =

Original: if a ∈ A t ,1 otherwise.

Original: Suppose an algorithm chooses arms S t ⊂ A in round t .

Original: Then

Original: P [ S t ∩ A t = ∅ ] ≥

Original: B − 1 (cid:89) i = (cid:18) 1 − |A t | |A t − 1 | − i (cid:19) ≥ (cid:18)

Original: 1 − |A t | |A t − 1 | − B (cid:19) B = (cid:18) 1 − 2 − B/ |A t (cid:19) B ≥ (cid:18) 1 − 3(cid:19)

Original: B = (cid:18) (cid:19) B provided that |A t | ≥ 32 B and t ≤ k , which holds when t ≤ k − log 2 ( B ) − log 2 (cid:0) 32 (cid:1) .

Original: If we set

Original: T = (cid:98) k − log 2 ( B ) − log 2 (cid:0) 32 (cid:1) (cid:99) , then the expected cost of any ﬁxed algorithm ALG is ≥ (cid:0) 14 (cid:1) B T =Ω (cid:16)(cid:0) 14 (cid:1) B (log 2 ( N ) − log 2 ( B )) (cid:17) .

Original: By construction, the cost of the best expert in hindsight is 0 .

Original: Sincethe expected regret is 0 , there exists ﬁxed cost functions c 1 , . .

Original: , c T such that the expected regret of

Original: ALG on this on this sequence is ≥ Ω (cid:16)(cid:0) 14 (cid:1) B (log 2 ( N ) − log 2 ( B )) (cid:17) . If N is not a [-power of 2, wecan just let A 
 ⊂ A be any subset-] of size 2 (cid:98) log 2 ( N ) (cid:99) and the asymptotic bounds remain the same. Proof of Proposition 5
Revised: We also highlight a [-power of 2, wecan just let A 
 ⊂ A be any subset-]number of areas for future work: (a) Lower bounds: Can we reduce the gap between the upper and lower bounds of this problem?

Original: Proof.

Original: Fix deterministic cost functions c 1 , . .

Original: . , c T .

Original: Writing F t for the σ -algebra generated by allactions and observations (as well as any other randomness) up to and including round t , for eacht ∈

Original: [ T ] let ˆ c t be a F t -measurable random function

Original: A → [0 , K ] such that E [ˆ c t ( a ) | F t − 1 ] =

Original: c t ( a ) foreach a .

Original: Assume an oblivious adversary and that w.l.o.g. instead of perturbations there is a ‘round zero’with costs ( − p ( a )) a ∈A where p ( a ) ∼ Exp( ε ) independently for each a ; deﬁne F 0 :

Original: = σ (( p ( a ))

Original: a ∈Ato be the σ -algebra generated by these and include it in each ( F t ) t ≥ 1 .

Original: Writing ˆ C i ( · ) :

Original: = (cid:80) it

Original: =

Original: 1 ˆ c t ( · ) for cumulative estimated reward and ˆ C ∗ i ( · ) :

Original: [-= (cid:80) it =
-]
Revised: [-= (cid:80) it =
-]

Original: ˆ c t ( · ) =

Original: ˆ C i ( · ) − p ( · )for the same but including the ‘round zero’ random initializations, deﬁne

Original: [Equation]

Original: for each i ∈ [ T ] .

Original: Let S i be the set of arms chosen by the algorithm at round i and a ∗ i be the best ofthese by perturbed estimated cost.

Original: We follow the argument from Lemma 3:

Original: [Equation]

Original: and so

Original: [Equation]

Original: Hence (using the tower law)

Original: [Equation]

Original: where I := { t ∈ [ T ] : a ∗ t (cid:54)∈ S t } .

Original: Noting that by Jensen’s inequality

Original: [Equation]

Original: ≥

Original: T

Original: [Equation]

Original: (where a ∗ is the best-in-hindsight arm) hence gives that the algorithm regret satisﬁes

Original: [Equation]

Original: Since E [ p ( a ∗ )] = 1 /ε as for any ﬁxed action, and max a ∈A p ( a ) is the maximum of |A| i.i.d.

Original: Exp( ε )random variables, so has expectation at most (1 + ln |A| ) /ε

Original: as argued in Kalai and Vempala [2005],taking expectations gives

Original: [Equation]

Original: It remains to upper-bound E [ |I| ] .

Original: This will bound the expected regret of the modiﬁed version of

Original: FPL which uses the same random perturbations in each round (i.e. p t ( a ) =

Original: p 1 ( a ) for t > 1 ).

Original: Sincethe expected cost incurred by the modiﬁed version of FPL doesn’t change if we draw new randomperturbations in each round (for any ﬁxed round t , the distribution of arms picked for both versionsof FPL is identical), the regret bound holds for the unmodiﬁed version of FTL as well.

Original: Fix t ∈ [ T ] and let V :=

Original: min a ∈A− S t ˆ C ∗ t − 1 ( a ) .

Original: So for any a , a ∈ S t iff ˆ C ∗ t − 1 ( a ) < V .

Original: Deﬁne

Original: E a :=

Original: { ˆ C ∗ t − 1 ( a ) <

Original: V − K } ; if this holds then a must have been ahead of every action a (cid:48) (cid:54)∈ S t byat least K and therefore cannot be overtaken by any such action, since the estimated costs are allupper-bounded by K .

Original: So{ a overtaken by some a (cid:48) (cid:54)∈ S t } ⊂ E ca .

Original: Note that{ a ∗ t (cid:54)∈ S t } =

Original: {∃ a (cid:48) ∈

Original: A − S t : ∀ a ∈ S t , a (cid:48) overtakes a at round t } =

Original: (cid:91) a (cid:48) ∈A− S t (cid:92) a ∈ S t { a (cid:48) overtakes a at round t } ⊂ (cid:92) a ∈ S t (cid:91) a (cid:48)

Original: ∈A− S t { a (cid:48) overtakes a at round t } =

Original: (cid:92) a ∈ S t { a overtaken by some a (cid:48) (cid:54)∈ S t } .

Original: Let G t :

Original: = σ ( S t , ( ˆ C ∗ t − 1 ( a )) a/ ∈ S t ) be the σ -algebra generated by the random set S t and the currentperturbed estimated cumulative costs of the actions not in it.

Original: So we have

Original: [Equation]

Original: a overtaken by some a

Original: [Equation]

Original: But, since V = min a ∈A− S t ˆ C ∗ t − 1 ( a ) , applying Lemma 9 gives us that

Original: [Equation]

Original: By the memoryless property of the exponential distribution, each term here just becomes

Original: [Equation]

Original: Thus P ( a ∗ t (cid:54)∈ S t | G t ) ≤ (1 − e − Kε ) B .

Original: Since this expression is deterministic and so triviallyindependent from the σ -algebra G t , this immediately implies that P ( a ∗ t (cid:54)∈ S t ) ≤ (1 − e − Kε ) B .

Original: The result then follows, since E [ |I| ] = (cid:80)

Original: Tt

Original: =1 P ( a ∗ t (cid:54)∈ S t ) ≤ T (1 − e − Kε ) B .

Original: B Section 3: Generalized regret bounds for Online Submodular FunctionMaximization

Original: Algorithm 2 OG hybrid ( B , (cid:101) B )

Original: Require: B ≥ (cid:101) B ≥ 1 .

Original: Assume for simplicity that (cid:101) B | B ; deﬁne L := B/ (cid:101) B .

Original: Let B 1 , . . .

Original: , B L be instances of FTL, each with budget (cid:101) B .

Original: for rounds t = 1 , . . .

Original: , T do

Original: Let S t, 0 = (cid:104)(cid:105) be the empty schedule.

Original: for i = 1 , . . .

Original: , L do

Original: Use B i to choose (cid:101) B actions a t ( i − 1) (cid:101) B +1 , . . .

Original: , a ti (cid:101) B .

Original: Set S t,i :=

Original: S t,i − 1 ⊕ (cid:104) a t

Original: ( i − 1) (cid:101) B +1 , . . .

Original: a ti (cid:101) B (cid:105) .

Original: end for

Original: Set S t := S t,L ; receive the job f t .

Original: for i = 1 , . . .

Original: , L do

Original: For each action a ∈ A feed back the cost c ( i ) t ( a ) :

Original: = 1 − ( f t ( (cid:104) a ∗ 1 ,t , . . .

Original: , a ∗ i − 1 ,t , a (cid:105) )

Original: − f t ( (cid:104) a ∗ 1 ,t , . . .

Original: , a ∗ i − 1 ,t (cid:105) )) to FTL instance B i .

Original: Deﬁne a ∗ i,t :=

Original: arg min j ∈ [ (cid:101) B ] c ( i ) t ( a ( i − 1) (cid:101) B + j ) .

Original: end for end for

Original: Proof of Theorem Before proving the theorem, we give a modiﬁcation to the original result from Streeter and Golovin[2008]. The problem setting they considered was slightly more general: Deﬁnition 4.
Revised: Improving lower bounds may require new techniques than are traditionally used in the OLwE setting.

Original: Let an action now be an activity-duration pair

Original: a = ( ν, τ ) ∈ V × (0 , ∞ )

Original: =

Original: A for someﬁxed ﬁnited set of activities V .

Original: The length (cid:96) ( S ) of a schedule S ∈ S is now the sum of the durationsof all the actions in S .

Original: Write S (cid:104) i (cid:105) for the preﬁx of length i of schedule S .

Original: The algorithm OG they introduced, which takes a budget B and experts algorithm E , is given in

Original: Algorithm 3 using our notation for ease of reference.

Original: Algorithm 3 OG( B, E )

Original: Require: B ≥ 1 .

Original: Let E 1 , . . .

Original: , E B be instances of experts

Original: algorithm E

Original: (e.g. Hedge).

Original: for rounds t = 1 , . . .

Original: , T do

Original: Let S t, 0 = (cid:104)(cid:105) be the empty schedule.

Original: for i = 1 , . . .

Original: B do

Original: Use E i to choose an action a ti =

Original: ( ν, τ ) ∈ A .

Original: With probability 1 /τ set S t,i :=

Original: S t,i − 1 ⊕ (cid:104) a ti (cid:105) , otherwise set S t,i :=

Original: S t,i − 1 .

Original: end for

Original: Set S t := S t,B ; receive the job f t .

Original: For each i ∈ [ B ] and each action a = ( ν, τ ) ∈

Original: A feed back the cost c ( i ) t ( a ) :

Original: =

Original: ( f t ( S t,i ⊕ (cid:104) a (cid:105) )

Original: − f t ( S t,i )) /τ to experts algorithm E i .

Original: end for

Original: We ﬁrst prove a lemma generalizing Theorem 6 in Streeter and Golovin [2008]:

Original: Lemma 11.

Original: Let f be any job and let ¯ G = (cid:104)

Original: ¯ g 1 , ¯ g 2 , . . .

Original: (cid:105) be an inﬁnite ‘greedy’ schedule satisfyingf ( ¯ G j ⊕ ¯ g j ) − f

Original: ( ¯ G j ) ¯ τ j ≥ max ( ν,τ ) ∈V× (0 , ∞ ) (cid:18) f

Original: ( ¯ G j ⊕ (cid:104) ( ν, τ ) (cid:105) )

Original: − f ( ¯ G j ) τ (cid:19) − ε j , j ≥for additive errors ε 1 , ε 2 , . .

Original: ≥ 0 , where ¯ g j =

Original: (¯ v j , ¯ τ j ) and ¯ G j = (cid:104)

Original: ¯ g 1 , . .

Original: , ¯ g j − 1 (cid:105) for each j ≥ 1 .

Original: Then for any L, B 0 ∈ N and for B 1 := (cid:80)

Original: Lj =1 ¯ τ j ,

Original: [Equation]

Original: where S ∗ B 0 := arg max S ∈S : (cid:96) ( S )=

Original: B 0 f ( S ) is the best schedule of length B 0 for f .

Original: Proof.

Original: For each j ∈ N write ∆ j :=

Original: f ( S ∗ B 0 ) − f ( ¯ G j ) .

Original: By Fact 2 from Streeter and Golovin [2008],for any j ∈ N , b > 0 and S ∈ S with (cid:96) ( S ) ≤ b ,

Original: [Equation]

Original: wheres j := max ( v,

Original: [Equation]

Original: ) ∈V×

Original: [Equation]

Original: [Equation]

Original: so in particular for any j

Original: [Equation]

Original: [Equation]

Original: giving ∆ j ≤ B 0 (cid:16) ∆ j −

Original: [Equation]

Original: +

Original: [Equation]

Original: Rearranging gives ∆ j +1 ≤ ∆ j (cid:16) 1 − ¯ τ j B 0 (cid:17) + ¯ τ j ε j for each j , and unrolling this inequality and usingthat 1 − ¯ τ j B 0 < 1 ∀ j as in Streeter and Golovin [2008] gives us

Original: [Equation]

Original: By deﬁnition B 1 = (cid:80)

Original: Lj =1 ¯ τ j ε j , and maximizing the product above subject to this constraint resultsin ¯ τ j = B 1 L for all j .

Original: Thus

Original: L (cid:89)

Original: [Equation]

Original: and so

Original: [Equation]

Original: giving f ( ¯ G (cid:104) T 1 (cid:105) )

Original: =

Original: f ( ¯ G L +1 ) >

Original: (1 − e − B 1 /B 0 ) f

Original: ( S ∗ B 0 ) − (cid:80)

Original: Lj =1 ¯ τ j ε j as required.

Original: Next we prove a generalized regret bound for the original OG algorithm:

Original: Lemma 12.

Original: For B ≥ B (cid:48) log T the algorithm OG, run using Hedge as the subroutine expertsalgorithm, produces a sequence of schedules S 1 , . .

Original: S B with regret

Original: E (cid:34) T (cid:88) t = f t ( S ∗ B (cid:48) ) −

Original: T (cid:88) t =

Original: f t ( S t

Original: [Equation]

Original: (cid:35)

Original: =

Original: O   E  

Original: B (cid:88)

Original: j =

Original: [Equation]

Original: T

Original: [Equation]

Original: j

Original: [Equation]     relative to S ∗ B (cid:48) :
Revised: (b) Plug-in algorithms:

Original: = arg max S ∈S : (cid:96) ( S )=

Original: B (cid:48) (cid:80) Tt =1 f t ( S ) , the best-in-hindsight ﬁxed schedule of length

Original: B (cid:48) , where R T, 1 ( E j ) is the 1-regret incurred by the j th experts algorithm.

Original: In particular, when run with Hedge as the subroutine experts algorithm, this is O (cid:0) √

Original: BT log N (cid:1) .

Original: Proof.

Original: Consider the quantity ρ B,B (cid:48) :

Original: = (cid:16) 1 − e − B/B (cid:48) (cid:17) (cid:80)

Original: Tt =1 f t ( S ∗ B (cid:48) ) − (cid:80) Tt =1 f t ( S t ) .

Original: As arguedin Streeter and Golovin [2008], we may view the sequence of actions a 1 i , . .

Original: . , a Ti selected by eachexperts algorithm E i as a single ‘meta-action’ ˜

Original: a i ∈

Original: A T ; so the schedules S 1 , . . .

Original: , S T output by OGcan be viewed as a single ‘meta-schedule’ ˜ S =

Original: (cid:104) ˜ a 1 , . .

Original: ˜ a B (cid:105) over A T which is a version of thegreedy schedule ¯ G B +1 for the job f = 1 T (cid:80) Tt =1 f t , and it may be assumed that each meta-action ˜ a ttakes unit time per job.

Original: Thus we may write

Original: [Equation]

Original: (after extending the domain of f appropriately).

Original: Applying Lemma 11 with L = B , B 0 = B (cid:48) ,

Original: B 1 = (cid:80)

Original: Bj =1 ¯ τ j = B (by the unit-time assumption) then immediately givesρ B,B (cid:48) <

Original: T

Original: B (cid:88)

Original: j = ¯ τ j ε j = T

Original: B (cid:88)

Original: j = ε j .

Original: Taking expectations,

Original: E [ ρ B,B (cid:48) ] ≤ T

Original: B (cid:88)

Original: j =

Original: E [ ε j ] =

Original: T

Original: B (cid:88)

Original: j =

Original: E (cid:20) R T, 1 ( E j )

Original: T (cid:21) where R T, 1 ( E j ) is the 1-regret incurred by the j th experts algorithm; here we used that E [ ε j ] =

Original: E [ R T, 1 ( E j ) /n ] as argued in Streeter and Golovin [2008].

Original: So E [ ρ B,B (cid:48) ] ≤ E (cid:104)(cid:80) Bj =1 R T, 1 ( E j ) (cid:105) .

Original: The result then follows quickly: since B ≥ B (cid:48) log T , so e − B/B (cid:48)

Original: ≤ e − ln T =

Original: T − 1 .

Original: Thusρ B,B (cid:48) ≥ (cid:0) 1 − T − 1 (cid:1)

Original: T (cid:88) t =

Original: f t ( S ∗ B (cid:48) )

Original: −

Original: T (cid:88) t =

Original: f t ( S t ) =

Original: R B (cid:48) − T −

Original: T (cid:88) t = f T ( S ∗ B (cid:48) )

Original: where R B (cid:48) :

Original: = (cid:80) Tt =1 f t ( S ∗ B (cid:48) )

Original: − (cid:80) Tt =1 f t ( S t ) is the regret of interest.

Original: Consequently,

Original: R B (cid:48) ≤

Original: ρ B,B (cid:48)

Original: + T −

Original: T (cid:88) t =

Original: f t ( S ∗ B (cid:48) )

Original: ≤ ρ B,B (cid:48)

Original: + T − 1 · T = ρ B,B (cid:48) + 1 .

Original: and the result follows.

Original: The bound E (cid:104)(cid:80) Bj =1 R T, 1 ( E j ) (cid:105) =

Original: O ( √ BT log N ) when using Hedge was shown in Streeter and

Original: Golovin [2008].

Original: Finally we prove the theorem on OG hybrid :

Original: Proof of Theorem 6.

Original: Note ﬁrst that under Assumption 1, any job f , any schedule S ∈ S and anysub-schedule S (cid:48) of S (i.e. the actions of S (cid:48) appear in order in S ) satisfyf ( S ) ≥

Original: f ( S (cid:48) )

Original: ; this is immediate using monotonicity and induction.

Original: Suppose for each i ∈

Original: [ L ] there is a ﬁctional experts algorithm (classical full feedback multi-armedbandit algorithm)

Original: E i which picks a ∗ i,t at each round t , and consider a hypothetical instance of thestandard algorithm OG run with time allowance L and these ﬁctional experts algorithms E 1 , . . .

Original: , E Las subroutines.

Original: Since L ≥ B (cid:48) log T (by our assumption that B ≥ B (cid:48) (cid:101) B log T ), by Lemma 12 the B (cid:48) -regret of our

Original: OG instance is upper-bounded in expectation by (cid:80) L i =1 R 1 ( E i ) , where R 1 ( E i ) is the total 1-regretexperienced by E i .

Original: But the payoff received by this OG instance at each round t is f ( (cid:104) a ∗ 1 ,t , . . .

Original: , a ∗ L,t (cid:105) ) , which by Appendix B is upper-bounded by f ( S t ) , the payoff of OG hybrid , since the actions a ∗ 1 ,t , . .

Original: . , a ∗ L,t appearin order in S t .

Original: So the B (cid:48) -regret R B (cid:48) of OG hybrid must be at most that of our ﬁctional OG instance,giving the upper bound

Original: [Equation]

Original: It remains to argue how large each of the regret of each of these ‘ﬁctional’ experts algorithms E i is.

Original: Writing a ∗∗ i = arg min a ∈A (cid:80)

Original: Tt =1 c ( i ) t ( a ) for the best-in-hindsight ﬁxed action under the costs passedto these subroutines, the regret incurred by E i is therefore

Original: [Equation]

Original: where R 1 ( B i ) is the 1-regret incurred by multitasking bandit algorithm B i .

Original: So by Appendix B

Original: [Equation]

Original: where E [ R 1 ( B )] is the expected 1-regret of any of the instances B 1 , . .

Original: , B L of B .

Original: Proof of Proposition

Original: Sketch proof.

Original: This is a special case of the more general result that the expected regret relative to thebest-in-hindsight ﬁxed set of size B (cid:48) is at most

Original: [Equation]

Original: where err B (cid:48) is the difference in cumulative cost between the best-in-hindsight set of B (cid:48)

Original: actions andthe set of the top B (cid:48) actions in hindsight on the given problem instance.

Original: The proof of this is a simple adaptation of the 1-regret argument, using “an action not in the top Benters the best B (cid:48) -set" as the event of interest; use the harmonic series form of the expectation of themax of exponential random variables to get a lower bound, and use a binomial counting argument tobound the probability of the event.

Original: [Table]

Original: C ExperimentsC.1 Full comparison of OG hybrid

Original: In Table 2 we give a more detailed comparison of FTL and OG with various instantiations of OG hybridon the hyperparameter-selection task from Section 4.

Original: Speciﬁcally, we include for each B and eachpossible pair ( B 1 , B 2 ) s.t.

Original: B 1 B 2 =

Original: B a version of OG hybrid with B 2 internal boxes and arm budget

Original: B 1 per box.

Original: As can be seen, in all cases decreasing the greediness and adding more arms per box isbeneﬁcial in this application.

Original: C.2 Synthetic tasks

Original: In this section we evaluate our algorithms on three synthetic tasks.

Original: In all cases,

Original: • let S ∗ be the best-in-hindsight set of B arms;

Original: • let S greedy be the greedy choice of B arms in hindsight;

Original: • let S top be the top B arms in hindsight.

Original: Task 1: The ﬁrst environment is one where S ∗ =

Original: S greedy and this set does better than S top ;greediness is better than picking the top B arms.

Original: There are |A| = 15 available arms and two typesof round, A and B , which occur with equal probability; costs are distributed within each roundaccording to Table 3.

Original: So the best ﬁxed arm set of any size up to 10 will be split evenly across arms{ 1 , 2 , 3 , 4 , 5 } and arms { 11 , 12 , 13 , 14 , 15 } —and will be the greedy choice—but for B ≤ 5 the top

Original: B arms will always be in { 1 , 2 , 3 , 4 , 5 } .

Original: We see in Fig. 1 that FTL does not outperform the greedyalgorithms on this task. Task 2: The second environment is one where (approximately) S ∗ = S greedy =
Revised: Are there other cases where using FPML in existing algorithms can lead to new theoretical results?

Original: S top ; greedinessis good but no better than picking the top B arms.

Original: There are |A| =

Original: 10 available arms and costsare distributed according to Table 4; because there are no groups of anticorrelated actions, the

Original: [Table]

Original: performance gap between the best set and the top B arms is trivially small.

Original: The results in Fig.

Original: 2 showthat FTL outperforms the greedy algorithms on this task.

Original: Task 3: The third environment is one where S ∗ =

Original: S top and this set does better better than S greedy ;greediness is worse than just picking the top B arms.

Original: Suppose there are |A| = 4 available arms and abudget of B = 3 .

Original: Costs are deterministic and listed in Table 5 for some parameter δ which we setto 0.01.

Original: The top 3 arms are S top = { 1 , 3 , 4 } and this is also the best-in-hindsight set S ∗ , incurringminimum cost 0 at each round.

Original: A quick calculation shows that the greedy choice S greedy is either{ 1 , 2 , 3 } or { 1 , 2 , 4 } , though, and either of these sets incur an average minimum cost of 1 / 8 − δ/ 4 ,substantially higher.

Original: Our empirical results in Table 6 show this gap in practice.

Original: [Table]

Original: [Table]

Original: [Table]

Original: C.3 Geometric resampling

Original: The geometric resampling technique used in the second and third partial feedback versions of FTL inthe experiments is adapted from Neu and Bartók [2013].

Original: At each round cost estimates

Original: [Equation]

Original: if a was pulled , otherwise are made, where ˆ q t,a is an estimate of the probability q t,a :=

Original: P ( arm a pulled at round t ) .

Original: Theseestimates are made by sampling 1ˆ q t,a ∼ Geom( q t,a ) , which is done by repeating the algorithm’sexecution at this round and counting how many trials are needed until a is pulled again.

Original: In practice,the number of repetitions must be capped and this introduces some bias to the estimates, but this isnot problematic in practice.

Original: In fact, there is a bias variance trade-off, because K = max a ∈A | ˆ c t ( a ) |is bounded by the number of samples we take.

Original: Therefore more samples lead to lower bias but highervariance.

Original: Using bounds similar to those of Proposition 5 as a guide (the bounds of Proposition 5 weresubsequently reﬁned after the experiments were concluded), we picked the number of samples to be(cid:18)

Original: [Equation]

Original: K = (cid:18) N (cid:16) TN ln ( N ) (cid:17) ˜ B (cid:19) 1 / (2 ˜ B +1) and ε = (cid:18) ln ( N )
Revised: And when are these new algorithms practically useful?

Original: T (cid:16) ln( N ) TN (cid:17) ˜ B (cid:19) 1 / (2 ˜ B +1) ,where ˜ B is the budget of each FTL-partial box.

Original: [Table]

Original: These estimators make complete use of the information received at each round, unlike the simpleone-arm uniform sampling, B arms exploiting version of FTL with partial feedback mentioned in

Original: Section 2.1.

Original: Moreover, the construction of cost estimates means no explicit exploration is necessary;an arm that hasn’t been pulled for several rounds will be overtaken in estimated cumulative cost byones that have, and so will eventually be pulled again, thus inducing a self-stabilizing property thatwould not occur if we used the same technique toestimate rewards r t ( a ) : = 1 − c t ( a ) instead. C.4 Methods Reward deﬁnitions: For the black-box optimization experiments in Section 4, the reward ( 1 − cost)for each black-box optimizer on each machine learning task (i.e. round) was deﬁned as follows.
Revised: (c) Which regret benchmarks are useful in practice: The experiments of Section 4 showed that algorithms designed to minimize regret with respect to a single arm can sometimes in practice outperform algorithms designed to minimize regret with respect to the stronger benchmark of the best subset of arms.

Original: Thisapproach was inspired heavily by the Bayesmark package used in the 2020 NeurIPS BBO Challengeand which we based our implementation on [Uber, 2020].

Original: Fix a round t and an optimizer a .

Original: Let opt t be an estimate of the global minimum classiﬁcation/regression loss achievable (at validation, not test) on the task corresponding to round t .

Original: Deﬁnerand t to be the mean performance of a random hyperparameter search on this task (i.e. the smallestloss achieved using any hyperparameter in the random search, averaged over trials).

Original: 6 Finally, letloss t ( a ) be the actual averaged minimum loss of the optimizer a on this problem.

Original: The reward is then deﬁned as

Original: [Equation]

Original: Conceptually, the reward is 0 when optimizer a performs as badly as a random search, and 1 when itperforms as well as is possible on this task.

Original: As per usual, the reward for a bandit algorithm

Original: selecting multiple optimizers at each round is thencalculated as the maximum of the rewards of each optimizer (equivalent to the minimum of costs).

Original: Bayesian optimizers used: The nine black-box optimization algorithms we ran the experiments in

Original: Section 4 over were as follows:

Original: Hyperopt [Bergstra et al., 2015]

Original: The AUCBanditMetaTechniqueA technique from OpenTuner [Ansel et al., 2014]

Original: The PSO_GA_Bandit technique from OpenTuner [Ansel et al., 2014]

Original: The PSO_GA_DE technique from OpenTuner [Ansel et al., 2014]

Original: PySOT [Eriksson et al., 2019]

Original: Scikit-Optimize [Head et al., 2018] using base estimator GBRT and acquisition objective

Original: gp_hedge

Original: Scikit-Optimize [Head et al., 2018] using base estimator GP and acquisition objective

Original: gp_hedge

Original: Scikit-Optimize [Head et al., 2018] using base estimator GP and acquisition objective LCB

Original: Random search

Original: The default settings of each package were used.

Original: D Section 5: Linear Programming

Original: Proof of Proposition 8 (Note: this proof was given for B = 1 in Arora et al.

Original: [2012] with slightlytighter bounds, and essentially remains unchanged for B ≥ 1 ).

Original: Proof.

Original: We run the FTL oracle with budget B , N = n arms, and ε =

Original: ((ln( N ) + 1) /T ) 1 / ( B +1) .

Original: Inround t ∈ [ T ] we do the following: Let d t be the joint distribution over N arms returned the FTLoracle in this round.

Original: We pass d t to the ( ρ, B ) -bounded oracle, and receive either a vector x t ∈ P or that no x t exists which satisﬁes the oracle problem.

Original: Let us ﬁrst suppose that we always receive anx t for each round.

Original: Then deﬁne the cost function c t ( i ) :

Original: =

Original: A i x t − b i ∈ [ − ρ, ρ ] and pass this to FTL.

Original: After T rounds, and by scaling and translating the cost functions to lie in [0 , 1] , Theorem 2 impliesthat ∀ j ∈ [ N ] [Equation] T
Revised: Are certain regret objectives better than others in different practical applications?

Original: By assumption of the ( ρ, B ) -bounded oracle, the left hand side is ≥ 0 .

Original: When T ≥

Original: [Equation]

Original: (cid:1)

Original: [Equation]

Original: (4 ρ ) B

Original: [Equation]

Original: B (1 + ln( N )) , it follows that x

Original: := (cid:80) Tt =1 x t T satisﬁes ∀ j ∈

Original: [ N ] , A j x ≥ b j − ε .

Original: Since P is convex, x ∈ P

Original: and we are done.

Original: Now suppose that in some round t we were told theoracle problem was not solvable.

Original: We claim that we can conclude that the problem is not feasible andwe are done.

Original: This is because if ∃ x ∈ P s.t.

Original: Ax ≥ b , then E ( i 1 ,...,i B ) ∼ d (cid:2) min i ∈

Original: [Equation]

Original: i 1 ,...,i B } A i x − b i (cid:3) ≥

Original: [-E ( i 1 ,...,i B ) ∼ d (cid:2) min i ∈{ i 1 ,...,i B } 0 (cid:3) = 
-]
Revised: [-E ( i 1 ,...,i B ) ∼ d (cid:2) min i ∈{ i 1 ,...,i B } 0 (cid:3) = 
-]

Original: and so the oracle problem would be solvable.