Original: Trading Off Resource Budgets For Improved Regret Bounds
Revised: Trading Off Resource Budgets For Improved Regret Bounds

Original: Abstract
Revised: Abstract

Original: In this work we consider a variant of adversarial online learning where in each round one picks B out of N arms and incurs cost equal to the minimum of the costs of each arm chosen.
Revised: In this work we consider a variant of adversarial online learning where in each round one picks B out of N arms and incurs cost equal to the minimum of the costs of each arm chosen.

Original: We propose an algorithm called Follow the Perturbed Multiple
Revised: We propose an algorithm called Follow the Perturbed Multiple

Original: Leaders (FPML) for this problem, which we show (by adapting the techniques of Kalai and Vempala [2005]) achieves expected regret O ( T 1 B
Revised: Leaders (FPML) for this problem, which we show (by adapting the techniques of Kalai and Vempala [2005]) achieves expected regret O ( T 1 B

Original: +1 ln( N ) BB +1 ) over time horizon T relative to the single best arm in hindsight.
Revised: +1 ln( N ) BB +1 ) over time horizon T relative to the single best arm in hindsight.

Original: This introduces a trade-off between the budget B and the single-best-arm regret, and we proceed to investigate several applications of this trade-off.
Revised: This introduces a trade-off between the budget B and the single-best-arm regret, and we proceed to investigate several applications of this trade-off.

Original: First, we observe that algorithms which use standard regret minimizers as subroutines can sometimes be adapted by replacing these subroutines with FPML, and we use this to generalize existing algorithms for Online Submodular Function Maximization [Streeter and Golovin, 2008] in both the full feedback and semi-bandit feedback settings.
Revised: First, we observe that algorithms which use standard regret minimizers as subroutines can sometimes be adapted by replacing these subroutines with FPML, and we use this to generalize existing algorithms for Online Submodular Function Maximization [Streeter and Golovin, 2008] in both the full feedback and semi-bandit feedback settings.

Original: Next, we empirically evaluate our new algorithms on an online black-box hyperparameter optimization problem.
Revised: Next, we empirically evaluate our new algorithms on an online black-box hyperparameter optimization problem.

Original: Finally, we show how FPML can lead to new algorithms for
Revised: Finally, we show how FPML can lead to new algorithms for

Original: Linear Programming which require stronger oracles at the beneﬁt of fewer oracle calls.
Revised: Linear Programming which require stronger oracles at the beneﬁt of fewer oracle calls.

Original:  Introduction
Revised:  Introduction

Original: Adversarial online learning is a well-studied framework for sequential decision making with numerous applications.
Revised: Adversarial online learning is a well-studied framework for sequential decision making with numerous applications.

Original: In each round t = 1 , . . .
Revised: In each round t = 1 , . . .

Original: , T , an adversary chooses a hidden cost function c t : A !
Revised: , T , an adversary chooses a hidden cost function c t : A !

Original: [0 , 1] from a set of arms A to costs in [0 , 1] .
Revised: [0 , 1] from a set of arms A to costs in [0 , 1] .

Original: An algorithm must then choose an arm a t 2 A and incurs cost c t ( a t ) .
Revised: An algorithm must then choose an arm a t 2 A and incurs cost c t ( a t ) .

Original: In the full feedback setting (Online Learning with Experts (OLwE)), the algorithm then observes the cost function c t .
Revised: In the full feedback setting (Online Learning with Experts (OLwE)), the algorithm then observes the cost function c t .

Original: In the partial feedback setting (Multi-Armed Bandits (MAB))
Revised: In the partial feedback setting (Multi-Armed Bandits (MAB))

Original: the algorithm only observes the incurred cost c t ( a t ) .
Revised: the algorithm only observes the incurred cost c t ( a t ) .

Original: The objective is to ﬁnd algorithms which minimize regret , deﬁned as the difference between the algorithm’s cumulative cost and the cumulative cost of the single best arm in hindsight.
Revised: The objective is to ﬁnd algorithms which minimize regret , deﬁned as the difference between the algorithm’s cumulative cost and the cumulative cost of the single best arm in hindsight.

Original: In this paper we consider a search-like variant of these problems where in each round one can pick a subset of arms S t ⇢ A with | S t | =
Revised: In this paper we consider a search-like variant of these problems where in each round one can pick a subset of arms S t ⇢ A with | S t | =

Original: B  1 , and keep the arm with the smallest cost .
Revised: B  1 , and keep the arm with the smallest cost .

Original: This variant appears naturally in many settings, including:
Revised: This variant appears naturally in many settings, including:

Original: Online algorithm portfolios [Gomes and Selman, 2001]: In each round t , one receives a problem instance x t , and can pick a subset S t ⇢ A of algorithms to run in parallel to solve x t .
Revised: Online algorithm portfolios [Gomes and Selman, 2001]: In each round t , one receives a problem instance x t , and can pick a subset S t ⇢ A of algorithms to run in parallel to solve x t .

Original: For example, x t could be a boolean satisﬁability (SAT) problem, and A could be a collection of different SAT solving algorithms.
Revised: For example, x t could be a boolean satisﬁability (SAT) problem, and A could be a collection of different SAT solving algorithms.

Original: We let c t ( a ) =
Revised: We let c t ( a ) =

Original: if a solves x t and c t ( a ) =otherwise.
Revised: if a solves x t and c t ( a ) =otherwise.

Original: Then if any a 2 S t ﬁnds a solution to x t we incur 0 cost in this round.
Revised: Then if any a 2 S t ﬁnds a solution to x t we incur 0 cost in this round.

Original: Another example is online hyperparameter optimization (see Section 4).
Revised: Another example is online hyperparameter optimization (see Section 4).

Original: [Table]
Revised: [Table]

Original: Online bidding [Chen et al., 2016a]: In each round t , an auctioneer sets up a ﬁrst-price auction for bidders S t ⇢ A .
Revised: Online bidding [Chen et al., 2016a]: In each round t , an auctioneer sets up a ﬁrst-price auction for bidders S t ⇢ A .

Original: Each bidder a 2 A has a price 1  c t ( a ) they are willing to pay, and the auctioneer receives max
Revised: Each bidder a 2 A has a price 1  c t ( a ) they are willing to pay, and the auctioneer receives max

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: t ( a ) , and so maximizing revenue is equivalent to minimizing costs.
Revised: t ( a ) , and so maximizing revenue is equivalent to minimizing costs.

Original: Adaptive network routing [Awerbuch and Kleinberg, 2008]: In each round t , a network router receives a data packet x t and can pick a selection of network routes S t ⇢ A to send it to its destination via in parallel.
Revised: Adaptive network routing [Awerbuch and Kleinberg, 2008]: In each round t , a network router receives a data packet x t and can pick a selection of network routes S t ⇢ A to send it to its destination via in parallel.

Original: Let the cost c t ( a ) of a route a
Revised: Let the cost c t ( a ) of a route a

Original: be the total time taken for x t to reach its destination via a ; the router receives cost min a 2 S t c t ( a ) equal to the smallest delay.
Revised: be the total time taken for x t to reach its destination via a ; the router receives cost min a 2 S t c t ( a ) equal to the smallest delay.

Original: In many applications the budget B is a restricted resource (e.g. compute time or number of cores) we would like to keep small; this paper studies how one can trade off budget resources for better guarantees on the standard regret objective.
Revised: In many applications the budget B is a restricted resource (e.g. compute time or number of cores) we would like to keep small; this paper studies how one can trade off budget resources for better guarantees on the standard regret objective.

Original: Formally, for any randomized algorithm ALG which chooses subset S t ⇢ A in round t , and thus incurs cost c t ( S t ) :
Revised: Formally, for any randomized algorithm ALG which chooses subset S t ⇢ A in round t , and thus incurs cost c t ( S t ) :

Original: = min
Revised: = min

Original: [Equation]
Revised: [Equation]

Original: 2 S t c t (
Revised: 2 S t c t (

Original: [Equation]
Revised: [Equation]

Original: ) , deﬁne
Revised: ) , deﬁne

Original: [Equation]
Revised: [Equation]

Original: to be the worst-case expected regret of ALG relative to the single best arm in hindsight, where the expectation is with respect to the randomness of ALG .
Revised: to be the worst-case expected regret of ALG relative to the single best arm in hindsight, where the expectation is with respect to the randomness of ALG .

Original: * What guarantees can we give on R ⇤ T as a function of our budget B ?
Revised: * What guarantees can we give on R ⇤ T as a function of our budget B ?

Original: In the full feedback setting when B = 1 , this is the standard OLwE problem where it is known that R ⇤ T = ⌦ ( p T ) and there are algorithms which achieve R ⇤ T 
Revised: In the full feedback setting when B = 1 , this is the standard OLwE problem where it is known that R ⇤ T = ⌦ ( p T ) and there are algorithms which achieve R ⇤ T 

Original: [Equation]
Revised: [Equation]

Original: [Lattimore and Szepesvári, 2020], where N = |A| .
Revised: [Lattimore and Szepesvári, 2020], where N = |A| .

Original: When B = N the algorithm which chooses S t =
Revised: When B = N the algorithm which chooses S t =

Original: A in each round achieves R ⇤ T = 0 .
Revised: A in each round achieves R ⇤ T = 0 .

Original: But what bounds on R ⇤ T can be achieved in the intermediate regime when 1 < B < N ?
Revised: But what bounds on R ⇤ T can be achieved in the intermediate regime when 1 < B < N ?

Original: To the best of our knowledge this question has not been directly answered by any prior work.
Revised: To the best of our knowledge this question has not been directly answered by any prior work.

Original: 1.1 Contributions
Revised: 1.1 Contributions

Original: Theoretical results: We present a new algorithm for this learning problem called Follow
Revised: Theoretical results: We present a new algorithm for this learning problem called Follow

Original: the
Revised: the

Original: Perturbed Multiple Leaders ( FPML ), and show that in the full feedback setting R ⇤ T ( FPML ) 
Revised: Perturbed Multiple Leaders ( FPML ), and show that in the full feedback setting R ⇤ T ( FPML ) 

Original: O ( T 1 B
Revised: O ( T 1 B

Original: +1 ln( N ) BB +1 ) .
Revised: +1 ln( N ) BB +1 ) .

Original: This allows for a direct trade-off between the budget B and the regret bound (in particular, allowing resources B  ⌦ (ln( T )) leads to regret constant in T ) and recovers the familiar O ( p
Revised: This allows for a direct trade-off between the budget B and the regret bound (in particular, allowing resources B  ⌦ (ln( T )) leads to regret constant in T ) and recovers the familiar O ( p

Original: T ln( N )) bound when B = 1 .
Revised: T ln( N )) bound when B = 1 .

Original: We then show that in the semi-bandit feedback setting (where the algorithm ﬁnds out only the costs of the arms it chooses) this bound can be converted to
Revised: We then show that in the semi-bandit feedback setting (where the algorithm ﬁnds out only the costs of the arms it chooses) this bound can be converted to

Original: R ⇤ T ( FPML )  O ( T 1 B +

Revised: R ⇤ T ( FPML )  O ( T 1 B +

Original: ( K ln( N ))
Revised: ( K ln( N ))

Original: BB +1 )
Revised: BB +1 )

Original: if one has unbiased cost estimators bounded in [0 , K ] .
Revised: if one has unbiased cost estimators bounded in [0 , K ] .

Original: We also consider the more general problem of Online Submodular Function Maximization (OSFM), for which prior work gives an online greedy algorithm OG [Streeter and Golovin, 2008].
Revised: We also consider the more general problem of Online Submodular Function Maximization (OSFM), for which prior work gives an online greedy algorithm OG [Streeter and Golovin, 2008].

Original: When given a budget of B per round, OG achieves regret O ( p
Revised: When given a budget of B per round, OG achieves regret O ( p

Original: TB ln( N )) with respect to (1  e  1 ) OPT( B ), where OPT( B ) is the performance of the best ﬁxed length- B schedule (see Section 3 for a formal deﬁnition of OSFM).
Revised: TB ln( N )) with respect to (1  e  1 ) OPT( B ), where OPT( B ) is the performance of the best ﬁxed length- B schedule (see Section 3 for a formal deﬁnition of OSFM).

Original: Note that in this guarantee the regret benchmark is a function of the algorithm budget.
Revised: Note that in this guarantee the regret benchmark is a function of the algorithm budget.

Original: By replacing a subroutine in OG with FPML , we generalize OG to a new algorithm OG hybrid .
Revised: By replacing a subroutine in OG with FPML , we generalize OG to a new algorithm OG hybrid .

Original: Unlike OG , OG hybrid is able to give regret bounds against benchmarks which are decoupled from the algorithm budget.
Revised: Unlike OG , OG hybrid is able to give regret bounds against benchmarks which are decoupled from the algorithm budget.

Original: This allows one to more easily quantify the trade-off of increasing the budget against a ﬁxed regret objective.
Revised: This allows one to more easily quantify the trade-off of increasing the budget against a ﬁxed regret objective.

Original: As a special case, we are able to show that having a budget of B = B 

Revised: As a special case, we are able to show that having a budget of B = B

Original: d ln( T ) 2 e per round allows one to achieve regret O ( B 0 ln( T ) ln
Revised: d ln( T ) 2 e per round allows one to achieve regret O ( B 0 ln( T ) ln

Original: ( N ))
Revised: ( N ))

Original: with respect to
Revised: with respect to

Original: OPT( B 0 ).
Revised: OPT( B 0 ).

Original: One interpretation of this result is that if you are willing to increase your budget (e.g. runtime) by a factor of ln( T ) 2 , you are able to improve your performance guarantee benchmark from (1  e  1 ) OPT( B 0 ) to OPT( B 0 ).
Revised: One interpretation of this result is that if you are willing to increase your budget (e.g. runtime) by a factor of ln( T ) 2 , you are able to improve your performance guarantee benchmark from (1  e  1 ) OPT( B 0 ) to OPT( B 0 ).

Original: Likewise, your regret growth rate in terms of the number of rounds changes from O ( p T ) to O (ln( T )) .
Revised: Likewise, your regret growth rate in terms of the number of rounds changes from O ( p T ) to O (ln( T )) .

Original: Finally, in Section 5 we show how to use FTML to generalize a technique for solving linear programs assuming access to an oracle which solves relaxed forms of the linear program.
Revised: Finally, in Section 5 we show how to use FTML to generalize a technique for solving linear programs assuming access to an oracle which solves relaxed forms of the linear program.

Original: To obtain an " - approximate solution to the linear program requires
Revised: To obtain an " - approximate solution to the linear program requires

Original: [Equation]
Revised: [Equation]

Original: oracle calls, where the parameters ( B, ⇢ ) are related to the power of the oracle and n is the number of linear constraints.
Revised: oracle calls, where the parameters ( B, ⇢ ) are related to the power of the oracle and n is the number of linear constraints.

Original: The case B = 1 coincides with known results.
Revised: The case B = 1 coincides with known results.

Original: Experimental results: We benchmark both FPML and OG hybrid on an online black-box hyperparameter optimization problem based on the 2020 NeurIPS BBO challenge [Turner et al., 2021].
Revised: Experimental results: We benchmark both FPML and OG hybrid on an online black-box hyperparameter optimization problem based on the 2020 NeurIPS BBO challenge [Turner et al., 2021].

Original: We ﬁnd that both these new algorithms outperform OG for various compute budgets.
Revised: We ﬁnd that both these new algorithms outperform OG for various compute budgets.

Original: We are able to explain why this happens for this speciﬁc dataset, and discuss the scenarios under which each algorithm would perform better.
Revised: We are able to explain why this happens for this speciﬁc dataset, and discuss the scenarios under which each algorithm would perform better.

Original: Techniques: Minimizing R ⇤ T is an important subroutine for a large variety of applications including
Revised: Techniques: Minimizing R ⇤ T is an important subroutine for a large variety of applications including

Original: Linear Programming, Boosting, and solving zero sum games [Arora et al., 2012].
Revised: Linear Programming, Boosting, and solving zero sum games [Arora et al., 2012].

Original: Traditionally an experts algorithm such as Hedge [Littlestone and Warmuth, 1994], which pulls a single arm per round, will be used as a subroutine to minimize R ⇤ T .
Revised: Traditionally an experts algorithm such as Hedge [Littlestone and Warmuth, 1994], which pulls a single arm per round, will be used as a subroutine to minimize R ⇤ T .

Original: We highlight how in the cases of OSFM and
Revised: We highlight how in the cases of OSFM and

Original: Linear Programming, one can simply replace a single arm R ⇤ T -minimizing subroutine with FPML and get performance bounds with little or no alteration to the original proofs.
Revised: Linear Programming, one can simply replace a single arm R ⇤ T -minimizing subroutine with FPML and get performance bounds with little or no alteration to the original proofs.

Original: The resulting algorithms have improved bounds (due to improved bounds on R ⇤ T when B > 1 ) at the cost of qualitatively changing the algorithm (e.g. requiring a larger budget or more powerful oracle).
Revised: The resulting algorithms have improved bounds (due to improved bounds on R ⇤ T when B > 1 ) at the cost of qualitatively changing the algorithm (e.g. requiring a larger budget or more powerful oracle).

Original: This is signiﬁcant because it highlights the potential of how bounds on R ⇤ T when B > 1 can lead to new results in other application areas.
Revised: This is signiﬁcant because it highlights the potential of how bounds on R ⇤ T when B > 1 can lead to new results in other application areas.

Original: In Section 2.1 we also highlight how the proof techniques of Kalai and Vempala [2005] for bounding R ⇤ T in the traditional experts setting can naturally be generalized to the case when B > 1 , which is of independent interest.
Revised: In Section 2.1 we also highlight how the proof techniques of Kalai and Vempala [2005] for bounding R ⇤ T in the traditional experts setting can naturally be generalized to the case when B > 1 , which is of independent interest.

Original: 1.2 Relation to prior work
Revised: 1.2 Relation to prior work

Original: One can alternatively formulate more gewe consider as receiving the maximum reward r t ( a ) = 1  c t ( a ) of each arm chosen instead of the minimum cost.
Revised: One can alternatively formulate more gewe consider as receiving the maximum reward r t ( a ) = 1  c t ( a ) of each arm chosen instead of the minimum cost.

Original: In this maximum of rewards formulation, the problem ﬁts within the OSFM framework where (a) all actions are unit-time and (b) the submodular job function is always a maximum of rewards.
Revised: In this maximum of rewards formulation, the problem ﬁts within the OSFM framework where (a) all actions are unit-time and (b) the submodular job function is always a maximum of rewards.

Original: The rewards formulation of the problem has also been separately studied as the K-MAX problem (here K = B ) Chen et al.
Revised: The rewards formulation of the problem has also been separately studied as the K-MAX problem (here K = B ) Chen et al.

Original: [2016a].
Revised: [2016a].

Original: In the OSFM setting, Streeter and Golovin [2008] give an online greedy approximation algorithm which guarantees
Revised: In the OSFM setting, Streeter and Golovin [2008] give an online greedy approximation algorithm which guarantees

Original: E [(1  e  1 ) OPT ( B )  Reward T ]  O ( p
Revised: E [(1  e  1 ) OPT ( B )  Reward T ]  O ( p

Original: TB ln( N )) in the full feedback adversarial setting, where
Revised: TB ln( N )) in the full feedback adversarial setting, where

Original: OPT ( B ) is the cumulative reward of the best ﬁxed subset of B arms in hindsight, and Reward T is the cumulative reward of the algorithm.
Revised: OPT ( B ) is the cumulative reward of the best ﬁxed subset of B arms in hindsight, and Reward T is the cumulative reward of the algorithm.

Original: A similar bound of O
Revised: A similar bound of O

Original: ( B p
Revised: ( B p

Original: TN ln( N )) can be given in a semifeedback setting.
Revised: TN ln( N )) can be given in a semifeedback setting.

Original: Conversely in the full feedback setting, Streeter and Golovin [2007] shows that any algorithm has worst-case regret E [ OPT ( B )  Reward T ]  ⌦ ( p
Revised: Conversely in the full feedback setting, Streeter and Golovin [2007] shows that any algorithm has worst-case regret E [ OPT ( B )  Reward T ]  ⌦ ( p

Original: TB ln( N/B ))
Revised: TB ln( N/B ))

Original: when one receives the maximum of rewards in each round.
Revised: when one receives the maximum of rewards in each round.

Original: Chen et al.
Revised: Chen et al.

Original: [2016a] study the K-MAX problem and other non-linear reward functions in the stochastic combinatorial multi-armed bandit setting.
Revised: [2016a] study the K-MAX problem and other non-linear reward functions in the stochastic combinatorial multi-armed bandit setting.

Original: Assuming the rewards satisfy certain distributional assumptions, they give an algorithm which achieves distributionindependent regret bounds of E [(1  " ) OPT ( B )  Reward T ]  O ( p
Revised: Assuming the rewards satisfy certain distributional assumptions, they give an algorithm which achieves distributionindependent regret bounds of E [(1  " ) OPT ( B )  Reward T ]  O ( p

Original: TBN ln( T )) for " > 0 with semi-bandit feedback.
Revised: TBN ln( T )) for " > 0 with semi-bandit feedback.

Original: Note however that we consider the adversarial setting in this paper.
Revised: Note however that we consider the adversarial setting in this paper.

Original: More broadly, these problems fall within the combinatorial online learning setting where an algorithm may pull a subset of arms in each round.
Revised: More broadly, these problems fall within the combinatorial online learning setting where an algorithm may pull a subset of arms in each round.

Original: Much prior work has focused on combinatorial bandits where the reward is linear in the subset of arms chosen, which can model applications including online advertising and online shortest paths [Cesa-Bianchi and Lugosi, 2012, Audibert et al., 2014, Combes et al., 2015].
Revised: Much prior work has focused on combinatorial bandits where the reward is linear in the subset of arms chosen, which can model applications including online advertising and online shortest paths [Cesa-Bianchi and Lugosi, 2012, Audibert et al., 2014, Combes et al., 2015].

Original: The case of non-linear reward is comparatively less studied, but having non-linear rewards (such as max) allows one to model a wider variety of problems including online expected utility maximization
Revised: The case of non-linear reward is comparatively less studied, but having non-linear rewards (such as max) allows one to model a wider variety of problems including online expected utility maximization

Original: [Li and Deshpande, 2011, Chen et al., 2016a].
Revised: [Li and Deshpande, 2011, Chen et al., 2016a].

Original: As some examples of prior work in the stochastic setting, [Gopalan et al., 2014] uses Thompson Sampling to deal with non-linear rewards of functions of subsets of arms (including the max function), but requires the rewards to come from a known parametric distribution.
Revised: As some examples of prior work in the stochastic setting, [Gopalan et al., 2014] uses Thompson Sampling to deal with non-linear rewards of functions of subsets of arms (including the max function), but requires the rewards to come from a known parametric distribution.

Original: Chen et al.
Revised: Chen et al.

Original: [2016b] considers a model where the subset of arms pulled is randomized based on pulling a ‘super-arm’, and the reward is a non-linear function of the values of the arms pulled.
Revised: [2016b] considers a model where the subset of arms pulled is randomized based on pulling a ‘super-arm’, and the reward is a non-linear function of the values of the arms pulled.

Original: In the adversarial setting, Han et al.
Revised: In the adversarial setting, Han et al.

Original: [2021] study the combinatorial
Revised: [2021] study the combinatorial

Original: MAB problem when rewards can be expressed as a d -degree polynomial.
Revised: MAB problem when rewards can be expressed as a d -degree polynomial.

Original: In contrast to prior work which focuses on giving algorithms which compete against benchmarks which have the same budget as the algorithm, this work is concerned with the trade-off between regret bounds and budget size.
Revised: In contrast to prior work which focuses on giving algorithms which compete against benchmarks which have the same budget as the algorithm, this work is concerned with the trade-off between regret bounds and budget size.

Original: We focus on giving regret bounds against OPT (1) , and we use this result in Section 3 to get regret bounds against OPT ( B 0 ) for B 0 < B in OSFM.
Revised: We focus on giving regret bounds against OPT (1) , and we use this result in Section 3 to get regret bounds against OPT ( B 0 ) for B 0 < B in OSFM.

Original: Decoupling the regret benchmark OPT ( B 0 ) from the algorithm budget B can be useful when one would like to control the strength of a regret bound against a speciﬁc target OPT ( B 0 ) for theoretical or applied reasons.
Revised: Decoupling the regret benchmark OPT ( B 0 ) from the algorithm budget B can be useful when one would like to control the strength of a regret bound against a speciﬁc target OPT ( B 0 ) for theoretical or applied reasons.

Original: For example Arora et al.
Revised: For example Arora et al.

Original: [2012] survey a wide variety of applications which rely on bounding R ⇤ T , but bounds such as E
Revised: [2012] survey a wide variety of applications which rely on bounding R ⇤ T , but bounds such as E

Original: [(1  e  1 ) OPT ( B )  Reward T ]  O ( p
Revised: [(1  e  1 ) OPT ( B )  Reward T ]  O ( p

Original: TB ln( N )) do not immediately imply useful bounds on OPT (1)  Reward T .
Revised: TB ln( N )) do not immediately imply useful bounds on OPT (1)  Reward T .

Original:  Follow the Perturbed Multiple Leaders
Revised:  Follow the Perturbed Multiple Leaders

Original: We begin by considering the full feedback setting.
Revised: We begin by considering the full feedback setting.

Original: We ﬁrst check that allowing the algorithm to choose B > 1 arms per round, while only competing against the best single ﬁxed arm in hindsight, does not make the problem trivial.
Revised: We ﬁrst check that allowing the algorithm to choose B > 1 arms per round, while only competing against the best single ﬁxed arm in hindsight, does not make the problem trivial.

Original: We do this by showing that any deterministic algorithm with budget B < N still achieves linear regret in the number of rounds.
Revised: We do this by showing that any deterministic algorithm with budget B < N still achieves linear regret in the number of rounds.

Original: This is achieved by setting c t ( a ) = 1 if a 2 S t , c t ( a ) = 0 otherwise.
Revised: This is achieved by setting c t ( a ) = 1 if a 2 S t , c t ( a ) = 0 otherwise.

Original: Proposition 1.
Revised: Proposition 1.

Original: In the full feedback setting, any deterministic algorithm with arm budget B  N per round has worst-case regret R ⇤ T 
Revised: In the full feedback setting, any deterministic algorithm with arm budget B  N per round has worst-case regret R ⇤ T

Original: [Equation]
Revised: [Equation]

Original: Likewise, it can be shown that the algorithm which chooses a uniformly random subset of B arms in each round has worst-case expected regret at least T (1  B N ) B (achieved by having one arm have cost 0 across all rounds and every other arm having cost 1 ).
Revised: Likewise, it can be shown that the algorithm which chooses a uniformly random subset of B arms in each round has worst-case expected regret at least T (1  B N ) B (achieved by having one arm have cost 0 across all rounds and every other arm having cost 1 ).

Original: These two observations show that any solution for achieving sub-linear regret in T requires randomization which depends in some non-trivial way on the prior observed costs even when B > 1 .
Revised: These two observations show that any solution for achieving sub-linear regret in T requires randomization which depends in some non-trivial way on the prior observed costs even when B > 1 .

Original: 2.1 Generalizing Follow the Perturbed Leader
Revised: 2.1 Generalizing Follow the Perturbed Leader

Original: Choosing the current lowest perturbed-cost arm in each round, Follow the Perturbed Leader ( FPL )
Revised: Choosing the current lowest perturbed-cost arm in each round, Follow the Perturbed Leader ( FPL )

Original: [Kalai and Vempala, 2005], is a well-known regret minimization technique which achieves optimal worst-case regret against adaptive adversaries in the OLwE setting.
Revised: [Kalai and Vempala, 2005], is a well-known regret minimization technique which achieves optimal worst-case regret against adaptive adversaries in the OLwE setting.

Original: In this section we generalize the
Revised: In this section we generalize the

Original: FPL algorithm to Follow the Perturbed Multiple Leaders ( FPML ).
Revised: FPL algorithm to Follow the Perturbed Multiple Leaders ( FPML ).

Original: In each round, FPML perturbs the cumulative costs of each arm by adding noise, and then picks the B arms with lowest cumulative perturbed cost.
Revised: In each round, FPML perturbs the cumulative costs of each arm by adding noise, and then picks the B arms with lowest cumulative perturbed cost.

Original: This is precisely FPL when B = 1 .
Revised: This is precisely FPL when B = 1 .

Original: We show how one can extend the proof techniques of Kalai and Vempala [2005] in a natural way to prove that FPML achieves worst-case regret R ⇤ T ( FPML )  2 T 1 B +1 (1 + ln( N ))
Revised: We show how one can extend the proof techniques of Kalai and Vempala [2005] in a natural way to prove that FPML achieves worst-case regret R ⇤ T ( FPML )  2 T 1 B +1 (1 + ln( N ))

Original: BB +1 .
Revised: BB +1 .

Original: Require: N  B  1 , " > 0 .
Revised: Require: N  B  1 , " > 0 .

Original: Initialize the cumulative cost C 

Revised: Initialize the cumulative cost C

Original: ( a ) 0 for each arm a 2 A .
Revised: ( a ) 0 for each arm a 2 A .

Original: for round t = 1 , . . .
Revised: for round t = 1 , . . .

Original: , T do 1.
Revised: , T do 1.

Original: For each arm a 2 A , draw a noise perturbation p t ( a ) ⇠ 1 " Exp.
Revised: For each arm a 2 A , draw a noise perturbation p t ( a ) ⇠ 1 " Exp.

Original: Calculate the perturbed cumulative costs for round t  1 , ˜
Revised: Calculate the perturbed cumulative costs for round t  1 , ˜

Original: C t  1 ( a )
Revised: C t  1 ( a )

Original: [Equation]
Revised: [Equation]

Original: Pull the B arms with the lowest perturbed cumulative costs according to ˜ C t  1 .
Revised: Pull the B arms with the lowest perturbed cumulative costs according to ˜ C t  1 .

Original: Break ties arbitrarily.
Revised: Break ties arbitrarily.

Original: Update the cumulative costs for each arm
Revised: .Update the cumulative costs for each arm

Original: C t  a )
Revised: C t ( a )

Original: [Equation]
Revised: [Equation]

Original:  a
Revised:  a

Original: [Equation]
Revised: [Equation]

Original: end for
Revised: end for

Original: Theorem 2.
Revised: Theorem 2.

Original: In the full feedback setting, where S t ⇢ A is the subset of arms chosen by FPML in round t , we have:
Revised: In the full feedback setting, where S t ⇢ A is the subset of arms chosen by FPML in round t , we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, for " =
Revised: In particular, for " =

Original: ((ln( N ) + 1) /T ) 1 / ( B +1) , we have
Revised: ((ln( N ) + 1) /T ) 1 / ( B +1) , we have

Original: [Equation]
Revised: [Equation]

Original: The proof follows the same three high level steps which appear in Kalai and Vempala [2005] for FPL , but we extend these ideas to the case where B > 1 .
Revised: The proof follows the same three high level steps which appear in Kalai and Vempala [2005] for FPL , but we extend these ideas to the case where B > 1 .

Original: We ﬁrst observe that the algorithm which picks the B lowest cumulative cost arms in each round only incurs regret when the best arm in round t is not one of the
Revised: We ﬁrst observe that the algorithm which picks the B lowest cumulative cost arms in each round only incurs regret when the best arm in round t is not one of the

Original: Lemma 3.
Revised: Lemma 3.

Original: Consider a ﬁxed sequence of cost functions c 1 , . .
Revised: Consider a ﬁxed sequence of cost functions c 1 , . .

Original: . , c T .
Revised: . , c T .

Original: Let a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds, breaking ties arbitrarily.
Revised: Let a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds, breaking ties arbitrarily.

Original: Let S ⇤ t :=
Revised: Let S ⇤ t :=

Original: { a ⇤ ,jt  1 | j 2 [ B ] } be the set of the B lowest cost arms at the end of round t  1 .
Revised: { a ⇤ ,jt  1 | j 2 [ B ] } be the set of the B lowest cost arms at the end of round t  1 .

Original: Then for each i 2 [ T ] ,
Revised: Then for each i 2 [ T ] ,

Original: [Equation]
Revised: [Equation]

Original: and R i  R i  1  1 [ a ⇤ , 1 i
Revised: and R i  R i  1  1 [ a ⇤ , 1 i

Original: [Equation]
Revised: [Equation]

Original: This is a generalization of the familiar result that when B = 1 , following the leader has regret bounded by the number of times the leader is overtaken [Kalai and Vempala, 2005].
Revised: This is a generalization of the familiar result that when B = 1 , following the leader has regret bounded by the number of times the leader is overtaken [Kalai and Vempala, 2005].

Original: The second step is to argue that if the cumulative costs are perturbed slightly, it becomes unlikely that the event { a ⇤ , 1 t 62 S ⇤ t } will occur.
Revised: The second step is to argue that if the cumulative costs are perturbed slightly, it becomes unlikely that the event { a ⇤ , 1 t 62 S ⇤ t } will occur.

Original: One way to see this is as follows: ﬁx a round t , and let C t  1 ( a ) be the cumulative cost of a at the end of round t  1 .
Revised: One way to see this is as follows: ﬁx a round t , and let C t  1 ( a ) be the cumulative cost of a at the end of round t  1 .

Original: Let M :=
Revised: Let M :=

Original: C t  1 ( a ⇤ ,B +1 ) .
Revised: C t  1 ( a ⇤ ,B +1 ) .

Original: Then every a 2 S ⇤ t has C t  1 ( a )  M .
Revised: Then every a 2 S ⇤ t has C t  1 ( a )  M .

Original: If it is also true that C t  1 ( a ) <
Revised: If it is also true that C t  1 ( a ) <

Original: M  c t ( a ) for any a 2 S ⇤ t then the event { a ⇤ , 1 t 62 S ⇤ t } cannot occur.
Revised: M  c t ( a ) for any a 2 S ⇤ t then the event { a ⇤ , 1 t 62 S ⇤ t } cannot occur.

Original: This is because C t ( a ) <
Revised: This is because C t ( a ) <

Original: ( M  c t ( a ))
Revised: ( M  c t ( a ))

Original: + c t ( a ) =
Revised: + c t ( a ) =

Original: M but any a 0 2 A  S t has C t ( a 0 )  M , so a 0 6 = a ⇤ , 1 t .
Revised: M but any a 0 2 A  S t has C t ( a 0 )  M , so a 0 6 = a ⇤ , 1 t .

Original: If we had initially perturbed each C t  1 ( a ) by subtracting independent exponential noise p ( a ) ⇠ 1 " Exp, then conditional on M the event { C t  1 ( a ) <
Revised: If we had initially perturbed each C t  1 ( a ) by subtracting independent exponential noise p ( a ) ⇠ 1 " Exp, then conditional on M the event { C t  1 ( a ) <

Original: M  c t ( a ) } is jointly independent for each a 2 S ⇤ t .
Revised: M  c t ( a ) } is jointly independent for each a 2 S ⇤ t .

Original: Moreover the probability of this inequality not holding is equal to P [ p ( a ) < v + c t ( a )
Revised: Moreover the probability of this inequality not holding is equal to P [ p ( a ) < v + c t ( a )

Original: | p ( a )  v ] for v equal to the unperturbed cost of a at round t  1 minus M , which is bounded by " c t ( a ) (due to the memorylessness property of the exponential distribution).
Revised: | p ( a )  v ] for v equal to the unperturbed cost of a at round t  1 minus M , which is bounded by " c t ( a ) (due to the memorylessness property of the exponential distribution).

Original: Lemma 4.
Revised: Lemma 4.

Original: Fix a sequence of cost functions c 1 , . .
Revised: Fix a sequence of cost functions c 1 , . .

Original: . , c T .
Revised: . , c T .

Original: Let C i ( a ) =
Revised: Let C i ( a ) =

Original: P it =1 c t ( a ) and ˜ C i ( a ) =
Revised: P it =1 c t ( a ) and ˜ C i ( a ) =

Original: C i ( a )  p ( a ) be the perturbed cumulative cost of arm a at the end of round i , where p ( a ) ⇠ 1 " Exp.
Revised: C i ( a )  p ( a ) be the perturbed cumulative cost of arm a at the end of round i , where p ( a ) ⇠ 1 " Exp.

Original: Let ˜ a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds using these perturbed costs, and let ˜ S
Revised: Let ˜ a ⇤ ,jt be the j th lowest cumulative cost arm in hindsight after the ﬁrst t rounds using these perturbed costs, and let ˜ S

Original: ⇤ t =
Revised: ⇤ t =

Original: { ˜ a ⇤ ,jt  1 | j 2 [ B ] } .
Revised: { ˜ a ⇤ ,jt  1 | j 2 [ B ] } .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Again, when B = 1 this argument and bound coincides with the argument given by Kalai and
Revised: Again, when B = 1 this argument and bound coincides with the argument given by Kalai and

Original: Vempala [2005].
Revised: Vempala [2005].

Original: The ﬁnal step is to combine Lemmas 3 and 4 to argue that FPML achieves expected regret at most
Revised: The ﬁnal step is to combine Lemmas 3 and 4 to argue that FPML achieves expected regret at most

Original: E h " B
Revised: E h " B

Original: P
Revised: P

Original: Tt =1 c t ( ˜ S ⇤ t ) i with respect to the perturbed cumulative cost ˜ C T .
Revised: Tt =1 c t ( ˜ S ⇤ t ) i with respect to the perturbed cumulative cost ˜ C T .

Original: Since max a 2 A E [ p ( a )]  1+ln( N ) " we can argue we also achieve low expected regret with respect to the unperturbed cost C T .
Revised: Since max a 2 A E [ p ( a )]  1+ln( N ) " we can argue we also achieve low expected regret with respect to the unperturbed cost C T .

Original: In the setting of this paper, drawing new random perturbations p t ( a ) in each round is not strictly necessary (we can take p t ( a ) =
Revised: In the setting of this paper, drawing new random perturbations p t ( a ) in each round is not strictly necessary (we can take p t ( a ) =

Original: p 1 ( a ) for t > 1 ), but it is necessary to achieve regret bounds when cost functions can depend on prior arm choices of the algorithm (the adaptive adversarial setting).
Revised: p 1 ( a ) for t > 1 ), but it is necessary to achieve regret bounds when cost functions can depend on prior arm choices of the algorithm (the adaptive adversarial setting).

Original: In the setting of this paper where the costs are ﬁxed, the expected regret in either case is the same.
Revised: In the setting of this paper where the costs are ﬁxed, the expected regret in either case is the same.

Original: Probabilistic guarantees: One advantage of this proof technique is that the regret is bounded using the positive random variable
Revised: Probabilistic guarantees: One advantage of this proof technique is that the regret is bounded using the positive random variable

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: i .
Revised: i .

Original: This means that one can apply methods like
Revised: This means that one can apply methods like

Original: Markov inequality to give a probabilistic guarantee of small regret, which is substantially stronger than saying the regret is small in expectation.
Revised: Markov inequality to give a probabilistic guarantee of small regret, which is substantially stronger than saying the regret is small in expectation.

Original: Comments on settings of parameters: When B = 1 we recover the standard O ( p
Revised: Comments on settings of parameters: When B = 1 we recover the standard O ( p

Original: T ln( N )) regret bound for the OLwE problem.
Revised: T ln( N )) regret bound for the OLwE problem.

Original: For B > 1 , the regret growth rate as a function of the number of rounds is T 1 B +1 .
Revised: For B > 1 , the regret growth rate as a function of the number of rounds is T 1 B +1 .

Original: In particular, when B = ⌦ (ln( T )) grows slowly with the number of rounds, the expected regret becomes O (ln( N )) and does not grow with the number of rounds T .
Revised: In particular, when B = ⌦ (ln( T )) grows slowly with the number of rounds, the expected regret becomes O (ln( N )) and does not grow with the number of rounds T .

Original: If we use a tighter inequality in the proof of Lemma 4, it is possible to get constant expected regret when
Revised: If we use a tighter inequality in the proof of Lemma 4, it is possible to get constant expected regret when

Original: B = ln( T ) ln
Revised: B = ln( T ) ln

Original: ( A ) grows slowly with the number of arms and rounds.
Revised: ( A ) grows slowly with the number of arms and rounds.

Original: Lower bounds: A standard technique for constructing lower bounds in the online experts setting with B = 1 is to consider costs which are i.i.d.
Revised: Lower bounds: A standard technique for constructing lower bounds in the online experts setting with B = 1 is to consider costs which are i.i.d.

Original: Bernoulli ( p ) [Lattimore and Szepesvári, 2020].
Revised: Bernoulli ( p ) [Lattimore and Szepesvári, 2020].

Original: Unfortunately this technique fails when B > 1 because the expected cost of the minimum of B i.i.d.
Revised: Unfortunately this technique fails when B > 1 because the expected cost of the minimum of B i.i.d.

Original: Bernoulli random variables is generally smaller than the expected cost of the best arm in hindsight unless p is very close to 1 .
Revised: Bernoulli random variables is generally smaller than the expected cost of the best arm in hindsight unless p is very close to 1 .

Original: We are able to show very weak lower bounds in the full feedback setting of R ⇤ T =
Revised: We are able to show very weak lower bounds in the full feedback setting of R ⇤ T =

Original: ⌦ (ln( N )) for constant B and T = ⌦ (ln( N )) , but there is a substantial gap between this and the upper bound.
Revised: ⌦ (ln( N )) for constant B and T = ⌦ (ln( N )) , but there is a substantial gap between this and the upper bound.

Original: We think constructing stronger lower bounds is an interesting problem for future work which may require new analysis techniques.
Revised: We think constructing stronger lower bounds is an interesting problem for future work which may require new analysis techniques.

Original: Partial feedback: The semi-bandit feedback setting is a form of partial feedback where the algorithm only observes the individual costs of the arms it pulls.
Revised: Partial feedback: The semi-bandit feedback setting is a form of partial feedback where the algorithm only observes the individual costs of the arms it pulls.

Original: It can be shown that passing unbiased cost function estimates to FPML results in a similar regret bound in the semi-bandit feedback setting; the result is speciﬁc to FPML and using unbiased cost functions does not generally work for any
Revised: It can be shown that passing unbiased cost function estimates to FPML results in a similar regret bound in the semi-bandit feedback setting; the result is speciﬁc to FPML and using unbiased cost functions does not generally work for any

Original: R ⇤ T -minimizing algorithm when B > 1 because of the non-linearity of the cost function.
Revised: R ⇤ T -minimizing algorithm when B > 1 because of the non-linearity of the cost function.

Original: In the case of FPML this is not an issue because the same bounding technique using Lemma 3 holds in expectation when using unbiased cost estimators.
Revised: In the case of FPML this is not an issue because the same bounding technique using Lemma 3 holds in expectation when using unbiased cost estimators.

Original: A naïve way to generate unbiased cost estimates in this setting is to use an additional arm to uniformly sample costs; in Section 4 we explore geometric sampling
Revised: A naïve way to generate unbiased cost estimates in this setting is to use an additional arm to uniformly sample costs; in Section 4 we explore geometric sampling

Original: [Neu and Bartók, 2013] for getting unbiased cost estimates which is effective in practice.
Revised: [Neu and Bartók, 2013] for getting unbiased cost estimates which is effective in practice.

Original: Proposition 5.
Revised: Proposition 5.

Original: Deﬁne the algorithm FPML-partial which simulates FPML , passing it unbiased cost estimates ˆ c t 2 [0 , K ] at round t 2 [ T ] , * and copies the arm choices of FPML .
Revised: Deﬁne the algorithm FPML-partial which simulates FPML , passing it unbiased cost estimates ˆ c t 2 [0 , K ] at round t 2 [ T ] , * and copies the arm choices of FPML .

Original: Then we have
Revised: Then we have

Original: R ⇤
Revised: R ⇤

Original: [Equation]
Revised: [Equation]

Original: For
Revised: For

Original: [Equation]
Revised: [Equation]

Original: O (
Revised: O (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  Generalized regret bounds for Online Submodular Function Maximization
Revised:  Generalized regret bounds for Online Submodular Function Maximization

Original: Streeter and Golovin [2008] considered the more general problem of Online Submodular Function
Revised: Streeter and Golovin [2008] considered the more general problem of Online Submodular Function

Original: Maximization which captures a number of previously studied problems as special cases.
Revised: Maximization which captures a number of previously studied problems as special cases.

Original: For the sake of emphasizing the key ideas, we restrict attention to the full feedback setting where each action has unit duration.
Revised: For the sake of emphasizing the key ideas, we restrict attention to the full feedback setting where each action has unit duration.

Original: The OSFM problem in this setting is as follows:
Revised: The OSFM problem in this setting is as follows:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Deﬁne a schedule to be a ﬁnite sequence of actions * a 2 A , and let S be the set of all schedules; the length ` ( S ) of a schedule S 2 S is the number of actions it contains.
Revised: Deﬁne a schedule to be a ﬁnite sequence of actions * a 2 A , and let S be the set of all schedules; the length ` ( S ) of a schedule S 2 S is the number of actions it contains.

Original: Deﬁne a job to be a function f : S !
Revised: Deﬁne a job to be a function f : S !

Original: [0 , 1] such that for any schedules S 1 , S 2 2 S and any action a 2 A :
Revised: [0 , 1] such that for any schedules S 1 , S 2 2 S and any action a 2 A :

Original: f ( S 1 )  f ( S 1  S 2 ) and f ( S 2 )
Revised: f ( S 1 )  f ( S 1  S 2 ) and f ( S 2 )

Original:  f ( S 1  S 2 ) (monotonicity) ; 2.
Revised:  f ( S 1  S 2 ) (monotonicity) ; 2.

Original: f ( S 1  S 2  h a i )  f ( S 1  S 2 )  f ( S 1  h a i )
Revised: f ( S 1  S 2  h a i )  f ( S 1  S 2 )  f ( S 1  h a i )

Original:  f ( S 1 ) (submodularity) .
Revised:  f ( S 1 ) (submodularity) .

Original: Deﬁnition 2 (Online Submodular Function Maximization) .
Revised: Deﬁnition 2 (Online Submodular Function Maximization) .

Original: The problem consists of a game with T rounds.
Revised: The problem consists of a game with T rounds.

Original: We are given some ﬁxed B > 0 and at each round t 2 [ T ] we must choose a schedule S t 2 S with ` ( S t )  B to be evaluated by a job f t which is only revealed after our choice.
Revised: We are given some ﬁxed B > 0 and at each round t 2 [ T ] we must choose a schedule S t 2 S with ` ( S t )  B to be evaluated by a job f t which is only revealed after our choice.

Original: The goal is to maximize the cumulative output Reward T :=
Revised: The goal is to maximize the cumulative output Reward T :=

Original: P
Revised: P

Original: Tt =1 f t ( S t ) .
Revised: Tt =1 f t ( S t ) .

Original: Streeter and Golovin [2008] propose an online greedy algorithm OG which achieves the guarantee (1  e  1 ) OPT ( B )  Reward T  O ( p
Revised: Streeter and Golovin [2008] propose an online greedy algorithm OG which achieves the guarantee (1  e  1 ) OPT ( B )  Reward T  O ( p

Original: TB ln( N )) in expectation, where OPT ( B ) is the cumulative reward of the best ﬁxed schedule of length B in hindsight.
Revised: TB ln( N )) in expectation, where OPT ( B ) is the cumulative reward of the best ﬁxed schedule of length B in hindsight.

Original: In this section we explain how to use
Revised: In this section we explain how to use

Original: FPML to extend their algorithm to allow a trade-off between budget resources and regret bounds.
Revised: FPML to extend their algorithm to allow a trade-off between budget resources and regret bounds.

Original: The algorithm OG has two key ideas.
Revised: The algorithm OG has two key ideas.

Original: Suppose that we start with an empty schedule S 0 :
Revised: Suppose that we start with an empty schedule S 0 :

Original: = ; .
Revised: = ; .

Original: The ﬁrst idea is that if we knew f 1 , . . .
Revised: The ﬁrst idea is that if we knew f 1 , . . .

Original: , f T in advance, we could greedily construct S i := S i  1  h arg max a i 2 A
Revised: , f T in advance, we could greedily construct S i := S i  1  h arg max a i 2 A

Original: P
Revised: P

Original: Tt =1 f t
Revised: Tt =1 f t

Original: ( S i  1  h
Revised: ( S i  1  h

Original: a
Revised: a

Original: i
Revised: i

Original: i )  f t
Revised: i )  f t

Original: ( S i  1 ) i for i = 1 , . . .
Revised: ( S i  1 ) i for i = 1 , . . .

Original: , B , where a i is the best greedy arm in hindsight for greedy round i .
Revised: , B , where a i is the best greedy arm in hindsight for greedy round i .

Original: It can be shown that submodularity then implies (1  e  1 ) OPT ( B ) 
Revised: It can be shown that submodularity then implies (1  e  1 ) OPT ( B ) 

Original: P
Revised: P

Original: Tt =1 f ( S B ) .
Revised: Tt =1 f ( S B ) .

Original: Since we don’t know f 1 , . .
Revised: Since we don’t know f 1 , . .

Original: , f T in advance, the second idea is to run B copies of a
Revised: , f T in advance, the second idea is to run B copies of a

Original: R ⇤ T regret minimizer, where the i th copy tries to compete with achieving the same improvement of cumulative reward as the best ﬁxed greedy action a i in hindsight.
Revised: R ⇤ T regret minimizer, where the i th copy tries to compete with achieving the same improvement of cumulative reward as the best ﬁxed greedy action a i in hindsight.

Original: The regret bound on the i th copy with respect to the best greedy arm in hindsight is O ( p T ln N ) ; across the B copies one can show the net regret compared to the ofﬂine greedy solution is bounded by O ( p TB ln N ) , which is where the ﬁnal bound comes from.
Revised: The regret bound on the i th copy with respect to the best greedy arm in hindsight is O ( p T ln N ) ; across the B copies one can show the net regret compared to the ofﬂine greedy solution is bounded by O ( p TB ln N ) , which is where the ﬁnal bound comes from.

Original: In summary, OG works as follows: for each round t 2 [ T ] , run B greedy rounds.
Revised: In summary, OG works as follows: for each round t 2 [ T ] , run B greedy rounds.

Original: In greedy round i = 1 , . . .
Revised: In greedy round i = 1 , . . .

Original: , B , pull the arm a ti proposed by the i th black box R ⇤ T -minimizer, set S t,i :=
Revised: , B , pull the arm a ti proposed by the i th black box R ⇤ T -minimizer, set S t,i :=

Original: S t,i  1  h a ti i , and feed back the greedy rewards r t,i ( a ) =
Revised: S t,i  1  h a ti i , and feed back the greedy rewards r t,i ( a ) =

Original: f t ( S t,i  1  h a i )  f t ( S t,i  1 ) to the i th black box.
Revised: f t ( S t,i  1  h a i )  f t ( S t,i  1 ) to the i th black box.

Original: We propose a hybrid version of OG , called OG hybrid , which for any budget B allows us to compete asymptotically well against OPT( B 0 ) for any chosen B 0  B/ ln( T ) .
Revised: We propose a hybrid version of OG , called OG hybrid , which for any budget B allows us to compete asymptotically well against OPT( B 0 ) for any chosen B 0  B/ ln( T ) .

Original: The algorithm OG hybrid is based on the following two changes to OG :
Revised: The algorithm OG hybrid is based on the following two changes to OG :

Original: Instead of having B greedy rounds, we have B 0 ln( T ) greedy rounds.
Revised: Instead of having B greedy rounds, we have B 0 ln( T ) greedy rounds.

Original: One can show that the extra factor of ln( T ) allows one to drop the (1  e  1 ) term in the regret bound.
Revised: One can show that the extra factor of ln( T ) allows one to drop the (1  e  1 ) term in the regret bound.

Original: Instead of running a one-arm-pulling R ⇤ T minimizer in each greedy round, we run FPML which pulls b B/B 

Revised: Instead of running a one-arm-pulling R ⇤ T minimizer in each greedy round, we run FPML which pulls b B/B

Original: ln( T ) c
Revised: ln( T ) c

Original: arms.
Revised: arms.

Original: This allows us to improve the regret bound for each of the B 0 ln( T )
Revised: This allows us to improve the regret bound for each of the B 0 ln( T )

Original: R ⇤ T -minimizers and directly translates to a tighter overall regret bound.
Revised: R ⇤ T -minimizers and directly translates to a tighter overall regret bound.

Original: Besides these two changes, the algorithm is identical to that in Streeter and Golovin [2008].
Revised: Besides these two changes, the algorithm is identical to that in Streeter and Golovin [2008].

Original: More generally, let OG hybrid ( B, ˜ B ) denote the algorithm where each FPML box has a budget of ˜ B and there are B/ ˜ B greedy rounds, so that the total number of arms pulled in each round is B .
Revised: More generally, let OG hybrid ( B, ˜ B ) denote the algorithm where each FPML box has a budget of ˜ B and there are B/ ˜ B greedy rounds, so that the total number of arms pulled in each round is B .

Original: This algorithm is a hybrid of OG and FPML in the sense that OG hybrid ( B, 1) is OG with Follow the
Revised: This algorithm is a hybrid of OG and FPML in the sense that OG hybrid ( B, 1) is OG with Follow the

Original: Perturbed Leader as the R ⇤ T -minimizing subroutine.
Revised: Perturbed Leader as the R ⇤ T -minimizing subroutine.

Original: On the other hand, OG hybrid ( B, B ) is FPML .
Revised: On the other hand, OG hybrid ( B, B ) is FPML .

Original: Varying ˜ B allows us to interpolate between these two algorithms by varying the budget we give to the FPML subroutines.
Revised: Varying ˜ B allows us to interpolate between these two algorithms by varying the budget we give to the FPML subroutines.

Original: On a technical note, because we are pulling ˜ B  1 arms for each greedy choice, we require a slight strengthening on the monotonicity condition which is common to many practical applications including the experiments we consider in the next section:
Revised: On a technical note, because we are pulling ˜ B  1 arms for each greedy choice, we require a slight strengthening on the monotonicity condition which is common to many practical applications including the experiments we consider in the next section:

Original: Assumption 1.
Revised: Assumption 1.

Original: In addition to monotonicity and submodularity, each job f : S !
Revised: In addition to monotonicity and submodularity, each job f : S !

Original: [0 , 1] also satisﬁes f ( S 1  S 2  S 3 )
Revised: [0 , 1] also satisﬁes f ( S 1  S 2  S 3 )

Original:  f ( S 1  S 3 ) for any schedules S 1 , S 2 , S 3 2 S .
Revised:  f ( S 1  S 3 ) for any schedules S 1 , S 2 , S 3 2 S .

Original: We can then give the following bounds:
Revised: We can then give the following bounds:

Original: Theorem 6.
Revised: Theorem 6.

Original: For any choices of B 0 , ˜ B , under Assumption 1 and with budget B := d Be
Revised: For any choices of B 0 , ˜ B , under Assumption 1 and with budget B := d Be

Original: B ln( T ) e , algorithm OG hybrid (B, ˜ B ) experiences expected regret
Revised: B ln( T ) e , algorithm OG hybrid (B, ˜ B ) experiences expected regret

Original: O ⇣
Revised: O [Equation]

Original: B 0 ln( T ) T 1 / ( e
Revised: B 0 ln( T ) T 1 / ( e

Original: B +1)
Revised: B +1)

Original: ln( N ) e
Revised: ln( N ) e

Original: B/ ( e
Revised: B/ ( e

Original: B +1) ⌘ relative to the best-in-hindsight ﬁxed schedule of length B 0 .
Revised: B +1) ⌘ relative to the best-in-hindsight ﬁxed schedule of length B 0 .

Original: In particular, if e
Revised: In particular, if e

Original: B =
Revised: B =

Original: d ln( T ) e (and so B =
Revised: d ln( T ) e (and so B =

Original: Bd ln( T )
Revised: Bd ln( T )

Original: e 2 ) the regret is bounded by O ( B 0 ln( T ) ln( N )) .
Revised: e 2 ) the regret is bounded by O ( B 0 ln( T ) ln( N )) .

Original: This bound allows us the ﬂexibility of trading off a schedule budget B for how tightly we would like to compete with the best ﬁxed schedule of length B 0  B in hindsight.
Revised: This bound allows us the ﬂexibility of trading off a schedule budget B for how tightly we would like to compete with the best ﬁxed schedule of length B 0  B in hindsight.

Original: Partial feedback: Streeter and Golovin [2008] extend their algorithm to handle partial feedback by replacing the R ⇤ T regret minimizers with the bandit algorithm Exp3 [Auer et al., 2002] which only requires feedback on the arms which are pulled.
Revised: Partial feedback: Streeter and Golovin [2008] extend their algorithm to handle partial feedback by replacing the R ⇤ T regret minimizers with the bandit algorithm Exp3 [Auer et al., 2002] which only requires feedback on the arms which are pulled.

Original: Likewise, one can replace FPML with FPMLpartial in OG hybrid to get an algorithm which gives regret bounds in a semi-bandit feedback setting (where we can observe f t ( S ) for any schedule S consisting of actions which were pulled in round t ).
Revised: Likewise, one can replace FPML with FPMLpartial in OG hybrid to get an algorithm which gives regret bounds in a semi-bandit feedback setting (where we can observe f t ( S ) for any schedule S consisting of actions which were pulled in round t ).

Original: We empirically compare the partial feedback versions of OG and OG hybrid in the next section.
Revised: We empirically compare the partial feedback versions of OG and OG hybrid in the next section.

Original:  Experiments: online hyperparameter optimization
Revised:  Experiments: online hyperparameter optimization

Original: The problem of black-box optimization —where a hidden function is to be minimized using as few evaluations as possible—has recently generated increased interest in the context of hyperparameter selection in deep learning [Snoek et al., 2012, Liu et al., 2020b, Bouthillier and Varoquaux, 2020].
Revised: The problem of black-box optimization —where a hidden function is to be minimized using as few evaluations as possible—has recently generated increased interest in the context of hyperparameter selection in deep learning [Snoek et al., 2012, Liu et al., 2020b, Bouthillier and Varoquaux, 2020].

Original: As a result, the 2020 NeurIPS BBO Challenge [Turner et al., 2021] invited participants’ optimizers to compete to ﬁnd the best possible conﬁgurations of several ML models on a number of common datasets, given a limited budget of training cycles for each.
Revised: As a result, the 2020 NeurIPS BBO Challenge [Turner et al., 2021] invited participants’ optimizers to compete to ﬁnd the best possible conﬁgurations of several ML models on a number of common datasets, given a limited budget of training cycles for each.

Original: One of the key ﬁndings was that sophisticated new algorithms are normally outperformed on average by techniques that ensemble
Revised: One of the key ﬁndings was that sophisticated new algorithms are normally outperformed on average by techniques that ensemble

Original: [Table] [Table]
Revised: [Table] [Table]

Original: existing methods [Liu et al., 2020a], i.e. optimizers tend to have varying strengths and weaknesses that are suited to different task types.
Revised: existing methods [Liu et al., 2020a], i.e. optimizers tend to have varying strengths and weaknesses that are suited to different task types.

Original: In a scenario where many hyperparameter selection problems are to be processed (e
Revised: In a scenario where many hyperparameter selection problems are to be processed (e

Original: [Table]
Revised: [Table]

Original: g. in a data center) and limited computing resources are available, it may thus be desirable to learn over time how best to choose B optimizers to apply independently to each problem (e
Revised: g. in a data center) and limited computing resources are available, it may thus be desirable to learn over time how best to choose B optimizers to apply independently to each problem (e

Original: [Table]
Revised: [Table]

Original: g. to run in parallel on B available CPU cores).
Revised: g. to run in parallel on B available CPU cores).

Original: This is a natural partial feedback application of bandit algorithms that pull multiple arms per round and receive the best score found across the optimizers which are run.
Revised: This is a natural partial feedback application of bandit algorithms that pull multiple arms per round and receive the best score found across the optimizers which are run.

Original: 4.1 Experimental setup
Revised: 4.1 Experimental setup

Original: We follow a similar approach to the NeurIPS BBO Challenge; the optimization problem at each round t 2 [ T ] is to choose the hyperparameters of either a multi-layer perceptron (MLP) or a lasso classiﬁer for one of 184 classiﬁcation tasks from the Pembroke Machine Learning Benchmark [Olson et al., 2017]
Revised: We follow a similar approach to the NeurIPS BBO Challenge; the optimization problem at each round t 2 [ T ] is to choose the hyperparameters of either a multi-layer perceptron (MLP) or a lasso classiﬁer for one of 184 classiﬁcation tasks from the Pembroke Machine Learning Benchmark [Olson et al., 2017]

Original: (so T = 368 ).
Revised: (so T = 368 ).

Original: At each round the bandit algorithm must select B Bayesian black-box optimizers from a choice of 9 to run in parallel on the current problem; the received reward * for each optimizer was calculated as a [0 , 1] -normalized measure of where the best training loss attained sits between (a) the expected best loss from a random hyperparameter search and (b) an estimate of the best possible loss attainable (the same approach used by the Bayesmark package [Uber, 2020]).
Revised: At each round the bandit algorithm must select B Bayesian black-box optimizers from a choice of 9 to run in parallel on the current problem; the received reward * for each optimizer was calculated as a [0 , 1] -normalized measure of where the best training loss attained sits between (a) the expected best loss from a random hyperparameter search and (b) an estimate of the best possible loss attainable (the same approach used by the Bayesmark package [Uber, 2020]).

Original: Rewards are only observed for the optimizers which are run (semi-bandit feedback).
Revised: Rewards are only observed for the optimizers which are run (semi-bandit feedback).

Original: For each budget level B = 1 , . . .
Revised: For each budget level B = 1 , . . .

Original: , 6 , we ran a benchmarking study comparing the performance of the following bandit algorithms in this setting: (1) FPML-partial ( B ) from Section 2.1, using geometric sampling [Neu and Bartók, 2013] to construct unbiased cost estimates (see appendix for details).
Revised: , 6 , we ran a benchmarking study comparing the performance of the following bandit algorithms in this setting: (1) FPML-partial ( B ) from Section 2.1, using geometric sampling [Neu and Bartók, 2013] to construct unbiased cost estimates (see appendix for details).

Original: (2) OG hybrid ( B, ˜ B ) from Section 3, with FPML-partial as a subroutine.
Revised: (2) OG hybrid ( B, ˜ B ) from Section 3, with FPML-partial as a subroutine.

Original: We do this for varying values of ˜ B to see how performance changes.
Revised: We do this for varying values of ˜ B to see how performance changes.

Original: (3) OG ( B ) , the partial feedback version of the original online greedy algorithm from Streeter and Golovin [2008].
Revised: (3) OG ( B ) , the partial feedback version of the original online greedy algorithm from Streeter and Golovin [2008].

Original: We benchmark performance against BIH ( B ) , the score of the Best ﬁxed subset of size B In Hindsight.
Revised: We benchmark performance against BIH ( B ) , the score of the Best ﬁxed subset of size B In Hindsight.

Original: In all cases the " parameter for the bandit subroutines
Revised: In all cases the " parameter for the bandit subroutines

Original: FPML-partial and Exp3 was set to their theoretically optimal values given B , N and T without any ﬁne-tuning (for FPML-partial we use Proposition 5).
Revised: FPML-partial and Exp3 was set to their theoretically optimal values given B , N and T without any ﬁne-tuning (for FPML-partial we use Proposition 5).

Original: We run each algorithm setting 100 times to estimate the mean and standard deviation of the performance.
Revised: We run each algorithm setting 100 times to estimate the mean and standard deviation of the performance.

Original: 4.2 Results and discussion
Revised: 4.2 Results and discussion

Original: As expected, OG hybrid ( B, 1) has very similar performance to OG ( B ) in all cases because they implement essentially the same algorithm (the difference being due to different choices of one-arm pulling R ⇤ T minimizers, FPML-partial (1) and Exp3 ).
Revised: As expected, OG hybrid ( B, 1) has very similar performance to OG ( B ) in all cases because they implement essentially the same algorithm (the difference being due to different choices of one-arm pulling R ⇤ T minimizers, FPML-partial (1) and Exp3 ).

Original: However, we notice that in every instance, allocating more budget to the FPML-partial subroutine in OG hybrid (increasing ˜ B ) while keeping the overall budget constant improved the average score performance.
Revised: However, we notice that in every instance, allocating more budget to the FPML-partial subroutine in OG hybrid (increasing ˜ B ) while keeping the overall budget constant improved the average score performance.

Original: This means that OG hybrid was always at least as good as OG for all parameter settings, and that FPML-partial outperformed both of these algorithms in all cases.
Revised: This means that OG hybrid was always at least as good as OG for all parameter settings, and that FPML-partial outperformed both of these algorithms in all cases.

Original: This is perhaps surprising, because OG and OG hybrid are designed to achieve low regret against the stronger benchmark of the best subset of B 0  1 arms in hindsight, while FPML-partial is only designed to achieve low regret with respect to the single best optimizer in hindsight.
Revised: This is perhaps surprising, because OG and OG hybrid are designed to achieve low regret against the stronger benchmark of the best subset of B 0  1 arms in hindsight, while FPML-partial is only designed to achieve low regret with respect to the single best optimizer in hindsight.

Original: Towards explaining this observation, we notice that for this particular dataset, the best subset of B arms in hindsight happens to be very similar to the set of the individual best performing
Revised: Towards explaining this observation, we notice that for this particular dataset, the best subset of B arms in hindsight happens to be very similar to the set of the individual best performing

Original: B arms in hindsight, and FPML achieves low regret with with respect to the latter (see Propositionbelow).
Revised: B arms in hindsight, and FPML achieves low regret with with respect to the latter (see Propositionbelow).

Original: This raises the interesting question of when certain regret objectives might be better than others in practice.
Revised: This raises the interesting question of when certain regret objectives might be better than others in practice.

Original: Proposition 7.
Revised: Proposition 7.

Original: In the full feedback setting, the expected regret of FPML with B arms relative to the set of the individual best performing B arms in hindsight is at most [1  B  1 + ln( |A| /B )] / "
Revised: In the full feedback setting, the expected regret of FPML with B arms relative to the set of the individual best performing B arms in hindsight is at most [1  B  1 + ln( |A| /B )] / "

Original: + T (  exp(  " B )) .
Revised: + T (  exp(  " B )) .

Original: Synthetic tasks: We also evaluated these partial feedback algorithms in a number of synthetic environments, exploring examples where (a) the optimal subset of B arms, (b) the subset of B arms chosen greedily, (c) the individual best performing B arms, perform in various ways relative to each other; OG approximates (b) and FPML-partial approximates (c) , so the closeness of either of these to (a) determines each algorithm’s performance.
Revised: Synthetic tasks: We also evaluated these partial feedback algorithms in a number of synthetic environments, exploring examples where (a) the optimal subset of B arms, (b) the subset of B arms chosen greedily, (c) the individual best performing B arms, perform in various ways relative to each other; OG approximates (b) and FPML-partial approximates (c) , so the closeness of either of these to (a) determines each algorithm’s performance.

Original: We ﬁnd that problem instances exist where OG outperforms FPML-partial and vice versa.
Revised: We ﬁnd that problem instances exist where OG outperforms FPML-partial and vice versa.

Original: See the appendix for details.
Revised: See the appendix for details.

Original:  Connections to other problems
Revised:  Connections to other problems

Original: Arora et al.
Revised: Arora et al.

Original: [2012] surveys a wide variety of algorithmic problems and shows how they can all be solved using a R ⇤ T minimizing subroutine in the standard OLwE setting with B = 1 .
Revised: [2012] surveys a wide variety of algorithmic problems and shows how they can all be solved using a R ⇤ T minimizing subroutine in the standard OLwE setting with B = 1 .

Original: The purpose of this section is to highlight the relative ease with which algorithms like FPML can sometimes be plugged in to existing algorithms which use an R ⇤ T minimizer as black box subroutine, with little or no alteration to their proofs of correctness.
Revised: The purpose of this section is to highlight the relative ease with which algorithms like FPML can sometimes be plugged in to existing algorithms which use an R ⇤ T minimizer as black box subroutine, with little or no alteration to their proofs of correctness.

Original: We already saw an example of this in Section 3, where the regret minimizing subroutine was replaced with FPML .
Revised: We already saw an example of this in Section 3, where the regret minimizing subroutine was replaced with FPML .

Original: The resulting algorithm gave improved regret guarantees at the cost of higher budget requirements, without changing the structure of the original proof given in Streeter and Golovin [2008].
Revised: The resulting algorithm gave improved regret guarantees at the cost of higher budget requirements, without changing the structure of the original proof given in Streeter and Golovin [2008].

Original: In this section we take Linear Programming as an example, and illustrate what its plug-in algorithm looks like.
Revised: In this section we take Linear Programming as an example, and illustrate what its plug-in algorithm looks like.

Original: We are unsure whether the resulting algorithm is necessarily useful because it requires a more powerful oracle than the one supposed in Arora et al.
Revised: We are unsure whether the resulting algorithm is necessarily useful because it requires a more powerful oracle than the one supposed in Arora et al.

Original: ; but we do think that exploring the algorithms which result from this plug-in technique more broadly may be an area of interest for future work.
Revised: ; but we do think that exploring the algorithms which result from this plug-in technique more broadly may be an area of interest for future work.

Original: 5.1 Linear Programming
Revised: 5.1 Linear Programming

Original: We consider the Linear Programming (LP) problem from Arora et al.
Revised: We consider the Linear Programming (LP) problem from Arora et al.

Original: [2012]: Given a convex set
Revised: [2012]: Given a convex set

Original: P , an n ⇥ m matrix A with entries in R , and a vector b 2 R n , the task is to ﬁnd an x 2 P such that
Revised: P , an n ⇥ m matrix A with entries in R , and a vector b 2 R n , the task is to ﬁnd an x 2 P such that

Original: Ax  b , or determine that no such x exists.
Revised: Ax  b , or determine that no such x exists.

Original: We assume that we have an oracle which solves the following easier problem (where A i denotes the i th row of A ):
Revised: We assume that we have an oracle which solves the following easier problem (where A i denotes the i th row of A ):

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: A ( ⇢ , B ) -bounded oracle for ⇢  0 is an algorithm, which when given a joint distribution d over [ n ] B , ﬁnds an x 2 P such that
Revised: A ( ⇢ , B ) -bounded oracle for ⇢  0 is an algorithm, which when given a joint distribution d over [ n ] B , ﬁnds an x 2 P such that

Original: [Equation]
Revised: [Equation]

Original: or determines that no such x exists.
Revised: or determines that no such x exists.

Original: If an x is found, then 8 i 2 [ n ] , | A i x  b i |  ⇢ .
Revised: If an x is found, then 8 i 2 [ n ] , | A i x  b i |  ⇢ .

Original: When B = 1 this is a simpliﬁed version of the oracle deﬁned in Arora et al.
Revised: When B = 1 this is a simpliﬁed version of the oracle deﬁned in Arora et al.

Original: [2012], and it is equivalent to an oracle which can ﬁnd x 2 P which satisﬁes a single linear constraint d > Ax  d > b (when viewing d as a vector in R n ).
Revised: [2012], and it is equivalent to an oracle which can ﬁnd x 2 P which satisﬁes a single linear constraint d > Ax  d > b (when viewing d as a vector in R n ).

Original: When B = n , solving this problem can be as hard as solving the original LP problem by jointly choosing i j to have probability mass 1 on the j th linear constraint.
Revised: When B = n , solving this problem can be as hard as solving the original LP problem by jointly choosing i j to have probability mass 1 on the j th linear constraint.

Original: 1 < B < n represents an intermediate regime where the oracle needs to ﬁnd an x which ‘fools’ the joint distribution d .
Revised: 1 < B < n represents an intermediate regime where the oracle needs to ﬁnd an x which ‘fools’ the joint distribution d .

Original: For simplicity, we assume that we have access to an oracle which takes as input bounded cost functions c 1 , . . .
Revised: For simplicity, we assume that we have access to an oracle which takes as input bounded cost functions c 1 , . . .

Original: , c t  1 , and outputs the joint distribution d t over arms of FPML in round t after observing cost functions c 1 , . . . ,
Revised: , c t  1 , and outputs the joint distribution d t over arms of FPML in round t after observing cost functions c 1 , . . . ,

Original: c t  1 .
Revised: c t  1 .

Original: In practice such an oracle could be achieved by e.g. sampling arm draws from FPML to approximate d t .
Revised: In practice such an oracle could be achieved by e.g. sampling arm draws from FPML to approximate d t .

Original: Proposition 8.
Revised: Proposition 8.

Original: Let " > 0 .
Revised: Let " > 0 .

Original: Suppose there exists a ( ⇢ , B ) -bounded oracle for the feasibility problem 9 x 2 P s.t.
Revised: Suppose there exists a ( ⇢ , B ) -bounded oracle for the feasibility problem 9 x 2 P s.t.

Original: Ax  b .
Revised: Ax  b .

Original: Then there is an algorithm which either ﬁnds an x 2 P s.t. 

Revised: Then there is an algorithm which either ﬁnds an x 2 P s.t.

Original: i 2 [ n ] , A i x  b i  " , or correctly concludes that the problem is infeasible.
Revised: i 2 [ n ] , A i x  b i  " , or correctly concludes that the problem is infeasible.

Original: The algorithm makes at most T
Revised: The algorithm makes at most T

Original: =  1 " 
Revised: =  1 "

Original: B +1 B (4 ⇢ )
Revised: B +1 B (4 ⇢ )

Original: B +1 B (1 + ln( n )) calls to the ( ⇢ , B ) -bounded oracle and FPML oracle, with a total runtime of O ( T ) .
Revised: B +1 B (1 + ln( n )) calls to the ( ⇢ , B ) -bounded oracle and FPML oracle, with a total runtime of O ( T ) .

Original: The proof technique for general B  1 is essentially identical to the proof given in Arora et al.
Revised: The proof technique for general B  1 is essentially identical to the proof given in Arora et al.

Original: [2012] for B = 1 except that we are able to use a stronger bound on R ⇤ T (see appendix for details).
Revised: [2012] for B = 1 except that we are able to use a stronger bound on R ⇤ T (see appendix for details).

Original: Comment on bound: When B = 1 , we require ⌦ (( 1 " ) 2 ) steps in order to ﬁnd an x which is " -close to satisfying the constraint.
Revised: Comment on bound: When B = 1 , we require ⌦ (( 1 " ) 2 ) steps in order to ﬁnd an x which is " -close to satisfying the constraint.

Original: The ( 1 " ) 2 term comes from the fact that when B = 1 , the average regret for OLwE is ⌦ ( p T/T ) =
Revised: The ( 1 " ) 2 term comes from the fact that when B = 1 , the average regret for OLwE is ⌦ ( p T/T ) =

Original: T  12 , so it takes T  ( 1 " ) 2 steps for the average regret to be  " .
Revised: T  12 , so it takes T  ( 1 " ) 2 steps for the average regret to be  " .

Original: The quadratic dependence on an accuracy parameter is therefore common in many applications which use
Revised: The quadratic dependence on an accuracy parameter is therefore common in many applications which use

Original: OLwE with B = 1 as a subroutine (including Boosting [Schapire, 1990] and solving zero sum games [Freund and Schapire, 1999]).
Revised: OLwE with B = 1 as a subroutine (including Boosting [Schapire, 1990] and solving zero sum games [Freund and Schapire, 1999]).

Original: For general B  1 , we only require O (( 1 " ) B +1 B ) steps (suppressing terms related to n and ⇢ ) for the average regret to be  " .
Revised: For general B  1 , we only require O (( 1 " ) B +1 B ) steps (suppressing terms related to n and ⇢ ) for the average regret to be  " .

Original: In the case of Linear Programming, this is at the expense of requiring a stronger oracle for the problem.
Revised: In the case of Linear Programming, this is at the expense of requiring a stronger oracle for the problem.

Original:  Conclusion
Revised:  Conclusion

Original: This paper presented a new algorithm, Follow the Perturbed Multiple
Revised: This paper presented a new algorithm, Follow the Perturbed Multiple

Original: Leaders, which allows one to directly trade off budget constraints for bounds on regret.
Revised: Leaders, which allows one to directly trade off budget constraints for bounds on regret.

Original: We showed how FPML can be used as a subroutine to generate new algorithms for Online Submodular Function Optimization and Linear
Revised: We showed how FPML can be used as a subroutine to generate new algorithms for Online Submodular Function Optimization and Linear

Original: Programming which trade off resources and oracle power for improved performance guarantees.
Revised: Programming which trade off resources and oracle power for improved performance guarantees.

Original: We also highlight a number of areas for future work: (a) Lower bounds: Can we reduce the gap between the upper and lower bounds of this problem?
Revised: We also highlight a number of areas for future work: (a) Lower bounds: Can we reduce the gap between the upper and lower bounds of this problem?

Original: Improving lower bounds may require new techniques than are traditionally used in the OLwE setting.
Revised: Improving lower bounds may require new techniques than are traditionally used in the OLwE setting.

Original: (b) Plug-in algorithms:
Revised: (b) Plug-in algorithms:

Original: Are there other cases where using FPML in existing algorithms can lead to new theoretical results?
Revised: Are there other cases where using FPML in existing algorithms can lead to new theoretical results?

Original: And when are these new algorithms practically useful?
Revised: And when are these new algorithms practically useful?

Original: (c) Which regret benchmarks are useful in practice: The experiments of Section 4 showed that algorithms designed to minimize regret with respect to a single arm can sometimes in practice outperform algorithms designed to minimize regret with respect to the stronger benchmark of the best subset of arms.
Revised: (c) Which regret benchmarks are useful in practice: The experiments of Section 4 showed that algorithms designed to minimize regret with respect to a single arm can sometimes in practice outperform algorithms designed to minimize regret with respect to the stronger benchmark of the best subset of arms.

Original: Are certain regret objectives better than others in different practical applications?
Revised: Are certain regret objectives better than others in different practical applications?