Non-monotonic Resource Utilization in the Bandits with Knapsacks Problem
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Bandits with knapsacks ( BwK ) [6] is an inﬂuential model of sequential decisionmaking under uncertainty that incorporates resource consumption constraints.
Ineach round, the decision-maker observes an outcome consisting of a reward and avector of nonnegative resource consumptions, and the budget of each resource isdecremented by its consumption.
In this paper we introduce a natural generalizationof the stochastic BwK problem that allows non-monotonic resource utilization.
Ineach round, the decision-maker observes an outcome consisting of a reward anda vector of resource drifts that can be positive, negative or zero, and the budgetof each resource is incremented by its drift.
Our main result is a Markov decisionprocess (MDP) policy that has constant regret against a linear programming (LP)relaxation when the decision-maker knows the true outcome distributions.
Webuild upon this to develop a learning algorithm that has logarithmic regret againstthe same LP relaxation when the decision-maker does not know the true outcomedistributions.
We also present a reduction from BwK to our model that shows ourregret bound matches existing results [14].
Introduction
Multi-armed bandits are the quintessential model of sequential decision-making under uncertaintyin which the decision-maker must trade-off between exploration and exploitation.
They have beenstudied extensively and have numerous applications, such as clinical trials, ad placements anddynamic pricing to name a few.
We refer the reader to Bubeck and Cesa-Bianchi [7], Slivkins[17], Lattimore and Szepesvári [13] for an introduction to bandits.
An important shortcoming of thebasic stochastic bandits model is that it does not take into account resource consumption constraintsthat are present in many of the motivating applications.
For example, in a dynamic pricing applicationthe seller may be constrained by a limited inventory of items that can run out well before the end ofthe time horizon.
The bandits with knapsacks ( BwK ) model [18, 19, 5, 6] remedies this by endowingthe decision-maker with some initial budget for each of m resources.
In each round, the outcomeis a reward and a vector of nonnegative resource consumptions, and the budget of each resourceis decremented by its consumption.
The process ends when the budget of any resource becomesnonpositive.
However, even this formulation fails to model that in many applications resources canget replenished or renewed over time.
For example, in a dynamic pricing application a seller mayreceive shipments that increase their inventory level.
Contributions In this paper we introduce a natural generalization of BwK by allowing nonmonotonic resource utilization.
The decision-maker starts with some initial budget for each of mresources.
In each round, the outcome is a reward and a vector of resource drifts that can be positive,negative or zero, and the budget of each resource is incremented by its drift.
A negative drift hasthe effect of decreasing the budget akin to consumption in BwK and a positive drift has the effect of increasing the budget.
We consider two settings: (i) when the decision-maker knows the trueoutcome distributions and must design a Markov decision process (MDP) policy; and (ii) when thedecision-maker does not know the true outcome distributions and must design a learning algorithm.
[Table]
OPT CB
LP 𝑂(1)
ETCB 𝑂(log𝑇)
Not
[Table]
for BwK.
Our main contribution is an MDP policy,
ControlBudget(CB) , that has constant regretagainst a linear programming (LP) relaxation.
Such a result was not known even for BwK .
Webuild upon this to develop a learning algorithm,
ExploreThenControlBudget(ETCB) , that haslogarithmic regret against the same LP relaxation.
We also present a reduction from BwK to our modeland show that our regret bound matches existing results.
Instead of merely sampling from the optimal probability distribution over arms, our policy samplesfrom a perturbed distribution to ensure that the budget of each resource stays close to a decreasingsequence of thresholds.
The sequence is chosen such that the expected leftover budget is a constantand proving this is a key step in the regret analysis.
Our work combines aspects of related work onlogarithmic regret for BwK [9, 14].
Related Work Multi-armed bandits have a rich history and logarithmic instance-dependent regretbounds have been known for a long time [12, 3].
Since then, there have been numerous papersextending the stochastic bandits model in a variety of ways [4, 16, 11, 6, 10, 2, 1].
To the best of our knowledge, there are three papers on logarithmic regret bounds for BwK .
Flajoletand Jaillet [9] showed the ﬁrst logarithmic regret bound for BwK .
In each round, their algorithmﬁnds the optimal basis for an optimistic version of the LP relaxation, and chooses arms from theresulting basis to ensure that the average resource consumption stays close to a pre-speciﬁed level.
Even though their regret bound is logarithmic in T and inverse linear in the suboptimality gap, itis exponential in the number of resources.
Li et al.
[14] showed an improved logarithmic regretbound that is polynomial in the number of resources, but it scales inverse quadratically with thesuboptimality gap and their deﬁnition of the gap is different from the one in Flajolet and Jaillet [9].
The main idea behind improving the dependence on the number of resources is to proceed in twophases: (i) identify the set of arms and binding resources in the optimal solution; (ii) in each round,solve an adaptive, optimistic version of the LP relaxation and sample an arm from the resultingprobability distribution.
Finally, Sankararaman and Slivkins [15] show a logarithmic regret bound for
BwK against a ﬁxed-distribution benchmark.
However, the regret of this benchmark itself with theoptimal MDP policy can be as large as O ( √ T )
Preliminaries2.1 Model
Let T denote a ﬁnite time horizon, X = { 1 , . . .
, k } a set of k arms, J = { 1 , . . .
, m } denote a setof m resources, and B 0 ,j = B denote the initial budget of resource j .
In each round t ∈ [ T ] , if thebudget of any resource is less than 1 , then X t = { 1 } .
Otherwise, X t = X .
The algorithm chooses anarm x t ∈ X t and observes an outcome o t =
( r t , d t, 1 , . . .
, d t,m ) ∈ [0 , 1] ×
[ − 1 , 1] m .
The algorithmearns reward r t and the budget of resource j ∈ J is incremented by drift d t,j as B t,j = B t − 1 ,j + d t,j .
Each arm x ∈ X has an outcome distribution over [0 , 1] ×
[ − 1 , 1] m and o t is drawn from the outcomedistribution of the arm x t .
We use µ ox = ( µ rx , µ d, 1 x , . . . , µ d,mx ) to denote the expected outcome vectorof arm x consisting of the expected reward and the expected drifts for each of the m resources.
Wealso use µ d,j = ( µ d,jx : x ∈ X ) to denote the vector of expected drifts for resource j .
We assumethat arm x 0 = 1 ∈ X is a null arm with three important properties: (i) its reward is zero a.s.; (ii) thedrift for each resource is nonnegative a.s.; and (iii) the expected drift for each resource is positive.
The second and third properties of the null arm plus the model’s requirement that x t = 1 if ∃ j s.t.
B t − 1 ,j < 1 ensure that the budgets are nonnegative a.s. and can be safely increased from 0 .
Our model is intended to capture applications featuring resource renewal, such as the following.
Ineach round, each resource gets replenished by some random amount independent of the past andthe chosen arm consumes some random amount of each resource.
If the consumption is less than replenishment, the resource gets renewed.
The random variable d t,j then models the net replenishmentminus consumption.
The full model presented above is more general because it allows both theconsumption and replenishment to depend on the arm pulled.
We consider two settings in this paper.
MDP setting The decision-maker knows the true outcome distributions.
In this setting the modelimplicitly deﬁnes an MDP, where the state is the budget vector, the actions are arms, and thetransition probabilities are deﬁned by the outcome distributions of the arms.
Learning setting The decision-maker does now know the true outcome distributions.
The goal is to design to an MDP policy for the ﬁrst setting and a learning algorithm for the second,and bound their regret against an LP relaxation as deﬁned in the next subsection.
2.2 Linear Programming Relaxation
Similar to Badanidiyuru et al.
[6, Lemma 3.1], we consider the following LP relaxation that providesan upper bound on the total expected reward of any algorithm:
[Equation]
Lemma 2.1.
The total expected reward of any algorithm is at most T · OPT LP .
The proof of this lemma, similar to those in existing works [2, 6], follows from the observations that(i) the variables p
= { p x : x ∈ X} can be interpreted as the probability of choosing arm x in a round;and (ii) if we set p x equal to the expected number of times x is chosen by an algorithm divided by T ,then it is a feasible solution for the LP.
Deﬁnition 2.1 (Regret) .
The regret of an algorithm A is deﬁned as R T ( A ) =
T · OPT LP − REW ( A ) ,where REW ( A ) denotes the total expected reward of A .
2.3 Assumptions
We assume that the initial budget of every resource is B ≤ T .
This assumption is without loss ofgenerality because otherwise we can scale the drifts by dividing them by the smallest budget.
Thisresults in a smaller support set for the drift distribution that is still contained in [ − 1 , 1] .
Our assumptions about the null arm x 0 are a major difference between our model and BwK .
In
BwK the budgets can only decrease and the process ends when the budget of any resource reaches 0 .
However, in our model the budgets can increase or decrease, and the process ends at the end of thetime horizon.
Our assumptions about the null arm allow us to increase the budget from 0 withoutmaking it negative.
A side-effect of this is that in our model we can even assume that B is a smallconstant because we can always increase the budget by pulling the null arm, in contrast to existingliterature on BwK that assume the initial budgets are large and often scale with the time horizon.
A standard assumption for achieving logarithmic regret in stochastic bandits is that the gap betweenthe total expected reward of an optimal arm and that of a second-best arm is positive.
There are a fewdifferent ways in which one could translate this to our model where the optimal solution is a mixtureover arms.
We make the following choice.
We assume that there exists a unique set of arms X ∗ thatform the support set of the LP solution and a unique set of resources J ∗ that correspond to bindingconstraints in the LP solution [14].
We deﬁne the gap of the problem instance in Deﬁnition 4.1 andour uniqueness assumption 1 implies that the gap is strictly positive.
We make a few separation assumptions parameterized by four positive constants that can be arbitrarilysmall.
First, the smallest magnitude of the drifts, δ drift = min {| µ d,jx | : x ∈ X , j ∈ J } , satisﬁesδ drift > 0 .
Second, the smallest singular value of the LP constraint matrix, denoted by σ min , satisﬁes0 < σ min < 1 .
Third, the LP solution p ∗ satisﬁes p ∗ x ≥ δ support > 0 for all x ∈ X ∗ .
Fourth, x ∈ X ∗ p ∗ x µ d,jx ≥ δ slack > 0 for all resources j / ∈ J ∗ .
The ﬁrst assumption is necessary for logarithmic regret bounds because otherwise one can show that the regret of the optimal algorithmfor the case of one resoure and one zero-drift arm is Θ( √ T ) (Appendix A).
The second and thirdassumptions are essentially the same as in existing literature on logarithmic regret bounds for
BwK [9, 14].
The fourth assumption allows us to design algorithms that can increase the budgets ofthe non-binding resources away from 0 , thereby reducing the number of times the algorithm has topull the null arm.
Otherwise, if they have zero drift, then, as stated above, the regret of the optimalalgorithm for the case of one resource and one zero-drift arm is Θ( √ T ) (Appendix A).
MDP Policy with Constant Regret
In this section we design an MDP policy, ControlBudget (Algorithm 2), with constant regret interms of T for the setting when the learner knows the true outcome distributions and our modelimplicitly deﬁnes an MDP (Section 2.1).
At a high level, ControlBudget , which shares similaritieswith Flajolet and Jaillet [9, Algorithm UCB-Simplex], plays arms to keep the budgets close to adecreasing sequence of thresholds.
The choice of this sequence allows us to show that the expectedleftover budgets and the expected number of null arm pulls are constants.
This is a key step in provingthe ﬁnal regret bound.
We start by considering the special case of one resource in Section 3.1 becauseit provides intuition for the general case of multiple resources in Section 3.2.
3.1 Special Case: One Resource
Since there is only one resource we drop the superscript j in this section.
We say that an arm x is apositive (resp.
negative) drift arm if µ dx > 0 (resp.
µ dx < 0 ).
The following lemma characterizes thepossible solutions of the LP (Eq.
Lemma 3.1.
The solution of the LP relaxation (Eq.
(1) ) is supported on at most two arms.
Furthermore, if T ≥ B / δ drift , then the solution belongs to one of three categories: (i) supported on asingle positive drift arm; (ii) supported on the null arm and a negative drift arm; (iii) supported on apositive drift arm and a negative drift arm.
The proof of this lemma follows from properties of LPs and a case analysis of which constraints aretight.
Our MDP policy, ControlBudget (Algorithm 1), deals with the three cases separately andsatisﬁes the following regret bound. Algorithm 1: =
Theorem 3.1.
If c ≥ 6 δ 2 drift , the MDP policy ControlBudget
(Algorithm 1) satisﬁes
[Equation]
where ˜ C
=
O (cid:18) δ − 4 drift ln (cid:18)(cid:16)
[Equation] − exp (cid:16) − [Equation] drift 8 (cid:17)(cid:17)
[Equation]
1 (cid:19) + δ − 1 drift (cid:0) 1 − exp (cid:0) δ 2 drift (cid:1)(cid:1) − 2 (cid:19) is a constant.
We defer all proofs in this section to Appendix B, but we include a proof sketch of most results in themain paper following the statement.
The proof of Theorem 3.1 follows from the following sequenceof lemmas.
Lemma 3.2.
If the LP solution is supported on a positive drift arm x p , then
[Equation]
where ˜ C
=
O (cid:18) δ − 3 drift ln (cid:18)(cid:16) 1 − exp (cid:16) − δ 2 drift 8 (cid:17)(cid:17) − 1 (cid:19)(cid:19) is a constant.
We can write the regret in terms of the norm of ξ = ( ξ x p ) , where ξ x p is the expected differencebetween the number of times x p is played by the LP and by ControlBudget .
This is equal to theexpected number of times the policy plays the null arm and, in turn, is equal to the expected numberof rounds in which the budget is below 1 .
Since both x 0 and x p have positive drift, this is a transientrandom walk that drifts away from 0 .
It is known that such a walk spends a constant number ofrounds in any state in expectation.
Lemma 3.3.
If the LP solution is supported on the null arm x 0 and a negative drift arm x n , then
[Equation]
where ˜ C
= O ( δ − 1 drift ) is a constant.
We can write the regret in terms of the norm of ξ = ( ξ x 0 , ξ x n ) , where ξ x is the expected differencebetween the number of times x is played by the LP and by ControlBudget .
Since both constraints(resource and sum-to-one) are tight, the lemma follows by writing ξ = D − 1 b and taking norms,where D is the LP constraint matrix and b = ( − E [ B T ] , 0) .
Lemma 3.4.
If the LP solution is supported on a positive drift arm x p and a negative drift arm x n ,then
[Equation]
where E [ N x 0 ] denotes the expected number null arm pulls and ˜ C
= O ( δ − 1 drift ) is a constant.
This lemma follows similarly to the previous one by writing regret in terms of the norm of ξ =( ξ x p , ξ x n ) and writing ξ = D − 1 b for b = ( − E [ B T ] , E [ N x 0 ]) .
Therefore, proving that R T ( ControlBudget ) is a constant in T requires proving that both theexpected leftover budget and expected number of null arm pulls are constants.
Intuitively, we couldensure E [ B T ] is small by playing the negative drift arm whenever the budget is at least 1 .
However,there is constant probability of the budget decreasing below 1 and the expected number of null armpulls becomes O ( T ) .
ControlBudget solves the tension between the two objectives by carefullychoosing a decreasing sequence of thresholds τ t .
The threshold is initially far from 0 to ensure lowprobability of pre-mature resource depletion, but decreases to 0 over time to ensure small expectedleftover budget and decreases at a rate that ensures the expected number of null arm pulls is a constant.
Lemma 3.5.
If the LP solution is supported on a positive drift arm x p and a negative drift arm x n ,and c ≥ 6 δ 2 drift , then
[Equation]
where ˜ C
=
O (cid:18) δ −
[Table]
a constant.
If the budget is below the threshold, i.e., B t − 1 < τ t for some t , then ControlBudget pulls x p until
B s ≥ τ s +1 for some s ≥ t .
Since x p has positive drift, the event that repeated pulls decrease thebudget towards 0 is a low probability event.
Using this, our choice of τ t = c log( T − t ) for anappropriate constant c , and summing over all rounds shows that the expected number of rounds inwhich the budget is less than 1 is a constant in T .
Lemma 3.6.
If the LP solution is supported on two arms, and c ≥ 6 δ 2 drift , then
E [ B T
[Equation]
≤ ˜ C, (7) where ˜ C
=
˜
O
{+(cid:16)(cid:0) 
 −+} exp (cid:0) δ 2 drift (cid:1)(cid:1) − 2 + δ − 2 drift (cid:17) is a constant.
If B t − 1 ≥ τ t , then ControlBudget pulls a negative drift arm x n .
We can upper bound the expectedleftover budget by conditioning on q , the number of consecutive pulls of x n at the end of the timeline.
The main idea in completing the proof is that (i) if q is large, then it corresponds to a low probabilityevent; and (ii) if q is small, then the budget in round T − q was smaller than τ q , which is a decreasingsequence in q , and there are few rounds left so the budget cannot increase by too much.
3.2 General Case: Multiple Resources
Now we use the ideas from Section 3.1 to tackle the case of m > 1 resources that is much morechallenging.
Generalizing Lemma 3.1, the solution of the LP relaxation (Eq.
(1)) is supported on atmost min { k, m } arms.
Informally, our MDP policy, ControlBudget (Algorithm 2), samples an armfrom a probability distribution that ensures drifts bounded away from 0 in the “correct directions”:(i) a binding resource j has drift at least γ t if B t − 1 ,j < τ t and drift at most − γ t if B t − 1 ,j ≥ τ t ; and(ii) a non-binding resource j has drift at least 1 2 γ t if B t − 1 ,j < τ t .
This allows us to show that theexpected leftover budget for each binding resource and the expected number of null arm pulls areconstants in terms of T .
2 Deﬁne
threshold τ t = c log( T − t ) .
3 for t = 1 , 2 , . . .
, T do if ∃ j ∈ J such that B t − 1 ,j < 1 then
Pull the null arm x 0 .
else
Deﬁne s t ∈ {± 1 } | X ∗ |− 1 × 0 as follows.
Let j denote the resource corresponding to row i ∈ [ | X ∗ | − 1] in the matrix D and vector b .
Then, the i th entry of s t is [- 
-]− 1 if
B t − 1 ,j < τ t and +1 otherwise.
Deﬁne γ t to be the solution to the following constrained optimization problem:
[Equation]
Sample an arm from the probability distribution p t = D − 1 ( b + γ t s t ) .
end 11 end
Theorem 3.2.
If c ≥ 6 γ ∗ 2 , the regret of ControlBudget (Algorithm 2) satisﬁes
[Equation]
(9) where γ ∗ (deﬁned in
Lemma 3.9) and ˜ C are constants with ˜ C
= O
[Equation]
We defer all proofs in this section to Appendix C, but we include a proof sketch of most results in themain paper following the statement.
The proof of Theorem 3.2 follows from the following sequenceof lemmas.
The next two lemmas are generalizations of Lemmas 3.3 and 3.4 with essentially thesame proofs.
Recall J ∗ denotes the unique set of resources that correspond to binding constraints inthe LP solution (Section 2.3).
Lemma 3.7.
If the LP solution includes the null arm x 0 in its support, then
[Equation]
where ˜ C
= O ( σ − 1min ) is a constant.
Lemma 3.8.
If the LP solution does not include the null arm x 0 in its support, then
[Equation]
where E [ N x 0 ] denotes the expected number of null arm pulls and
˜ C
= O ( σ − 1min ) is a constant.
Lemmas 3.10 and 3.11 are generalizations of Lemmas 3.5 and 3.6 with similar proofs after takinga union bound over resources.
But we ﬁrst need Lemma 3.9 that lets us conclude there is drift ofmagnitude at least γ ∗ > 0 in the “correct directions” as stated earlier.
Lemma 3.9.
[9, Lemma 14]
In each round t ,
[Equation]
The proof of this lemma is identical to Flajolet and Jaillet [9, Lemma 14] but we provide a proof inthe appendix for completeness.
Lemma 3.10.
If the LP solution does not include the null arm in its support, then
[Equation]
[Equation]
where ˜ C
= O (cid:18) m ( γ ∗ ) − 3 ln (cid:18)(cid:16) (1 − exp (cid:16) − γ ∗ 2 8 (cid:17)(cid:17) − 1 (cid:19)(cid:19) is a constant.
Lemma 3.11.
If the LP solution is supported on more than one arm, then for all j ∈ J ∗
[Equation]
(13) where ˜ C
=
˜ O (cid:16)(cid:0) 1 − exp( γ ∗ 2 ) (cid:1) − 2 + ( γ ∗ ) − 2 (cid:17) is a constant.
A subtle but important point is that the regret analysis does not require ControlBudget to knowthe true expected drifts in order to ﬁnd the probability vector p t .
It simply requires the algorithm to know X ∗ , J ∗ , and ﬁnd any probability vector p t that ensures drifts bounded away from0 in the “correct directions” as stated earlier.
We use this property in our learning algorithm,
ExploreThenControlBudget (Algorithm 3), in the next section.
Learning Algorithm with Logarithmic Regret
In this section we design a learning algorithm, ExploreThenControlBudget (Algorithm 3), withlogarithmic regret in terms of T for the setting when the learning does not know the true distributions.
Our algorithm, which can be viewed as combining aspects of Li et al.
[14, Algorithm 1] and Flajoletand Jaillet [9, Algorithm UCB-Simplex], proceeds in three phases.
It uses phase one of Li et al.[14 , Algorithm 1] to identify the set of optimal arms X ∗ and the set of binding constraints J ∗ byplaying arms in a round-robin fashion, and using conﬁdence intervals and properties of LPs.
Thisis reminiscent of successive elimination [8], except that the algorithm tries to identify the optimalarms instead of eliminating suboptimal ones.
In the second phase the algorithm continues playing thearms in X ∗ in a round-robin fashion to shrink the conﬁdence radius further.
In the third phase thealgorithm plays a variant of the MDP policy ControlBudget
(Algorithm 2)
with a slighly differentoptimization problem for γ t because it only has empirical esitmates of the drifts.
4.1 Additional Notation and Preliminaries
For all arms x ∈ X and rounds t ≥ k , deﬁne the upper conﬁdence bound (UCB) of the expectedoutcome vector µ ox as UCB t ( x ) =
¯ o t ( x ) + rad t ( x ) , where rad t ( x ) = (cid:112) 8 n t ( x ) − 1 log T denotesthe conﬁdence radius, n t ( x ) denotes the number of times x has been played before t , and ¯ o t ( x ) =n t ( x ) {+− 
+}
(cid:80) t
o t 1 [ x t = x ] denotes the empirical mean outcome vector of x .
The lower conﬁdencebound (LCB) is deﬁned similarly as LCB t ( x ) =
¯ o t ( x ) − rad t ( x ) .
For all arms x ∈ X , let OPT − x denote the value of the LP relaxation (Eq.
(1)) with the additionalconstraint p x = 0 , and for all resources j ∈ J , let OPT − j denote the value when the objective has an extra − (cid:80) x
p x µ d,jx + B / T term [14].
Intuitively, these represent how important it is to play arm xor make the resource constraint for j a binding constraint.
Deﬁne the UCB of OPT − x to be the valueof the LP when the expected outcome is replaced by its UCB, and denote this by UCB t ( OPT − x ) .
The LCB for OPT − x , and UCB and LCB for OPT − j and OPT LP are deﬁned similarly.
Deﬁnition 4.1 (Gap [14]) .
The gap of the problem instance is deﬁned as
[Equation]
4.2 Learning Algorithm and Regret Analysis
end
if ∃ j ∈ J such that B t − 1 ,j < 1 then
Pull the null arm x 0 .
else
Deﬁne s t as in Algorithm 2.
Choose ( γ t , p t ) = max γ ∈ [0 , 1] γ such that there exists a probability vector p satisfying [Equation]
[Equation]
Sample an arm from the probability distribution p t .
end 20 end
Theorem 4.1.
If c ≥ 6 γ ∗ , the regret of ExploreThenControlBudget (Algorithm 3) satisﬁes
[Equation]
where γ ∗ (deﬁned in Lemma 3.9), ˜ C (cid:48) and ˜ C are constants with ˜ C (cid:48) denoting the constant in Theorem 3.2 and
[Equation]
We defer all proofs in this section to Appendix D, but we include a proof sketch of mostresults in the main paper following the statement.
We refer to the three while loops of
ExploreThenControlBudget (Algorithm 3) as the three phases.
The lemmas and corollariesfollowing the theorem below show that the ﬁrst phase consists of at most logarithmic number ofrounds.
It is easy to see that the second phase consists of at most logarithmic number of rounds.
The third phase plays a variant of the MDP policy ControlBudget
(Algorithm 2).
There exists a feasible solution to the optimization problem in Line 17 that ensures drifts bounded away from0 in the “correct directions” (Appendix D.3).
Combining the analysis of the ﬁrst two phases with
Theorem 3.2 lets us conclude that ExploreThenControlBudget has logarithmic regret.
Deﬁnition 4.2 (Clean Event) .
The clean event is the event such that for all x ∈ X and all t ≥ k , (i)µ 0 x ∈ [ LCB t ( x ) , UCB t ( x )] ; and (ii) after the ﬁrst n pulls of the null arm the sum of the drifts for eachresource is at least w , where w = 4096 km 2 log T δ 2 drift ∆and n = 2 wµ dx 0 .
Lemma 4.1.
The clean event occurs with probability at least 1 − 5 mT − 2 .
The lemma follows from Hoeffding’s inequality.
Since the complement of the clean event contributes
O ( mT − 1 ) to the regret, it sufﬁces to bound the regret conditioned on the clean event.
Lemma 4.2.
If the clean event occurs, then UCB t ( OPT LP ) − LCB t ( OPT LP ) ≤ 8 m σ min rad t .
A similarstatement is true for OPT − x and OPT − j for all x ∈ X and j ∈ J .
The proof follows from a perturbation analysis of the LP and uses the conﬁdence radius to bound theperturbations in the rewards and drifts.
Corollary 4.1.
If the clean event occurs and n t ( x ) > 2048 m 2 log T σ 2min ∆for all x ∈ X , then
UCB t ( OPT − x ) <
LCB t ( OPT LP ) and UCB t ( OPT − j ) <
LCB t ( OPT LP ) (20) for all x ∈ X ∗ and j ∈ J \ J ∗ .
This follows from substituting the bound on n t ( x ) into the deﬁnition of rad t ( x ) and applying Lemma 4.2.
In the worst case, each pull of an arm can cause the budget to drop below 1 ,but the clean event implies that the ﬁrst n pulls of x 0 have enough total drift to allow n t ( x ) pulls ofeach non-null arm in phase 1.
This allows us to upper bound the duration of phase 1 as follows.
Corollary 4.2.
If the clean event occurs, then phase 1 of ExploreThenControlBudget has at most
[Equation]
rounds, where ˜ C
= O (cid:16)
[Equation]
(cid:17) is a constant.
4.3 Reduction from BwK
Theorem 4.2.
Suppose BT ≥ δ drift .
Consider a BwK instance such that (i) for each arm and resource,the expected consumption of that resource differs from BT by at least δ drift ; and (ii) all the otherassumptions required by Theorem 4.1 (Section 2.3) are also satisﬁed.
Then, there is an algorithm for
BwK whose regret satisﬁes the same bound as in Theorem 4.1 with the same constant ˜ C .
Due to space constraints, we present the reduction in Appendix D.4.
Conclusion
In this paper we introduced a natural generalization of BwK that allows non-monotonic resourceutilization.
We ﬁrst considered the setting when the decision-maker knows the true distributionsand presented an MDP policy with constant regret against an LP relaxation.
Then we consideredthe setting when the decision-maker does not know the true distributions and presented a learningalgorithm with logarithmic regret against the same LP relaxation.
Finally, we also presented areduction from BwK to our model and showed a regret bound that matches existing results [14].
An important direction for future research is to obtain optimal regret bounds.
The regret bound forour algorithm scales as O (poly( k )poly( m )∆ − 2 ) , where k is the number of arms, m is the numberof resources and ∆ is the suboptimality parameter.
A modiﬁcation to our algorithm along the linesof Flajolet and Jaillet [9, algorithm UCB-Simplex] that considers each support set of the LP solutionexplicitly leads to a regret bound that scales as O (poly( k )2 m ∆ − 1 ) .
It is an open question, evenfor BwK , to obtain a regret bound that scales as O (poly( k )poly( m )∆ − 1 ) or show that the trade-offbetween the dependence on the number of resources and the suboptimality parameter is unavoidable.
Another natural follow-up to our work is to develop further extensions, such as considering aninﬁnte set of arms [11], studying adversarial observations [4, 10], or incorporating contextualinformation [16, 1] as has been the case elsewhere throughout the literature on bandits.