Non-stationary Bandits with Knapsacks
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we study the problem of bandits with knapsacks (BwK) in a nonstationary environment.
The BwK problem generalizes the multi-arm bandit (MAB)problem to model the resource consumption associated with playing each arm.
Ateach time, the decision maker/player chooses to play an arm, and s/he will receive areward and consume certain amount of resource from each of the multiple resourcetypes.
The objective is to maximize the cumulative reward over a finite horizonsubject to some knapsack constraints on the resources.
Existing works study the
BwK problem under either a stochastic or adversarial environment.
Our paperconsiders a non-stationary environment which continuously interpolates betweenthese two extremes.
We first show that the traditional notion of variation budget isinsufficient to characterize the non-stationarity of the BwK problem for a sublinearregret due to the presence of the constraints, and then we propose a new notionof global non-stationarity measure.
We employ both non-stationarity measures toderive upper and lower bounds for the problem.
Our results are based on a primaldual analysis of the underlying linear programs and highlight the interplay betweenthe constraints and the non-stationarity.
Finally, we also extend the non-stationaritymeasure to the problem of online convex optimization with constraints and obtainnew regret bounds accordingly.
Introduction
The multi-armed bandit (MAB) problem characterizes a problem for which a limited amount ofresource must be allocated between competing (alternative) choices in a way that maximizes theexpected gain.
The bandits with knapsacks (BwK) problem generalizes the multi-armed banditsproblem to allow more general resource constraints structure on the decisions made over time, inaddition to the customary limitation on the time horizon.
Specifically, for the BwK problem, thedecision maker/player chooses to play an arm at each time period; s/he will receive a reward andconsume certain amount of resource from each of the multiple resource types.
Accordingly, theobjective is to maximize the cumulative reward over a finite time horizon and subject to an initialbudget of multiple resource types.
The BwK problem was first introduced by Badanidiyuru et al.[2013] as a general framework to model a wide range of applications, including dynamic pricing andrevenue management [Besbes and Zeevi, 2012], Adwords problem [Mehta et al., 2005] and more.
The standard setting of the BwK problem is stochastic where the joint distribution of reward andresource consumption for each arm remains stationary (identical) over time.
Under such setting, alinear program (LP), that takes the expected reward and resource consumption of each arm as input,both serves as the benchmark for regret analysis and drives the algorithm design [Badanidiyuru et al.
,2013, Agrawal and Devanur, 2014].
Notably, a static best distribution prescribed by the LP’s optimalsolution is used for defining the regret benchmark.
An alternative setting is the adversarial BwKproblem where the reward and the consumption may no long follow a distribution and they can bechosen arbitrarily over time.
Under the adversarial setting, a sublinear regret is not achievable in the worst case; Immorlica et al.
[2019] derive a O (log T ) competitive ratio against the static bestdistribution benchmark which is aligned with the static optimal benchmark in the adversarial banditsproblem [Auer et al., 1995].
Another key of the BwK problem is the number of resource types d .
When d = 1 , one optimal decision is to play the arm with largest (expected) reward to (expected)resource consumption ratio, where the algorithm design and analysis can be largely reduced to the
MAB problem.
When d > 1 , the optimal decision in general requires to play a combination of arms(corresponding the optimal basis of the underlying LP).
Rangi et al.
[2018] focus on the case of d = 1and propose an EXP3-based algorithm that attains a regret of O ( √ mB log m ) against the best fixeddistribution benchmark.
Their result thus bridges the gap between the stochastic BwK problem andthe adversarial BwK problem for the case of d = 1 .
The difference between the cases of d = 1 andd > 1 is also exhibited in the derivation of problem-dependent regret bounds for the stochastic BwKproblem [Flajolet and Jaillet, 2015, Li et al., 2021, Sankararaman and Slivkins, 2021].
In this paper, we study the non-stationary BwK problem where the reward and the resource consumption at each time are sampled from a distribution as the stochastic BwK problem but the distributionmay change over time.
The setting relaxes the temporally i.i.d.
assumption in the stochastic settingand it can be viewed as a soft measure of adversity.
We aim to relate the non-stationarity (or adversity)of the distribution change with the best-achievable algorithm performance, and thus our result bridgesthe two extremes of BwK problem: stochastic BwK and adversarial BwK. We consider a dynamicbenchmark to define the regret; while such a benchmark is aligned with the dynamic benchmark inother non-stationary learning problem [Besbes et al., 2014, 2015, Cheung et al., 2019, Faury et al.
,2021], it is stronger than the static distribution benchmark in adversarial BwK
[Rangi et al., 2018,
Immorlica et al., 2019].
Importantly, we use simple examples and lower bound results to show that thetraditional non-stationarity measures such as change points and variation budget are not suitable forthe BwK problem due to the presence of the constraints.
We introduce a new non-stationarity measurecalled global variation budget and employ both of this new measure and the original variation budgetto capture the underlying non-stationarity of the BwK problem.
We analyze the performance of asliding-window UCB-based BwK algorithm and derive a near-optimal regret bound.
Furthermore,we show that the new non-stationarity measure can also be applied to the problem of online convexoptimization with constraints (OCOwC) and extend the analyses therein.
1.1 Related literature
The study of non-stationary bandits problem begins with the change-point or piecewise-stationarysetting where the distribution of the rewards remains constant over epochs and changes at unknowntime instants [Garivier and Moulines, 2008, Yu and Mannor, 2009].
The prototype of non-stationaryalgorithms such as discounted UCB and sliding-window UCB are proposed and analyzed in [Garivierand Moulines, 2008] to robustify the standard UCB algorithm against the environment change.
Theprevalent variation budget measure V = (cid:80) T − 1 t =1 ∥P t − P t +1 ∥ (where P t and the norm bear differentmeaning under different context) is later proposed and widely studied under different contexts, suchas non-stationary stochastic optimization (Besbes et al.
[2015]), non-stationary MAB (Besbes et al.[2014]), non-stationary linear bandits (Cheung et al.
[2019]), and non-stationary generalized linearbandits (Faury et al.
[2021]) problems.
In general, these works derive lower bound of Ω( V 1 3 T 2 3 ) ,and propose algorithms that achieve near-optimal regret of ˜ O
( V 13 T 23 ) .
Cheung et al.
[2019] and
Faury et al.
[2021] require additional orthogonality assumption on the decision set to attain suchupper bound; without this assumption, a regret bound of ˜ O
( V 15 T 45 ) can be obtained [Faury et al.
With the soft measure of non-stationarity, the existing results manage to obtain sublinearregret bounds in T against dynamic optimal benchmarks.
In contrast, a linear regret in T is generallyinevitable against the dynamic benchmark when the underlying environment is adversarial.
Weremark that while all these existing works consider the unconstrained setting, our work complementsthis line of literature with a proper measure of non-stationarity in the constrained setting.
Another related stream of literature is the problem of online convex optimization with constraints(OCOwC) which extends the OCO problem in a constrained setting.
There are two types of constraintsconsidered: the long-term constraint [Jenatton et al., 2016, Neely and Yu, 2017] and the cumulativeconstraint [Yuan and Lamperski, 2018, Yi et al., 2021].
The former defines the constraint violation by∥ ( (cid:80)
Tt
=1 g t ( x t ))
+ ∥ whilst the latter defines it by (cid:80)
Tt =1 ∥ ( g t ( x t ))
+ ∥ where ( · ) + is the positive-partfunction.
The existing works mainly study the setting where g t =
g for all t and g is known a priori.
Neely and Yu [2017] considers a setting where g t is i.i.d.
generated from some distribution.
In this paper, we show that our non-stationarity measure naturally extends to this problem and derivesbounds for OCOwC when g t ’s are generated in a non-stationary manner.
Problem Setup
We first introduce the formulation of the BwK problem.
The decision-maker/learner is given a fixedfinite set of arms A (with |A| = m ) called as action set .
There are d knapsack constraints with aknown initial budget of B j for j ∈ [ d ] .
Without loss of generality, we assume B j = B for all j. Thereis a finite time horizon T , which is also known in advance.
At each time t = 1 , ..., T , the learnermust choose either to play an arm i t or to do nothing but wait.
If the learner plays the arm i at time t ,s/he will receive a reward r t,i ∈ [0 , 1] and consume c t,j,i ∈ [0 , 1] amount of each resource j fromthe initial budget B .
As the convention, we introduce a null arm to model “doing nothing” whichgenerates a reward of zero and consumes no resource at all.
We assume ( r t , c t ) is sampled from somedistribution P t independently over time where r t = { r t,i } i ∈ [ m ] and c t = { c t,j,i } i ∈ [ m ] ,j ∈ [ d ] .
In the 105 stochastic BwK problem, the distribution P t remains unchanged over time, while in the adversarial
BwK problem, P t is chosen adversarially.
In our paper, we allow P t to be chosen adversarially, whilewe use some non-stationarity measure to control the extent of adversity in choosing P t ’s.
At each time t , the learner needs to pick i t using the past observations until time t − 1 but without observing the outcomes of time step t .
The resource constraints are assumed to be hard constraints, i.e., the learner must stop at the earliest time τ when at least one constraint is violated, i.e. (cid:80) τt =1 c t,j,i t >
B , or the time horizon T is exceeded.
The objective is to maximize the expected cumulative reward until time τ , i.e. E [ (cid:80) τ − 1 t =1 r t,i t ] .
To measure the performance of a learner, we define the regret of the algorithm/policy π adopted by the learner as
[Equation]
Here OPT ( T ) denotes the expected cumulative reward of the optimal dynamic policy given all theknowledge of P t ’s in advance.
Its definition is based on the dynamic optimal benchmark whichallows the arm play decisions/distributions to change over time.
2.1 A Motivating Example
The conventional variation budget is defined by
[Equation]
By twisting the definition of the metric dist ( · , · ) , it captures many of the existing non-stationarymeasures for unconstrained learning problems.
Now we use a simple example to illustrate why V nolonger fits for the constrained setting.
Similar examples have been used to motivate algorithm designand lower bound analysis in [Golrezaei et al., 2014, Cheung et al., 2019, Jiang et al., 2020], but havenot been yet be exploited in a partial-information setting such as bandits problems.
Consider a BwK problem instance that has two arms (one actual arm and one null arm), and a singleresource constraint with initial capacity of T 2 .
Without loss of generality, we assume T is even.
Thenull arm has zero reward and zero resource consumption throughout the horizon, and the actual armalways consumes 1 unit of resource (deterministically) for each play and outputs 1 unit of reward(deterministically) for the first half of the horizon, i.e., when t = 1 , ..., T 2 .
For the second half of thehorizon t = T 2 + 1 , ..., T , the reward of the actual arm will change to either 1 + ∆ or 1 − ∆ , andthe change happens adversarially.
For this problem instance, the distribution P t only changes once,i.e., V = ∆ (varying up to constant due to the metric definition).
But for this problem instance, aregret of T · ∆4 is inevitable.
To see this, if the player plays the actual arm no less than T 4 times, thenthe distributions of the second half can adversarially change to the reward 1 + ∆ , and this will resultin a T · ∆4 regret at least.
The same for the case of playing the actual arm for the case of no more than
T 4 times, and we defer the formal analysis to the proof of the lower bounds in Theorem 2.
This problem instance implies that a sublinear dependency on T cannot be achieved with merely thevariation budget V to characterize the non-stationarity.
Because with the presence of the constraint(s), the arm play decisions over time are all coupled together not only through the learning procedure,but also through the “global” resource constraint(s).
For the unconstrained problems, the nonstationarity affects the effectiveness of the learning of the system; for the constrained problems, thenon-stationarity further challenges the decision making process through the lens of the constraints.
2.2 Non-stationarity Measure and Linear Programs
We denote the expected reward vector as µ t = { µ t,i } i ∈ [ m ] and the expected consumption matrix as
C t = { C t,j,i }
j ∈
[ d ] ,i ∈ [ m ] , i.e.,
[Equation]
We first follow the conventional variation budget and define the local non-stationarity budget :
[Equation]
We refer to the measure as a local one in that they capture the local change of the distributionsbetween time t and time t + 1 .
Next, we define the global non-stationarity budget :
[Equation]
where ¯ µ = 1 T (cid:80) Tt =1 µ t and ¯ C = 1 T (cid:80) Tt =1 C t .
These measures capture the total deviations for all theµ t ’s and C t from their global averages.
By definition, W 1 and W 2 upper bound V 1 and V 2 (up to aconstant), so they can be viewed as a more strict measure of non-stationarity than the local budget.
Inthe definition of W 2 , the L 1 norm is not essential and it aims to sharpen the regret bounds.
All the existing analyses of the BwK problem utilize the underlying linear program (LP) and establishthe LP’s optimal objective value as an upper bound of the regret benchmark OPT ( T ) .
In a nonstationary environment, the underlying LP is given by
[Equation]
where B =
( B, ..., B ) ⊤ and ∆ m denotes the m -dimensional standard simplex.
We know that
[Equation]
In the rest of our paper, we will use LP( { µ t } , { C t } , T ) for the analysis of regret upper bound.
Weremark that in terms of this LP upper bound, the dynamic benchmark allows the x t to take differentvalues, while the static benchmark will impose an additional constraint to require all the x t be thesame.
For notation simplicity, we introduce the following linear growth assumption.
All the results in thispaper still hold without this condition.
[Equation]
Assumption 1 (Linear Growth) .
We have the resource budget B = bT for some b > 0 .
Define the single-step LP by
[Equation]
where b = ( b, ..., b ) ⊤ .
The single-step LP’s optimal objective value can be interpreted as the singlestep optimal reward under a normalized resource budget b .
Throughout the paper, we will use the dual program and the dual variables to relate the resourceconsumption with the reward, especially for the non-stationary environment.
The dual of thebenchmark LP ( { µ t } , { C t } , T ) is
[Equation]
where 1 m denotes an m -dimensional all-one vector.
Here we denotes one optimal solution as( q ∗ , α ∗ ) .
The dual of the single-step LP ( µ t , C t ) is
[Equation]
Here we denotes one optimal solution as ( q ∗ t , α ∗ t ) .
The dual optimal solutions q ∗ and q ∗ t are alsoknown as the dual price, and they quantify the cost efficiency of each arm play.
Define
[Equation]
The quantity ¯ q captures the maximum amount of achievable reward by each unit of resource consumption.
We will return with more discussion on this quantity ¯ q after we present the regret bound.
Lemma 1.
We have the following upper bound on ¯ q,
[Equation]
Proposition 1.
We have
T (cid:88)
[Equation]
Proposition 1 relates the optimal value of the benchmark LP( { µ t } , { C t } , T ) with the optimal valuesof the single-step LPs.
To interpret the bound, LP( { µ t } , { C t } , T ) works as an upper bound of the OPT ( T ) in defining the regret, and the summation of LP( µ t , C t ) corresponds to the total rewardobtained by evenly allocating the resource over all time periods.
In a stationary environment, thesetwo are the same as the optimal decision naturally corresponds to an even allocation of the resources.
In a non-stationary environment, it can happen that the optimal allocation of the resource correspondsan uneven one for LP( { µ t } , { C t } , T ) .
For the problem instance in Section 2.1, the optimal allocationmay be either to exhaust all the resource in first half of time periods or preserve all the resource forthe second half.
In such case, forcing an even allocation will reduce the total reward obtained.
Theproposition tells that the reduction can be bounded by 2 W 1 + 2¯ qW 2 where the non-stationarity inresource consumption W 2 is weighted by the dual price upper bound
¯ q.
Sliding-Window UCB for Non-stationary BwK
In this section, we adapt the standard sliding-window UCB algorithm for the BwK problem (Algorithm1) and derive a near-optimal regret bound.
The algorithm will terminate when any type of the resources is exhausted.
At each time t , it constructs standard sliding-window confidence bounds for the rewardand the resource consumption.
Specifically, we define the sliding-window estimators byˆ µ ( w ) t,i :=
(cid:80) t
[Equation]
1 s = [Equation] ( t
[Equation] w ) r t,j · 1 { i s = i } n ( w ) t,i +, ˆ C [Equation] [Equation]
(cid:80)
[Equation]
=
[Equation]
n (
[Equation]
[Equation]
[Equation]
+, where n ( w ) t,i = (cid:80) t − 1 s =1 ∨ ( t − w ) 1 { i s = i } denotes the number of times that the i -th arm has been playedin the last w time periods.
To be optimistic on the objective value, UCBs are computed for rewardsand LCBs are computed for the resource consumption, respectively.
With the confidence bounds, thealgorithm solves a single-step LP to prescribe a randomized rule for the arm play decision.
Our algorithm can be viewed as a combination of the standard sliding-window UCB algorithm[Garivier and Moulines, 2008, Besbes et al., 2015] with the UCB for BwK algorithm
[Agrawal and
Devanur, 2014].
It makes a minor change compared to [Agrawal and Devanur, 2014] which solves asingle-step LP with a shrinkage factor (1 − ϵ ) on the right-hand-side.
The shrinkage factor thereinensures that the resources will not be exhausted until the end of the horizon, but it is not essentialto solving the problem.
For simplicity, we choose the more natural version of the algorithm whichdirectly solves the single-step LP.
We remark that the knowledge of the initial resource budget Band the time horizon T will only be used for defining the right-hand-side of the constraints for this
LP(UCB t ( µ t ) , LCB t ( C t )) .
Algorithm
Input: Initial resource , time horizon T
Output: Arm play indices { i t } ’s 1: while t ≤ T do 2: s
1 c then 3:
Break 4: % Terminate the procedure if any resource exhausted.
5: end if 6:
Construct confidence bounds
[Equation]
Solve the single-step problem LP(UCB t ( µ t ) , LCB [Equation] ( C t )) 8:
Denote its optimal solution by x ∗ t =
( x ∗ t, 1 , ..., x ∗ t,m ) 9:
Pick arm i t randomly according to x ∗ t , i.e., P ( i t = i ) =
x ∗ t,i 10:
Observe the realized reward r t and resource consumption c t,j for j ∈ [ d ] 11: end while
Now we begin to analyze the algorithm’s performance.
For starters, the following lemma states astandard concentration result for the sliding-window confidence bound.
Lemma 2.
The following inequalities hold for all t = 1 , ..., T with probability at least 1 − 13 T :
[Equation]
C t,j,i , ∀ j, i where the UCB and LCB estimators are defined in Algorithm 1.
With Lemma 2, we can employ a concentration argument to relate the realized reward (or resourceconsumption) with the reward (or resource consumption) of the LP under its optimal solution.
In
Lemma 3, recall that τ is the termination time of the algorithm where some type of resources isexhausted, and x ∗ s is defined in Algorithm 1 as the optimal solution of the LP solved at time s .
Lemma 3.
For Algorithm 1, the following inequalities hold for all t ≤ min { τ, T } ,
[Equation]
with probability at least 1 − 1 T .
We note that the single-step LP’s optimal solution is always subject to the resource constraints.
Sothe second group of inequalities in Lemma 3 implies the following bound on the termination time τ .
Recall that b is the resource budget per time period; for a larger b , the resource consumption processbecomes more stable and the budget is accordingly less likely to be exhausted too early.
Corollary 1.
If we choose w 2 = min {+(cid:110) ⌈ m 13 V − 
+}
T 23 log 13 (12 mdT 3 ) ⌉ , T (cid:111) in Algorithm 1, thefollowing inequality holds
[Equation]
with probability at least 1 − 12 T .
To summarize, Lemma 3 compares the realized reward with the cumulative reward of the single-step
LPs, and Corollary 1 bounds the termination time of the algorithm.
Recall that Proposition 1 relatesthe cumulative reward of the single-step LPs with the underlying LP – the regret benchmark.
Puttingtogether these results, we can optimize w 1 and w 2 by choosingw [Equation]
(cid:110)
[Equation]
and then obtain the final regret upper bound as follows.
Theorem 1.
Under Assumption 1, the regret of Algorithm 1 is upper bounded as
[Equation]
[Equation] where π 1 denotes the policy specified by Algorithm 1 and ˜ O ( · ) hides the universal constant and thelogarithmic factors.
Theorem 1 provides a regret upper bound for Algorithm 1 that consists of several parts.
The firstpart of the regret bound is on the order of 1 b √ mT and it captures the regret when the underlyingenvironment is stationary.
The remaining parts of the regret bound characterize the relation betweenthe intensity of non-stationarity and the algorithm performance.
The non-stationarity from both thereward and the resource consumption will contribute to the regret bound and that from the resourceconsumption will be weighted by a factor of 1 b or q (See Lemma 1 for the relation between these two).
For the local non-stationarity V 1 and V 2 , the algorithm requires a prior knowledge of them to decidethe window length, aligned with the existing works on non-stationarity in unconstrained settings.
Forthe global non-stationarity W 1 and W 2 , the algorithm does not require any prior knowledge and theywill contribute additively to the regret bound.
Together with the lower bound results in Theorem 2,we argue that the regret bound cannot be further improved even with the knowledge of W 1 and W 2 .
When the underlying environment degenerates from a non-stationary one to a stationary one, all theterms related to V 1 , V 2 , W 1 and W 2 will disappear and then the upper bound in Theorem 1 matches the regret upper bound for the stochastic BwK setting.
In Theorem 1, we choose to represent theupper bound in terms of b and T so as to reveal its dependency on T and draw a better comparisonwith the literature on unconstrained bandits problem.
We provide a second version of Theorem1 in Appendix D that matches the existing high probability bounds using OPT ( T ) [Badanidiyuruet al., 2013, Agrawal and Devanur, 2014].
In contrast to the Θ(log T ) -competitiveness result in theadversarial BwK [Immorlica et al., 2019], our result implies that with a property measure of thenon-stationarity/adversity, the sliding-window design provides an effective approach to robustify thealgorithm performance when the underlying environment changes from stationary to non-stationary,and the according algorithm performance will not drastically deteriorate when the intensity of thenon-stationarity is small.
When the resource constraints become non-binding for the underlying LPs, the underlying environment degenerates from a constrained setting to an unconstrained setting.
We separate the discussionfor the two cases: (i) the benchmark LP and all the single-step LPs have only non-binding constraints;(ii)
the benchmark LP have only non-binding constraints but some single-step LP have bindingconstraints.
For case (i), the regret bound in Theorem 1 will match the non-stationary MAB bound[Besbes et al., 2014].
For case (ii), the match will not happen and this is inevitable.
We elaborate thediscussion in Section C.
Theorem 2 (Regret lower bounds) .
The following lower bounds hold for any policy π ,
(i) Reg( π, T ) = Ω( m 13 V 13 1 T 23 ) . (ii) Reg( π, T ) =
Ω( 1 b · m 13 V 13 2 T 23 ) .
(iii) Reg( π, T ) =
Ω( W 1 + ¯ qW 2 ) .
Theorem 2 presents a few lower bounds for the problem.
The first and the second lower bounds areadapted from the lower bound example in non-stationary MAB [Besbes et al., 2014] and the thirdlower bound is adapted from the motivating example in 2.1.
There are simple examples where eachone of these three lower bounds dominates over the other two.
In this sense, all the non-stationarityrelated terms in the upper bound of Theorem 1 are necessary including the parameters 1 /b and¯ q .
There is one gap between the lower bound and the upper bound with regard to the number ofconstraints d in the term related to V 2 .
We leave it as future work to reduce the factor to log dwith some finer analysis.
Furthermore, we provide a sharper definition of the global nonstationaritymeasure W minand W minin replacement of W 1 and W 2 in Appendix B.
It makes no essential changeto our analysis, and the two measures coincide with each other on the lower bound problem instance.
We choose to use W 1 and W 2 for presentation simplicity, while W minand W mincan capture themore detailed temporal structure of the nonstationarity.
The discussion leaves an open question thatwhether the knowledge of some additional structure of the environment can further reduce the globalnon-stationarity.
Extension to Online Convex Optimization with Constraints
In this section, we show how our notion of non-stationarity measure can be extended to the problemof online convex optimization with constraints (OCOwC).
Similar to BwK, OCOwC also models asequential decision making problem under the presence of constraints.
Specifically, at each time t ,the player chooses an action x t from some convex set X .
After the choice, a convex cost functionf t : X → R and a concave resource consumption function g t
= ( g t, 1 , ...., g t,d ) :
X → R d arerevealed.
As in the standard setting of OCO, the functions f t is adversarially chosen and thus a staticbenchmark is consider and defined by
[Equation]
[ d
While the existing works consider the case when g t ’s are static or sample i.i.d.
from some distribution
P .
We consider a non-stationary setting where g t may change adversarially over time.
We define a global non-stationarity measure by
[Equation]
where ¯ g j = 1 T (cid:80) Tt =1 g t,j and ∥ f ∥ ∞ : = sup x ∈X | f ( x ) | .
The OCOwC problem considers the following bi-objective performance measure:
[Equation]
where ( · ) + denotes the positive-part function and π denotes the policy/algorithm.
In analogous to the single-step LPs, we consider an optimization problem with more restrictedconstraints as
[Equation]
Denote its optimal solution as x ∗ ′ , and its dual optimal solution as q ∗ ′ .
The following propositionrelates the two optimal objective values.
Assumption 2.
We assume that Slater’s condition holds for both the standard OCOwC program
OPT ( T ) and the restricted OCOwC program OPT ′ ( T ) .
We assume that f t , ∇ f t , g t,i , and ∇ g t,i are uniformly bounded on X and that X itself is bounded.
Moreover, we assume that their dual optimal solutions are uniformly bounded by ¯ q , i.e.
[Equation]
The following proposition relates the two optimal objective values.
Proposition 2.
For OCOwC problem, under Assumption 2, we have
[Equation]
Utilizing the proposition, we can show that the gradient-based algorithm of [Neely and Yu, 2017]achieves the following regret for the setting of OCO with non-stationary constraints.
Moreover, wefurther extend the results and discuss in Appendix E on an oblivious adversarial setting where g t issampled from some distribution P t and the distribution P t may change over time.
Theorem 3.
Under Assumption 2, the Virtual Queue Algorithm of [Neely and Yu, 2017] for any
OCOwC problem (denoted by π 2 ) produces a decision sequence { x t } such that
[Equation]
The theorem tells that the non-stationarity when measured properly will not drastically deteriorate theperformance of the algorithm for the OCOwC problem as well.
Moreover, the non-stationarity willnot affect the constraint violation at all.
Together with the results for the BwK problem, we argue thatthe new global non-stationarity measure serves as a proper one for the constrained online learningproblems.
Note that the upper and lower bounds match up to a logarithmic factor (in a worst-casesense) subject to the non-stationarity measures.
The future direction can be to refine the bounds ina more instance-dependent way and to identify useful prior knowledge on the non-stationarity forbetter algorithm design and analysis.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]