Online Convex Optimization with Hard Constraints: Towards the Best of Two Worlds and Beyond
Abstract
This paper considers online convex optimization with hard constraints and analyzesachievable regret and cumulative hard constraint violation (violation for short).
Theproblem distinguishes itself from online convex optimization with soft constraints,where a violation at one round can be compensated/cancelled by a conservativedecision at a different round.
We propose a RECtiﬁed Online Optimization algorithm (RECOO) and consider two settings: ﬁxed constraints and adversarialconstraints.
Both settings have been considered in the literature.
Compared withexisting results, RECOO achieves the best of two worlds and beyond.
For theﬁxed-constraints setting, RECOO achieves O ` ?
T ˘ regret and O p 1 q violation,where T is the learning horizon.
The best known results in this case are O p?
T qregret and O `
T 1 { 4 ˘ violation.
For the adversarial-constraints setting, it guarantees
O p?
T q regret and O p T 3 { 4 q violation, which match the best existing results.
Whenthe loss functions are strongly convex, RECOO can guarantee O p log T q regretand O p 1 q violation for ﬁxed constraints, and O p log T q regret and O p?
T log T qviolation for adversarial constraints.
Both these results are order-wise better thanthe existing bounds.
The regret and violation bounds mentioned above use thebest ﬁxed decision in hindsight as the baseline.
This paper further considers adynamic baseline where the comparator sequence is time-varying.
This papershows that RECOO not only improves the existing results in the ﬁxed-constraintssetting but also for the ﬁrst time, guarantees dynamic regret and violation boundsin the adversarial-constraints setting.
Our experiment results conﬁrm that RECOOoutperforms several existing algorithms for both ﬁxed and adversarial constraints.
Introduction
Online convex optimization (OCO) is a general framework for modelling and studying online decisionmaking in uncertain environments [23, 10, 21] where the learner adapts its decisions to minimize aloss function or to maximize a utility function when interacting with the environment in real-time.
OCO has a broad range of applications such as online advertising [18, 3], resource allocation innetwork systems [9, 2, 32], load balancing in queueing systems [13, 15], and personalized healthcare[35, 25], etc.
Speciﬁcally, given an unknown convex loss function f t p¨q , the learner makes decisionx t at each round to minimize the total loss over T rounds, i.e. min x P X ř Tt “ 1 f t p x t q , where the lossof decision x t is revealed after x t is executed.
In practice, the decisions are often subject to a varietyof operational constraints.
For example, in online advertisement systems, users, who submit bids toadvertise their items in order to maximize their click-through-rates, often have a weekly or monthlybudget; and in a patient onboarding system, the hospital assigns an incoming patients to a medicalunit to optimize the quality of treatment subject to personnel and medical resource constraints.
Toconsider these applications, we study constrained online convex optimization (COCO) with g t p¨q being a convex constraint function and g t p x t q ď 0 being the constraint at round t .
In COCO, thelearner aims at minimizing the total loss while keeping the constraint violation to its minimum.
One possible method to satisfy the constraints is to project the decision x t into the feasible setat each round [10].
However, such projection may not be possible when the constraints g t p¨q areadversarial and unknown before hand.
Even if g t p¨q is known before hand, the projection-basedmethod (e.g., projected online gradient descent) may require heavy computation when g t p¨q is acomplicated function because it is equivalent to solve a constrained quadratic optimization problem[12, 17, 33]. This motivates projection-free online learning methods including online Frank-Wolfealgorithm [12] where the projection operator at each round is by a linear programming with theexact same constraint functions.
Recently, a sequence of studies have considered COCO with softconstraints, where the constraints are allowed to be violated as long as they are satisﬁed in the longterm [17, 14, 20, 31, 24, 5, 4, 33, 26] (in other words, the severity of constraint violation is evaluatedbased on ř t g t p x t q ).
We call them soft constraints because the constraint violation at one roundcould be compensated in a different round.
In other words, a decision sequence might have zeroconstraint violation based on the metric despite violating the constraint(s) at almost every round.
Forexample, consider a decision sequence such that t g t p x t qu “ t´ 1000 , 1 , 1 , 1 , . .
, 1 , 1 u with 1000ones.
For such a sequence, we have ř τt “ 1 g t p x t q ď 0 for all τ ď T, but the constraint is violated atall rounds except the ﬁrst round.
While some constraints such as budget or fairness constraints arenaturally soft constraints, a stronger notion of constraint and constraint violation is needed for otherapplications, in particular, for safety-critical applications.
In this paper, we consider a stronger notionof constraint violation as in [34, 29, 30]
[Equation]
“g ` t p x t q , (1) where that the operator p¨q ` “ max p¨ , 0 q is imposed on each round and all violations during thedecision process are added up.
We call this hard constraint and the metric cumulative hard constraintviolation .
If a decision sequence has a small V p T q , it implies the constraints are satisﬁed most of thetime during the learning.
For example, consider the same example, V p T q “ 1000 instead of 0 underthis stronger notion of constraint violation.
In this paper, we consider the following two settings:
• Fixed constraints g t p x q “ g p x q , @ t, where the constraint functions are the same across the timebut they are not necessary to be known when making decision at round t . Note the setting ofknown and ﬁxed constraints in [14, 17, 29, 33] is a special case of ours.• Adversarial constraints g t p x q , where the constraint function g t p x q is unknown when makingdecision at round t and can be arbitrarily and adversarially chosen, as in [24, 20, 30].
We will show that RECOO is a single algorithm that achieves small regret and violation in bothsettings.
Next, we summarize our main contributions and compare them with the most related works.
1.1 Main contributions
Algorithm: This paper develops a RECtiﬁed Online Optimization algorithm, called RECOO, for
COCO-Hard (COCO with hard constraints).
RECOO is a unifying algorithm for both ﬁxed andadversarial constraints.
By introducing rectiﬁers in both “decision-making" and “penalty update"components, RECOO maintains a minimum penalty price for constraint violation and encouragesconservative/pessimistic decisions, which is the key to minimizing the cumulative hard constraintviolation.
Regret and violation bounds: We ﬁrst consider the best ﬁxed decision in hindsight as the baseline(also called the static baseline).
The main theoretical results are summarized below.
• For ﬁxed constraints, RECOO achieves O p?
T q regret and O p 1 q violation for convex lossfunctions, and O p log T q regret and O p 1 q violations for strongly-convex loss functions.
• For adversarial constraints, RECOO achieves O p ?
T q regret and O p T 3 { 4 q violation for convex loss functions, and O p log T q regret and O p?
T log T q violation for strongly convex loss functions.
We then consider a dynamic baseline where the comparator is time-varying with path length
P T andestablish the following results.
• For ﬁxed constraints, RECOO achieves O p?
P T T q regret and O p log T q violation, where P T isthe path length of the feasible comparator sequence deﬁned by P T “ ř T ´ 1 t “ 1 } y t ` 1 ´ y t } withg t p y t q ď 0 , @ t P r T s as in [36, 29].•
For adversarial constraints, RECOO achieves O ` P T ?
T ˘ regret and O p T 3 { 4 q violation.
The comparison with most related works are summarized in Tables 1 and 2, from which we can seethat RECOO achieves and improves the state-of-the-art results for both settings (“N/A" means noresults are reported).
Below are a few highlights we would like to mention:
• For ﬁxed constraints and the static baseline, RECOO not only improves the results in [29] butalso answers a conjecture in [29] that O p 1 q violation may be achievable under Slater’s condition(i e., there exists a positive (cid:15) ą 0 and x P X such that g t p x q ď ´ (cid:15), @ t P r T s ).
We proved thatthe conjecture is true even without Slater’s condition.
• For adversarial constraints, the static baseline, and strongly convex loss functions, RECOOreduces the O p T c q regret and O p T 1 ´ c { 2 q violation in [30] to O p log T q and O p?
T log T q ,respectively.• For ﬁxed constraints and the dynamic baseline, RECOO improves the violation from O p?
T q in[29] to O p log T q , while maintaining the same regret bound.• For adversarial constraints and the dynamic baseline, RECOO achieves O ` P T ?
T ˘ regret andO p T 3 { 4 q violation, which is the ﬁrst regret and violation bounds in this setting.
Reference Loss Function
[Table] [Table]
, Violation [34]
[Equation]
N/ [Equation] [Table]
[Table]
Analysis: The key method to establish these results is by imposing the rectiﬁed penalty when makingdecisions, which enable us to develop a simple yet effective self-bounding technique to quantify thetrade-off between the regret and the cumulative hard constraints.
These methods are tailored for thecumulative hard constraints and are different from the drift-plus-penalty method used in the relatedworks [29, 30, 33], where the constraint violations are established by bounding the dual variables(also called virtual queues).
1.2 Related work
COCO has been studied in the literature [17, 24, 20, 16, 6, 22, 34, 29, 30].
We have already presenteda detailed comparison with the existing results on COCO-Hard.
We next review some of recentresults on COCO-Soft, COCO with soft constraints.
COCO-Soft with ﬁxed constraints: [17] studies COCO-Soft and proposed an algorithm thatachieves O p?
T q regret and O p T 3 { 4 q violation.
This result has been extended in [14] to characterize the regret-violation tradeoff, in particular, the proposed algorithm achieves O p T max t c, 1 ´ c u qregret and O p T 1 ´ c { 2 q violation, where c controls the trade-off.
Assuming Slater’s condition holds,the algorithm in [33] achieves O p ?
T q regret and O p 1 q violation and is designed based on a powerful“drift-plus-penalty” method, which inspires some of the design in RECOO.
COCO-Soft with adversarial constraints: Adversarial constraints are more difﬁcult to satisfy buthave been considered in the literature [24, 20, 16, 6, 22].
For COCO-Soft with adversarial constraints,the authors in [24] developed an online mirrored descent type algorithm that achieves O p?
T q regretand O p T 3 { 4 q violation.
Later, [6, 16] generalized the baseline in [24] and still achieve O p?
T q regretand O p T 3 { 4 q violation.
With Slater’s condition, [20] presents an online gradient descent algorithmbased on the drift-plus-penalty method [19], which achieves O p?
T q regret and O p?
T q violation and[22] extended it to an online optimization with sub-modular losses.
Note the key improvement inthese work is the Lyapunov drift technique that can provide a reﬁned bound on virtual queues (ordual variables) with Slater’s condition, which thus achieves a smaller soft constraint violation.
Itremains open that whether O p T 3 { 4 q violation can be reduced with adversarial constraints (soft orhard) while keeping O p?
T q regret without Slater’s condition.
COCO-Hard
In this section, we formally deﬁne COCO-Hard.
Consider the following online convex optimizationproblem: At each round t P r T s , the learner makes decision x t and then observes the loss f t p x t qand constraint function g t p¨q after the decision is executed.
The goal of the learner is to generatea decision sequence t x 1 , x 2 , . . .
, x T ´ 1 , x T u to minimize the total loss ř
Tt “ 1 f t p x t q and the hardconstraint violation ř Tt “ 1 g ` t p x t q .
To quantify the performance of an online convex optimization algorithm for COCO-Hard, we willﬁrst compare it a static baseline, called the best ﬁxed decision in hindsight, which is the solution tothe following ofﬂine COCO.
Ofﬂine COCO
The ofﬂine COCO is formulated as follows [Equation] [Equation] f
[Equation]
p
[Equation]
q subject to
[Equation]
g
[Equation]
ď
[Equation] ď [Equation] (2) where X is a “simple” convex set (e.g., positive quadrant or probability simplex), t f t p¨qu t are convexloss functions and t g t p¨qu t are convex constraint functions.
The optimal solution x ˚ to ofﬂine COCOis called the best single decision in hindsight, a widely used baseline for COCO [10, 17, 24, 31].
Based on ofﬂine COCO, the regret and constraint violation of an online algorithm are deﬁned below.
Regret and cumulative hard constraint violation (or violation for short)
We deﬁne the regretand cumulative constraint violation with respect to the baseline as follows
[Equation]
RECOO: a rectiﬁed online optimization algorithm
We present RECOO, an online optimization algorithm with the rectiﬁed decision and penalty update.
RECOO — A Rectiﬁed Online Optimization Algorithm
Initialization: x 0 P X , Q p 0 q “ 0 , f 0 p x q “ g 0 p x q “ 0 , @ x P X and learning rates α t , η t , γ t .
For t “ 1 , ¨ ¨ ¨ , T, • Set: ˆ g ` t ´ 1 p x q γ t ´
1 g ` t ´ 1 p x q • Rectiﬁed decision: ﬁnd the optimal solution of x t by solving:
[Equation]
• Observe: value ∇ f t p x t q and constraint function g t p¨q .
• Rectiﬁed penalty update : update the estimates of penalty variables Q p t q as follows:
[Equation]
In RECOO, the learning rates α t , η t , γ t are chosen differently depending on the loss function isconvex or strongly convex, but are oblivious to the type of constraints (ﬁxed or adversarial).
Thechoices of learning rates can be found in the theorem statements.
We explain the intuition behind RECOO and the importance of the “rectiﬁers”.
The Lagrange function of the ofﬂine optimization problem (2) is deﬁned to be
[Equation]
where t λ t u are the dual variables associated with the constraints in (2).
Since we have no prior knowledge of f t p¨q when making decision x t , we estimate it with the ﬁrst-order approximation at x t ´ 1 based on the historical information as follows
[Equation]
Moreover, we replace the original constraint function g t p¨q with ˆ g ` t ´ 1 p¨q and the dual variables λ twith Q p t ´ 1 q such that Q p t ´ 1 q ˆ g ` t ´ 1 p x q is a rectiﬁed approximator of λ t g t p x q .
We also added thethe regularization (or smooth) term α t } x ´ x t } 2 that helps the stability of the algorithm.
Note thisdesign is related to penalty-based proximal optimization where we aim to minimize an approximatedf t p x q w.r.t.
proximal operator on the “old” rectiﬁed function ˆ g ` t ´ 1 p x q . In the penalty update of Q p t q , we ﬁrst rectify the original constraint function g t p¨q with ˆ g ` t p¨q andadd it to Q p t ´ 1 q such that penalty increases when violation occurs in each round.
Further werectify Q p t q with a round-dependent constant η t to impose a “minimum” penalty price.
The designof rectiﬁed penalty update induces conservative decisions in the decision-making step to minimizeconstraint violation.
Note that it is different with the traditional primal-dual algorithm that doesnot rectify the constraint violation and impose a minimum penalty price, and when the price (dualvariable) is zero, the algorithm can take very aggressive decisions that lead to large hard violation.
This is not a problem when primal-dual algorithm is used as a numerical method for solving aconstrained optimization problem, but leads to overly aggressive decisions and large violation whenapplying it to COCO.
For a similar reason, RECOO rectiﬁes the amount of violation g t p x q in thedecision-making step so the violation does not become negative to prevent overly optimistic decisions.
We will see that the rectiﬁers in both decision-making and penalty update leads to an upper boundof “regret + violation” as a whole (Lemma 1) such that we can establish regret and (hard) constraintviolation directly. These techniques are different from the primal-dual optimization that quantiﬁesthe constraint violation indirectly by bounding the dual variables/virtual queues. Finally, we comment that our algorithm only needs to solve an “almost” unconstrained optimizationproblem ( X is usually a simple set like the box constraints). Therefore, the gradient-based methodsare sufﬁcient to ﬁnd the minimizer or we might even ﬁnd its close-form with the inverse operationof the function by taking “gradient “ zero”.
Our algorithm is more efﬁcient than the projectiononline gradient descent algorithm [37] and online Frank-Wolfe algorithm
[12], and has the similarcomplexity with [29, 33]. We next analyze the regret and violation of RECOO as deﬁned in (3) and (4) based on the followingstandard technical assumptions on the feasible set, loss and constraint functions.
Assumption 1 The feasible set X is convex with diameter D such that } x ´ x 1 } ď D, @ x, x 1 P X .
Assumption 2 The loss function is convex and Lipschitz continuous with Lipschitz constant F suchthat | f t p x q ´ f t p x 1 q| ď F } x ´ x 1 } , @ x, x 1 P X , @
t.
Assumption 3 The constraint function is convex and Lipschitz continuous with Lipschitz constant Gsuch that | g t p x q ´ g t p x 1 q| ď G } x ´ x 1 } , @ x, x 1 P X , @ t.
For ease of exposition, assume g 1 p
q ď 0 .
Note Assumption 1 implies the feasible set X is bounded.
Assumptions 2 and 3 mean the loss andconstraint functions have bounded gradients.
With these assumption, we are ready to present ourmain result of RECOO for both ﬁxed and adversarial constraints.
Theorem 1 Choose α t “ ?
t, η t “ ? t, and γ t “ t 12 ` ε , where ε ą 0 .
Under Assumptions 1-3,
RECOO algorithm achieves the following regret and constraint violation bounds:
[Equation]
Remark 1 For the ﬁxed constraints, Theorem 1 establishes the optimal order-wise results of O p?
T qregret and O p 1 q violation since Ω p T q regret is the lower bound for OCO with general convex lossand O p 1 q is the best one can have.
For the adversarial constraints, by carefully choosing the learningrates in RECOO algorithm, we can establish a trade-off of O p T c q regret and O p T 1 ´ c { 2 q cumulativeviolation where c P r 1 { 2 , 1 q (see details of Corollary 1 in Appendix C).
Next, we show Theorem 1 can be further improved when the loss function is strongly convex.
Assumption 4 The loss functions f t p¨q , @ t, are µ -strongly convex in X for µ ą 0 , i.e. f t p x 1 q ěf [Equation] ´ [Equation]
, ∇ [Equation] x [Equation]
P X , @ t. With Assumption 4, we are able to establish a O p log T q regret and violation bounds at the same or asmaller order.
Theorem 2 Choose the learning rates to be α t “ µt 2 , η t “ ?
t, γ t “ t 12 ` ε , @
t P r T s , where ε ą 0 .
Under Assumptions 1-4, RECOO achieves the following regret and violation bounds:
[Equation]
ﬁxed or adversarial constraints) ,
[Equation]
ﬁxed constraints
[Equation]
and [Equation] Remark 2 The O p log T q regret is a well-known result in OCO (unconstrained) for a strongly convexloss function [11].
Therefore, for ﬁxed constraints, our results are order-wise sharp because O p 1 qviolation is the best one can achieve.
Extension to a dynamic baseline
In this paper, we establish the main results against a static baseline x ˚ t “ x ˚ , @ t, to convey theidea of RECOO algorithm that can improve the violation in Theorems 1 and 2.
In fact, RECOOcan be generalized to a dynamic baseline as in [36, 29], where x ˚ t ` 1 could be different with x ˚ t butwith limited variation deﬁned by P T “ ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
Deﬁne R dynamic p T q : “ ř Tt “ 1 f t p x t q ´ 219 ř
Tt “ 1 f t p x ˚ t q .
Without any modiﬁcations, we show that RECOO algorithm can achieve O p?
T p 1 `
P T qq regret and O p 1 q violation in the following corollary.
Theorem 3 Under Assumptions 1-3, let the learning rates be α t “ ?
t, η t “ ?
t, γ t “ t 12 ` ε , @
t Pr T s , where ε ą 0 .
Let t x
˚ t u be the optimal solution to (2) with P T “ ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
RECOOalgorithm achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
V p T q ď F 2 ` FD ˆ 1 ` 1 ε ˙ ` D 2 (ﬁxed constraints), and
V p T q ď ˆ
F 2 ` G4 ` FD ˆ 5 ` 1 ε ˙ ` 2 D˙
T 34 (adversarial constraints) .
Remark 3 RECOO algorithm can achieve O `
P T ?
T ˘ regret and O p 1 q violation for ﬁxed constraints, which improves O `
P T ?
T ˘ regret and O p T 1 { 4 q violation in [29].
Moreover, RECOOalgorithm achieve O `
P T ?
T ˘ regret and O p T 3 { 4 q violation for adversarial constraints, which is newin the literature.
As shown in Corollary 2 in Appendix F, combining RECOO with expert-trackingtechniques proposed in [36], we achieve O ` ?
P T T ˘ regret and O p log T q violation for the case ofﬁxed constraints, which improves O ` ?
P T T ˘ regret and O p?
T q violation in [29].
Next, we provide the detailed analysis on Theorem 1.
The proofs of Theorem 2 and Theorem 3 canbe found in Appendix.
3.1 Proof of Theorem 1
We ﬁrst introduce a key lemma that establishes an upper bound on “regret plus violation” at eachround t, which we called self-bounding property.
Lemma 1 (Self-Bounding Property) Recall that x ˚ denotes the optimal solution to ofﬂine COCO(2) and x t be the optimal solution returned by RECOO algorithm.
We have
[Equation]
Based on Lemma 1, we establish the regret and the constant violation bounds in Theorem 1.
Proof of Theorem 1: ﬁxed constraints
Based on (5), we have
[Equation]
[Equation]
[Equation]
which we will use next to establish the bounds on regret and violation.
Regret bound: We sum up Inequality (6) for t “ 1 , ¨ ¨ ¨ , T and have
[Equation]
where the ﬁrst inequality holds by dropping the last negative term and the last inequality holds by
Assumption 1.
Choose α t “ ?
t, we have
[Equation]
Violation bound: We substitute the deﬁnition of ˆ g ` t p¨q in (7) and have
[Equation]
where the second inequality holds by Q p t
q ě η t according to the rectiﬁed penalty update Q p t q “max
[Equation]
˘ .
The inequality implies that
T ÿ
[Equation]
where the second inequality holds by Lemma 6, which includes the detailed calculations of the threeterms above.
Note that the left-hand-side of inequality (8) is not the violation because the indexmismatch (the violation at round t is g ` t p x t q not g ` t p x t ` 1 q ).
For ﬁxed constraints, i.e. g t p x q “ g p x q , the inequality above implies that V p T q : “ ř Tt “ 1 g ` p x t q ď
F 2 ` FD ` 1 ` 1 ε ˘ ` D 2 .
We have proved the ﬁrst part of Theorem 1 for the ﬁxed constraints.
Let uscontinue with (8) to prove the second part of Theorem 1 for adversarial constraints.
Proof of Theorem 1: adversarial constraints
To quantify V p T q : “ ř Tt “ 1 g ` t p x t q for the adversarial constraints, we need to establish the relationship between g ` t p x t q and g ` t p x t ` 1 q to address the index mismatch, which can be bounded by} x t ` 1 ´ x t } 2 as shown in the following lemma.
Lemma 2 Under Assumptions 1-3, RECOO achieves for any β ą
[Equation]
The next lemma further quantiﬁes ř T t “ 1 } x t ` 1 ´ x t } 2 .
Lemma 3 Under Assumptions 1-3, RECOO achieves
[Equation]
Take summation of this equality in Lemma 2 from t “ 1 to T, and we have
[Equation]
1 { 4 , (9) where the last inequality holds by letting β “ T 1 { 4 .
Combining (8) and (9) completes the proof.
Experiments
In this section, we present numerical experiments for evaluating the performance of RECOO withﬁxed and adversarial constraints.
We compared RECOO with the algorithms in [34, 29] for theﬁxed-constraints setting; and with the algorithm in [30] for the adversarial-constraints setting.
Allresults are obtained by averaging over 500 trials and reported with 95% conﬁdence interval.
Fixed constraints Similar to [29], we considered loss functions f t p x q “ x θ p t q , x y , where θ p t q istime varying and unknown at round t ; and the ﬁxed constraint function Ax ď b, where x P R 2 ,
A P R 3 ˆ 2 and b P R 3 .
We chose the number of total rounds to be T “ 5000 , and chose θ p t q “r θ 1 p t q , θ 2 p t q , θ 3 p t qs , where θ 1 p t q „ U p´ t 1 { 10 , ` t 1 { 10 q , θ 2 p t q „ U p´ 1 , 0 q when t P r 1 , 1500 s Yr 2000 , 3500 s Y r 4000 , 5000 s and θ 2 p t q P U p 0 , 1 q otherwise; θ 3 p t q “ p´ 1 q µ p t q where µ p t q is arandom permutation of vector r 1 : 5000 s .
Let X “ t x |
ď x 1 ď 1 , 0 ď x 2 ď 1 u , A i,j „
U p 0 . 1 , 0 .
5 q , 1 ď i ď 3 , 1 ď j ď 2 , and b i „ U p 0 , 0 .
3 q , 1 ď i ď 3 , respectively.
We compared RECOO with Algorithm in [34] and Algorithm 1 and Algorithm 2 in [29], wherelearning rates are summarized in Table 3 in Appendix H as suggested in [34] and [29].
Figure1a illustrates the trajectories of the cumulative losses ř Tt “ 1 f t p x t q , where we observe all of thesealgorithms have similar trends while RECOO achieves the best cumulative losses.
Figure 1b shows RECOO achieves the smallest cumulative violation.
In particular, the (mean, variance) pair of RECOO for loss and violation are p´ 510 .
78 q and p 0 .
008 q at the end of learning horizon,respectively; that of Algorithm 1 in [29] are p´ 507 .
46 q and p 0 .
007 q ; that of Algorithm 2in [29] are p´ 509 . 09 , 36 .
73 q and p 0 .
008 q ;that of Algorithm 1 in [34] are p´ 504 .
82 q andp 980 .
49 q .
These results and Figure 1 conﬁrm that RECOO outperforms the existing algorithmsw.r.t.
both cumulative loss and violation.
[Figure]
Experiment with adversarial constraints We considered an OCO with adversarial constraintssimilar as the one in [30].
The loss function was chosen to be f t p x q “ 12 } H p t q ¨ x ´ y p t q} 2 with
H p t q P R 4 ˆ 10 , x P R 10 and y p t q P R 4 where H i,j p t q „ U p´ 1 , 1 q , 1 ď i ď 4 , 1 ď j ď 10 ,and y i p t
q “ ř i,jj “ 1 H i,j p t q ` ε i with ε i being the standard normal random variable for any i. Theconstraint functions were chosen to be g t p x q “ A p t q ¨ x ´ b p t q with A p t q P R 2 ˆ 10 and b p t q P R 2 ,where A i,j p t q „ U p 0 , 2 q , @
i, j, t and b i p t
q „ U p 0 , 1 q , @
i, t, respectively.
We compared RECOO with Algorithm 1 in [30], where learning rates are summarized in Table 4in Appendix H. Figure 2 includes the cumulative losses and violations.
In particular, The (mean,variance) pair of RECOO for loss and violation are p 42510 .
05 q and p 713 . 45 , 1 .
60 q at theend of learning horizon, respectively while that of Algorithm 1 in [30] are p 43011 .
17 q andp 1684 .
17 , 2 . 85 q .
It is justiﬁed RECOO performs better, especially in terms of the cumulative violation,which further shows the “rectiﬁed” design reduces the cumulative hard constraint violation.
[Figure]
Experiment of online job scheduling in distributed data centers We also tested our algorithmwith the experiment of online job scheduling in a distributed data center similar as in [31].
Ouralgorithm outperforms the baselines in [31] and [30].
The details can be found in Appendix G.
Conclusions
In this paper, we studied online convex optimization with hard constraints (COCO-Hard) under twosettings: ﬁxed constraints and adversarial constraints.
We proposed a RECtiﬁed Online Optimizationalgorithm (RECOO) and proved it achieves the best of two worlds and beyond.
The algorithmimproves the best-known results for the ﬁxed-constraints, matches the best results for the adversarialconstraints, outperforms the state-of-art results when the loss functions are strongly-convex.
Theexperiments conﬁrmed our theoretical results.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No] We didn’t ﬁnd any major limita-
tions in applying our results to constrained online convex programming.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
N/ A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Auxiliary lemmas
The following lemma provides a useful upper bound on the optimal value of strongly convex function[31, 8, 28].
Lemma 4 Let X be a convex set.
Let h : X Ñ R be α -strongly convex function on X and x opt bean optimal solution of h, i.e., x opt “ arg min x P X h p x q .
Then, h p x opt q ď h p x q ´ α 2 } x ´ x opt } 2 forany x P X .
Proof The proof of the lemma is based on the deﬁnition of α -strongly convex functions and theﬁrst-order optimality condition.
Deﬁne the subgradient of h p x q to be ∇ h p x q , According to thedeﬁnition of strong convexity, we have
[Equation]
Deﬁne x opt “ arg min x P X h p x q .
Let y “ x opt in (10) , we have
[Equation]
We then conclude the proof based on the ﬁrst-order optimality condition that for any x P X , x ∇ h p x opt q , x ´ x opt y ě 0 .
The following lemma is the key to bridge the regret and the constraint violation.
Lemma 5 Deﬁne
[Equation]
Let x t ` 1 be the optimal solution returned by RECOO, i.e., x t ` 1 “ arg min x P X h t p x q .
We have forany x P X that
[Equation]
Proof The proof is a direct application of Lemma 4.
Note that h t p x q is 2 α t -strongly convex becausex ∇ f t p x t q , x ´ x t y ` Q p t q ˆ g ` t p x q is convex in x and α t } x ´ x t } 2 is 2 α t -strongly convex.
The following lemma is to provide the detailed calculations required for obtaining inequality (8).
Lemma 6 Under Assumptions 1-3, we have T ÿ
[Equation]
Proof Under Assumptions 1-3, we calculate these three terms as follows:
[Equation]
B Proof of lemmas in Theorem
B.1 Proof of Lemma
We prove the key self-bounding property in this section.
Since (11) holds for any x P X in Lemma 5,let x “ x ˚ such that
[Equation]
We add f t p x t q to both sides of (12) that
[Equation]
where the second inequality holds because f t p¨q is a convex function; the third inequality holdsbecause x ˚ is an feasible point g t p x ˚
q ď 0 such that ˆ g ` t p x ˚ q “ 0 holds.
By moving f t p x ˚ q to the left-hand side and α t } x t ` 1 ´ x t } 2 to the right-hand side of (13), respectively,we have
[Equation]
where the last inequality holds by Assumption
that
[Equation]
B.2 Proof of Lemma
Since g t p x q is convex function, we have g ` t p x q to be convex because max over convex functions is convex.
Denoting by ∇ g ` t p x q the subgradient of g ` t p x q , we have
[Equation]
Let y “ x t [Equation] x “ x t `
[Equation]
we have [Equation] where the second inequality holds because of Cauchy-Schwarz inequality; the third inequality holdsbecause of Assumption 3.
Take summation of the equality above from 1 to T, we have
[Equation]
B.3 Proof of Lemma
According to Lemma 5, we have
[Equation]
[Table]
where the last inequality holds because of the convexity of f t p¨q .
By g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 ,we rearrange the inequality above and have
[Equation]
Take summation of the inequality above from t “ 1 to T, we have
[Equation]
where the second inequality holds because
[Equation]
the last inequality holds because of α t “ ?
t and Assumption 1.
The proof is completed.
C Corollary
Corollary 1 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s ,where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO algorithm achieves the following bounds on the regret andcumulative constraint violations:
[Equation]
, for both types of constraints,
[Equation]
for ﬁxed constraints, and
[Equation]
for adversarial constraints
[Equation]
The proof of corollary follows almost the same steps as in Theorem 1 where c “ 1 { 2 .
Based on (5)in Lemma 1, we still have
[Equation]
which is used to establish the bounds of regret and violation.
Regret Bound: we take summation of the inequality (14) from t “ 1 , ¨ ¨ ¨ , T and have
[Equation]
where the last inequality holds by Assumption 1.
Choose α t “ t c , we have T
[Equation]
It implies that
[Equation]
which gives the regret bound with O p T max t c, 1 ´ c u q .
Cumulative constraint violation bound: For constraint violation, we still have (15) and
[Equation]
Set γ t “ t c and η t “ t c ` ε , where c P r 1 { 2 , 1 q and ε ą 0 , this implies that
T
[Equation]
By Lemma 6, we establish
[Equation]
which proves V p T q : “ ř Tt “ 1 g ` p x
[Equation]
q ď F 2 ` FD ` FDε ` D 2 .
Let’s continue with (16) to prove thesecond part of Corollary 1 for the adversarial constraints.
Recall Lemma 2 that under Assumptions1-3, RECOO achieves for any β ą
[Equation]
From Lemma 2, it is required to quantify the stability term } x t ´ x t ` 1 } 2 that is established in thefollowing lemma.
Lemma 7 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s ,where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO achieves
[Equation]
Proof From Lemma 5,we still have:
[Equation]
According to g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 and rearrange the inequality, we have
[Equation]
Take summation of the inequality above, we have
T ÿ
[Equation]
q .
Since α t “ t c , we have:
T ÿ
[Equation]
which proves Lemma 7.
Recall in Lemma 2 and we have
[Equation]
˙ [Equation] β ˆ D
[Equation]
` 2 FD 1 ´ cT [Equation] c ˙ . Let β “ T c { 2 , we establish
[Equation]
[Equation]
which proves the corollary.
D Proof of Theorem
In this section, we prove Theorem 2 which considers strongly convex loss functions.
According tothe deﬁnition of µ -strongly convex, we have
[Equation]
We start with inequality (12) by using Lemma 5:
[Equation]
(18) where the last inequality holds according to the strongly convex condition of f t p¨q in (17).
Byfollowing the same steps that lead to inequality (5), we have
[Equation]
[Equation]
This is the key “self-bounding” property when f t p¨q is strongly convex function, from which weobtain [Equation]
[Equation]
Recall that α t “ µt 2 , η t “ ?
t, and γ t “ t 12 ` ε , ε ą 0 .
We next establish the regret and violationbounds.
Regret bound: From (20), we immediately have
[Equation]
¯
By the choice of α t “ µt 2 , we have
T ÿ
[Equation]
Following the same steps as in the proof of Theorem 1, we establish
T
[Equation]
(22) which implies that
[Equation]
D.1 Violation bound: ﬁxed constraints
For ﬁxed constraints, inequality (23) implies V p T q : “ ř Tt “ 1 g ` p x t q ď F 2 µ ` FD ` 1 ` 1 (cid:15) ˘ becausethe constraint is ﬁxed.
We have proved the ﬁrst part of Theorem 2 for the ﬁxed constraints.
Let’scontinue with (23) to prove the second part of Theorem 2 for the adversarial constraints.
D.2 Violation bound: adversarial constraints
By Lemma 2, we have
[Equation]
From (18), we again establish the bound on ř
Tt “ 1 } x t ´ x t ` 1 } 2 as follows
[Equation]
(25) where the second inequality holds because
[Equation]
the last inequality holds because of α t “ µt 2 and Assumption 1.
We combine (24) and (25) withβ “ a
T {p 1 ` log T q as follows
[Equation]
Combining the inequality above with inequality (23), we establish the second part of Theorem 2 foradversarial constraints.
E Proof of Theorem 3
We prove Theorem 3 with a dynamic baseline similar to that in [29], which is the solution to thefollowing ofﬂine OCO with constraints:
[Equation]
where (28) imposes a path-length constraint on the baseline solution which limits change of t x t u .
Note that the solution to (26)-(28) with P T “ 0 reduces to the best ﬁxed decision in hindsight.
Lett x ˚ t u be the optimal solution to (26)-(28).
We deﬁne the regret and cumulative constraint violation asfollows
[Equation]
Next, we state the key self-bounding property for establishing Theorem 3.
Lemma 8 Let t x t u be the decision sequence generated by RECOO.
Under Assumptions 1-3, thefollowing inequality holds for any sequence t y t u with y t P X , @ t,
[Equation]
Proof According to Lemma 5, we have
[Equation]
We add f t p x t q to both sides of the inequality above
[Equation]
which proves the lemma because
[Equation]
Regret bound with dynamic baseline: Let the optimal sequence be t x ˚ t u .
Substitute t y t u “ t x ˚ t uin (31).
By g t p x ˚ t q ď 0 , we have
[Equation]
Take the summation of the inequality above from t “ 1 to T, we have
[Equation]
Recall α t “ ?
t, we have
[Equation]
For the second term, we have
[Equation]
where the ﬁrst inequality holds because of Assumption 1 and the last inequality holds because
T ÿ t “
[Equation]
Therefore
[Equation]
we prove the regret in Theorem 3 as follows
[Equation]
Cumulative hard constraint violation bound: When α t “ η t “ ?
t, γ t “ t 12 ` ε with ε ą 0 , theproof follows the same steps in the proof of Theorem 1 in Section 3.1 because the deﬁnition ofconstraint violation is the same.
F Reﬁned results of RECOO with expert-tracking under the dynamicbaseline
Motivated by [36], we combine RECOO with expert tracking techniques in [7] to improve theperformance bounds w.r.t.
P T , similar in [29].
The intuition of the algorithm is to set up N parallelexperts ( N RECOO algorithms) and to track the best one.
We state RECOO with expert trackingalgorithm as follows.
A Rectiﬁed Online Optimization with Expert-Tracking Algorithm
Initialization ( N Experts): f 0 p x q “ g 0 p x q “ 0 , @ x P X , x i, 0 P X , Q i p 0 q “ 0 .
The learning ratesα i,t , η i,t , γ t , κ and w i, 1 “ N ` 1 i p i ` 1 q N , @
i P r N s .
For t “ 1 , ¨ ¨ ¨ , T,
• Set:
ˆ g ` t ´ 1 p x q “ γ t ´ 1 g ` t ´ 1 p x q .•
Rectiﬁed decision: ﬁnd the optimal solution of x i,t for each expert and output x t :x i,t “ arg min [Equation] X
[Equation]
f t ´
[Equation]
p x i,t ´
[Equation]
x
[Equation]
x i
[Equation]
x t “ N ÿ [Equation] [Equation] w
[Equation]
t x
[Equation]
,t • Observe: ∇ f t p¨q and g t p¨q .
• Rectiﬁed penalty update: update Q i p t q and w i,t as follows:
[Equation]
Before presenting the main result of RECOO with expert-tracking algorithm, we impose an additionalassumption on the loss functions as in [36].
Assumption 5 The loss functions are bounded by a constant C such that | f t p x q| ď C, @
x P X , @
t.
We are ready to show RECOO with expert-tracking algorithm improves Theorem 3 in the following.
Corollary 2 Let N “
[Equation]
\ ` 1 , κ “ 1 {?
T .
Let the learning rates be α
i,t “ 575 ? t { 2 i ´ 1 , η i,t “ [-
-]2 i ´ 1 ?
t, γ t “ t 12 ` ε , @
i P r N s , where ε ą 0 .
Let t x
˚ t u be the optimal solutionto (2) with P T “ ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
Under Assumptions 1-3, RECOO with expert-tracking algorithm achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
We ﬁrst introduce Lemma 1 in [36], which quantiﬁes the difference between the weighted output ofall experts with the best expert.
Lemma 9 (Lemma 1 in [36]) Let t x
i,t u and t x t u be the sequence generated by RECOO with experttracking algorithm, we have
[Equation]
Substitute the learning rate of α i,t in (32) in the proof of Theorem 3, we have
[Equation] [Equation]
Recall N “
X 12 log 2 p 1 ` T q \ ` 1 , there exists i 0 “
X 12 log 2 p 1 ` P T D q \ ` 1 P r N s such that
[Equation]
Regret bound with dynamic baseline: Let i “ i 0 in (33) and by Lemma 9, we have:
[Equation]
where the second and the last inequalities hold by using (34).
Moreover, we have
[Equation]
where the ﬁrst inequality holds by the convexity of f t p¨q and the second inequality holds by Lemma9.
Recall κ “ 1 ? T and w i, 1 “ N ` 1 i p i ` 1 q N , we have
[Equation]
Combine all these inequalities, we have:
[Equation]
T which proves the regret in Corollary 2 and establish O p?
P T T q regret bound.
Violation bound: Since g ` t p x q is convex, we have:
T ÿ
[Equation]
`
[Equation]
` 1 q .
By the inequality (31), we have
[Equation]
α t ` α i
[Equation]
t
[Equation]
˚ t
[Equation]
x ˚ t ´ x i
[Equation]
t ` 1 } 2 .
Let α i,t “ ?
t { 2 i ´ 1 , η i,t “ 2 i ´ 1 ?
t, γ t “ t 12 ` ε , ε ą 0 , we haveg ` t p x i,t ` 1 q ď | f t p x ˚ t q ´ f t p x i
[Equation]
t q| η i
[Equation]
t γ t `
F
[Equation]
α i
[Equation]
t η i,t γ t ` α i
[Equation]
t η i
[Equation]
t γ t p} x ˚ t
´ x i,t } 2 ´ } x ˚ t
´ x i,t ` 1 } 2 q ď
[Equation]
FD t
[Equation]
F
[Equation] [Equation] [Equation]
12 `
[Equation]
p
[Equation]
´
[Equation]
t } 2 ´ } x ˚ t ´ x i,t ` 1 } 2 q , which implies
[Equation]
ÿ
[Equation]
Thus we have
[Equation]
which proves the violation in Corollary 2.
G Experiment of online job scheduling in a distributed data center
In this section, we tested our algorithm with the experiment of online job scheduling in a distributeddata center similar as in [31].
We considered a distributed data center with server clusters locatedat different regions.
The incoming jobs arrive at a front-end load balancer and will be scheduledto different clusters to fulﬁll the service.
The service capability of a cluster is a function w.r.t.
itsenergy consumption and the energy prices vary across locations and times.
The goal is to minimizethe energy cost while guaranteeing real-time processing of safety-critical jobs.
This problem can beformulated as a constrained online convex optimization problem and solved by our algorithm.
Speciﬁcally, we considered r “ 10 regions, each region has 10 clusters, and each time/round t has 5minutes.
Let x t P R 100 be the energy allocation vector of all clusters at round t, where the i th entryis the energy allocation of cluster i .
Let f t p x t q “ x c t , x t y , where c t P R 100 are the energy prices attime t. Let g t p x t q “ λ t ´ ři “ 1 h i p x t,i q , where λ t is the number of job arrivals during time t and 608 h i p x t,i q “ 4 log p 1 ` 4 x t,i q is the service capacity of cluster i at round t .
The constraint violationrepresents the number of delayed jobs (jobs not severed in real-time).
In the experiment, we usedthe electricity price trace (i.e., t c t u ) between 05 { 01 { 2017 and 05 { 10 { 2017 at 10 different regions in
New York city from NYISO [1].
We calibrated job arrivals of a realistic trafﬁc pattern from [27] witha non-stationary Poisson process t λ t u to replace the stationary trafﬁc studied in [31].
We compared our algorithm with [30] and [31], where learning rates are summarized in Table 5 in
Appendix H. We plotted average energy costs and constraint violations in Figure 3.
It shows that ouralgorithm achieves better performance on the loss and constraint violation (the number of delayedjobs) compared to [30] and [31].
[Figure]
H Hyperparameters in our experiments
In this section, we provide the hyperparameters of tested algorithms in our three experiments.
The parameters in the experiment with ﬁxed constraints (Figure 1) are summarized in Table 3.
[Table]
Algorithm 1 in [34]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Algorithm 1 in [29]
[Equation]
Algorithm 2 in [29]
[Equation]
[Table]
[Equation]
[Table]
The parameters in the experiment with ﬁxed constraints (Figure 2) are summarized in Table 4.
[Table]
Algorithm 1 in [30]
[Table]
The parameters in the experiment with ﬁxed constraints (Figure 3) are summarized in Table 5.
[Table]
Algorithm 1 in [30]
[Table]
Algorithm 1 in [31]
[Table]