Online Convex Optimization with Hard Constraints: Towards the Best of Two Worlds and Beyond
Abstract
This paper considers online convex optimization with hard constraints and analyzes achievable regret and cumulative hard constraint violation (violation for short).
The problem distinguishes itself from online convex optimization with soft constraints, where a violation at one round can be compensated/cancelled by a conservative decision at a different round.
We propose a RECtiﬁed Online Optimization algorithm (RECOO) and consider two settings: ﬁxed constraints and adversarial constraints.
Both settings have been considered in the literature.
Compared with existing results, RECOO achieves the best of two worlds and beyond.
For the ﬁxed-constraints setting, RECOO achieves O ` ?
T ˘ regret and O p 1 q violation, where T is the learning horizon.
The best known results in this case are O p?
T q regret and O `
T 1 { 4 ˘ violation.
For the adversarial-constraints setting, it guarantees
O p?
T q regret and O p T 3 { 4 q violation, which match the best existing results.
When the loss function is strongly convex, RECOO can guarantee O p log T q regret and O p 1 q violation for ﬁxed constraints, and O p log T q regret and O p?
T log T q violation for adversarial constraints.
Both these results are order-wise better than the existing bounds.
The regret and violation bounds mentioned above use the best ﬁxed decision in hindsight as the baseline.
This paper further considers a dynamic baseline where the comparator sequence is time-varying.
This paper shows that RECOO not only improves the existing bounds for the ﬁxed-constraints setting but also for the ﬁrst time, establishes dynamic regret and violation bounds for the adversarial-constraints setting.
Our experiment results conﬁrm that RECOO outperforms several existing algorithms for both ﬁxed and adversarial constraints.
Introduction
Online convex optimization (OCO) is a general framework for modelling and studying online decisionmaking in uncertain environments [24, 10, 21], where the learner adapts its decisions to minimize a loss function or to maximize a utility function when interacting with the environment in real-time.
OCO has a broad range of applications such as online advertising [18, 3], resource allocation in network systems [9, 2, 33], load balancing in queueing systems [13, 15], and personalized healthcare [36, 26], etc.
Speciﬁcally, given an unknown convex loss function f t p¨q , the learner makes decision x t at each round to minimize the total loss over T rounds, i.e. min x P X ř Tt “ 1 f t p x t q , where the loss of decision x t is revealed after x t is executed.
In practice, the decisions are often subject to a variety of operational constraints.
For example, in online advertisement systems, users, who submit bids to advertise their items in order to maximize their click-through-rates, often have a weekly or monthly budget; and in a patient onboarding system, the hospital assigns an incoming patient to a medical unit to optimize the quality of treatment subject to personnel and medical resource constraints.
To consider these applications, we study constrained online convex optimization (COCO) with g t p¨q being a convex constraint function and g t p x t q ď 0 being the constraint at round t .
In COCO, the learner aims at minimizing the total loss while keeping the constraint violation to its minimum.
One possible method to satisfy the constraints is to project the decision x t into the feasible set at each round [10].
However, such projection may not be possible when the constraints g t p¨q are adversarial and unknown before hand.
Even if g t p¨q is known before hand, projection-based methods (e.g., projected online gradient descent) may require heavy computation when g t p¨q is a complicated function because it is equivalent to solving a constrained quadratic optimization problem [12, 17, 34].
This has motivated projection-free online learning methods such as the online Frank-Wolfe algorithm [12] where the projection operator at each round is by a linear programming with the exact same constraint functions.
Recently, a sequence of studies have considered COCO with soft constraints, where the constraints are allowed to be violated at some rounds as long as they are satisﬁed in the long term [17, 14, 20, 32, 25, 5, 4, 34, 27] (in other words, the severity of constraint violation is evaluated based on ř t g t p x t q ).
We call them soft constraints because the constraint violation at one round could be compensated in a different round.
In other words, a decision sequence might have zero constraint violation based on the metric despite violating the constraint(s) at almost every round.
For example, consider a decision sequence such that t g t p x t qu “ t´ 1000 , 1 , 1 , 1 , . .
, 1 , 1 u withones.
For such a sequence, we have ř τt “ 1 g t p x t q ď 0 for all τ ď T, but the constraint is violated at all rounds except the ﬁrst round.
While some constraints such as budget or fairness constraints are naturally soft constraints, stronger notions of constraint and constraint violation are needed for other applications, in particular, for safety-critical applications.
In this paper, we consider a stronger notion of constraint violation as in [35, 30, 31]
[Equation]
where that the operator p¨q ` “ max p¨ , 0 q is imposed at every round and all violations during the decision process are added up.
We call this hard constraint and the metric cumulative hard constraint violation .
If a decision sequence has a small V p T q , it implies the constraints are satisﬁed most of the time during the learning.
For example, consider the same example, V p T q “ 1000 instead of 0 under this stronger notion of constraint violation.
In this paper, we consider the following two settings:
• Fixed constraints g t p x q “ g p x q , @ t, where the constraint function remains the same over time but is not necessarily known to the learner.
Note the setting of known and ﬁxed constraints in [14, 17, 30, 34] is a special case of ours. • Adversarial constraints g t p x q , where the constraint function g t p x q is unknown when making decision at round t and can be arbitrarily and adversarially chosen, as in [25, 20, 31].
We will show that RECOO is a unifying algorithm that achieves small regret and violation in both settings.
Next, we summarize our main contributions and compare them with the most related works.
1.1 Main contributions
Algorithm: This paper develops a RECtiﬁed Online Optimization algorithm, called RECOO, for
COCO-Hard (COCO with hard constraints).
RECOO is a unifying algorithm for both ﬁxed and adversarial constraints.
By introducing rectiﬁers in both “decision-making" and “penalty update" components, RECOO maintains a minimum penalty price for constraint violation and encourages conservative/pessimistic decisions, which is the key to minimizing the cumulative hard constraint violation.
Regret and violation bounds: We ﬁrst consider the best ﬁxed decision in hindsight as the baseline (also called the static baseline).
The main theoretical results are summarized below.
• For ﬁxed constraints, RECOO achieves O p ?
T q regret and O p 1 q violation for convex loss functions, and O p log T q regret and O p 1 q violations for strongly-convex loss functions.
• For adversarial constraints, RECOO achieves O p?
T q regret and O p T 3 { 4 q violation for convex loss functions, and O p log T q regret and O p?
T log T q violation for strongly convex loss functions.
We then consider a dynamic baseline where the comparator is time-varying with path length P T and establish the following results
• For ﬁxed constraints, RECOO achieves O p?
P T T q regret and O p log T q violation, where P T is the path length of the feasible comparator sequence deﬁned by P T “ ř T ´ 1 t “ 1 } y t ` 1 ´ y t } with g t p
y t q ď 0 , @
t P r T s as in [37, 30].
• For adversarial constraints, RECOO achieves O ` P T ?
T ˘ regret and O p T 3 { 4 q violation.
The comparison with most related works are summarized in Tables 1 and 2, from which we can see that RECOO achieves and improves the state-of-the-art results for both settings (“N/A" means no results to the best of our knowledge).
Below are a few highlights we would like to mention:
• For ﬁxed constraints and the static baseline, RECOO not only improves the results in [30] but also answers a conjecture in [30] that O p 1 q violation may be achievable under Slater’s condition (the condition holds when there exist a positive (cid:15) ą 0 and x P X such that g t p x q ď ´ (cid:15), @
t P r T s ).
We proved that the conjecture is true even without Slater’s condition.
• For adversarial constraints, the static baseline, and strongly convex loss functions, RECOO reduces the O p T c q regret and O p T {+
+} ´ c { 2 q violation in [31] to O p log T q and O p?
T log T q , respectively. • For ﬁxed constraints and the dynamic baseline, RECOO improves the violation from O p?
T q in [30] to O p log T q , while maintaining the same regret bound. • For adversarial constraints and the dynamic baseline, RECOO achieves O ` P T ?
T ˘ regret and O p T 3 { 4 q violation, which are the ﬁrst regret and violation bounds in this setting.
Reference [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
Reference
[Table]
Analysis: The key idea behind these results is the rectiﬁed penalty imposed when making decisions, which enable us to develop a simple yet effective self-bounding relation that quantiﬁes the trade-off between the regret and the cumulative hard constraints.
The analysis is tailored for the cumulative hard constraints and is different from the drift-plus-penalty method used in [30, 31, 34], where the constraint violations have been established by bounding the dual variables (also called virtual queues) [Table]
1.2 Related work
COCO has been studied in the literature [17, 25, 20, 16, 6, 22, 35, 30, 31].
We have already presented a detailed comparison with the existing results on COCO-Hard.
We next review some of recent results on COCO-Soft, COCO with soft constraints.
COCO-Soft with ﬁxed constraints: [17] studies COCO-Soft and proposed an algorithm that achieves O p?
T q regret and O p T 3 { 4 q violation.
This result has been extended in [14] to characterize the regret-violation tradeoff, in particular, the proposed algorithm achieves O p T max t c, 1 ´ c u q regret and O p T 1 ´ c { 2 q violation, where c controls the trade-off.
Assuming Slater’s condition holds, the algorithm in [34] achieves O p?
T q regret and O p 1 q violation and is designed based on a powerful “drift-plus-penalty” method, which inspires some of the design in RECOO.
COCO-Soft with adversarial constraints: Adversarial constraints are more difﬁcult to satisfy but have been considered in the literature [25, 20, 16, 6, 22].
For COCO-Soft with adversarial constraints, the authors in [25] developed an online mirrored descent type algorithm that achieves O p?
T q regret and O p T 3 { 4 q violation.
Later, [6, 16] generalized the baseline in [25] and still achieve O p?
T q regret and O p T 3 { 4 q violation.
With Slater’s condition, [20] presents an online gradient descent algorithm based on the drift-plus-penalty method [19], which achieves O p?
T q regret and O p?
T q violation. [22] extended the result to an online optimization with sub-modular losses.
Note the key to reducing the constraint violation in these works is a reﬁned bound on virtual queues (or dual variables) using the Lyapunov drift method under Slater’s condition.
It remains open that whether O p T 3 { 4 q violation can be reduced with adversarial constraints (soft or hard) while keeping O p?
T q regret without Slater’s condition.
COCO-Hard
In this section, we formally deﬁne COCO-Hard.
Consider the following online convex optimization problem: At each round t P r T s , the learner makes decision x t and then observes the loss f t p x t q and constraint function g t p¨q after the decision is executed.
The goal of the learner is to generate a decision sequence t x 1 , x 2 , . . .
, x T ´ 1 , x T u to minimize the total loss
[Equation]
and the hard constraint violation ř Tt “ 1 g ` t p x t q .
To quantify the performance of an online convex optimization algorithm for COCO-Hard, we will ﬁrst compare it with a static baseline, called the best ﬁxed decision in hindsight, which is the solution to the following ofﬂine COCO.
Ofﬂine COCO
The ofﬂine COCO is formulated as follows
[Equation]
where X is a “simple” convex set (e.g., positive quadrant or probability simplex), t f t p¨qu t are convex loss functions and t g t p¨qu t are convex constraint functions.
The optimal solution x ˚ to ofﬂine COCO is called the best single decision in hindsight, a widely used baseline for COCO [10, 17, 25, 32].
Based on ofﬂine COCO, the regret and constraint violation of an online algorithm are deﬁned below.
Regret and cumulative hard constraint violation (or violation for short)
[Equation]
RECOO: A rectiﬁed online optimization algorithm
We present RECOO, an online optimization algorithm with the rectiﬁed decision and penalty update.
RECOO — A Rectiﬁed Online Optimization Algorithm
Initialization: x 0 P X , Q p 0 q “ 0 , f 0 p x q “ g 0 p x q “ 0 , @ x P X and learning rates α t , η t , γ t .
For t “ 1 , ¨ ¨ ¨ , T,
• Set: ˆ g ` t ´ 1 p x q
[Equation]
γ t
[Equation]
1 g ` t ´ 1 p x
[Equation]
• Rectiﬁed decision: ﬁnd the optimal solution of x t by solving:
[Equation]
• Observe: value ∇ f t p x t q and constraint function g t p¨q .
• Rectiﬁed penalty update: update the estimates of penalty variables Q p t q as follows:
[Equation]
In RECOO, the learning rates α t , η t , γ t are chosen differently depending on whether the loss function is convex or strongly convex, but are oblivious to the type of constraints (ﬁxed or adversarial).
The choices of learning rates can be found in the theorem statements.
We explain the intuition behind RECOO and the importance of the “rectiﬁers”.
The Lagrange function of the ofﬂine optimization problem (2) is deﬁned to be
[Equation]
where t λ t u are the dual variables associated with the constraints in (2).
Since we have no prior knowledge of f t p¨q when making decision x t , we estimate it with the ﬁrst-order approximation at x t ´ 1 based on the historical information as follows
[Equation]
Moreover, we replace the original constraint function g t p¨q with ˆ g ` t ´ 1 p¨q and the dual variables λ t with Q p t ´ 1 q such that Q p t ´ 1 q ˆ g ` t ´ 1 p x q is a rectiﬁed approximator of λ t g t p x q .
We also added the the regularization (or smooth) term α t } x ´ x t } 2 that helps the stability of the algorithm.
Note this design is related to penalty-based proximal optimization where we aim to minimize an approximated f t p x q w.r.t.
proximal operator on the “old” rectiﬁed function ˆ g ` t ´ 1 p x q .
In the penalty update of Q p t q , we ﬁrst rectify the original constraint function g t p¨q with ˆ g ` t p¨q and add it to Q p t ´ 1 q such that penalty increases when violation occurs in each round.
Further we rectify Q p t q with a round-dependent constant η t to impose a “minimum” penalty price.
The design of rectiﬁed penalty update induces conservative decisions in the decision-making step to minimize constraint violation.
Note that it is different with the traditional primal-dual algorithm that does not rectify the constraint violation and impose a minimum penalty price, and when the price (dual variable) is zero, the algorithm can take very aggressive decisions that lead to large hard violation.
This is not a problem when primal-dual algorithm is used as a numerical method for solving a constrained optimization problem, but leads to overly aggressive decisions and large violation when applying it to COCO.
For a similar reason, RECOO rectiﬁes the amount of violation g t p x q in the decision-making step so the violation does not become negative to prevent overly optimistic decisions.
We will see that the rectiﬁers in both decision-making and penalty update leads to an upper bound on “regret + violation” as a whole (Lemma 1), which further leads separate upper bounds on regret and (hard) constraint violation.
This approach is different from the primal-dual optimization that quantiﬁes the constraint violation indirectly by bounding the dual variables/virtual queues.
Finally, we comment that our algorithm only needs to solve an “almost” unconstrained optimization problem ( X is usually a simple set like the box constraints).
Therefore, we might ﬁnd its close-form with the inverse operation of the function by taking “gradient “ zero” or gradient-based methods are sufﬁcient to ﬁnd the minimizer.
For example, we could use the accelerated proximal-gradient method in [23] to ﬁnd the minimizer with a linear and dimension-free converge rate.
Our algorithm is more efﬁcient than the projection online gradient descent algorithm [38] and online Frank-Wolfe algorithm
[12], and has similar complexity with [30, 34].
Moreover, our algorithm does not assume any information on the type of constraints apriori and projection-based methods are only feasible when the constraint set is available beforehand.
We next analyze the regret and violation of RECOO as deﬁned in (3) and (4) based on the following standard technical assumptions on the feasible set, loss and constraint functions.
Assumption 1 The feasible set X is convex with diameter D such that } x ´ x 1 } ď D, @ x, x 1 P X .
Assumption 2 The loss function is convex and Lipschitz continuous with Lipschitz constant F such that | f t p x q ´ f t p x 1 q| ď F } x ´ x 1 } , @ x, x 1 P X , @
t.
Assumption 3 The constraint function is convex and Lipschitz continuous with Lipschitz constant G such that | g t p x q ´ g t p x 1 q| ď G } x ´ x 1 } , @ x, x 1 P X , @ t.
For ease of exposition, assume g 1 p
q ď 0 .
Note Assumption 1 implies the feasible set X is bounded.
Assumptions 2 and 3 mean the loss and constraint functions have bounded gradients.
With these assumptions, we are ready to present our main result of RECOO for both ﬁxed and adversarial constraints.
Theorem 1 Choose α t “ ?
t, η t “ ? t, and γ t “ t 12 ` ε , where ε ą 0 .
Under Assumptions 1-3,
RECOO algorithm achieves the following regret and constraint violation bounds:
[Equation]
Remark 1 For ﬁxed constraints, Theorem 1 establishes the optimal order-wise results of O p?
T q regret and O p 1 q violation since Ω p? T q regret is the well-known lower bound for OCO with general convex loss and O p 1 q is the best one can have.
For adversarial constraints, by carefully choosing the learning rates in RECOO, we can establish a trade-off of O p T c q regret and O p T 1 ´ c { 2 q cumulative violation where c P r 1 { 2 , 1 q (see details of Corollary 1 in Appendix C).
Moreover, Theorem 1 suggests that a large (cid:15) may imply small upper bounds of constraint violation.
Our experiment results in Section 4 show that a small (cid:15) (e.g., 0 .
01 or 0 . 1 ) is sufﬁcient to keep the violation small.
Next, we show Theorem 1 can be further improved when the loss function is strongly convex.
Assumption 4 The loss functions f t p¨q , @ t, are µ -strongly convex in X for µ ą 0 , i.e. f t p x 1 q ě f t p x q ` x x 1 ´ x, ∇ f t p x qy ` µ 2 } x ´ x 1 } 2 , @ x, x 1 P X , @
t.
With Assumption 4, we are able to establish a O p log T q regret whiling keeping the violation bounds at the same or a smaller order.
Theorem 2 Choose the learning rates to be α t “ µt 2 , η t “ ?
t, γ t “ t 12 ` ε , @
t P r T s , where ε ą 0 .
Under Assumptions 1-4, RECOO achieves the following regret and violation bounds:
[Equation]
Remark 2 The O p log T q regret is a well-known result in OCO (unconstrained) for a strongly convex loss function [11].
Therefore, for ﬁxed constraints, our results are order-wise sharp because O p 1 q violation is the best one can achieve.
Extension to a dynamic baseline
So far, the main results are against a static baseline x ˚ t “ x ˚ , @ t. We next analyze the performance of RECOO against a dynamic baseline as in [37, 30], where x ˚ t ` 1 could be different from x ˚ t but with limited variation such that ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } ď P T .
We deﬁne the regret against the dynamic baseline to be R dynamic p T q : “ ř Tt “ 1 f t p x t q ´ ř
Tt “ 1 f t p x ˚ t q .
Without any modiﬁcations, we show that RECOO algorithm can achieve O p ?
T p 1 ` P T qq regret and O p 1 q violation in the following theorem.
Theorem 3 Under Assumptions 1-3, let the learning rates be α t “ ?
t, η t “ ?
t, γ t “ t 12 ` ε , @
t P r T s , where ε ą 0 .
Let t x
˚ t u be the optimal solution to (2) with an additional constraint that ř T ´t “ 1 } x t ` 1 ´ x t } ď P T .
RECOO achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
or adversarial constraints) , [Equation] constraints)
[Equation] and
[Equation]
constraints [Equation] Remark 3 RECOO achieves O `
P T ?
T ˘ regret and O p 1 q violation for ﬁxed constraints, which improves O `
P T ?
T ˘ regret and O p T 1 { 4 q violation in [30].
Moreover, RECOO achieves O ` P T ?
T ˘ regret and O p T 3 { 4 q violation for adversarial constraints, which is a new result.
As shown in Corollary 2 in Appendix F, combining RECOO with expert-tracking proposed in [37], we achieve O ` ?
P T T ˘ regret and O p log T q violation for ﬁxed constraints, which improves O ` ?
P T T ˘ regret and O p?
T q violation in [30].
Next, we provide the detailed analysis of Theorem 1.
The proofs of Theorem 2 and Theorem 3 can be found in Appendix.
3.1 Proof of Theorem
We ﬁrst introduce a key lemma that establishes an upper bound on “regret plus violation” at each round t, which we called self-bounding property.
Lemma 1 (Self-Bounding Property) Let x be any feasible solution to ofﬂine COCO (2) and x t be the optimal solution returned by the RECOO algorithm.
We have
[Equation]
Deﬁne x ˚ to be the optimal solution to ofﬂine COCO in (2) and let x “ x ˚ in Lemma 1, we establish the regret and the constant violation bounds in Theorem 1.
ﬁxed constraints
Note that Q p t q ˆ g ` t p x t ` 1 q is nonnegative.
Based on (5), we have
[Equation]
which we will use next to establish the bounds on regret and violation
[Equation]
Regret bound: We sum up inequality (6) for t “ 1 , ¨ ¨ ¨ , T and have T
[Equation]
where the ﬁrst inequality holds by dropping the last negative term and the last inequality holds by
Assumption 1.
Choose α t “ ?
t, we have
[Equation]
Violation bound: We substitute the deﬁnition of ˆ g ` t p¨q in (7) and have
[Equation]
where the second inequality holds by Q p t
q ě η t according to the rectiﬁed penalty update Q p t q “ max
[Equation]
η
[Equation]
˘ .
The inequality implies that
[Equation]
where the second inequality holds by Lemma 6, which includes the detailed calculations of the three terms above.
Note that the left-hand-side of inequality (8) is not the violation because the index mismatch (the violation at round t is g ` t p x t q not g ` t p x t ` 1 q ).
For ﬁxed constraints, i.e. g t p x q “ g p x q , the inequality above implies that V p T q : “ ř T t “ 1 g ` p x t q ď
F 2 ` FD ` 1 ` 1 ε ˘ ` D 2 .
We have proved the ﬁrst part of Theorem 1 for the ﬁxed constraints.
Let us continue with (8) to prove the second part of Theorem 1 for adversarial constraints.
Proof of Theorem 1: adversarial constraints
To quantify V p T q : “ ř Tt “ 1 g ` t p x t q for the adversarial constraints, we need to establish the relationship between g ` t p x t q and g ` t p x t ` 1 q to address the index mismatch, which can be bounded by } x t ` 1 ´ x t } 2 as shown in the following lemma.
Lemma 2 Under Assumptions 1-3, RECOO achieves for any β ą[-
-]
[Equation]
The next lemma further quantiﬁes ř Tt “ 1 } x t ` [Equation] x t } 2 .
Lemma 3 Under Assumptions 1-3, RECOO achieves
[Equation]
Take summation of this equality in Lemma 2 from t “ 1 to T, and we have
[Equation]
where the last inequality holds by letting β “ T 1 { 4 .
Combining (8) and (9) completes the proof.
Experiments
In this section, we present synthetic and real dataset experiments for evaluating the performance of RECOO with ﬁxed and adversarial constraints.
We compared RECOO with the algorithms in [35, 30] for the ﬁxed-constraints setting; with the algorithm in [31] for the adversarial-constraints setting; and with the algorithms in [31, 32] for the real-dataset setting.
All results are obtained by averaging over 500 trials and reported with 95% conﬁdence interval.
Fixed constraints Similar to [30], we considered loss functions f t p x
[Equation] “ x θ p [Equation] , x y , where θ p t q is time varying and unknown at round t ; and the ﬁxed constraint function Ax ď b, where x P R 2 ,
A P R 3 ˆ 2 and b P R 3 .
We chose the number of total rounds to be T “ 5 , 000 , and chose θ p t q “ r θ 1 p t q , θ 2 p t q , θ 3 p t qs , where θ 1 p t q „ U p´ t 1 { 10 , ` t 1 { 10 q ; θ 2 p t q „ U p´ 1 , 0 q when t P r 1 , 1500 s Y r 2000 , 3500 s Y r 4000 , 5000 s and θ 2 p t q P U p 0 , 1 q otherwise; and θ 3 p t q “ p´ 1 q µ p t q where µ p t q is a random permutation of vector r 1 : 5000 s .
Let X “ t x |
ď x 1 ď 1 , 0 ď x 2 ď 1 u , A i,j „
U p 0 . 1 , 0 .
5 q , 1 ď i ď 3 , 1 ď j ď 2 , and b i „ U p 0 , 0 .
3 q , 1 ď i ď 3 , respectively.
We compared RECOO with the algorithm in [35] and Algorithm 1 and Algorithm 2 in [30], where the learning rates are summarized in Table 3 in Appendix G, which are the values suggested in [35] and [30].
Figure 1a illustrates the trajectories of the cumulative loss ř Tt “ 1 f t p x t q , where we observe that all algorithms have similar trends while RECOO achieves the best cumulative loss.
Figure 1b shows RECOO achieves the smallest cumulative violation.
In particular, the (mean, variance) pair of RECOO for loss and violation are p´ 510 .
78 q and p 0 .
008 q at the end of the learning horizon; that of Algorithm 1 in [30] are p´ 507 . 17 , 36 .
46 q and p 0 .
007 q ; that of Algorithm 2 in [30] are p´ 509 . 09 , 36 .
73 q and p 0 .
008 q ; and that of Algorithm 1 in [35] are p´ 504 .
82 q and p 980 .
49 q .
These results and Figure 1 conﬁrm that RECOO outperforms the existing algorithms w.r.t.
both cumulative loss and violation.
[Figure]
[Figure]
Experiment with adversarial constraints We considered an OCO with adversarial constraints similar as the one in [31].
The loss function was chosen to be f t p x q “ 1 2 } H p t q ¨ x ´ y p t q} 2 with
H p t q P R 4 ˆ 10 , x P R 10 and y p t q P R 4 where H i,j p t q „ U p´ 1 , 1 q , 1 ď i ď 4 , 1 ď j ď 10 , and y i p t
q “ ř i,jj “ 1 H i,j p t q ` ε i with ε i being the standard normal random variable for any i. The constraint functions were chosen to be g t p x q “ A p t q x ´ b p t q with A p t q P R 2 ˆ 10 and b p t q P R 2 , where A i,j p t q „ U p 0 , 2 q , @
i, j, t and b i p t
q „ U p 0 , 1 q , @
i, t, respectively.
We compared RECOO with Algorithm 1 in [31], where the learning rates are summarized in Table 4 in Appendix G. Figure 2 includes the cumulative losses and violations.
In particular, the (mean, variance) pair of RECOO for loss and violation are p 42510 .
05 q and p 713 . 45 , 1 .
60 q at the end of learning horizon, respectively while that of Algorithm 1 in [31] are p 43011 .
17 q and p 1684 .
17 , 2 . 85 q .
Therefore, RECOO performs better, especially in terms of the cumulative violation, which further shows the “rectiﬁed” design reduces the cumulative hard constraint violation.
Experiment of online job scheduling in distributed data centers We also tested our algorithm for online job scheduling in a distributed data center similar as in [32]. We considered a distributed data center with server clusters located at different regions.
The incoming jobs arrive at a front-end load balancer and will be scheduled to different clusters to fulﬁll the service. The service capability of a cluster is a function w.r.t. its energy consumption and the energy prices vary across locations and times. The goal is to minimize the energy cost while guaranteeing real-time processing of safety-critical jobs.
This problem can be formulated as a constrained online convex optimization problem and solved by our algorithm. Speciﬁcally, we considered r “ 10 regions, each region has 10 clusters, and each time/round t hasminutes. Let x t P R 100 be the energy allocation vector of all clusters at round t, where the i th entry is the energy allocation of cluster i . Let f t p x t q “ x c t , x t y , where c t P R 100 are the energy prices at time t. Let g t p x t q “ λ t ´ ři “ 1 h i p x t,i q , where λ t is the number of job arrivals during time t and h i p x t,i q “ 4 log p 1 ` 4 x t,i q is the service capacity of cluster i at round t .
The constraint violation represents the number of delayed jobs (jobs not severed in real-time). In the experiment, we used the electricity price trace (i.e., t c t u ) between 05 { 01 { 2017 and 05 { 10 { 2017 at 10 different regions in New York city from NYISO [1].
We calibrated job arrivals of a realistic trafﬁc pattern from [28] with a non-stationary Poisson process t λ t u to replace the stationary trafﬁc studied in [32].
We compared our algorithm with [31] and [32], where learning rates are summarized in Table 5 in
Appendix G. We plotted average energy costs and constraint violations in Figure 3. It shows that our algorithm achieves better performance on the loss and constraint violation (the number of delayed jobs) compared to [31] and [32].
[Figure]
Conclusions
In this paper, we studied online convex optimization with hard constraints (COCO-Hard) under two settings: ﬁxed constraints and adversarial constraints.
We proposed a RECtiﬁed Online Optimization algorithm (RECOO) and proved it achieves the best of two worlds and beyond.
The algorithm improves the best-known results for the ﬁxed-constraints, matches the best results for the adversarialconstraints, outperforms the state-of-art results when the loss functions are strongly-convex.
The experiments conﬁrmed our theoretical results.
A Auxiliary lemmas
The following lemma provides a useful upper bound on the optimal value of strongly convex function [32, 8, 29].
Lemma 4 Let X be a convex set.
Let h : X Ñ R be α -strongly convex function on X and x opt be an optimal solution of h, i.e., x opt “ arg min x P X h p x q .
Then, h p x opt q ď h p x q ´ α 2 } x ´ x opt } 2 for any x P X .
Proof The proof of the lemma is based on the deﬁnition of α -strongly convex functions and the ﬁrst-order optimality condition.
Deﬁne the subgradient of h p x q to be ∇ h p x q , According to the deﬁnition of strong convexity, we have
[Equation]
Deﬁne x opt “ arg min x P X h p x q .
Let y “ x opt in (10) , we have
[Equation]
We then conclude the proof based on the ﬁrst-order optimality condition that for any x P X , [Equation]
The following lemma is the key to bridge the regret and the constraint violation.
Lemma 5 Deﬁne
[Equation]
Let x t ` 1 be the optimal solution returned by RECOO, i.e., x t ` 1 “ arg min x P X h t p x q .
We have for any x P X that
[Equation]
Proof The proof is a direct application of Lemma 4.
Note that h t p x q is 2 α t -strongly convex because x ∇ f t p x t q , x ´ x t y ` Q p t q ˆ g ` t p x q is convex in x and α t } x ´ x t } 2 is 2 α t -strongly convex.
The following lemma is to provide the detailed calculations required for obtaining inequality (8) [Equation]
Lemma 6 Under Assumptions 1-3, we have
[Equation]
Proof Under Assumptions 1- [Equation] we calculate these three terms as follows:
[Equation]
B Proof of lemmas in TheoremB.1 Proof of Lemma
We prove the key self-bounding property in this section.
Since (11) holds for any x P X in Lemma 5, we have [Equation]
[Equation]
We add f t p x t q to both sides of (12) that
[Equation]
where the second inequality holds because f t p¨q is a convex function and the third inequality holds because x is an feasible solution to g t p x {+q ď 
+}
so ˆ g ` t p x q “ 0 .
By moving f t p x q to the left-hand side and α t } x t ` 1 ´ x t } 2 to the right-hand side of (13) [Equation] respectively [Equation]
we have
[Equation]
where the last inequality holds by Assumption
that
[Equation]
B.2 Proof of Lemma
Since g t p x q is convex function [Equation] we have g ` t p x q to be convex because max over convex functions is convex.
Denoting by ∇ g ` t p x q the subgradient of g ` t p x q , we have
[Equation]
Let y “ x t , x “ x t ` 1 , we have
[Equation]
where the second inequality holds because of Cauchy-Schwarz inequality and the third inequality holds because of Assumption
[Equation]
B.3 Proof of Lemma
According to Lemma 5, we have
[Equation]
where the last inequality holds because of the convexity of f t p¨q .
By g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 , we rearrange the inequality above and have
[Equation]
Summing the inequality above from t “ 1 to T, we have T ÿ
[Equation]
where the second inequality holds because
[Equation]
the last inequality holds because of α t “ ?
t and Assumption 1.
The proof is completed
[Equation]
C Corollary
Corollary 1 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s , where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
for both types of constraints,
[Equation]
for ﬁxed constraints
[Equation]
and
[Equation]
for adversarial constraints .
The proof of corollary closely follows the proof of Theorem 1 where c “ 1 { 2 .
Based on (5) in Lemma 1, we continue to have
[Equation]
[Equation]
which will be used to establish the bounds on regret and violation.
Regret Bound: we take summation of inequality (14) from t “ 1 , ¨ ¨ ¨ , T and have
[Equation]
where the last inequality holds by Assumption
[Equation] Choosing α t “ t c
[Equation] we have
[Equation]
It implies that
[Equation]
which gives the regret bound with O p T max t c, 1 ´ c u q .
Cumulative constraint violation bound: For constraint violation, we still have (15) and
[Equation]
By setting γ t “ t c and η t “ t c ` ε , where c P r 1 { 2 , 1 q and ε ą 0 , we obtain
T ÿ
[Equation]
By Lemma 6, we establish
[Equation]
For ﬁxed constraints, the inequality above proves that V p T q :
[Equation]
We next continue to prove the second part of Corollary 1 for adversarial constraints.
Recall Lemmathat under Assumptions 1-3, RECOO achieves for any β ą[-
-]
[Equation]
We next establish an upper bound on } x t ´ x t ` 1 } 2 in the following lemma.
Lemma 7 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s , where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO achieves
T ÿ
[Equation]
Proof Recall from Lemma 5 we have
[Equation]
Since g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 , by rearranging the inequality, we have
[Equation]
Summing the inequality above for t “ 1 , ¨ ¨ ¨ , T , we have T ÿ t “} x t ` 1 ´ x [Equation] [Equation]
´ x
[Equation]
Since α t “ t c , we have
T ÿ
[Equation]
which prove Lemma 7.
Recall in Lemma 2 and we have
[Equation]
Letting β “ T c { 2 , we establish
[Equation]
which proves the corollary.
D Proof of Theorem
In this section, we prove Theorem 2 which considers strongly convex loss functions.
According to the deﬁnition of µ -strongly convex, we have
[Equation]
We start with inequality (12) by using Lemma 5:
[Equation]
(18) where the last inequality holds according to the strongly convex condition of f t p¨q in (17).
By following the same steps that lead to inequality (5), we have
[Equation]
[Equation]
This is the key “self-bounding” property when f t p¨q is strongly convex function, from which we obtain
[Equation]
Recall that α t “ µt 2 , η t “ ?
t, and γ t “ t 12 ` ε , ε ą 0 .
We next establish the regret and violation bounds
[Equation]
Regret bound: From (20), we immediately have
[Equation]
¯
By choosing α t “ µt [Equation] we have
[Equation]
Following the same steps as in the proof of Theorem 1, we can have
T ÿ
[Equation]
which implies that
[Equation]
D.1 Violation bound: ﬁxed constraints
For ﬁxed constraints, inequality (23) implies for ﬁxed constraints V p T q : “ ř Tt “ 1 g ` p x t q ď F 2 µ `
[Equation]
˘ because the constraint is ﬁxed.
We next continue with (23) to prove the second part of Theorem 2 for the adversarial constraints.
D.2 Violation bound: adversarial constraints
By Lemma [Equation] we have
[Equation]
From (18) [Equation] we again establish a bound on ř
Tt “ 1 } x t ´ x t ` 1 } 2 as follows
[Equation]
[Equation]
where the second inequality holds because
[Equation]
; and the last inequality holds because of α t “ µt 2 and Assumption 1.
We combine (24) and (25) with β “ a
T {p 1 ` log T q as follows
[Equation]
Combining the inequality above with inequality (23), we establish the second part of Theorem 2 for adversarial constraints.
E Proof of Theorem
We prove Theorem 3 with a dynamic baseline similar to that in [30], which is the solution to the following ofﬂine OCO with constraints:
[Equation]
where (28) imposes a path-length constraint on the baseline solution which limits the change of t x t u .
Note that the solution to (26)-(28) with P T “ 0 reduces to the best ﬁxed decision in hindsight.
Let t x ˚ t u be the optimal solution to (26)-(28).
We deﬁne the regret and cumulative constraint violation as follows
[Equation]
Next, we state the key self-bounding property for establishing Theorem 3 by letting x “ x ˚ t in
Lemma 1.
Lemma 8 Let t x t u be the decision sequence generated by RECOO.
Under Assumptions 1-3, the following inequality holds for any sequence t x ˚ t u with x ˚ t P X , @ t,
[Equation]
[Equation]
Regret bound with dynamic baseline: Take the summation of the inequality in (31) above from t “ 1 to T, we have
[Equation]
[Equation]
Recall α t “ ?
t, we have
[Equation]
For the second term [Equation] we have
[Equation]
where the ﬁrst inequality holds because of Assumption 1 and the last inequality holds because
T ÿ
[Equation]
Therefore, we prove the regret in Theorem 3 as follows
[Equation]
Cumulative hard constraint violation bound: When α t “ η t “ ?
t, γ t “ t 12 ` ε with ε ą 0 , the proof follows the same steps in the proof of Theorem 1 in Section 3.1 because the deﬁnition of constraint violation is the same.
F Reﬁned results of RECOO with expert-tracking under the dynamic baseline
Motivated by [37], we combine RECOO with the expert tracking method in [7] to improve the performance bounds w.r.t.
P T , similar as in [30].
The intuition of the algorithm is to set up N parallel experts ( N RECOO algorithms) and to track the best one.
We state RECOO with expert tracking algorithm as follows.
A Rectiﬁed Online Optimization with Expert-Tracking
Initialization ( N Experts): f 0 p x q “ g 0 p x q “ 0 , @ x P X , x i, 0 P X , Q i p 0 q “ 0 .
The learning rates α i,t , η i,t , γ t , κ and w i, 1 “ N ` 1 i p i ` 1 q N , @
i P r N s .
For t “ 1 , ¨ ¨ ¨ , T, • Set: ˆ g
` t ´
p x q
[Equation]
γ t
[Equation]
` t ´ 1 p x q . • Rectiﬁed decision: ﬁnd the optimal solution of x i,t for each expert and output x t : x i,t
[Equation]
arg min
[Equation]
P X
[Equation]
• Observe: ∇ f t p¨q and g t p¨q .
• Rectiﬁed penalty update: update Q i p t q and w i,t as follows:
[Equation]
Before presenting the main result of RECOO with expert-tracking algorithm [Equation] we impose an additional assumption on the loss functions as in [37] [Equation]
Assumption 5 The loss functions are bounded by a constant C such that | f t p x q| ď C, @
x P X , @
t.
We are ready to show that RECOO with expert-tracking improves the bounds in Theorem 3 in the following corollary.
Corollary 2 Let N “
[Equation]
` 1 , κ “ 1 {?
T .
Let the learning rates be α
i,t “ ? t { 2 i ´ 1 , η i,t “ {+
+} i ´ 1 ?
t, γ t “ t 12 ` ε , @
i P r N s , where ε ą 0 .
Let t x
˚ t u be the optimal solution to (2) with P T ě ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
Under Assumptions 1-3, RECOO with expert-tracking achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
We ﬁrst introduce Lemma 1 in [37], which quantiﬁes the difference between the weighted output of all experts with the best expert.
Lemma 9 (Lemma 1 in [37]) Let t x
i,t u and t x t u be the sequence generated by RECOO with experttracking, we have
[Equation]
Substitute the learning rate of α i,t in (32) in the proof of Theorem 3, we have T
[Equation] [Equation]
Recall N “
X 12 log 2 p 1 ` T q \ ` 1 and P t ď TD , so there exists i 0 “ [Equation] such that
[Equation]
Regret bound with dynamic baseline: Letting i “ i 0 in (33), we have T ÿ
[Equation]
˙ where the second and the last inequalities hold by using (34).
Moreover, we have
T ÿ
[Equation]
where the ﬁrst inequality holds by the convexity of f t p¨q and the second inequality holds by Lemma 9.
Recall κ “ 1 ? T and w i, 1 “ N ` 1 i p i ` 1 q N , so we have
[Equation]
Combining all these inequalities, we have
[Equation]
which proves the regret in Corollary 2 and establishes the O p?
P T T q regret bound.
Violation bound: Since g ` t p x q is convex, we have
T
[Equation]
According to inequality (31), we have
[Equation]
Letting α i,t “ ?
t { 2 i ´ 1 , η i,t “ 2 i ´ 1 ?
t, γ t “ t 12 ` ε , ε ą 0 , we have
[Equation]
`
[Equation]
p
[Equation]
i
[Equation]
t `
[Equation]
η
[Equation]
which implies
T
[Equation]
Thus we have
[Equation]
which proves the violation bound in Corollary
[Equation]
G Hyperparameters used in our experiments
In this section, we provide the hyperparameters of tested algorithms in our three experiments.
The parameters in the experiment with ﬁxed constraints (Figure 1) are summarized in Table 3.
[Table]
[Equation]
Algorithm 1 in [35]
[Equation]
Algorithm 1 in [30]
[Equation]
Algorithm 2 in [30]
[Equation]
The parameters in the experiment with ﬁxed constraints (Figure 2) are summarized in Table 4.
[Table]
Algorithm 1 in [31]
[Table]
The parameters in the experiment with ﬁxed constraints (Figure 3) are summarized in Table 5.
[Table]
Algorithm 1 in [31]
[Table]
Algorithm 1 in [32]
[Equation]
[Table]