Online Convex Optimization with Hard Constraints: Towards the Best of Two Worlds and Beyond
Abstract
This paper considers online convex optimization with hard constraints and analyzesachievable regret and cumulative hard constraint violation (violation for short).
Theproblem distinguishes itself from online convex optimization with soft constraints,where a violation at one round can be compensated/cancelled by a conservative decision at a different round.
We propose a RECtiﬁed Online Optimization algorithm(RECOO) and consider two settings: ﬁxed (known) constraints and adversarial (unknown) constraints.
Both settings have been considered in the literature.
Comparedwith existing results, RECOO achieves the best of two worlds and beyond.
Forthe ﬁxed-constraints setting, RECOO achieves O ` ?
T ˘ regret and O p 1 q violation,where T is the learning horizon.
The best known results in this case are O p?
T qregret and O `
T 1 { 4 ˘ violation.
For the adversarial-constraints setting, it guarantees
O p?
T q regret and O p T 3 { 4 q violation, which match the best existing results.
Whenthe loss functions are strongly convex, RECOO can guarantee O p log T q regretand O p 1 q violation for ﬁxed constraints, and O p log T q regret and O p?
T log T qviolation for adversarial constraints.
Both these results are order-wise better thanthe existing bounds.
The regret and violation bounds mentioned above use thebest ﬁxed decision in hindsight as the baseline.
This paper further considers adynamic baseline where the comparator sequence is time-varying.
This papershows that RECOO not only improves the existing results in the ﬁxed-constraintssetting but also for the ﬁrst time, guarantees dynamic regret and violation boundsin the adversarial-constraints setting.
Our experiment results conﬁrm that RECOOoutperforms several existing algorithms for both ﬁxed and adversarial constraints.
Introduction
Online convex optimization (OCO) is a general framework for modelling and studying online decisionmaking in uncertain environments [21, 9, 19], where the learner adapts its decisions to minimize aloss function or to maximize a utility function when interacting with the environment in real-time.
OCO has a broad range of applications such as online advertising [16, 2], resource allocation innetwork systems [8, 1, 29], load balancing in queueing systems [11, 13], and personalized healthcare[32, 23], etc.
Speciﬁcally, given an unknown convex loss function f t p¨q , the learner makes decisionx t at each round to minimize the total loss over T rounds, i.e. min ř Tt “ 1 f t p x t q , where the loss ofdecision x t is revealed after x t is executed.
In practice, the decisions are often subject to a varietyof operational constraints.
For example, in online advertisement systems, users, who submit bids toadvertise their items in order to maximize their click-through-rates, often have a weekly or monthlybudget; and in a patient onboarding system, the hospital assigns an incoming patients to a medicalunit to optimize the quality of treatment subject to personnel and medical resource constraints.
Toconsider these applications, we study constrained online convex optimization (COCO) with g t p¨q being a convex constraint function and g t p x t q ď 0 being the constraint at round t .
In COCO, thelearner aims at minimizing the total loss while keeping the constraint violation to its minimum.
One possible method to satisfy the constraints is to project the decision x t into the feasible set at eachround [9].
However, such projection may not be possible when the constraints g t p¨q are adversarial andunknown before hand.
It also may require heavy computation when g t p¨q is a complicated function[15, 30].
This motivates a sequence of studies on COCO with soft constraints, where the constraintsare allowed to be violated as long as they are satisﬁed in the long term [15, 12, 18, 28, 22, 4, 3, 30, 24](in other words, the severity of constraint violation is evaluated based on ř t g t p x t q ).
We call themsoft constraints because the constraint violation at one round could be compensated in a differentround.
In other words, a decision sequence might have zero constraint violation based on the metricdespite violating the constraint(s) at almost every round.
For example, consider a decision sequencesuch that t g t p x t qu “ t´ 1000 , 1 , 1 , 1 , . .
, 1 , 1 looooooomooooooon 1, u .
For such a sequence, we have ř τt “ 1 g t p x t q ď 0 forall τ ď T, but the constraint is violated at all rounds except the ﬁrst round.
While some constraintssuch as budget or fairness constraints are naturally soft constraints, a stronger notion of constraintand constraint violation is needed for other applications, in particular, for safety-critical applications.
In this paper, we consider a stronger notion of constraint violation as in [31, 26, 27]
[Equation]
where that the operator p¨q ` “ max p¨ , 0 q is imposed on each round and all violations during thedecision process are added up.
We call this hard constraint and the metric cumulative hard constraintviolation .
If a decision sequence has a small V p T q , it implies the constraints are satisﬁed most of thetime during the learning.
For example, consider the same example, V p T q “ 1000 instead of 0 underthis stronger notion of constraint violation.
In this paper, we consider the following two settings:
• Fixed constraints g t p x q “ g p x q , @ t, where the constraint function is known (ﬁxed) when makingdecision at round t as in [15, 12, 30, 26]. • Adversarial constraints g t p x q , where the constraint function g t p x q is unknown when making decision at round t and can be arbitrarily and adversarially chosen, as in [22, 18, 27].
We will show that RECOO is a single algorithm that achieves small regret and violation in bothsettings.
Next, we summarize our main contributions and compare them with the most related works.
1.1 Main contributions
Algorithm: This paper develops a RECtiﬁed Online Optimization algorithm, called RECOO, for
COCO-Hard (COCO with hard constraints).
RECOO is a unifying algorithm for both ﬁxed andadversarial constraints.
By introducing rectiﬁers in both “decision-making" and “penalty update"components, RECOO maintains a minimum penalty price for constraint violation and encouragesconservative/pessimistic decisions, which is the key to minimizing the cumulative hard constraintviolation.
Regret and violation bounds: We ﬁrst consider the best ﬁxed decision in hindsight as the baseline(also called the static baseline).
The main theoretical results are summarized below.
• For ﬁxed constraints, RECOO achieves O p?
T q regret and O p 1 q violation for convex lossfunctions, and O p log T q regret and O p 1 q violations for strongly-convex loss functions.•
For adversarial constraints, RECOO achieves O p?
T q regret and O p T 3 { 4 q violation for convex loss functions, and O p log T q regret and O p?
T log T q violation for strongly convex lossfunctions.
We then consider a dynamic baseline where the comparator is time-varying with path length
P T andestablish the following results.
• For ﬁxed constraints, RECOO achieves O p?
P T T q regret and O p log T q violation, where P T isthe path length of the comparator sequence [33, 26].•
For adversarial constraints, RECOO achieves O ` P T ?
T ˘ regret and O p T 3 { 4 q violation.
The comparison with most related works are summarized in Tables 1 and 2, from which we can seethat RECOO achieves and improves the state-of-the-art results for both settings (“N/A" means noresults are reported).
Below are a few highlights we would like to mention:
[Table] • For ﬁxed constraints and the static baseline, RECOO not only improves the results in [26] butalso answers a conjecture in [26] that O p 1 q violation may be achievable under Slater’s condition.
We proved that the conjecture is true even without Slater’s condition.
[Table] • For adversarial constraints, the static baseline, and strongly convex loss functions, RECOOreduces the O p T c q regret and O p T 1 ´ c { 2 q violation in [27] to O p log T q and O p? T log T q ,respectively.
[Table]
• For ﬁxed constraints and the dynamic baseline, RECOO improves the violation from O p?
T q in[26] to O p log T q , while maintaining the same regret bound. [Table] • For adversarial constraints and the dynamic baseline, RECOO achieves O ` P T ?
T ˘ regret andO p T 3 { 4 q violation, which is the ﬁrst regret and violation bounds in this setting. Reference [Table]
Reference Loss Function
[Table] [Table]
[Table]
RECOO
[Table]
Analysis: The key method to establish these results is by imposing the rectiﬁed penalty when makingdecisions, which enable us to develop a simple yet effective self-bounding technique to quantify thetrade-off between the regret and the cumulative hard constraints.
These methods are tailored for thecumulative hard constraints and are different from the drift-plus-penalty method used in the relatedworks [26, 27, 30], where the constraint violations are established by bounding the dual variables(also called virtual queues).
1.2 Related work
COCO has been studied in the literature [15, 22, 18, 14, 5, 20, 31, 26, 27].
We have already presenteda detailed comparison with the existing results on COCO-Hard.
We next review some of recentresults on COCO-Soft, COCO with soft constraints.
COCO-Soft with ﬁxed constraints: [15] studies COCO-Soft and proposed an algorithm thatachieves O p?
T q regret and O p T 3 { 4 q violation.
This result has been extended in [12] to characterize the regret-violation tradeoff, in particular, the proposed algorithm achieves O p T max t c, 1 ´ c u qregret and O p T 1 ´ c { 2 q violation, where c controls the trade-off.
Assuming Slater’s condition holds,the algorithm in [30] achieves O p?
T q regret and O p 1 q violation and is designed based on a powerful“drift-plus-penalty” method, which inspires some of the design in RECOO.
COCO-Soft with adversarial constraints: Adversarial constraints are more difﬁcult to satisfy buthave been considered in the literature [22, 18, 14, 5, 20].
For COCO-Soft with adversarial constraints,the authors in [22] developed an online mirrored descent type algorithm that achieves O p?
T q regretand O p T 3 { 4 q violation.
Later, [5, 14] generalized the baseline in [22] and still achieve O p ?
T q regret and O p T 3 { 4 q violation.
With Slater’s condition, [18] presents an online gradient descent algorithmbased on the drift-plus-penalty method [17], which achieves O p?
T q regret and O p?
T q violationand [20] extended it to an online optimization with sub-modular losses.
It remains open that whether
O p T 3 { 4 q violation can be reduced with adversarial constraints (soft or hard) while keeping O p?
T qregret without Slater’ condition.
COCO-Hard
In this section, we formally deﬁne COCO-Hard.
Consider the following online convex optimizationproblem: At each round t P r T s , the learner makes decision x t and then observes the loss f t p x t qand constraint function g t p¨q after the decision is executed.
The goal of the learner is to generatea decision sequence t x 1 , x 2 , . . .
, x T ´ 1 , x T u to minimize the total loss
[Equation]
the
[Equation]
[Equation]
To quantify the performance of an online convex optimization algorithm for COCO-Hard, we willﬁrst compare it a static baseline, called the best ﬁxed decision in hindsight, which is the solution tothe following ofﬂine COCO.
Ofﬂine COCO
The ofﬂine COCO is formulated as follows
[Equation]
where X is a “simple” convex set (e.g., positive quadrant or probability simplex), t f t p¨qu t are convexloss functions and t g t p¨qu t are convex constraint functions.
The optimal solution x ˚ to ofﬂine COCOis called the best single decision in hindsight, a widely used baseline for COCO [9, 15, 22, 28].
Based on ofﬂine COCO, the regret and constraint violation of an online algorithm are deﬁned below.
Regret and cumulative hard constraint violation (or violation for short)
We deﬁne the regretand cumulative constraint violation with respect to the baseline as follows
[Equation]
RECOO: a rectiﬁed online optimization algorithm
We present RECOO, an online optimization algorithm with the rectiﬁed decision and penalty update.
RECOO — A Rectiﬁed Online Optimization Algorithm
Initialization: x 0 P X , Q p 0 q “ 0 , f 0 p x q “ g 0 p x q “ 0 , @ x P X and learning rates α t , η t , γ t .
For t “ 1 , ¨ ¨ ¨ , T,• Set: ˆ g ` t ´ 1 p x q “ γ t ´ 1 g ` t ´ 1 p x q .
• Rectiﬁed decision: ﬁnd the optimal solution of x t by solving:
[Equation]
• Observe: value ∇ f t p x t q and constraint function g t p¨q .
• Rectiﬁed penalty update: update the estimates of penalty variables Q p t q as follows:
[Equation]
In RECOO, the learning rates α t , η t , γ t are chosen differently depending on the loss function isconvex or strongly convex, but are oblivious to the type of constraints (ﬁxed or adversarial).
Thechoices of learning rates can be found in the theorem statements.
We explain the intuition behind RECOO and the importance of the “rectiﬁers”.
The Lagrange function of the ofﬂine optimization problem (2) is deﬁned to be
[Equation]
where t λ t u are the dual variables associated with the constraints in (2).
Since we have no prior knowledge of f t p¨q when making decision x t , we estimate it with the ﬁrst-order approximation at x t ´ 1 based on the historical information as follows
[Equation]
Moreover, we replace the original constraint function g t p¨q with ˆ g ` t ´ 1 p¨q and the dual variables λ twith Q p t ´ 1 q such that Q p t ´ 1 q ˆ g ` t ´ 1 p x q is a rectiﬁed approximator of λ t g t p x q .
We also added thethe regularization (or smooth) term α t } x ´ x t } 2 that helps the stability of the algorithm.
In the penalty update of Q p t q , we ﬁrst rectify the original constraint function g t p¨q with ˆ g ` t p¨q and addit to Q p t ´ 1 q such that penalty increases when violation occurs in each round.
Further we rectify Q p t qwith a round-dependent constant η t to impose a “minimum” penalty price.
The design of rectiﬁedpenalty update induces conservative decisions in the decision-making step to minimize constraintviolation.
Note that it is different with the traditional primal-dual algorithm that does not rectifythe constraint violation and impose a minimum penalty price, and when the price (dual variable)is zero, the algorithm can take very aggressive decisions that lead to large hard violation.
This isnot a problem when primal-dual algorithm is used as a numerical method for solving a constrainedoptimization problem, but leads to overly aggressive decisions and large violation when applying it to COCO.
For a similar reason, RECOO rectiﬁes the amount of violation g t p x q in the decision-makingstep so the violation does not become negative to prevent overly optimistic decisions.
We will see thatthe rectiﬁers in both decision-making and penalty update lead to a small cumulative hard constraintviolation ř Tt “ 1 g ` t p x t q .
We next analyze the regret and violation of RECOO as deﬁned in (3) and (4) based on the followingstandard technical assumptions on the feasible set, loss and constraint functions.
Assumption 1 The feasible set X is convex with diameter D such that } x ´ x 1 } ď D, @ x, x 1 P X .
Assumption 2 The loss function is Lipschitz continuous with Lipschitz constant F such that | f t p x q´f t p x 1 q| ď F } x ´ x 1 } , @ x, x 1 P X , @ t.
Assumption 3 The constraint function is Lipschitz continuous with Lipschitz constant G such that| g t p x q ´ g t p x 1 q| ď G } x ´ x 1 } , @ x, x 1 P X , @ t.
For ease of exposition, assume g 1 p
q ď 0 .
Note Assumption 1 implies the feasible set X is bounded.
Assumptions 2 and 3 mean the loss andconstraint functions have bounded gradients.
With these assumption, we are ready to present ourmain result of RECOO for both ﬁxed and adversarial constraints.
Theorem 1 Choose α t “ ?
t, η t “ ? t, and γ t “ t 12 ` ε , where ε ą 0 .
Under Assumptions 1-3,
RECOO algorithm achieves the following regret and constraint violation bounds:
[Equation]
constraints
[Equation]
` D 2 (ﬁxed constraints), and
[Equation] constraints) [Equation] Remark 1 For the ﬁxed constraints, Theorem 1 establishes the optimal order-wise results of O p ?
T qregret and O p 1 q violation since Ω p T q regret is the lower bound for OCO with general convex lossand O p 1 q is the best one can have.
For the adversarial constraints, by carefully choosing the learningrates in RECOO algorithm, we can establish a trade-off of O p T c q regret and O p T 1 ´ c { 2 q cumulativeviolation where c P r 1 { 2 , 1 q (see details of Corollary 1 in Appendix C).
Next, we show Theorem 1 can be further improved when the loss function is strongly convex.
Assumption 4 The loss functions f t p¨q , @ t, are µ -strongly convex in X for µ ą 0 , i.e. f t p x 1 q ěf t p x q ` x x 1 ´ x, ∇ f t p x qy ` µ 2 } x ´ x 1 } 2 , @ x, x 1 P X , @
t.
With Assumption 4, we are able to establish a O p log T q regret and violation bounds at the same or asmaller order.
Theorem 2 Choose the learning rates to be α t “ µt 2 , η t “ ?
t, γ t “ t 12 ` ε , @
t P r T s , where ε ą 0 .
Under Assumptions 1-4, RECOO achieves the following regret and violation bounds:
[Equation]
or
[Equation]
constraints
[Equation]
Remark 2 The O p log T q regret is a well-known result in OCO (unconstrained) for a strongly convexloss function [10].
Therefore, for ﬁxed constraints, our results are order-wise sharp because O p 1 qviolation is the best one can achieve.
Extension to a dynamic baseline
In this paper, we establish the main results against a static baseline x ˚ t “ x ˚ , @ t, to convey theidea of RECOO algorithm that can improve the violation in Theorems 1 and 2.
In fact, RECOOcan be generalized to a dynamic baseline as in [33, 26], where x ˚ t ` 1 could be different with x ˚ t butwith limited variation deﬁned by P T “ ř T [Equation]
Deﬁne [Equation] Tt “ 1 f t p x t q ´ 199 ř
Tt “ 1 f t p x ˚ t q .
Without any modiﬁcations, we show that RECOO algorithm can achieve O p?
T p 1 `
P T qq regret and O p 1 q violation in the following corollary.
Theorem 3 Under Assumptions 1-3, let the learning rates be α t “ ?
t, η t “ ?
t, γ t “ t 12 ` ε , @
t Pr T s , where ε ą 0 .
Let t x
˚ t u be the optimal solution to (2) with P T “ ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
RECOOalgorithm achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
Remark 3 RECOO algorithm can achieve O `
P T ?
T ˘ regret and O p 1 q violation for ﬁxed constraints, which improves O `
P T ?
T ˘ regret and O p T 1 { 4 q violation in [26].
Moreover, RECOOalgorithm achieve O `
P T ?
T ˘ regret and O p T 3 { 4 q violation for adversarial constraints, which is newin the literature.
As shown in Corollary 2 in Appendix F, combining RECOO with expert-trackingtechniques proposed in [33], we achieve O ` ?
P T T ˘ regret and O p log T q violation for the case ofﬁxed constraints, which improves O ` ?
P T T ˘ regret and O p?
T q violation in [26].
Next, we provide the detailed analysis on Theorem 1.
The proofs of Theorem 2 and Theorem 3 canbe found in Appendix.
3.1 Proof of Theorem
We ﬁrst introduce a key lemma that establishes an upper bound on “regret plus violation” at eachround t, which we called self-bounding property.
Lemma 1 (Self-Bounding Property) Recall that x ˚ denotes the optimal solution to ofﬂine COCO(2) and x t be the optimal solution returned by RECOO algorithm.
We have
[Equation]
Based on Lemma 1, we establish the regret and the constant violation bounds in Theorem 1.
Proof of Theorem 1: ﬁxed constraints
Based on (5), we have
[Equation]
which we will use next to establish the bounds on regret and violation.
Regret bound: We sum up Inequality (6) for t “ 1 , ¨ ¨ ¨ , T and have
[Equation]
where the ﬁrst inequality holds by dropping the last negative term and the last inequality holds by
Assumption 1.
Choose α t “ ?
t, we have
[Equation]
Violation bound: We substitute the deﬁnition of ˆ g ` t p¨q in (7) and haveg ` t
[Equation]
where the second inequality holds by Q p t
q ě η t according to the rectiﬁed penalty update Q p t q “max
[Equation]
˘ .
The inequality implies that
[Equation]
[Equation]
where the second inequality holds by Lemma 6, which includes the detailed calculations of the threeterms above.
Note that the left-hand-side of inequality (8) is not the violation because the indexmismatch (the violation at round t is g ` t p x t q not g ` t p x t ` 1 q ).
For ﬁxed constraints, i.e. g t p x q “ g p x q , the inequality above implies that V p T q : “ ř Tt “ 1 g ` p x t q ď
F 2 ` FD ` 1 ` 1 ε ˘ ` D 2 .
We have proved the ﬁrst part of Theorem 1 for the ﬁxed constraints.
Let uscontinue with (8) to prove the second part of Theorem 1 for adversarial constraints.
Proof of Theorem 1: adversarial constraints
To quantify V p T q : “ ř Tt “ 1 g ` t p x t q for the adversarial constraints, we need to establish the relationship between g ` t p x t q and g ` t p x t ` 1 q to address the index mismatch, which can be bounded by} x t ` 1 ´ x t } 2 as shown in the following lemma.
Lemma 2 Under Assumptions 1-3, RECOO achieves for any β ą
[Equation]
The next lemma further quantiﬁes ř Tt “ 1 } x t ` 1 ´ x t } 2 .
Lemma 3 Under Assumptions 1-3, RECOO achieves
[Equation]
Take summation of this equality in Lemma 2 from t “ 1 to T, and we have T
[Equation]
where the last inequality holds by letting β “ T 1 { 4 .
Combining (8) and (9) completes the proof.
Experiments
In this section, we present numerical experiments for evaluating the performance of RECOO withﬁxed and adversarial constraints.
We compared RECOO with the algorithms in [31, 26]for theﬁxed-constraints setting; and with the algorithm in [27] for the adversarial-constraints setting.
Allresults are obtained by averaging over 500 trials and reported with 95% conﬁdence interval.
Fixed constraints Similar to [26], we considered loss functions f t p x q “ x θ p t q , x y , where θ p t q istime varying and unknown at round t ; and the ﬁxed constraint function Ax ď b, where x P R 2 ,
A P R 3 ˆ 2 and b P R 3 .
We chose the number of total rounds to be T “ 5000 , and chose θ p t q “r θ 1 p t q , θ 2 p t q , θ 3 p t qs , where θ 1 p t q „ U p´ t 1 { 10 , ` t 1 { 10 q , θ 2 p t q „ U p´ 1 , 0 q when t P r 1 , 1500 s Yr 2000 , 3500 s Y r 4000 , 5000 s and θ 2 p t q P U p 0 , 1 q otherwise; θ 3 p t q “ p´ 1 q µ p t q where µ p t q is arandom permutation of vector r 1 : 5000 s .
Let X “ t x |
ď x 1 ď 1 , 0 ď x 2 ď 1 u , A i,j „
U p 0 . 1 , 0 .
5 q , 1 ď i ď 3 , 1 ď j ď 2 , and b i „ U p 0 , 0 .
3 q , 1 ď i ď 3 , respectively.
We compared RECOO with Algorithm in [31] and Algorithm 1 and Algorithm 2 in [26], where learning rates are summarized in Table 3 as suggested in [31] and [26].
Figure 1a illustrates the trajectoriesof the cumulative losses ř Tt “ 1 f t p x t q , where we observe all of these algorithms have similar trendswhile RECOO achieves the best cumulative losses.
Figure 1b shows RECOO achieves the smallestcumulative violation.
In particular, the (mean, variance) pair of RECOO for loss and violation arep´ 510 .
78 q and p 0 .
008 q at the end of learning horizon, respectively; that of Algorithm 1in [26] are p´ 507 .
46 q and p 0 .
007 q ; that of Algorithm 2 in [26] are p´ 509 . 09 , 36 .
73 q andp 0 .
008 q ;that of Algorithm 1 in [31] are p´ 504 .
82 q and p 980 .
49 q .
These resultsand Figure 1 conﬁrm that RECOO outperforms the existing algorithms w.r.t.
both cumulative lossand violation.
[Table]
[Equation]
Algorithm 1 in [31]
[Equation]
Algorithm 1 in [26]
[Equation]
Algorithm
[Equation] [Equation] [Figure] Experiment with adversarial constraints We considered an OCO with adversarial constraintssimilar as the one in [27].
The loss function was chosen to be f t p x q “ 12 } H p t q ¨ x ´ y p t q} 2 with
H p t q P R 4 ˆ 10 , x P R 10 and y p t q P R 4 where H i,j p t q „ U p´ 1 , 1 q , 1 ď i ď 4 , 1 ď j ď 10 ,and y i p t
q “ ř i,jj “ 1 H i,j p t q ` ε i with ε i being the standard normal random variable for any i. Theconstraint functions were chosen to be g t p x q “ A p t q ¨ x ´ b p t q with A p t q P R 2 ˆ 10 and b p t q P R 2 ,where A i,j p t q „ U p 0 , 2 q , @
i, j, t and b i p t
q „ U p 0 , 1 q , @
i, t, respectively.
We compared RECOO with Algorithm 1 in [27], with α t “ η t “ ? t , γ t “ t 1 { 2 ` 0 . 01 in RECOO; andα t “ 0 . 8 {? t, β t “ 5 {? t and γ t “ 0 . 5 {? t for Algorithm 1 in [27] (these are the optimized learningrates). Figure 2 includes the cumulative losses and violations.
In particular, The (mean, variance)pair of RECOO for loss and violation are p 42510 .
05 q and p 713 . 45 , 1 .
60 q at the end of learninghorizon, respectively while that of Algorithm 1 in [27] are p 43011 .
17 q and p 1684 .
17 , 2 . 85 q .
Itis justiﬁed RECOO performs better, especially in terms of the cumulative violation, which furthershows the “rectiﬁed” design reduces the cumulative hard constraint violation.
[Figure]
Conclusions
In this paper, we studied online convex optimization with hard constraints (COCO-Hard) undertwo settings: ﬁxed (known) constraints and adversarial (unknown) constraints.
We proposed a RECtiﬁed Online Optimization algorithm (RECOO) and proved it achieves the best of two worldsand beyond.
The algorithm improves the best-known results for the ﬁxed-constraints, matches thebest results for the adversarial-constraints, outperforms the state-of-art results when the loss functionsare strongly-convex.
The experiments conﬁrmed our theoretical results.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No] We didn’t ﬁnd any major limita-
tions in applying our results to constrained online convex programming.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A](
e) Did you discuss whether the data you are using/
curating contains personally identiﬁableinformation or offensive content?
[N/A
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Auxiliary lemmas
The following lemma provides a useful upper bound on the optimal value of strongly convex function[28, 7, 25].
Lemma 4 Let X be a convex set.
Let h : X Ñ R be α -strongly convex function on X and x opt bean optimal solution of h, i.e., x opt “ arg min x P X h p x q .
Then, h p x opt q ď h p x q ´ α 2 } x ´ x opt } 2 forany x P X .
Proof The proof of the lemma is based on the deﬁnition of α -strongly convex functions and theﬁrst-order optimality condition.
Deﬁne the subgradient of h p x q to be ∇ h p x q , According to thedeﬁnition of strong convexity, we have
[Equation]
Deﬁne x opt “ arg min x P X h p x q .
Let y “ x opt in (10) , we have
[Equation]
We then conclude the proof based on the ﬁrst-order optimality condition that for any x P X , x ∇ h p x opt q , x ´ x opt y ě 0 .
The following lemma is the key to bridge the regret and the constraint violation.
Lemma 5 Deﬁne
[Equation]
Let x t ` 1 be the optimal solution returned by RECOO, i.e., x t ` 1 “ arg min x P X h t p x q .
We have forany x P X that
[Equation]
Proof The proof is a direct application of Lemma 4.
Note that h t p x q is 2 α t -strongly convex becausex ∇ f t p x t q , x ´ x t y ` Q p t q ˆ g ` t p x q is convex in x and α t } x ´ x t } 2 is 2 α t -strongly convex.
The following lemma is to provide the detailed calculations required for obtaining inequality (8) [Equation]
Lemma 6 Under Assumptions 1-3, we have
[Equation]
Proof Under Assumptions 1- [Equation] we calculate these three terms as follows:
[Equation]
B Proof of lemmas in Theorem 1B.1 Proof of Lemma 1
We prove the key self-bounding property in this section.
Since (11) holds for any x P X in Lemma 5,let x “ x ˚ such that
[Equation]
We add f t p x t q to both sides of (12) that
[Equation]
where the second inequality holds because f t p¨q is a convex function; the third inequality holdsbecause x ˚ is an feasible point g t p x ˚
q ď 0 such that ˆ g ` t p x ˚ q “ 0 holds.
By moving f t p x ˚ q to the left-hand side and α t } x t ` 1 ´ x t } 2 to the right-hand side of (13), respectively,we have
[Equation]
where the last inequality holds by Assumption
that
[Equation]
B.2 Proof of Lemma 2
Since g t p x q is convex function, we have g ` t p x q to be convex because max over convex functions is convex.
Denoting by ∇ g ` t p x q the subgradient of g ` t p x q , we have
[Equation]
Let y “ x t , x “ x t ` 1 , we have
[Equation]
where the second inequality holds because of Cauchy-Schwarz inequality; the third inequality holdsbecause of Assumption 3.
Take summation of the equality above from 1 to T, we have
[Equation]
B.3 Proof of Lemma
According to Lemma 5, we have
[Equation]
[Table]
ď f
[Table]
p
[Table]
ď
[Table]
where the last inequality holds because of the convexity of f t p¨q .
By g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 ,we rearrange the inequality above and have
[Equation]
Take summation of the inequality above from t “ 1 to T, we have
[Equation]
where the second inequality holds because
[Equation]
the last inequality holds because of α t “ ?
t and Assumption 1.
The proof is completed.
C Corollary
Corollary 1 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s ,where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO algorithm achieves the following bounds on the regret andcumulative constraint violations:
[Equation]
for both types of constraints,
[Equation]
ﬁxed constraints [Equation] and
[Equation]
for adversarial constraints
[Equation]
The proof of corollary follows almost the same steps as in Theorem 1 where c “ 1 { 2 .
Based on (5)in Lemma 1, we still have
[Equation]
which is used to establish the bounds of regret and violation.
Regret Bound: we take summation of the inequality (14) from t “ 1 , ¨ ¨ ¨ , T and have
[Equation]
where the last inequality holds by Assumption 1.
Choose α t “ t c , we have
[Equation]
It implies that
[Equation]
which gives the regret bound with O p T max t c, 1 ´ c u q .
Cumulative constraint violation bound: For constraint violation, we still have (15) andg `
[Equation]
p x
[Equation]
Set γ t “ t c and η t “ t c ` ε , where c P r 1 { 2 , 1 q and ε ą 0 , this implies that
T
[Equation]
By Lemma 6, we establish
[Equation]
[Equation]
which proves V p T q : “ ř Tt “ 1 g ` p x
[Equation]
q ď F 2 ` FD ` FDε ` D 2 .
Let’s continue with (16) to prove thesecond part of Corollary 1 for the adversarial constraints.
Recall Lemma 2 that under Assumptions1-3, RECOO achieves for any β ą
[Equation]
From Lemma 2, it is required to quantify the stability term } x t ´ x t ` 1 } 2 that is established in thefollowing lemma.
Lemma 7 Under Assumptions 1-3, let the learning rates be α t “ t c , η t “ t c , γ t “ t c ` ε , @
t P r T s ,where c P r 1 { 2 , 1 q and ε ą 0 .
RECOO achieves
[Equation]
Proof From Lemma 5,we still have:
[Equation]
According to g ` t p x t ` 1 q ě 0 and g ` t p x ˚ q “ 0 and rearrange the inequality, we have
[Equation]
Take summation of the inequality above, we have
T ÿ
[Equation]
} 2 q .
Since α t “ t c , we have:
T ÿ
[Equation]
which proves Lemma 7.
Recall in Lemma 2 and we have
[Equation]
˙ .
Let β “ T c { 2 , we establish
[Equation]
which proves the corollary.
D Proof of Theorem
In this section, we prove Theorem 2 which considers strongly convex loss functions.
According tothe deﬁnition of µ -strongly convex, we have
[Equation]
We start with inequality (12) by using Lemma 5:
[Equation]
(18) where the last inequality holds according to the strongly convex condition of f t p¨q in (17).
Byfollowing the same steps that lead to inequality (5), we have
[Equation]
[Equation]
This is the key “self-bounding” property when f t p¨q is strongly convex function, from which weobtain
[Equation]
Recall that α t “ µt 2 , η t “ ?
t, and γ t “ t 12 ` ε , ε ą 0 .
We next establish the regret and violationbounds.
Regret bound: From (20), we immediately have
[Equation]
¯
By the choice of α t “ µt 2 , we have
T ÿ
[Equation]
Following the same steps as in the proof of Theorem 1, we establish
T
[Equation]
(22) which implies that
[Equation]
D.1 Violation bound:
ﬁxed constraints
For ﬁxed constraints, inequality (23) implies V p T q : “ ř Tt “ 1 g ` p x t q ď F 2 µ ` FD ` 1 ` 1 (cid:15) ˘ becausethe constraint is ﬁxed.
We have proved the ﬁrst part of Theorem 2 for the ﬁxed constraints.
Let’scontinue with (23) to prove the second part of Theorem 2 for the adversarial constraints.
D.2 Violation bound: adversarial constraints
By Lemma 2, we have
[Equation]
From (18), we again establish the bound on ř
Tt “ 1 } x t ´ x t ` 1 } 2 as follows
[Equation]
(25) where the second inequality holds because
[Equation]
the last inequality holds because of α t “ µt 2 and Assumption 1.
We combine (24) and (25) withβ “ a
T {p 1 ` log T q as follows
[Equation]
Combining the inequality above with inequality (23), we establish the second part of Theorem 2 foradversarial constraints.
E Proof of Theorem 3
We prove Theorem 3 with a dynamic baseline similar to that in [26], which is the solution to thefollowing ofﬂine OCO with constraints:
[Equation]
where (28) imposes a path-length constraint on the baseline solution which limits change of t x t u .
Note that the solution to (26)-(28) with P T “ 0 reduces to the best ﬁxed decision in hindsight.
Lett x ˚ t u be the optimal solution to (26)-(28).
We deﬁne the regret and cumulative constraint violation asfollows
[Equation]
Next, we state the key self-bounding property for establishing Theorem 3.
Lemma 8 Let t x t u be the decision sequence generated by RECOO.
Under Assumptions 1-3, thefollowing inequality holds for any sequence t y t u with y t P X , @ t,
[Equation]
Proof According to Lemma 5, we have
[Equation]
We add f t p x t q to both sides of the inequality above
[Equation]
which proves the lemma because
[Equation]
Regret bound with dynamic baseline: Let the optimal sequence be t x ˚ t u .
Substitute t y t u “ t x ˚ t uin (31).
By g t p x ˚ t q ď 0 , we have
[Equation]
Take the summation of the inequality above from t “ 1 to T
[Equation]
we have
[Equation]
Recall α t “ ?
t, we have
[Equation]
For the second term, we have
[Equation]
where the ﬁrst inequality holds because of Assumption 1 and the last inequality holds because
T ÿ t “
[Equation]
Therefore
[Equation]
we prove the regret in Theorem 3 as follows
[Equation]
Cumulative hard constraint violation bound: When α t “ η t “ ?
t, γ t “ t 12 ` ε with ε ą 0 , theproof follows the same steps in the proof of Theorem 1 in Section 3.1 because the deﬁnition ofconstraint violation is the same.
F Reﬁned results of RECOO with expert-tracking under the dynamicbaseline
Motivated by [33], we combine RECOO with expert tracking techniques in [6] to improve theperformance bounds w.r.t.
P T , similar in [26].
The intuition of the algorithm is to set up N parallelexperts ( N RECOO algorithms) and to track the best one.
We state RECOO with expert trackingalgorithm as follows.
A Rectiﬁed Online Optimization with Expert-Tracking Algorithm
Initialization ( N Experts): f 0 p x q “ g 0 p x q “ 0 , @ x P X , x i, 0 P X , Q i p 0 q “ 0 .
The learning ratesα i,t , η i,t , γ t , κ and w i, 1 “ N ` 1 i p i ` 1 q N , @
i P r N s .
For t “ 1 , ¨ ¨ ¨ , T,
• Set:
ˆ g ` t ´ 1 p x q “ γ t ´ 1 g ` t ´ 1 p x q .•
Rectiﬁed decision: ﬁnd the optimal solution of x i,t for each expert and output x t :x i,t “
[Equation]
min
[Equation]
X
[Equation]
[Equation]
x t
[Equation]
N ÿ
[Equation]
x
[Equation]
,t • Observe: ∇ f t p¨q and g t p¨q .
• Rectiﬁed penalty update: update Q i p t q and w i,t as follows:
[Equation]
Before presenting the main result of RECOO with expert-tracking algorithm, we impose an additionalassumption on the loss functions as in [33].
Assumption 5 The loss functions are bounded by a constant C such that | f t p x q| ď C, @
x P X , @
t.
We are ready to show RECOO with expert-tracking algorithm improves Theorem 3 in the following.
Corollary 2 Let N “
[Equation]
[Equation]
\ ` 1 , κ “ 1 {?
T .
Let the learning rates be α
i,t “ 543 ? t { 2 i ´ 1 , η i,t “ {+
+} i ´ 1 ?
t, γ t “ t 12 ` ε , @
i P r N s , where ε ą 0 .
Let t x
˚ t u be the optimal solutionto (2) with P T “ ř T ´ 1 t “ 1 } x ˚ t ` 1 ´ x ˚ t } .
Under Assumptions 1-3, RECOO with expert-tracking algorithm achieves the following bounds on the regret and cumulative constraint violations:
[Equation]
We ﬁrst introduce Lemma 1 in [33], which quantiﬁes the difference between the weighted output ofall experts with the best expert.
Lemma 9 (Lemma 1 in [33]) Let t x
i,t u and t x t u be the sequence generated by RECOO with experttracking algorithm, we have
[Equation]
Substitute the learning rate of α i,t in (32) in the proof of Theorem 3, we have
[Equation] [Equation]
Recall N “
X 12 log 2 p 1 ` T q \ ` 1 , there exists i 0 “
X 12 log 2 p 1 ` P T D q \ ` 1 P r N s such that
[Equation]
Regret bound with dynamic baseline: Let i “ i 0 in (33) and by Lemma 9, we have:
[Equation]
where the second and the last inequalities hold by using (34).
Moreover, we have
[Equation]
where the ﬁrst inequality holds by the convexity of f t p¨q and the second inequality holds by Lemma9.
Recall κ “ 1 ? T and w i, 1 “ N ` 1 i p i ` 1 q N , we have
[Equation]
Combine all these inequalities, we have:
[Equation]
T which proves the regret in Corollary 2 and establish O p?
P T T q regret bound.
Violation bound: Since g ` t p x q is convex, we have:
T
[Equation]
g `
[Equation]
1 q .
By the inequality (31), we have
[Equation]
α t
[Equation]
α
[Equation]
x
[Equation]
t ´ x i
[Equation]
` 1 } 2 .
Let α i,t “ ?
t { 2 i ´ 1 , η i,t “ 2 i ´ 1 ?
t, γ t “ t 12 ` ε , ε ą 0 , we haveg ` t p x i,t ` 1 q ď | f t p x ˚ t q ´ f t p x i
[Equation]
t q| η i
[Equation]
t γ t `
[Equation]
i
[Equation]
t η i
[Equation]
t γ t ` α i
[Equation]
t η i
[Equation]
t γ t p} x ˚ t
´ x i
[Equation]
t } 2 ´ } x ˚ t ´ x i
[Equation]
t ` 1 } 2 q ď
[Equation]
FD t
[Equation]
F
[Equation] [Equation]
} 2 ´ } x
[Equation]
t
[Equation]
x i,t
[Equation]
[Equation]
q
[Equation]
which implies
[Equation]
Thus we have
[Equation]
which proves the violation in Corollary 2.