A Unifying Framework for Online Optimization with Long-Term Constraints
Abstract
We study online learning problems in which a decision maker has to take a sequence of decisions subject to m long-term constraints .
The goal of the decision maker is to maximize their total reward, while at the same time achieving small cumulative constraints violation across the T rounds.
We present the ﬁrst best-of-both-world type algorithm for this general class of problems, with no-regret guarantees both in the case in which rewards and constraints are selected according to an unknown stochastic model, and in the case in which they are selected at each round by an adversary.
Our algorithm is the ﬁrst to provide guarantees in the adversarial setting with respect to the optimal ﬁxed strategy that satisﬁes the long-term constraints.
In particular, it guarantees a ρ/ (1 + ρ ) fraction of the optimal reward and sublinear regret, where ρ is a feasibility parameter related to the existence of strictly feasible solutions.
Our framework employs traditional regret minimizers as black-box components.
Therefore, by instantiating it with an appropriate choice of regret minimizers it can handle the full-feedback as well as the bandit-feedback setting.
Moreover, it allows the decision maker to seamlessly handle scenarios with nonconvex rewards and constraints.
We show how our framework can be applied in the context of budget-management mechanisms for repeated auctions in order to guarantee long-term constraints that are not packing ( e.g. , ROI constraints).
Introduction
We study online learning problems where a decision maker takes decisions over T rounds.
At each round t , the decision x t ∈ X is chosen before observing a reward function f t together with a set of m time-varying constraint functions g t .
The decision maker is allowed to make decisions that are not feasible, provided that the overall sequence of decisions obeys the long-term constraints (cid:80)
Tt =1 g t ( x t ) ≤ 0 , up to a small cumulative violation across the T rounds.
The problem becomes that of ﬁnding a sequence of decisions x t which guarantees a reward close to that of the best ﬁxed decision in hindsight while satisfying long-term constraints.
This type of framework was ﬁrst proposed by Mannor et al.
[36], and it has numerous applications ranging from wireless communication [36] and multi-objective online classiﬁcation [13], to safe online learning [4].
Mannor et al.
[36] show that guaranteeing sublinear regret and sublinear cumulative constraints violation is impossible even when f t and g t are simple linear functions.
Therefore, previous works either focus on the case in which constraints are generated i.i.d.
according to some unknown stochastic
[Table]
model, without providing any guarantees for the adversarial case, or provide results for adversariallygenerated constraints under some strong assumptions on the structure of the problem or using a weaker baseline (a detailed discussion of related works can be found in Appendix A).
A few examples in the latter case are [16, 21, 40, 42].
In the former setting ( i.e. , stochastic constraints), Wei et al.
[41] consider a weaker baseline that is feasible for each constraint g t , going against the basic idea of long-term constraints.
A notable exception is the work by Yu et al.
[44], who employ the same baseline as ours, and provide an upper bound of ˜ O
( T 1 / 2 ) for both regret and constraints violation (see Table 1).
We also mention that there are some works studying the problem in which constraints are static (see, e.g. , [31, 35, 43, 45]), or focus on speciﬁc types of constraints, such as knapsack constraints [7, 30].
Our framework differs from those works as we deal with arbitrary and timevarying constraints.
Moreover, it also extends the online convex optimization framework introduced by Zinkevich [46] by allowing for general non-convex loss functions f t , arbitrary feasibility sets X , and arbitrary time-varying long-term constraints.
1.1 Original contributions
Given the negative result by Mannor et al.
[36], a natural question is what kind of guarantees we can reach in the adversarial setting, when adopting the standard baseline of the best ﬁxed decision in hindsight satisfying (in expectation) the long-term constraints.
We provide the ﬁrst positive result going in this direction, by designing a no-
α -regret algorithm that guarantees a sublinear cumulative constraints violation.
Moreover, we make a step forward in the line of work initiated by Bubeck and Slivkins [14], by showing that our algorithm is also the ﬁrst best-of-both-worlds algorithm for problems with arbitrary long-term constraints.
This allows our algorithm to guarantee good worstcase performance (adversarial case), while being able to exploit well-behaved problem instances (stochastic case).
The only assumption which we require is the existence of a decision that is strictly feasible with respect to the sequence of constraints.
We denote by ρ the “margin” by which this decision is strictly feasible (see Section 2 for a deﬁnition).
At the same time, we show that even without this assumption, we can recover sublinear regret and violation with stochastic constraints.
Previous work usually assumes that ρ is a given constant .
In that case, our algorithm matches the guarantees by Yu et al.
[44] when constraints are generated i.i.d.
according to an unknown distribution, and has no-
α -regret with α = ρ/ (1 + ρ ) in the adversarial case (see Table 1).
Our algorithm only requires a lower bound on the real value of the feasibility parameter ρ .
In the stochastic case, the lower bound may even be unknown, and the algorithm can efﬁciently estimate it from data.
Moreover, we argue that if ρ is allowed to depend on T and take arbitrarily small values, then there are certain values ( ρ ≤ T − 1 / 4 ), for which any regret bound depending on 1 /ρ would be useless ( i.e. , not sublinear in T , see Section 3).
This setting is usually overlooked by previous work, which assumes ρ to be a given constant.
We show that, in the case of an arbitrary feasibility parameter ρ , in the
stochastic setting
our algorithm guarantees an upper bound of ˜ O
( T 3 / 4 ) for regret and cumulative constraints violation.
Our framework employs traditional regret minimizers as black-box components.
Therefore, by instantiating it with an appropriate choice of regret minimizers it can handle full-feedback as well as bandit-feedback settings.
In the former case, after playing x t , the decision maker gets to observe f t and g t , while in the latter case only the realized values f t ( x t ) and g t ( x t ) are observed.
Moreover, this allows the decision maker to seamlessly handle scenarios with non-convex reward and constraints, by employing a suitable regret minimizer for non-convex losses (see, e.g. , [39]).
Our algorithm is based on a two-stage approach in which primal and dual players interact through Lagrangian games .
In the ﬁrst ( play ) phase, the primal player tries to balance out the maximization of their rewards with constraints violation.
In the second ( recovery ) phase, the primal player only makes “safe decisions” to avoid violating constraints too much.
It is possible to prove that, in the case of stochastic rewards and constraints, the algorithm never enters phase two.
This property is particularly relevant for budget-pacing mechanisms in repeated auctions, since it is related to how budget is allocated.
Our framework can also be instantiated to perform budget allocation subject to constraints that were previously not tractable by traditional mechanisms, such as ROI constraints [11, 22].
Preliminaries
The decision maker has a non-empty set of available strategies
X (this set may be non-convex, integral, and even non-compact).
In each round t ∈ [ T ] , 2 the decision maker ﬁrst chooses x t ∈ X , and the environment selects a reward function f t : X → [0 , 1] and a constraint function g t : X → [ − 1 , 1] m conditioned on the past history of play up to time t − 1 ( i.e. , the environment chooses f t and g t without knowledge of x t ).
Notice that both f t and g t need not be convex.
The latter speciﬁes a set of m constraints of the form g t ( x ) ≤ 0 , with g t,i ( x ) ≤ 0 denoting the i -th constraint.
In the following, we denote as F , respectively G , the set of all the possible f t , respectively g t , functions ( e.g. , F and
G may contain all the Lipschitz-continuous functions deﬁned over X ).
At each round t ∈ [ T ] , the decision maker can condition their decision on prior feedbacks and on the sequence of prior decisions x 1 , . .
, x t − 1 , but no information about future rewards and constraint functions is available.
2.1 Strong duality through strategy mixtures
In the following, we deﬁne the optimization problem (Problem LP f,g ) which is used to deﬁne the baselines against which we compare the performances of the decision maker.
Such a problem involves probabilistic mixtures of strategies in X , which are crucial in order to recover strong duality.
First, we introduce the set of probability measures on the Borel sets of X .
We refer to such a set as the set of strategy mixtures , denoted as Ξ .
We endow X with the Lebesgue σ -algebra, and we assume that all the functions in F and G are measurable with respect to every probability measure ξ ∈ Ξ .
This ensures that the various expectations taken are well-deﬁned, since the functions are assumed to be bounded above, and they are therefore integrable.
In the following, for ease of presentation and with a slight abuse of notation, whenever we write a ξ ∈ Ξ in place of an x ∈ X , we mean that we are taking the expectation with respect to the probability measure ξ .
For instance, given f ∈ F and g ∈ G , we have that f ( ξ ) =
E x ∼ ξ f ( x ) and g ( ξ ) =
E x ∼ ξ g ( x ) .
Then, given two functions f ∈ F and g ∈ G , we deﬁne the following optimization problem, which chooses the strategy mixture ξ ∈ Ξ that maximizes the expected reward encoded by f , while guaranteeing that the constraints encoded by g are satisﬁed in expectation.
[Equation]
We denote by d g ∈ [ − 1 , 1] the largest possible value for which there exists a strategy mixture ξ ∈ Ξ satisfying the constraints g ( ξ ) ≤ 0 by a margin of at least d g .
Formally,
[Equation]
In order to ensure that OPT f,g is always well deﬁned, we assume that it is always the case that d g ≥ 0 .
Notice that, if d g > 0 , then Problem LP f,g satisﬁes
Slater’s condition.
In the following, we prove some auxiliary results relating to Problem LP f,g that will be useful in the rest of the paper.
First, we introduce a Lagrangian relaxation of the problem.
x natural numbers.
Focusing on the case g t ( x ) ≤ 0 is w.l.o.g.
since any set of constraints can be represented in such a form.
The optimal ﬁxed strategy mixture provides an arguably stronger baseline than the optimal ﬁxed strategy.
In stochastic settings, this baseline is related to the best dynamic policy.
In particular, if we consider the case in which the observed functions are deﬁned as the average of functions f t and g t across the T rounds, then the optimal mixture provides the same utility as the best dynamic policy (see [7] for a similar result).
Deﬁnition 2.1 (Lagrangian Function) .
Given two arbitrary functions f ∈ F and g ∈ G , the
Lagrangian function L f,g : Ξ × R m ≥
→ R of Problem LP f,g is deﬁned as
[Equation]
If Problem LP f,g satisﬁes
Slater’s condition, then Theorem 1 of Chapter 8.3 in [34] readily gives us that strong duality holds even if f and g are arbitrary non-convex functions.
Formally:
Corollary 2.2.
Given f ∈ F and g ∈ G such that d g > 0 , it holds
[Equation]
Next, we show that, if d g > 0 , then strong duality holds even when we restrict the admissible dual vectors λ ∈ R m ≥ 0 to the set D d g , where, for any q ∈ R > 0 , we let D q :=
(cid:8) λ ∈ R m ≥ 0 : (cid:107) λ (cid:107) 1 ≤ 1 /q (cid:9) (omitted proofs can be found in Appendix B).
Theorem 2.3.
Given f ∈ F and g ∈ G such that d g > 0 , it holds
[Equation]
2.2 Stochastic vs. adversarial: baselines and feasibility
We consider several settings, differing in how functions f t and g t are selected, either stochastically or adversarially .
We say that functions f t (respectively g t ) are selected stochastically, when they are independently drawn according to a given probability measure µ F over F (respectively µ G over G ).
Instead, we say that functions f t (respectively g t ) are selected adversarially if each f t (respectively g t ) is chosen by an adversary based on the sequence of prior decisions, namely x 1 , . . .
, x
t − 1 .
Consistently with previous work (see, e.g. , [36]), we compare the performance of the decision maker (in terms of reward cumulated over the T rounds) against the baseline T OPT ¯ f, ¯ g (as deﬁned by
Problem LP ¯ f, ¯ g ), where ¯ f and ¯ g are suitably-deﬁned functions.
In particular:
• When functions f t , respectively g t , are selected stochastically, then we deﬁne function ¯ f , respectively ¯ g , so that ¯ f ( x ) :=
E f ∼ µ F [ f ( x )] , respectively ¯ g ( x ) :
=
E g ∼ µ G [ g ( x )] .
• When functions f t , respectively g t , are selected adversarially, then we deﬁne function ¯ f , respectively ¯ g , so that ¯ f ( x ) :
= 1 T (cid:80)
Tt =1 f t ( x ) , respectively ¯ g ( x ) :
= 1 T (cid:80)
Tt =1 g t ( x ) .
Intuitively, in the stochastic case, the baseline is instantiated with an expectation of functions taken with respect to the probability measure µ F (respectively µ G ).
Instead, in the adversarial case, the baseline uses the average of functions f t (respectively g t ) observed over the T rounds.
Let us remark that, when the set X is compact convex and functions f t and g t are convex, then
Problem LP ¯ f, ¯ g deﬁning our baselines can be equivalently re-written by using strategies x ∈ X rather than strategy mixtures ξ ∈ Ξ , since there always exists an optimal solution to Problem LP ¯ f, ¯ g
that places all the probability mass on a single strategy.
Our goal is to design online algorithms for the decision maker that output a sequence of decisions x 1 , . . .
, x T such that both the cumulative regret with respect to the performance of the baseline, deﬁned as R T :=
T OPT ¯ f, ¯ g − (cid:80) Tt =1 f t ( x t ) , and the cumulative constraints violation , deﬁned as
V T := max i ∈ [ m ] (cid:80) Tt =1 g t,i ( x t ) , grow sublinearly in the number of rounds T .
In conclusion, we introduce a problem-speciﬁc parameter that is strictly related to the feasibility of
Problem LP ¯ f, ¯ g .
We call it the feasibility parameter ρ ∈ R , which is formally deﬁned as follows:
• When functions g t are selected stochastically, ρ := sup ξ ∈
Ξ min i ∈
[ m ] − ¯ g i ( ξ ) .
• When functions g t are selected adversarially, ρ := sup ξ ∈
Ξ min t ∈ [ T ] min i ∈ [ m ] − g t,i ( ξ ) .
Intuitively, in the stochastic case, ρ is equal to d ¯ g , while in the adversarial case it is computed similarly, but considering the worst case with respect to the functions g t observed at each round t .
Notice that, when ρ > 0 , Slater’s condition is satisﬁed for Problem LP
¯ f, ¯ g .
In the following, we denote by ξ ∗ ∈
Ξ a strategy mixture that is optimal for Problem LP ¯ f, ¯ g .
Moreover, we always assume that functions f t and g t are such that Problem LP ¯ f, ¯ g is feasible, and we let ξ ◦ ∈ Ξ be the feasible strategy mixture that is optimal for the problem deﬁning ρ .
2.3 Regret minimizers
A regret minimizer (RM) for a set W is an abstract model for a decision maker repeatedly interacting with an environment.
At each t , a RM performs two operations: (i) N EXT E LEMENT () , which outputs an element w t ∈ W ; and (ii)
O BSERVE
U TILITY ( · ) , which updates the internal state of the RM using the feedback received from the environment.
This is either a utility function u t : W → [ a, b ] having range [ a, b ] ⊆ R ( full feedback ) or only the value u t ( w t ) ( bandit feedback ), with u t possibly depending adversarially on w 1 , . . .
, w t − 1 .
The objective of the RM is to output a sequence w 1 , . . .
, w T of points in W so that its cumulative regret , deﬁned as sup w ∈W (cid:80) Tt =1 ( u t ( w ) − u t ( w t )) , grows asymptotically sublinearly in T .
See [19] for a review of the various RMs available in the literature.
For the ease of presentation, we introduce the concept of regret minimizer constructor , which is a procedure, say I NIT ( W , [ a, b ] , η ) , that builds a RM on the basis of the three parameters given as input.
In particular, the procedure returns a RM instantiated for the set W , working with utility functions having range [ a, b ] , and such that its cumulative regret is guaranteed to grow sublinearly in the time horizon T with probability at least 1 − η .
A unifying meta-algorithm
In this section, we present our meta-algorithm.
Its core idea is to instantiate suitable pairs of RMs, where one is working in the domain X of primal variables and the other in a suitable subset of the domain R m + of dual variables.
At each round t ∈ [ T ] , the algorithm makes the RMs “play” against each other in a Lagrangian game , where the utility functions observed by them are related to the
Lagrangian function L f t ,g t ( x , λ ) of Problem LP f t ,g t .
Algorithm 1 provides the pseudo-code of the meta-algorithm, which takes as input: the total number of rounds T , a failure probability δ ∈ (0 , 1) such that the guarantees provided by the algorithm hold with probability at least 1 − δ , and a lower bound ˆ ρ ≥ 0 on the value of the feasibility parameter ρ .
Algorithm description.
The algorithm works in two phases.
In the ﬁrst one, called play phase , the algorithm builds a primal RM, called R P I , working in the primal domain X and a dual RM, called
R D I , operating on the subset D ˜ ρ of the dual domain R m
+ , where ˜ ρ is set in Line 1.
The algorithm makes the two RMs playing against each other (see the call L AGRANGIAN G AME ( R P
I , R D
I , 1) ) until either the cumulative violation V t incurred by the algorithm exceeds a given threshold (see Line 4, where M ˜ ρ is deﬁned in Equation (2)) or round T is reached.
Then, in the second phase, called recovery phase , the algorithm constructs a new pair of primal, dual RMs, with the latter working on the ( m − 1) -dimensional simplex ∆ m .
The recovery phase uses the remaining rounds to make these new RMs play against each other, with the primal RM observing modiﬁed utility functions that do not account for functions f t (see the call L AGRANGIAN G AME ( R P II , R D II , 0) ).
Intuitively, this is needed in order to ensure that the algorithm plays strategies x
t that satisfy the constraints, thus balancing out the cumulative constraint violation accumulated in the ﬁrst phase.
The pseudocode describing one “play” between two RMs, called R P and R D , is deﬁned by the sub-procedure
L AGRANGIAN G AME ( R P , R D , v ) in Algorithm 2.
The additional parameter v ∈ { 0 , 1 } is used to control the feedback fed into the primal RM R P ; speciﬁcally, if v = 1 , then R P observes a utility function that also accounts for f t (play phase), otherwise, if v = 0 , the observed utility function only accounts for the term depending on g t (recovery phase).
Regret minimizer constructors.
Algorithm 1 also needs access to two suitably-deﬁned regret minimizer constructors, namely I NIT P ( W , [ a, b ] , η ) and I NIT D
( W , [ a, b ] , η ) , where the former is used
1: ˜ ρ ← max (cid:8) ˆ ρ/ 2 , T − 1 / 4 (cid:9) , η ← δ/ 3 , t ←(cid:46)
Phase
I: Play 2: R P
I ← I NIT P (cid:0) X , (cid:2) − 1 / ˜ ρ, 1 + 1 / ˜ ρ (cid:3) , η (cid:1) 3: R D
I ← I NIT D (cid:0) D ˜ ρ , (cid:2) − 1 / ˜ ρ, 1 / ˜ ρ (cid:3) , 0 (cid:1) 4: while V t ≤ ( T − t )
˜ ρ + M ˜ ρ − 1 ∧ t ≤ T do 5: x t ← L AGRANGIAN G AME ( R P
I , R D
I , 1) 6: t ← t +7: T 1 ← t −(cid:46)
Phase II: Recovery 8: R P II ← I NIT P
( X , [ − 1 , 1] , η ) 9: R D II ← I NIT D (∆ m , [ − 1 , 1] , 0) 10: while t ≤ T do 11: x t ← L AGRANGIAN G AME ( R P II , R D II , 0) 12: t ← t +
AME ( R P , R D , v ) 1: x t ← R P .
N EXT E LEMENT () 2: λ t ← R D .
N EXT E LEMENT () 3: Play x t and get f t and g t (cid:46)
Full f.
Play x t and get f t ( x t ) and g t ( x t ) (cid:46)
Bandit f.
(cid:46) Primal RM update 4: Let u P t : x (cid:55)→ vf t ( x ) −(cid:104) λ t , g t ( x ) (cid:105) (cid:46)
Full f. u P t ( x t ) ← vf t ( x t ) − (cid:104) λ
t , g t ( x t ) (cid:105) (cid:46)
Bandit f.
5: R P .
O BSERVE
U TILITY ( u P t ) (cid:46)
Full f.
R P .
O BSERVE
U TILITY ( u P t ( x t ))
(cid:46) Bandit f.
(cid:46)
Dual RM update 6: Let u D t : λ (cid:55)→ (cid:104) λ , g t ( x ) (cid:105) 7: R D .
O BSERVE
U TILITY ( u D t ) to build RMs working in the primal domain and the latter for those operating on the dual one.
Their actual implementation depends on the speciﬁc problem at hand.
In the following, we let E P t,η be the regret upper bound (on t ∈ [ T ] rounds) for primal RMs R P dealing with utility functions having range [0 , 1] , as returned by the call I NIT P
( X , [0 , 1] , η ) .
Notice that, when the range is [ a, b ] , the same
RM can be adopted by ﬁrst normalizing utility values, so that the resulting regret upper bound is ( b − a ) E P t,η .
As for dual RMs R D , we let E D t be the regret upper bound (on t ∈ [ T ] rounds) provided by the RM deﬁned for the set ∆ m , while E D t / ˜ ρ is the upper bound for the dual RM instantiated on the set D ˜ ρ .
Notice that, since dual RMs always have full feedback, we can safely assume that the regret bounds E D t hold deterministically.
We also assume that RMs provide bounds that increase with the number of rounds, i.e. , such that E P t,η ≤ E P t (cid:48) ,η and E D t ≤ E D t (cid:48) for all t ≤ t (cid:48) .
How to construct RMs.
I NIT D can be implemented by using online mirror descent (OMD) with domain ∆ m (or D 1 ) and a negative entropy regularizer.
Since the utility function u D t is linear in λ , we get a regret bound for the primal RM of E D T
= O ( (cid:112) T log( m ))
(see, e.g. , [12, 37]).
The design of
I NIT P depends on the structure of X and functions f t and g t .
For instance, in convex settings with full feedback we can employ OMD [29], while with bandit feedback we can use [15].
Finally, for non-convex functions we can employ, e.g. , the RMs in [39].
All these RMs guarantee ˜ O ( √ T ) regret.
How to get away with no knowledge of ρ .
In Section 7, we show that a lower bound ˆ ρ is not necessary when functions g t are selected stochastically.
Indeed, it is sufﬁcient to add a preliminary phase to Algorithm 1, which is used to infer a suitable lower bound on ρ from experience.
In order to do this, only √ T rounds are needed, so that the bounds on the cumulative regret and the cumulative constraint violation achieved by the algorithm are not compromised.
When functions g t are chosen adversarially, as we show in Section 7, it is impossible to compute a lower bound on the feasibility parameter ρ by only adding a preliminary phase to Algorithm 1 which uses √ T rounds.
Remark 3.1 (Dependence on the lower bound ˆ ρ ) .
Algorithm 1 can take as input any ˆ ρ ≥ 0 .
However, since our regret bounds include a factor 1 / ˜ ρ , by choosing the trivial lower bound ˆ ρ = 0 we incur in a regret of ˜ O ( √ T/ ˜ ρ ) =
˜ O ( T 3 / 4 ) .
In order to obtain optimal bounds, we would like to have ˜ ρ
= Ω( ρ ) .
Remark 3.2 (Dependence on the feasibility parameter ρ ) .
In our analysis, we include the dependence on the feasibility parameter ρ in the regret and constraint violation bounds achieved by the algorithm.
As customary, the goal is devising bounds of the form poly(instance) · h ( T ) , where the ﬁrst term is a polynomial function of the parameters deﬁning the problem instance, and h ( T ) =
o ( T ) .
Therefore, we cannot include a factor 1 /ρ in the regret bounds if ρ can be arbitrarily small.
Even from a practical standpoint, when ρ is too small, a 1 /ρ regret bound is too large to be signiﬁcant.
For those reasons, we set ˜ ρ in Algorithm 1 to be the maximum between the feasibility parameter lower bound ˆ ρ and T − 1 / 4 .
Notice that the value T − 1 / 4 has been carefully chosen so as to minimize the maximum between the cumulative regret and the cumulative constraint violation when the lower bound on the feasibility parameter ˆ ρ is too small.
Analysis with stochastic constraints and adversarial rewards
We start by analyzing the performance of our meta-algorithm (Algorithm 1) when the reward and constraint functions are selected stochastically and adversarially, respectively.
Given t ∈ [ T ] and η ∈ (0 , 1) , we let E t,η := (cid:112) 8 t log(18 mt 2 /η ) be the value bounding differences between expectations and empirical means of constraint functions, obtained by applying the AzumaHoeffding inequality, and holding with probability at least 1 − η .
Given γ ∈ (0 , 1) , we also let
[Equation]
which is a recurring term related to the maximum violation that Algorithm 1 accepts in play phase.
First, we introduce a useful event E that encompasses all the cases in which Algorithm 1 successfully terminates.
Then, Lemma 4.2 shows that such an event holds with probability at least 1 − δ .
In particular, E holds when the regret bounds of R P I and R P II hold, and, additionally, the differences between expectations and empirical means of constraint functions are bounded as desired.
Deﬁnition 4.1.
We denote with E the event in which Algorithm 1 satisﬁes the following conditions (recall that η = δ/ 3 ): (i) the regret incurred by R P
I after T 1 rounds is upper bounded by E P T 1 ,η ; (ii) the regret cumulated by R P II after the remaining T − T 1 rounds is upper bounded by E P T − T 1 ,η ; and (iii) for every pair of rounds t, t (cid:48) ∈
[ T ] : t ≤ t (cid:48) and resource i ∈ [ m ] it holds:
• (cid:12)(cid:12)(cid:12)(cid:80) t (cid:48) τ
= t g τ,i ( x τ ) − (cid:80) t (cid:48) τ
= t ¯ g i ( x τ ) (cid:12)(cid:12)(cid:12) ≤ E t (cid:48) − t,η , • (cid:12)(cid:12)(cid:12)(cid:80) t (cid:48) τ
= t
λ τ g τ,i ( x τ ) − (cid:80) t (cid:48) τ
= t
λ τ ¯ g i ( x τ ) (cid:12)(cid:12)(cid:12) ≤ E t (cid:48) − t,η max τ ∈ [ T ]: t ≤ τ ≤ t (cid:48) ||
λ
τ || 1 , • (cid:12)(cid:12)(cid:12)(cid:80) t (cid:48) τ
= t g τ,i ( ξ ) − (cid:80) t (cid:48) τ
= t ¯ g i ( ξ ) (cid:12)(cid:12)(cid:12) ≤ E t (cid:48) − t,η for ξ ∈ { ξ ∗ , ξ ◦ } , • (cid:12)(cid:12)(cid:12)(cid:80) t (cid:48) τ
= t
λ τ g τ,i ( ξ ) − (cid:80) t (cid:48) τ
= t
λ τ ¯ g i ( ξ ) (cid:12)(cid:12)(cid:12) ≤ E t (cid:48) − t,η
[Table]
∈
[Table]
≤ τ ≤
[Table]
τ
[Table]
∗
[Table]
ξ ◦
[Table]
Lemma 4.2.
After running Algorithm 1, the event E holds with probability at least 1 − δ .
Next, we lower bound the cumulative reward obtained by Algorithm 1 during the play phase.
Intuitively, we show that, if the cumulative constraints violation is large, then the decisions x t in the ﬁrst T 1 rounds provide a per-round reward much higher than that achievable by ξ ∗ .
This allows us to employ the following recovery phase to decrease constraints violation cumulated in the play phase, while also ensuring that the cumulative regret stays low at the end of the algorithm.
Formally:
Lemma 4.3.
If event E holds, then after round T 1 of Algorithm 1 the following inequality holds
[Equation]
In the recovery phase, the only goal of Algorithm 1 is to decrease constraints violation.
In the following Lemma 4.4, we show that, at each round of the recovery phase, the algorithm is “close” to satisfying (in expectation) all the constraints by at least ρ .
Formally:
Lemma 4.4.
If event E holds, then after Algorithm 1 halts, the following holds for every i ∈ [ m ] :
[Equation]
Now, we are ready to present the two main results of this section.
First, we provide a bound on the cumulative regret and constraints violation when the lower bound ˆ ρ is sufﬁciently large.
Condition 4.5.
It holds that ˆ ρ ≥ 2 T − 1 / 4 .
Notice that, under Condition 4.5, ˜ ρ
= ˆ ρ/ 2 .
This gives us the following result:
Theorem 4.6.
Suppose that functions f t and g t are selected adversarially and stochastically, respectively.
If Condition 4.5 is satisﬁed, then, with probability at least 1 − δ , Algorithm 1 provides
R T ≤ 1˜
[Equation]
E T,
[Equation]
+ (cid:16)
[Equation]
+ 2˜
[Equation]
(cid:17) E P T,
[Equation]
+ 1˜ ρ E D T and V T ≤ M ˜ ρ + 2 E P T,η + E D T + E T,η .
Finally, we also prove that even if Condition 4.5 is not satisﬁed, i.e. , the lower bound ˆ ρ is not sufﬁciently large, the following holds:
Theorem 4.7.
Suppose that functions f t and g t are selected adversarially and stochastically, respectively.
Algorithm 1 guarantees that the following bounds hold with probability at least 1 − δ :
R T ≤ T 1 / 4 E T,η + (cid:0)
+ 2 T 1 / 4 (cid:1) E P T,η + T 1 / 4 E D T and V T ≤ T 3 / 4 + M T − 1 / 4 + 2 E P T,η + E D T + E T,η .
Remark 4.8.
Notice that, by using primal and dual RMs whose regret bounds are of the order of ˜
O ( √ T ) , Theorem 4.6 allows us to recover ˜ O ( √ T/ ˆ ρ ) regret and ˜ O ( √ T/ ˆ ρ ) constraints violation for the case in which Condition 4.5 holds.
Theorem 4.7 still provides ˜ O
( T 3 / 4 ) regret and constraints violation when the condition is not met, which is necessary the case when ρ = 0 .
Analysis with stochastic constraints and stochastic rewards
In this section, we focus on the case in which both reward and constraint functions are selected stochastically.
In this setting, we are able to show that Algorithm 1 never enters the recovery phase.
As we argue in Section 8, this is an important property for budget-management applications, since it is related to the round in which the budget is fully depleted.
In order to prove our result, we extend the event E to capture also the Azuma-Hoeffding bounds for the reward functions, which are stochastic in this setting.
7 The core idea that we exploit to prove our result is that we can think of the two RMs as if they are playing a stochastic repeated zero-sum game, which is the repeated Lagrangian game whose functions are sampled according to the probability measures µ F and µ G .
By Theorem 2.3, strong duality holds, and the game has an equilibrium.
Hence, it is possible to show that the per-round utility of the primal RM is close to the value of the game, which is OPT ¯ f, ¯ g .
At the same time, it is possible to show that, if the cumulative constraints violation becomes large during the play phase (and, thus, T 1 < T ), then the per-round utility of the primal RM is below OPT ¯ f, ¯ g , reaching a contradiction that proves the following theorem.
Theorem 5.1.
Suppose that functions f t and g t are selected stochastically.
With probability at least 1 − δ , Algorithm 1 never enters the recovery phase, namely T 1 = T .
Notice that regret bounds analogous to the ones in Theorems 4.6 and 4.7 also hold in the case in which both reward and constraint functions are selected stochastically.
Analysis with adversarial constraints
In this section, we study settings in which the constraint functions g t are selected adversarially.
As shown by Mannor et al.
[36], it is impossible to obtain sublinear cumulative regret and constraints violation when using our baseline, i.e. , the best ﬁxed strategy mixture ξ ∗ satisfying (in expectation) the long-tern constraints.
However, we show that it is possible to achieve a ρ/ (1 + ρ ) fraction of the cumulative reward obtained by always playing ξ ∗ , while guaranteeing sublinear constraints violation.
The dependence of the approximation factor on the feasibility parameter ρ is similar to the dependence on the per-round budget in problems with budget constraints (see the related works in
Appendix A for more details).
Moreover, as we discuss later in Section 8, when restricted to the case of budget constraints and adversarial reward/cost functions, our approximation factor matches the state-of-the-art bounds provided by Castiglioni et al.
As a ﬁrst step to prove our result, we provide a lower bound on the cumulative reward of the primal
RM during the play phase.
We show that it achieves at least a ρ/ (1 + ρ ) fraction of the value obtained by the optimal solution in the ﬁrst T 1 rounds.
Moreover, the algorithm provides an additional utility compensating for the last rounds in which the algorithm only focuses in satisfying the constraints.
Finally, we show that, in the recovery phase, the constraints are satisﬁed by at least ρ at each round, up to a term related to the regret of R P II and R D II , proving the following theorem.
Theorem 6.1.
Suppose that functions f t and g t are selected adversarially.
If Condition 4.5 is satisﬁed, then, with probability at least 1 − 23 δ , Algorithm 1 guarantees that the following holds:
[Equation]
A similar result can be also derived for the case of stochastic rewards and adversarial constraints.
Corollary 6.2.
Suppose functions f t and g t are selected stochastically and adversarially, respectively.
If Condition 4.5 is satisﬁed, then, with probability at least 1 − δ , Algorithm 1 provides (cid:80) Tt =1 f t ( x t ) ≥
[Equation]
Remark 6.3.
By using primal and dual RMs whose regret bounds are of the order of ˜ O ( √ T ) ,
Theorem 6.1 and Corollary 6.2 allows us to recover (cid:80)
Tt =1 f t ( x t ) ≥ ρ 1+ ρ (cid:80)
Tt =
OPT ¯ f, ¯ g − ˜ O ( √ T/ ˆ ρ ) , and ˜ O ( √ T/ ˆ ρ ) constraints violation for the case in which Condition 4.5 holds.
How to get away with no knowledge about the feasibility parameter
We show how to extend Algorithm 1 in order to deal with settings in which a lower bound on the feasibility parameter ρ is not known, when functions g t are selected stochastically.
We propose
Algorithm 3, which runs Algorithm 1 by ﬁrst devoting a given number T 0 < T of rounds to inferring a suitable lower bound ˆ ρ on the feasibility parameter ρ .
Ideally, we would like to have ˆ ρ = Ω( ρ ) , so as to recover bounds of the order ˜ O ( √ T/ρ ) .
In particular, we show that we can run Algorithm 3 with
T
= T 1 / 2 to obtain an approximation of ρ that has an additive approximation error of the order T 1 / 4 .
This is sufﬁcient to get ˆ ρ = Ω( ρ ) , since a good approximation of ρ is only needed when ρ ≥ T 1 / 4 .
1: R P ← I NIT P (cid:0) X , (cid:2) − 1 , 1 (cid:3) , δ (cid:1) 2: R D ← I NIT D (∆ m , [ − 1 , 1] , 0) 3: t ←4: while t ≤ T 0 : do 5: x t ← L AGRANGIAN G AME ( R P , R D , 0) 6: t ← t +7:
ˆ ρ ← − 1 T 0 (cid:16) max i ∈ [ m ] (cid:80) T 0 t =1 g t,i ( x t ) + E T 0 ,δ (cid:17) 8: Run Algorithm 1 with T − T 0 , δ , and ˆ ρ as inputs
Let us remark that our approach only works when constraints functions g t are selected stochastically.
When these are chosen adversarially, it is easy to see that it is impossible to compute a lower bound on the feasibility parameter ρ by only using the ﬁrst rounds.
For instance, think of a setting in which ρ is very large in the ﬁrst rounds, while it becomes small during the last ones.
To exploit the guarantees of Algorithm 1 presented in the previous sections, it is enough to show that, after the ﬁrst T 0 rounds of Algorithm 3, ˆ ρ ≤ ρ holds with high probability.
Lemma 7.1.
If T 0 = √ T , then Algorithm 3 guarantees that ˆ ρ ≤ ρ with probability at least 1 − δ .
In order to recover a good estimate of ρ , we need the value of ρ to be sufﬁciently large.
Condition 7.2.
It holds that ρ ≥ 2 T 0 (2 E T 0 ,δ + 2 E P T 0 ,δ + E D T 0 ) .
Remark 7.3.
Notice that, by using primal and dual RMs whose regret bounds are of the order ˜
O ( √ T ) and by setting T 0 = √ T , Condition 7.2 is satisﬁed when ρ = ω ( T − 1 / 4 ) .
Next, we show that ˆ ρ = Ω( ρ ) , which allows us to exploit the guarantees proved for Algorithm 1 in order to provide analogous ones for Algorithm 3.
Formally:
Lemma 7.4.
By setting T 0 = √ T , and assuming that Condition 7.2 is satisﬁed, after T 0 rounds of
Algorithm 3 we have that ˆ ρ ≥ ρ/ 2 with probability at least 1 − 2 δ .
By applying the results of the previous sections on the guarantees of Algorithm 1, and by using primal and dual RMs whose regret bounds are of the order ˜ O ( √ T ) , we get ˜ O ( √ T/ρ ) and ˜ O ( √ T/ρ ) regret and violation bounds, respectively, when the functions g t are selected stochastically.
Applications to repeated auctions settings
Internet advertising platforms usually operationalize large auction markets by using proxy bidders that place bids in repeated auctions on the advertisers’ behalf.
A proxy-bidder selects bids according to a budget-pacing mechanism , which manages the usage of the advertisers’ budget over time [1, 22, 9].
In this section, we discuss the application of our framework to budget-management in auctions, arguing that it can deal with more general constraints on ad slots allocation with respect to what is currently achievable with multiplicative pacing algorithms, which manage only knapsack constraints .
We consider the problem faced by a bidder who takes part in a sequence of repeated auctions.
We focus on the case of second-price and ﬁrst-price auctions, since they are the de facto standard in large
Internet advertising platforms.
At each round t ∈ [ T ] , the bidder observes their valuation v t from a ﬁnite set of n v possible valuations V ⊂ [0 , 1] .
Such valuation models targeting preferences of the advertiser.
Then, the bidder chooses a bid b t ∈ B , where B ⊂ [0 , 1] is a ﬁnite set of n b possible bids such that 0 ∈ B ( i.e. , the bidder is allowed to skip items without incurring in any cost).
The utility of the bidder depends on the largest among competing bids, denoted by β t .
In particular, the utility is computed as f t ( b t ) =
( v t − c t ( b t )) 1 { b t ≥ β t } , where the cost c t is such that c t ( b t ) =
β 1 { b t ≥ β t } in second-price auctions, and c t ( b t ) =
b t 1 { b t ≥ β t } for ﬁrst-price ones.
Finally, the bidder has a target per-round budget of ρ > 0 , which yields an overall budget B := ρT that limits the total spending over the T rounds.
In the case of budget-constrained bidding, a strictly feasible solution can be easily achieved by always bidding 0 .
Using the target per-round budget ρ = B/T we can write the budget constraint as (cid:80) t ∈ [ T ] g t ( b t ) ≤ 0 , with g t ( b ) =
c t ( b ) − ρ for any b ∈ B .
Notice that, in this setting, we have the same feasibility parameter ρ for both the stochastic and the adversarial case.
As a benchmark to evaluate the algorithm, we consider the best feasible static policy π : V → B .
The set of static policies can be represented by X :=
B n v , where a vector b ∈ B n v encodes the policy’s bids for each possible valuation.
To apply our framework to this problem, it is sufﬁcient to design a primal regret minimizer constructor (recall that, in order to design dual RMs, we can employ OMD).
This can be implemented by instantiating a regret minimizer E XP 3.P [5] for each possible valuation in
V .
Given a failure probability ν ∈ (0 , 1) , each RM guarantees a regret bound O ( (cid:112) Tn b log( n b /ν )) with probability at least 1 − ν .
Thus, given a desired failure probability η ∈ (0 , 1) , by setting ν = η/n v we get that, with probability at least 1 − η , the bounds of all the RMs hold.
Hence, by a union bound, we get that the regret of a primal RM is E P T,η
= O ( n v (cid:112)
Tn b log( n b n v /η )) .
Guaranteed budget completion in the stochastic case.
The crux of budget-pacing mechanisms is ensuring that the advertisers’ budget is not depleted too early (thereby missing potentially valuable future advertising opportunities), while being fully depleted within the planned duration of the campaign.
Theorem 5.1 shows that, when inputs are generated according to some stochastic model, Algorithm 1 never enters the recovery phase.
This is crucial in the context of budget-pacing mechanisms, because whenever the algorithm enters the recovery phase it will converge to always bid 0 in order to mitigate constraints violation.
Therefore, the bidder could miss out on potentially valuable items.
Moreover, if the platform wanted to guarantee that the bidder does not spend more than the budget B , it would be enough to set a virtual budget of B − ˜ O
( T 1 / 2 ) to compensate for the potential constraints violation.
Finally, we argue that, in large-scale markets, an individual bidder has almost no impact on prices, and, thus, stochastic behavior of costs is a reasonable assumption.
Adversarial case.
Theorem 6.1 of [17] shows how to construct an algorithm that provides a ρ fraction of the optimal utility for problems with budget constraints and adversarial inputs.
The ratio ρ/ (1 + ρ ) obtained in Theorem 6.1 matches such result.
The latter assumes that rewards and costs are in [0 , 1] , and, thus, g t ∈ [ − ρ, 1 − ρ ] (as they only model budget constraints).
However, in our case we have g t ∈ [ − 1 , 1] .
By normalizing the former range to match with ours, we get g t ∈ [ − ρ/ (1 − ρ ) , 1] .
Therefore, the feasibility parameter would be ρ (cid:48)
=
ρ/ (1 − ρ ) .
By rewriting our guarantees as a function of ρ , we get ρ (cid:48) /
(1 + ρ (cid:48) )
= ρ , which is the same guarantee of [17].
Handling ROI constraints.
Traditional budget-pacing mechanisms (see, e.g. , [11, 8]) are based on primal-dual algorithms that are near optimal in settings with knapsack constraints only, and they cannot be generalized to deal with other types of long-term constraints.
However, there are many realworld situations in which guaranteeing other types of constraints is crucial for practical applications (see, e.g. , [26, 25]).
One example is the case of return on investment (ROI) constraints [6, 26, 32].The recent work by Golrezaei et al.
[25] presents a threshold-based algorithm for repeated second-price auctions under budget and ROI constraints.
Our framework allows advertisers to reach a target ROI while keeping expenses under control also in the setting of repeated ﬁrst-price auctions.
In particular, given a target ROI ω , we deﬁne the ROI constraints as g t ( b t ) =
( ω − v t /b t ) 1 { b t ≥ β t } ≤ 0 .
Then, it is enough to instantiate our framework as described before to immediately get that the cumulative violation of the budget and ROI constraints are upper bounded by ˜ O
( T 1 / 2 ) .
This holds both in the fully stochastic and in the fully adversarial setting.
See Appendix C for a detailed discussion on the types of constraints that our framework can handle. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We present the goal of our paper and summarize the theoretical results.
In the Original Contribution section, we underline the novelty of our work and in the Related Work section we compare our results with those in the literature (see also Table 1).
(b) Did you describe the limitations of your work?
[Yes] The assumptions characterizing the settings in which our results hold are clearly stated throughout the paper.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The results presented in this work does not have any foreseeable negative societal consequence.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] In each theorem statement we provide all the assumptions needed for the result to hold.
(b) Did you include complete proofs of all theoretical results?
[Yes] They are provided in Appendix B and Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]