Combinatorial Bandits with Linear Constraints: Beyond Knapsacks and Fairness
Abstract
This paper proposes and studies for the first time the problem of combinatorial multi-armed bandits with linear long-term constraints.
Our model generalizes and unifies several prominent lines of work, including bandits with fairness constraints, bandits with knapsacks (BwK), etc.
We propose an upper-confidence bound LPstyle algorithm for this problem, called UCB-LP, and prove that it achieves a logarithmic problem-dependent regret bound and zero constraint violations in expectation.
In the special case of fairness constraints, we further provide a sharper constant regret bound for UCB-LP.
Our regret bounds outperform the existing literature on BwK and bandits with fairness constraints simultaneously.
We also develop another low-complexity version of UCB-LP and show that it yields ˜ O ( √ T ) problem-independent regret and zero constraint violations with high-probability.
Finally, we conduct numerical experiments to validate our theoretical results.
Introduction
In this paper we study the problem of combinatorial bandits with long-term linear constraints.
Our model captures important application scenarios like ad placement in online advertising systems [41], real-time traffic scheduling in wireless networks and task assignment in crowdsourcing platforms [29] etc.
Although being studied for the first time, our model subsumes several well-known problems in the Constrained Multi-Armed Bandit (CMAB) literature, including bandits with knapsacks, bandits with fairness constraints etc.
Details about these problems and how they fit into our framework are provided in Section 1.
Specifically, we consider an agent’s online decision problem faced with a fixed finite set of N arms labelled 1 , 2 , ..., N , within the time horizon T .
At each round t ( 1 ≤ t ≤ T ), every arm i ∈
[ N ] is associated with a random reward f i ( t ) ∈ [0 , 1] sampled from a time-invariant distribution P i .
The reward f i ( t ) and its distribution P i are unknown to the agent a priori .
The mean reward of distribution
P i is denoted as µ i ∈ [0 , 1] .
We denote µ = ( µ 1 , ..., µ N ) ⊤ ∈ [0 , 1] N the mean reward vector, and define µ ∗ := max i ∈
[ N ] µ i as the maximum mean reward value of all arms.
At the beginning of each round t , the agent is allowed to pull multiple, but no more than m arms.
At round t , the action taken by the agent is represented by an action vector a ( t ) =
( a 1 ( t ) , ..., a N ( t ))
⊤ ∈ { 0 , 1 } N , where a i ( t )
=
if and only if arm i is pulled.
The set of all feasible action vectors is defined as
A = { a | a ∈ { 0 , 1 } N , || a || 1 ≤ m } .
After taking action a ( t ) , the agent can observe reward from each pulled arm.
Summing up the reward from the pulled arms, at round t , the agent receives a total reward of R t := (cid:80) Ni =
1 f i ( t ) a i ( t ) .
Beyond the standard combinatorial bandit setting above, we consider that the agent is subject to some constraints g ( · ) at every round t , defined as g ( a ( t )) :
= [ g 1 ( a ( t )) , g 2 ( a ( t )) , ..., g K ( a ( t ))]
⊤ , where g 1 , g 2 , . .
, g K : R N → R are linear functions.
The goal of the agent is to maximize the accumulated expected reward up to the time horizon T , while satisfying the constraints in the long term, i.e.,
[Equation]
(The comparison operator ≤ is coordinate-wise)
Define OPT(T) as the expected accumulated reward in T rounds of the optimal policy satisfying the long term constraints.
The agent’s performance is measured in terms of regret and constraint violations defined respectively as
[Equation]
where the expectation is taken w.r.t.
the randomness of the reward and algorithm’s internal randomness.
Consider the following linear programming problem (LP):
[Equation]
[4] showed that OPT ( T ) ≤ T · OPT LP .
We denote the set of optimal solutions to LP (2) as X ∗ , and define x ∗ ∈
X ∗ as one of these optimal solutions.
Following a common approach, we use the optimal randomized policy x ∗ as the benchmark in the regret:
[Equation]
1.1 Representative problems and related works
In this section, we outline several motivating and representative problems which fit into our general formulation and review the literature closely related to them (listed in Table 1).
Problem-dependent parameters ∆ min and ∆ will be formally defined in Section 2.
[Table]
[Table] [Table]
(1) ˜ O
( N ) [12]
SINGLE - ARM , FAIRNESS
N/A
O ( N log T/ ∆)
N/A ˜ O ( N ) (T HIS WORK )
UCB-LP
COMBINATORIAL , LINEAR
N/A
O ( mN log T/ ∆ min )
LP (T HIS WORK ) UCB-LP
COMBINATORIAL , FAIRNESS
N/A
O (1) ˜ O
( N ) (T HIS WORK )
UCB-PLLP
COMBINATORIAL , LINEAR
N/A
O ( m √ T log T )
( HIGH - PROB )
˜ O ( N )
Bandits with knapsacks.
The BwK (Bandits with Knapsacks) problem with deterministic costs studied in [18, 42, 40] assumes that there are K resources consumed over time, each with budgets
B 1 , B 2 , . . .
, B K respectively.
Every resource i ∈
[ K ] is associated with a fixed consumption vector λ
i =
( λ i, 1 , λ i, 2 , . . .
, λ i,N ) ∈
R N ≥ 0 .
The agent maximizes the accumulated reward subject to the budget constraints (cid:80) i ∈
[ T ] λ ⊤ i a ( t ) ≤ B i , ∀ i ∈
[ K ] .
One can see that, this problem is equivalent with the special case of our setting where the linear constraint functions have the form g ( a ( t )) =
( λ ⊤ 1 a ( t ) − B 1 /T, . . .
, λ ⊤ K a ( t ) − B K /T ) .
The existing BwK literature [1, 4, 17] has derived algorithms that achieve the problem-independent regret bounds of the similar order ˜ O ( √ KNT ) .
Such order of the regret bound is also obtained by [41, 42, 28] for combinatorial setting.
However, the problem-dependent regret of BwK with deterministic costs is less explored.
[18] achieved one regret bound of O (cid:16) min { N, K } (cid:0) N + KK (cid:1) log T ∆ min (cid:17) with unsatisfying exponential dependence on K and N .
They also provide another regret bound of
O ((min { N, K } ) 3 log T ∆ 2min ) with polynomial dependence on N, K but require an additional assumption of the optimal solution of LP (2) being unique.
Later [40] improves this regret to O ( N log T ∆ min ) also under the assumption that LP (2) has an unique optimal solution.
But their algorithm requires the knowledge of some parameters of the problem instance a priori (characterized by ∆ min and µ ∗ ), which is unpractical.
With the “best-arm-optimality” assumption, i.e., there is an optimal policy that only pulls one arm, [42] achieved the regret of O ( N · log T/G 2 LAG ) ( G LAG is their defined Lagrangian gap) when there is only one resource ( K = 2) , whose practicability is also restricted.
Very recently, [15] derives a regret bound of O (log 2 T ) for combinatorial setting and one resource constraint, which is the best so far in the area of combinatorial BwK but still sub-optimal in terms of T .
For BwK problems with stochastic costs, [18, 47, 46, 42, 30, 7] obtained logarithmic regrets under different restrictive assumptions, e.g., non-degeneracy of (2), K = 2 (single source) or “best-armoptimality”.
[18, 42] showed that it is impossible to achieve any problem-dependent regret bound of o ( √ T ) without additional assumptions in general.
Bandits with fairness constraints.
Recently, [29, 48, 39] studied the problem of bandits with fairness constraints.
Under their setting, the agent maximizes the cumulative expected reward, and needs to ensure that each arm i ∈ [ N ] is pulled for at least r i ∈ (0 , 1) fraction of times at the end of
T rounds.
In our model, such fairness constraints are equivalent with the following special kind of linear long-term constraints: g ( a ( t ))
= − a ( t ) + r , where r =
( r 1 , r 2 , . . .
, r N ) ⊤ .
[29] first studied the combinatorial (sleeping) bandits with fairness constraints.
Their algorithm
LFG combines virtual queue technique and UCB learning.
LFG yields ˜ O
( √ T ) regret and sublinear ( o ( T )) constraint violations.
[20] replaced the UCB learning with Thompson Sampling in LFG and obtained performance guarantees with the same order.
Later [48] improved the constraint violations bound to O ( √ T ) with the same regret order by using online convex optimization techniques and
RRS rounding.
A big advancement is made by recent works [12, 39] that they achieve O (log T ) and o (log T ) regret bounds, respectively, for MAB ( m = 1 ) with fairness constraints based on the modified
UCB1 algorithm.
And [16] achieved a "penalized" regret bound of O (log T ) in the single-arm setting.
Bandits with group fairness.
In scenarios like ad-display optimization, the agent is subject to the group fairness constraint, e.g., arms belonging to one group should be pulled more frequently than arms belonging to another group [35], or the arm with higher average reward should be pulled more times than the arm with lower average reward [23], etc.
This problem also fits into our formulation of linear constraints.
Only problem-dependent regrets of ˜ O ( √ T ) are obtained in related works.
Other related literature.
A large body of literature (e.g., [10, 45]) derived O (log T ) regret bounds for unconstrained combinatorial bandits.
[22, 24] studied the BwK problem under the adversarial setting.
[2, 38, 3, 34] studied constrained linear bandits.
Our framework is also related to online convex optimization with long-term constraints (e.g., [33, 11, 51, 50]), where the agent faces several convex constraints and these constraints need to be satisfied in the long term.
We remark that, in this setting, the full reward function at each round would be revealed after the decision making, which is in contrast to our setting that we do not have such observation due to the semi-bandit feedback.
1.2 Discussion: significance of linear constraints
In this section, we discuss the significance of generalization to linear constraints.
Previous literature on constrained bandits only studies constraints with specific forms, namely, the fairness constraint and the knapsacks constraint, etc.
As we only require g ( · ) to be linear, our model not only subsumes both of them as its special cases, but also solves a larger group of constraints, enabling broader applications.
For example: 1.
Our model for the first time addresses scenarios where both fairness and knapsack constraints exists simultaneously.
Our model solves the case of weighted fairness constraints, i.e., weighted sums of pulls of each arm are required to be larger than a threshold: (cid:80) N i
=1 κ ij h
i ( T ) ≥ r j T, where κ ij is the j th weight of the i th arm.
In contrast, traditional fairness constraints treat each arm as unweighted and could be seen as a special case of weight fairness constraints.
In fact, there are many applications where combinatorial bandits with various complicated linear constraints are required.
We show some concrete examples as follows.
(a)
In crowdsourcing, where a group of workers are assigned with tasks to achieve high accuracy, a set of complex linear constraints occurs: each tasks should have enough workers, while each worker should have a fair workload.
(b)
In network routing where multiple paths (each path consists of a series of links) needs to be selected to send the traffic within the time budget and the bandwidth constraints.
(c) Case of Internet of Things where a set of sensors need to be selected at each round to guarantee the QoS requirement [21] (e.g., throughout, mean-delay) under budget constraint on data collection cost (e.g., energy consumption).
1.3 Our contributions
Our main contributions are summarized below.
(a) We define a general formulation termed combinatorial bandits with linear long-term constraints.
In contrast to previously outlined pieces of work, we consider the problem in its full generality and do not assume or require any prior knowledge of the problem instance.
We design an upper-confidence bound LP-style algorithm named UCB-LP, and develop a novel analytical technique to build a relationship between LP solution (distribution support over arms) and "reward allocation", with which we show that UCB-LP guarantees a problem-dependent regret of O ( mN log T ∆ min ) and no constraint violation in expectation.
To the best of our knowledge, this is the first logarithmic regret bound for bandits with long term linear constraints under the combinatorial setting.
(b) For the special case of fairness constraints,
we show that
UCB-LP
achieves
O (1) regret
and guarantees zero expected constraint violation at the
same
time.
To the best of our knowledge, both the regret and the constraint violations outperform all existing works on combinatorial bandits with fairness constraints.
We also show that UCB-LP has a low running time of ˜ O
( N ) in this special case
(c) To overcome the
potentially
time-consuming
LP
in UCB-
LP we further develop a low-
complexity
version of UCB-LP, called UCB-PLLP, since it builds on the Lagrangian (L) of UCB-LP and pessimistically (
P
tracks
the constraint violations.
We
show
that it yields ˜ O ( √ T ) problem-independent regret and guarantees zero constraint violations for any τ ≤ T with high-probability.
The computational complexity of UCB-PLLP is ˜ O ( N ) .
Main results
In this section, we present our algorithm and corresponding performance analysis for our general formulation.
All the proofs of listed lemmas, propositions and corollaries are deferred to the supplementary material.
Our experimental results are given in the suppmetary material.
2.1 Preliminary: notations and existing techniques
Notations.
For every arm i , define h
i
t ) :
= (cid:80) t − 1 τ =1 a i ( τ ) as the number of pulls of it at the beginning of round t , and ¯ µ i ( t ) :
= 1 h i ( t ) (cid:80) t − 1 τ =1 a i ( τ ) f i ( τ ) as its empirical reward estimate at round t .
Denote the feasible region of LP (2) as D :=
{ x | x ∈ [0 , 1] N , || x || 1 ≤ m, g ( x ) ≤ 0 } .
In this paper, for any vector v , we use v i to denote its i th coordinate.
For any event E , we use E to denote its negation.
Extreme points and general sub-optimality measure.
Note that the feasible region D to LP (2) is a convex polytope, and we let B be the set of its extreme points.
An extreme point of D is a point in D which does not lie in any open line segment joining two points of D .
It is well-known in the theory of LP that extreme points and basic feasible solutions are equivalent, and any LP attains its optimal value at an extreme point [6].
Recall that x
∗ is an optimal solution to LP (2), and we define the sub-optimality gap for any x ∈ B as ∆ x := ⟨ µ , x ∗ ⟩ − ⟨ µ , x ⟩ .
Define ∆ min :=
min x ∈B\X ∗ ∆ x .
Since
B is finite, ∆ min is well-defined and strictly positive.
The same definition of the sub-optimality gap is also used in [18].
We note that ∆ min can be seen as a generalization of the minimum sub-optimality gap in standard MAB problem defined as ∆ :=
min i : µ i ̸ = µ ∗ | µ ∗ − µ i | .
Specifically, under the standard
MAB setting, m = 1 , B = { a | a ∈ { 0 , 1 } N , || a || 1 ≤ 1 } , ∆ min coincides with ∆ .
In this paper, we will state our problem-dependent regret bounds in terms of ∆ min .
2.2 The general algorithm UCB-LP and its performance analysis
Now we introduce our algorithm UCB-LP for combinatorial bandits with long-term linear constraints.
UCB-LP is a generalization of SemiBwK algorithm [41] to the general linear constraints setting which chooses arms through randomized policy.
In our setting, one main challenge is that no super-arm is optimal across all rounds, but there exists an optimal sampling distribution over arms and the intuition behind UCB-LP is to identify such distribution and sampling arms based on it.
At each round, UCB-LP consists of two stages.
In the first stage, we first compute the truncated UCB estimate vector ˆ µ ( t ) ∈
R N defined as ˆ µ i ( t ) =
min (cid:110) ¯ µ i ( t ) + (cid:113) 2ln
t h i ( t ) , 1 (cid:111) , ∀ i ∈ [ N ] .
Then we solve the following
LP and get an optimal solution x ( t ) ∈ B :
[Equation]
(Note that LP (2) and LP (4) have the same feasible region D , hence the same set of extreme points
B .)
Here we require the solved optimal solution x ( t ) to be an extreme point of LP (4), i.e., x ( t ) ∈ B .
In fact, this is naturally satisfied by many LP algorithms, e.g., Simplex Method.
Even if the optimal solution of LP is not an extreme point, it can be efficiently converted to one by tightening some slack constraints in (4) [36, 19].
The second stage is to construct a distribution π t ( · ) over A with expectation x ( t ) , i.e., (cid:80) a ∈A π t ( a ) · a = x ( t ) , and sample super-arm a ( t ) ∼ π t .
Since x ( t ) ∈ D ⊆
Conv ( A ) , such π t exists and can be generated via a convex decomposition of x ( t ) .
Although there are many randomized rounding methods with O ( N 2 ) running time to achieve this, we show in the supplementary material that computing π t and sampling a ( t ) can be finished in O ( N log N ) time.
The idea of sampling arms maintaining the marginal distribution are also used in [13, 52].
Finally, we pull the arms according to a ( t ) , observe the reward value of pulled arms, and update the statistics.
Although motivated by SemiBwK algorithm [41], UCB-LP deals with general linear constraint function without any assumptions and our goal is to derive a problem-dependent regret bound.
Therefore, new techniques have to be developed for the analysis of UCB-LP.
The following theorem provides the generic bounds of regret and constraint violations for UCB-LP.
Theorem 1 UCB-LP satisfies
[Equation]
Proof sketch of Theorem 1: Note that (6) is straightforward from LP (4) and the fact that g ( · ) is linear.
Now we present the main idea of proving (5).
To obtain the regret bound, we cannot directly apply the traditional analysis from bandit community here as we cannot bound h i ( t ) and the algorithm might favor sub-optimal arms even if they have already been pulled for Ω(log T ) times due to the structural property of the LP.
Instead, we bound the number of times UCB-LP fails to yield the optimal policy, i.e., x ( t ) / ∈ X ∗ .
The most natural idea to achieve this is to bound the number of times x ( t ) =
x for every sub-optimal policy x ∈ B \ X ∗ .
However, such an idea fails in the sense that the resulting bound would scale with |B| , which is exponentially large.
Addressing this technical challenge is nontrivial and previous works circumvented this problem by assuming special structures (e.g., [18] and [40] assumed |X ∗ | = 1 , [42] assumed only one resource and || x ∗ ||
= 1 , etc) or using the prior knowledge of some problem-dependent parameters [40].
In our proof, we overcome this difficulty directly by handling the case x ( t ) /
∈ X ∗ with the idea of "regret allocation".
Define w ( t ) =
( (cid:113) 2ln
t h i ( t ) , .., (cid:113) 2ln
t h N ( t ) )
⊤ .
Since the regret only occurs when x / ∈ X ∗ , we claim that ⟨ µ , x ∗ ⟩ ≤ ⟨ ˆ µ ( t ) , x ( t ) ⟩ , ⟨ ˆ µ ( t ) , x ( t ) ≥ ⟨ µ ( t ) , x ( t ) ⟩ + ∆ x ( t ) and ⟨ w , x ( t ) ⟩ ≥ ∆ x ( t ) / 2 all hold with high probability in such case since ⟨ ˆ µ ( t ) , x ( t ) ⟩ ≥ (LP property) ⟨ ˆ µ ( t ) , x ∗ ⟩ ≥ ( high-prob ) ⟨ µ , x ∗ ⟩ ≥ ⟨ µ , x ( t ) ⟩ + ∆ x ( t ) ⇒ ⟨ ˆ µ ( t ) − µ , x ( t ) ⟩ ≥ ( high-prob )
∆ x ( t ) ⇒ ⟨ 2 w , x ( t ) ⟩ ≥ ( high-prob )
∆ x ( t ) ⇒ ⟨ w , x ( t ) ⟩ ≥ ( high-prob )
∆ x ( t ) / 2 .
With these properties, when x / ∈ X ∗ , we could allocate the incurred regret R t =
E [ ⟨ µ , x ∗ ⟩ − ⟨ µ , x ( t ) ⟩ ] =
∆ x ( t ) to every base arm in the following way, and we will argue that this allocation is correct since:
[Equation]
Thus each arm contributes 2 x i ( t ) (cid:113)
2ln
t h i ( t ) to the regret ∆ x ( t ) .
Next we claim that the sum above regret allocation on all arms is dominated by the arms in the set V ( t )
= { i : h i ( t ) ≤ 32 m 2 ln t ∆ 2min } , i.e., 12 (cid:80) i ∈ [ N ] 2 x i ( t ) (cid:113) 2ln
t h i ( t ) ≤ (cid:80) i ∈
V ( t ) 2 x i ( t ) (cid:113)
2ln
t h i ( t ) , which is derived by:
[Equation]
Therefore, the following total regret decomposition holds with high-probability:
[Equation]
Define G
i ) =
{ t : i ∈ V ( t ) } , and T i = arg max τ ∈ G
( i ) τ .
Continuing from (7) we obtain
[Equation]
The remaining problem is to bound 4 E [ (cid:80) i ∈ N (cid:80) T i t =1 x i ( t ) (cid:113)
2ln
t h i ( t ) ] , which is solved by the following lemma.
Lemma
[Equation]
Applying Lemma 1 yields 4 E [ (cid:80)
[Equation]
≤
[Equation]
+ 1) (
Since T i ∈ V ( T i )) ≤ 12 (cid:80)
i ∈
[ N ] (cid:112) 2 ln T (32 m 2 ln T/ ∆ 2min + 1) =
O ( mN ∆ min ln T ) .
Putting all things together completes the proof of Theorem 1.
Proof of Lemma 1: Our proof idea is using the facts (a): 1 / √ x ≤ 3( √ x + 1 − √ x ) , x ≥ 1 ; (b): E [ (cid:112) h
i ( τ + 1)
|F t ] =
E [ x i ( τ ) (cid:112) h i ( τ ) + 1 |F t − 1 ] + E [(1 − x i ( τ ))
(cid:112) h i ( τ ) |F
t − 1 ] , where F t = { x ( τ ) } tτ =1 .
Then E [ (cid:80) tτ
=1 E [ x i ( τ ) (cid:112)
ln τ/h
[Equation]
τ ) |F
[Equation]
]] ≤ √ ln tE [
[Equation]
tτ =
[Equation]
[Table]
Comparison with previous results.
Several previous works (e.g., [18, 42, 40, 39]) studying bandits with long term constraints have also achieved O (log T ) regret bounds.
Our regret bound has four major improvements over theirs: (a) Our regret bound has a better dependence on ∆ min and N .
(b)
Previous regret bounds only apply to the single-armed setting, while ours applies to the combinatorial setting.
(c) Our regret bound is valid for all linear constraints, while theirs is only valid for specific kind of constraints, either BwK with detereministic costs, or fairness constraints (which are the special cases of linear constraints).
(d) Almost all of them require additional assumptions or knowledge of some parameters of the problem instance a prior.
For example, [18] explicitly admitted that their (poly-)logarithmic regret and the corresponding analysis only valid under the assumption of |X ∗ | = 1 .
(See Appendix A.1 and Assumption 9 in their arxiv version.)
However, |X ∗ | = 1 does not always hold.
When consider BwK problem with only one resource, if there exist arms p, q such that µ p λ p = µ q λ q , where λ i is the amount of resource consumed by arm i if it gets pulled, then LP (2) has non-unique optimal solution, i.e., |X ∗ | ̸ = 1 .
Even consider problem of bandits with fairness constraints, if there exist two arms with the same required probability of being pulled, the optimal solution to LP (2) also may not be unique.
Beyond assuming |X ∗ | = 1 , the algorithms in [40] still require some prior knowledge of the problem instance (characterized by µ ∗ and ∆ min ).
Substantial assumptions including only one resource and “best-arm-optimality” are also required in [42].
On the contrary, we do not require any assumptions and prior knowledge of the problem instance.
Reduction to unconstrained (combinatorial) bandits setting.
When there are no constraints, i.e., g ( t ) = 0 , UCB-LP reduces to the standard (Comb) UCB1 algorithm [25] and we could also recover the results of [25] in such case.
This further justifies the tightness of our regret bound.
2.3 Achieving constant regret for fairness constraints
In this section, we consider the special case of fairness constraints.
Following the same setting in the literature, e.g., [29, 48, 39], each arm i ∈
[ N ] is required to be pulled at least r i ∈ (0 , 1) fraction of times, and (cid:80) i ∈
[ N ] r i < m .
Namely, the agent has to satisfy h
i ( T ) ≥ r i T, ∀ i ∈ [ N ] .
Define r = ( r 1 , r 2 , . . .
, r N ) ⊤ , then the equivalent linear long term constraint function g ( · ) under our setting is g ( a ( t ))
= − a ( t ) + r .
In the fairness constraints setting, the main challenge is that the time horizon
T is unknown to the algorithm beforehand, and thus the “forced exploration” trick cannot be directly applied here.
A lot of work (See Table 1) has sprung up recently to tackle this difficulty, but the best previous result is only ˜ O ( √ T ) under the combinatorial setting.
The following theorem shows that
UCB-LP could guarantee a constant regret.
Theorem 2 Define r min := min i ∈
[ N ] r i .
In the case of fairness constraints, UCB-LP guarantees that
[Equation]
Basic analysis of Theorem 2.
To derive a constant regret, one natural idea is to show that the quantity
E [ h i ( T )] − r i · T is bounded for every sub-optimal arm i (whose mean reward is not Top- m ).
However, this is not the case in classic bandit analysis (e.g., for UCB-based algorithms) and thus cannot directly apply here.
Our main proof idea is to show that UCB-LP only chooses a sub-optimal distribution over arms (i.e., x ( t ) / ∈ X ∗ ) a limited number of times for fairness constraints.
To achieve this, we first transform the event of x ( t ) /
∈ X ∗ to the events associated with arm’s UCB estimate error, i.e., we use the following lemma to characterize the case of x ( t ) /
∈ X ∗ by the sensitivity analysis of LP (4).
Lemma
If || ˆ µ ( t ) − µ || 1 < ∆ min , then x ( t ) ∈
X ∗ .
In other words, when x ( t ) / ∈ X ∗ we have || ˆ µ ( t ) − µ || 1 > ∆ min .
And then we just have to prove that || ˆ µ ( t ) − µ || 1 > ∆ min ⇒ max
i ∈
[ N ] | ˆ µ i ( t ) − µ i | ≥ ∆ min /N only happens a finite number of times.
This is a little counterintuitive for UCB-style algorithms, but it holds for UCB-LP in the fairness setting as UCB-LP guarantees that every arm has a positive probability of being pulled at each round which leads to the diminishing confidence width of all arms and the bonus of concentration inequality.
In particular, by martingale analysis and extending the concentration bound to a random process that evolves over time, we prove that there exists a constant c >
(e.g.,
[Equation]
such that h i ( t ) ≥ 8 N 2 ∆ 2min ln t holds with high-probability for each arm i when t > c , which gives that | ˆ µ i ( t ) − µ i | ≤ ∆ min /N holds with high-probability when t > c.
It is worth noting that such properties do not hold for classic bandit algorithms (e.g., algorithms based on the UCB1 framework).
And to the best of our knowledge there is no such results in the literature.
Finally, to fit into our Lemma 2, we handle the regret as Regret T ≤ m (cid:80) Tt =1 Pr[ x ( t ) /
∈ X ∗ ] ≤ mc + m (cid:80)
Tt =
c +
Pr[ x ( t ) / ∈
X ∗ ] ≤ mc + m (cid:80)
Tt =
c +
Pr[ || ˆ µ ( t ) − µ || 1 ≥ ∆ min ] .
Put all things together then we complete the proof.
Breaking the Ω(log T ) lower bound.
The constant regret bound given by Theorem 2 might seem counter-intuitive at first glance, since the lower bound of regret under classical MAB setting is Ω(log T ) [26].
However, note that the fairness level r i is required to be strictly positive, and the benchmark policy in the regret also needs to satisfy the long term constraints.
This distinguishes our setting from the classical MAB setting.
Thus, the traditional Ω(log T )
lower bound no longer applies.
The reason why UCB-LP achieves a constant regret is: UCB-LP guarantees that every arm has a positive probability of being pulled at each round.
This leads to more accurate reward estimates of arms and a diminishing gap between ˆ µ ( t ) and µ , which makes UCB-LP output the sub-optimal action distribution only a finite number of times.
That said, it still requires non-trivial techniques to establish a constant regret bound.
In fact, even O (log T ) regret bound has not yet been achieved for combinatorial setting in prior works.
Note that our regret bound has a quadratic dependence on 1 /r min , which goes to infinity as r → 0 .
In fact, when r = 0 , i.e., there are no fairness constraints, our problem degenerates to classical MAB setting where achieving constant regret bound is impossible.
This indicates that such dependence on extra parameters is inevitable for any constant regret bound.
Of course, the general O (log T ) regret bound developed in Theorem 1 still applies when r → 0 , since it does not depend on 1 /r min .
˜ O ( N ) running time.
The structural property of the fairness constraints allows us to obtain a closed form solution to LP (4), as shown in the following proposition.
Proposition 1 For each round t , rearrange the coordinates of UCB estimate vector ˆ µ ( t ) in descending order such that ˆ µ σ t 1 ( t ) ≥ ˆ µ σ t 2 ( t ) ≥ · · · ≥ ˆ µ σ tN ( t ) , where σ t 1 , . . .
, σ tN is a permutation of 1 , . . .
, N .
Then one of the optimal extreme points to LP (4) (denoted as x ( t ) ) has the following form:
[Equation]
where k t =
min { q ∈ Z + | (cid:80) qi
=1 (1 − r σ ti ) ≥ m − (cid:80) Ni =1 r i } .
Proposition 1 immediately implies that UCB-LP is computationally efficient when dealing with fairness constraints.
To solve LP (4), one can simply sort all coordinates of ˆ µ ( t ) in O ( N log N ) time, then compute k t and x ( t ) according to the closed form expression in proposition 1 (note that x ( t ) in
Proposition 1 also lies in B ).
Since the running time of computing distribution π t and sampling a ( t ) is also O
( N log N ) , we conclude that the time average complexity of UCB-LP is O ( N log N ) =
˜ O
( N ) .
Another constant regret bound.
Proposition 1 further provides two important implications: (a) Since LP (2) has the same form with LP (4), the optimal solution x ∗ to LP (2) can also be written in closed form in the same manner to Proposition 1.
Specifically, rearrange the coordinates of µ in descending order as µ σ 1 ≥ µ σ 2 ≥ · · · ≥ µ σ N , and define k = min { q ∈ Z + | (cid:80) qi
=1 (1 − r σ i ) ≥ m − (cid:80)
Ni =1 r i } .
Then x ∗ has the following form:
[Equation]
(b)
The optimal solution x ( t ) to LP (4) only depends
on
the relative order, not the absolute value of ˆ
µ
1 ( t ) ,
ˆ µ
N ( t ) .
This motivates us to characterize the sensitivity of LP (4) from a new perspective.
Intuitively, if ˆ µ and µ are close enough such that the relative order between the coordinates is (at least partially) preserved, then the optimal solutions of LP (4) and LP (2) will coincide.
The above two implications motivate us to define a new parameter ϵ :
=
min µ i ̸ = µ σk | µ i − µ σ k | , and propose the following corollary to characterize the sensitivity of LP (4).
Corollary
If || ˆ µ ( t ) − µ || ∞ <
ϵ 2 , then x ( t ) ∈
X ∗ .
Proof sketch of Corollary 1.
Proposition 1 shows that when r is fixed, for ∀ i ̸
= σ tk t , the value of x i ( t ) only depends on whether ˆ µ i ( t ) <
ˆ µ σ tkt ( t ) or not.
Similarly, for ∀ i ̸
= σ k , the value of x ∗ i only depends on whether µ i < µ σ k or not.
This implies that, if k t = k, σ tk t = σ k , and ˆ µ i ( t ) − ˆ µ σ tkt ( t ) have the same sign with µ i − µ σ k , then x i ( t ) =
x ∗ i .
In other words, if µ i > µ σ k ⇒ ˆ µ i ( t ) > ˆ µ σ k ( t ) and µ i < µ σ k ⇒ ˆ
µ i ( t ) <
ˆ µ σ k ( t ) holds for ∀ i ∈
[ N ] , then x ( t ) = x ∗ .
When || ˆ µ ( t ) − µ || ∞ < ϵ 2 , by the definition of ϵ , for ∀
i we have µ i > µ σ k ⇒ µ i ≥ µ σ k + ϵ ⇒ ˆ
µ i ( t ) > µ i − ϵ 2 ≥ µ σ k + ϵ 2 >
ˆ µ σ k ( t ) and µ i < µ σ k ⇒ µ i ≤ µ σ k − ϵ ⇒ ˆ
µ i ( t ) <
µ i + ϵ 2 ≤ µ σ k − ϵ 2 < ˆ µ σ k ( t ) .
Then x ( t ) = x ∗ .
With Corollary 1, we derive another constant regret bound of UCB-LP in the following theorem.
Theorem 3 UCB-LP also guarantees the following regret bound for fairness constraints,
[Equation]
In the supplementary material, we show that ∆ min = ϵ = ∆ when our formulation reduces to the classical MAB setting, which suggests that ∆ min and ϵ are both reasonable generalizations of ∆ .
Although ϵ and ∆ min are generally incomparable, we believe that the regret bound given by Theorem 3 is tighter than Theorem 2.
To provide evidences for this, in supplementary material, we investigate the closed form expression of ∆ min .
We show that ϵ = ∆ ≥ ∆ min = ∆(1 − (cid:80) i r i ) in the special case of m = 1 .
Intuitively, ϵ characterizes the structural properties of fairness constraints, while ∆ min is more general and applies to all linear constraints.
Remark 1 LP-sensitivity arguments are not new in bandit analyses and was firstly shown in [42] which used a technique based on LP-sensitivity to analyze the UcbBwK algorithm [1] for BwK problem in the single-arm setting.
Here we would like to point out that the technique based on
LP-sensitivity we used is completely different from theirs.
Specifically, their analysis relies on the assumptions of single resource and that the best distribution over arms reduces to the best fixed arm, while we do not require any assumptions.
Moreover, our technique is also non-standard as our sensitivity analysis (Lemma 2, Corollary 1) takes full advantage of the fairness structure (e.g., closed-form solution of LP).
Thus, our result is sharper than standard results about LP sensitivity and leads to the O (1) regret bound.
Algorithm
1: Initialization: A = { x | x ∈ { 0 , 1 } N , || x || 1 ≤ m } 2: for round t = 1 , ...
T − 1 do 3:
Compute UCBs: ˆ µ i ( t ) =
min { µ i ( t ) + (cid:113) 2ln
t h i ( t ) , 1 } , ∀ i.
Update the primal iterate: a ( t ) =
arg max a ∈A ⟨ ˆ µ ( t ) − α t (cid:80)
Kk =1 ∇ g k ( a ( t − 1))
Q k ( t ) , a ⟩ 5:
Play arm i and receive f i ( t ) if a i ( t )
= 1 .
Update the virtual queues: Q ( t + 1) =
[ Q ( t ) + g ( a ( t ))
+ ϵ t
I ] + .
Update the statistics: h i ( t + 1) , µ i ( t + 1) , ∀ i.
8: end for
Except [42] studies the BwK problem based on the LP methodology, [5] also develops an algorithm based on LP methodology to track the (contextual) blocking bandit problem wherein once an arm is pulled it cannot be played again for a fixed number of consecutive rounds.
Here we argue that their LP-sensitivity-based proof techniques cannot obtain our regret bounds.
Specifically, [5] utilizes techniques in [10] and [44], where [10] proposes the general CMAB framework and [44] improves upon its regret bounds.
The core of their techniques is to maintain a set of counters for every arm i and allocate the regret to arms according to these counters.
However, this regret allocation leads to the arms in the support of LP solution associated with the same term while their probability of being triggered by the LP-solution is different.
This coarseness makes the final regret bound to scale with the size of the support, transforming the regret bound from our result of Θ( mN ln T/ ∆ min ) to Θ( N 2 ln T/ ∆ min ) .
In contrast, our analysis is more fine-grained since we do not maintain a counter for every arm i and allocate the regret according to the counter value, but allocate the regret to arm i according to the number of times it is pulled directly.
More specifically, we only allocate the regret to arms in V ( t ) :
= { i : h i ( t ) ≤ 32 m 2 ln t ∆ 2min } , i.e., the “dominant arms” whose pulls is no more than
O ( m 2 ln t/ ∆ 2min ) times.
Thus, each arm is bounded according to its probability of being triggered, and one no longer needs to distinguish arms in the support and other arms.
Remark
When the long-term constraints only need to be satisfied in expectation, someone may adopt the well-established linear bandit algorithms like LinUCB [14] to track and identify the optimal sampling distribution over arms (optimal extreme point).
We pointed out that applying LinUCB to our model involves solving an NP-hard optimization problem and fails to achieve a logarithmic regret bound of O ( mN log T ) as our algorithm UCB-LP does.
In contrast, UCB-LP has a much lower computational complexity than the reduced LinUCB algorithm since it is polynomial-time computable.
Furthermore, UCB-LP can guarantee a bounded regret for fairness setting.
The reason why LinUCB produces sub-optimal results is that LinUCB ignores the structural properties of the constraints (e.g., concentration property bonus caused by the fairness constraints) and observations about each individual pulled arm.
2. ˜ O ( N ) running time version of UCB-LP and its theoretical performance
As mentioned earlier, UCB-LP might be computationally inefficient when constraint function g ( · ) is complicated.
In this section, we present the low computational-complexity version of UCB-LP, the UCB-PLLP algorithm, for problems of combinatorial bandits with linear long-term constraint.
Note that the main computational bottleneck of UCB-LP is caused by the constraint g ( x ) ≤ 0 .
To be computationally efficient, we optimize the (partial) Lagrangian of LP (4) at each round t :
[Equation]
| 1 ≤ m.
In (8) λ t is the Lagrange multiplier associated with the constraints at round t .
The main challenge is how to design λ t to make a good balance between reward maximization and long-term constraints satisfaction.
To address this challenge, we construct a virtual queue Q ( t ) to keep track of the “debt” of constraint violations up to round t , i.e., Q ( t ) =
[ Q ( t − 1) + g ( a ( t − 1))] + , and let λ t = α t Q ( t ) , where α t could be time-varying.
However, this Lagrange multiplier deign may lead to large constraint violations as g ( · ) becomes a “soft” constraint in LP (8) while in LP (4) it is a “hard” constraint.
To yield lower constraint violations, we incorporate the virtual queue update with “pessimistic” mechanism [34] so that the virtual queues overestimate the constraint violations, which is a novelty of our UCB-PLLP.
Although the idea of using a “pessimistic” mechanism has been exploited in linear
(contextual) bandits with safety constraints (e.g., [2, 38], etc), the pessimism in our algorithm is achieved via adding a time-varying tightness constant to virtual queue update, i.e., Q ( t ) =
[ Q ( t − 1) + g ( a ( t − 1))
+ ϵ t − 1 I ] + , which is completely different from the pessimism in previous works.
Since g ( · ) is linear, the optimization problem (8) has an integral closed-form solution and is equivalent to the following optimization problem:
[Equation]
We illustrate this algorithmic approach in Algorithm 1.
Note that if we set α t = η and ϵ t = 0 , the algorithm LBF in [29] is our special case as ∇ g k ( · ) = − e k and K = N for fairness constraints setting.
Apparently, the computational-complexity of UCB-PLLP is essentially the same as choosing the top m arms with maximum positive compound value, which is ˜ O
( N ) .
Theoretical performance of UCB-PLLP.
Here we present the regret bound and constraint violations for UCB-PLLP.
Our result relies on a mild assumption of Slater condition (Interior condition), i.e., there exists a δ > 0 and ˆ x ∈ D such that g (ˆ x ) ≤ − δ I .
Note that Slater condition automatically holds for fairness constraints as r i < 1 , ∀ i, and (cid:80) i ∈
[ N ] r i < m .
It is also a default assumption in BwK literature (They all assume the null arm denoted by 0 exists, i.e, λ i,
= 0 , ∀ i ∈
[ K ] ).
Theorem 4 Set ϵ t
=
O ( δ √ t ) and α t
= O ( Nδ √ t ) , then UCB-PLLP achieves
[Equation]
We give the proof of Theorem 4 in supplementary material.
Our proof technique is based on the
Lyapunov-drift analysis for queueing systems.
The bounds in Theorem 4 are sharp in the perspective that the regret bound matches the problem-independent regret of UCB1 algorithm in standard MAB setting, and the probability of constraints not being violated converges to 1 , i.e, holds asymptotically almost surely.
Intuition behind Theorem 4.
Here we explain the intuition why UCB-PLLP has such performance guarantees.
Since α = O (1 / √ t ) , the reward term dominates the whole term in (9) when Q k ( t ) =
o ( √ t ) .
If Q k ( t ) = ω ( √ t ) , the term containing virtual queues dominates the reward term, and UCB-PLLP tends to reduce the virtual queues Q k ( t ) .
Slater’s condition implies that there exists a policy that can reduce Q k ( t ) by a constant (related to δ ) in each round.
Therefore, the algorithm takes at most O ( √ t ) rounds to reduce Q k ( t ) to o ( √ t ) , which may lead to O ( √ t ) increase of the regret.
Thus, we could derive that Q k ( t ) =
O ( √ t ) .
Recall that Q k ( t +1) ≥ (cid:80)
Tt =1 g k ( a ( t ))
+ (cid:80)
Tt =1 ϵ t ⇒ Vio k ( T ) =
(cid:80) Tt =1 g k
( a ( t )) ≤
Q k ( T + 1) − (cid:80) Tt =1 ϵ t , we can obtain zero constraint violation via choosing proper ϵ t .
Then, the high probability constraint violation guarantee is established by bounding the exponential moment of the virtual queues since Pr( Vio k ( T ) ≥ 0)
≤ Pr( Q k ( T + 1) ≥ (cid:80)
[Table]
Remark
Although the virtual queue techniques have been used for various constrained online learning problems in the literature (e.g., [8, 9, 29, 43, 20, 49, 27, 34, 32, 31]), our virtual queue update rule differs from theirs in the pessimistic mechanism via adding a time-varying tightness constant.
Beyond the update rule of virtual queue, we also employ some new techniques in our Lyapunov analysis.
For example, we establish a bound on the exponential moment of the virtual queue length (Lemma 6), which is the central focus of our high-probability guarantee on zero constraint violations.
We also establish an upper bound on the ϵ t -tight term (incurred by our pessimistic mechanism) via comparing the optimal solution to the original LP problem and that to its -tightened version based on LP-sensitivity (lemmas 3 and 4).
These analysis techniques are not present in these works.
2.5 Experiments
The results of our numerical experiments are given in the supplementary material.
Acknowledgments and Disclosure of Funding
The work of Siwei Wang is supported in part by the National Natural Science Foundation of China
Grant 62106122.
A Proofs for Subsection 2. A.1 O ( N log N ) running time method for decomposition
Recall that A
= { a | a ∈ { 0 , 1 } N , || a || 1 ≤ m } and we define the m -set as ˜ A
= { a | a ∈ { 0 , 1 } N , || a || 1 = m } .
Note that ˜
A ⊆ A .
[52] showed in their Appendix B.2 that, for any x ∈ Conv ( ˜ A ) , one can compute a distribution π over set ˜
A such that (cid:80) a ∈ ˜
A π ( a ) · a = x , and sample the action vector a ∼ π in O ( N log N ) time.
In our setting, we only need to do the same thing for A .
Now we show that their sampling scheme can be easily generalized from ˜ A to A .
In fact, to sample an action vector with an expected value of x , if x = 0 , then the sampling is trivial, otherwise we first compute a vector ˜ x := x · m || x || 1 .
Note that ˜ x ∈ Conv ( ˜ A ) , so we can call the sampling scheme in [52] with input ˜ x as a subroutine.
Denote the output of their sampling scheme (i.e., the sampled action vector) as ˜ a .
Then ˜ a ∈ ˜ A ⊆ A .
Finally, with probability 1 − || x || 1 m , we choose action vector a = 0 as the final outcome of sampling; and with probability || x
|| 1 m , we choose action vector ˜ a as the final outcome of sampling.
It can be easily verified that the expected value of the sampled action vector equals to x .
A.2 Proof of Theorem
Proof: Note that (6) is straightforward from LP (4) and the fact that g ( · ) is linear.
Now we are going to proving (5).
Since E π t [ a i ( t )]
= x i ( t ) and the two random variables f i ( t ) and a i ( t ) are independent, therefore,
[Equation]
For convenience, we redefine h i ( t ) :
= (cid:80) t − 1 τ =1 a i ( τ )+1 , i.e., h i ( t ) ≥ 1 , and related confidence bounds at time t still hold with probability 1 − t 2 .
We also define w ( t ) =
( (cid:113) 2ln
t h i ( t ) , .., (cid:113) 2ln
t h N ( t ) )
⊤ .
When x / ∈ X ∗ , the following inequalities hold with high-probability 1 − t 2 :
[Equation] [Equation]
Where (11) and (12) are due to
[Equation]
(13) is given by continuing from (14), i.e.,
[Equation]
⇒ ⟨
[Equation]
µ ( t ) − µ
[Equation]
x (
[Equation]
x ( t
[Equation]
⇒ ⟨ 2 w
[Equation]
x ( t
[Equation]
≥ ∆ x ( t ) ⇒ ⟨ w , x ( t ) ⟩ ≥ ∆ x ( t ) /
[Equation]
We first decompose the regret as:
[Equation]
Define V ( t )
=
{ i : h i ( t ) ≤ 32 m 2 ln t ∆ 2min } .
When I { x ( t ) / ∈ X ∗ , ⟨ r , x ( t ) ⟩ ≥ ∆ x ( t ) / 2 } = 1 we have
[Equation]
i
[Equation] [Equation]
] x
[Equation]
(15) due to the facts that
[Equation]
and
[Equation]
x
[Equation]
Therefore,
Regret T
[Equation]
Define G
i ) =
{ t : i ∈ V ( t ) }
[Table]
and T i =
arg max τ ∈ V ( t ) τ .
Then we obtain
[Table] [Table]
[Equation]
B Proofs for Subsection 2. B.1 Proof of Theorem
Proof: Firstly, plug (10) into the definition of regret in (3), we decompose the regret as
[Equation]
In (18), c is a parameter to be determined later, (a) holds due to Lemma 2 and (b) is because
[Equation]
Note that ˆ µ i ( t ) − µ i ≥ ∆ min /N implies that at least one of the following two events must happen:
[Equation]
The probability of the first event could be bounded with Chernoff-Hoeffding inequality: Pr[ F i ( t )]
≤ t − 2 .
For the second event J i ( t ) , notice that
[Equation]
The next step is to set c to be sufficiently large such that
[Equation]
We set c =
[Equation]
and claim that it satisfies (20).
The proof is by basic algebra and we defer the details to Appendix B.3.
According to (19) and (20), ∀ t ≥ c , J i ( t ) implies
[Equation]
where (a) in (21) is because the fairness constraints guarantee that x i ( τ ) ≥ r
i , ∀ τ ∈ [ t ] .
Consequently,
[Equation]
Define a filtration up to time t : H t = { ( f ( τ ) , a ( τ )) } tτ =1 , where f ( τ ) :
= ( f 1 ( τ ) , f 2 ( τ ) , . . .
, f N ( τ )) ⊤ .
Note that a i ( t ) − x i ( t ) is a martingale difference with respect to the filtration H t .
From Azuma-
Hoeffding inequality we have, ∀ t ≥ c,
[Equation]
Therefore, ∀ t ≥ c
Pr[
[Equation]
On the other hand, using Chernoff-Hoeffding inequality we get
[Equation]
Then Pr[ | ˆ µ i ( t ) − µ i | ≥ ∆ min /N ] ≤ 3 /t 2 , ∀ t ≥ c.
Continuing from (18), the regret can be bounded as
Regret T
[Equation]
+ m
N
[Equation]
This completes the proof.
B.2 Proof of Lemma
Proof: To derive a contradiction, we assume that x ( t ) / ∈
X ∗ when || ˆ µ ( t ) − µ || 1 < ∆ min .
Note that x ( t ) ∈
( B \ X ∗ ) , then the definition of ∆ min and x ( t ) implies
[Equation]
and ˆ
[Equation]
t )
[Equation]
x ∗ .
Combining the above two inequality gives
[Equation]
which contradicts the conditions
||
ˆ µ ( t ) − µ || 1 < ∆ min and || x ( t ) − x ∗ || ∞ ≤ 1 .
B.3 Proof of of the adequacy of c
Now we prove that, in the proof of Theorem 2, the constant c defined as c =
[Equation]
satisfies (20).
For the simplicity of notation, define γ = 32 N 2 r 2min ∆ 2min , then c = γ ln 2 γ .
First we show that c ln c ≥ γ .
In fact,
[Equation]
The last inequality in (23) holds because ln γ ≥ ln(32) >
2 and ln γ − 1 ≥ ln ln γ .
Thus, c ln c ≥ γ .
Since t ≥ c ≥ 32 > e and the function t ln t monotonically increases for t > e , we have t ln t ≥ c ln c ≥ γ, ∀ t ≥ c .
Define a shorthand ω :
= (cid:113) t ln t .
Then ω = (cid:113) t ln t ≥ √ γ =
[Equation]
√ 2 N r min ∆ min .
Consider ∀ t ≥ c, we have
[Equation]
The definition of r min implies that r i ω − 1 ≥ 4 √
Nr
i r min ∆ min − 1 ≥ 4 √ 2 N ∆ min − 1 ≥ √ 2 N ∆ min .
Then ( r i ω − 1) ω ≥ √ 2 N ∆ min · 4 √ 2 N ∆ min = 8 N 2 ∆ 2min , the correctness of the last inequality in (24) has been verified.
So (20) is proved.
B.4 Proof of Proposition
Proof: Due to the definition of k t , we have (cid:88)
i<k t (
[Equation]
ti
[Equation]
< m −
N (cid:88)
[Equation]
and k t (cid:88)
[Equation]
=
[Equation]
ti
[Equation]
m −
N (cid:88)
[Equation]
The above two inequalities imply that x ( t ) is feasible, i.e., x ( t ) ∈ D .
Then x ( t ) is an extreme point in B since there are N constraints in LP (4) that are active (tight) at point x
( t ) .
Now to prove this proposition, we only need to prove that for any x ′ ∈ D , ⟨ ˆ µ ( t ) , x ( t ) ⟩ ≥ ⟨ ˆ µ ( t ) , x ′ ⟩ .
In fact, ⟨ ˆ µ ( t ) , x ( t ) ⟩ − ⟨
[Equation]
( t ) , x ′ ⟩ = (cid:88)
[Equation]
where the first inequality holds because x σ ti ( t )
= 1 ≥ x ′ σ ti for all i < k t , and x σ ti ( t ) =
r σ ti ≤ x ′ σ ti for all i > k t .
Thus, x ( t ) is optimal and we complete the proof.
B.5 Proof of Corollary
Proof: According to Proposition 1, when r is fixed, for ∀ i ̸
= σ tk t , the value of x i ( t ) only depends on whether ˆ µ i ( t ) <
ˆ µ σ tkt ( t ) or not.
Similarly, for ∀ i ̸
= σ k , the value of x ∗ i only depends on whether µ i < µ σ k or not.
This implies that, if k t = k, σ tk t = σ k , and ˆ µ i ( t ) − ˆ µ σ tkt ( t ) have the same sign with µ i − µ σ k , then x i ( t ) =
x ∗ i .
In other words, if µ
[Equation]
µ
[Equation]
⇒
[Equation]
t ) and µ
[Equation]
< µ
[Equation]
k ⇒ ˆ µ
[Equation]
t ) <
ˆ µ
[Equation]
k ( t ) holds for ∀ i ∈
[ N ] , then x ( t ) = x ∗ .
When || ˆ µ ( t ) − µ || ∞ < ϵ 2 , by the definition of ϵ , for ∀
i we have
[Equation]
and
[Equation]
Then x ( t ) = x ∗ .
B.6 Proof of Theorem
Proof: In the same manner to (18), except that corollary 1 instead of Lemma 2 is applied, we have
[Equation]
where c ′ is still a parameter to be determined later, as in the the proof of Theorem 2.
Note that the last line in (25) is just the last line in (18), except that ∆ min /N is replaced with ϵ 2 .
Therefore we can set c ′ =
[Equation]
in the same manner as in the proof of Theorem 2.
The remaining part of the proof is almost the same as the proof of Theorem 2, and we omit the details.
B.7 Bridging ϵ , ∆ min and ∆ .
In this subsection, we investigate the closed-form solution of ∆ min in our formulation in some special cases of the fairness constraints.
Based on the derived closed form expression, we provide evidences for the belief that the regret bound given by Theorem 3 is tighter than Theorem 2.
Specifically, we show that when m = 1 , ∆ min = ϵ (1 − (cid:80) i r i ) , which immediately implies that the regret bound given by Theorem 3 is tighter than Theorem 2.
We also provide a better understanding of the relationship between the two optimality gaps ∆ min and ϵ used in this paper and the traditional optimality gap ∆ used in classical MAB setting.
In fact, both ∆ min and ϵ could be seen as a generalized version of ∆ , as both of them coincides with ∆ under the classical MAB setting.
Consider the case when m = 1 .
In this special case, ∆ min can be written in closed form, and we prove that ∆ min ≤ ϵ .
In fact, if m = 1 , the extreme points of D are
[Equation]
Then for ∀ j ∈ [ N ] , we have
[Equation]
Hence max x ∈B ⟨ µ , x ⟩ = max j ( ⟨ µ , r ⟩ + µ j (1 − (cid:80) i r i )) = ⟨ µ , r ⟩ + µ ∗ (1 − (cid:80) i r i ) , and ∆ min can be written in closed form as
[Equation]
On the other hand, when m = 1 , k = min { l | (cid:80) li =1 (1 − r σ i ) ≥ 1 − (cid:80) Ni =1 r i , l ∈ Z + } = 1 .
Note that µ σ 1 = µ ∗ due to the definition of σ , then
[Equation]
So we have ∆ min =
∆(1 − (cid:80) i r i ) <
∆ =
ϵ , this implies that when m = 1 , the regret bound in
Theorem 3 is tighter than the regret bound in Theorem 2.
When m > 1 , obtaining the closed form expression of ∆ min is complicated, and it is hard to establish any direct inequality between the two regret bounds.
However, based on the insights given by analyzing the case of m = 1 , we believe that the regret bound in Theorem 3 is also likely to be tighter for general m values.
We also note that, under the classical MAB setting, m = 1 and there is no fairness constraints, i.e., r = 0 , both ∆ min and ϵ coincides with ∆ .
This validates the reasonability and tightness of the two optimality gaps used in this paper.
C Proofs for Subsection 2.
Preliminary.
Since D = { x | x ∈ [0 , 1] N , || x || 1 ≤ m, g ( x ) ≤ 0 } is bounded, the function g k ( · ) , ∀ k ∈ [ K ] , has bounded function value on D .
In other words, there exists a constant U >such that g k ( x ) ≤ U, ∀ x ∈ D , k ∈
[ K ] .
C.1 Proof of Theorem
Firstly, we give a proof sketch of Theorem 4.
The high-level idea of our proof for regret bound is decomposing the regret as follows,
Regret T ≤ T µ · x ∗ − E [
T (cid:88) t = µ · x ( t )]
= T µ · ( x ∗ − a ϵ t , ∗ ) (cid:124) (cid:123)(cid:122) (cid:125) term
A + E [
T (cid:88) t
= (ˆ µ ( t ) · x ϵ t , ∗ − ˆ µ ( t ) · a ( t ))]
(cid:124) (cid:123)(cid:122) (cid:125) term B + E
T (cid:88) t
= ( µ − ˆ µ ( t )) ·
x ϵ t , ∗ ] (cid:124) (cid:123)(cid:122)
[Equation]
where x ϵ t , ∗ is one of the optimal solutions to the ” ϵ t -pessimistic” version of LP (2), i.e.,
[Equation]
To prove the regret bound in Theorem 4, we bound the term B using the Lyapunov-drift analysis [37, 29], etc.
Term A could be bounded based on the intuition that adding ϵ t -tightness to the feasible region only incurs O ( ϵ t ) loss in the optimal objective value.
Terms C and D could be regarded as "reward mismatch" and we bound them using the similar analysis from [1, 41] based on the concentration bound.
For the constraint violations bound at any τ ≤ T , we firstly characterize it by the bound of || Q ( τ ) || 1 .
Then we use the drift property of Q ( · ) and convergence time analysis from random process community to bound e || Q ( τ ) || 1 , which could be transformed into high-probability upper bound of || Q ( τ ) || 1 via
Markov inequality.
Now we give a proof of Theorem 4.
To obtain the regret bound, as mentioned before, we first decompose the regret expression as follows,
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
[Equation]
[Equation]
We next present a sequence of lemmas that bounds the terms above.
Lemma
The algorithm UCB-PLLP guarantees that
[Equation]
Lemma
For any sequence { ϵ t } t which satisfies 0 ≤
[Equation]
≤ δ, ∀ t , we have that
[Equation]
Lemma 5 ∀ a ∈ A , the following inequality holds
[Equation]
Then according to Lemma 4, we bound term
A as follows,
[Equation]
Term B could be upper-bounded based on the Lemma 3:
[Equation]
And Lemma 5 gives
[Equation]
[Equation]
Thus, combine the (30), (31), and (32), we have the following upper-bound for regret,
Regret T ≤ m
[Equation]
T
[Equation]
Let p =
δ 4 and q = 8 N √ K δ .
Since α t = q √ t and ϵ t = p √ t , (33) yields
Regret T ≤ m
[Equation]
δ
[Equation] [Equation]
where (a) in (34) follows from the fact that || Q ( t ) || 1 ≤ Ut + (cid:80) tτ
=
1 ϵ τ ≤ t ( U + ϵ 0 ) ; (b) in (34) holds due to ϵ t ≤ δ , (cid:80)
Tt =1 1 √ t ≤ 2 √ T and (cid:80) Tt =1 t − 32 ≤ 2 .
To obtain the constraint violations bound, according to the update rule of Q ( t ) , we have
Q k ( t + 1) ≥
T (cid:88) t =
g k ( a ( t ))
+
T (cid:88) t =
ϵ t ⇒
[Equation]
Let η =
[Equation]
Combine (35) with the Markov inequality gives
[Equation]
where (a) in (36) comes from the following lemma:
Lemma 6 When ϵ ≤ δ 4 , UCB-PLLP guarantees that
[Equation]
where η =
[Equation]
This completes the proof.
C.2 Proof of Lemma
Proof Define Lyapunov drift ∆( t ) =
12 ( || Q ( t + 1) ||
2 − || Q ( t ) || 2 ) .
According to the evolution dynamics of Q ( t ) , we have
[Equation]
where (a) in (38) holds due to g k ( · ) is bounded by U .
Recall that A
= { a | a ∈ { 0 , 1 } N , || a || 1 ≤ m } and D = { x | x ∈ [0 , 1] N , || x
|| 1 ≤ m, g ( x ) ≤ 0 } , and define ˆ D = { x | x ∈ [0 , 1] N , || x
|| 1 ≤ m } then we have
[Equation] [Equation]
Since a ( t ) = arg max a ∈A (cid:68) ˆ µ ( t ) − α t (cid:80)
Kk =1 ∇ g k ( a ( t − 1))
Q k ( t ) , a (cid:69) , then for any y ∈ D we have (cid:42) ˆ µ ( t ) − α t
K (cid:88) k
= ∇ g k ( a ( t − 1))
Q k ( t ) , a ( t ) (cid:43) ≥ (cid:42) ˆ µ ( t ) − α t
K (cid:88) k
= ∇ g k ( a ( t − 1))
Q k ( t ) , y (cid:43) ⇔ ⟨ ˆ µ ( t ) , a ( t ) ⟩ − α t
K (cid:88) k
=
Q k ( t ) ⟨∇ g k ( a ( t − 1)) , a ( t ) ⟩ ≥ ⟨ ˆ µ ( t ) , y ⟩ − α t
K (cid:88) k
=
Q k ( t ) ⟨∇ g k ( a ( t − 1)) , y ⟩ ⇔ ⟨ ˆ µ ( t ) , a ( t ) ⟩ − α t
K (cid:88) k
=
Q k ( t ) ⟨∇ g k ( a ( t − 1)) , a ( t ) − a ( t − 1) ⟩ ≥ ⟨ ˆ µ ( t ) , y ⟩ − α t
K (cid:88) k
=
Q k ( t ) ⟨∇ g k ( a ( t − 1)) , y − a ( t − 1) ⟩ ( a ) ⇔ ⟨ ˆ µ ( t ) , a ( t ) ⟩ − α t K (cid:88) k
=
Q k ( t ) g k ( a ( t )) ≥ ⟨ ˆ µ ( t ) , y ⟩ − α t
K (cid:88) k
=
Q k ( t ) g k ( y ) ⇔ ⟨ ˆ µ ( t ) , a ( t ) ⟩ − α t [ g ( a ( t ))] T Q ( t ) ≥ ⟨ ˆ µ ( t ) , y ⟩ − α t [ g ( y )]
[Equation]
) where (a) holds since g k ( · ) is linear.
The above inequality is also equivalent to:
[Equation]
Divide both sides of (39) by α t and add it to inequality (38)
[Equation]
then we obtain
[Equation]
Rearrange terms yields
[Equation]
Denote H ( t ) =
[ Q ( t ) , ˆ µ ( t )] the current state.
Substitute y by x ϵ t , ∗ in (41) and conditional on
H ( t ) =
h , we have
[Equation]
where (a) and (b) in (42) come from the fact that g ( x ϵ t , ∗ ) + ϵ t I ≤ 0 .
Thus, the Lyapunov drift ∆( t ) satisfies
[Equation]
(43) Take expectations with respect to H ( t ) , multiply α t on both sides and conduct a telescope summing over t lead to
E [
[Equation]
where (a) in (44) holds since α t is non-increasing over time t ; (b) in (44) holds due to || Q (1) ||
= 0 .
Then we complete the proof.
C.3 Proof of Lemma
Proof Due to the Slater condition, there exists a δ > 0 and ˆ x ∈ D such that g (ˆ x ) ≤ − δ I .
Define x ϵ t =
(1 − ϵ t δ ) x ∗ +
[Equation]
t δ ˆ x .
Since 0 ≤ ϵ t ≤ δ and
[Equation]
we can verify that x ϵ t ∈ D and
[Equation]
where the second equality in (45) holds due to the linearity of g ( · ) .
Hence x ϵ t is exactly the feasible solution to the " ϵ t -tightness" version of the optimization problem (2) and hence we have µ · x ϵ t ≤ µ · x ϵ t , ∗ .
So we can obtain
[Equation]
This completes the proof.
C.4 Proof of Lemma
Proof We prove the Lemma 5 by following the standard analysis from the traditional bandits community.
First we have the following supportive lemma.
Lemma 7 (Hoeffding’s lemma)
With probability at least 1 − 2 T − 1 , we have
[Equation]
(cid:115) 2 ln T
N i ( t ) .
By the definition of ˆ µ ( t ) , ∀ a ∈
A we have |
T (cid:88) t = a · ˆ µ ( t ) − a · µ | ≤ |
N (cid:88) i =
T (cid:88) t = a i ( µ i ( t ) − µ i ) | +
N (cid:88) i =
T (cid:88) t
= (cid:115) 2 ln t
N i ( t ) a i (48)
Using Lemma 7, we have that with probability 1 − 2 T − 1 , |
N (cid:88) i =
T (cid:88) t = a i ( µ i ( t ) − µ i ) | ≤
N (cid:88) i = |
T (cid:88) t = a i ( µ i ( t ) − µ i ) | ≤ (cid:88) a i >
T (cid:88) t =
| µ i ( t ) − µ i | ≤ (cid:88) a i >
T (cid:88) t
= (cid:115) 2 ln T
N i ( t ) (49)
Combine (48) with (49), then with 1 − 2 T − 1 we obtain |
T (cid:88) t = a · ˆ µ ( t ) − a · µ | ≤ (cid:88) a i >
T (cid:88) t
= (cid:115) 2 ln T
N i ( t ) +
N (cid:88) i =
T (cid:88) t
= (cid:115) 2 ln t
N i ( t ) a i ≤ 2 (cid:88) a i >
T (cid:88) t
= (cid:115) 2 ln T
N i ( t ) ( a ) ≤ 2 √ 2 ln T (cid:88)
a i >(cid:112) 2 N
[Equation]
) where (a) in (50) comes from the fact that (cid:80) ti =1 1 √ i ≤ 2 √ t .
Therefore, we can get
E [ |
T (cid:88)
[Equation]
) where the first inequality in (51) holds since | a · ˆ µ ( t ) − a · µ | ≤
|| x
||
|| ˆ µ ( t ) − µ || ∞ ≤ 2 m.
And we complete the proof.
C.5 Proof of Lemma
Proof According to (41), for any y ∈ D , we have
[Equation]
Substitute y by ˆ x into (52) we get
[Equation]
where (a) in (53) holds due to the Slater condition.
(53) implies that
[Equation]
Next we define ˜ L ( t ) =
|| Q ( t ) || , and when ˜ L ( t ) ≥ 2 αt N +2 K
( U 2 + ϵ 2 t ) δ , we have
[Equation]
where (a) in (55) follows from (54); (b) in (55) is due to ϵ t ≤ δ 4 .
Besides, based on the update rule of Q ( t ) , the following inequality holds,
[Equation]
Define η =
[Equation]
3 , then we can derive that
[Equation]
according to the lemma below.
Lemma 8 (Lemma 3.8 in [2]) Let S ( t ) be the state of Markov chain, L ( t ) be a Lyapunov function and its drift denotes ∆( t ) =
L ( t + 1) − L ( t ) .
When the following two conditions satisfied
• There exists constant γ > 0 , increasing sequence { θ } Tt =1 such that E [∆( t ) | S ( t )
=
s ] ≤ − γ when L ( t ) ≥ θ t .
• || L (
t + 1) − L ( t ) || ≤ v holds for any t .
Then we have
[Equation]
where η = γ [max {
[Equation]
γ
[Equation] [Equation]
+
[Equation]
Since || Q ( t ) || 1 ≤ √ K || Q ( t ) || , according to (57) it is obvious that
[Equation]
Then we complete the proof.
D Numerical experiments
In this section, we conduct numerical experiments for fairness constraints and knapsacks constraints, respectively, to validate the theoretical guarantees of our algorithms.
D.1 Fairness constraints
For the simulation setup, we set N = 4 , m = 2 , µ = (0 . 4 , 0 . 5 , 0 . 6 , 0 . 7) and r = (0 . 5 , 0 . 6 , 0 . 4 , 0 . 3) .
The representative baselines we used are: LFG [29] with η = O ( 1 √ T ) , and CMF [48] with α = ∞ .
We do not compare Thompson-Sampling-based algorithm [20] here as it requires additional knowledge on the prior distributions of µ and the latent distributions of the arms, which is not assumed in other algorithms.
In fact, their algorithm is the same as LFG except for the way of estimating rewards, which leads to almost the same performance.
We do not compare [39, 12, 16] as their algorithms only work for m = 1 .
For fair comparisons, we use the same confidence bound in all algorithms.
We simulate our algorithms and baselines for T = 3 × 10 4 rounds.
Every point in the figure is averaged over 100 independent trials.
Results and analysis.
Figure 1 shows the time-averaged regret and total constraint violations of all compared algorithms.
From this figure, we can see that UCB-LP indeed guarantees a constant regret bound and zero constraint violations in expectation, which validates our theoretical results.
The empirical performance of UCB-PLLP in our experiment also coincides with the results of Theorem 3.1.
However, UCB-PLLP achieves a regret that is closed to constant in our experiment.
It is not strange that LFG achieves negative regret as it compromises constraint violation to pull high-reward arms.
For CMF, the empirical performance also matches its theoretical results.
D.2 Knapsacks constraints
For the simulation setup, we choose N = 5 and K = 3 .
The mean rewards and mean costs of other arms are generated as follows: (a) µ 1 = 0 .
5 , λ 1 ,j = 0 .
45 , ∀ j ∈ [3]; (b) for all arms 2 ≤ x ≤ 5 , µ x is sampled from Uniform([ 0 .
5 − 2 σ, 0 .
5 − σ ]), and λ x,j is sampled from Uniform([ 0 . 45 + σ, 0 .
45 + 2 σ ]), ∀ j ∈ [3] , where σ = 0 .
We also set B j = 0 .
45 T, ∀ j ∈ [ d ] .
The baselines we used are these works which obtained logarithmic regret for deterministic costs: UCB-Simplex [18], and BNPA [40] with ϵ
= O ( log TT ) .
We do not compare UCB-Simplex-v2 [18] here as it is the same as BNPA with ϵ =(
but when ϵ > 0 , the regret upper bound of BNPA is better than it).
We do not compare [42, 15] as their algorithms and results require restrictive assumptions like only one resource.
Since all baselines only work for m = 1 , we set m = 1 in our simulation setup.
All algorithms are simulated on the same datasets.
Every point in every figure is averaged over 50 independent trials.
Results and analysis.
Since all algorithms can guarantee zero constraint violations in expectation, we do not show the comparison of their constraint violations.
Figure 2 (a) shows that the ratio of regret to log T of all algorithms except UCB-PLLP approaches a constant, i.e., achieving a logarithmic regret.
This is consistent with their theoretical results.
We can also see from this figure that UCB-LP has a best empirical performance under our experimental setup.
Figure 2 (b) presents the running time of these algorithms, which highlights the advantage of UCB-PLLP in computational complexity.