On Kernelized Multi-Armed Bandits with Constraints
Abstract
We study a stochastic bandit problem with a general unknown reward function and a general unknown constraint function.
Both functions can be non-linear (even non-convex) and are assumed to lie in a reproducing kernel Hilbert space (RKHS) with a bounded norm.
In contrast to safety-type hard constraints studied in prior works, we consider soft constraints that may be violated in any round as long as the cumulative violations are small.
Our ultimate goal is to study how to utilize the nature of soft constraints to attain a ﬁner complexity-regret-constraint trade-off in the kernelized bandit setting.
To this end, leveraging primal-dual optimization, we propose a general framework for both algorithm design and performance analysis.
This framework builds upon a novel sufﬁcient condition, which not only is satisﬁed under general exploration strategies, including upper conﬁdence bound (UCB),
Thompson sampling (TS), and new ones based on random exploration , but also enables a uniﬁed analysis for showing both sublinear regret and sublinear or even zero constraint violation.
We demonstrate the superior performance of our proposed algorithms via numerical experiments based on both synthetic and real-world datasets.
Along the way, we also make the ﬁrst detailed comparison between two popular methods for analyzing constrained bandits and Markov decision processes (MDPs) by discussing the key difference and some subtleties in the analysis, which could be of independent interest to the communities.
Introduction
Stochastic bandit optimization of an unknown function f has recently gained increasing popularity due to its widespread real-life applications such as recommendations [1], cloud resource conﬁgurations [2], and wireless power control [3].
At each time t , an action x t is selected and then a (noisy) bandit reward feedback y t is observed.
The goal is to maximize the cumulative reward, or equivalently minimize the total regret due to not choosing the optimal action in hindsight.
To capture a generic function, researchers have turned to nonparametric models of f via Gaussian process or reproducing kernel Hilbert space (RKHS), which are able to uniformly approximate an arbitrary continuous function over a compact set [4].
In this paper, as in [5, 6], we consider the agnostic setting (i.e., frequentist-type), where f is assumed to be a ﬁxed function in an RKHS with a bounded norm (i.e., a measure of smoothness).
We call this setting frequentist-type kernelized bandits (KB).
In addition to a generic non-linear or even non-convex function f , another common feature in practical applications is that there often exist additional constraints in the decision-making process such as hard-constraint like safety or soft-constraint like cost.
To this end, there have been exciting recent advances in the theoretical analysis of constrained kernelized bandits.
In particular, [7–9] propose algorithms with convergence guarantees, while [10], to the best our knowledge, is the ﬁrst 36th Conference on Neural Information Processing Systems (NeurIPS 2022). work that establishes regret bounds for their developed algorithm, although under the Bayesian-typesetting.
These algorithms mainly focus on KB with a hard constraint such as safety, i.e., the selected action in each round needs to satisfy the constraint with a high probability.
Hence, compared to the unconstrained case, additional computation is required to construct a safe action set in each round, which not only incurs additional complexity burdens, but often leads to conservative performance.
Motivations.
In practice, there are also many applications that involve soft constraints that may be violated in any round.
The goal is to maximize the total reward while minimizing the total constraint violations.
For example, in a wireless networking system, the reward could be the throughput while the constraint is that the average energy consumption is below a threshold.
In this case, the constraint can be violated at any time, and moreover, using less energy at one time instant allows one to use more in the next time instant, i.e., “compensation” across time (see a formal deﬁnition in Eq.(1))
Furthermore, existing provably efﬁcient algorithms [7–10] largely build on upper conﬁdence bound (UCB) exploration, which often has inferior empirical performance compared to Thompson sampling (TS) exploration.
Hence, another key question is whether one can design provably efﬁcient CKB algorithms with general explorations.
In summary, the following fundamental theoretical question remains open:
Can a ﬁner complexity-regret-constraint trade-off be attained in CKB under general explorations?
Contributions.
In this paper, we take a systematic approach to afﬁrmatively answer the above fundamental question.
In particular, we tackle the complexity-regret-constraint trade-off by formulating
KB under soft constraints as a stochastic bandit problem where the objective is to maximize the cumulative reward while minimizing the cumulative constraint violations and maintaining the same computation complexity as in the unconstrained case.
Our detailed contributions are as follows.
• We develop a uniﬁed framework for CKB based on primal-dual optimization, which can guarantee both sublinear reward regret and sublinear total constraint violation under a class of general exploration strategies, including UCB, TS, and new effective ones (e g., random exploration) under the same complexity as the unconstrained case.
We also show that by introducing slackness in the dual update, one can trade regret to achieve bounded or even zero constraint violation.
This framework builds upon a novel sufﬁcient condition, which not only facilitates the design of new CKB algorithms but provides a uniﬁed view in the performance analysis.
• We demonstrate the superior performance of our proposed algorithms via numerical experiments based on both synthetic and real-world data.
In addition, we discuss the beneﬁts of our algorithms in terms of various practical considerations such as low complexity, scalability, robustness, and ﬂexibility.
• We also provide the ﬁrst detailed comparison between two popular methods for analyzing constrained bandits in general.
Speciﬁcally, the ﬁrst one is based on convex optimization tool as in [11, 12], which is also the inspiration for our paper.
The other one is based on Lyapunov-drift argument as in [13–15].
We discuss the key difference in terms of regret and constraint violation analysis in these two methods and highlight the subtlety in applying a standard queueing technique (i
e., Hajek lemma [16]) to bound the constraint violation in the second method.
We believe this provides a clear picture on the methodology, which is of independent interest to the communities.
1.1 Related Work
In the special cases of KB, including multi-armed bandits (MAB) and linear bandits (e.g., KB with a linear kernel), there is a large body of work on bandits with different types of constraints, including knapsack bandits [17–19], conservative bandits [20–22], bandits with fairness constraints [23, 24], bandits with hard safety constraints [25–27], and bandits with cumulative soft constraints [28, 13].
Among them, the bandit setting with cumulative soft constraints is the closest to ours.
In particular, [13] considers linear bandits under UCB exploration and a zero constraint violation is attained via the Lyapunov-drift method.
However, it is unclear how to generalize it to handle general exploration strategies such as TS; see a further discussion in Section 4. process and the observation noise is Gaussian.
In contrast, in our considered frequentist-type KB, f is a ﬁxed function in an RKHS and the noise can be any sub-Gaussian.
A better regret bound can often be achieved in the easier Bayesian-type KB setting [6].
Broadly speaking, our work is also related to reinforcement learning (RL) with soft constraints, i.e., constrained MDPs.
In particular, our analysis is inspired by those on constrained MDPs [11, 12] (which is another popular method to handle constrained bandits and MDPs via convex optimization tools), but has signiﬁcant differences.
First, in those works, the constraint violation is e O ( p T ) .
In contrast, ours can attain bounded and even zero constraint violations by introducing the slackness in the dual update.
Second, they only consider UCB-type exploration, but our algorithms can be equipped with various exploration strategies (including UCB), thanks to our general sufﬁcient condition.
Third, they focus on either tabular or linear function approximation settings.
In contrast, both objective and constraint functions we consider can be nonlinear .
There are also recent works on constrained MDPs that claim to achieve bounded or zero constraint violation [14, 15] based on the Lyapunov-drift method.
However, as in the bandit case, it is unclear how to generalize it to handle general explorations beyond UCB.
Finally, the theoretical understanding of unconstrained
KB (a.k.a.
Gaussian process bandit) is well-established, including regret upper bounds under UCB and TS [5, 29, 6] and lower bounds [30, 31].
We remark that our work is mainly a theory-guided study.
In a more practical area of KB, i.e., Bayesian optimization (BO), there have been many BO algorithms developed for the constrained setting; see [32–34] and the references therein.
Although these algorithms enjoy good performance in various practical settings, their theoretical performance guarantees are still unclear.
Problem Formulation and Preliminaries
We consider a stochastic bandit optimization problem with soft constraints.
In particular, in each round t 2 { 1 , 2 , . .
, T } , a learning agent chooses an action x t 2 X and receives a bandit reward feedback r t = f ( x t ) + ⌘ t , where ⌘ t is a zero-mean noise.
The learning agent also observes a bandit constraint feedback c t = g ( x t ) + ⇠ t , where ⇠ t is a zero-mean noise.
By soft constraints, the goal here is to maximize the cumulative reward while minimizing the cumulative total violation.
Learning Problem.
Deﬁne cumulative regret and constraint violation as follows:
[Equation]
where x ⇤ : = argmax { x 2 X : g ( x )  0 } f ( x ) and [ · ] + := max {· , 0 } .
The goal of the learning agent is to achieve both sublinear regret and sublinear constraint violation.
In fact, we will establish bounds on the following stronger version of regret.
Speciﬁcally, let ⇡ be a probability distribution over the set of actions
X (i.e., stochastic policy), and let E ⇡ [ f ( x )] := R x 2 X f ( x ) ⇡ ( x ) dx and
E ⇡ [ g ( x )] := R x 2 X g ( x ) ⇡ ( x ) dx .
We compare our achieved reward with the following baseline optimization problem: max ⇡ { E ⇡ [ f ( x )] : E ⇡ [ g ( x )]  0 } where both f and g are known, and ⇡ ⇤ is its optimal solution.
Now, our stronger regret is deﬁned as
[Equation]
Clearly, we have R ( T )  R + ( T ) 2 .
Throughout the paper, we assume the following commonly used condition in constrained optimization literature; see also [13, 35, 11].
Assumption 1 (Slater’s condition) .
There is a constant  > 0 such that there exists a probability distribution ⇡ 0 that satisﬁes E ⇡
[ g ( x )]    .
Without loss of generality, we assume   1 .
This is a quite mild assumption since it only requires that one can ﬁnd a stochastic policy under which the expected cost is less than a strictly negative value.
This is in sharp constraint to existing
KB algorithms for hard constraints that typically require the existence of an initial safe action [9, 10].
In this paper, we consider the frequentist-type regularity assumption that is typically used in uncontrained KB works (e.g., [5, 6]).
Speciﬁcally, we assume that f is a ﬁxed function in an RKHS with a bounded norm.
In particular, the RKHS for f is denoted by H k , which is completely determined by the corresponding kernel function k : X ⇥ X !
R . Any function h 2 H k satisﬁes the reproducing property : h ( x ) =
h h, k ( · , x ) i H k , where h · , · i H k is the inner product deﬁned on H k .
Similarly, for the unknown constraint function g , we assume that g is a ﬁxed function in the RKHS deﬁned by a R [Equation] [Equation]
kernel function
e k , and the RKHS for g is denoted by H e k .
We assume that the following boundedness property holds throughout the paper.
Assumption 2 (Boundedness) .
We assume that k f k H k  B and k ( x, x )  1 for any x 2 X and that the noise ⌘ t is i.i.d.
R -sub-Gaussian.
Similarly, we assume that k g k H e k  G and e k ( x, x )  1 for any x 2 X and that the noise ⇠ t is i.i.d. e
R -sub-Gaussian.
Gaussian Process Surrogate Model.
We use a Gaussian process (GP), denoted by GP (0 , k ( · , · )) , as a prior for the unknown function f , and a Gaussian likelihood model for the noise variables ⌘ t , which are drawn from N (0 ,  ) and are independent across t .
Note that this GP surrogate model is used for algorithm design only; it does not change the fact that f is a ﬁxed function in H k and that the noise ⌘ t can be sub-Gaussian (i.e., an agnostic setting [6]).
Let [ t ] := { 1 , 2 , . . .
, t } .
Conditioned on a set of observations H t = { ( x s , r s ) , s 2 [ t ] } , by the properties of GP [36], the posterior distribution for f is GP ( µ t ( · ) , k t ( · , · )) , where
[Equation]
[Equation]
(4) in which k t ( x ) :=
[ k ( x 1 , x ) , . . .
, k ( x t , x )] T , K t :=
[ k ( x u , x v )] u,v 2 [ t ] , and R t is the (noisy) reward vector [ r 1 , r 2 , . . .
, r t ] T .
In particular, we also deﬁne  2 t ( x ) : =
k t ( x, x ) .
Let K A :=
[ k ( x, x 0 )] x,x 0 2 A for A ✓ X .
We deﬁne the maximum information gain as  t ( k, X ) :
= max A ✓ X : | A | = t 12 ln | I t +   1 K A | where I t is the t ⇥ t identity matrix.
The maximum information gain plays a key role in the regret bounds of GP-based algorithms.
While  t ( k, X ) depends on the kernel k and domain
X , we simply use  t whenever the context is clear.
For instance, if X is compact and convex with dimension d , then we have  t
= O ((ln t ) d +1 ) for squared exponential kernel k SE ,  t
=
O
( d ln t ) for linear kernel [6] and  t =
˜ O ( T d 2 ⌫ + d ) (where ⌫ is a hyperparameter) for Matérn kernel k Matérn [37].
Similarly, the learning agent also uses a GP surrogate model for g , i.e., a GP prior GP (0 , e k ( · , · )) and a Gaussian noise N (0 , e  ) .
Conditioned on a set of observations e
H t = { ( x s , c s ) , s 2 [ t ] } , the posterior distribution for g is GP ( e µ t ( · ) , e k t ( · , · )) , where e µ t and e k t are computed in the same way.
A Uniﬁed Framework for Constrained Kernelized Bandits
In this section, leveraging primal-dual optimization, we develop a uniﬁed framework for both algorithm design and performance analysis in constrained kernelized bandits.
In particular, we ﬁrst propose a “master” algorithm called CKB (constrained KB), which can be equipped with very general exploration strategies.
Then, we develop a novel sufﬁcient condition, which not only provides a uniﬁed analysis of regret and constraint violation, but also facilitates the design of new exploration strategies (and hence new CKB algorithms) with rigorous performance guarantees.
Algorithm.
We ﬁrst explain our “master” algorithm CKB in Algorithm 1, which is based on primaldual optimization.
Let the Lagrangian of the baseline problem max ⇡ { E ⇡ [ f ( x )] : E ⇡ [ g ( x )]  0 } be L ( ⇡ ,  ) :
=
E ⇡ [ f ( x )]   E ⇡ [ g ( x )] and the associated dual problem is deﬁned as D (  ) :
= max ⇡ L ( ⇡ ,  ) with the optimal dual variable being  ⇤ : = argmin   0 D (  ) .
Note that since both f and g are unknown, the agent has to ﬁrst generate estimates of them (i.e., f t and g t , respectively) based on exploration strategies A f and A g , which capture the tradeoff between exploration and exploitation (line 3).
Then, both estimates will be truncated according to the range of f and g , respectively (lines 4-5) (where Proj is the projection operator).
The truncation is necessary for our analysis, but it does not impact the regret bound since it will not lead to loss of useful information.
Then, lines 6-7 correspond to the primal optimization step that approximates D (  t ) (i.e., approximate
L by ¯ L with f and g replaced by ¯ f t and ¯ g t ).
The reason behind line 7 is that one of the optimal solutions for max ⇡ ¯ L ( ⇡ ,  t ) is simply argmax x ( ¯ f t ( x )   t ¯ g t ( x )) .
Then, line 8 is the dual update that minimizes D (  t ) with respect to  by taking a projected gradient step with 1 /V being the step size.
The parameter ⇢ is chosen to be larger than the optimal dual variable  ⇤ , and hence the projected interval [0 , ⇢ ] includes the optimal dual variable.
This can be achieved since the optimal dual variable is bounded under Slater’s condition, and in particular, we have  ⇤  ( E ⇡ ⇤ [ f ( x )]  E ⇡
[ f ( x )]) /  by [38, Theorem 8.42].
Finally, line 8 is the posterior update via standard GP regression for both f and g as computed in (3) and (4) with  2 t ( x ) =
k t ( x, x ) and e  2 t ( x ) =
e k t ( x, x ) .
Remark 1 (Computational complexity) .
CKB enjoys the same computational complexity as the standard unconstrained case (e.g., [5]) since the additional dual update is a simple projection and 1: Parameters: V , ⇢ ,  1 = 0 , µ 0 ( x ) = e µ 0 ( x ) = 0 ,  0 ( x ) =
e  0 ( x ) = 1 , 8 x , exploration strategies A f and A g 2: for batch t =1 , 2 , . . . do 3: Based on posterior models, generate f t and g t using A f and A g , respectively 4:
Truncate f t as ¯ f t ( x ) =
Proj [  B,B ] f t ( x ) 5:
Truncate g t as ¯ g t ( x ) =
Proj [  G,G ] g t ( x ) 6:
Pseudo-acquisition function: b z  t ( x ) =
¯ f t ( x )   t ¯ g t ( x ) 7:
Choose primal action x t = argmax x 2 X b z  t ( x ) ; observe r t and c t 8:
Update dual variable:  t
+1 =
Proj [0 , ⇢ ] ⇥  t + 1 V ¯ g t ( x t )
⇤ 9:
Posterior model: update ( µ t ,  t ) and ( e µ t , e  t ) via GP regression using new data ( x t , r t , c t ) 10: end for the primal optimization keeps the same ﬂavor as the unconstrained case, i.e., without constructing a speciﬁc safe set as in existing constrained KB algorithms designed for hard constraints.
We call CKB a “master” algorithm as it allows us to employ different exploration strategies (or called acquisition functions ) (i.e., A f and A g ).
Therefore, one fundamental question is: How to design efﬁcient exploration strategies such that favorable performance can be guaranteed?
In the following, we ﬁrst use UCB-type exploration as an example to gain useful insights, which in turn will facilitate the development of a novel sufﬁcient condition.
This condition not only is satisﬁed under very general exploration strategies, but also enables a uniﬁed analytical framework for showing both sublinear regret and sublinear constraint violation in constrained kernelized bandits.
We
Deﬁnition 1 (GP-UCB and GP-TS Explorations) .
Suppose the posterior distribution for a black-box function h in round t is given by GP ( b µ t  1 ( · ) , b k t  1 ( · , · )) and b  t is an increasing sequence.
(i)
The estimate of h in round t under GP-UCB exploration is h t ( · ) = b µ t  1 ( · ) + b  t b  t  1 ( · ) , where b  2 t  1 ( x ) :=
b k t  1 ( x, x ) for all x 2 X .
(ii)
The estimate of h in round t under GP-TS exploration is h t ( · ) ⇠ GP ( b µ t  1 ( · ) , b  2 t b k t  1 ( · , · )) .
3.1 Warm Up: CKB with GP-UCB Exploration
In this section, we instantiate CKB with GP-UCB exploration called CKB-UCB , as a warm-up.
In particular, in CKB-UCB , A f is a GP-UCB exploration (see Deﬁnition 1) with a positive b  t sequence (i.e., optimistic with respect to reward), and A g is a GP-UCB exploration with a negative b  t sequence (i.e., optimistic with respect to cost).
This instantiation enjoys the following performance guarantee.
Theorem 1.
Suppose ⇢  4 B/  , V = G p T/ ⇢ , A f is a GP-UCB exploration with b  t =
t := B + R p 2(  t  1 + 1 + ln(2 / ↵ )) , and A g is a GP-UCB exploration with b  t =  e  t :=  ( G +
R p 2( e  t  1 + 1 + ln(2 / ↵ ))) .
Under Slater’s condition in Assumption 1 and regularity assumptions in Assumption 2, CKB-UCB achieves the following bounds simultaneously with probability at least 1  ↵ for any ↵ 2 (0 , 1) :
[Equation]
where C := max { B, G } and [Equation]
Remark 2.
The (reward) regret here is the stronger version, i.e., R + ( T ) .
Compared to the unconstrained case, the regret bound has an additional term ⇢ G p T , which roughly captures the impact of the constraint.
As in the unconstrained case, one can plug in different  T and e  T to see that both regret and constraint violation are sublinear for commonly used kernels.
For example, for an SE kernel, both  T and e  T are on the order of (ln T )
d +1 .
Finally, the standard “doubling trick” can be used to design an anytime algorithm (i.e., without the knowledge of T ) with regret and constraint violation bounds of the same order.
Proof Sketch of Theorem 1.
The key step is to establish a sublinear bound on the term R + ( T ) +
P Tt =1 g ( x t ) for any  2 [0 , ⇢ ] .
Then, with convex optimization tools, one can establish regret and constraint violation bound.
To this end, By the dual update of CKB, we can ﬁrst show that
R + [Equation]
2 , where
[Equation]
Thus, the analysis for each choice of exploration strategy only differ in how to bound T 1 + T 2 .
Under GP-UCB exploration, the term T 1  0 by the optimism while T 2 can be upper bounded by T
[Equation]
[Equation]
., standard predication error bound.
Building on the bound on T 1 + T 2 , one can then show the required results.
3.2 A Sufﬁcient Condition for Provably Efﬁcient Explorations
The above analysis reveals that the key step in obtaining sublinear performance guarantees of
Algorithm 1 is to ﬁnd a sublinear bound on T 1 + T 2 , which depends on the choice of exploration strategies.
To go beyond GP-UCB exploration, we will establish a sufﬁcient condition on general exploration strategies (i.e., A f and A g ), which guarantees a sublinear bound on T 1 + T 2 and hence sublinear regret and sublinear constraint violation.
We ﬁrst present the intuition behind the key components of our sufﬁcient condition.
Inspired by [39], it sufﬁces to focus on the following three nice events so as to bound T 1 + T 2 in (5)-(6):
E est
[Equation]
x,
[Equation] [Equation]
x,
[Equation] [Equation]
x, t ) } , [Equation]
[Equation]
x
[Equation]
[Equation] [Equation] [Equation]
[Equation]
E antig,t , in which
[Equation]
To see the intuition, ﬁrst suppose that events E est and E conct hold with high probability.
Then, it is easy to see that the estimates are close to the true functions, and hence, one can derive a bound on Tin (6).
Now, suppose that events E est and E antit hold with some positive probability.
Then, one can see that the estimates are optimistic with respect to the true functions when evaluated at the optimal points.
This probabilistic optimism is the key to bounding T 1 in (5).
Note that GP-UCB exploration is optimistic with probability one by deﬁnition (see Deﬁnition 1), and hence, T 1  0 always holds.
To formally state our result, let us deﬁne the ﬁltration F t as all the history up to the end of round t .
Let E t [ · ] := E [ ·|F t  1 ] and P t ( · ) :
= P [ ·|F t  1 ] .
Then, we have the following sufﬁcient condition.
Assumption 3 (Sufﬁcient Condition) .
The sufﬁcient condition includes two parts: (1) (Probability condition) P ( E est )  1  p 1 , P t ( E conct )  1  p 2 ,t , and P t  E antit   p 3 > 0 for some time-dependent sequences c (1) f,t , c (1) g,t , c (2) f,t , and c (2) g,t .
(2) (Boundedness condition)
(i)
There exists a positive probability p 4 such that 1 + 2 ( p 3  p 2 ,t )  1 /pfor all t ; (ii) There exist some functions of T : c f ( T ) , c g ( T ) such that c (1) f,t + c (2) f,t  c f ( T ) and c (1) g,t + c (2) g,t  c g ( T ) for all t ; (iii) P
Tt =1 p 2 ,t  C 0 for some constant C 0 .
Remark 3.
The above sufﬁcient condition generalizes existing similar results [39–41] in several aspects.
First, existing works mainly focus on the MAB or linear bandit settings, which are special cases of our KB setting (e.g., choosing a linear kernel leads to linear bandit).
Second, while existing works only establish bounds on the expected regret, we aim to establish a high-probability bound.
As a result, we need the additional boundedness condition, which, however, is simply for technical reasons.
Third, in contrast to existing works that consider the unconstrained case only, we consider the constrained case, which is more challenging.
Speciﬁcally, it requires that E antit holds under policy ⇡ ⇤ (i.e., the expectation over ⇡ ⇤ ) rather than under a single optimal action x ⇤ .
With the above sufﬁcient condition, we have the following general performance bounds for CKB.
Theorem 2.
Suppose ⇢  4 B/  and V = G p T/ ⇢ .
Let  := B + ⇢ G .
Assume that CKB is equipped with exploration strategies that satisfy the sufﬁcient condition in Assumption 3.
Then, under Slater’s condition in Assumption 1 and regularity assumptions in Assumption 2, CKB achieves the following bounds on regret and constraint violation with probability at least 1  ↵  p 1 for any ↵ 2 (0 , 1) :
[Equation]
In the following, we will show that Theorem 2 provides a uniﬁed view of the regret and constraint violation performance for various CKB algorithms, thanks to our sufﬁcient condition.
In particular, we ﬁrst show that existing exploration strategies, such as GP-UCB and GP-TS, satisfy our sufﬁcient condition, given by the following corollary.
Corollary 1.
Let  t =
B + R p 2(  t  1 + 1 + ln(1 / ↵ f [Equation] and e
[Equation] ln [Equation]
(i).
GP-UCB with
b
t =
t and b  t =  e  t for A f and A g , respectively, satisﬁes
the sufﬁcient condition.
(ii).
GP-TS with [Equation]  t and b  t =  e  t for A f and A g , respectively, satisﬁes the sufﬁcient condition when ⇡ ⇤ is a deterministic policy.
The sufﬁcient condition also enables us to design CKB algorithms with new exploration strategies.
In the following, inspired by [40], we propose a new GP-based exploration strategy, which aims to strike a balance between GP-UCB and GP-TS explorations.
Deﬁnition 2 (RandGP-UCB Exploration) .
Suppose that the posterior distribution for a black-box function h in round t is given by GP ( [Equation] µ t  [Equation]
[Equation]
[Equation] k t  [Equation] [Equation]
[Equation]
Then, the estimate of h in round t under
RandGP-UCB exploration strategy is h [Equation] = [Equation]
where b Z t ⇠ [Equation] D for some distribution
[Equation] In contrast to GP-UCB, RandGP-UCB replaces the deterministic conﬁdence bound by a randomized one.
Compared to GP-TS, RandGP-UCB uses “coupled” noise in the sense that all the actions share the same noise b Z t rather than “decoupled” and correlated noise in GP-TS.
This subtle difference will not only help to eliminate the additional factor p ln( |X| ) in GP-TS due to the use of union bound, but also allow us to deal with a general stochastic ⇡ ⇤ .
Corollary 2.
Let  t , e  t be the same as in Corollary 1.
RandGP-UCB with b D = N (0 ,  2 t ) and b
D = N (0 , e  2 t ) for A f and A g , respectively, satisﬁes the sufﬁcient condition.
Thus, one can instantiate CKB with RandGP-UCB exploration to obtain a new algorithm called
CKB-Rand with performance guarantees given by Theorem 2.
Note that RandGP-UCB with other distributions b D can also satisfy the sufﬁcient condition (as discussed in Appendix C).
Discussions
In this section, we discuss several possible questions one may have at this moment about our CKB algorithm.
The ﬁrst natural question to ask is whether one can further improve the constraint violation bound.
It turns out that with a minor modiﬁcation of CKB algorithm, one can achieve a bounded and even zero constraint violation by trading the regret slightly (but still the same order as before).
The modiﬁcation is to introduce a slackness given by " in the dual update in Algorithm 1, i.e.,  t +1 =
Proj [0 , ⇢ ]
[Equation] ⇤ with "   / 2 .
Intuitively speaking, this can be viewed as if one is working on a new pessimistic constraint function.
After obtaining the constraint violation under this new hypothetic constraint, one can subtract " T to ﬁnd the true constraint violation under the true function g .
The catch here is that one needs also change the baseline problem to the following one: max ⇡ { E ⇡ [ f ( x )] : E ⇡ [ g ( x )] + "  0 } so that it matches the new pessimistic constraint function.
Let ⇡ ⇤ " be the optimal solution to this new problem and the obtained regret is only with respect to ⇡ ⇤ " rather than the original ⇡ ⇤ .
Thus, we need to further bound the following difference
T E ⇡ ⇤ [ f ( x )]  T E ⇡ ⇤ "
[ f ( x )] to obtain the true regret bound, which is given in Appendix E.
[Figure]
Another question one may ask is why we are able to go beyond UCB-type exploration compared to existing works (e.g., [13–15]) that only establish performance guarantees under UCB.
The answer is that they resort to another popular method for constrained optimization based on the so-called
Lyapunov-drift argument.
However, this method relies heavily on the optimism under UCB to derive both regret and constraint violation bounds.
Thus, it is difﬁcult to generalize it for other exploration strategies even for simpler linear bandit problems.
In contrast, our analysis relies only on the bound for T 1 + T 2 , which allows us to consider more general exploration strategies.
In Appendix F, we also discuss how to apply the Lyapunov-drift based method to analyze UCB exploration in our setting, through which we highlight more differences and some subtleties in Lyapunov-drift method
[Figure]
Simulation Results
Although our work is mainly a theoretical study
[Figure]
we conduct simulations to compare the performance of our algorithms (i.e., CKB-UCB , CKB-TS , and CKB-Rand , that is, CKB with GP-UCB, GPTS, and RandGP-UCB explorations, respectively) with existing safe KB algorithms based on both synthetic and real-world datasets.
In particular, we consider the two most recent safe KB algorithms:
StageOpt [28] (which has a superior performance compared to SafeOpt [27]) and SGP-UCB [3].
Our goal is to show that our proposed CKB algorithms can trade a slight performance in constraint violation (i.e., soft constraint) for improvement in the reward regret, computation complexity and ﬂexible implementations, i.e., a better complexity-regret-constraint trade-off.
5.1 Synthetic Data and Light-Tailed Real-World Data
Synthetic Data.
The domain X is generated by discretizing [0 , 1] uniformly into 100 points.
The objective function f ( · ) = [Equation] pi =1 a i k ( · , x i ) is generated by uniformly sampling a i 2 [  1 , 1] and support points x i 2 X with p = 100 .
With the same manner, we generate the constraint function g .
The kernel is k se with parameter l = 0 .
Other parameters include B , R and  t are set similar as in the unconstrained case (e.g., [5]).
Light-Tailed Real-World Data.
We use the light sensor data collected in the CMU Intelligent
Workplace in Nov 2005, which is available online as Matlab structure 3 and contains locations ofsensors, 601 train samples and 192 test samples.
We use it in the context of ﬁnding the maximum average reading of the sensors.
In particular, f is set as empirical average of the test samples, with B set as its maximum, and k is set as the empirical covariance of the normalized train samples.
The constraint is given by g ( · ) =  f ( · )+ h with h = B/ 2 .
We perform 50 trials (each with T = 10 , 000 ) and plot the mean of the cumulative regret along with the error bars, as shown in Fig.
Regret.
Our three CKB algorithms achieve a better (or similar) regret performance compared to the existing safe BO algorithms (see Figures 1(a) and 1(b)).
Among the three CKB algorithms,
CKB-Rand appears to have reasonably good performance at all times.
Constraint violation.
Since we have V ( T ) = 0 under all the algorithms, we study the total number of rounds where the constraint is violated, denoted by N .
In the synthetic data setting, our proposed
CKB algorithms have N  5 over T = 10 , 000 rounds; in the real-world data setting, CKB-UCB enjoys N = 0 and CKB-Rand has an average N = 38 over a horizon T = 1 , 000 .
Furthermore, we plot the stronger cumulative constraint violations given by
P
Tt =
[ g ( x t )] + as shown in Figure 1(c),
[Figure]
from which we can see that all CKB algorithms achieve sublinear performance even with respect to this stronger metric, i.e., violation cancellation across rounds is not allowed.
Practical considerations.
Our proposed CKB algorithms have the same computational complexity as the unconstrained case.
In particular, they scale linearly with the number of actions in the discretedomain case 4 .
On the other hand, StageOpt scales quadratically due to the construction of the safe set, and SGP-UCB requires the additional random initialization stage, which leads to linear regret at the beginning of the learning process.
Moreover, standard methods for improving the scalability of unconstrained KB can be naturally applied to our CKB algorithms [42].
Finally, both StageOpt and
SGP-UCB require the knowledge of a safe action (i.e., one that satisﬁes the constraint) in advance, and moreover, StageOpt requires f to be Lipschitz and needs to estimate the Lipschitz constant, which impacts the robustness.
In contrast, CKB algorithms for soft constraints only require a mild
Slater’s condition as in Assumption 1, which does not require the existence of a safe action.
5.2 Heavy-Tailed Real-World Data
We further compare different constrained KB algorithms in a new real-world dataset, which demonstrates a heavy-tailed noise.
Note that sub-Gaussian noise is required in all the existing theoretical works (including our work).
We use this dataset to test the robustness of various constrained KB algorithms.
The experimental results tend to show that our three CKB algorithms are more robust in terms of heavy-tailed noise, which is common in practical applications.
The detail of this real-world dataset is deferred to Appendix D.
Regret.
We plot both cumulative regret and time-average regret in this setting (see Figures 2 (a) and (b)).
We can observe that in the presence of heavy-tailed noise, our three CKB algorithms have signiﬁcant performance gain over existing safe KB algorithms.
Constraint violation.
We focus on the strong metric, i.e., the number of rounds where the constraint is violated, denoted by N .
We have that CKB-Rand enjoys an average N = 21 and CKB-UCB has an average N = 47 within the horizon of T = 10 , 000 .
We also plot stronger cumulative constraint violations given by
P
Tt =
[ g ( x t )] + as shown in Figure 2(c), from which we can see that all CKB algorithms achieve sublinear performance even with respect to this stronger metric.
Conclusion
We presented a general framework for constrained KB with soft constraints via primal-dual optimization.
Armed with our developed sufﬁcient condition, this framework not only allows us to design provably efﬁcient (i.e., sublinear reward regret and sublinear total constraint violation) CKB algorithms with both UCB and TS explorations, but presents a uniﬁed method to design new effective ones.
By introducing slackness, our algorithm can also attain a bounded or even zero constraint violation while still achieving a sublinear regret.
We further perform simulations on both synthetic data and real-world data that corroborate our theoretical results.
Along the way, we also present the ﬁrst detailed discussion on two existing methods for analyzing constrained bandits and MDPs by highlighting interesting insights.
For future work, one interesting direction is to build upon recent advances in unconstrained KB that attain an improved regret (e.g., [43–46]) to study the corresponding constrained case in the hope to maintain the same improvement.
Acknowledgement
We would like to thank the anonymous reviewers for their helpful comments.
XZ is partially supported by NSF grant CNS-2153220.
BJ is partially supported by NSF grant CNS-2112694.
[-C ′ and p 
-]
[-π 2 / 3 and p 
-]