On Kernelized Multi-Armed Bandits with Constraints
Anonymous Author(s)
AffiliationAddress email
Abstract
We study a stochastic bandit problem with a general unknown reward function anda general unknown constraint function.
Both functions can be non-linear (evennon-convex) and are assumed to lie in a reproducing kernel
Hilbert space (RKHS)with a bounded norm.
In contrast to safety-type hard constraints studied in priorworks, we consider soft constraints that may be violated in any round as long asthe cumulative violations are small.
Our ultimate goal is to study how to utilize thenature of soft constraints to attain a finer complexity-regret-constraint trade-off inthe kernelized bandit setting.
To this end, leveraging primal-dual optimization, wepropose a general framework for both algorithm design and performance analysis.
This framework builds upon a novel sufficient condition, which not only is satisfiedunder general exploration strategies, including upper confidence bound (UCB),
Thompson sampling (TS), and new ones based on random exploration , but alsoenables a unified analysis for showing both sublinear regret and sublinear or evenzero constraint violation.
We demonstrate the superior performance of our proposedalgorithms via numerical experiments based on both synthetic and real-worlddatasets.
Along the way, we also make the first detailed comparison between twopopular methods for analyzing constrained bandits and Markov decision processes(MDPs) by discussing the key difference and some subtleties in the analysis, whichcould be of independent interest to the communities.
Introduction
Stochastic bandit optimization of an unknown function f has recently gained increasing popularity dueto its widespread real-life applications such as recommendations [1], cloud resource configurations [2],and wireless power control [3].
At each time t , an action x t is selected and then a (noisy) banditreward feedback y t is observed.
The goal is to maximize the cumulative reward or equivalentlyminimize the total regret due to not choosing the optimal action in hindsight.
To capture a genericfunction, researchers have turned to nonparametric models of f via Gaussian process or reproducingkernel Hilbert space (RKHS), which are able to uniformly approximate an arbitrary continuousfunction over a compact set [4].
In this paper, as in [5, 6], we consider the agnostic setting (i.e.,frequentist-type), where f is assumed to be a fixed function in an RKHS with a bounded norm (i.e., ameasure of smoothness).
We call this setting frequentist-type kernelized bandits (KB).
In addition to a generic non-linear or even non-convex function f , another common feature inpractical applications is that there often exist additional constraints in the decision-making processsuch as hard-constraint like safety or soft-constraint like cost.
To this end, there have been excitingrecent advances in the theoretical analysis of constrained kernelized bandits.
In particular, [7–9]propose algorithms with convergence guarantees, while [10], to the best our knowledge, is the firstwork that establishes regret bounds for their developed algorithm, although under the Bayesian-type setting.
These algorithms mainly focus on KB with a hard constraint such as safety, i.e., the selectedaction in each round needs to satisfy the constraint with a high probability.
Hence, compared to theunconstrained case, additional computation is required to construct a safe action set in each round,which not only incurs additional complexity burdens, but often leads to conservative performance.
Motivations.
In practice, there are also many applications that involve soft constraints that may beviolated in any round.
The goal is to maximize the total reward while minimizing the total constraintviolations.
To give a concrete example, let us consider resource configuration in cloud computingplatforms where the objective is to minimize the cost while guaranteeing that the latency is below athreshold, e.g., 95% percentile latency. In this case, the latency of a job could be above the thresholdas long as the fraction of violations is small, e.g., less than 5% . Another example is throughputmaximization under energy constraints in wireless communications where energy consumptionconstraint is often a soft cumulative one.
In both examples, one fundamental question is whether thenature of soft constraints can be utilized to design constrained KB (CKB) algorithms with the samecomplexity as the unconstrained case while attaining a better reward performance compared to thehard constraints.
Furthermore, existing provably efficient algorithms [7–10] largely build on upperconfidence bound (UCB) exploration, which often has inferior empirical performance comparedto Thompson sampling (TS) exploration.
Hence, another key question is whether one can designprovably efficient CKB algorithms with general explorations.
In summary, the following fundamentaltheoretical question remains open:
Can a finer complexity-regret-constraint trade-off be attained in CKB under general explorations?
Contributions.
In this paper, we take a systematic approach to affirmatively answer the above fundamental question.
In particular, we tackle the complexity-regret-constraint trade-off by formulating
KB under soft constraints as a stochastic bandit problem where the objective is to maximize thecumulative reward while minimizing the cumulative constraint violations and maintaining the samecomputation complexity as in the unconstrained case.
Our detailed contributions are as follows.
• We develop a unified framework for CKB based on primal-dual optimization, which canguarantee both sublinear reward regret and sublinear total constraint violation under a classof general exploration strategies, including UCB, TS, and new effective ones (e.g., randomexploration) under the same complexity as the unconstrained case.
We also show that byintroducing slackness in the dual update, one can trade regret to achieve bounded or evenzero constraint violation.
This framework builds upon a novel sufficient condition, whichnot only facilitates the design of new CKB algorithms but provides a unified view in the
performance analysis.•
We demonstrate the superior performance of our proposed algorithms via numerical experiments based on both synthetic and real-world data.
In addition, we discuss the benefits of ouralgorithms in terms of various practical considerations such as low complexity, scalability,robustness, and flexibility.• We also provide the first detailed comparison between two popular methods for analyzingconstrained bandits in general.
Specifically, the first one is based on convex optimizationtool as in [11, 12], which is also the inspiration for our paper.
The other one is based onLyapunov-drift argument as in [13–15].
We discuss the key difference in terms of regret andconstraint violation analysis in these two methods and highlight the subtlety in applying astandard queueing technique (i.e., Hajek lemma [16]) to bound the constraint violation inthe second method.
We believe this provides a clear picture on the methodology, which is ofindependent interest to the communities.
1.1 Related Work
In the special cases of KB, including multi-armed bandits (MAB) and linear bandits (e g., KB with alinear kernel), there is a large body of work on bandits with different types of constraints, includingknapsack bandits [17–19], conservative bandits [20–22], bandits with fairness constraints [23, 24],bandits with hard safety constraints [25–27] and bandits with cumulative soft constraints [28,13].
Among them, the bandit setting with cumulative soft constraints is the closest to ours.
Inparticular, [13] considers linear bandits under UCB exploration and a constraint is to it handle general
Broadly speaking, our work is also related to reinforcement learning (RL) with soft constraints, i.e.,constrained MDPs.
In particular, our analysis is inspired by those on constrained MDPs [11, 12](which is another popular method to handle constrained bandits and MDPs via convex optimizationtools), but has significant differences.
First, in those works, the constraint violation is (cid:101) O ( √ T ) .
Incontrast, ours can attain bounded and even zero constraint violations by introducing the slacknessin the dual update.
Second, they only consider UCB-type exploration, but our algorithms canbe equipped with various exploration strategies (including UCB), thanks to our general sufficientcondition.
Third, they focus on either tabular or linear function approximation settings.
In contrast,both objective and constraint functions we consider can be nonlinear .
There are also recent workson constrained MDPs that claim to achieve bounded or zero constraint violation [14, 15] based onthe Lyapunov-drift method.
However, as in the bandit case, it is unclear how to generalize it tohandle general explorations beyond UCB.
Finally, the theoretical understanding of unconstrained
KB (a.k.a.
Gaussian process bandit) is well-established, including regret upper bounds under UCBand TS [5, 29, 6] and lower bounds [30, 31].
We remark that our work is mainly a theory-guidedstudy.
In a more practical area of KB, i.e., Bayesian optimization (BO), there have been many BOalgorithms developed for the constrained setting; see [32–34] and the references therein.
Althoughthese algorithms enjoy good performance in various practical settings, their theoretical performanceguarantees are still unclear.
Problem Formulation and Preliminaries
We consider a stochastic bandit optimization problem with soft constraints.
In particular, in eachround t ∈ { 1 , 2 , . .
, T } , a learning agent chooses an action x t ∈ X and receives a bandit rewardfeedback r t = f ( x t ) + η t , where η t is a zero-mean noise.
The learning agent also observes a banditconstraint feedback c t = g ( x t ) + ξ t , where ξ t is a zero-mean noise.
By soft constraints, the goal hereis to maximize the cumulative reward while minimizing the cumulative total violation.
Learning Problem.
Define cumulative regret and constraint violation as follows:
[Equation]
where x ∗ := argmax { x ∈X : g ( x ) ≤ 0 } f ( x ) and [ · ] + := max {· , 0 } .
The goal of the learning agentis to achieve both sublinear regret and sublinear constraint violation.
In fact, we will establishbounds on the following stronger version of regret.
Specifically, let π be a probability distributionover the set of actions
X (i.e., stochastic policy), and let E π [ f ( x )] := (cid:82) x ∈X f ( x ) π ( x ) dx and
E π [ g ( x )] := (cid:82) x ∈X g ( x ) π ( x ) dx .
We compare our achieved reward with the following baselineoptimization problem: max π { E π [ f ( x )] : E π [ g ( x )] ≤ 0 } where both f and g are known, and π ∗ isits optimal solution.
Now, our stronger regret is defined as
[Equation]
Clearly, we have R ( T ) ≤ R + ( T ) 2 .
Throughout the paper, we assume the following commonly usedcondition in constrained optimization literature; see also [13, 35, 11].
Assumption 1 (Slater’s condition) .
There is a constant δ > 0 such that there exists a probabilitydistribution π 0 that satisfies E π
[ g ( x )] ≤ − δ .
Without loss of generality, we assume δ ≤ 1 .
This is a quite mild assumption since it only requires that one can find a stochastic policy underwhich the expected cost is less than a strictly negative value.
This is in sharp constraint to existing
KB algorithms for hard constraints that typically require the existence of an initial safe action [9, 10].
In this paper, we consider the frequentist-type regularity assumption that is typically used in uncontrained KB works (e.g., [5, 6]).
Specifically, we assume that f is a fixed function in an RKHS with abounded norm.
In particular, the RKHS for f is denoted by H k , which is completely determined bythe corresponding kernel function k : X × X → R .
Any function h ∈ H k satisfies the reproducingproperty : h ( x ) = ⟨ h, k ( · , x ) ⟩ H k , where ⟨· , ·⟩ H k is the inner product defined on H k .
Similarly, forthe unknown constraint function g , we assume that g is a fixed function in the RKHS defined by a kernel function (cid:101) k , and the RKHS for g is denoted by H (cid:101) k .
We assume that the following boundednessproperty holds throughout the paper.
Assumption 2 (Boundedness) .
We assume that ∥ f ∥ H k ≤ B and k ( x, x ) ≤ 1 for any x ∈ X andthat the noise η t is i.i.d.
R -sub-Gaussian.
Similarly, we assume that ∥ g ∥ H (cid:101) k ≤ G and (cid:101) k ( x, x ) ≤ 1 forany x ∈ X and that the noise ξ t is i.i.d.
(cid:101) R -sub-Gaussian.
Gaussian Process Surrogate Model.
We use a Gaussian process (GP), denoted by GP (0 , k ( · , · )) , asa prior for the unknown function f , and a Gaussian likelihood model for the noise variables η t , whichare drawn from N (0 , λ ) and are independent across t .
Note that this GP surrogate model is used foralgorithm design only; it does not change the fact that f is a fixed function in H k and that the noiseη t can be sub-Gaussian (i.e., an agnostic setting [6]).
Let [ t ] := { 1 , 2 , . . .
, t } .
Conditioned on a setof observations H t
= { ( x s , r s ) , s ∈ [ t ] } , by the properties of GP [36], the posterior distribution forf is GP ( µ t ( · ) , k t ( · , · )) , where
[Equation]
in which k t ( x ) :=
[ k ( x 1 , x ) , . . .
, k ( x t , x )] T , K t [Equation] =
[Equation] [Equation] [Equation] ] , and R t is the (noisy) rewardvector [ r 1 , r 2 , . . .
, r t ] T .
In particular, we also define σ 2 t ( x ) :=
k t ( x, x ) .
Let K A :=
[ k ( x, x ′ )] x,x ′ ∈ Afor A ⊆ X .
We define the maximum information gain as γ t ( k, X ) :
= max A ⊆X : | A | = t 12 ln | I t +λ − 1 K A | where I t is the t × t identity matrix.
The maximum information gain plays a key role in theregret bounds of GP-based algorithms.
While γ t ( k, X ) depends on the kernel k and domain X , wesimply use γ t whenever the context is clear.
For instance, if X is compact and convex with dimensiond , then we have γ t =
O ((ln t ) d +1 ) for squared exponential kernel k SE , γ t
=
O
( t d ( d +1) 2 ν + d ( d +1) ln t )(where ν is a hyperparameter) for Matérn kernel k Matérn , and γ t = O ( d ln t ) for linear kernel [6].
Similarly, the learning agent also uses a GP surrogate model for g , i.e., a GP prior GP (0 , (cid:101) k and a Gaussian noise N (0 , (cid:101) λ ) .
Conditioned on a set of observations (cid:101) H t
= { ( x s , c s ) , s ∈ [ t ] } , theposterior distribution for g is GP ( (cid:101) µ t ( · ) , (cid:101) k t ( · , · )) , where (cid:101) µ t and (cid:101) k t are computed in the same way.
A Unified Framework for Constrained Kernelized Bandits
In this section, leveraging primal-dual optimization, we develop a unified framework for bothalgorithm design and performance analysis in constrained kernelized bandits.
In particular, we firstpropose a “master” algorithm called CKB (constrained KB), which can be equipped with very generalexploration strategies.
Then, we develop a novel sufficient condition, which not only provides aunified analysis of regret and constraint violation, but also facilitates the design of new explorationstrategies (and hence new CKB algorithms) with rigorous performance guarantees.
Algorithm.
We first explain our “master” algorithm CKB in Algorithm 1, which is based on primaldual optimization.
Let the Lagrangian of the baseline problem max π { E π [ f ( x )] : E π [ g ( x )] ≤ 0 }be L ( π, ϕ ) :
= E π
[ f ( x )] − ϕ E π [ g ( x )] and the associated dual problem is defined as D ( ϕ ) :
=max π L ( π, ϕ ) with the optimal dual variable being ϕ ∗ := argmin ϕ ≥ 0 D ( ϕ ) .
Note that since both fand g are unknown, the agent has to first generate estimates of them (i.e., f t and g t , respectively)based on exploration strategies
A f and A g , which capture the tradeoff between exploration andexploitation (line 3).
Then, both estimates will be truncated according to the range of f and g ,respectively (lines 4-5) (where Proj is the projection operator).
The truncation is necessary for ouranalysis, but it does not impact the regret bound since it will not lead to loss of useful information.
Then, lines 6-7 correspond to the primal optimization step that approximates D ( ϕ t ) (i.e., approximate
L by ¯ L with f and g replaced by ¯ f t and ¯ g t ).
The reason behind line 7 is that one of the optimalsolutions for max π ¯ L ( π, ϕ t ) is simply argmax x ( ¯ f t ( x ) − ϕ t ¯ g t ( x )) .
Then, line 8 is the dual updatethat minimizes D ( ϕ t ) with respect to ϕ by taking a projected gradient step with 1 /V being the stepsize.
The parameter ρ is chosen to be larger than the optimal dual variable ϕ ∗ , and hence the projectedinterval [0 , ρ ] includes the optimal dual variable.
This can be achieved since the optimal dual variableis bounded under Slater’s condition, and in particular, we have ϕ ∗ ≤ ( E π ∗ [ f ( x )] − E π
[ f ( x )]) /δby
[37, Theorem 8.42].
Finally, line 8 is the posterior update via standard GP regression for both fand g as computed in (2) and (3) with σ 2 t ( x ) =
k t ( x, x ) and (cid:101) σ 2 t ( x ) = (cid:101) k t
( x, x ) .
Remark 1 (Computational complexity) .
CKB enjoys the same computational complexity as thestandard unconstrained case (e.g., [5]) since the additional dual update is a simple projection and the primal optimization keeps the same flavor as the unconstrained case, i.e., without constructing aspecific safe set as in existing constrained KB algorithms designed for hard constraints.
We call CKB a “master” algorithm as it allows us to employ different exploration strategies (or calledacquisition functions ) (i.e., A f and A g ).
Therefore, one fundamental question is: How to designefficient exploration strategies such that favorable performance can be guaranteed?
In the following,we first use UCB-type exploration as an example to gain useful insights, which in turn will facilitatethe development of a novel sufficient condition.
This condition not only is satisfied under very generalexploration strategies, but also enables a unified analytical framework for showing both sublinearregret and sublinear constraint violation in constrained kernelized bandits.
We first introduce standard UCB and TS explorations under GP as in [5].
Definition 1 (GP-UCB and GP-TS Explorations) .
Suppose the posterior distribution for a black-boxfunction h in round t is given by GP ( (cid:98) µ t − 1 ( · ) , (cid:98) k t − 1 ( · , · )) and (cid:98) β t is an increasing sequence
(i)
The estimate of h in round t under GP-UCB exploration is h t ( · ) = (cid:98) µ t − 1 ( · ) + (cid:98) β t (cid:98) σ t − 1 ( · ) , where(cid:98)
σ 2 t − 1 ( x ) := (cid:98) k t − 1 ( x, x ) for all x ∈ X .
(ii)
The estimate of h in round t under GP-TS exploration is h t ( · ) ∼ GP ( (cid:98) µ t − 1 ( · ) , (cid:98) β 2 t (cid:98) k t − 1 ( · , · )) .
3.1 Warm Up: CKB with GP-UCB Exploration
In this section, we instantiate CKB with GP-UCB exploration called CKB-UCB , as a warm-up.
Inparticular, in CKB-UCB , A f is a GP-UCB exploration (see Definition 1) with a positive (cid:98) β t sequence(i.e.
, optimistic with respect to reward), and A g is a GP-UCB exploration with a negative (cid:98) β t sequence(i.e.
, optimistic with respect to cost).
This instantiation enjoys the following performance guarantee.
Theorem 1.
Suppose ρ ≥ 4 B/δ , V = G √ T/ρ , A f is a GP-UCB exploration with (cid:98) β t = β t :=
B + R (cid:112) 2( γ t − 1 + 1 + ln(2 /α )) , and A g is a GP-UCB exploration with (cid:98) β t = − (cid:101) β t := − ( G +
R (cid:112) 2( (cid:101) γ t − 1 + 1 + ln(2 /α ))) .
Under Slater’s condition in Assumption 1 and regularity assumptionsin Assumption 2, CKB-UCB achieves the following bounds simultaneously with probability at least1 − α for any α ∈ (0 , 1) :
[Equation]
where C := max { B, G } and (cid:98) γ T := max { γ T , (cid:101) γ T } .
Remark 2.
The (reward) regret here is the stronger version, i.e., R + ( T ) .
Compared to the unconstrained case, the regret bound has an additional term ρG √ T , which roughly captures the impact ofthe constraint.
As in the unconstrained case, one can plug in different γ T and (cid:101) γ T to see that bothregret and constraint violation are sublinear for commonly used kernels.
For example, for an SEkernel, both γ T and (cid:101) γ T are on the order of (ln T )
d +1 .
Finally, the standard “doubling trick” can beused to design an anytime algorithm (i.e., without the knowledge of T ) with regret and constraintviolation bounds of the same order.
Proof Sketch of Theorem 1.
The key step is to establish a sublinear bound on the term R + ( T ) +ϕ (cid:80) Tt
=1 g ( x t ) for any ϕ ∈ [0 , ρ ] .
Then, with convex optimization tools, one can establish regretand constraint violation bound.
To this end, By the dual update of CKB, we can first show that
R + ( T ) + ϕ (cid:80)
Tt
=1 g ( x t ) ≤ T 1 + T 2 + V 2 ϕ 2 + 12 V TG 2 , where
[Equation]
Thus, the analysis for each choice of exploration strategy only differ in how to bound T 1 + T 2 .
Under GP-UCB exploration, the term T 1 ≤ 0 by the optimism while T 2 can be upper boundedby T 2 ≤ 2 β T (cid:80) Tt =1 σ t − 1 ( x t ) + 2 ϕ (cid:101) β T
[Equation]
Tt =1 (cid:101) σ t − 1 ( x t ) = O ( β [Equation] √ γ
[Equation]
T + ϕ [Equation] β T (cid:112)(cid:101) γ T T ) , i.e., standardpredication error bound.
Building on the bound on T 1 + T 2 , one can then show the required results.
3.2 A Sufficient Condition for Provably Efficient Explorations
The above analysis reveals that the key step in obtaining sublinear performance guarantees of
Algorithm 1 is to find a sublinear bound on T 1 + T 2 , which depends on the choice of explorationstrategies.
To go beyond GP-UCB exploration, we will establish a sufficient condition on generalexploration strategies (i.e., A f and A g ), which guarantees a sublinear bound on T 1 + T 2 and hencesublinear regret and sublinear constraint violation.
We first present the intuition behind the key components of our sufficient condition.
Inspired by [38],it suffices to focus on the following three nice events so as to bound T 1 + T 2 in (4)-(5):
E est
[Equation]
[Equation]
[Equation]
:=
[Equation]
E antig,t , in which
[Equation]
To see the intuition, first suppose that events E est and E conct hold with high probability.
Then, it iseasy to see that the estimates are close to the true functions, and hence, one can derive a bound on T 2in (5).
Now, suppose that events E est and E antit hold with some positive probability.
Then, one cansee that the estimates are optimistic with respect to the true functions when evaluated at the optimalpoints.
This probabilistic optimism is the key to bounding T 1 in (4).
Note that GP-UCB explorationis optimistic with probability one by definition (see Definition 1), and hence, T 1 ≤ 0 always holds.
To formally state our result, let us define the filtration F t as all the history up to the end of round t .
Let E t [ · ] := E [ ·|F t − 1 ] and P t ( · ) :
= P [ ·|F t − 1 ] .
Then, we have the following sufficient condition.
Assumption 3 (Sufficient Condition) .
The sufficient condition includes two parts:(1)
(Probability condition) P ( E est ) ≥ 1 − p 1 , P t ( E conct ) ≥ 1 − p 2 ,t , and P t (cid:0) E antit (cid:1) ≥ p 3 > 0 forsome time-dependent sequences c (1) f,t , c (1) g,t , c (2) f,t , and c (2) g,t .(2)
(Boundedness condition)
(i)
There exists a positive probability p 4 such that 1 + 2 ( p 3 − p 2 ,t ) ≤ 1 /p 4for all t ; (ii) c (1) f,t + c (2) f,t ≤ c f ( T ) and c (1) g,t + c (2) g,t ≤ c g ( T ) for all t ; (iii) (cid:80) Tt =1 p 2 ,t ≤ C ′ for someconstant C ′ .
Remark 3.
The above sufficient condition generalizes existing similar results [38–40] in severalaspects.
First, existing works mainly focus on the MAB or linear bandit settings, which are specialcases of our KB setting (e.g., choosing a linear kernel leads to linear bandit).
Second, while existingworks only establish bounds on the expected regret, we aim to establish a high-probability bound.
As a result, we need the additional boundedness condition, which, however, is simply for technicalreasons.
Third, in contrast to existing works that consider the unconstrained case only, we considerthe constrained case, which is more challenging.
Specifically, it requires that E antit holds underpolicy π ∗ (i.e., the expectation over π ∗ ) rather than under a single optimal action x ∗ .
With the above sufficient condition, we have the following general performance bounds for CKB.
Theorem 2.
Suppose ρ ≥ 4 B/δ and V = G √ T/ρ .
Let κ := B + ρG .
Assume that CKB is equippedwith exploration strategies that satisfy the sufficient condition in Assumption 3.
Then, under Slater’scondition in Assumption 1 and regularity assumptions in Assumption 2, CKB achieves the followingbounds on regret and constraint violation with probability at least 1 − α − p 1 for any α ∈ (0 , 1) :
[Equation]
In the following, we will show that Theorem 2 provides a unified view of the regret and constraintviolation performance for various CKB algorithms, thanks to our sufficient condition.
In particular,we first show that existing exploration strategies, such as GP-UCB and GP-TS, satisfy our sufficientcondition, given by the following corollary.
Corollary 1.
Let β t =
B + R (cid:112) 2( γ t − 1 + 1 + ln(1 /α f )) and (cid:101) β t =
B + R (cid:112) 2( (cid:101) γ t − 1 + 1 + ln(1 /α g )) .(i).
GP-UCB with (cid:98) β t = β t and (cid:98) β t = − (cid:101) β t for A
f and A g , respectively, satisfies the sufficient condition. (ii).
GP-TS with (cid:98) β t = β t and (cid:98) β t = − (cid:101) β t for A f and A g , respectively, satisfies the sufficient conditionwhen π ∗ is a deterministic policy.
The sufficient condition also enables us to design CKB algorithms with new exploration strategies.
In the following, inspired by [39], we propose a new GP-based exploration strategy, which aims tostrike a balance between GP-UCB and GP-TS explorations.
Definition 2 (RandGP-UCB Exploration) .
Suppose that the posterior distribution for a black-boxfunction h in round t is given by GP ( (cid:98) µ t − 1 ( · ) , (cid:98) k t − 1 ( · , · )) .
Then, the estimate of h in round t under
RandGP-UCB exploration strategy is h t ( · ) = (cid:98) µ t − 1 ( · ) + (cid:98) Z t (cid:98) σ t − 1 ( · ) , where (cid:98)
Z t ∼ (cid:98) D for somedistribution (cid:98) D and (cid:98) σ 2 t − 1 ( x ) = (cid:98) k t − 1 ( x, x ) for all x ∈ X .
In contrast to GP-UCB, RandGP-UCB replaces the deterministic confidence bound by a randomizedone.
Compared to GP-TS, RandGP-UCB uses “coupled” noise in the sense that all the actions sharethe same noise (cid:98) Z t rather than “decoupled” and correlated noise in GP-TS.
This subtle difference willnot only help to eliminate the additional factor (cid:112) ln( |X| ) in GP-TS due to the use of union bound,but also allow us to deal with a general stochastic π ∗ .
Corollary 2.
Let β t , (cid:101) β t be the same as in Corollary 1.
RandGP-UCB with (cid:98) D = N (0 , β 2 t ) and(cid:98) D = N (0 , (cid:101)
β 2 t ) for A f and A g , respectively, satisfies the sufficient condition.
Thus, one can instantiate CKB with RandGP-UCB exploration to obtain a new algorithm called
CKB-Rand with performance guarantees given by Theorem 2.
Note that RandGP-UCB with otherdistributions (cid:98) D can also satisfy the sufficient condition (as discussed in Appendix C).
Discussions
In this section, we discuss several possible questions one may have at this moment about our CKBalgorithm.
The first natural question to ask is whether one can further improve the constraint violationbound.
It turns out that with a minor modification of CKB algorithm, one can achieve a boundedand even zero constraint violation by trading the regret slightly (but still the same order as before).
The modification is to introduce a slackness given by ε in the dual update in Algorithm 1, i.e.,ϕ t +1 =
Proj [0 ,ρ ] (cid:2) ϕ t + 1 V
(¯ g t ( x t ) + ε ) (cid:3) with ε ≤ δ/ 2 .
Intuitively speaking, this can be viewed as ifone is working on a new pessimistic constraint function.
After obtaining the constraint violationunder this new hypothetic constraint, one can subtract εT to find the true constraint violation underthe true function g .
The catch here is that one needs also change the baseline problem to the followingone: max π { E π [ f ( x )] : E π [ g ( x )] + ε ≤ 0 } so that it matches the new pessimistic constraintfunction.
Let π ∗ ε be the optimal solution to this new problem and the obtained regret is only withrespect to π ∗ ε rather than the original π ∗ .
Thus, we need to further bound the following difference
T E π ∗ [ f ( x )] − T E π ∗
ε [ f ( x )] to obtain the true regret bound, which is given in Appendix E.
[Figure]
Another question one may ask is why we are able to go beyond UCB-type exploration compared toexisting works (e.g., [13–15]) that only establish performance guarantees under UCB.
The answeris that they resort to another popular method for constrained optimization based on the so-called
Lyapunov-drift argument.
However, this method relies heavily on the optimism under UCB to deriveboth regret and constraint violation bounds.
Thus, it is difficult to generalize it for other explorationstrategies even for simpler linear bandit problems.
In contrast, our analysis relies only on the boundfor T 1 + T 2 , which allows us to consider more general exploration strategies.
In Appendix F, we alsodiscuss how to apply the Lyapunov-drift based method to analyze UCB exploration in our setting,through which we highlight more differences and some subtleties in Lyapunov-drift method.
Simulation Results
Although our work is mainly a theoretical study, we conduct simulations to compare the performanceof our algorithms (i.e., CKB-UCB , CKB-TS , and CKB-Rand , that is, CKB with GP-UCB, GPTS, and RandGP-UCB explorations, respectively) with existing safe KB algorithms based on bothsynthetic and real-world datasets.
In particular, we consider the two most recent safe KB algorithms:
StageOpt [28] (which has a superior performance compared to SafeOpt [27]) and SGP-UCB [3].
Our goal is to show that our proposed CKB algorithms can trade a slight performance in constraintviolation (i.e., soft constraint) for improvement in the reward regret, computation complexity andflexible implementations, i.e., a better complexity-regret-constraint trade-off.
5.1 Synthetic Data and Light-Tailed Real-World Data
Synthetic Data.
The domain X is generated by discretizing [0 , 1] uniformly into 100 points.
Theobjective function f ( · ) = (cid:80) pi
=1 a i k ( · , x i ) is generated by uniformly sampling a i ∈ [ − 1 , 1] andsupport points x i ∈ X with p = 100 .
With the same manner, we generate the constraint function g .
The kernel is k se with parameter l = 0 .
Other parameters include B , R and γ t are set similar as inthe unconstrained case (e.g., [5]).
Light-Tailed Real-World Data.
We use the light sensor data collected in the CMU Intelligent
Workplace in Nov 2005, which is available online as Matlab structure 3 and contains locations of 41sensors, 601 train samples and 192 test samples.
We use it in the context of finding the maximumaverage reading of the sensors.
In particular, f is set as empirical average of the test samples, with Bset as its maximum, and k is set as the empirical covariance of the normalized train samples.
Theconstraint is given by g ( · ) = − f ( · )+ h with h = B/ 2 .
We perform 50 trials (each with T = 10 , 000 )and plot the mean of the cumulative regret along with the error bars, as shown in Fig.
Regret.
Our three CKB algorithms achieve a better (or similar) regret performance compared tothe existing safe BO algorithms (see Figures 1(a) and 1(b)).
Among the three CKB algorithms,
CKB-Rand appears to have reasonably good performance at all times.
Constraint violation.
Since we have V ( T ) = 0 under all the algorithms, we study the total numberof rounds where the constraint is violated, denoted by N .
In the synthetic data setting, our proposed
CKB algorithms have N ≤ 5 over T = 10 , 000 rounds; in the real-world data setting, CKB-UCBenjoys N = 0 and CKB-Rand has an average N = 38 over a horizon T = 1 , 000 .
Furthermore, weplot the stronger cumulative constraint violations given by (cid:80) Tt =
[ g ( x t )] + as shown in Figure 1(c),
[Figure]
from which we can see that all CKB algorithms achieve sublinear performance even with respect tothis stronger metric, i.e., violation cancellation across rounds is not allowed.
Practical considerations.
Our proposed CKB algorithms have the same computational complexityas the unconstrained case.
In particular, they scale linearly with the number of actions in the discretedomain case 4 .
On the other hand, StageOpt scales quadratically due to the construction of the safeset, and SGP-UCB requires the additional random initialization stage, which leads to linear regret atthe beginning of the learning process.
Moreover, standard methods for improving the scalability ofunconstrained KB can be naturally applied to our CKB algorithms [41].
Finally, both StageOpt and
SGP-UCB require the knowledge of a safe action (i.e., one that satisfies the constraint) in advance,and moreover, StageOpt requires f to be Lipschitz and needs to estimate the Lipschitz constant,which impacts the robustness.
In contrast, CKB algorithms for soft constraints only require a mild
Slater’s condition as in Assumption 1, which does not require the existence of a safe action.
5.2 Heavy-Tailed Real-World Data
We further compare different constrained KB algorithms in a new real-world dataset, which demonstrates a heavy-tailed noise.
Note that sub-Gaussian noise is required in all the existing theoreticalworks (including our work).
We use this dataset to test the robustness of various constrained KBalgorithms.
The experimental results tend to show that our three CKB algorithms are more robust interms of heavy-tailed noise, which is common in practical applications.
The detail of this real-worlddataset is deferred to Appendix D.
Regret.
We plot both cumulative regret and time-average regret in this setting (see Figures 2 (a)and (b)).
We can observe that in the presence of heavy-tailed noise, our three CKB algorithms havesignificant performance gain over existing safe KB algorithms.
Constraint violation.
We focus on the strong metric, i.e., the number of rounds where the constraintis violated, denoted by N .
We have that CKB-Rand enjoys an average N = 21 and CKB-UCB hasan average N = 47 within the horizon of T = 10 , 000 .
We also plot stronger cumulative constraintviolations given by (cid:80)
Tt =
[ g ( x t )] + as shown in Figure 2(c), from which we can see that all CKBalgorithms achieve sublinear performance even with respect to this stronger metric.
Conclusion
We presented a general framework for constrained KB with soft constraints via primal-dual optimization.
Armed with our developed sufficient condition, this framework not only allows us todesign provably efficient (i.e., sublinear reward regret and sublinear total constraint violation)
CKBalgorithms with both UCB and TS explorations, but presents a unified method to design new effectiveones.
By introducing slackness, our algorithm can also attain a bounded or even zero constraintviolation while still achieving a sublinear regret.
We further perform simulations on both syntheticdata and real-world data that corroborate our theoretical results.
Along the way, we also present the first detailed discussion on two existing methods for analyzing constrained bandits and MDPs by highlighting interesting insights