A Robust Phased Elimination Algorithm for Corruption-Tolerant Gaussian Process Bandits
Abstract
We consider the sequential optimization of an unknown, continuous, and expensive to evaluate reward function, from noisy and adversarially corrupted observed rewards.
When the corruption attacks are subject to a suitable budget C and the function lives in a Reproducing Kernel Hilbert Space (RKHS), the problem can be posed as corrupted Gaussian process (GP) bandit optimization .
We propose a novel robust elimination-type algorithm that runs in epochs, combines exploration with infrequent switching to select a small subset of actions, and plays each action for multiple time instants.
Our algorithm, Robust GP Phased Elimination (RGP-PE) , successfully balances robustness to corruptions with exploration and exploitation such that its performance degrades minimally in the presence (or absence) of adversarial corruptions.
When T is the number of samples and  T is the maximal information gain, the corruption-dependent term in our regret bound is O ( C  3 / 2 T ) , which is signiﬁcantly tighter than the existing O ( C p T  T ) for several commonly-considered kernels.
We perform the ﬁrst empirical study of robustness in the corrupted GP bandit setting, and show that our algorithm is robust against a variety of adversarial attacks.
Introduction
Black-box optimization is a fundamental problem with broad applications including hyperparameter tuning [42], robotics [34] and chemical design [20], among others.
To make the problem tractable, a variety of smoothness properties have been adopted, and Reproducing Kernel Hilbert Space (RKHS) functions have proved to provide a versatile framework that can be tackled via Gaussian process (GP) methods [43, 15].
This problem is referred to as GP bandits or kernelized bandits .
While an extensive line of works have established GP bandit algorithms and regret bounds, settings with adversarial corruptions have only arisen relatively recently.
Such corruptions may come in the form of outliers [38, 41], perturbations of sampled inputs [5, 40, 16], adversarial noise in the rewards [8], or perturbations of the ﬁnal recommendation [7].
In this work, we are interested in the setting of adversarial noise in the rewards, in which the performance of standard non-robust GP bandit algorithms can deteriorate signiﬁcantly (see Fig. 1).
The ﬁrst work considering this setting [8] established regret bounds for various algorithms depending on the degree of knowledge on the corruption level C (deﬁned formally in Section 2).
A key limitation in their regret bound is that the main corruption-dependent term, C , and the usual uncorrupted regret term, which is p T or higher (with time horizon T ), are multiplied together .
That is, the dependence 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
with no attacks and the two most effective corruption attacks on the Robot3D pushing task.
As the number of samples t increases, the performance of non-robust GP-UCB deteriorates signiﬁcantly under both attacking strategies, while the performance of the proposed algorithm remains robust.
on C is multiplicative with respect to the uncorrupted bound.
Analogous studies of bandits with independent arms [35, 21] or linear rewards [9] suggest that additive dependence may be possible, but this has remained very much open in the GP bandit setting.
In this paper, we address this fundamental gap in the literature by introducing a novel algorithm in which the uncorrupted term and the C -dependent term are clearly decoupled, and the latter is only multiplied by a kernel-dependent function of T that can be much smaller than p T .
Related work.
The closest work to ours is [8], which also considers the corruption-tolerant GP bandit setting.
In that work, the authors propose a conﬁdence-bound-based algorithm with enlarged conﬁdence.
As outlined above, the regret bound therein scales as O ( C p T  T ) , and the possibility of additive C dependence was left as an open problem.
The question of additive vs. multiplicative dependence ﬁrst arose in multi-armed bandits with independent arms, with an initial work [35] being multiplicative, and a subsequent work [21] improving to additive.
Closer to our setup (and in fact a special case of it via the linear kernel) is the case of corrupted stochastic linear bandits, in which additive dependence was obtained in [9], with the corruption term more precisely being O ( Cd 3 / 2 log T ) under mild assumptions.
1 Our main result will achieve a similar bound as a special case, while being much more general due to handling general kernels, and adopting GP-based algorithmic and mathematical techniques that have minimal overlap with the linear setting.
Other less related results for corrupted linear bandits (e.g., contextual or instance-dependent) are given in [31] and [50].
Adversarial corruptions of the rewards were also considered for GP bandits in [28], but with the key difference of considering a weaker adversary that does not know the chosen action when choosing the corruption term.
This distinction has a considerable effect on the problem, leading to signiﬁcantly different algorithms, and with the setting of [28] leading to a GP-UCB-style regret bound in which the corruptions only impact the constant factors.
In our setting, the effect of corruptions is much more signiﬁcant, and we know from [9] that this is unavoidable in general.
Other less related notions of robustness in GP bandits have included outliers [38], misspeciﬁcation [13, 6], input noise [5, 40, 16], risk-aversion [39, 11, 37], and corruptions in the ﬁnal recommendation [7, 29].
Moreover, other settings with adversarial corruptions have included multi-armed bandit and online [21, 25, 24, 3], active [14], reinforcement learning [36, 49, 4], and multi-agent RL [33].
Corruption-robustness have been considered in other sequential decision making problems including multi-armed bandits and predictioin with expert advice / online learning.
Contributions.
We provide a novel algorithm for GP bandit optimization with adversarial corruptions, that attains the ﬁrst regret bound to avoid multiplying the uncorrupted part by the corruption level
C .
Our algorithm crucially incorporates a rare switching idea, along with a non-standard robust estimator, enlarged conﬁdence bounds, and a minimal number of plays of each selected action; see
Sections 2.1 and 3 for details.
To our knowledge, we are the ﬁrst to use rare switching to achieve adversarial robustness; previous works instead used it for reducing computational complexity.
We show that our regret bound is provably near-optimal for the SE kernel, and recovers recentlyestablished bounds for stochastic linear bandits [9] that are also known to be near-optimal.
For the
Matérn kernel, the degree of tightness depends on the dimension and smoothness parameter, but our bound strictly improves on that of [8] in all scaling regimes where the latter is non-trivial (i.e., sub-linear in T ); see Table 1 on Page 7 for a summary.
We demonstrate that our algorithm is able to successfully defend against various attacks, including those proposed in [22].
On the technical side, we note that the GP setting dictates the use of a signiﬁcantly different algorithm compared to linear bandits, and a technical analysis with only minor overlap.
To highlight this, in
Appendix E, we explore an approach based on a direct reduction to linear bandits (followed by using the algorithm in [9]), and show that it yields strictly worse regret scaling than our main result.
Problem Setting and Preliminaries
We consider the Gaussian process bandit (i.e., kernelized bandit) problem, in which the goal of the learner is to maximize the collected rewards by sequentially querying the unknown reward function f : X ! R over T rounds.
In particular, at every time t , the learner selects x t 2 X and receives
[Equation]
where ✏ t is assumed to be  -sub-Gaussian with independence over time steps, and  is also known.
We consider the corrupted setting in which, besides the stochastic noise, the observations at every time step are adversarially corrupted, so that the learner observes
[Equation]
Following [8], we make the following assumptions on the adversary:
• The adversary knows the true reward function f ( · ) , and, at every round t , it observes x t before deciding upon the corruption
c t .
• The total adversarial corruption budget over T rounds is bounded as follows:
[Equation]
In this paper, we focus primarily on the case where C is known to the learner, but we also discuss in Section 3.4 how our results have implications for the case of unknown C .
The domain X is assumed to either be ﬁnite, or a compact subset of R d for some dimension d (e.g.,
X = [0 , 1] d ).
In either case, X is endowed with a continuous, positive semideﬁnite kernel function k ( · , · ) :
X ⇥ X ! R that is normalized to satisfy k ( x, x 0 )  1 for all x, x 0 2 X .
We further assume that f has a bounded norm in the corresponding Reproducing Kernel Hilbert Space (RKHS)
H k , i.e., k f k k  B (see Appendix A for more details).
This assumption permits the construction of conﬁdence bounds via Gaussian process (GP) models (Section 3.2).
The learner’s performance is measured using the widely-considered notion of cumulative regret:
[Equation]
4) and we are interested in the joint dependence of R T on C and T .
As noted in [35] and [8], one could alternatively deﬁne the cumulative regret with respect to the corrupted values (i.e., f ( x ) + c t ), and these notions coincide to within an additive term of 2 C .
2.1 Gaussian Process Model under Corruptions
In the standard (non-corrupted) setting, previous algorithms use (i) zero-mean GP priors for modeling the uncertainty in f (i.e., they assume f ⇠ GP (0 , k ) ), and (ii) Gaussian likelihood models for the observations.
As more data points become available, Bayesian posterior updates are then performed according to a misspeciﬁed model in which the noise variables ✏ t =
y t  f ( x t ) are assumed to be drawn independently across t from N (0 ,  ) , where  is a hyperparameter that may differ from the true noise variance  2 .
In particular, in the absence of corruptions, given a sequence of points { x 1 , . . .
, x t } and their noisy observations { y 1 , . . .
, y t } , the posterior mean and variance are given by
[Equation]
where k t ( x ) =
[Equation]
x i , [Equation] i =1 , K t = ⇥ k
( x t , x t 0 )
⇤ t,t 0 is the kernel matrix, and Y t 2 R t contains the non-corrupted observations up to time t
, i.e., Y t [ i ] =
y i for i 2 [ t ] .
In the corrupted setting, given the inputs { x 1 , . . .
, x t } and their corrupted observations { e y 1 , . . .
, e y t } (with e y i
=
y i
+ c i ), we propose the following non-standard robust posterior mean estimator:
[Equation]
(7) where e Y t 2 R t and e
[Equation] i [Equation] e
[Equation] = [Equation] 1 { x
[Equation]
x j } for i 2 [ t ] .
Intuitively, the averaging of terms corresponding to identical actions is done in order to diminish the impact of corruption, and this will be a crucial component of our analysis.
In our algorithm, besides e µ t ( · ) , we will also make use of the standard posterior variance  2 t ( · ) as given in Eq.
(6); the use of this quantity is intuitively reasonable because GP posterior variances do not depend on the observations.
The main quantity that characterizes the regret bounds in the non-corrupted setting (and is also useful in our setting) is the maximum information gain [43], deﬁned at time t as [Equation]
[Equation]
Robust GP Phased Elimination 3.1 Algorithm and Conﬁdence Bounds
Our algorithm works in epochs indexed by h = 0 , 1 , . .
, H  1 , each of which consists of sampling a batch of points.
The epoch lengths may be chosen adaptively, and hence H may not be deterministic, but we will ensure with probability one that H  ¯ H with ¯ H = log 2 T .
The length of epoch h is denoted by u h , so that
[-that (cid:80) H − 1 h =
 u h = T .-][Equation]
The algorithm and analysis are based on the widespread notion of conﬁdence bounds.
While our conﬁdence bounds will be expanded to account for corruptions, it is useful to consider the following generic assumption regarding non-corrupted observations (although the algorithm cannot access these, they will appear in our mathematical analysis).
Assumption 1 (Regular conﬁdence bounds) .
Let µ ( h ) ( x ) and  ( h ) ( x ) denote the posterior mean and standard deviation computed (hypothetically) using only the non-corrupted observations { ( x i , y i ) } u h i =1 in epoch
h using Eqs.
(5) and (6) .
We assume that given  2 (0 , 1) , there exists a sequence of parameters
h =  h (  ) which is non-decreasing in h and yields with probability at least 1   that
[Equation]
simultaneously for all h  0 and x 2 X .
Speciﬁc choices of  h satisfying this assumption will be considered in Section 3.2.
Similarly to previous kernelized algorithms (e.g., [8, 6]), our proposed algorithm makes use of enlarged conﬁdence bounds.
Hence, our ﬁrst result concerns concentration of an RKHS member under corrupted observations, where we make use of the proposed estimator from Eq.
Lemma 2 (Corrupted conﬁdence bounds) .
Under Assumption 1, let e µ ( h ) ( x ) denote the posterior mean based on only the corrupted observations { ( x i , e y i ) }
u h i =1 in epoch
h using Eq.
(7) , and let u min  1 denote the minimum number of times any single action from { x i } u h i =1 is played, i.e., u min = min x 2 { x 1 ,...,x uh }
P u h i =
1 1 { x i = x } .
Then, with probability at least 1   , it holds for all x 2 X and h  0 that
[Equation]
The conﬁdence-bound enlargement is proportional to the total amount of corruption C .
This bears some similarity to the conﬁdence intervals used in [8, Lemma 2], but we note the following important differences:
• We make use of a novel kernelized mean estimator (Eq.
(7)) that takes average over rewards corresponding to the same played action;
[Equation]
o
• Our enlargement term is O
( C p u h u min ) , as opposed to O ( C ) used in [8, Lemma 2].
We will typically apply this lemma with p u h u min ⌧ 1 , so that our conﬁdence width is much smaller.
For the second of these, the intuition is that if the same action is played multiple times it becomes harder for the adversary to hide the true value (i.e., since the rewards of the same played actions are averaged, the adversary needs to spend more of its budget corrupting the reward).
The Robust GP-Phased Elimination algorithm (Algorithm 1) proceeds in epochs (indexed by h ) of exponentially increasing length u h .
At every round t (where t 2 { 1 , . . .
, l h } and l h =
2 h +1 ) within an epoch h , the algorithm selects an action maximizing a posterior uncertainty computed at some (possibly strictly earlier) time t 0 :
[Equation]
where X h denotes the set of active actions in epoch h .
The selected action is then added to S h which is a set that contains distinct actions selected in epoch h .
The key idea behind using t 0 instead of t in Eq.
(11) is to ensure that our algorithm rarely switches , based on a condition relating to the information gain (Line 6), meaning that the same action x t is typically selected multiple times.
Whenever there are ties, they are resolved arbitrarily but consistently over rounds (i.e., if  t 0 ( · ) does not change, the same points are selected).
Based on Lines 6 to 9, we update t 0 and recompute  t 0 ( x ) only when det( I t +   1 K t ) increases by a constant factor ⌘ .
Related ideas of rare switching have appeared in the literature (e.g., [1, 47, 19]), but to our knowledge we are the ﬁrst to use this idea in the kernelized bandit problem to provide an algorithm that includes an explicit switching condition for improving robustness.
Intuitively, by rarely switching, we obtain more samples of the same point, allowing us to average more of them together and making the “averaged” observation harder to corrupt.
Concurrent work also used rare switching to reduce GP posterior computation, noting that the computation time can be made to scale (cubically) with the number of unique points [12].
This beneﬁt also applies directly to our algorithm, and we exploit it to run large- T experiments in Section 4.
After the set S h is constructed, we deﬁne ⇠ h ( x ) =
P lhi =1 1 { x = x i } l h for every x 2 S h , representing the empirical frequency of selecting x t 2 X h in l h rounds.
The algorithm then plays actions from S h only, where the number of times each action x from S h is played is denoted by u h ( x ) =
d l h max { ⇠ h ( x ) , } e .
Here, the truncation parameter ensures that each action from S h is played sufﬁciently many times; this idea was used for corrupted linear bandits in [9].
Our theory suggests a particular choice of ; see Theorem 3.
Each action x 2 S h is played for u h ( x ) times in an arbitrary order, leading to the total epoch length
u h = P x
[Equation]
h u h ( x ) .
Based on the received noisy and potentially corrupted rewards { x j , e y j } u h j =1 , the algorithm updates its estimates e µ ( h ) ( · ) and  ( h ) ( · ) according to Eq.
(7) and Eq.
Finally, each epoch h ends by updating the set of active actions
X h
+1 .
To do so, we use the conﬁdence bounds from Lemmawith u min = l h , where l h is a lower bound on the number of times each distinct action from S h is played.
These conﬁdence bounds are valid in the sense that the true function is contained within the conﬁdence bounds with high probability.
The deﬁnition of X h +
(Line 15) ensures that with high probability, the optimal action is never eliminated.
Besides the standard exploration/exploitation trade-off (controlled via  h ), our algorithm additionally balances robustness to corruptions.
This is done via two parameters: the switching parameter ⌘ and truncation parameter .
We set these parameter to ensure that the number of distinct actions played per epoch is sufﬁciently small, while the number of plays per each such action is sufﬁciently large.
This trade-off is non-trivial; for example, in the case
that C =
(i.e., the non-corrupted setting), resampling the same actions (controlled via ) increases the regret.
Main result.
We now present our main theoretical result, where we use O ⇤ ( · ) notation to hide constants and dimension-independent log factors.
We treat the RKHS norm bound B as being ﬁxed, so its dependence is also hidden in O ( · ) or O ⇤ ( · ) notation.
Theorem 3 (Main result) .
Under the preceding setup and Assumption 1, for any corruption budget
C  0 , Algorithm 1 with a constant switching parameter ⌘ > 1 and truncation parameter [Equation]
satisﬁes the following with probability at least 1   :
[Equation]
3.2 Applications to Speciﬁc Conﬁdence Bounds
Now we discuss speciﬁc choices of  h satisfying Assumption 1, and the resulting ﬁnal regret bounds.
We observe that the actions in each ﬁxed epoch are sampled non-adaptively, and the resulting GP posterior formed only depends on the points in that epoch.
As noted in [32], these conditions are sufﬁcient to make use of the following conﬁdence bounds for non-adaptive sampling.
Lemma 4.
[45, Theorem 1] When { x i } ti =1 are selected independently of all the observations { y i } ti =1 , it holds for any ﬁxed x 2 X and any t  1 with probability at least 1   that | µ t ( x )  f ( x ) |   B +  p  [Equation]
For ﬁnite domains, applying the union bound leads to a choice of  h for the proposed algorithm such that  ¯ H only contributes to logarithmic terms in the cumulative regret.
Corollary 5.
Deﬁning ¯
[Equation]
=
[Equation] + [Equation]  q 2 log |X|  , we have that Assumption 1 holds with
h = ¯  h
(  h ) and  h = 6  ( h +1) 2 ⇡ 2 .
Hence, with probability at least 1   , Algorithm 1 with switching parameter ⌘ > 1 , truncation parameter =
[Equation]
⌘ 2  T , and [-(cid:80) H − 1 h =
 δ h ≤ (cid:80) ∞-] h as above achieves
[Equation]
This corollary is obtained by noting that the error probability is at most  as desired, since a union bound over X gives a per-epoch term of at most  h , and
[Equation]
For general (possibly continuous) domains, one option is to set  h according to a widely-used conﬁdence bound as follows, though we will shortly discuss improved choices.
[Table] Ours
Linear p Td + Cd p
Td + Cd 3 /p Td + Cd 3 / SE p T (log T ) d/ 2 + C (log T )
d/p T (log T ) d + C p T (log T ) d/p T (log T ) d + C (log T ) 3 d/ Matérn
T ⌫ + d 2 ⌫ + d + C ⌫ d + ⌫ T dd + ⌫ T 2 ⌫ +3 d 4 ⌫ +2 d + CT ⌫ + d 2 ⌫ + d T ⌫ + d 2 ⌫ + d + CT 3 d 4 ⌫ +2 d
For the SE and Matérn kernels, the upper bounds are from [8] and the lower bounds are
For the linear kernel, the existing bounds are from [9], except the p Td lower bound which is from [17].
Lemma 6.
[15, Theorem 2]
For any (possibly adaptive) sampling strategy, it holds with probability at least 1   that | µ t ( x )  f ( x ) |  [Equation] for all x 2 X and t  1 .
By a similar argument to Corollary 5 and the fact that  t is increasing in t , we obtain the following.
Corollary 7.
If u h  ¯ u h almost surely, then deﬁning ˇ  h (  ) =
B +  p 2(  ¯ u h + 1 + ln(1 /  )) , we have that Assumption 1 holds with  h =
ˇ  h (  h ) and  h = 6  ( h +1) [-
 π-]2 ⇡ 2 .
Hence, with probability at least 1   , Algorithm 1 with a constant switching parameter ⌘ > 1 , truncation parameter = ln ⌘ 2  T , and  h as above achieves
[Equation]
where we crudely selected ¯ u h = T .
While this regret bound can be signiﬁcantly weaker than Corollary 5 due to the O ⇤ ( p T  T ) term, we can also obtain an analog of Corollary 5 (i.e., attaining the improved dependence in Eq.
(13)) for continuous domains, under the mild assumption that functions in the RKHS are Lipschitz continuous (which is true for the kernels we consider below).
A crude approach is to have the algorithm use a very ﬁne discretization [26, 32], and a more sophisticated approach is to only discretize as part of the analysis [45].
The details can be found in the preceding references, and we avoid repeating them.
3.3 Comparisons to Existing Bounds
We specialize our regret bound in Eq.
(13) to speciﬁc kernels by substituting  T =
O ⇤ ( d ) for the linear kernel,  T =
O ⇤ ((log T ) d ) for the SE kernel, and  T
= O ⇤ ( T d 2 ⌫ + d ) for the Matérn kernel [43].
The resulting regret bounds are shown in Table 1 (omitting constants and dimension-independent log factors), along with the best known existing upper and lower bounds.
We observe the following:
• For the linear kernel, we recover the recent upper bound of [9], and this is tight up to the presence of d vs. d 3 / 2 in the corrupted part.
• For the SE kernel, we match the lower bound of [10] up to small changes in the implied constant in each (log T ) ⇥ ( d ) term.
In contrast, the existing upper bound of [8] incurs a much larger p T term in the corrupted part.
• For the Matérn kernel, compared to the existing result in [8], we obtain an improvement in the non-corrupted part recently established in [32], matching the non-corrupted lower bound.
In the corrupted part, the existing result has a better exponent to T when ⌫ < d 2 , whereas ours is better when ⌫ > d 2 , in particular approaching zero (instead of 12 ) as ⌫ ! 1 and nearly matching the lower bound in this limit.
However, when ⌫ < d 2 we ﬁnd that the non-corrupted part in [8] is super-linear in
T , making the bound trivial.
Hence, our bound is better whenever non-trivial scaling is attained.
The bounds based on a reduction to linear bandits, which we derive in Appendix E, are omitted in
Table 1.
We brieﬂy note that they are able to provide a similar upper bound to our main one under the SE kernel, but are always strictly worse under the Matérn kernel.
3.4 Implications for the Unknown C Setting
While we have focused on the case of known C , an idea from a concurrent work [23] (on linear bandits) can be used to transfer our main result to a setting with unknown C .
The idea is that if the parameter C is used by the algorithm but C true is the amount of corruption actually used by the adversary, then the analysis goes through unchanged as long as C  C true .
Hence, we may cautiously choose a large value of C to cover more values of C true .
As an important special case, we may choose C such that the corrupted and uncorrupted regret terms are of the same order; for instance, in (13), setting C
= O  p T  T  gives R T
= O ⇤ ( p T  T ) . Hence, we ﬁnd that any corruption level C true up to O  p T  T  only affects the constant (or possibly logarithmic) factors, and the precise corruption level does not need to be known.
For particularly smooth kernels such as linear and SE (with constant dimension), the scaling O  p T  T  reduces to O ⇤ ( p T ) .
This may not seem as high as ideal, but at least in the case of linear bandits, it is known to be the best we can hope for unless the algorithm attains signiﬁcantly higher uncorrupted regret [9, 23].
Speciﬁcally, if optimal O ⇤ ( p T ) uncorrupted regret is attained, then linear regret is unavoidable when C = ! ( p T ) .
See [23] for similar statements with the dependence on d included.
The overall picture remains less complete for general kernels, but the preceding discussion reveals that our results for known C do have important implications for the unknown C setting.
Experiments
We experimentally evaluate the performance of our proposed algorithm, along with two baselines, one robust and one non-robust.
Our experiments serve as a proof of concept for our proposed approach, but also highlight possible remaining gaps between theory and practice, e.g., arising from large constant factors in the regret bounds.
We emphasize that our contributions are primarily theoretical.
Algorithms.
We consider the following three algorithms:
RGP-PE:
Robust GP-Phased Elimination with constant  h ; this is a slight variation of Corollary 5 in which the number of epochs H turns out to be a small constant in our experiments.
GP-UCB: a representative non-robust fully sequential algorithm with slowly growing  t , where t 2 [ T ] [43, Algorithm 1].
RGP-UCB: the robust version of GP-UCB with
slowly growing  t [8, Algorithm 1], where the only difference from GP-UCB is that the theoretical coefﬁcient of  t  1 in the UCB is  t + C p  .
We found the term  h + C p
u h l h  multiplying  ( h ) in Algorithm [-
-]1 to be overly conservative, so we instead replace it by  h + b · C p u h (since l h and u h are similar, we replace p u h l h by 1 p u h ), where b 2 (0 , 1] is an additional parameter controlling the degree of exploration and robustness.
Similarly, in
RGP-UCB we use the coefﬁcient  t + b · C p  .
The remaining parameters  h and  t are speciﬁed below.
Synthetic Function.
We produce a synthetic 2D function f 1 , shown in Figure 4 of the supplementary material, which is randomly sampled from a Gaussian Process with zero mean and the SE kernel with lengthscale l = 0 .
The domain X of f 1 contains 100 points obtained by evenly splitting [  5 , 5]into a 10 ⇥ 10 grid.
We use the true kernel as the prior for all three algorithms, and use  h = 4 for
RGP-PE, and  t = p log t/ 2 for GP-UCB and RGP-UCB.
Robot Pushing Objective Function.
We consider the deterministic robot pushing objective function on a 2D plane introduced in [48], which aims to ﬁnd suitable parameters to push an object to the target location r g .
We use the Robot3d function, which takes the robot location ( r x , r y ) and pushing duration t r as a 3D input, and outputs the reversed distance between the pushed robot location and the target location r g , i.e.,
[Equation]
where push( · ) outputs the pushed robot location.
We let the domain X contain 100 points ( r x , r y , t r ) randomly sampled from [  5 , 5] 2 ⇥ [1 , 30] , and the target location r g is set to be (3 , 2) .
Since the lengthscale of the SE kernel with maximum likelihood given the noiseless data is 1 . 94 ⇡ 2 , we use the SE kernel with l = 2 as prior for all three algorithms.
We found it beneﬁcial for all algorithms to be slightly more explorative for this function, and accordingly use  h = 6 for KE and  t = 2 p log t for GP-UCB and RGP-UCB.
Attack Methods.
We consider the following ﬁve attack methods, which continue until the corruption budget is exhausted:
• Clipping:
This attack proposed in [22] perturbs f and produces another reward function
e f whose optima are in some region R target that does not contain x ⇤ by setting
[Equation]
where e x
⇤ = arg max x 2 R target f ( x ) .
We let  = 0 .
5 and choose R target = { ( x 1 , x 2 ) 2 X : x 1  x 2 } for f 1 , and R target = { ( r x , r y , t r ) 2 X : r x  0 } for the function Robot3D. • Aggressive Subtraction (AggSub): This attack proposed in [22] sets
[Equation]
for some h max > f ( x ⇤ )  f ( e x ⇤ ) .
We use the same R target as the Clipping attack and let h max = 1 for f 1 and h max = 3 for Robot3D. •
Top- K : When x is one of the top K remaining actions, this attack perturbs the reward down to  1 .
We consider both K = 3 and K = 5 .
• Flip: This attack simply ﬂips the reward from f ( x ) to  f ( x ) .
Both this attack and the previous one are variations of attacks considered for linear bandits in [9].
For the algorithms, we consider C = 50 and C = 100 .
By default, the attack starts at t = 1 , but for the robust algorithms RGP-PE and RGP-UCB, we also conduct experiments with a later attack, where (i) the attack in RGP-PE starts when at least one action is eliminated from the domain; and (ii) the attack in RGP-UCB starts when at least one action has UCB strictly lower than max x 2 X LCB( x ) .
We let T = 50000 , 2  = 0 .
02 , and  = 1 for all three algorithms, b = 0 .
1 for RGP-PE and
RGP-UCB, and = 0 .
5 , ⌘ = 2 for RGP-PE.The results are produced by performing 10 trials and plotting the average cumulative regret, with error bars indicating one standard deviation.
Comparison of Algorithms.
As shown in Figures 2 and 3, the non-robust algorithm GP-UCB succeeds when no attack is applied.
However, the cumulative regret for f 1 associated with the
Clipping, AggSub, Top-3, and Top-5 attacks grow linearly, indicating that these four attacks succeed in driving GP-UCB towards a suboptimal action.
Similarly, the Top-3 and Top-5 attacks incur linear regret for Robot3D. In contrast, we ﬁnd that RGP-PE has only one action remaining at the end of the 13th epoch, and manages to defend against all ﬁve attack methods for both functions.
The baseline robust algorithm RGP-UCB also successfully defends against all the attacks, and generally has lower cumulative regret than RGP-PE, despite RGP-PE having a stronger regret guarantee.
There are at least two possibly reasons for this: (i) The analysis of RGP-UCB in [8] could be loose, with a tighter analysis potentially giving an additive dependence similar to Theorem 3, and (ii) the strong scaling laws in our theory may still leave room for improvements in the constant factors (or logarithmic).
Further addressing these ﬁndings remains an interesting direction for future work.
We note that even in the more specialized problem of corrupted stochastic linear bandits, analogous practical limitations of a phased elimination algorithm were observed in [9].
Later Attack.
We observe that RGP-PE and RGP-UCB are also able to defend against the later attack, and their performance is similar to when the attack starts from the beginning.
There are only two trials of RGP-PE (budget C = 100 and Top-5 attack on Robot3D in Figure 3), in which the only action remaining at the end of the 13th epoch is slightly suboptimal.
In Appendix F, we additionally show the experiment results for f 1 with C = 100 , and Robot3D with C = 50 .
Conclusion
We have provided a new algorithm for corruption-tolerant GP bandits based on phased elimination, incorporating a key idea of rare switching based on a certain condition relating to the information gain, along with a robust estimator, enlarged conﬁdence bounds, and truncation to ensure a minimal number of plays of each selected action.
Our regret bound recovers the best known existing bound under the linear kernel, is provably near-optimal under the SE kernel, and improves on the best existing bound in all cases where the latter is non-trivial.
Acknowledgment
This project has received funding from the European Research Council (ERC) under the European
Unions Horizon 2020 research and innovation programme grant agreement
No 815943.
J. Scarlett was supported by the Singapore National Research Foundation (NRF) under grant number R-252000-A74-281.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[-(47), we multiply and divide by (cid:80) x ∈X ,u t ( x ) (cid:54) =
-]
[-(cid:54) =
-]
[-(cid:54) =
-]
[-(cid:54) =
-]
[-(cid:54) =
-]
[N/A] 5.
[-i =
-]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[-i +
-]
[-=
-]
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[-Then, we note that u 
-]
[-=1 N 
-]
[-Di =
-]