Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptions
Anonymous Author(s)
AffiliationAddress email
Abstract
We study the linear contextual bandit problem in the presence of adversarialcorruption, where the reward at each round is corrupted by an adversary, andthe corruption level (i.e., the sum of corruption magnitudes over the horizon) is
C ≥ 0 .
The best-known algorithms in this setting are limited in that they eitherare computationally inefficient or require a strong assumption on the corruption,or their regret is at least C times worse than the regret without corruption.
In thispaper, to overcome these limitations, we propose a new algorithm based on theprinciple of optimism in the face of uncertainty.
At the core of our algorithm is aweighted ridge regression where the weight of each chosen action depends on itsconfidence up to some threshold.
We show that for both known C and unknown
C cases, our algorithm with proper choice of hyperparameter achieves a regretthat nearly matches the lower bounds.
Thus, our algorithm is nearly optimal up tologarithmic factors for both cases.
Notably, our algorithm achieves the near-optimalregret for both corrupted and uncorrupted cases ( C = 0 ) simultaneously.
Introduction
We study linear contextual bandits with adversarial corruptions.
At each round, the agent observesan decision set provided by the environment and selects an action from the decision set.
Thenan adversary corrupts the reward of the action selected by the agent.
The agent then receives thecorrupted reward of the selected action and proceeds until K rounds.
The agent’s goal is to minimizethe regret Regret ( K ) which is the difference between the optimal accumulated reward and theselected accumulated reward.
This problem can be regarded as a combination of the two classicalbandit problems, stochastic bandits and adversarial bandits (Lattimore and Szepesvári, 2018).
Inpractice, the contextual bandits with adversarial corruptions can describe many popular decisionmaking problems such as pay-per-click advertisements with click fraud (Lykouris et al., 2018) andrecommendation system with malicious users (Deshpande and Montanari, 2012).
Lykouris et al.
(2018) first studied the multi-armed bandit with adversarial corruptions.
Specifically,let C denote the corruption level which is the sum of the corruption magnitudes at each round.
Lykouris et al.
(2018) proposed an algorithm with a regret that is C times worse than the regretwithout corruption.
Later, Gupta et al.
(2019a) proposed an improved algorithm whose regret consistsof two terms: a corruption-independent term that matches the optimal regret for multi-armed banditwithout corruption, and a corruption-dependent term that is linear in C and independent of K , i.e.,
Regret ( K ) =
o ( K ) + O ( C ) .
The lower bound proved in Gupta et al.
(2019a) suggests that thelinear dependence on C is near-optimal.
Such a regret structure reveals an desirable property ofcorruption-robust bandit algorithms, that is, the algorithm should perform nearly the same as thebandit algorithms without corruption when the corruption level C is small or diminishes.
Based on the above observation, a natural question arises:
Can we design computationally efficient algorithms for linear contextual bandits with corruption thatcan attain the best possible regret, similar to those in multi-armed bandits?
Some previous works have attempted to answer this question for the simpler stochastic linear banditsetting, where the decision sets at each round are identical and finite.
Li et al.
(2019) studied thestochastic linear bandits and proposed an instance-dependent regret bound.
Later on, Bogunovic et al.(2021) studied the same problem and proposed an algorithm that achieves a regret with the corruptionterm depending on C linearly and on K logarithmically.
However, these algorithms are limited to thestochastic linear bandit setting since their algorithm design highly relies on the experiment designand arm-elimination techniques that require a multiple selection of the same action and can onlyhandle finite decision set.
They are not applicable to contextual bandits, where the decision set ischanging over time and can even be infinite.
For the more general linear contextual bandit setting,
Bogunovic et al.
(2021) proved that a simple greedy algorithm based on linear regression can attainan ideal corruption term that has a linear dependence on C and a logarithmic dependence on K , undera stringent diversity assumption on the contexts.
Lee et al.
(2021) proposed an algorithm and thecorruption term in its regret depends on C linearly and on K logarithmically, but only holds for therestricted case when the corruption at each round is a linear function of the action.
Without specialassumptions on the contexts or corruptions, Zhao et al.
; Ding et al.
(2021) proposed a variantof the OFUL algorithm (Abbasi-Yadkori et al., 2011) and its regret has a corruption term dependingon K polynomially.
Recently, Wei et al.
(2022) proposed a Robust VOFUL algorithm that achievesa regret with a corruption term linearly dependent on C ′ 1 and only logarithmically dependent on
K .
However, Robust VOFUL is computationally inefficient since it needs to solve a maximizationproblem over a nonconvex confidence set that is defined as the intersection of exponential number ofsets, and its regret has a loose dependence on context dimension d .
In addition, Wei et al.
(2022) alsoproposed a Robust OFUL algorithm and provided a regret guarantee that has a linear dependenceon a different notion of corruption level C r 2 , which is strictly larger than the corruption level Cconsidered in the previous work and the current paper.
Thus, the above question remains open.
In this paper, we give an affirmative answer to the above question.
We summarize our contributionsas follows.
• We propose a computationally efficient algorithm based on the principle of optimism in the faceof uncertainty (Abbasi-Yadkori et al., 2011), named Confidence-Weighted OFUL (CW-OFUL).At the core of our algorithm is a weighted ridge regression where the weight of each chosen armis adaptive to its confidence, which is defined as the truncation of the inverse exploration bonus.
Intuitively, such a weighting strategy prevents the algorithm from exploiting the contexts whoserewards are more likely corrupted by a large amount.• For the case when the corruption level C is known to the agent, we show that the proposed algorithmenjoys a regret Regret ( K )
= (cid:101) O
( d √ K + dC ) , where d is the dimension of the contexts, C is thecorruption level and K is the number of total iterations.
The first term matches the regret lowerbound of linear contextual bandits without corruption Ω( d √ K ) (Lattimore and Szepesvári, 2018).The second term matches the lower bound on the corruption term in regret Ω( dC )
(Bogunovicet al., 2021).
They together suggest that our algorithm is not only robust but also near-optimal upto logarithmic
factors.•
For the case when the corruption level C is unknown to the agent, we show that CW-OFUL enjoysan (cid:101) O ( d √ K ) regret for the case C ≤ √ K , with proper choice of the hyperparameter.
Surprisingly,by proving a lower bound on the regret, we show that our regret upper bound is already optimal forall algorithms that achieve a near-optimal regret bound for uncorrupted bandits.
We compare our regret bounds with previous ones in Table 1.
We can see that
our algorithm matchesthe lower bound up to logarithmic factors in both known C and unknown C cases and therefore isnearly optimal
[Table]
1.1 Additional Related Work
Bandits with Misspecification.
Bandits with misspecification can be seen as a special case ofbandit with adversarial corruption since it is corrupted relative evenly at each round.
Let ϵ be themisspecification level.
Ghosh et al.
(2017) firstly studied the stochastic linear bandits and proved asublinear regret when ϵ is small.
Lattimore and Szepesvari (2019) studied the stochastic linear banditsetting under milder assumptions.
With the knowledge of ϵ , they proposed an algorithm with an(cid:101) O ( (cid:112) dK log( N )+ ϵ √ dK ) regret, where d is the dimension of the contextual vector, N is the numberof arms.
Their regret bound matches their proved lower bound up to logarithmic factors.
Foster et al.(2020) further considered the more general linear contextual bandits with misspecification whenϵ is unknown to the agent, and proposed an algorithm equipped with a CORRAL meta algorithm(Agarwal et al., 2017) to deal with the unknown ϵ .
Their algorithm enjoys an (cid:101)
O
( d √ K + ϵ √ dK )regret.
Krishnamurthy et al.
(2021) proposed an algorithm without using a meta algorithm whichhas the same order of regret as Foster et al.
Our algorithm can be directly applied to themisspecification setting by choosing the corruption level C to be Kϵ , which immediately gives us an(cid:101)
O ( d √ K + dKϵ ) regret upper bound.
Bandits with Adversarial Rewards.
There exists a large body of literature on the problems ofadversarial multi-armed bandits (Auer et al., 2002; Bubeck and Cesa-Bianchi, 2012).
There is alsoa line of works trying to design algorithms that can achieve near-optimal regret bounds for bothstochastic bandits and adversarial bandits simultaneously (Bubeck and Slivkins, 2012; Seldin and
Slivkins, 2014; Auer and Chiang, 2016; Seldin and Lugosi, 2017; Zimmert and Seldin, 2019; Leeet al., 2021).
However, most of these algorithms focus on the general adversarial reward setting
0 ) -diverse if for any a = µ + ξ with µ ∈ R d and ξ ∼ D , λ min ( E ξ ∈D
[ aa ⊤ | θ ⊤ ξ ≥ b ]) ≥ λ 0 holds for all θ ∈ R d and b ∈ R satisfying b ≤ r ∥ θ ∥ 2 .
The lower bound under a large corruption level C ≥ √ K only holds for algorithms that can achieve nearoptimal regret for uncorrupted bandits.
It is possible for an algorithm that does not achieve the optimal regret for uncorrupted bandits (e.g., R K
= O
( K 0 . 75 ) ) to achieve a sub-linear regret in the presence of corruptions. without specifying the total amount of corruption. One of the notable exceptions is Lee et al.(2021), which assumed that the adversarial corruptions are generated through the inner product ofan adversarial vectors and the contextual vector.
As a comparison, our algorithm and result do notneed such additional assumption on the structure of the corruption.
Our algorithm can be applied toboth corrupted and uncorrupted settings with different choices of hyperparameters, and achieves anear-optimal regret for both cases.
Notation We use lower case letters to denote scalars, and use lower and upper case bold face lettersto denote vectors and matrices respectively.
We denote by [ n ] the set { 1 , . . . , n } .
For a vectorx ∈ R d and a positive semi-definite matrix Σ ∈ R d × d , we denote by ∥ x ∥ 2 the vector’s ℓ 2 normand by ∥ x ∥ Σ =
√ x ⊤ Σx the Mahalanobis norm.
For two positive sequences { a n } and { b n } withn = 1 , 2 , . . .
, we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb nholds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such thata n ≥ Cb n holds for all n ≥ 1 .
We use (cid:101) O ( · ) to further hide the polylogarithmic factors.
We use 1 {·}to denote the indicator function.
Preliminaries
In this section, we introduce the setting of linear contextual bandit with adversarial corruption.
Linear contextual bandit with corruption.
We define linear contextual bandits with corruption asfollows: at the beginning of each round k ∈ [ K ] , the agent receives a decision set D k ∈ R d from theenvironment and it chooses an action (i.e., arm, contextual vector) x ∈ D k .
After choosing the actionx k at round k , the environment generates the corresponding r k based on the stochastic linear modelr k
= ⟨ θ ∗ , x ⟩ + η k , where θ ∗ ∈ R
d is an unknown environment parameter and η k is the stochasticnoise.
After seeing the stochastic reward r k , the adversary (i.e., attacker) introduces an adversarialcorruption c k onto the reward, which may depend on the decision set D k , action x k , stochastic rewardr k .
Finally, the agent observes the corrupted reward (cid:98) r k
=
⟨ θ ∗ , x ⟩ + η k + c k at round k .
Following
Abbasi-Yadkori et al.
(2011), we make the following assumptions on the bandit model.
Assumption 2.1.
The linear contextual bandit satisfies the following conditions:
• At each round k and any action x ∈ D k , we have ∥ x ∥ 2 ≤ L .
• For the unknown environment parameter θ ∗ , it satisfies ∥ θ ∗ ∥ 2 ≤ S .
• At each round k , the corresponding stochastic noise η k is conditional R -sub-Gaussian, i
e.,
[Equation]
[Equation] [Equation]
[Equation]
(cid:3) ≤ exp( R 2 λ 2 / 2) .
Regret.
The goal of the agent is to minimize the pseudo-regret in the first K rounds, which is definedas follows:
[Equation]
Corruption level.
To measure the level of adversarial corruptions, we define the corruption level as
C :=
(cid:80)
Kk =
| c k | .
With this definition, we say a linear contextual bandit problem is C -corrupted ifand only if the corruption level is no larger than C .
Remark 2.2.
The adversary in our setting and the corresponding definition of corruption level isthe same as that in Bogunovic et al.
(2021) and slightly different from that in prior works such as
Lykouris et al.
; Gupta et al. (2019b); Zhao et al.
More specifically, in these works, theadversarial corruption c k is chosen before the choice of action x k ∈ D k .
Since the actions selectedby the agent may not be deterministic, the adversary chooses different corruption
c k, x for differentaction x ∈ D k .
With this notion of corruption, the corresponding corruption level is defined as
C ′ =
[Equation] max x [Equation]
k | c k, x | .
As a comparison, our adversary chooses the corruption after observingthe action x k and for the corruption level.
We have
[Equation]
which implies that our corruption level C is always no larger than the corruption level C ′ in Lykouriset al.
Gupta et al. (2019b) Zhao et al.
Algorithms
In this section, we review existing algorithms for linear contextual bandits (and stochastic linearbandits) and discuss their limitations when they are applied to the adversarial corruption setting.
Thenwe present our algorithm CW-OFUL and illustrate how our algorithm design can overcome the theselimitations.
3.1 Existing Algorithms
We begin with reviewing the classical OFUL algorithm (Abbasi-Yadkori et al., 2011).
Under
Assumption 2.1, at round k , OFUL estimates θ ∗ by online ridge regression over all the past observedactions and rewards, i.e.,
[Equation]
With θ k in hand, OFUL constructs a confidence set for θ ∗ as follows C
[Equation]
,where β is the confidence radius and Σ k
= λ
I + (cid:80) k − 1 i =1 x i x ⊤ i is the covariance matrix of contextsx
i , i = 1 , . . .
, k .
Without corruption, it is known that setting β = (cid:101) O
( R √ d ) guarantees that θ ∗ ∈ C kwith high probability, which further leads to a sublinear regret (cid:101)
O ( d √ K ) .
However, with corruption,such a choice of β is not sufficient.
To see why, we take a closer look at the closed-form solution θ kto (3.1):
[Equation]
By simple calculation and assuming λ to be a constant, we can show that ∥ θ k − θ ∗ ∥ Σ k can be upperbounded by [Equation] θ [Equation]
Σ k
[Equation]
O (cid:18) (cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88) i
= x
i η i (cid:13)(cid:13)(cid:13)(cid:13)
Σ − 1 k (cid:124) (cid:123)(cid:122) (cid:125) I+ (cid:13)(cid:13)(cid:13)(cid:13)
k − 1 (cid:88)
i =
x i c i (cid:13)(cid:13)(cid:13)(cid:13) Σ − 1 k
(cid:124) (cid:123)(cid:122) (cid:125) I(cid:19) .
The first term I 1 is corruption-independent and bounded by (cid:101) O
( R √ d ) according to Abbasi-Yadkoriet al.
The challenge is to bound the second term I 2 , which depends on the corruption.
Existingapproaches (Zhao et al., 2021; Ding et al., 2021) bound I 2 by triangle inequality and Cauchy-Schwarzinequality,
[Equation]
where C is the corruption level and max 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k is bounded by the crude upper bound
L/ √ λ .
Unfortunately, such a bound makes the confidence radius be the order of O ( R √ d + C ) ,which eventually leads to an term O ( C √ K ) in the regret, which is C times worse than the regretwithout corruption.
In order to obtain a tighter bound of I 2 , for stochastic linear bandits, Bogunovic et al.
(2021)proposed a Robust Phase Elimination (RPE) algorithm, which employs optimal design (Lattimoreand Szepesvári, 2018) to select the arms.
In this setting, the decision set is finite and fixed over time,i.e., D k = D for all k ∈ [ K ] and |D| ≤ ∞ .
More specifically, RPE divides the time horizon intoseveral phases. Within each phase, RPE performs linear regression on a multiset A ⊂ D , which is the G-optimal design of D . Here the multiset means A has duplicate elements.
Let Σ be the covariancematrix defined over A , then the following upper bound holds (Lattimore and Szepesvári, 2018):
[Equation]
By choosing a large enough |A| , (3.3) provides a uniformly small upper bound formax 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k for any k .
Substituting (3.3) back into (3.2) with |A| =
O ( C 2 ) , we can Algorithm 1 CW- Require: Regularization parameter λ , confidence radius β and threshold parameter α 1: for round k
= 1 , 2 , .. do 2:
Set Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i 3:
Set
b k = (cid:80) k − 1 i =
w i x i r i and θ k
= Σ − 1 k b k 4:
Receive the decision set D k 5:
Choose action x k ← argmax x ∈D k θ ⊤ k x + β (cid:113) x ⊤ Σ − 1 k x 6:
Set w k = min { 1 , α/ ∥ x k ∥ Σ − 1 k } 7: end for show that I 2 is bounded by some small constant, which therefore eliminates the O ( C √ K ) term in thefinal regret.
Although the optimal design-based approach RPE (Bogunovic et al., 2021) successfullyeliminates the multiplicative term C √ K , it is not applicable to our linear contextual bandit setting:(1) it needs to select a multiset from the decision set, which is impossible for the general contextualbandit setting; (2) the complexity of optimal design introduces some additional quadratic term C 2 intheir final regret, which makes their algorithm non-optimal (See Bogunovic et al.
(2021) for moredetails).
3.2 Our Algorithm
As we have seen before, it is pivotal to bound the corruption-dependent term
I 2 tightly.
To overcomethe limitations of existing approaches, we propose a fundamentally new approach and present
our
CW-OFUL in Algorithm 1.
At a high level, Algorithm 1 is an extension of the OFUL algorithm(Abbasi-Yadkori et al., 2011), which is also based on the principle of optimism in the face ofuncertainty.
Our algorithm assigns a weight w k to each selected action x k .
More specifically, at round k , we usethe following weighted ridge regression to estimated the unknown vector θ ∗ :
[Equation]
The closed-form solution to the above optimization problem is displayed in Line 3 of Algorithm 1.
While weighted ridge regression is not new and has been used in prior work on bandits (Kirschner and
Krause, 2018; Zhou et al., 2021; Russac et al., 2019), the setting, motivation and the choice of weightare fundamentally different.
More specifically, we choose the weight as the truncation of the inverseexploration bonus, which is w k = min (cid:110) 1 , α/ ∥ x k ∥ Σ − 1 k (cid:111) .
Here α > 0 is a threshold parameter.
Wecan see that for action x k with a large exploration bonus ∥ x k ∥ Σ − 1 k (low confidence), CW-OFUL willassign a small weight to it to avoid the potentially large regret caused by both the stochastic noise andthe adversarial corruption.
On the other hand, for the action with a small exploration bonus (highconfidence), CW-OFUL will assign a large weight to it (it can be as large as 1 ).
Another interestingobservation is that by setting α to be sufficiently large, the weight will become 1 for every action,and CW-OFUL will degenerate to OFUL (Abbasi-Yadkori et al., 2011).
As a comparison, Kirschner and Krause (2018)
; Zhou et al.
(2021) used the inverse of the noisevariance as the weight to normalize the noise and derived tight variance-dependent regret guarantees.
Russac et al.
(2019) set the weight as a geometric sequence to perform moving average to deal withthe non-stationary environment.
To see how our choice of weight can lead to tighter regret, we first write down the closed-formsolution to (3.4) [Equation] +
[Equation]
where the covariance matrix Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i .
With some calculation and assuming λ tobe a constant, we can obtain
[Equation]
I 1 is the corruption-independent term and can still be bounded by (cid:101)
O ( R √ d ) according to AbbasiYadkori et al.
For I 2 , we have
[Equation]
It is evident that with our carefully designed weight, the corruption-dependent term I 2 can beuniformly bounded by some constant Cα , the same as that in Bogunovic et al. (2021).
Therefore, bysetting α to be sufficiently small, our CW-OFUL can get rid of the C √ K term in the final regret.
Main Results
In this section, we present the main theoretical guarantees of CW-OFUL.
4.1 Known Corruption Level C
We first consider the case when C is known to the agent.
In this case, we choose α = R √ d/C .
The following lemma characterizes the estimation error of θ k with respect to θ ∗ , which is a formalsummary of our discussion in Section 3.
Lemma 4.1.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 ,set the confidence radius β = R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) + √ λS + αC in Algorithm 1, then withprobability at least 1 − δ , for every round k , the estimator θ k satisfies that ∥ θ k − θ ∗ ∥
Σ k ≤ β .
The following theorem provides the regret bound of Algorithm 1.
Theorem 4.2.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 ,set the confidence radius β in Algorithm 1 as follows:
[Equation]
Then with probability at least 1 − δ , its regret in the first K rounds is upper bounded by
Regret ( K ) =
O
[Equation]
+ dC (cid:113) log 2 (cid:0) (1 + KL
[Equation]
[Equation]
) /δ (cid:1)(cid:19) .
In addition, if choosing α =
( R √ d + √ λS ) /C
and λ =
R 2 /S 2 , its regret can be upper bounded by
[Equation]
A few remarks about Theorem 4.2 are in order.
Remark 4.3.
Compared with the (cid:101) O
( d √ K + dC √ K ) regret proved in Zhao et al.
(2021); Dinget al.
(2021), our algorithm improves the multiplicative dependence on corruption level C to additivedependence.
In particular, CW-OFUL achieves the same order of regret as the uncorrupted settingwhen C
=
O ( √ K ) , and it attains a sublinear regret as long as C
= o ( K ) .
In sharp contrast, thealgorithm proposed in Zhao et al.
(2021) achieves the same order of regret as the uncorrupted settingonly
when C =
O (1) , and has a sublinear regret only when
C = o ( √ K ) .
Remark 4.4.
We also compare our result with that in Wei et al.
The Robust + OFULalgorithm in Wei et al.
(2022) achieves an (cid:101)
O
( d √ K + C r ) regret with C r = (cid:113) T (cid:80)
Kk =1 c 2 k , whichwill degenerate to (cid:101) O
( d √ K + d √ KC ) in the worst case.
Their regret guarantee is always worsethan ours when C < √ K .
In addition, according to the discussion in Remark 2.2, Theorem 4.2 alsoimplies an (cid:101) O (cid:0) √ dK + dC ′ (cid:1) regret under the notion of the corruption level C ′ .
In contrast, the Robust
VOFUL algorithm in Wei et al.
(2022) has an (cid:101) O
( d 4 . 5 √ K + d 4 C ′ ) regret, which is also inferior toour regret.
Furthermore, Robust VOFUL is computationally inefficient.
Remark 4.5.
We further compare our result with previous additive regrets derived for stochasticlinear bandits.
Let D k
= D be the decision set.
Compared with the O ( (cid:112) dK log |D| + Cd 3 / 2 ) regretfor stochastic linear bandit with corruption derived in Bogunovic et al.
(2021), our regret improvesthe corruption term by a factor of √ d .
Note that the √ d difference in the leading √ K term betweenour regret and theirs is caused by the fact that Bogunovic et al.
(2021) considered the finite-armsetting, while we consider the infinite-arm setting.
Our algorithm will have the same regret as theirswhen |D| =
O ( exp ( d )) .
Remark 4.6.
For the uncorrupted setting where C = 0 , Theorem 4.2 suggests that the thresholdparameter α should be set to infinity.
Then by Line 6 in Algorithm 1, each weight w k becomes 1 , and
CW-OFUL degenerates to OFUL.
Meanwhile, the regret in Theorem 1 also becomes (cid:101) O
( d √ K ) thatmatches the regret of OFUL (Abbasi-Yadkori et al., 2011).
Lower bound.
Next we refer to two existing lower bound results to show that when C is known,our (cid:101) O ( d √ K + dC ) regret is optimal up to logarithmic factors.
The first proposition shows that the(cid:101) O
( d √ K ) corruption-independent term in our regret is near-optimal.
Proposition 4.7 (Theorem 24.2, Lattimore and Szepesvári 2018) .
Assume d ≤ 2 K , R = 1 and
D k
= {∥ x ∥ 2 ≤ 1 } for all k ≥ 1 .
Then for any algorithm, there exists a environment parameter vectorθ ∗ ∈ R d satisfying ∥ θ ∗ ∥
22 = d 2 / (48 K ) such that E ( Regret ( K )) ≥ d √ K/ (16 √ 3) .
The second proposition suggests that the O ( dC ) corruption term in our regret is optimal.
Proposition 4.8 (Theorem 3, Bogunovic et al. 2021) .
For any dimension d , for any algorithm thathas the knowledge of C , there exists an instance satisfying with probability at least 0 .
5 , Regret ( K ) =
Ω( dC ) .
Combining Propositions 4.7 and 4.8, we can conclude that for any algorithm, there exists a corruptedbandit instance such that the algorithm suffers at least Ω(max { d √ K, dC } ) regret.
Such a lowerbound matches our upper bound up to logarithmic factors.
Therefore, our algorithm is nearly optimal.
Misspecified linear bandits.
We consider the misspecified linear bandit setting which assumes thatthe corruption at each round is uniformly bounded by ϵ .
Clearly, the misspecified linear bandit is aspecial case of corrupted linear contextual bandit with C = Kϵ . Theorem 4.2 suggests that a directapplication of our algorithm to this special setting incurs an (cid:101) O ( d √ K + dKϵ ) regret, which differsfrom the near-optimal regret (cid:101) O
( d √ K + √ dKϵ ) (Lattimore and Szepesvari, 2019; Foster et al.,
2020)by a √ d factor on the corruption term.
Whether our algorithm is able to achieve the near-optimalregret for both misspecified linear bandit and corrupted linear contextual bandit simultaneouslyremains an open question.
Instance-dependent regret bounds.
CW-OFUL also enjoys an instance-dependent regret bound.
Due to space limit, we defer it to Appendix A.
4.2 Unknown Corruption Level C
Now we consider the case when C is unknown.
Our solution is quite simple for this case: weintroduce a tuning parameter ¯ C , which can be viewed as an estimate of C , and select the thresholdparameter α as Theorem 4.2 suggests.
The following theorem gives the regret upper bound of
CW-OFUL for the unknown C case.
Theorem 4.9.
Under the same conditions of Theorem 4.2 except that we set α = ( R √ d + √ λS ) / ¯ Cwith ¯ C being an estimated corruption level, λ = R 2 /S 2 and β = 2 R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) +2 √ λS in Algorithm 1.
Its regret can be upper bounded by
• If 0 ≤ C ≤ ¯ C , then with probability at least 1 − δ , we have Regret ( K ) = (cid:101)
O ( dR √ K + d ¯ C ) .• If C > ¯ C , we have Regret ( K ) =
O ( K ) .
In addition, if we set ¯ C = √ K , then when 0 ≤ C ≤ √ K , the regret is upper bounded by (cid:101)
O
( d √ K ) .
Remark 4.10.
Zhao et al.
(2021) proposed an (cid:101)
O ( C 2 d √ K ) regret with unknown C = Ω(1) .
Compared with their result, our regret (with ¯ C = √ K ) is strictly better in the corruption term.
Bogunovicet al.
(2021) proposed an (cid:101) O
( (cid:112) dK log |D| + Cd 1 .
5 + C 2 ) regret for the stochastic linear banditwith unknown C , in the regime C
= (cid:101) O ( √ K/d ) , where D is the finite decision set.
Such a regretbecomes (cid:101) O ( d √ K + Cd 1 .
5 + C 2 ) when the size of D becomes exponentially large in d or eveninfinite.
Compared with their regret, our regret is not only smaller, but also holds for a wider regime(i.e.
, C
= O ( √ K ) ).
Compared with the greedy algorithm in Bogunovic et al.
(2021), our result doesnot rely on the stringent ( r, λ 0 ) -diverse property assumption on the contexts.
Remark 4.11.
We also compare our result (choosing ¯ C = √ K ) with those in Wei et al.
(2022) for theunknown C case.
Wei et al.
(2022) proposed a COBE + OFUL algorithm with an (cid:101) O
( d √ K + C r ) regret,and a COBE + VOFUL algorithm with an (cid:101)
O ( d 4 .
5 √ K + d 4 C ′ ) regret, analogous to their results for theknown C case discussed in Remark 4.4.
Our CW-OFUL enjoys a better regret than COBE + OFULfor all C , and it is better than COBE + VOFUL for C < √ K .
In addition, for the modified notion ofcorruption level C ′ , if we choose the basic algorithm in COBE (Wei et al., 2022) as our CW-OFULalgorithm, then Theorem 3 in Wei et al.
(2022) suggests that COBE+CW-OFUL can deal withunknown corruption level C ′ and obtained an (cid:101)
O ( d √ K + dC ′ ) regret guarantee, which matches theregret of CW-OFUL algorithm with known corruption level C ′ .
Note that COBE + VOFUL is alsocomputationally inefficient.
With ¯ C = √ K , for the case when 0 ≤ C ≤ √ K , our regret result is already near-optimal, due tothe lower bound for the uncorrupted bandit in Proposition 4.7.
Now we show that our O ( K ) bound,seemingly trivial, is actually optimal for a large class of bandit algorithms.
In detail, the followingtheorem provides a lower bound result for any algorithm for the unknown C case.
This is an extensionof the lower bound result in Bogunovic et al.
from d = 2 to general d .
Theorem 4.12.
For any algorithm, let R K be an upper bound of Regret ( K ) such that for any banditinstance satisfying Assumption 2.1 with C = 0 , it satisfies the E (cid:2)
Regret ( K ) (cid:3) ≤ R K ≤ O ( K ) , wherethe expectation is with respect to the possible randomness of the algorithm and the stochastic noise.
Then for the general case with C = Ω( R K /d ) , such an algorithm will have E (cid:2) Regret ( K ) (cid:3)
=
Ω( K ) .
Remark 4.13.
Consider a class of algorithms A whose worst-case regret is R K in the uncorruptedcase. Here we only need to consider Ω( d √ K ) ≤ R K ≤ O ( K ) , since for any algorithm, Ω( d √ K )is the lowest possible worst-case regret (Lattimore and Szepesvári, 2018) and O ( K ) is the highestpossible regret. We first show that CW-OFUL belongs to A .
Choosing ¯ C = R K /d , Theorem 4.9immediately suggests that CW-OFUL enjoys a R K regret in the uncorrupted case (i.e., C = 0 ). Thus CW-OFUL belongs to A .
Then we will show that CW-OFUL is the best possible one in A . Onthe one hand, Theorem 4.9 suggests that CW-OFUL suffers a linear regret when C > ¯ C = R K /d . On the other hand, Theorem 4.12 shows that any algorithm with R K regret in the uncorrupted caseshould have a linear regret when C = Ω( R K /d ) . These together imply that CW-OFUL is optimalwithin A .
Conclusion and Future Work
In this work, we study corrupted linear contextual bandits.
We propose a CW-OFUL algorithm basedon a weighted ridge regression with truncated inverse exploration bonus weights.
We show that forboth cases when the corruption level C is known or unknown to the agent, CW-OFUL achieves aregret that matches the lower bound up to logarithmic factors.
We are also interested in achieving the optimal regret when specializing our algorithm to the misspecified linear contextual bandits.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We study corrupted linear contextual bandits andpropose a computation efficient algorithm with a near-optimal regret guarantee.(b)
Did you describe the limitations of your work?
[Yes] See the discussion aboutMisspecified linear bandits in subsection 4.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our workfocuses on a purely theoretical problem and does not have any negative social impact.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption2.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]