Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptions
Abstract
We study the linear contextual bandit problem in the presence of adversarial corruption, where the reward at each round is corrupted by an adversary, and the corruption level (i.e., the sum of corruption magnitudes over the horizon) is
C ≥ 0 .
The best-known algorithms in this setting are limited in that they either are computationally inefficient or require a strong assumption on the corruption, or their regret is at least C times worse than the regret without corruption.
In this paper, to overcome these limitations, we propose a new algorithm based on the principle of optimism in the face of uncertainty.
At the core of our algorithm is a weighted ridge regression where the weight of each chosen action depends on its confidence up to some threshold.
We show that for both known C and unknown
C cases, our algorithm with proper choice of hyperparameter achieves a regret that nearly matches the lower bounds.
Thus, our algorithm is nearly optimal up to logarithmic factors for both cases.
Notably, our algorithm achieves the near-optimal regret for both corrupted and uncorrupted cases ( C = 0 ) simultaneously.
Introduction
We study linear contextual bandits with adversarial corruptions.
At each round, the agent observes a decision set provided by the environment and selects an action from the decision set.
Then an adversary corrupts the reward of the action selected by the agent.
The agent then receives the corrupted reward of the selected action and proceeds until K rounds.
The agent’s goal is to minimize the regret Regret ( K ) , which is the difference between the optimal accumulated reward and the selected accumulated reward.
This problem can be regarded as a combination of the two classical bandit problems, stochastic bandits and adversarial bandits (Lattimore and Szepesvári, 2018).
In practice, the contextual bandits with adversarial corruptions can describe many popular decisionmaking problems such as pay-per-click advertisements with click fraud (Lykouris et al., 2018) and recommendation system with malicious users (Deshpande and Montanari, 2012).
Lykouris et al.
(2018) first studied the multi-armed bandit with adversarial corruptions.
Specifically, let C denote the corruption level which is the sum of the corruption magnitudes at each round.
Lykouris et al.
(2018) proposed an algorithm with a regret that is C times worse than the regret without corruption.
Later, Gupta et al.
(2019a) proposed an improved algorithm whose regret consists of two terms: a corruption-independent term that matches the optimal regret for multi-armed bandit without corruption, and a corruption-dependent term that is linear in C and independent of K , i.e.,
Regret ( K ) =
o ( K ) + O ( C ) .
The lower bound proved in Gupta et al.
(2019a) suggests that the linear dependence on C is near-optimal.
Such a regret structure reveals an desirable property of corruption-robust bandit algorithms, that is, the algorithm should perform nearly the same as the bandit algorithms without corruption when the corruption level C is small or diminishes.
Based on the above observation, a natural question arises:
Can we design computationally efficient algorithms for linear contextual bandits with corruption that can attain the best possible regret, similar to those in multi-armed bandits?
Some previous works have attempted to answer this question for the simpler stochastic linear bandit setting, where the decision sets at each round are identical and finite.
Li et al.
(2019) studied the stochastic linear bandits and proposed an instance-dependent regret bound.
Later on, Bogunovic et al.
(2021) studied the same problem and proposed an algorithm that achieves a regret with the corruption term depending on C linearly and on K logarithmically.
However, these algorithms are limited to the stochastic linear bandit setting since their algorithm design highly relies on the experiment design and arm-elimination techniques that require a multiple selection of the same action and can only handle fixed decision set.
They are not applicable to contextual bandits, where the decision set is changing over time and can even be infinite.
For the more general linear contextual bandit setting,
Bogunovic et al.
(2021) proved that a simple greedy algorithm based on linear regression can attain an ideal corruption term that has a linear dependence on C and a logarithmic dependence on K , under a stringent diversity assumption on the contexts.
Lee et al.
(2021) proposed an algorithm and the corruption term in its regret depends on C linearly and on K logarithmically, but only holds for the restricted case when the corruption at each round is a linear function of the action.
Without special assumptions on the contexts or corruptions, Zhao et al.
; Ding et al.
(2021) proposed a variant of the OFUL algorithm (Abbasi-Yadkori et al., 2011) and its regret has a corruption term depending on K polynomially.
Recently, Wei et al.
(2022) proposed a Robust VOFUL algorithm that achieves a regret with a corruption term linearly dependent on C ′ 1 and only logarithmically dependent on
K .
However, Robust VOFUL is computationally inefficient since it needs to solve a maximization problem over a nonconvex confidence set that is defined as the intersection of exponential number of sets, and its regret has a loose dependence on context dimension d .
In addition, Wei et al.
(2022) also proposed a Robust OFUL algorithm and provided a regret guarantee that has a linear dependence on a different notion of corruption level C r 2 , which is strictly larger than the corruption level C considered in the previous work and the current paper.
Thus, the above question remains open.
In this paper, we give an affirmative answer to the above question.
We summarize our contributions as follows.
• We propose a computationally efficient algorithm based on the principle of optimism in the face of uncertainty (Abbasi-Yadkori et al., 2011), named Confidence-Weighted OFUL (CW-OFUL).
At the core of our algorithm is a weighted ridge regression where the weight of each chosen arm is adaptive to its confidence, which is defined as the truncation of the inverse exploration bonus.
Intuitively, such a weighting strategy prevents the algorithm from exploiting the contexts whose rewards are more likely corrupted by a large amount.
• For the case when the corruption level C is known to the agent, we show that the proposed algorithm enjoys a regret Regret ( K )
=
(cid:101) O
( d √ K + dC ) , where d is the dimension of the contexts, C is the corruption level and K is the number of total iterations.
The first term matches the regret lower bound of linear contextual bandits without corruption Ω( d √ K ) (Lattimore and Szepesvári, 2018).
The second term matches the lower bound on the corruption term in regret Ω( dC ) (Bogunovic et al., 2021).
They together suggest that our algorithm is not only robust but also near-optimal up to logarithmic factors.
• For the case when the corruption level C is unknown to the agent, we show that CW-OFUL enjoys an (cid:101) O
( d √ K ) regret for the case C ≤ √ K , with proper choice of the hyperparameter.
Surprisingly,
by proving a lower bound on the regret
[Table]
we show that our regret upper bound is already optimal for all algorithms that achieve a near-optimal regret bound for uncorrupted bandits.
We compare our regret bounds with previous ones in Table 1.
We can see that our algorithm matches the lower bound up to logarithmic factors in both known C and unknown C cases, and therefore is nearly optimal.
[Table]
[Equation] [Equation]
Unknown
Yes
Strong
[Equation]
OFUL (Remark 4.
[Equation]
Unknown
Yes
Weak
Lower bound( Lattimore and Szepesvári 2018)
[Equation]
Unknown
N/
[Equation]
Strong (Theorem 4.12)
[Equation]
1.1 Additional Related Work
Bandits with Misspecification.
Bandits with misspecification can be seen as a special case of bandit with adversarial corruption since it is corrupted relative evenly at each round.
Let ϵ be the misspecification level.
Ghosh et al.
(2017) firstly studied the stochastic linear bandits and proved a sublinear regret when ϵ is small.
Lattimore and Szepesvari (2019) studied the stochastic linear bandit θ ∈ R d and b ∈ R satisfying b ≤ r ∥ θ ∥ 2 .
The lower bound under a large corruption level C ≥ √ K only holds for algorithms that can achieve nearoptimal regret for uncorrupted bandits.
It is possible for an algorithm that does not achieve the optimal regret for uncorrupted bandits (e.g., R K
= O
( K 0 . 75 ) )
to achieve a sub-linear regret in the presence of corruptions.
setting under milder assumptions.
With the knowledge of ϵ , they proposed an algorithm with an (cid:101) O
( (cid:112) dK log( N )+ ϵ √ dK ) regret, where d is the dimension of the contextual vector, N is the number of arms.
Their regret bound matches their proved lower bound up to logarithmic factors.
Foster et al.
(2020) further considered the more general linear contextual bandits with misspecification when ϵ is unknown to the agent, and proposed an algorithm equipped with a CORRAL meta algorithm
(Agarwal et al., 2017) to deal with the unknown ϵ .
Their algorithm enjoys an (cid:101)
O
( d √ K + ϵ √ dK ) regret.
Krishnamurthy et al.
(2021) proposed an algorithm without using a meta algorithm which has the same order of regret as Foster et al.
Our algorithm can be directly applied to the misspecification setting by choosing the corruption level C to be Kϵ , which immediately gives us an (cid:101)
O ( d √ K + dKϵ ) regret upper bound.
Bandits with Adversarial Rewards.
There exists a large body of literature on the problems of adversarial multi-armed bandits (Auer et al., 2002; Bubeck and Cesa-Bianchi, 2012).
There is also a line of works trying to design algorithms that can achieve near-optimal regret bounds for both stochastic bandits and adversarial bandits simultaneously (Bubeck and Slivkins, 2012; Seldin and
Slivkins, 2014; Auer and Chiang, 2016; Seldin and Lugosi, 2017; Zimmert and Seldin, 2019; Lee et al., 2021).
However, most of these algorithms focus on the general adversarial reward setting without specifying the total amount of corruption.
One of the notable exceptions is Lee et al.
(2021), which assumed that the adversarial corruptions are generated through the inner product of an adversarial vectors and the contextual vector.
As a comparison, our algorithm and result do not need such additional assumption on the structure of the corruption.
Our algorithm can be applied to both corrupted and uncorrupted settings with different choices of hyperparameters, and achieves a near-optimal regret for both cases.
Notation We use lower case letters to denote scalars, and use lower and upper case bold face letters to denote vectors and matrices respectively.
We denote by [ n ] the set { 1 , . . . , n } .
For a vector x ∈ R d and a positive semi-definite matrix Σ ∈ R d × d , we denote by ∥ x ∥ 2 the vector’s ℓ 2 norm and by ∥ x
∥ Σ = √ x ⊤
Σx the Mahalanobis norm.
For two positive sequences { a n } and { b n } with n = 1 , 2 , . . .
, we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb n holds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such that a n ≥ Cb n holds for all n ≥ 1 .
We use (cid:101) O ( · ) to further hide the polylogarithmic factors.
We use 1 {·} to denote the indicator function.
Preliminaries
In this section, we introduce the setting of linear contextual bandit with adversarial corruption.
Linear contextual bandit with corruption.
We define linear contextual bandits with corruption as follows: at the beginning of each round k ∈ [ K ] , the agent receives a decision set D k ⊆ R d from the environment and it chooses an action (i.e., arm, contextual vector) x ∈ D k . After choosing the action x k at round k , the environment generates the corresponding r ′ k
based on the stochastic linear model r ′ k = ⟨ θ ∗ , x ⟩ + η k , where θ ∗ ∈ R
d is an unknown environment parameter and η k is the stochastic noise.
After seeing the stochastic reward r ′ k , the adversary (i.e., attacker) introduces an adversarial corruption c k onto the reward, which may depend on the decision set D k , action x k , stochastic reward r ′ k .
Finally, the agent observes the corrupted reward r k = ⟨ θ ∗ , x ⟩ + η k + c k at round k .
Following
Abbasi-Yadkori et al.
(2011), we make the following assumptions on the bandit model.
Assumption 2.1.
The linear contextual bandit satisfies the following conditions:
• At each round k and any action x ∈ D k , we have ∥ x ∥ 2 ≤ L .
• For the unknown environment parameter θ ∗ , it satisfies ∥ θ ∗ ∥ 2 ≤ S .
• At each round k , the corresponding stochastic noise η k is conditional R -sub-Gaussian, i
e.,
[Equation]
Regret.
The goal of the agent is to minimize the pseudo-regret in the first K rounds, which is defined as follows:
[Equation]
Corruption level.
To measure the level of adversarial corruptions, we define the corruption level as
C :=
(cid:80)
Kk =
| c k | .
With this definition, we say a linear contextual bandit problem is C -corrupted if and only if the corruption level is no larger than C .
Remark 2.2.
The adversary in our setting and the corresponding definition of corruption level is the same as that in Bogunovic et al.
(2021) and slightly different from that in prior works such as
Lykouris et al.
; Gupta et al. (2019b); Zhao et al.
More specifically, in these works, the adversarial corruption c k is chosen before the choice of action x k ∈ D k .
Since the actions selected by the agent may not be deterministic, the adversary chooses different corruption
c k, x for different action x ∈ D k .
With this notion of corruption, the corresponding corruption level is defined as
[Equation]
| .
As a comparison, our adversary chooses the corruption after observing the action x k and for the corruption level.
We have
[Equation]
which implies that our corruption level C is always no larger than the corruption level C ′ in Lykouris et al.
; Gupta et al. (2019b); Zhao et
[Equation]
Algorithms
In this section, we review existing algorithms for linear contextual bandits (and stochastic linear bandits) and discuss their limitations when they are applied to the adversarial corruption setting.
Then we present our algorithm CW-OFUL and illustrate how our algorithm design can overcome the these limitations.
3.1 Existing Algorithms
We begin with reviewing the classical OFUL algorithm (Abbasi-Yadkori et al., 2011).
Under
Assumption 2.1, at round k , OFUL estimates θ ∗ by online ridge regression over all the past observed actions and rewards, i.e.,
[Equation]
With θ k in hand, OFUL constructs a confidence set for θ ∗ as follows C k
= (cid:110) θ
[Equation]
∥ θ
[Equation]
, where β is the confidence radius and Σ k
= λ
I + (cid:80) k − 1 i =1 x i x ⊤ i is the covariance matrix of contexts x
i , i = 1 , . . .
, k .
Without corruption, it is known that setting β = (cid:101) O
( R √ d ) guarantees that θ ∗ ∈ C k with high probability, which further leads to a sublinear regret (cid:101) O ( d √ K ) .
However, with corruption, such a choice of β is not sufficient.
To see why, we take a closer look at the closed-form solution θ k to (3.1):
[Equation]
By simple calculation and assuming λ to be a constant, we can show that ∥ θ k − θ ∗ ∥ Σ k can be upper bounded by
[Equation]
− 1 k (cid:124) (cid:123)(cid:122) (cid:125) I(cid:19) .
The first term I 1 is corruption-independent and bounded by (cid:101) O
( R √ d ) according to Abbasi-Yadkori et al.
The challenge is to bound the second term I 2 , which depends on the corruption.
Existing approaches (Zhao et al., 2021; Ding et al., 2021) bound I 2 by triangle inequality and Cauchy-Schwarz inequality,
[Equation]
Algorithm
Require: Regularization parameter λ , confidence radius β and threshold parameter α 1: for round k
= 1 , 2 , .. do 2:
Set Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i 3:
Set
b k = (cid:80) k − 1 i =
w i x i r i and θ k
= Σ − 1 k b k 4:
Receive the decision set D k 5:
Choose action x k ← argmax x ∈D k θ ⊤ k x + β (cid:113) x ⊤ Σ − 1 k x 6:
Set w k = min { 1 , α/ ∥ x k ∥ Σ − 1 k } 7: end for where C is the corruption level and max 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k is bounded by the crude upper bound
L/ √ λ .
Unfortunately, such a bound makes the confidence radius be the order of O ( R √ d + C ) , which eventually leads to an term O ( C √ K ) in the regret, which is C times worse than the regret without corruption.
In order to obtain a tighter bound of I 2 , for stochastic linear bandits, Bogunovic et al.
(2021) proposed a Robust Phase Elimination (RPE) algorithm, which employs optimal design (Lattimore and Szepesvári, 2018) to select the arms.
In this setting, the decision set is finite and fixed over time, i.e., D k
= D for all k ∈
[ K ] and |D| ≤ ∞ .
More specifically, RPE divides the time horizon into several phases.
Within each phase, RPE performs linear regression on a multiset A ⊂ D , which is the
G-optimal design of D .
Here the multiset means A has duplicate elements.
Let Σ be the covariance matrix defined over A , then the following upper bound holds (Lattimore and Szepesvári, 2018):
[Equation]
By choosing a large enough |A| , (3.3) provides a uniformly small upper bound for max 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k for any k .
Substituting (3.3) back into (3.2) with |A| =
O ( C 2 ) , we can show that I 2 is bounded by some small constant, which therefore eliminates the O ( C √ K ) term in the final regret.
Although the optimal design-based approach RPE (Bogunovic et al., 2021) successfully eliminates the multiplicative term C √ K , it is not applicable to our linear contextual bandit setting: (1) it needs to select a multiset from the decision set, which is impossible for the general contextual bandit setting; (2) the complexity of optimal design introduces some additional quadratic term C 2 in their final regret, which makes their algorithm non-optimal (See Bogunovic et al.
(2021) for more details).
3.2 Our Algorithm
As we have seen before, it is pivotal to bound the corruption-dependent term
I 2 tightly.
To overcome the limitations of existing approaches, we propose a fundamentally new approach and present
our
CW-OFUL in Algorithm 1.
At a high level, Algorithm 1 is an extension of the OFUL algorithm (Abbasi-Yadkori et al., 2011), which is also based on the principle of optimism in the face of uncertainty.
Our algorithm assigns a weight w k to each selected action x k .
More specifically, at round k , we use the following weighted ridge regression to estimate the unknown vector θ ∗ :
[Equation]
The closed-form solution to the above optimization problem is displayed in Line 3 of Algorithm 1.
While weighted ridge regression is not new and has been used in prior work on bandits (Kirschner and
Krause, 2018; Zhou et al., 2021; Russac et al., 2019), the setting, motivation and the choice of weight are fundamentally different.
More specifically, we choose the weight as the truncation of the inverse exploration bonus, which is w k = min (cid:110) 1 , α/ ∥ x k ∥ Σ − 1 k (cid:111) .
Here α > 0 is a threshold parameter.
We can see that for action x k with a large exploration bonus ∥ x
k ∥ Σ − 1 k (low confidence), CW-OFUL will assign a small weight to it to avoid the potentially large regret caused by both the stochastic noise and the adversarial corruption.
On the other hand, for the action with a small exploration bonus (high confidence), CW-OFUL will assign a large weight to it (it can be as large as 1 ).
Another interesting observation is that by setting α to be sufficiently large, the weight will become 1 for every action, and CW-OFUL will degenerate to OFUL (Abbasi-Yadkori et al., 2011).
As a comparison, Kirschner and Krause (2018)
; Zhou et al.
(2021) used the inverse of the noise variance as the weight to normalize the noise and derived tight variance-dependent regret guarantees.
Russac et al.
(2019) set the weight as a geometric sequence to perform moving average to deal with the non-stationary environment.
To see how our choice of weight can lead to tighter regret, we first write down the closed-form solution to (3.4)
[Equation]
where the covariance matrix Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i .
With some calculation and assuming λ to be a constant, we can obtain
[Equation]
I 1 is the corruption-independent term and can still be bounded by (cid:101)
O ( R √ d ) according to AbbasiYadkori et al.
For I 2 , we have
[Equation]
It is evident that with our carefully designed weight, the corruption-dependent term I 2 can be uniformly bounded by some constant Cα , the same as that in Bogunovic et al.
Therefore, by setting α to be sufficiently small, our CW-OFUL can get rid of the C √ K term in the final regret.
Main Results
In this section, we present the main theoretical guarantees of CW-OFUL.
4.1 Known Corruption Level C : Upper Bound
We first consider the case when C is known to the agent.
In this case, we choose α = R √ d/C .
The following lemma characterizes the estimation error of θ k with respect to θ ∗ , which is a formal summary of our discussion in Section 3.
Lemma 4.1.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 , set the confidence radius β = R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) + √ λS + αC in Algorithm 1, then with probability at least 1 − δ , for every round k , the estimator θ k satisfies that
∥ θ k − θ ∗ ∥
Σ k ≤ β .
The following theorem provides the regret bound of Algorithm 1.
Theorem 4.2.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 , set the confidence radius β in Algorithm 1 as follows:
[Equation]
Then with probability at least 1 − δ , its regret in the first K rounds is upper bounded by
Regret ( K ) =
O
[Equation]
+ dS √
[Equation]
α
[Equation]
In addition, if choosing α =
( R √ d + √ λS ) /C
and λ =
R 2 /S 2 , its regret can be upper bounded by
Regret ( K ) =
(cid:101) O
( d √ K + dC ) .
A few remarks about Theorem 4.2 are in order.
Remark 4.3.
Compared with the (cid:101) O
( d √ K + dC √ K ) regret proved in Zhao et al.
; Ding et al.
(2021), our algorithm improves the multiplicative dependence on corruption level C to additive dependence.
In particular, CW-OFUL achieves the same order of regret as the uncorrupted setting when
C
=
O
( √ K ) ,
and it attains a sublinear regret as long as C
= o ( K ) .
In sharp contrast, the algorithm proposed in Zhao et al.
(2021) achieves the same order of regret as the uncorrupted setting only when
C = O (1) , and has a sublinear regret only when C = o ( √ K ) .
Remark 4.4.
We also compare our result with that in Wei et al.
The Robust + OFUL algorithm in Wei et al.
(2022) achieves an (cid:101)
O
( d √ K + C r ) regret with C r = (cid:113) T (cid:80)
Kk =1 c 2 k , which will degenerate to (cid:101) O
( d √ K + d √ KC ) in the worst case.
Their regret guarantee is always worse than ours when C < √ K .
In addition, according to the discussion in Remark 2.2, Theorem 4.2 also implies an (cid:101) O (cid:0) d √ K + dC ′ (cid:1) regret under the notion of the corruption level C ′ .
In contrast, the Robust
VOFUL algorithm in Wei et al.
(2022) has an (cid:101) O
( d 4 . 5 √ K + d 4 C ′ ) regret, which is also inferior to our regret.
Furthermore, Robust VOFUL is computationally inefficient.
Remark 4.5.
We further compare our result with previous additive regrets derived for stochastic linear bandits.
Let D k
= D be the decision set.
Compared with the O ( (cid:112) dK log |D| + Cd 3 / 2 ) regret for stochastic linear bandit with corruption derived in Bogunovic et al.
(2021), our regret improves the corruption term by a factor of √ d .
Note that the √ d difference in the leading √ K term between our regret and theirs is caused by the fact that Bogunovic et al.
(2021) considered the finite-arm setting, while we consider the infinite-arm setting.
Our algorithm will have the same regret as theirs when |D| =
O ( exp ( d )) .
Remark 4.6.
For the uncorrupted setting where C = 0 , Theorem 4.2 suggests that the threshold parameter α should be set to infinity.
Then by Line 6 in Algorithm 1, each weight w k becomes 1 , and
CW-OFUL degenerates to OFUL.
Meanwhile, the regret in Theorem 1 also becomes (cid:101) O
( d √ K ) that matches the regret of OFUL (Abbasi-Yadkori et al., 2011).
4.2 Known Corruption Level C: Lower Bound
In this subsection, we refer to two existing lower bound results to show that when C is known, our (cid:101) O
( d √ K + dC ) regret is optimal up to logarithmic factors.
The first proposition shows that the (cid:101) O ( d √ K ) corruption-independent term in our regret is near-optimal.
Proposition 4.7 (Theorem 24.2, Lattimore and Szepesvári 2018) .
Assume d ≤ 2 K , R = 1 and
D k
= {∥ x ∥ 2 ≤ 1 } for all k ≥ 1 .
Then for any algorithm, there exists an environment parameter vector θ ∗ ∈ R d satisfying ∥ θ ∗ ∥
22 = d 2 / (48 K ) such that E ( Regret ( K )) ≥ d √ K/ (16 √ 3) .
The second proposition suggests that the O ( dC ) corruption term in our regret is optimal.
Proposition 4.8 (Theorem 3, Bogunovic et al. 2021) .
For any dimension d , for any algorithm that has the knowledge of C , there exists an instance satisfying with probability at least 0 .
5 , Regret ( K ) =
Ω( dC ) .
Combining Propositions 4.7 and 4.8, we can conclude that for any algorithm, there exists a corrupted bandit instance such that the algorithm suffers at least Ω(max { d √ K, dC } ) regret.
Such a lower bound matches our upper bound up to logarithmic factors.
Therefore, our algorithm is nearly optimal.
4.3 Unknown Corruption Level C : Upper Bound
Now we consider the case when C is unknown.
Our solution is quite simple for this case: we introduce a tuning parameter ¯ C , which can be viewed as an estimate of C , and select the threshold parameter α as Theorem 4.2 suggests.
The following theorem gives the regret upper bound of
CW-OFUL for the unknown C case.
Theorem 4.9.
Under the same conditions of Theorem 4.2 except that we set α = ( R √ d + √ λS ) / ¯ C with ¯ C being an estimated corruption level, λ = R 2 /S 2 and β = 2 R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) + 2 √ λS in Algorithm 1.
The regret of CW-OFUL can be upper bounded in the following two cases:
• If the corruption level C satisfies that 0 ≤ C ≤ ¯ C , then with probability at least 1 − δ , the regret is upper bounded by Regret ( K ) = (cid:101) O ( dR √ K + d ¯ C ) .
• If the corruption level C satisfies that C > ¯ C , the regret is upper bounded by Regret ( K )
=
O ( K ) .
In addition, if we set the estimation ¯ C = √ K , then when 0 ≤ C ≤ √ K , the regret is upper bounded by (cid:101)
O
( d √ K ) .
Remark 4.10.
Zhao et al.
(2021) proposed an (cid:101)
O ( C 2 d √ K ) regret with unknown C = Ω(1) .
Compared with their result, our regret (with ¯ C = √ K ) is strictly better in the corruption term.
Bogunovic et al.
(2021) proposed an (cid:101) O
( (cid:112) dK log |D| + Cd 1 .
5 + C 2 ) regret for the stochastic linear bandit with unknown C , in the regime C
= (cid:101) O ( √ K/d ) , where D is the finite decision set.
Such a regret becomes (cid:101) O ( d √ K + Cd 1 .
5 + C 2 ) when the size of D becomes exponentially large in d or even infinite.
Compared with their regret, our regret is not only smaller, but also holds for a wider regime (i.e., C
=
O
( √ K ) ).
Compared with the greedy algorithm in Bogunovic et al.
(2021), our result does not rely on the stringent ( r, λ 0 ) -diverse property assumption on the contexts.
Remark 4.11.
We also compare our result (choosing ¯ C = √ K ) with those in Wei et al.
(2022) for the unknown C case.
Wei et al.
(2022) proposed a COBE + OFUL algorithm with an (cid:101) O
( d √ K + C r ) regret, and a COBE + VOFUL algorithm with an (cid:101)
O ( d 4 .
5 √ K + d 4 C ′ ) regret, analogous to their results for the known C case discussed in Remark 4.4.
Our CW-OFUL enjoys a better regret than COBE + OFUL for all C , and it is better than COBE + VOFUL for C < √ K .
In addition, for the modified notion of corruption level C ′ , if we choose the basic algorithm in COBE (Wei et al., 2022) as our CW-OFUL algorithm, then Theorem 3 in Wei et al.
(2022) suggests that COBE+CW-OFUL can deal with unknown corruption level C ′ and obtained an (cid:101)
O ( d √ K + dC ′ ) regret guarantee, which matches the regret of CW-OFUL algorithm with known corruption level C ′ .
Note that COBE + VOFUL is also computationally inefficient.
4.4 Unknown Corruption Level C : Lower Bound
With ¯ C = √ K , for the case when 0 ≤ C ≤ √ K , our regret result is already near-optimal, due to the lower bound for the uncorrupted bandit in Proposition 4.7.
Now we show that our O ( K ) bound, seemingly trivial, is actually optimal for a large class of bandit algorithms.
In detail, the following theorem provides a lower bound result for any algorithm for the unknown C case.
This is an extension of the lower bound result in Bogunovic et al.
from d = 2 to general d .
Theorem 4.12.
For any algorithm Alg , let R K be an upper bound of Regret ( K ) such that for any bandit instance satisfying Assumption 2.1 with C = 0 , it satisfies the E (cid:2)
Regret ( K ) (cid:3) ≤ R K ≤ O ( K ) , where the expectation is with respect to the randomness of the algorithm and the stochastic noise.
Then for the general case with C = Ω( R K /d ) , such an algorithm will have E (cid:2) Regret ( K ) (cid:3)
=
Ω( K ) .
Remark 4.13.
If we selects the estimated corruption ¯ C = Ω( R K /d ) , Theorem 4.9 immediately implies that CW-OFUL enjoys a O ( R K ) regret when corruption level C < Ω( R K /d ) and O ( K ) regret when corruption level C ≥ Ω( R K /d ) .
Compared with the algorithm Alg , Theorem 4.
suggests that CW-OFUL is no worse than the algorithm Alg no matter whether the corruption level
C < Ω( R K /d ) .
More discussion can be found in Appendix A.1.
Conclusion and Future Work
In this work, we study corrupted linear contextual bandits.
We propose a CW-OFUL algorithm based on a weighted ridge regression with truncated inverse exploration bonus weights.
We show that for both cases when the corruption level C is known or unknown to the agent, CW-OFUL achieves a regret that matches the lower bound up to logarithmic factors.
We are also interested in achieving the optimal regret when specializing our algorithm to the misspecified linear contextual bandits.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers and area chair for their helpful comments.
JH, DZ and QG are supported in part by the National Science Foundation CAREER Award 1906169 and the Sloan
Research Fellowship.
TZ is supported in part by the GRF 16201320.
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We study corrupted linear contextual bandits and propose a computation efficient algorithm with a near-optimal regret guarantee.
(b) Did you describe the limitations of your work?
[Yes] See the discussion about Misspecified linear bandits in subsection 4.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work focuses on a purely theoretical problem and does not have any negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 2.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]