Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptions
Abstract
We study the linear contextual bandit problem in the presence of adversarial corruption, where the reward at each round is corrupted by an adversary, and the corruption level (i.e., the sum of corruption magnitudes over the horizon) is
C ≥ 0 .
The best-known algorithms in this setting are limited in that they either are computationally inefficient or require a strong assumption on the corruption, or their regret is at least C times worse than the regret without corruption.
In this paper, to overcome these limitations, we propose a new algorithm based on the principle of optimism in the face of uncertainty.
At the core of our algorithm is a weighted ridge regression where the weight of each chosen action depends on its confidence up to some threshold.
We show that for both known C and unknown
C cases, our algorithm with proper choice of hyperparameter achieves a regret that nearly matches the lower bounds.
Thus, our algorithm is nearly optimal up to logarithmic factors for both cases.
Notably, our algorithm achieves the near-optimal regret for both corrupted and uncorrupted cases ( C = 0 ) simultaneously.
Introduction
We study linear contextual bandits with adversarial corruptions.
At each round, the agent observes a decision set provided by the environment and selects an action from the decision set.
Then an adversary corrupts the reward of the action selected by the agent.
The agent then receives the corrupted reward of the selected action and proceeds until K rounds.
The agent’s goal is to minimize the regret Regret ( K ) , which is the difference between the optimal accumulated reward and the selected accumulated reward.
This problem can be regarded as a combination of the two classical bandit problems, stochastic bandits and adversarial bandits (Lattimore and Szepesvári, 2018).
In practice, the contextual bandits with adversarial corruptions can describe many popular decisionmaking problems such as pay-per-click advertisements with click fraud (Lykouris et al., 2018) and recommendation system with malicious users (Deshpande and Montanari, 2012).
Lykouris et al.
(2018) first studied the multi-armed bandit with adversarial corruptions.
Specifically, let C denote the corruption level which is the sum of the corruption magnitudes at each round.
Lykouris et al.
(2018) proposed an algorithm with a regret that is C times worse than the regret without corruption.
Later, Gupta et al.
(2019a) proposed an improved algorithm whose regret consists of two terms: a corruption-independent term that matches the optimal regret for multi-armed bandit without corruption, and a corruption-dependent term that is linear in C and independent of K , i.e.,
Regret ( K ) =
o ( K ) + O ( C ) .
The lower bound proved in Gupta et al.
(2019a) suggests that the linear dependence on C is near-optimal.
Such a regret structure reveals an desirable property of corruption-robust bandit algorithms, that is, the algorithm should perform nearly the same as the bandit algorithms without corruption when the corruption level C is small or diminishes.
Based on the above observation, a natural question arises:
Can we design computationally efficient algorithms for linear contextual bandits with corruption that can attain the best possible regret, similar to those in multi-armed bandits?
Some previous works have attempted to answer this question for the simpler stochastic linear bandit setting, where the decision sets at each round are identical and finite.
Li et al.
(2019) studied the stochastic linear bandits and proposed an instance-dependent regret bound.
Later on, Bogunovic et al.
(2021) studied the same problem and proposed an algorithm that achieves a regret with the corruption term depending on C linearly and on K logarithmically.
However, these algorithms are limited to the stochastic linear bandit setting since their algorithm design highly relies on the experiment design and arm-elimination techniques that require a multiple selection of the same action and can only handle fixed decision set.
They are not applicable to contextual bandits, where the decision set is changing over time and can even be infinite.
For the more general linear contextual bandit setting,
Bogunovic et al.
(2021) proved that a simple greedy algorithm based on linear regression can attain an ideal corruption term that has a linear dependence on C and a logarithmic dependence on K , under a stringent diversity assumption on the contexts.
Lee et al.
(2021) proposed an algorithm and the corruption term in its regret depends on C linearly and on K logarithmically, but only holds for the restricted case when the corruption at each round is a linear function of the action.
Without special assumptions on the contexts or corruptions, Zhao et al.
; Ding et al.
(2021) proposed a variant of the OFUL algorithm (Abbasi-Yadkori et al., 2011) and its regret has a corruption term depending on K polynomially.
Recently, Wei et al.
(2022) proposed a Robust VOFUL algorithm that achieves a regret with a corruption term linearly dependent on C ′ 1 and only logarithmically dependent on
K .
However, Robust VOFUL is computationally inefficient since it needs to solve a maximization problem over a nonconvex confidence set that is defined as the intersection of exponential number of sets, and its regret has a loose dependence on context dimension d .
In addition, Wei et al.
(2022) also proposed a Robust OFUL algorithm and provided a regret guarantee that has a linear dependence on a different notion of corruption level C r 2 , which is strictly larger than the corruption level C considered in the previous work and the current paper.
Thus, the above question remains open.
In this paper, we give an affirmative answer to the above question.
We summarize our contributions as follows.
• We propose a computationally efficient algorithm based on the principle of optimism in the face of uncertainty (Abbasi-Yadkori et al., 2011), named Confidence-Weighted OFUL (CW-OFUL).
At the core of our algorithm is a weighted ridge regression where the weight of each chosen arm is adaptive to its confidence, which is defined as the truncation of the inverse exploration bonus.
Intuitively, such a weighting strategy prevents the algorithm from exploiting the contexts whose rewards are more likely corrupted by a large amount.
• For the case when the corruption level C is known to the agent, we show that the proposed algorithm enjoys a regret Regret ( K )
=
(cid:101) O
( d √ K + dC ) , where d is the dimension of the contexts, C is the corruption level and K is the number of total iterations.
The first term matches the regret lower bound of linear contextual bandits without corruption Ω( d √ K ) (Lattimore and Szepesvári, 2018).
The second term matches the lower bound on the corruption term in regret Ω( dC ) (Bogunovic et al., 2021).
They together suggest that our algorithm is not only robust but also near-optimal up to logarithmic factors.
• For the case when the corruption level C is unknown to the agent, we show that CW-OFUL enjoys an (cid:101) O
( d √ K ) regret for the case C ≤ √ K , with proper choice of the hyperparameter.
Surprisingly,
by proving a lower bound on the regret
[Table]
we show that our regret upper bound is already optimal for all algorithms that achieve a near-optimal regret bound for uncorrupted bandits.
We compare our regret bounds with previous ones in Table 1.
We can see that our algorithm matches the lower bound up to logarithmic factors in both known C and unknown C cases, and therefore is nearly optimal.
[Table]
[Equation] [Equation]
Unknown
Yes
Strong
[Equation]
OFUL (Remark 4.
[Equation]
Unknown
Yes
Weak
Lower bound( Lattimore and Szepesvári 2018)
[Equation]
Unknown
N/
[Equation]
Strong (Theorem 4.12)
[Equation]
1.1 Additional Related Work
Bandits with Misspecification.
Bandits with misspecification can be seen as a special case of bandit with adversarial corruption since it is corrupted relative evenly at each round.
Let ϵ be the misspecification level.
Ghosh et al.
(2017) firstly studied the stochastic linear bandits and proved a sublinear regret when ϵ is small.
Lattimore and Szepesvari (2019) studied the stochastic linear bandit θ ∈ R d and b ∈ R satisfying b ≤ r ∥ θ ∥ 2 .
The lower bound under a large corruption level C ≥ √ K only holds for algorithms that can achieve nearoptimal regret for uncorrupted bandits.
It is possible for an algorithm that does not achieve the optimal regret for uncorrupted bandits (e.g., R K
= O
( K 0 . 75 ) )
to achieve a sub-linear regret in the presence of corruptions.
setting under milder assumptions.
With the knowledge of ϵ , they proposed an algorithm with an (cid:101) O
( (cid:112) dK log( N )+ ϵ √ dK ) regret, where d is the dimension of the contextual vector, N is the number of arms.
Their regret bound matches their proved lower bound up to logarithmic factors.
Foster et al.
(2020) further considered the more general linear contextual bandits with misspecification when ϵ is unknown to the agent, and proposed an algorithm equipped with a CORRAL meta algorithm
(Agarwal et al., 2017) to deal with the unknown ϵ .
Their algorithm enjoys an (cid:101)
O
( d √ K + ϵ √ dK ) regret.
Krishnamurthy et al.
(2021) proposed an algorithm without using a meta algorithm which has the same order of regret as Foster et al.
Our algorithm can be directly applied to the misspecification setting by choosing the corruption level C to be Kϵ , which immediately gives us an (cid:101)
O ( d √ K + dKϵ ) regret upper bound.
Bandits with Adversarial Rewards.
There exists a large body of literature on the problems of adversarial multi-armed bandits (Auer et al., 2002; Bubeck and Cesa-Bianchi, 2012).
There is also a line of works trying to design algorithms that can achieve near-optimal regret bounds for both stochastic bandits and adversarial bandits simultaneously (Bubeck and Slivkins, 2012; Seldin and
Slivkins, 2014; Auer and Chiang, 2016; Seldin and Lugosi, 2017; Zimmert and Seldin, 2019; Lee et al., 2021).
However, most of these algorithms focus on the general adversarial reward setting without specifying the total amount of corruption.
One of the notable exceptions is Lee et al.
(2021), which assumed that the adversarial corruptions are generated through the inner product of an adversarial vectors and the contextual vector.
As a comparison, our algorithm and result do not need such additional assumption on the structure of the corruption.
Our algorithm can be applied to both corrupted and uncorrupted settings with different choices of hyperparameters, and achieves a near-optimal regret for both cases.
Notation We use lower case letters to denote scalars, and use lower and upper case bold face letters to denote vectors and matrices respectively.
We denote by [ n ] the set { 1 , . . . , n } .
For a vector x ∈ R d and a positive semi-definite matrix Σ ∈ R d × d , we denote by ∥ x ∥ 2 the vector’s ℓ 2 norm and by ∥ x
∥ Σ = √ x ⊤
Σx the Mahalanobis norm.
For two positive sequences { a n } and { b n } with n = 1 , 2 , . . .
, we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb n holds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such that a n ≥ Cb n holds for all n ≥ 1 .
We use (cid:101) O ( · ) to further hide the polylogarithmic factors.
We use 1 {·} to denote the indicator function.
Preliminaries
In this section, we introduce the setting of linear contextual bandit with adversarial corruption.
Linear contextual bandit with corruption.
We define linear contextual bandits with corruption as follows: at the beginning of each round k ∈ [ K ] , the agent receives a decision set D k ⊆ R d from the environment and it chooses an action (i.e., arm, contextual vector) x ∈ D k .
After choosing the action x k at round k , the environment generates the corresponding r k based on the stochastic linear model r k = ⟨ θ ∗ , x ⟩ + η k , where θ ∗ ∈ R
d is an unknown environment parameter and η k is the stochastic noise.
After seeing the stochastic reward r k , the adversary (i.e., attacker) introduces an adversarial corruption c k onto the reward, which may depend on the decision set D k , action x k , stochastic reward r k .
Finally, the agent observes the corrupted reward (cid:98) r k
=
⟨ θ ∗ , x ⟩ + η k + c k at round k .
Following
Abbasi-Yadkori et al.
(2011), we make the following assumptions on the bandit model.
Assumption 2.1.
The linear contextual bandit satisfies the following conditions:
• At each round k and any action x ∈ D k , we have ∥ x ∥ 2 ≤ L .
• For the unknown environment parameter θ ∗ , it satisfies ∥ θ ∗ ∥ 2 ≤ S .
• At each round k , the corresponding stochastic noise η k is conditional R -sub-Gaussian, i
[Equation]
e.,
[Equation]
Regret.
The goal of the agent is to minimize the pseudo-regret in the first K rounds, which is defined as follows:
[Equation]
Corruption level.
To measure the level of adversarial corruptions, we define the corruption level as
C :=
(cid:80)
Kk =
| c k | .
With this definition, we say a linear contextual bandit problem is C -corrupted if and only if the corruption level is no larger than C .
Remark 2.2.
The adversary in our setting and the corresponding definition of corruption level is the same as that in Bogunovic et al.
(2021) and slightly different from that in prior works such as
Lykouris et al.
; Gupta et al. (2019b); Zhao et al.
More specifically, in these works, the adversarial corruption c k is chosen before the choice of action x k ∈ D k .
Since the actions selected by the agent may not be deterministic, the adversary chooses different corruption
c k, x for different action x ∈ D k .
With this notion of corruption, the corresponding corruption level is defined as
[Equation]
| .
As a comparison, our adversary chooses the corruption after observing the action x k and for the corruption level.
We have
[Equation]
which implies that our corruption level C is always no larger than the corruption level C ′ in Lykouris et al.
; Gupta et al. (2019b); Zhao et
[Equation]
Algorithms
In this section, we review existing algorithms for linear contextual bandits (and stochastic linear bandits) and discuss their limitations when they are applied to the adversarial corruption setting.
Then we present our algorithm CW-OFUL and illustrate how our algorithm design can overcome the these limitations.
3.1 Existing Algorithms
We begin with reviewing the classical OFUL algorithm (Abbasi-Yadkori et al., 2011).
Under
Assumption 2.1, at round k , OFUL estimates θ ∗ by online ridge regression over all the past observed actions and rewards, i.e.,
[Equation]
With θ k in hand, OFUL constructs a confidence set for θ ∗ as follows C k
= (cid:110) θ
[Equation]
∥ θ
[Equation]
, where β is the confidence radius and Σ k
= λ
I + (cid:80) k − 1 i =1 x i x ⊤ i is the covariance matrix of contexts x
i , i = 1 , . . .
, k .
Without corruption, it is known that setting β = (cid:101) O
( R √ d ) guarantees that θ ∗ ∈ C k with high probability, which further leads to a sublinear regret (cid:101) O ( d √ K ) .
However, with corruption, such a choice of β is not sufficient.
To see why, we take a closer look at the closed-form solution θ k to (3.1):
[Equation]
By simple calculation and assuming λ to be a constant, we can show that ∥ θ k − θ ∗ ∥ Σ k can be upper bounded by
[Equation]
− 1 k (cid:124) (cid:123)(cid:122) (cid:125) I(cid:19) .
The first term I 1 is corruption-independent and bounded by (cid:101) O
( R √ d ) according to Abbasi-Yadkori et al.
The challenge is to bound the second term I 2 , which depends on the corruption.
Existing approaches (Zhao et al., 2021; Ding et al., 2021) bound I 2 by triangle inequality and Cauchy-Schwarz inequality,
[Equation]
Algorithm
Require: Regularization parameter λ , confidence radius β and threshold parameter α 1: for round k
= 1 , 2 , .. do 2:
Set Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i 3:
Set
b k = (cid:80) k − 1 i =
w i x i r i and θ k
= Σ − 1 k b k 4:
Receive the decision set D k 5:
Choose action x k ← argmax x ∈D k θ ⊤ k x + β (cid:113) x ⊤ Σ − 1 k x 6:
Set w k = min { 1 , α/ ∥ x k ∥ Σ − 1 k } 7: end for where C is the corruption level and max 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k is bounded by the crude upper bound
L/ √ λ .
Unfortunately, such a bound makes the confidence radius be the order of O ( R √ d + C ) , which eventually leads to an term O ( C √ K ) in the regret, which is C times worse than the regret without corruption.
In order to obtain a tighter bound of I 2 , for stochastic linear bandits, Bogunovic et al.
(2021) proposed a Robust Phase Elimination (RPE) algorithm, which employs optimal design (Lattimore and Szepesvári, 2018) to select the arms.
In this setting, the decision set is finite and fixed over time, i.e., D k
= D for all k ∈
[ K ] and |D| ≤ ∞ .
More specifically, RPE divides the time horizon into several phases.
Within each phase, RPE performs linear regression on a multiset A ⊂ D , which is the
G-optimal design of D .
Here the multiset means A has duplicate elements.
Let Σ be the covariance matrix defined over A , then the following upper bound holds (Lattimore and Szepesvári, 2018):
[Equation]
By choosing a large enough |A| , (3.3) provides a uniformly small upper bound for max 1 ≤ j ≤ k − 1 ∥ x j ∥ Σ − 1 k for any k .
Substituting (3.3) back into (3.2) with |A| =
O ( C 2 ) , we can show that I 2 is bounded by some small constant, which therefore eliminates the O ( C √ K ) term in the final regret.
Although the optimal design-based approach RPE (Bogunovic et al., 2021) successfully eliminates the multiplicative term C √ K , it is not applicable to our linear contextual bandit setting: (1) it needs to select a multiset from the decision set, which is impossible for the general contextual bandit setting; (2) the complexity of optimal design introduces some additional quadratic term C 2 in their final regret, which makes their algorithm non-optimal (See Bogunovic et al.
(2021) for more details).
3.2 Our Algorithm
As we have seen before, it is pivotal to bound the corruption-dependent term
I 2 tightly.
To overcome the limitations of existing approaches, we propose a fundamentally new approach and present
our
CW-OFUL in Algorithm 1.
At a high level, Algorithm 1 is an extension of the OFUL algorithm (Abbasi-Yadkori et al., 2011), which is also based on the principle of optimism in the face of uncertainty.
Our algorithm assigns a weight w k to each selected action x k .
More specifically, at round k , we use the following weighted ridge regression to estimate the unknown vector θ ∗ :
[Equation]
The closed-form solution to the above optimization problem is displayed in Line 3 of Algorithm 1.
While weighted ridge regression is not new and has been used in prior work on bandits (Kirschner and
Krause, 2018; Zhou et al., 2021; Russac et al., 2019), the setting, motivation and the choice of weight are fundamentally different.
More specifically, we choose the weight as the truncation of the inverse exploration bonus, which is w k = min (cid:110) 1 , α/ ∥ x k ∥ Σ − 1 k (cid:111) .
Here α > 0 is a threshold parameter.
We can see that for action x k with a large exploration bonus ∥ x
k ∥ Σ − 1 k (low confidence), CW-OFUL will assign a small weight to it to avoid the potentially large regret caused by both the stochastic noise and the adversarial corruption.
On the other hand, for the action with a small exploration bonus (high confidence), CW-OFUL will assign a large weight to it (it can be as large as 1 ).
Another interesting observation is that by setting α to be sufficiently large, the weight will become 1 for every action, and CW-OFUL will degenerate to OFUL (Abbasi-Yadkori et al., 2011).
As a comparison, Kirschner and Krause (2018)
; Zhou et al.
(2021) used the inverse of the noise variance as the weight to normalize the noise and derived tight variance-dependent regret guarantees.
Russac et al.
(2019) set the weight as a geometric sequence to perform moving average to deal with the non-stationary environment.
To see how our choice of weight can lead to tighter regret, we first write down the closed-form solution to (3.4)
[Equation]
where the covariance matrix Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i .
With some calculation and assuming λ to be a constant, we can obtain
[Equation]
I 1 is the corruption-independent term and can still be bounded by (cid:101)
O ( R √ d ) according to AbbasiYadkori et al.
For I 2 , we have
[Equation]
It is evident that with our carefully designed weight, the corruption-dependent term I 2 can be uniformly bounded by some constant Cα , the same as that in Bogunovic et al.
Therefore, by setting α to be sufficiently small, our CW-OFUL can get rid of the C √ K term in the final regret.
Main Results
In this section, we present the main theoretical guarantees of CW-OFUL.
4.1 Known Corruption Level C : Upper Bound
We first consider the case when C is known to the agent.
In this case, we choose α = R √ d/C .
The following lemma characterizes the estimation error of θ k with respect to θ ∗ , which is a formal summary of our discussion in Section 3.
Lemma 4.1.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 , set the confidence radius β = R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) + √ λS + αC in Algorithm 1, then with probability at least 1 − δ , for every round k , the estimator θ k satisfies that
∥ θ k − θ ∗ ∥
Σ k ≤ β .
The following theorem provides the regret bound of Algorithm 1.
Theorem 4.2.
Suppose that Assumption 2.1 holds.
For any 0 < δ < 1 and corruption budget C ≥ 0 , set the confidence radius β in Algorithm 1 as follows:
[Equation]
Then with probability at least 1 − δ , its regret in the first K rounds is upper bounded by
Regret ( K ) =
O
[Equation]
+ dS √
[Equation]
α
[Equation]
In addition, if choosing α =
( R √ d + √ λS ) /C
and λ =
R 2 /S 2 , its regret can be upper bounded by
Regret ( K ) =
(cid:101) O
( d √ K + dC ) .
A few remarks about Theorem 4.2 are in order.
Remark 4.3.
Compared with the (cid:101) O
( d √ K + dC √ K ) regret proved in Zhao et al.
; Ding et al.
(2021), our algorithm improves the multiplicative dependence on corruption level C to additive dependence.
In particular, CW-OFUL achieves the same order of regret as the uncorrupted setting when
C
=
O
( √ K ) ,
and it attains a sublinear regret as long as C
= o ( K ) .
In sharp contrast, the algorithm proposed in Zhao et al.
(2021) achieves the same order of regret as the uncorrupted setting only when
C = O (1) , and has a sublinear regret only when C = o ( √ K ) .
Remark 4.4.
We also compare our result with that in Wei et al.
The Robust + OFUL algorithm in Wei et al.
(2022) achieves an (cid:101)
O
( d √ K + C r ) regret with C r = (cid:113) T (cid:80)
Kk =1 c 2 k , which will degenerate to (cid:101) O
( d √ K + d √ KC ) in the worst case.
Their regret guarantee is always worse than ours when C < √ K .
In addition, according to the discussion in Remark 2.2, Theorem 4.2 also implies an (cid:101) O (cid:0) d √ K + dC ′ (cid:1) regret under the notion of the corruption level C ′ .
In contrast, the Robust
VOFUL algorithm in Wei et al.
(2022) has an (cid:101) O
( d 4 . 5 √ K + d 4 C ′ ) regret, which is also inferior to our regret.
Furthermore, Robust VOFUL is computationally inefficient.
Remark 4.5.
We further compare our result with previous additive regrets derived for stochastic linear bandits.
Let D k
= D be the decision set.
Compared with the O ( (cid:112) dK log |D| + Cd 3 / 2 ) regret for stochastic linear bandit with corruption derived in Bogunovic et al.
(2021), our regret improves the corruption term by a factor of √ d .
Note that the √ d difference in the leading √ K term between our regret and theirs is caused by the fact that Bogunovic et al.
(2021) considered the finite-arm setting, while we consider the infinite-arm setting.
Our algorithm will have the same regret as theirs when |D| =
O ( exp ( d )) .
Remark 4.6.
For the uncorrupted setting where C = 0 , Theorem 4.2 suggests that the threshold parameter α should be set to infinity.
Then by Line 6 in Algorithm 1, each weight w k becomes 1 , and
CW-OFUL degenerates to OFUL.
Meanwhile, the regret in Theorem 1 also becomes (cid:101) O
( d √ K ) that matches the regret of OFUL (Abbasi-Yadkori et al., 2011).
4.2 Known Corruption Level C: Lower Bound
In this subsection, we refer to two existing lower bound results to show that when C is known, our (cid:101) O
( d √ K + dC ) regret is optimal up to logarithmic factors.
The first proposition shows that the (cid:101) O ( d √ K ) corruption-independent term in our regret is near-optimal.
Proposition 4.7 (Theorem 24.2, Lattimore and Szepesvári 2018) .
Assume d ≤ 2 K , R = 1 and
D k
= {∥ x ∥ 2 ≤ 1 } for all k ≥ 1 .
Then for any algorithm, there exists an environment parameter vector θ ∗ ∈ R d satisfying ∥ θ ∗ ∥
22 = d 2 / (48 K ) such that E ( Regret ( K )) ≥ d √ K/ (16 √ 3) .
The second proposition suggests that the O ( dC ) corruption term in our regret is optimal.
Proposition 4.8 (Theorem 3, Bogunovic et al. 2021) .
For any dimension d , for any algorithm that has the knowledge of C , there exists an instance satisfying with probability at least 0 .
5 , Regret ( K ) =
Ω( dC ) .
Combining Propositions 4.7 and 4.8, we can conclude that for any algorithm, there exists a corrupted bandit instance such that the algorithm suffers at least Ω(max { d √ K, dC } ) regret.
Such a lower bound matches our upper bound up to logarithmic factors.
Therefore, our algorithm is nearly optimal.
4.3 Unknown Corruption Level C : Upper Bound
Now we consider the case when C is unknown.
Our solution is quite simple for this case: we introduce a tuning parameter ¯ C , which can be viewed as an estimate of C , and select the threshold parameter α as Theorem 4.2 suggests.
The following theorem gives the regret upper bound of
CW-OFUL for the unknown C case.
Theorem 4.9.
Under the same conditions of Theorem 4.2 except that we set α = ( R √ d + √ λS ) / ¯ C with ¯ C being an estimated corruption level, λ = R 2 /S 2 and β = 2 R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ
(cid:1) + 2 √ λS in Algorithm 1.
The regret of CW-OFUL can be upper bounded in the following two cases:
• If the corruption level C satisfies that 0 ≤ C ≤ ¯ C , then with probability at least 1 − δ , the regret is upper bounded by Regret ( K ) = (cid:101) O ( dR √ K + d ¯ C ) .
• If the corruption level C satisfies that C > ¯ C , the regret is upper bounded by Regret ( K )
=
O ( K ) .
In addition, if we set the estimation ¯ C = √ K , then when 0 ≤ C ≤ √ K , the regret is upper bounded by (cid:101)
O
( d √ K ) .
Remark 4.10.
Zhao et al.
(2021) proposed an (cid:101)
O ( C 2 d √ K ) regret with unknown C = Ω(1) .
Compared with their result, our regret (with ¯ C = √ K ) is strictly better in the corruption term.
Bogunovic et al.
(2021) proposed an (cid:101) O
( (cid:112) dK log |D| + Cd 1 .
5 + C 2 ) regret for the stochastic linear bandit with unknown C , in the regime C
= (cid:101) O ( √ K/d ) , where D is the finite decision set.
Such a regret becomes (cid:101) O ( d √ K + Cd 1 .
5 + C 2 ) when the size of D becomes exponentially large in d or even infinite.
Compared with their regret, our regret is not only smaller, but also holds for a wider regime (i.e., C
=
O
( √ K ) ).
Compared with the greedy algorithm in Bogunovic et al.
(2021), our result does not rely on the stringent ( r, λ 0 ) -diverse property assumption on the contexts.
Remark 4.11.
We also compare our result (choosing ¯ C = √ K ) with those in Wei et al.
(2022) for the unknown C case.
Wei et al.
(2022) proposed a COBE + OFUL algorithm with an (cid:101) O
( d √ K + C r ) regret, and a COBE + VOFUL algorithm with an (cid:101)
O ( d 4 .
5 √ K + d 4 C ′ ) regret, analogous to their results for the known C case discussed in Remark 4.4.
Our CW-OFUL enjoys a better regret than COBE + OFUL for all C , and it is better than COBE + VOFUL for C < √ K .
In addition, for the modified notion of corruption level C ′ , if we choose the basic algorithm in COBE (Wei et al., 2022) as our CW-OFUL algorithm, then Theorem 3 in Wei et al.
(2022) suggests that COBE+CW-OFUL can deal with unknown corruption level C ′ and obtained an (cid:101)
O ( d √ K + dC ′ ) regret guarantee, which matches the regret of CW-OFUL algorithm with known corruption level C ′ .
Note that COBE + VOFUL is also computationally inefficient.
4.4 Unknown Corruption Level C : Lower Bound
With ¯ C = √ K , for the case when 0 ≤ C ≤ √ K , our regret result is already near-optimal, due to the lower bound for the uncorrupted bandit in Proposition 4.7.
Now we show that our O ( K ) bound, seemingly trivial, is actually optimal for a large class of bandit algorithms.
In detail, the following theorem provides a lower bound result for any algorithm for the unknown C case.
This is an extension of the lower bound result in Bogunovic et al.
from d = 2 to general d .
Theorem 4.12.
For any algorithm Alg , let R K be an upper bound of Regret ( K ) such that for any bandit instance satisfying Assumption 2.1 with C = 0 , it satisfies the E (cid:2)
Regret ( K ) (cid:3) ≤ R K ≤ O ( K ) , where the expectation is with respect to the randomness of the algorithm and the stochastic noise.
Then for the general case with C = Ω( R K /d ) , such an algorithm will have E (cid:2) Regret ( K ) (cid:3)
=
Ω( K ) .
Remark 4.13.
If we selects the estimated corruption ¯ C = Ω( R K /d ) , Theorem 4.9 immediately implies that CW-OFUL enjoys a O ( R K ) regret when corruption level C < Ω( R K /d ) and O ( K ) regret when corruption level C ≥ Ω( R K /d ) .
Compared with the algorithm Alg , Theorem 4.
suggests that CW-OFUL is no worse than the algorithm Alg no matter whether the corruption level
C < Ω( R K /d ) .
More discussion can be found in Appendix A.1.
Conclusion and Future Work
In this work, we study corrupted linear contextual bandits.
We propose a CW-OFUL algorithm based on a weighted ridge regression with truncated inverse exploration bonus weights.
We show that for both cases when the corruption level C is known or unknown to the agent, CW-OFUL achieves a regret that matches the lower bound up to logarithmic factors.
We are also interested in achieving the optimal regret when specializing our algorithm to the misspecified linear contextual bandits.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers and area chair for their helpful comments.
JH, DZ and QG are supported in part by the National Science Foundation CAREER Award 1906169 and the Sloan
Research Fellowship.
TZ is supported in part by the GRF 16201320.
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We study corrupted linear contextual bandits and propose a computation efficient algorithm with a near-optimal regret guarantee.
(b) Did you describe the limitations of your work?
[Yes] See the discussion about Misspecified linear bandits in subsection 4.1.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work focuses on a purely theoretical problem and does not have any negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumption 2.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional Results A.1 Discussion on the Lower Bound for Unknown Corruption Level C
Consider a class of algorithms A whose worst-case regret is R K in the uncorrupted case.
Here we only need to consider Ω( d √ K ) ≤ R K ≤ O ( K ) , since for any algorithm, Ω( d √ K ) is the lowest possible worst-case regret (Lattimore and Szepesvári, 2018) and O ( K ) is the highest possible regret.
We first show that CW-OFUL belongs to A .
Choosing ¯ C
= R K /d , Theorem 4.9 immediately suggests that CW-OFUL enjoys a R K regret in the uncorrupted case (i.e., C = 0 ).
Thus CW-OFUL belongs to A .
Then we will show that CW-OFUL is the best possible one in A .
On the one hand,
Theorem 4.9 suggests that CW-OFUL suffers a linear regret when C > ¯ C = R K /d .
On the other hand, Theorem 4.12 shows that any algorithm with R K regret in the uncorrupted case should have a linear regret when C = Ω( R K /d ) .
These together imply that CW-OFUL is optimal within A .
A.2 Discussion on the Misspecified Linear Bandits
We consider the misspecified linear bandit setting which assumes that the corruption at each round is uniformly bounded by ϵ .
Clearly, the misspecified linear bandit is a special case of corrupted linear contextual bandit with C =
Kϵ .
Theorem 4.2 suggests that a direct application of our algorithm to this special setting incurs an (cid:101)
O ( d √ K + dKϵ ) regret, which differs from the near-optimal regret (cid:101)
O ( d √ K + √ dKϵ ) (Lattimore and Szepesvari, 2019; Foster et al., 2020) by a √ d factor on the corruption term.
Whether our algorithm is able to achieve the near-optimal regret for both misspecified linear bandit and corrupted linear contextual bandit simultaneously remains an open question.
B Experiments
[Figure]
[Figure]
In this section, we run experiments and evaluate the performance of our algorithm CW-OFUL with an unknown corruption level C , which corroborates our theory.
The code and data for our experiments can be found on Github 3 .
Model Parameters We construct a linear bandit instance with dimension d = 5 and the true model parameter θ ∗ is denoted by [Equation] [Equation] .com
[Equation]
uclaml/CW-OFUL
During each round k ∈ [ K ] , the decision set D k consists of 20 different actions and each action is uniformly sampled from the space [+1 / √ d, − 1 / √ d ] d , which satisfies the positive minimum eigenvalue assumption for greedy algorithm (Bogunovic et al., 2021).
In addition, after choosing the action, x k in round k ∈ [ K ] , an 0 . 1 -Gaussian noise η k will be added to the reward.
Attack method For the attack method, we choose the flip- θ attack.
More specifically, with a corruption level of C , the adversary tricks the learner by flipping the value, i.e., r t ( x t ) =
−⟨ x t , θ ∗ ⟩ + η k in the first C rounds.
In the remaining rounds, the adversary does not corrupt the reward.
Results and discussions In our experiments, we make a simulation with the total number of rounds K = 10000 (repeating 100 times and taking the average) and corruption levels from { 0 , 50 , 100 , ..., 450 } (10 different corruption levels ).
We applied our CW-OFUL algorithm and compared its performance with greedy (Bogunovic et al., 2021) and OFUL (Abbasi-Yadkori et al., 2011).
The experimental results are shown in Figure 1 These simulation results suggest that our CW-OFUL algorithm outperforms both the Greedy and OFUL algorithms.
Specifically, Figure 1(a) displays the cumulative regret of our algorithm, OFUL, and Greedy algorithm under the same number of rounds.
They show that our algorithm slightly outperforms them in terms of regret.
Figure 1(b)
plots the cumulative regret versus the unknown corruption level.
We can see that all of the three algorithms demonstrate an additive linear dependence on the unknown corruption level C , which corroborate our theoretical guarantee.
C Instance-dependent Regrets
Prior works (Lykouris et al., 2018; Li et al., 2019; Zhao et al., 2021) have proved instance-dependent regret bounds for corruption-robust linear bandits.
We show that CW-OFUL also enjoys an instancedependent regret bound.
Following Abbasi-Yadkori et al.
(2011), we define the minimal suboptimality gap as follows.
Definition C.1 (Minimal sub-optimality gap) .
For each round k ∈ [ K ] and any action x ∈ D k , the sub-optimality gap ∆ x ,k is defined as
[Equation]
and the minimal sub-optimality gap is defined as
[Equation]
We assume that the minimal sub-optimality gap is strictly positive.
Assumption C.2.
The minimal sub-optimality gap is strictly positive, i.e., ∆ > 0 .
Under the assumption of positive minimal sub-optimality, the following theorem provides an instancedependent regret guarantee for CW-OFUL.
Theorem C.3.
Under the same conditions of Theorem 4.2, with high probability at least 1 − δ , the regret of Algorithm 1 in the first K rounds is upper bounded by
Regret ( K ) ≤ O
[Equation]
In addition, if choosing α =
( R √ d + √ λS ) /C
and λ =
R 2 /S 2 , the regret can be upper bounded by
[Equation]
Remark C.4.
Our regret is strictly better than the (cid:101) O
( d 2 . 5 C/ ∆ + d 6 / ∆ 2 ) regret proved by Li et al.
(2019) under a stronger assumption.
Meanwhile, Zhao et al.
(2021) implies an (cid:101) O
( d 2 C/ ∆) regret for their algorithm under the known C case, which is also worse than our result.
D Overview of Key Proof Techniques
In this section, we give an overview of the main technical difficulty and our proof technique to derive
Theorem 4.2.
By the standard regret decomposition technique from Abbasi-Yadkori et al.
, we upper bound the regret by the sum of the exploration bonuses times the confidence radius:
[Equation]
Lemma 4.1 suggests β ∼ R √ d + αC .
Therefore, we only need to bound the summation of the exploration bonuses.
For the basic case when w k = 1 , we bound it using the elliptical potential lemma (Abbasi-Yadkori et al., 2011) as follows
[Equation]
which contributes to the corruption-independent term dR √ K in our regret.
For the case when w k < 1 , however, we are facing the weighted covariance matrix and cannot directly use the elliptical potential lemma.
A trivial approach is to lower bound the weights by their uniform lower bound, i.e.,
[Equation]
By the definition of the weight w i in Algorithm 1 and a crude upper bound for the exploration bonus, we conclude from the definition of w k that w k = Ω( α ) .
Substituting it into (D.3), we only obtain a regret (cid:101) O ( √ dK/α ) , which is not satisfying.
To overcome this issue, we recall the definition for weight w k < 1 in Algorithm 1: w k = α/ ∥ x k ∥ Σ − 1 k
and we can bound the summation of the exploration bonuses as
[Equation]
Combining the results in (D.2) and (D.4) into (D.1), we can prove the final regret.
E Proof of Theorem 4.
In this section, we provide the proof of Theorem 4.2.
For simplicity, we use E to denote the following event:
[Equation]
Lemma 4.1 shows that Pr( E ) ≥ 1 − δ .
Lemma E.1.
If setting the confidence radius β = R (cid:113) d log (cid:0) (1 + KL 2 /λ ) /δ (cid:1) + αC + √ λS in
Algorithm 1, then on the event E , for each round k ∈ [ K ] , the regret at round k is upper bounded by
[Equation]
Proof of Theorem 4.2.
Based on the event E , the regret in the first K round can be decomposed into two parts based on the weight w k :
[Equation]
[Equation]
where the inequality holds due to the Lemma E.1 with the fact that the suboptimality in each round k is no more than 2 .
For the term I 1 , we consider for all rounds k ∈
[ K ] with w k =
and we assume these rounds can be listed as { k 1 , .., k m } for simplicity.
With this notation, for each i ≤ m , we can construct the auxiliary covariance matrix
A i
=
λ
I +
[Equation]
i − 1 j =1 x k j x ⊤ k j .
Due to the definition of original covariance matrix Σ k in Algorithm (Line 2), we have
[Equation]
According to Lemma J.4, it further implies that for vector x k i , we have
[Equation]
Therefore, the term I 1 can be bounded by
[Equation]
where the first inequality holds since β ≥ 1 , the second inequality holds due to (E.2), the third inequality holds due to Cauchy-Schwarz inequality, the last inequality holds due to Lemma J.3 with the facts that m ≤ K and ∥ x k i ∥ 2 ≤ L .
For the second term I 2 , according to the definition for weight w k < 1 in Algorithm 1, we have w k = α/ (cid:113) x
⊤ k Σ − 1 k x k , which implies that
[Equation]
where the second equation holds due to the definition of weight w k .
Now, we assume the rounds with weight w k < 1 can be listed as { k 1 , .., k m } for simplicity.
In addition, we introduce the auxiliary vector x ′ i as x ′ i = √ w k
i
x
k i and matrix Σ
′ i as Σ
′ i
=
λ
I + i − 1 (cid:88)
j = w k j x k j x ⊤ k j = λ
I
+ i − 1 (cid:88)
j = x ′ j ( x ′ j ) ⊤ .
According to Lemma J.4, we have ( Σ ′ i ) − 1 ⪰ Σ − 1 k i .
Therefore, for each i ∈ [ m ] , we have
[Equation]
where the inequality holds due to ( Σ ′ i ) − 1 ⪰ Σ − 1 k i .
Now, taking a summation of (E.5) over all rounds k i , we have
[Equation]
where the first inequality holds due to (E.5), the second inequality holds due to Lemma J.3 with the facts that m ≤ K .
Substituting the result in (E.6) into (E.4), the term I 2 can be upper bounded by
I 2 ≤
[Equation]
Finally, substituting the results in (E.3) and (E.7) into (E.1), the regret can be upper bounded by
Regret ( K ) ≤
[Equation]
(cid:112)
[Equation]
+
[Equation]
+
[Equation]
α
[Equation]
×
[Equation]
Therefore, we complete the proof of Theorem 4.2.
F Proof of Theorem C.
In this section, we present the detailed proof of Theorem 4.2.
Proof of Theorem C.3.
Based on the event E , the regret in round k ∈ [ K ] is upper bounded by
[Equation]
On the other hand, according to Assumption C.2, the regret in round k ∈
[ K ] satisfies that ∆ k = 0 or ∆ k ≥ ∆ .
Combining these two results, for round k ∈ [ K ] with uncertainty 2 β (cid:113) x ⊤ k Σ − 1 k x k < ∆ , the regret must satisfy ∆ k = 0 .
Therefore, the regret in the first K rounds can be decomposed to two part based on the weight w k and exploration bonus (cid:113) x ⊤ k Σ − 1 k x k :
Regret ( K ) =
[Equation] [Equation]
where the inequality holds due to Lemma E.1 with the fact that the suboptimality in each round is no more than 2 .
Notice that the term J 2 is equal to the term I 2 in the proof of Theorem 4.2 (See (E.1)) and with the same argument, it can be upper bounded by
[Equation]
where the inequality comes from (E.7).
For the term J 1 , we consider for all rounds k ∈
[ K ] with w k = 1 and exploration bonus 2 β (cid:113) x ⊤ k Σ − 1 k x k ≥ ∆ .
For simplicity, we assume these rounds can be listed as { k 1 , .., k m } .
With this notation, for each i ≤ m , we can construct the auxiliary covariance matrix
A i
=
λ
I + (cid:80) i − 1 j =1 x k j x ⊤ k j .
Due to the definition of original covariance matrix Σ k in Algorithm (Line 2), we have
[Equation]
According to Lemma J.
[Equation]
it further implies that for vector x k i , we have
[Equation]
Therefore
[Equation]
the term J 1 can be bounded by
[Equation]
where the first inequality holds since β ≥ 1 , the second inequality holds due to (F.3), the third inequality holds due to Cauchy-Schwarz inequality, the fourth inequality holds due to Lemma J. with the facts that m ≤ K and ∥ x k i ∥ 2 ≤ L .
On the other hand, the term J 1 is lower bounded by
[Equation]
where the inequality holds due to the definition of k i with the fact that ∆ ≤ 2 .
Combining the upper and lower bound for term J 1 , we have
[Equation]
which further implies that
[Equation]
Substituting the upper bound of m in (F.6) into (F.4), the term J 1 can be upper bounded by
[Equation]
Finally, substituting the upper bounds of term J 2 in (F.2) and term J 1 in (F.7) into (F.1), the regret can be upper bounded by
[Equation]
Therefore, we complete the proof of Theorem C.3.
G Proof of Theorem 4.
Proof of Theorem 4.9.
We discuss two cases here.
• For the case C ≤ ¯ C , we know that ¯ C is still a valid upper bound of the corruption level.
Thus, CW-OFUL with a ¯ C corruption level runs successfully, and its regret is upper bounded by (cid:101)
O ( dR √ K + d ¯ C ) =
(cid:101)
O ( dR √ K + d ¯ C ) as Theorem 4.2 suggests.
• For the case C = Ω( ¯ C ) , CW-OFUL can not guarantee a sublinear regret.
Thus a trivial regret bound (i.e., regret at each round is bounded by 2) applies.
H Proof of Theorem 4.
We introduce our proof of Theorem 4.12, which is adapted from Bogunovic et al.
Proof of Theorem 4.12.
In this proof, we consider an arbitrary algorithm satisfying the conditions in the statement of Theorem 4.12, which will run K rounds for any bandit instance.
We consider an uncorrupted bandit instance A 0 defined as follows.
A 0 has the decision sets D k = D .
Here D =
{ a i } 1 ≤ i ≤ d , where a i =
e i is the basis in the d -dimensional space.
Let θ ∗
= (1 / 4 , 1 / 8 , . . .
, 1 /(cid:124) (cid:123)(cid:122) (cid:125) ( d − 1) − times ) ∈
R d and ϵ i = 0 .
It is easy to see that the optimal policy is to select a 1 at each round, and the regret to select a sub-optimal arm is 1 / 8 .
Since the regret of the algorithm without corruption satisfies E (cid:2)
Regret ( K ) (cid:3) < R K , and all the regret comes from selecting a 2 , . .
. , a d , we have the expected number of rounds to select a 2 , . .
. , a d is at most R K / (1 / 8) = 8 R K .
Then by the pigeonhole principle, there exists some 2 ≤ i ≤ d such that the expected number of times to select a i is less than 8 R K / ( d − 1) .
Without loss of generality, we suppose i = 2 .
Then by Markov inequality, with probability at least 1 / 2 , the number of times to select a 2 is less than 16 R K / ( d − 1) .
Next, we consider a corrupted bandit instance A 1 defined as follows.
A 1 has the same decision set
D = { e i } as A 0 , while it has a different θ ∗ 1 = (1 / 4 , 3 / 8 , 1 / 8 , . .
, 1 /(cid:124) (cid:123)(cid:122) (cid:125) ( d − 2) − times ) .
A 1 is also noiseless, i.e., ϵ i = 0 .
Unlike A 0 , we have an adversary to attack A 1 as follows: whenever a 2 is selected and the total corruption level up to the previous step is no more than 4 R K / ( d − 1) − 1 / 4 , the adversary corrupts the reward from 3 / 8 to 1 / 8 .
Otherwise, the adversary stops to corrupt the reward.
With this adversary, the corruption level C is upper bounded by 4 R K / ( d − 1) − 1 / 4 + 1 / 4 =
Ω( R K /d ) .
For this adversary, since for A 1 , each selection of a 2 returns a reward 1 / 8 , then the agent can not tell the difference between A 0 and A 1 until the total corruption level reaches the threshold 4 R k / ( d − 1) and the adversary stops to corrupt the reward.
Therefore, the sequence of rounds for the agent to select a 2 with A 1 instance is the same as the sequence for the agent to select a 2 with
A 0 , until the number of rounds to select action a 2 reaches 4 R k / ( d − 1) / (1 / 4)
=
16 R k / ( d − 1) .
However, when the total number of times to select a 2 is less than 16 R K / ( d − 1) , the agent cannot differentiate A 0 and A 1 and will follow the same action sequence as A 0 .
In this case, since for A 1 , a 2 is the optimal action, and all the other actions suffer a 1 / 8 regret, then the regret on A 1 is at least 1 / 8 · ( K − 16 R K / ( d − 1))
=
Ω( K ) , where we use the fact that R K ≤ O ( K ) .
Therefore, with probability at least 1 / 2 , the regret is at least Ω( K ) , which further implies that the expected regret is lower bounded by E [ Regret(K) ] ≥ 1 / 2 × Ω( K ) =
Ω( K ) .
Thus, we finish the proof of
Theorem 4.12.
I Proof of Lemmas in Sections 4, D and Appendix E I.1 Proof of Lemma 4.
Proof of Lemma
[Equation]
According to the definition of estimated vector θ k in Algorithm 1 (Line 3)
[Equation]
we have
[Equation]
This equation further implies that the difference between estimated vector θ k and the unknown vector θ ∗ can be decomposed as:
[Equation]
where the inequality holds due to the fact that ∥ a + b + c ∥ Σ k ≤ ∥ a ∥ Σ k + ∥ b ∥ Σ k + ∥ c ∥ Σ k .
For the stochastic error term I 1 , it can be bounded by the concentration Lemma J.2 in AbbasiYadkori et al.
More specifically, we introduce the auxiliary vector x ′ i and noise η ′ i such that x ′ i = √ w i x i and η ′ i = √ w i η i .
According to the definition of weight θ i in Algorithm (Line 6), both of these two situations satisfies that the weight θ i is bounded by w i ≤ 1 .
Since the original vector x i satisfies that ∥ x i ∥ 2 ≤ L and the original stochastic noise η i is R -sub Gaussian, these results further imply that
[Equation]
With this notation
[Equation]
the covariance matrix Σ k and the stochastic error term I 1 can be rewritten and bounded as:
[Equation]
where the first inequality holds due to Lemma J.2 and the second inequality holds due to the facts that
[Equation]
For the corruption error term I 2 , it can be bounded by
[Equation]
where the first inequality holds due to the fact that ∥ a + b ∥ 2 ≤ ∥ a ∥ 2 + ∥ b ∥ 2 , the second inequality holds due to the definition of weight w i in Algorithm (Line 6) with the fact that Σ k ⪰ Σ i and the last inequality holds due to the definition of corruption level C .
For the regularization error term I 3 , we have
[Equation]
where the first inequality holds due to (cid:13)(cid:13)
θ ∗ (cid:13)(cid:13)
Σ k ≤ ∥
θ ∗ ∥ 2 / (cid:112) λ min ( Σ k ) with the fact that Σ k
= λ
I + (cid:80) k − 1 i =
w i x i x ⊤ i ⪰ λ I and the last inequality holds due to the assumption that ∥ θ ∗ ∥ 2 ≤ S .
Finally, substituting the results in (I.2), (I.3) and (I.4) into (I.1), we have ∥
[Equation]
Therefore, we finish the proof of Lemma 4.1.
I.2 Proof of Lemma E.
Proof of Lemma E.1.
Firstly, on the event E , for each round k ∈ [ K ] and each action x ∈ D k , we have
[Equation]
where the first inequality holds due to the Cauchy-Schwarz inequality and the last inequality holds due to the definition of E in Lemma 4.1.
(I.5) shows that our estimator in Algorithm 1 is optimistic for each action x ∈ D k .
For simplicity, we denote the optimal action at round k as x ∗ = arg max x ∈D k
( θ ∗ )
⊤ x and (I.5) further implies that the regret at round k can be upper bounded by
[Equation]
where the first inequality holds due to (I.5), the second inequality holds due to the selection rule in Algorithm (Line 5), the third inequality holds due to the Cauchy-Schwarz inequality and the last inequality holds due to the definition of E in Lemma 4.1.
Thus, we finish the proof of Lemma
E.1.
J Auxiliary Lemmas
Lemma J.1 (Azuma–Hoeffding inequality, Cesa-Bianchi and Lugosi 2006) .
Let { η k } Kk =1 be a martingale difference sequence with respect to a filtration {G k } satisfying | η k | ≤ R for some constant
R , η k is G k +1 -measurable, E (cid:2)
η k |G k (cid:3) = 0 .
Then for any 0 < δ < 1 , with high probability at least 1 − δ , we have
[Equation]
Lemma J.2 (Lemma 9 in Abbasi-Yadkori et al. 2011) .
Let { ϵ k } Kk =1 be a real-valued stochastic process with corresponding filtration {F k } Kk =
such that ϵ k is F k -measure and ϵ k is conditionally
R -sub-Gaussian, i.e.
[Equation]
Let { x k } Kk =1 be an R d -valued stochastic process where x k is F k − 1 -measurable and for any k ∈
[ K ] , we further define Σ k
= λ
I + (cid:80) k
i =
1 x i x ⊤ i .
Then with probability at least 1 − δ , for all k ∈ [ K ] , we have
[Equation]
Lemma J.3 (Lemma 11 in Abbasi-Yadkori et al. 2011) .
Let { x k } Kk =1 be a sequence of vectors in
R d , matrix Σ 0 a d × d positive definite matrix and define Σ k = Σ
+ (cid:80) ki
=1 x i x ⊤ i , then we have
[Equation]
In addition, if ∥ x i ∥ 2 ≤ L holds for all i ∈ [ K ] , then k
[Equation]
Lemma J.4 (Corollary 7.7.4.
(a) in Horn and Johnson 2012) .
Let A , B be a Hermitian matrix in R d × d and suppose A , B ≻ 0 , then A ⪰ B
if and only if B − 1 ⪰ A − 1 .