PopArt: Efficient Sparse Regression and Experimental Design for Optimal Sparse Linear Bandits
Abstract
In sparse linear bandits, a learning agent sequentially selects an action and receive reward feedback, and the reward function depends linearly on a few coordinates of the covariates of the actions.
This has applications in many real-world sequential decision making problems.
In this paper, we propose a simple and computationally efficient sparse linear estimation method called P OP A RT that enjoys a tighter ℓrecovery guarantee compared to Lasso (Tibshirani, 1996) in many problems.
Our bound naturally motivates an experimental design criterion that is convex and thus computationally efficient to solve.
Based on our novel estimator and design criterion, we derive sparse linear bandit algorithms that enjoy improved regret upper bounds upon the state of the art (Hao et al., 2020), especially w.r.t.
the geometry of the given action set.
Finally, we prove a matching lower bound for sparse linear bandits in the data-poor regime, which closes the gap between upper and lower bounds in prior work.
Introduction
In many modern science and engineering applications, high-dimensional data naturally emerges, where the number of features significantly outnumber the number of samples.
In gene microarray analysis for cancer prediction [19], for example, tens of thousands of genes expression data are measured per patient, far exceeding the number of patients.
Such practical settings motivate the study of high-dimensional statistics, where certain structures of the data are exploited to make statistical inference possible.
One representative example is sparse linear models [13], where we assume that a linear regression task’s underlying predictor depends only on a small subset of the input features.
On the other hand, online learning with bandit feedback, due to its practicality in many applications such as online news recommendations [16] or clinical trials [17, 23], has attracted a surge of research interests.
Of particular interest is linear bandits, where in n rounds, the learner repeatedly takes an action A t (e.g., some feature representation of a product or a medicine) from a set of available actions A ⊂ R d and receives a reward r t = ⟨ θ ∗ ,A t ⟩ + η t as feedback where η t ∈ R is an independent zero-mean, σ -sub-Gaussian noise.
Sparsity structure is abundant in linear bandit applications: for example, customers’ interests on a product depend only on a number of its key specs; the effectiveness of a medicine only depends on a number of key medicinal properties, which means that the unknown parameter θ ∗ sparse; i.e., it has a small number of nonzero entries.
Early studies [2, 6, 15] on sparse linear bandits have revealed that leveraging sparsity assumptions yields bandit algorithms with lower regret than those provided by full-dimensional linear bandit algorithms [3, 4, 8, 1].
However, most existing studies either rely on a particular arm set (e.g., a norm ball), which is unrealistic in many applications, or use computationally intractable algorithms.
If we consider an arbitrary arm set, however, the optimal worst-case regret is Θ (√ sdn ) where s is the sparsity level of θ ∗ , which means that as long as n =
O ( sd ) , there exists an instance for which the algorithm suffers a linear regret [14].
This is in stark contrast to supervised learning
R EGRET B OUND
D ATA - POOR
A SSUMPTIONS
H AO ET AL .
[12] ˜ O
( s 2 / 3 C − 2 / 3 min n 2 / 3 )
✓
A SPANS R d
H AO ET AL .
[12] Ω ( s 1 / 3 κ − 2 / 3 n 2 / 3 ) ✓
A SPANS
[Table]
d
A LGORITHM 3 (
[Table]
URS )
˜ O
( s 2 / 3 H 2 /
[Table]
n 2 / 3 )
✓
A SPANS
[Table]
d
T HEOREM 5 (
[Table]
URS )
[Table]
[Table] [Table]
) ✓
[Table]
H
[Table]
AL .
[Table]
where it is possible to enjoy nontrivial prediction error bounds for n
= o ( d ) [11].
This motivates a natural research question: Can we develop computationally efficient sparse linear bandit algorithms that allow a generic arm set yet enjoy nonvacuous bounds in the data-poor regime by exploiting problem-dependent characteristics?
The seminal work of Hao et al.
[12] provides a positive answer to this question.
They propose algorithms that enjoy nonvacuous regret bounds with an arbitrary arm set in the data poor regime using Lasso.
Specifically, they have obtained a regret bound of ˜ O
(C min − 2 / 3 s 2 / 3 n 2 / 3 ) where C min is an arm-set-dependent quantity.
However, their work still left a few open problems.
First, their regret upper bound does not match with their lower bound Ω (C min − 1 / 3 s 1 / 3 n 2 / 3 ) .
Second, it is not clear if
C min is the right problem-dependent constant that captures the geometry of the arm set.
In this paper, we make significant progress in high-dimensional linear regression and sparse linear bandits, which resolves or partly answers the aforementioned open problems.
First (Section 3), we propose a novel and computationally efficient estimator called P OP A RT (POPulation covariance regression with hARd Thresholding) that enjoys a tighter ℓ 1 norm recovery bound than the de facto standard sparse linear regression method Lasso in many problems.
Motivated by the ℓ 1 norm recovery bound of P OP A RT , we develop a computationally-tractable design of experiment objective for finding the sampling distribution that minimize the error bound of P OP A RT , which is useful in settings where we have control on the sampling distribution (such as compressed sensing).
Our design of experiments results in an ℓ 1 norm error bound that depends on the measurement set dependent quantity denoted by H 2 ∗ (see Eq.
(5) for precise definition) that is provably better than C min −that appears in the ℓ 1 norm error bound used in Hao et al.
[12], thus leading to an improved planning method for sparse linear bandits.
Second (Section 4), Using P OP A RT , we design new algorithms for the sparse linear bandit problem, and improve the regret upper bound of prior work [12]; see Tablefor the summary.
Third (Section 5), We prove a matching lower bound in data-poor regime, showing that the regret rate obtained by our algorithm is optimal.
The key insight in our lower bound is a novel application of the algorithmic symmetrization technique [21].
Unlike the conjecture of Hao et al.
[12, Remark 4.5], the improvable part was not the algorithm but the lower bound for sparsity s .
We empirically verify our theoretical findings in Section 6 where P OP A RT shows a favorable performance over Lasso.
Finally, we conclude our paper with future research enabled by P OP A RT in
Section 7.
For space constraint, we discuss related work in Appendix A but closely related studies are discussed in depth throughout the paper.
Problem Definition and Preliminaries
Sparse linear bandits.
We study the sparse linear bandit learning setting, where the learner is given access to an action space
A ⊂ { a ∈ R d ∶ ∥ a ∥ ∞ ≤ 1 } , and repeatedly interacts with the environment as follows: at each round t = 1 ,...,n , the learner chooses some action A t ∈ A , and receives reward feedback r t = ⟨ θ ∗ ,A t ⟩+ η t , where θ ∗ ∈ R
d is the underlying reward predictor, and η t is an independent zero-mean σ -subgaussian noise.
We assume that θ ∗ is s -sparse; that is, it has at most s nonzero entries.
The goal of the learner is to minimize its pseudo-regret defined as
[Equation]
Experimental design for linear regression.
In the experimental design for linear regression problem, one has a pool of unlabeled examples X , and some underlying predictor θ ∗ to be learned.
Querying the label of x , i.e. conducting experiment x , reveals a random label y
=
⟨ θ ∗ ,x ⟩ + η associated with it, where η is a zero mean noise random variable.
The goal is to accurately estimate θ ∗ , while using as few queries x as possible.
Definition 1.
(Population covariance matrix Q ) Let P(X) be the space of probability measures over X with the Borel σ -algebra, and define the population covariance matrix for the distribution µ ∈ P(X) as follows:
[Equation]
Classical approaches for experimental design focus on finding a distribution µ such that its induced population covariance matrix Q ( µ ) has properties amenable for building a low-error estimator, such as D-, A-, G-optimality [10].
Compatibility condition for Lasso.
For a positive definite matrix Σ ∈ R d × d and a sparsity level s ∈ [ d ] ∶= { 1 ,...,d } , we define its compatibility constant ϕ 20 ( Σ ,s ) as follows:
[Equation]
where v S ∈ R d denotes the vector that agrees with v in coordinates in S and 0 everywhere else and v − S ∈ R
d denotes v − v S .
Notation.
Let e i be the i -th canonical basis vector.
We define [ x ] = { 1 , 2 ,...,x } .
Let supp ( θ ) be the set of coordinate indices i where θ i ≠ 0 .
We use a ≲ b to denote that there exists an absolute constant c such that a ≤ cb .
Improved Linear Regression and Experimental Design for Sparse Models
In this section, we discuss our novel sparse linear estimator P OP A RT for the setting where the population covariance matrix is known and show its strong theoretical properties.
We then present a variation of P OP A RT called W ARM -P OP A RT that amends a potential weakness of P OP A RT , followed by our novel experimental design for P OP A RT and discuss its merit over prior art.
P OP A RT (POPulation covariance regression with hARd Thresholding).
Unlike typical estimators for the statistical learning setup, our main estimator P OP A RT described in Algorithm 1 takes the population covariance matrix denoted by Q as input.
We summarize our assumption for P OP A RT .
Assumption 1.
(Assumptions on the input of P OP A RT )
There exists µ such that the input data points {( X t ,Y t )} nt = 1 satisfy that X t i.i.d.
∼ µ and Q = Q ( µ ) ∶=
E X ∼ µ [ XX ⊺ ] .
Furthermore, Y t = ⟨ θ ∗ ,X t ⟩+ η t with η t being zero-mean σ -subgaussian noise.
Also, R 0 ≥ max a ∈A ∣⟨ a,θ ∗ − θ 0 ⟩∣ . Algorithm θ { ˜ θ ti = ⟨ ˜ θ t e i ⟩} nt =
α δ 2 d α
= ¿``(cid:192) 2log 2 dδ n ( R 20 + σ 2 )( Q − 1 ) ii ( 1 + 2log 2 dδ n − 2log 2 dδ ) 7: λ ( θ ′ ∶= [ θ i 1 (∣ θ ′ i ∣ > λ i )] di = 1 where λ i is defined in Proposition 1. P OP A RT consists of several stages.
In the first stage, for each ( X t ,Y t ) pair, we create a onesample estimator ˜ θ t (step 4).
The estimator, ˜ θ t , can be viewed as a generalization of doubly-robust estimator [7, 9] for linear models.
Specifically, it is the sum of two parts: one is the pilot estimator θ that is a hyperparameter of P OP A RT ; the other is Q ( µ ) − 1 X t ( Y t − ⟨ X t ,θ 0 ⟩) , an unbiased estimator of the difference θ ∗ − θ 0 .
Thus, it is not hard to see that ˜ θ t is an unbiased estimator of θ ∗ .
As we will see in Theorem 1, the variance of ˜ θ t is smaller when θ 0 is closer to θ ∗ , showing the advantage of allowing a pilot estimator θ 0 as input.
If no good pilot estimator is available a priori, one can set θ 0 = 0 .
From the discussion above, it is natural to take an average of ˜ θ t .
Indeed, when n is large, the population covariance matrix Q ( π ) is close to empirical covariance matrix ˆ Q ∶= 1 n ∑ nt = 1 X t X ⊺ t , which makes ˆ θ avg ∶= 1 n ∑ nt = 1 ˜ θ t close to the ordinary least squares estimator ˆ θ OLS =
ˆ Q − 1 ( 1 n ∑ nt = 1 X t Y t ) .
However, for technical reasons, the concentration property of ˜ θ avg is hard to establish.
This motivates P OP
A RT ’s second stage (step 6), where, for each coordinate i ∈ [ d ] , we employ Catoni’s estimator [18] (see
Appendix B for a recap) to obtain an intermediate estimate for each θ ∗ i , namely θ ′ i .
To use Catoni’s estimator, we need to have an upper bound of the variance of θ ′ i for its α i parameter.
A direct calculation yields that, for all i ∈
[ d ] and t ∈ [ n ] ,
[Equation]
where ˜ θ ti ∶= ⟨ ˜ θ t ,e i ⟩ .
This implies that ( R 2 0 + σ 2 ) max i ( Q ( µ ) − 1 ) ii is an upper bound of Var ( ˜ θ ti ) .
By the standard concentration inequality of Catoni’s estimator (see Lemma 1), we obtain the following estimation error guarantee for θ ′ i ; the proof can be found in Appendix C.1.
Hereafter, all proofs are deferred to appendix unless noted otherwise.
Proposition 1.
Suppose Assumption 1 holds.
In P OP A RT , for i ∈ [ d ] , if n ≥ 2ln 2 dδ , the following inequality holds with probability 1 − δd :
[Equation]
Proposition 1 shows that, for each coordinate i , ( θ ′ i − λ i ,θ
′ i
+ λ i ) forms a confidence interval for θ ∗ i .
Therefore, if 0 ∉ ( θ ′ i − λ i ,θ ′ i
+ λ i ) , we can infer that θ ∗ i ≠ 0 , i.e., i ∈ supp ( θ ∗ ) .
Based on the observation above, P OP A RT ’s last stage (step 7) performs a hard-thresholding for each of the coordinates of θ ′ , using the threshold
λ i for coordinate
i .
Thanks to the thresholding step, with high probability, ˆ θ ’s support is contained in that of θ ∗ , which means that all coordinates i outside the support of θ ∗ (typically the vast majority of the coordinates when s ≪ d ) satisfy ˆ θ i
= θ ∗ i = 0 .
Meanwhile, for coordinate i ’s in supp ( θ ∗ ) , the estimated value ˆ θ i is not too far from θ ∗ i .
The following theorem states P OP A
RT ’s estimation error bound in terms of its output ˆ θ ’s ℓ ∞ , ℓ 0 , and ℓ 1 errors, respectively.
We remark that replacing hard thresholding in the last stage with soft thresholding enjoys similar guarantees.
Theorem 1.
Take Assumption 1.
Let H 2 ( Q ) ∶= max i ∈
[ d ] ( Q − 1 ) ii .
Then, P OP A RT has the following guarantees with probability at least 1 − δ :
[Equation]
(ii) supp ( ˆ θ ) ⊂ supp ( θ ∗ )
so ∥ ˆ θ − θ ∗ ∥ 0 ≤ s , (iii) ∥
ˆ θ − θ ∗ ∥ 1 ≤ 2 s √ 4 ( R 20 + σ 2 ) H 2 ( Q ( µ ))
[Equation]
Interestingly, P OP A RT has no false positive for identifying the sparsity pattern and enjoys an ℓ ∞ error bound, which is not available from Lasso, to our knowledge.
Unfortunately, a direct comparison with
Lasso is nontrivial since the largest compatibility constant ϕ 20 ( ˆΣ ,s ) is defined as the solution of the optimization problem (2), let alone the fact that ϕ 20 ( ˆΣ ,s ) is a function of the empirical covariance matrix.
While we leave further investigation as future work, our experiment results in Sectionsuggest that there might be a case where P OP A RT makes a meaningful improvement over Lasso.
Proof of Theorem 1.
Let λ ∶= max
[Equation]
log
[Equation]
From Proposition 1 and the union bound, one can check that
[Equation]
with probability 1 − δ .
Therefore, the coordinates in supp ( θ ∗ ) c will be thresholded out because of ∥ θ ′ − θ ∗ ∥ ∞ ≤ λ .
Therefore, (ii) holds and for all i ∈ supp ( θ ∗ ) c , ∣ ˆ θ i − θ ∗ i ∣ = 0 .
By definition, ˆ θ = clip λ ( θ ′ ) , we can say that ∥ ˆ θ − θ ′ ∥ ∞ ≤ λ .
Plus, by Eq.
(3), ∥ θ ′ − θ ∗ ∥ ∞ ≤ λ .
By the triangle inequality, ∥ θ ∗ − ˆ θ ∥ ∞ ≤ 2 λ .
Therefore, (i) holds.
Lastly, (iii) can be argued as follows:
[Equation]
W ARM -P OP
A RT :
Improved guarantee by warmup.
One drawback of the P OP A RT estimator is that its estimation error scales with √ R 20 + σ 2 , which can be very large when R 0 is large.
One may attempt to use the fact that P OP A RT allows a pilot estimator θ 0 to address this issue since R 0 gets smaller as θ 0 is closer to θ ∗ .
However, it is a priori unclear how to obtain a θ 0 close to θ ∗ as θ ∗ is the unknown parameter that we wanted to estimate in the first place.
To get around this “chicken and egg” problem, we propose to introduce a warmup stage, which we call W ARM
-P OP
A RT (Algorithm 2).
W ARM -P OP
A RT consists of two stages.
For the first warmup stage, the algorithm runs P OP A RT with the zero vector as the pilot estimator and with the first half of the samples to obtain a coarse estimator denoted by ˆ θ 0 which guarantees that for large enough n 0 , ∥ ˆ θ 0 − θ ∗ ∥ 1 ≤ σ .
In the second stage, using ˆ θ 0 as the pilot estimator, it runs P OP A RT on the remaining half of the samples.
1: Input: Samples δ .
The following corollary states the estimation error bound of the output estimator ˆ θ .
Compared with
P OP A RT ’s ℓ 1 recovery guarantee,
W ARM -P OP
A RT ’s ℓ 1 recovery guarantee (Equation (4)) has no dependence on R max ; its dependence on R max only appears in the lower bound requirement for n 0 .
Corollary 1.
Take Assumption 1 without the condition on R 0 .
Assume that R max ≥ max a ∈A ∣⟨ a,θ ∗ ⟩∣ , and n 0 >
32 s 2 ( R 2max + σ 2 ) H 2 ( Q ( µ )) σlog 2 dδ .
Then, W ARM -P OP
A RT has, with probability at least 1 − 2 δ ,
[Equation]
Remark 1.
In Algorithm 2, we choose P OP A RT as our coarse estimator, but we can freely change the coarse estimation step (step 3) to other principled estimation methods (such as Lasso) without affecting the main estimation error bound (4); the only change will be the lower bound requirement of n 0 to another problem-dependent constant.
Remark 2.
W ARM -P OP
A RT requires the knowledge of R max , an upper bound of max a ∈A ∣⟨
θ ∗ ,a ⟩∣ ; this requirement can be relaxed by changing the last argument of the coarse estimation step (step 3) from R max , to some function f ( n 0 ) such that f ( n 0 ) =
ω ( 1 ) and f ( n 0 ) =
o (√ n 0 ) (say, σn 14 0 ); with this change, a result analogous to Corollary 1 can be proved with a different lower bound requirement of n 0 .
A novel and efficient experimental design for sparse linear estimation.
In the experimental design setting where the learner has freedom to design the underlying sampling distribution µ , the ℓ 1 error bound of P OP A RT and W ARM -P OP
A RT naturally motivates a design criterion.
Specifically, we can choose µ that minimizes H 2 ( Q ( µ )) , which gives the lowest estimation error guarantee.
We denote the optimal value of H 2 ( Q ( µ )) by
[Equation]
The minimization of H 2 ( Q ( µ )) is a convex optimization problem, which admits efficient methods for finding the solution.
Intuitively, H 2 ∗ captures the geometry of the action set A .
To compare with previous studies that design a sampling distribution for Lasso, we first review the standard ℓ 1 error bound of Lasso.
Theorem 2.
(Buhlmann and van de Geer [5, Theorem 6.1])
With probability at least 1 − 2 δ , the ℓ 1 -estimation error of the optimal Lasso solution ˆ θ Lasso [5, Eq.
(2.2)] with λ = √ 2log ( 2 d / δ )/ n satisfies
[Equation]
where ϕ 0 ( ˆΣ ,s ) 2 is the compatibility constant with respect to the empirical covariance matrix ˆΣ =n ∑ nt = 1 X t X ⊺ t and the sparsity s in Eq.
Ideally, for Lasso, experiment design which minimizes the compatibility constant will guarantee the best estimation error bound within a fixed number of samples n .
However, naively, the computation of the compatibility constant is intractable since Eq.
(2) is a combinatorial optimization problem which is usually difficult to compute.
One simple approach taken by Hao et al.
[12] is to use the following computationally tractable surrogate of ϕ 20 ( ˆΣ ,s ) :
[Equation]
where λ min ( A ) denotes the minimum eigenvalue of a matrix A .
With the choice of sampling distribution µ = argmax
[Equation]
, and n ≥ ˜Ω ( s ⋅ polylog [Equation] ) , with high probability, ϕ 20 ( ˆΣ ,s ) ≥
C min / 2 holds [20, Theorem 1.8], and one can replace ϕ 0 ( ˆΣ ,s ) to C min / 2 in Theorem 2 to get the following corollary:
Corollary 2.
With probability at least 1 − exp (− cn ) − 2 δ for some universal constant c , the ℓ 1estimation error of the optimal Lasso solution ˆ θ Lasso satisfies
[Equation]
The following proposition shows that our estimator has a better error bound compared to the surrogate experimental design for Lasso of Hao et al.
Proposition 2.
We have H 2 ∗ ≤ C − 1min ≤ dH 2 ∗ .
Furthermore, there exist arm sets for which either of the inequalities is tight up to a constant factor.
Therefore, our new estimator has ℓ 1 error guarantees at least a factor C − 1 / 2 min better than that provided by [12], as follows: when we choose the µ as the solution of the Eq.
(5), then ( RHS of (4) )
≲ sσH ∗
[Equation]
In addition, we also prove that there exists a case where our estimator has an d / s -order better error bound compared to the traditional lasso bound in Theorem 2, although this is not in terms of the compatibility constant of the empirical covariance matrix ˆΣ .
Proposition 3.
There exists an action set A and an absolute constant C 1 > 0 such that
[Equation]
For the detailed proof about Proposition 2 and Proposition 3, see Section D in Appendix.
Improved Sparse Linear Bandits using W ARM -P OP A RT
We now apply our new W ARM -P OP
A RT sparse estimation algorithm to design new sparse linear bandit algorithms.
Following prior work [12], we adopt the classical Explore-then-Commit (ETC) framework for algorithm design, and use P OP A RT with experimental design to perform exploration.
As we will see, the tighter ℓ 1 estimation error bound of our P OP A RT -based estimators helps us obtain an improved regret bound.
Algorithm 3 Explore then commit with W ARM -P OP
A RT reward parameter R max denote the do
9: end for
Sparse linear bandit with W ARM -P OP
A RT .
Our first new algorithm, Explore then Commit with
W ARM -P OP
A RT (Algorithm 3), proceeds as follows.
For the exploration stage, which consists of the first n 0 rounds, it solves the optimization problem (5) to find µ ∗ , the optimal sampling distribution for P OP A RT and samples from it to collect a dataset for the estimation of θ ∗ .
Then, we use this dataset to compute the W ARM -P OP
A RT estimator ˆ θ .
Finally, in the commit stage, which consists of the remaining n − n 0 rounds, we take the greedy action with respect to ˆ θ .
We prove the following regret guarantee of Algorithm 3:
Theorem 3.
If Algorithm 3 has input time horizon n > 16 √ 2 R max ( R 2max + σ 2 ) 3 / 2 H 2 ∗ s 2 σlog 2 dδ , action set A ⊂ [− 1 , + 1 ] d , and exploration length n 0 = 4 ( s 2 σ 2 H 2 ∗ n 2 log 2 dδ R − 2max ) 13 , λ 1 = 4 σ √ H 2 ∗ n 0 log 2 dδ , then with probability at least 1 − 2 δ , Reg ( n ) ≤ 8 R 1 / 3max ( s 2 σ 2 H 2 ∗ n 2 log 2 dδ ) 13 .
Proof.
From Corollary 1, ∥ ˆ θ − θ ∗ ∥ 1 ≤ 2 sλ 1 with probability at least 1 − 2 δ .
Therefore, with probability 1 − 2 δ ,
[Equation]
δ and optimizing the right hand side with respect to n 0 leads to the desired upper bound.
Compared with Hao et al.
[12]’s regret bound ˜ O
(( R max s 2 σ 2 C min − 2 n 2 ) 1 / 3 ) 1 , Algorithm 3’s regret bound ˜ O
(( R max s 2 σ 2 H 2 ∗ n 2 ) 1 / 3 ) is at most ˜ O
(( R max s 2 σ 2 C min − 1 n 2 ) 1 / 3 ) , which is at least a factor
C min 13 smaller.
As we will see in Section 5, we show that the regret upper bound provided by
Theorem 3 is unimprovable in general, answering an open question of [12].
Improved upper bound with minimum signal condition.
Our second new algorithm, Algorithm 4, similarly uses W ARM -P OP
A RT under an additional minimum signal condition.
Assumption 2 (Minimum signal) .
There exists a known lower bound m > 0 such that min j ∈ supp ( θ ∗ ) ∣ θ ∗ j ∣ > m .
At a high level, Algorithm 4 uses the first n 2 rounds for identifying the support of θ ∗ ; the ℓ ∞ recovery guarantee of W ARM -P OP
A RT makes it suitable for this task.
Under the minimal signal condition and a large enough n 2 , it is guaranteed that ˆ θ 2 ’s support equals exactly the support of θ ∗ .
After identifying the support of θ ∗ , Algorithm 4 treats this as a s -dimensional linear bandit problem by discarding the remaining d − s coordinates of the arm covariates, and perform phase elimination algorithm [14,
Section 22.1] therein.
The following theorem provides a regret upper bound of Algorithm 4.
[Equation]
[Equation]
[Equation]
[Equation]
Theorem 4.
If Algorithm 4 has input time horizon n > max
[Equation]
σ
[Equation]
H
[Equation]
m 2 , [Equation] s [Equation]
R 2max +
[Equation]
action set A ⊂ [− 1 , 1 ] d , upper bound of the reward R max , then with probability at least 1 − 2 δ , the following regret upper bound of the Algorithm 4 holds: for universal constant C > 0 ,
[Equation]
For sufficiently large n , the second term dominates, and we obtain an O (√ sn ) regret upper bound.
Theorem 4 provides two major improvements compared to Hao et al.
[12, Algorithm 2].
First, when m is moderately small (so that the first subterm in the first term dominates), it shortens the length of the exploration phase n 2 by a factor of s ⋅ C min H 2 ∗ .
Second, compared with the regret bound ˜ O (
[Equation]
√ sn ) provided by [12], our main regret term ˜ O (√ sn ) is more interpretable and can be much lower.
Matching lower bound
We show the following theorem that establishes the optimality of Algorithm 3.
This solves the open problem of Hao et al.
[12, Remark 4.5] on the optimal order of regret in terms of sparsity and action set geometry in sparse linear bandits.
Theorem 5.
For any algorithm, any s,d,κ that satisfies d ≥ max ( n 1 / 3 s 4 / 3 κ − 4 / 3 , ( s + 1 ) 2 ) and n > 8 κs 2 , there exists a linear bandit environment an action set A and a s -sparse θ ∈ R d , such that
[Equation]
[Equation]
We give an overview of our lower bound proof techniques, and defer the details to Appendix F.
Change of measure technique.
Generally, researchers prove the lower bound by comparing two instances based on the information theory inequalities, such as Pinsker’s inequality, or BregtanolleHuber inequality.
In this proof, we also use two instances θ and θ ′ , but we use the change of measure technique, to help lower bound the probability of events more freely.
Specifically, for any event A ,
[Equation]
Symmetrization.
We utilize the algorithmic symmetrization technique of Simchowitz et al.
Stoltz et al.
[22], which makes it suffice to focus on proving lower bounds against symmetric algorithms.
Definition 2 (Symmetric Algorithm) .
An algorithm Alg is symmetric if for any permutation π ∈
Sym ( d ) , θ ∈ R d , { a t } nt
[Equation]
A
[Equation]
where for vector v , π ( v ) ∈ R d denotes its permuted version that moves v i to the π ( i ) -th position.
This approach can help us to exploit the symmetry of θ ′ to lower bound the right hand side of (8); below, Π ∶= { π ′ ∶ π
( θ ′ )
= θ ′ } is the set of permutations that keep θ ′ invariant, and A is an event invariant under Π :
[Equation] [Equation]
⎞
[Equation]
⎤⎥⎥⎥⎥⎦ which helps us use combinatorial tools over the actions for the lower bound proof.
Experimental results
We evaluate the empirical performance of P OP A RT and our proposed experimental design, along with its impact on sparse linear bandits.
One can check our code from here: https://github.com/ jajajang/sparse .
[Figure]
For sparse linear regression and experimental design, we compare our algorithm P OP A
RT with µ being the solution of (5) with two baselines.
The first baseline denoted by C min -Lasso is the method proposed by Hao et al.
[12] that uses Lasso with sampling distribution µ defined by (6).
The second baseline is H 2 -Lasso, uses Lasso with sampling distribution µ defined by (5), which is meant to observe if Lasso can perform better with our experimental design and to see how P OP A RT is compared with Lasso as an estimator since they are given the same data.
Of course, this experimental design is favored towards P OP A RT as we have optimized the design for it, so our intention is to observe if there ever exists a case where P OP A RT works better than Lasso.
For sparse linear bandits, we run a variant of our Algorithm 3 that uses W ARM -P OP A RT in place of
P OP A RT for simplicity.
As a baseline, we use ESTC [12].
For both methods, we use the exploration length prescribed by theory.
We consider two cases:
• Case 1: Hard instance where H 2 ∗ ≪ C − 1min .
We use the action set constructed in Appendix D. where H 2 ∗ and C min shows a gap of Θ ( d ) .
We choose d = 10 , s = 2 , σ = 0 .
• Case 2.
General unit vectors.
In this case, we choose d = 30 , s = 2 , σ = 0 .
1 and the action set A consists of ∣A∣ = 3 d = 90 uniformly random vectors on the unit sphere.
We run each method 30 times and report the average and standard deviation of the ℓ 1 estimation error and the cumulative regret in Figure 1.
Observation.
As we expected from the theoretical analysis, our estimator and bandit algorithm outperform the baselines.
In terms of the ℓ 1 error, for both cases, we see that P OP A RT converges much faster than C min -Lasso for large enough n .
Interestingly, H 2 -Lasso also improves by just using the design computed for P OP A RT in case 1.
At the same time, H 2 -Lasso is inferior than P OP A RT even if they are given the same data points.
While the design was optimized for P OP A RT and P OP A RT has the benefit of using the population covariance, which is unfair, it is still interesting to observe a significant gap between P OP A RT and Lasso.
For sparse linear bandit experiments, while ESTC requires exploration time almost the total length of the time horizon, ours requires a significantly shorter exploration phase in both cases and thus suffers much lower regret.
Conclusion
We have proposed a novel estimator P OP A RT and experimental design for high-dimensional linear regression.
P OP
A RT has not only enabled accurate estimation with computational efficiency but also led to improved sparse linear bandit algorithms.
Furthermore, we have closed the gap between the lower and upper regret bound on an important family of instances in the data-poor regime.
Our work opens up numerous future directions.
For P OP A RT , we speculate that ( Q ( µ ) − 1 ) ii is the statistical limit for testing whether θ ∗ i = 0 or not – it would be a valuable investigation to prove or disprove this.
We believe this will also help investigate whether the dependence on H 2 ∗ in our regret upper bound is unimprovable (note our matching lower bound is only for a particular family of instances).
Furthermore, it would be interesting to investigate whether we can use P OP A RT without relying on the population covariance; e.g., use estimated covariance from an extra set of unlabeled data or find ways to use the empirical covariance directly.
For sparse linear bandits, it would be interesting to develop an algorithm that achieves the data-poor regime optimal regret and data-rich regime optimal regret √ sdn simultaneously.
Furthermore, it would be interesting to extend our result to changing arm set, which poses a great challenge in planning.
Acknowledgments and Disclosure of Funding
We thank Ning Hao for helpful discussions on theoretical guarantees of Lasso.
Kwang-Sung Jun is supported by Data Science Academy and Research Innovation & Impact at University of Arizona.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] We are not aware of any negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] But it was run on a 5-year-old laptop and no significant computation was required.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[No] (b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]
Appendix
Table of Contents
[Table] [Table]
D Proof
2 and Proposition
[Table]
(9 holds . . . . . . . . . . . . . . . . . . . .
E Proofs
Sparse Linear Bandits
[Table]
F
Proof
of
Lower
Bound
[Table] [Table]
G Experiment details
A Related work
Sparse linear bandits.
The sparse linear bandit problem is a natural extension of sparse linear regression to the bandit setup where the goal is to enjoy low regret in the high-dimensional setting by levering the sparsity of the unknown parameter θ ∗ .
The first study we are aware of is Abbasi-Yadkori et al.
[1] that achieves a ˜ O (√ sdn ) regret bound with a computationally intractable method, which is later shown to be optimal by Lattimore and Szepesvári [14, Section 24] yet is not computationally efficient.
Since then, several approaches have been proposed.
A large body of literature either assumes that the arm set is restricted to a continuous set (e.g., a norm ball) [4, 15] or that the set of available arms at every round is drawn in a time-varying manner, and playing arms greedily still induces a ‘nice’ arm distribution such as satisfying compatibility or restricted eigenvalue conditions [2, 13, 21, 17].
These assumptions allow them to leverage existing theoretical guarantees of Lasso.
In contrast, we follow Hao et al.
[10] and consider arm sets that are fixed throughout the bandit game without making further assumptions about the arm set.
While this setup is interesting in its own for not having restrictive assumptions, it is also an important stepping stone towards efficient bandit algorithms for the more generic yet challenging setup of changing arm sets without any distributional assumptions.
Our work is a direct improvement over Hao et al.
[10], in that we close the gap between upper and lower bounds on the optimal worst-case regret; we refer to Table 1 for a detailed comparison.
Sparse linear regression.
Natural attempts for solving sparse linear bandits are to turn to existing results from sparse linear regression.
While best subset selection (BSS) is a straightforward approach of trying all the possible sparsity patterns that achieves good guarantees, its computational complexity is prohibitive [8].
As a computationally efficient alternative, Lasso is arguably the most popular approach for sparse linear regression for its simplicity and effectiveness [24].
However, Lasso has an inferior ℓ 1 norm error bound than BSS, perhaps due to its bias [25].
Rather than turning to existing results from sparse linear regression, we propose a novel estimator, P OP A RT , by leveraging the fact that the setup allows us to design the sampling distribution, which allows a better ℓ 1 norm error bound than Lasso except for the dependence on the range of the mean response variable.
Experimental design.
In the linear bandit field, researchers often use experimental design to get the best estimator within the limited budget [22, 23, 3, 7, 16].
Especially, there were a few attempts using the population covariance based estimator instead of the traditional empirical covariance matrix [16, 23].
However, our study is the first approach that designs the experiment for minimizing the variance of each coordinate of the estimator uniformly, to the best of our knowledge.
For experimental design for sparse linear regression, Ravi et al.
[18] propose heuristic approaches that ensures the design distribution satisfy incoherence conditions and restricted isometry property (RIP).
Eftekhari et al.
[6] study the design of c -optimal experiments in sparse regression models, where the goal is to estimate ⟨ c,θ ∗ ⟩ with low error for some c ∈ R d ; our experimental design task can be seen as simultaneously estimating ⟨ c,θ ∗ ⟩ for all c = e 1 ,...,e d .
Huang et al.
[11] propose algorithms for optimal experimental design, tailored to minimizing the asymptotic variance of the debiased Lasso estimator [12].
In contrast, our results are based on finite-sample analyses.
In the theoretical computer science literature, a line of work on sketching also provides provably compressed sensing and sparse recovery algorithms [See 9, for an overview]; however, they mostly focus on using measurements (covariates) that are in { 0 , 1 } d and {− 1 , 1 } d , as opposed to general measurement sets in R d .
Regression with the population covariance matrix.
There are a few studies that consider regression with the population covariance matrix: Camilleri et al.
[3] devise the novel scheme for the experimental design for the kernel bandits and obtain a new estimator called RIPS that leverages the population covariance matrix and robust mean estimator like P OP A RT .
Mason et al.
[16] solve the level set estimation problem using RIPS.
Tao et al.
[23] also employ a similar estimator, but they do not use robust mean estimators and result in a weaker form of error bound involving additional lower order terms.
The main difference of our work from all these papers is that they do not address sparse linear models.
In particular, they do not perform thresholding nor provide l ∞ or l 1 recovery guarantees for the sparse parameter.
B Catoni’s Estimator
Definition 3 (Catoni’s estimator [5]) .
For the i.i.d random variables Z 1 , ⋯ ,Z n , Catoni’s mean estimator Catoni ({ Z i } ni = 1 ,δ,α ) with error rate δ and the weight parameter α is defined as the unique value y which satisfies
[Equation]
where ψ ( x ) ∶= sign ( x ) log ( 1 + ∣ x ∣ + x 2 / 2 ) .
Lemma 1 (Catoni’s estimator guarantee [5]) .
For the i.i.d random variable X 1 , ⋯ ,X n with mean µ , let ˆ µ be their Catoni’s estimator with error rate δ with the weight parameter α ∶= ¿ ``(cid:192)
[Equation]
Then with probability at least 1 − 2 δ , the following inequality holds:
[Equation]
C Proofs for P OP A RT and W ARM -P OP A RT C.1 Proof for Proposition
Proof.
To lighten the notation, in this proof, we write Q = Q ( µ ) , and let ( ˜ θ,A,η ) denote random vectors distributed identically to ( ˜ θ 1 ,A 1 ,η 1 ) .
First, observe that E [ ˜ θ ]
= θ ∗ .
We now use the law of total variance to decompose the covariance matrix of ˜ θ , by first conditioning on X :
[Equation]
[ ˜ θ ∣ X ]) ⊺ ]
For the first term,
[Equation] [Equation]
For the second term,
[Equation]
Combining the above two bounds, we have Var ( ˜ θ ) =
E [( ˜ θ − θ ∗ )( ˜ θ − θ ∗ ) ⊺ ] ⪯ ( R 20 + σ 2 ) Q − 1 .
Therefore, we can bound Var ( ˜ θ i ) as follows:
[Equation]
By the theoretical guarantee of the Catoni’s estimator (Lemma 1 in the Appendix), the desired inequality holds.
C.2 Full version of Corollary 1 and its proof
Corollary 3.
If W ARM -P OP
A RT receives inputs { X t ,Y t } n 0 t = 1 drawn from µ , Q ( µ ) , failure probability δ , and R max such that R max ≥ max a ∈A ∣⟨ a,θ ∗ ⟩∣ , and n 0 >
32 s 2 ( R 2max + σ 2 ) H 2 ( Q ( µ )) σlog 2 dδ , then all the following items hold with probability at least 1 − 2 δ :
(i) ∥
ˆ θ − θ ∗ ∥ ∞ ≤ 8 σH ( Q ) √
ln 2 dδ n(ii) supp ( ˆ θ ) ⊂ supp ( θ ∗ )
so ∥ ˆ θ − θ ∗ ∥ 0 ≤ s (iii) ∥
ˆ θ − θ ∗ ∥ 1 ≤ 8 sσH ( Q ) √
ln 2 dδ n
Proof.
Since n 0 is sufficiently large, from the Theorem 1 with R
= R max we can say that ∥ θ 0 − θ ∗ ∥ 1 ≤ σ .
Applying Theorem 1 again with R 0 = σ we can get all (i), (ii), (iii) directly.
D Proof of Proposition 2 and Proposition
First, we will prove H 2 ∗ ≤ C − 1min ≤ dH 2 ∗ .
We will deal with equality in the next subsection.
Proof.
For any positive definite matrix Q ∈ R d × d ,
[Equation]
Now, let the solution of the Eq.
(5) and Eq.
(6) as µ H and µ C , respectively.
Then, by the rightmost inequality of (9) we have
[Equation]
and by the leftmost inequality of the (9) we have
[Equation]
Therefore, the inequality part of the Proposition 2 holds.
D.1 The case of when the equality in Eq. (9 holds
For an example of H 2 ∗ =
C − 1min , consider A = { e i ∣ i = 1 , ⋯ ,d } ; it can be seen that H 2 ∗ =
C − 1min = d .
We now present an example where C min =
Θ ( dH 2 ∗ ) .
Consider A
= { a 1 ,...,a d } , where
[Equation]
and we will calculate H 2 ( Q ( π )) and λ min ( Q ( π )) for the optimal sampling distributions π to achieve
H 2 ∗ and C min , respectively.
D.1.1 Prove that the optimal π satisfies π ( a 2 ) = π ( a 3 ) = ⋯ = π ( a d )
We will first show that for both objectives H 2 ( Q ( π )) and λ min ( Q ( π )) , there exists an optimal sampling distribution π such that π ( a 2 ) =
π ( a 3 ) = ⋯ =
π ( a d ) .
Denote by a ∶= π ( a 1 ) .
Fix a .
For notational convenience, let π ( a i ) ∶=
b i and b = ( b 2 ,b 3 , ⋯ ,b d ) ∈ R
d − 1 .
Then the covariance matrix Q ( π ) (abbreviated as Q ) has the following form:
[Equation]
⎤⎥⎥⎥ ⎥⎥⎥⎥⎥⎥⎦
[Equation]
After some calculation, one can get the determinant det ( Q ) = a ( Π di = 2 b i ) d d and the cofactor
[Equation]
and therefore
[Equation]
When a is a fixed parameter, ( Q − 1 ) ii = d 2 a + db i and therefore the arg max i ( Q − 1 ) ii = arg min b i .
Under the constraint ∑ di = 2 b i = 1 − a , the optimal solution is reached when b 2 = b 3 = ⋯ =
b d .
For the C min case, we will utilize symmetry of λ min ( Q ( π )) .
Note that λ min ( Q ) is a concave function w.r.t Q .
Suppose that the ( b ′ 2 ,b ′ 3 , ⋯ ,b ′ d ) = arg b max λ min ( Q ( b )) .
Then from the symmetry, for any cyclic permutation P , all ( b ′ P i ( 2 ) ,
b ′ P
i ( 3 ) , ⋯ ,b ′ P
i ( d ) )
i = 1 , ⋯ ,d − 1 are also the maximum.
Therefore, by the Jensen’s inequality,
[Equation]
Therefore, we can conclude b 2 = b 3 = ⋯ =
b d = 1 − a d − 1 is indeed an optimal choice when a is fixed.
From now on, consider only the strategy π that satisfies π ( a 2 )
=
π ( a 3 ) = ⋯ =
π ( a d ) for this section, and let a = π ( a 1 ) and b = π ( a 2 ) .
Then a + ( d − 1 ) b = 1 .
Now the covariance matrix induced by π is of the following form:
[Equation]
⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦ (11)
D.1.2 Calculating H 2 ∗
One can calculate det ( Q ) =
ad ( bd ) d − 1 (using again the cofactor method) and the cofactor
[Equation]
otherwise and therefore
[Equation]
otherwise ( Q − 1 ) ii is always larger than ( Q − 1 ) 11 , and by taking derivatives, the b that minimizes the ( Q − 1 ) 22 is √ dd − 1 − 1 = 1 √ d
[Equation]
1 (√ d +√ d − 1 ) , and the corresponding ( Q − 1 ) 22 =
d ( √ d +
[Equation]
) 2 = Θ ( d 2 ) .
In this case, a = d − √ d ( d − 1 ) (close to 1/2).
D.1.3 Calculating C min
The eigenspectrum of Q is
[Equation]
Therefore,
[Equation]
Note that for the optimal π , a > 0 ; otherwise Q ( π ) is not invertible.
Therefore, b = 1 − a d − 1 < 1 d − 1 .
Hence, one can check that √( 1 + ( d 2 − 2 d + 2 ) b ) 2 − 4 b + 4 ( d − 1 ) b 2 ≥ 1 , so 1 λ min =
[Equation]
) b 2 1 −( d − 1 ) b ) .
Now we investigate arg min b 1 λ min ( b ) .
Recall that a + ( d − 1 ) b = 1 , b < 1 /( d − 1 ) .
Thus, − 4 b + 4 ( d − 1 ) b 2 < 0 .
Therefore, the big square root term is smaller than ( 1 + ( d 2 − 2 d + 2 ) b ) , which means if we let f ( b ) =
d ( 1 +( d 2 − 2 d + 2 ) b ) 2 b ( 1 −( d − 1 ) b ) , then f
[Equation]
[Equation]
It remains to calculate min b f ( b ) .
A few derivative calculations show that b ∗ = arg min b f ( b ) = −( d − 1 )+√ d 3 − 1 d 3 − 1 −( d − 1 ) 2 =
Θ ( d − 3 / 2 ) and min b f ( b ) =
Θ ( d 3 ) .
D.1.4 Lower bound of 1 ϕ 20 ( Q ( π ) ,s )
It is difficult to directly calculate the compatibility constant of Q ( π ) , but we can bound it using the diagonal entries of Q ( π ) .
Note that
[Equation]
and therefore 1 ϕ 2 ≥
[Equation]
We will use the notation in D.1.1.
Denote by a ∶= π ( a 1 ) .
For notational convenience, let π ( a i ) ∶=
b i and b = ( b 2 ,b 3 , ⋯ ,b d ) ∈ R
d − 1 .
Then the covariance matrix Q ( π ) (abbreviated as Q ) has the following form:
[Equation]
⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦
[Equation]
[Equation]
From the basic constraint a + ∑ di = 2 b i = 1 , min i ∈[ d ] Q ii = min i ∈{ 2 , ⋯ ,d } b i d
= O ( 1 d 2 ) .
Thereforeϕ 2 = Ω
( d 2 / s ) .
This means even for the best case of the compatibility constant cannot beat the recovery bound of P OP A RT in this action set.
E Proofs for Sparse Linear Bandits E.1 Proof of Theorem
Proof.
From the (i) in Corollary 3, when n 2 = 256 σ 2 H 2 ∗ mlog dδ , with probability at least 1 − 2 δ ,
[Equation]
Therefore, with probability at least 1 − 2 δ , for any index i ∈ supp ( θ ∗ ) C , ˆ θ i = 0 , and for any index j ∈ supp ( θ ∗ ) , ∣ ˆ θ j ∣ > ∣ θ ∗ j ∣ − m 2 > 0 .
Thus, supp ( θ ∗ ) =
supp ( ˆ θ ) with probability at least 1 − 2 δ .
After that, we use the following result about the restricted phase retrieval [14]:
Theorem 6.
(Lattimore and Szepesvári [14], Theorem 22.1)
The n-steps regret of phase elimination algorithm satisfies
[Equation]
for an appropriately chosen universal constant C > 0 .
F Proof of Lower Bound
In this section, we prove Theorem 5.
We start with a restatement of it.
Theorem 7.
(Restatement of the Theorem 5)
For any algorithm, any s,d,κ that satisfies s > 80 ,κ ∈ ( 0 , 1 ) ,d ≥ max ( n 1 / 3 s 4 / 3 κ − 4 / 3 , ( s + 1 ) 2 ) , n > 8 κs 2 , there exists a linear bandit environment with an action set A and a s -sparse θ ∈ R d , such that C min (A) − 1 ≤ κ − 2 , R max ≤ 2 , σ = 1 , and
Reg n ≥ Ω ( κ − 2 / 3 s 2 / 3 n 2 / 3 ) .
In the lower bound instance that establishes Theorem 7, we will prove that 2 κ − 2 ≥ C − 1min ≥ H 2 ∗ (see
Section F.2.7), and conclude that our ˜ O ( H 2 / 3 ∗ s 2 / 3 n 2 / 3 ) regret upper bound of Algorithm 3 has a matching lower bound and conclude that the algorithm and the lower bound are both optimal in this setting.
For convenience, throughout the rest of this section, we prove the following slight variant of Theorem 7, where the dimensonality is d + 1 as opposed to d , and the sparsity is 2 s + 1 as opposed to s ; note that the changes of these parameters do not affect the orders of the regret bounds in terms of them.
Theorem 8.
For any algorithm, any s,d,κ that satisfies s > 40 and is a multiple of 4, κ ∈ ( 0 , 1 ) ,d ≥ max ( n 1 / 3 s 4 / 3 κ − 4 / 3 , ( s + 1 ) 2 ) , n > 8 κs 2 , there exists a linear bandit environment an action set A and a ( 2 s + 1 ) -sparse θ ∈ R d + 1 , such that C min (A) − 1 ≤ 2 κ − 2 , R max ≤ 2 , σ = 1 , and
Reg n ≥ Ω ( κ − 2 / 3 s 2 / 3 n 2 / 3 ) .
Construction Following the standard minimax lower bound and hypothesis testing terminology, we will often refer to an underlying reward predictor θ ∈ R d + 1 as a hypothesis .
Let
[Equation] [Equation]
where ϵ
= κ − 2 / 3 s − 1 / 3 n − 1 / 3 .
We will use Θ s and Θ 2 s as our hypothesis space throughout the proof.
We construct a low-regret action set I and an informative action set H as follows:
[Equation]
H
[Equation]
j
[Equation]
where κ ∈ ( 0 , 1 ) is a constant.
The action set is the union
A
= I ∪ H .
The linear bandit environment parameterized by θ ∈ R d + 1 is defined as: given action taken A t , its reward r t = ⟨ θ,A t ⟩ + η t , where η t ∼ N ( 0 , 1 ) is an independently drawn standard Gaussian noise.
Note that by construction, η t is σ 2 -subgaussian with σ = 1 .
Notations In this section, we will use A =
( A 1 , ⋯ ,A n ) ∈
A n as the random variable about the history of actions.
For a ∈ A n , let T (H ; a ) =
∑ nt = 1 1 ( a t ∈ H) , which represents the total number of pulls of arms in H in the learning process.
For the brevity of the notation, we will write the random variable
T (H ; A ) as T (H) .
Let Sub x = { S ⊂
[ d ]∣∣ S ∣
= x } , the set of subsets of [ d ] which has x elements.
In subsequent proofs, given a bandit algorithm Alg and an bandit environment θ , we use P θ, Alg and E θ, Alg to denote probability and expectation under the probability space induced by the interaction history between them.
For any set of indices S ∈ 2 [ d + 1 ] , let Sym ( S ) be the symmetric group of the set S , and let Π S = {
σ ∈ Sym ([ d + 1 ])
∶ σ ( j ) =
j for all j ∈ [ d + 1 ]/ S } be the set of permutations which permutes only the indices in S , and let Π a ∶ b
= Π { a,a + 1 , ⋯ ,b } .
Structure of the
section Here is the high level idea of the proof structure.
• Reduction using symmetrization (Section F.1) : First, we will prove that the regret lower bound of the symmetric sparse linear bandit algorithms (see Definition 7) is also the lower bound of the general sparse linear bandit algorithms (Lemma 5).
Keen readers would note that our action set construction is symmetric except for ( d + 1 ) -th coordinate, and this is for exploiting the symmetry.
By focus on proving lower bounds for symmetric algorithms, we can exploit the favorable combinatorial properties of our action spaces to establish tighter lower bounds.
• Count the number of mistakes (Section F.2.1): Next, we will prove the core proposition of the lower bound proof, Proposition 4.
This proposition can be summarized as, ‘the learning agent has to pull sufficiently large number of arms in H (informative actions with high regret) to make less mistakes’, where ‘mistakes’ refers to coordinates in the support of θ that has not been ‘hit’ sufficiently by the agent via pulling the low-regret arms
I (See Equation (13) for a formal definition).
This implies an inherent tension between pulling informative, high regret arms H and pulling low regret arms I , which eventually leads to the desired lower bound in Theorem 5.
• Lower bound on symmetric algorithms (Section F.2.2) :
Now it remains to show the proof of Proposition 4.
Here, to improve the Ω ( s 1 / 3 n 2 / 3 ) regret lower bound proved by Hao et al.
[10] to Ω ( s 2 / 3 n 2 / 3 ) , we deviate from their usage of Bretagnolle-Huber inequality, and take a novel combination of various techniques such as change of measure technique, combinatorial calculation by utilizing symmetry (Claim 1).
F.1 Algorithmic symmetrization: reducing lower bounds for general algorithms to symmetric algorithms
In this section, we show how a proof of a lower bound for generic algorithms can be reduced to that of permutation-symmetric (abbrev.
symmetric) (augmented) algorithms (Definition 7), specifically
Lemma 5.
To introduce symmetric algorithms, let us first define some useful terminology.
Definition 4.
(Augmented bandit algorithm; Augmentation of an algorithm).
Define an augmented bandit algorithm as: at time t , choose action A t based on its historical observations ( A s ,r s ) t − 1 s = 1 ; finally it outputs ˆ S ⊂ [ d ] , such that ∣ ˆ S ∣ =
d 2 , and for all i ∈ ˆ
[Equation]
in other words all elements in ˆ S are among the top d 2 most frequently chosen coordinates (including ties) when restricted to arm pull history on I
Given a bandit algorithm Alg , define its augmentation ̃
Alg as: at time step t , use Alg to output A t based on all historical observations ( A s ,r s ) t − 1 s = 1 ; finally, output ˆ S ⊂
[ d ] as its top d 2 coordinates i ∈ [ d ] , with the largest value of ∑ nt = 1 ∣ A t,j ∣ I ( A t ∈ I) , breaking ties in dictionary order.
In other
words, the elements in ˆ S are the top d 2 most frequently chosen coordinates when restricted to arm pull history on I .
Definition 5 (Frequent coordinate set) .
Let U( a ) =
{ U ∈ Sub d / 2 ∣∀ i ∈ U, ∑ nt = 1 ∣ a ti ∣ 1 ( a t ∈ I) ≥ ( d / 2 ) - max {∑ nt = 1 ∣ a tj ∣ 1 ( a t ∈ I)} dj = 1 } where k - max S for a set S ⊆
R is the k -th largest member of
S .
With this notation, for any augmented bandit algorithm, its output ˆ S ∈ U( A ) .
As a result, we will mainly focus on ( a ,U ) ∈
A n × Sub d / 2 such that U ∈ U( a ) , but for the lemmas we keep the generality and consider any U ⊂ [ d ] .
Remark 3.
From the above definitions, it can be readily seen that Alg ’s augmentation, ̃ Alg , is a valid augmented bandit algorithm.
Remark 4.
̃
Alg outputs ˆ S by breaking ties in dictionary order.
While this breaks symmetry by favoring coordinates with lower indices, as we will see in our reduction proof (proof of Lemma 5), we do not require ̃ Alg to be symmetric (we will define symmetry momentarily in Definition 7); instead, we will work on a symmetrized version of ̃ Alg (Definition 10).
Definition 6 (Permutaion over sets of coordinates, and vectors in R d + 1 ) .
Given a permutation σ ∈ Π 1 ∶ d :
• For a subset of coordinates S ⊂
[ d ] , define σ ( S ) ⊂
[ d ] as σ ( S ) ∶= { σ ( i ) ∶ i ∈ S } .
• For vector v ∈ R d + 1 , define σ ( v ) =
( v σ − 1 ( 1 ) ,...,v σ − 1 ( d + 1 ) )
=
P σ v ∈ R d + 1 as the permuted version of v using σ , where P σ =
( e σ ( 1 ) ,...,e σ ( d ) ,e σ ( d + 1 ) )
∈ R ( d + 1 )×
( d + 1 ) is the permutation matrix 2 induced by σ and e j denotes j -th standard basis.
Note that for every i ∈
[ d + 1 ] , σ ( e i ) =
e σ ( i ) .
• For sequence of actions a = ( a 1 ,...,a n ) ∈
A n , define σ ( a ) =
( σ ( a 1 ) ,...,σ ( a n )) as its permuted version using σ .
We will frequently apply the above vector permutation operation in our subsequent proofs, where the vector v ∈ R d + 1 are often taken as actions A t or hypotheses (underlying reward predictors) θ .
Now we are ready to define symmetric augmented bandit algorithms, a special class of bandit algorithms we will focus on.
Definition 7 (Symmetric augmented bandit algorithm) .
An augmented bandit algorithm ̃ AlgS is said to be symmetric , if for any σ ∈ Π 1 ∶ d and any a = ( a 1 ,...,a n ) ∈ A n and U ⊂ [ d ] ,
[Equation]
Note that the above permutation symmetry notion is slightly different from Simchowitz et al.
[20] – here we only consider permutations
in Π 1 ∶ d , i
e
, over the first
d
coordinates (out of all
d +coordinates), whereas Simchowitz et al.
[20] consider permutations over all coordinates (arms).
For symmetric augmented bandit algorithms, we have the following elementary property.
Lemma 2.
For every symmetric augmented bandit algorithm ̃ AlgS , any σ ∈ Π 1 ∶ d and any function f
∶ A n × 2 [ d ] → R ,
[Equation]
Definition 8 (Permutation-invariant action space) .
An action space A is said to be permutationinvariant, if for any π ∈ Π 1 ∶ d ,
[Equation]
By our construction in the beginning of Section F, our action space
A
=
I ∪ H is permutation invariant.
Definition 9 (Permuted algorithm) .
For an augmented bandit algorithm ̃
Alg on a permutationinvariant action space A , and a permutation π ∈ Π 1 ∶ d , define its π -permuted version ̃
Alg π as: first 2 Here we use σ ’s row representation.
permute the [ d ] coordinates using π , and run ̃ Alg with the permuted coordinates.
Formally, at every time step t :
• ̃
Alg outputs some action A ′ t ∈ A , and ̃
Alg π accordingly outputs action
A t = π − 1 ( A ′ t ) ∈
A • Receives reward r t = ⟨ θ,A t ⟩ + η t = ⟨ π ( θ ) ,π ( A t )⟩ + η t = ⟨ π ( θ ) ,A ′ t ⟩ + η t • ̃
Alg outputs ˆ S ′ , and ̃
Alg π outputs ˆ S =
π − 1 ( ˆ S ′ ) .
The following lemma follows straightforwardly from the definition of ̃ Alg π :
Lemma 3.
• For any a = ( a 1 ,...,a n ) ∈
A n and U ⊂ [ d ] , P θ, ̃
Alg π ( A = a , ˆ S = U )
=
P π ( θ ) , ̃ Alg ( A = π ( a ) , ˆ S =
π ( U )) .
• For any function f
∶ A n × 2 [ d ] → R , E θ, ̃
Alg π [ f ( A , ˆ S )]
=
E π ( θ ) , ̃ Alg
[ f ( π − 1 ( A ) ,π − 1 ( ˆ S ))] .
Definition 10.
For an augmented bandit algorithm ̃
Alg on a permutation-invariant action space A , define its symmetrized version ̃
AlgP as: first, choosing π uniformly at random from Π 1 ∶ d , then, run ̃
Alg π on the bandit environment for n rounds.
Lemma 4.
We have the following:
P θ, ̃
AlgP (⋅) =
1 ∣ Π 1 ∶ d ∣ ∑ π ∈ Π 1 ∶ d P θ, ̃ Alg π (⋅) , and E θ, ̃
AlgP [⋅] =
∣ Π 1 ∶ d ∣ ∑ π ∈ Π 1 ∶ d E θ, Alg π [⋅] .
̃
AlgP is a symmetric augmented bandit algorithm.
The definition below formalizes the (pseudo-)regret notion under a specific hypothesis, which provides useful clarifications when using the averaging hammer to argue regret lower bounds.
Definition 11.
Define
[Equation]
as the pseudo-regret of a sequence of actions ( A t ) nt = 1 under hypothesis θ .
The main result of this section is the following lemma that reduces proving lower bounds for general algorithms to proving lower bounds for symmetric augmented algorithms.
Lemma 5.
If for all symmetric augmented bandit algorithms ̃ AlgS , there exists some θ ∈ Θ s ∪ Θ 2 s such that E θ, ̃
AlgS [ Reg ( n,θ )] ≥ R , then, for all bandit algorithms Alg , there exists some θ ′ ∈
Θ s ∪ Θ 2 s such that E θ ′ ,
Alg [ Reg ( n,θ ′ )] ≥ R .
In light of this lemma, in Section F.2, we focus on showing regret lower bounds on symmetric augmented bandit algorithms under hypotheses in Θ s ∪ Θ 2 s .
F.1.1 Deferred Proofs
Proof of Lemma 2.
For any f ∶ A n × 2 [ d ] → R ,
E θ, ̃ AlgS [ f ( A , ˆ S )]
[Equation]
Proof of Lemma 3.
For the first item, denote by A ′ =
( A ′ 1 ,...,A ′ n ) ; for any a = ( a 1 ,...,a n ) ∈ A n and U ⊂ [ d ] ,
[Equation]
switching to ̃
[Equation]
’s perspective)
The second item is the direct consequence of the first item by the following calculation.
[Equation]
Proof of Lemma
[Equation]
The first item follows from the definition of ̃ AlgP .
For the second item
[Equation]
for any permutation σ ∈ Π 1 ∶ d and action history a ∈ A n ,
P
[Equation]
̃
[Equation]
the first item
[Equation] [Equation]
the first item
[Equation]
where in step (*)
[Equation]
we use the observation that for any σ ∈ Π 1 ∶ d , { π ○ σ ∶ π ∈ Π 1 ∶ d } = Π 1 ∶ d .
Proof of Lemma 5.
Given any bandit algorithm Alg
[Equation]
denote by ̃
AlgP as the symmetrized version of its augmentation (Definitions 4 and 10).
Since ̃ AlgP is a symmetric augmented algorithm
[Equation]
by assumption
[Equation]
we have
[Equation]
there exists some θ ∈ Θ s ∪ Θ 2 s ,
[Equation]
[Equation] [Equation]
(Definition 11)
[Equation]
( Alg and ̃ Alg take the same action sequence)
By the pigeonhole principle, there exists π ∈ Π 1 ∶ d which satisfies E π ( θ ) , Alg [ Reg ( n,π ( θ ))] ≥ R , and
this π ( θ ) ∈ Θ s ∪ Θ 2 s is the desired θ ′ in Lemma 5.
F
Lower bound
F.2.1 Count the number of mistake
From now on, by Lemma 5, we will focus on proving regret lower bound for any symmetric augmented algorithm ̃
AlgS (Definition 7).
For the brevity of notation, we omit the ̃ AlgS notations from P and E .
For each history of action A ∈
A n , we define the M θ ( ˆ S ) , the number of false negative mistakes respect to θ as follows:
[Equation]
Let s be multiple of 4.
For ξ = 14 , we like to show the following claim.
Proposition 4.
If P θ ( M θ ( ˆ S ) ≥ s / 4 ) ≤ ξ for all θ ∈ Θ s ∪ Θ 2 s , then ∃ θ ′ ∈
Θ s such that E θ [ T (H)] ≥ Ω ( 1 κ 2 ϵ 2 )
Given the above proposition, we are now ready to prove Theorem 8.
Proof of Theorem 8.
Let a ∗ = arg max a ∈A ⟨ θ,a ⟩ .
If we note that ∑ nt = 1 ∣ A tj ∣ 1 ( A t ∈ I) ≤ 4 s ( n − T
(H))
d for all j ∉ ˆ S (if not, it contradicts ∑ d j = 1 ∑ n t = 1 ∣ A tj ∣ 1 ( A t ∈ I) ≤ 2 s ( n − T
(H)) ) and a ∗ =
( θ + e d + 1 )/ ϵ ∈
I , we can lower bound the regret as follows:
[Equation]
} ( a ∗ j − A tj ) + 1 ( A t ∈ H)] ( ∀ j ∈ [ d ] ∖ supp ( θ ) , θ j = 0 and for all j ∈ supp ( θ ) ∖ { d + 1 } , θ j = ± ϵ )
=
ϵ n ∑ t =
∑ j ∈ supp ( θ )/{ d + 1 } ( 1 − A tj ) + T (H) ( a ∗ =
( θ + e d + 1 )/ ϵ ) ≥ ϵ n ∑ t =
∑ j ∈ M θ ( ˆ S ) ( 1 − ∣
A tj 1 ( A t ∈ I)∣ − ∣ A tj 1 ( A t ∈ H)∣)
+ T (H) ≥
ϵ n ∑ t =
∑ j ∈ M θ ( ˆ S ) ( 1 − ∣
A tj 1 ( A t ∈ I)∣)
+ T (H)( 1 − ϵκ ∣ M θ
( ˆ S )∣) ≥ ϵ
∑ j ∈ M θ ( ˆ S ) n ∑ t =( 1 − ∣ A tj 1 ( A t ∈ I)∣) ( ϵκ ∣ M θ ( ˆ S )∣ ≤ 2 ϵκs ≤ 1 ) ≥
ϵ ∑ j ∈ M θ ( ˆ S )
[ n − n ∑ t =∣ A tj ∣ 1 ( A t ∈ I)] ≥ ϵ
∑ j ∈ M θ ( ˆ S ) ( n − 4 sn d ) ( ∑ nt = 1 ∣ A tj ∣ 1 ( A t ∈ I) ≤ 4 s ( n − T
(H))
d ) ≥ ϵs 4 ( n − 4 sn d ) and therefore 1 ( M θ ( ˆ S ) ≥ s 4 )
Reg n ≥
ϵ s 4 ( n − 4 snd ) .
Therefore, we can lower bound the regret further as follows: If there exists a θ ∈ Θ s ∪ Θ 2 s that satisfies P θ ( M θ ( ˆ S ) ≥ s / 4 ) ≥ ξ , then
E θ
[ Reg n ] ≥ E θ [ 1 ( M θ ( ˆ S ) ≥ s 4 ) Reg n ]
[Equation]
On the other hand, if P θ ( M θ ( ˆ S ) ≥ s / 4 ) ≤ ξ for all θ ∈ Θ s ∪ Θ 2 s , then by Proposition 4 ∃ θ ′ such that E θ ′
[ T (H)] ≥ Ω ( 1 κ 2 ϵ 2 ) .
Combining with the fact that for every θ ∈ Θ s ∪ Θ 2 s , max a ∈I ⟨ θ,a ⟩ − max a ∈H ⟨ θ,a ⟩ ≥ sϵ − ( κsϵ − 1 ) ≥ Ω ( 1 ) , we can finally lower bound the regret as
[Equation]
By construction, ϵ = κ − 2 / 3 s − 1 / 3 n − 1 / 3 , we get the desired regret lower bound Ω ( κ − 2 / 3 s 2 / 3 n 2 / 3 ) result for both cases.
F.2.2 Proof of Proposition
Let’s assume that ∀ θ ∈ Θ s ∪ Θ 2 s , P θ ( M θ ( ˆ S ) ≥ s / 4 ) ≤ ξ .
Define
[Equation]
The following advantage θ ′ θ this way.
Lemma 6.
Let ϕ ∈ R d , π ∈ Π , E 1 ,E 2 ∈ Sub d / 2 be the elements which satisfies π ( ϕ )
= ϕ and π ( E 1 ) =
E 2 .
For any symmetric algorithm AlgS , we have that
[Equation]
Proof.
To see this, note that:
[Equation]
Let Q m ∶= { S ∈ Sub d / 2 ∣ M θ ( S )
= m } and Q ′ m
= { S ∈ Sub d / 2 ∣ M θ ′ ( S )
= m } be the set of size d /sets which has exactly m mistakes with θ and θ ′ , respectively.
Lemma 7.
For our θ , for each E 1 ,E 2 ∈ Q 14 s − l , there exists π ∈ { π 1 ○ π 2 ∣ π 1 ∈ Π 1 ∶ 2 s ,π 2 ∈ Π 2 s + 1 ∶ d } which satisfies π ( E 1 ) =
E 2 .
Similarly, for each E ′ 1 ,E ′ 2 ∈ Q ′
s − l , there exists π ∈ { π 1 ○ π 2 ∣ π 1 ∈ Π 1 ∶ s ,
π 2 ∈ Π s + 1 ∶ d } which satisfies π ( E 1 )
=
E 2 .
Proof.
Since ∣ E 1 ∩ [ 1 ∶ 2 s ]∣ = ∣ E 2 ∩ [ 1 ∶ 2 s ]∣ and ∣ E 1 ∩ [ 2 s + 1 ∶ d ]∣ = ∣ E 2 ∩ [ 2 s + 1 ∶ d ]∣ , there exists π ∈ { π 1 ○ π 2 ∣ π 1 ∈ Π 1 ∶ 2 s ,π 2 ∈ Π 2 s + 1 ∶ d } which satisfies π ( E 1 )
=
E 2 .
Similar proof holds also for
E ′ 1 ,E ′ 2 .
Then,
[Equation]
Let l ∈ { 1 ,...,s / 4 }
[Equation]
and let R a = Q 34 s − a ∩ Q ′ 14 s − a .
For a ∈ R d and r ∈ R
[Equation]
and ϕ ∈ { θ,θ ′ }
[Equation]
let p ϕ ( r ∣ a )
= 1 √ 2 π exp (− ( r −⟨ a,ϕ ⟩) 2) be the probability density function of the reward r when the action a is given under hypothesis ϕ .
Now pick one element E ∈ R l .
Then,
[Equation]
6 and 7)
[Equation]
Here
[Equation]
we will use the following Claim 1 to bound this empirical KL divergence to the fixed constant.
Claim 1.
[Equation]
where KL ( ρ,δ,τ ) = 12 ϵ 2 ( 1 + ρ )( 4 s 3 nd + 77 sκ 2 τ ) +
1 ρ ln 1 δ
Now, decide ρ and δ later and continuing from the previous inequality with KL ( ρ,δ,τ ) = 12 ϵ 2 (
[Equation]
6 and 7)
[Equation]
For the last inequality
[Equation]
we used the following lemma.
Lemma 8.
For d > ( s + 1 ) 2 , s > 5 , and l ∈ [ s 4 ] , we have ∣ Q ′ s {+
+} − l ∣ ∣ R l ∣ < s .
In short,
[Equation]
Let Y =
min
[Equation]
∣ .
Then,
[Equation]
Summing up both sides for l ∈ [ s 4 ] ,
[Equation]
Setting δ =
( 4 ξ s 2 Y ) ρρ + 1 , ρ = 3 and rearranging the last equation with sufficiently large s we get:
[Equation]
We are talking about the data-poor regime where d > κ − 4 / 3 s 4 / 3 n 1 / 3 .
So setting ϵ = κ − 2 / 3 s − 1 / 3 n 1 /and using the following Lemma 9 leads the conclusion that the order of E θ ′
[ T (H)] ≥ Ω ( 1 κ 2 ϵ 2 ) .
Lemma 9.
ln Y
= Ω ( s ) , and for sufficiently large s , ln s 2 Y 4 ξ ≤ 2ln Y 4 ξ
F.2.3 Proof of the Claim
This claim consists of two parts - prove the relationship between empirical KL and the true KL divergence, and the KL divergence calculation part.
We will prove the first inequality as follows.
Lemma 10.
For ρ > 0 , let
[Equation]
Then, P θ ′
( B θ,θ ′ ( ρ ))
≤ δ .
Proof.
Let J t ∶= ln ( p θ ′ ( r t ∣ a t ) p θ ( r t ∣ a t ) ) .
Now let H t = exp ( ρ (∑ ts = 1 J s − ( 1 + ρ ) E θ ′ ,s − 1 [ J s ∣ a s ]) with H 0 = 1 .
Now we will prove that for all ρ > 0 the { H t } Tt = 0 is a non-negative super-martingale.
[Equation]
Now the proof boils down to check the case when E θ ′ ,t −
[Equation] [Equation]
])] ≤
[Equation]
Fortunately, we can explicitly calculate J t .
Let µ t = ⟨ θ,a t ⟩ and µ ′ t = ⟨ θ ′ ,a t ⟩ .
Then
[Equation] [Equation]
where KL ( p θ ′ (⋅∣ a t ) ,p θ (⋅∣ a t ))
=
( µ t − µ ′ t ) 2.
Now by the 1-subgaussianity of r t − µ ′ t ,
E θ ′ ,t
[Equation]
The last inequality holds because of the 1 -subgaussian property of the η t .
Therefore, E θ ′ ,t − 1 [ H t ] ≤
H t − 1 , and therefore { H t } Tt = 0 is a supermartingale.
Finally, using Ville’s maximal inequality on H t we can achieve the desired probability inequality
[Equation]
By the above Lemma
[Equation]
one can induce the following relationship.
[Equation]
∣
[Equation] [Equation]
Now the main part of the proof is bounding (X).
[Equation]
Define Π = { π 1 ○ π 2 ∶ π 1 ∈ Sym ([ 1 ∶ s ]) ,π 2 ∈
Sym ([ s + 1 ∶ d ])} .
Importantly, for any π ∈ Π , π ( θ ′ )
= θ ′ .
We focus on the first term in the above expression:
[Equation]
Now for any realization of A , ˆ S , namely, a = ( a 1 ,...,a n ) ∈ A n ,u ⊂ [ d ] , we examine the quantity
[Equation]
Claim 2.
For any set u such that ∣ u ∣ = d 2 , and any a 1 ,...,a n ,
[Equation]
Proof.
If u ∉ Q ′ 14 s − l , then for any permutation σ ∈ Π
[Equation]
it must be the case that σ − 1 ( u ) ∉ Q ′ 14 s − l
[Equation]
and therefore, σ − 1 ( u ) ∉ R l .
In this case, both sides are equal to zero and the claim is trivially true
[Equation]
Otherwise, u ∈ Q ′
14 s − l .
Define Π legal,u = { σ ∈ Π ∶ σ − 1 ( u ) ∈ R l } .
Using this notation, the left hand side can be equivalently written as:
[Equation] [Equation]
We defer Lemma 12 and Lemma 11 to Section F.2.4.
Proof of Claim 1 basically ends after combining
Lemma 10 and the above claim.
However, for the clear description how this was used in the main proof, we now continue Equation (17), which, using the above claim, is at least
[Equation]
Therefore
[Equation]
in conclusion,
[Equation]
⎠
F.2.4 Deferred
proof
for Claim
Lemma 11.
For any u ∈ Q ′ 14 s − l ,
[Equation]
Proof.
It suffices to prove ∣ Π legal,u ∣∣ Q ′
14 s − l ∣ =
∣ R l ∣∣ Π ∣ .
To see this, first note that for any ˆ u ∈ Q 14 s − l , ∣ Π legal, ˆ u ∣ =
∣ Π legal,u ∣ .
Next, note,
[Equation]
Lemma 12.
Assume s ≤ √ d and d ≥ 16 .
For any a ∈ A n and u ∈ U( a ) with u ∈ Q ′ 14 s − l ,
[Equation]
Proof.
Let us first focus on t ∶ a t ∈
I where I is the reward arm set.
[Equation]
a th ∣ 1 ( σ − 1 ( j ) = h ) ≤ sϵ 2 2 s ∑ j = s +∑ t ∶ a t ∈I d
∑ h = s +∣ a th ∣ ⋅ ∣ Π legal ( l,u )∣ ∑ σ ∈ Π legal ( l,u ) 1 ( σ − 1 ( j ) = h )
Let us compute ∑ σ ∈ Π legal ( l,u ) 1 ( σ − 1 ( j ) = h ) .
WLOG, let us consider the following specific u : u = { 1 ,..., 34 s + l, 32 s + 1 ,..., 32 s + k } where k = d 2 − ( 34 s + l ) .
Define Seg 1 =
[ 1 ∶ s ] , Seg 2 =
[ s + 1 ∶ 2 s ] , and Seg
= [ 2 s + 1 ∶ d ] .
Note that u has exactly s − ( s / 4 − l )
= 34 s + l members in Seg 1 , so a legal permutation σ − 1 should choose s /coordinates from the remaining members of u (there are k of them) and choose s / 2 coordinates from { s + 1 ,..., 32 s } ∪ { 32 s + k + 1 ,...,d } then put those two into Seg 2 .
Therefore, we need to consider the following two cases:
• h ∈ Seg 2 : Permutations that take one correct coordinate and puts it into a correct coordinate (correct =
Seg 2 ).
Thus
we need to multiply
the following
two: – the segment 1: s ! – the segment 2: ( k − 1 s 2 − 1 )( d − s − k s 2 )( s − 1 ) !
( d − 2 s ) !
• h ∈ Seg 3 : take one correct coordinate and puts it into an incorrect coordinate.
– the segment 1: s ! – the segment 2: ( k
[Equation]
Note that
[Equation]
For the first case,
[Equation]
For the second case, we get
[Equation]
Plugging in the definition of k , we get that
[Equation]
For the second part,
[Equation]
⎞⎟⎠
[Equation]
[Table]
[Equation]
⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠
To compute ( Z ′ 2 ) , let us define the following where the first two are false positives w.r.t.
supp ( θ ′ ) and the last two are true negatives w.r.t.
supp ( θ ′ ) :
[Equation]
Let c + ,c − ,m + ,m − be ∣ C + ∣ , ∣ C − ∣ , ∣ M + ∣ , ∣ M − ∣ , respectively.
Note that c
+ + c −
=
d 2 − 34 s − l
=
k and m + + m − =
d 2 − 14 s + l = d − s − k .
Then, • When A tπ − 1 ( a ) A tπ − 1 ( b ) = κ 2 : – Common part COM =
( s − 2 ) !
( d − 2 s ) !
s !
– A tπ − 1 ( a ) ∈ C
+ ,
A tπ − 1 ( b ) ∈ C + : c + × ( c + − 1 ) × ( k − 2 s 2 − 2 )
( d − s − k s 2 ) COM
A tπ − 1 ( a ) ∈ M + ,A tπ − 1 ( b ) ∈ M
+ : m + × ( m + − 1 ) ×
( k s 2 )( d − s − k − 2 s 2 − 2 )
COM –
A tπ − 1 ( a ) ∈ C − ,A tπ − 1 ( b ) ∈ C − : c − × ( c − − 1 ) × ( k − 2 s 2 − 2 )( d − s
− k s 2 ) COM –
A tπ − 1 ( a ) ∈ M − ,A tπ − 1 ( b ) ∈ M − : m − × ( m − − 1 ) ×
( k s 2 )( d − s − k − 2 s 2 − 2 )
COM –
A tπ − 1 ( a ) ∈ C
+ ,
A tπ − 1 ( b ) ∈ M + and opposite: 2 × c + × m + × ( k − 1 s 2 − 1 )( d − s − k − 1 s 2 − 1 )
COM
– A tπ − 1 ( a ) ∈ C − ,A tπ − 1 ( b ) ∈ M − and opposite: 2 × c − × m − × ( k − 1 s 2 − 1 )( d − s − k − 1 s 2 − 1 )
COM • When A tπ − 1 ( a ) A tπ − 1 ( b )
= − κ 2 : – A tπ − 1 ( a ) ∈ C
+ ,A tπ − 1 ( b ) ∈ C − and opposite: 2 × c + × c − ×
( k − 2 s 2 − 2 )( d − s −
k s 2 ) COM –
A tπ − 1 ( a ) ∈ M + ,A
tπ − 1 ( b ) ∈ M − and opposite: 2 × m + × m − × ( k s 2 )( d − s − k − 2 s 2 − 2 )
COM –
A tπ − 1 ( a ) ∈ C
+ ,
A tπ − 1 ( b ) ∈ M − and opposite: 2 × c + × m − ×
k − 1 s 2 − 1 )( d − s − k − 1 s 2 − 1 ) COM –
A tπ − 1 ( a ) ∈ M + ,A tπ − 1 ( b ) ∈ C − and opposite: 2 × m + × c − ×
( k − 1 s 2 − 1 )( d − s − k − 1 s 2 − 1 )
COM
To evaluate ( Z ′ 2 ) , note that
[Equation]
Assuming d ≥ s 2 and s ≥
[Equation]
we have that k ∧ ( d − s − k ) ≥ d 2 √ 2 .
This also implies that s ≥ 2 , k ≥ 2 , and d − s − k ≥ 2 .
Note that
[Equation]
Then,
[Equation] [Equation]
[Equation]
For ( Z ′ 2 , 1 ) , let w =
( c + + m + )−( c − + m − ).
Note ∣ 2 w ∣ ≤ √ 2 d ln2 d + s .
Then,
• c + + c − =
k , m + + m − =
d − s − k • c + + m + = d − s 2 + w , c − + m − =
d − s 2 − w .
• c + = d − s 2 + w − m + and c − =
d − s 2 − w − m −
Thus, c + − c − = 2 w − ( m + − m − ) .
So, c + − c −
[Equation]
+ m
[Equation] [Equation]
Thus, using s ≥ 2 and d > ln d ,
[Equation] [Equation]
For ( Z ′ 2 , 2 ) , using k ∧ ( d − s − k ) ≥ d 4 ,
[Equation]
=
[Equation]
Thus, ( Z ′ 2 ) ≤ 76 κ 2 d .
Altogether,
[Equation]
where the last inequality is by ( s − 1 ) <
d .
F
2.5 Proof of Lemma
Proof.
We will use the following Stirling’s approximation for calculating the scale of Y .
Lemma 13.
(Stirling’s approximation [19])
For x ≥ 1 ,
[Equation]
To be more specific, there exists an absolute constant s 0 = 10 such that for all x > s 0 ,
[Equation]
From this lemma, we can derive the following estimation of the combination.
For x,y > s 0 ,
[Equation]
Now by definition, ∣ Q a ∣ =
( d − 2 s d 2 − 2 s + a )( 2 sa ) , and ∣ Q ′
b ∣ =
( d − s d 2 − s + b )( sb ) .
For sufficiently large s > 4 s 0 and d > s 2 we have the following approximation
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
l
[Table]
⎛ ⎝
[Table]
+
[Table]
+ l
[Table]
⎞
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
From the Taylor’s expansion, one can achieve the following inequality.
Lemma 14.
For all y > 2 and x > y 2 , (
[Equation]
− yx
[Equation]
x
[Equation]
Proof.
[Equation]
Similarly,
[Equation]
Therefore,
[Equation]
and thus Y =
min l ∈[ s / 4 ] ∣ Q 34 s − l ∣ ∣ Q ′ s 4 − l
[Equation]
≥
[Equation]
min l ∈[ s / 4 ] ( 2 s 34 s − l
[Equation]
( s 14 s − l ) where
C = e − 2 ( 11 . 01 ) 3 14 is a universal constant.
From the formula ( nk + 1 ) =
( nk ) × n − k k + 1 , one can check that min l ∈[ s 4 ] (
[Equation]
where the last one is from Lemma 13 again and C ′ =
[Equation]
01 3 × √ 25 is another universal constant.
In short,
[Equation]
where the last inequality is from 2 × ( 85 ) 5 = 20 . 97 ⋯ > 1 . 3 × 2 4 Therefore, we can conclude that ln Y = Ω ( s ) .
F.2.6 Proof of Lemma
[Equation]
[Equation]
From the combinatorial formula, ( d − s d 2 − 14 s + l ) =
∑ sa = {+
+}
( d − 2 s d 2 − 14 s + l − a )( sa ) .
Now we will analyze the size of f ( a ) ∶= ( d − 2 s d 2 − 14 s + l − a )( sa ) for each a .
First, note that for any x ∈ N , ( 2 xy ) is increasing for y ∈ { 0 , 1 , ⋯ ,x } , and decreasing for y ∈ { x,x + 1 , ⋯ , 2 x } .
Therefore, ( d − 2 s d 2 − 14 s + l − a ) is an increasing function in a ≤ 34 s − l and ( sa ) is also an increasing function in a ≤ s 2 .
Therefore, f ( a ) is increasing function for a ∈ { 0 , ⋯ ,s / 2 } .
By the similar logic, f ( a ) is a decreasing function for a ∈ { 34 s − l,s } .
Now the region we have to take care is s 2 ≤ a ≤ 34 s − l .
In this case, (
[Equation] [Equation]
( l ≤
[Equation]
s and a ≥ s 2 )
[Equation]
− [Equation] −
[Equation]
s +≤ 1 + s −
[Equation]
{+− s = s +s ( d > s 
+} + 2 s − 2 ) and ( s a + 1 )/
( s a ) ≤ 1 − s
+ 1 = s s
+therefore f ( a + 1 ) f
( a ) ≤ s +s s s + 2 =which means f ( a ) is a nonincreasing function with respect to a ∈ { s 2 , ⋯ ,s } , and therefore max a ∈[ s ] ( d − 2 s d 2 − 14 s + l − a )( sa ) =
( d − 2 s d 2 − 34 s + l )( s s 2 ) .
Plus, by the similar calculation from above and using the condition s > 5 , we can show that f ( 0 ) <
1 s f ( 1 ) <
1 s f ( s 2 ) and f ( s ) <
1 s f ( s − 1 ) < 1 s f ( s 2 ) which means f ( s 2 ) >
f ( 0 ) + f ( s ) .
Therefore, ∣ Q ′ 14 s − l ∣ =
f ( 0 )
+ f ( s )
+ ∑ s − 1 a = 1 f ( a ) ≤ sf ( s 2 ) =
s ∣ R l ∣
F .2.7 Scale of κ 2 with respect to C min
Last thing we have to deal with is connecting κ 2 to C min and H 2 ∗ .
Lemma 15.
C min (H) ≥ κ
Proof.
By the maximality of C min (H) , C min (H) ≥ λ min ( Q ( Unif (H))) .
If we prove that λ min ( Q ( Unif (H))) ≥ κ 2 2 , then the proof is done.
Let’s define H d as
[Equation]
It is the set of first d coordinate vectors of H .
We can express Q ( Unif (H)) in terms of Q ( Unif (H d )) as follows
[Equation]
Therefore, the proof boils down to calculate Q ( Unif (H d )) .
We can connect this matrix to
Q ( Unif ({− κ,κ } d )) by the following method
[Equation]
Now
[Equation]
since Rademacher is 1 sub-Gaussian random variable
[Equation]
∑ ni = 1 a
i is κ √ d sub-Gaussian random variable when a ∼ Unif ({− κ,κ }) .
Therefore,
[Equation]
The last inequality is by the traditional Chernoff method [14].
Therefore
[Equation]
we can rewrite
Q ( Unif (H d )) as
[Equation]
Therefore
[Equation]
(Existing results about this positive definite matrix analysis)
[Equation]
Since every element in {− κ,κ } has ℓ 2 -norm √ dκ ,
[Equation]
and by simple symmetry one can calculate λ min ( Q ( Unif ({− κ,κ } d )))
= κ 2 .
Therefore,
[Equation]
G Experiment details
• Case 1 - ℓ 1 estimation error experiment – θ = − e
+ e i , i ∈ { 2 , ⋯ ,d } chosen uniformly random before the start of the experiment.
– Dimension d = 10 , sparsity s =– Action set A = { e 1 + 1 √ d e i ∣ i = 2 , ⋯ ,d } ∪ { 1 √ d e 1 } – T = 1000 , 2000 , ⋯ ,– σ = 0 .
– Repetition: 30 times for each exploration time.
• Case 1 - bandit experiment – θ = e
+ e i , i ∈ { 2 , ⋯ ,d } chosen uniformly random before the start of the experiment.
– Dimension d = 10 , sparsity s =
– Action set A = { e 1 + 1 √ d e i ∣ i = 2 , ⋯ ,d } ∪ { 1 √ d e 1 } – T =– σ = 0 .
– Repetition: 30 times • Case 2 - ℓ 1 estimation error experiment – θ = e i + e j , i,j ∈ [ d ] chosen uniformly random before the start of the experiment.
– Dimension d = 30 , sparsity s =– Action set A : 90 Uniform random vectors over S d − 1 before the start of the round, where S d − 1 = { v ∈ R d ∣∥ v ∥ 2 = 1 } – T = 1000 , 2000 , ⋯ ,– σ = 0 .
– Repetition: 30 times for each exploration time.
• Case 2 - bandit experiment – θ = e i + e j , i,j ∈ [ d ] chosen uniformly random before the start of the experiment.
– Dimension d = 30 , sparsity s =– Action set A : 90 Uniform random vectors over S d − 1 before the start of the round.
– T =– σ = 0 .
– Repetition: 30 times for each exploration time.