I MPACT OF R EPRESENTATION L EARNING IN L INEAR B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study how representation learning can improve the efﬁciency of bandit problems.
We study the setting where we play T linear bandits with dimension d concurrently, and these T bandit tasks share a common k ( (cid:28) d ) dimensional linear representation.
For the ﬁnite-action setting, we present a new algorithm which achieves (cid:101)
O ( T √ kN + √ dkNT ) regret, where N is the number of rounds we play for each bandit.
When T is sufﬁciently large, our algorithm signiﬁcantly outperforms the naive algorithm (playing T bandits independently) that achieves (cid:101)
O ( T √ dN ) regret.
We also provide an Ω( T √ kN + √ dkNT ) regret lower bound, showing that our algorithm is minimax-optimal up to poly-logarithmic factors.
Furthermore, we extend our algorithm to the inﬁnite-action setting and obtain a corresponding regret bound which demonstrates the beneﬁt of representation learning in certain regimes.
We also present experiments on synthetic and realworld data to illustrate our theoretical ﬁndings and demonstrate the effectiveness of our proposed algorithms.
I NTRODUCTION
This paper investigates the beneﬁt of using representation learning for sequential decision-making problems.
Representation learning learns a joint low-dimensional embedding (feature extractor) from different but related tasks and then uses a simple function (often a linear one) on top of the embedding (Baxter, 2000; Caruana, 1997; Li et al., 2010)
The mechanism behind is that since the tasks are related, we can extract the common information more efﬁciently than treating each task independently.
Empirically, representation learning has become a popular approach for improving sample efﬁciency across various machine learning tasks (Bengio et al., 2013).
In particular, recently, representation learning has become increasingly more popular in sequential decision-making problems (Teh et al., 2017; Taylor & Stone, 2009; Lazaric & Restelli, 2011; Rusu et al., 2015; Liu et al., 2016; Parisotto et al., 2015; Higgins et al., 2017; Hessel et al., 2019; Arora et al., 2020; D’Eramo et al., 2020).
For example, many sequential decision-making tasks share the same environment but have different reward functions.
Thus a natural approach is to learn a succinct representation that describes the environment and then make decisions for different tasks on top of the learned representation.
While representation learning is already widely applied in sequential decision-making problems empirically, its theoretical foundation is still limited.
One important problem remains open:
When does representation learning provably improve efﬁciency of sequential decision-making problems?
We take a step to characterize the beneﬁt of representation learning in sequential decision-making problems.
We tackle the above problem in the linear bandits setting, one of the most fundamental and popular settings in sequential decision-making problems.
This model is widely used in applications as such clinical treatment, manufacturing process, job scheduling, recommendation systems, etc (Dani et al., 2008; Chu et al., 2011).
We study the multi-task version of linear bandits, which naturally models the scenario where one needs to deal with multiple different but closely related sequential decision-making problems concurrently.
We will mostly focus on the ﬁnite-action setting.
Speciﬁcally, we have T tasks, each of which is governed by an unknown linear coefﬁcient θ t ∈ R d .
At the n -th round, for each task t ∈ [ T ] , the player chooses an action a n,t that belongs to a ﬁnite set, and receive a reward r n,t with expectation E r n,t = (cid:104) θ t , x n,t,a n,t (cid:105) where x n,t,a n,t represents the context of action a n,t .
For this problem, a straightforward approach is to treat each task independently, which leads to (cid:101)
O
( T √ dN ) 1 total regret.
Can we do better?
Clearly, if the tasks are independent, then by the classical Ω( √ dN ) per task lower bound for linear bandit, it is impossible to do better.
We investigate how representation learning can help if the tasks are related.
Our main assumption is the existence of an unknown linear feature extractor B ∈ R
d
× k with k (cid:28) d and a set of linear coefﬁcients { w t } Tt =1 such that θ t =
Bw t .
Under this assumption, the tasks are closely related as B is a shared linear feature extractor that maps the raw contexts x n,t,a ∈ R d to a low-dimensional embedding B (cid:62) x n,t,a ∈ R k .
In this paper, we focus on the regime where k (cid:28) d, N, T .
This regime is common in real-world problems, e.g., computer vision, where the input dimension is high, the number of data is large, many task are related, and there exists a low-dimension representation among these tasks that we can utilize.
Problems with similar assumptions have been studied in the supervised learning setting (Ando & Zhang, 2005).
However, to our knowledge, this formulation has not been studied in the bandit setting.
Our Contributions
We give the ﬁrst rigorous characterization on the beneﬁt of representation learning for multi-task linear bandits.
Our contributions are summarized below.
• We design a new algorithm for the aforementioned problem.
Theoretically, we show our algorithm incurs (cid:101) O ( √ dkTN + T √ kN ) total regret in N rounds for all T tasks.
Therefore, our algorithm outperforms the naive approach with O ( T √ dN ) regret.
To our knowledge, this is the ﬁrst theoretical result demonstrating the beneﬁt of representation learning for bandits problems.
• To complement our upper bound, we also provide an Ω( √ dkTN + T √ kN ) lower bound, showing our regret bound is tight up to polylogarithmic factors.
• We further design a new algorithm for the inﬁnite-action setting, which has a regret (cid:101) O
( d 1 . 5 k √ TN + kT √ N ) , which outperforms the naive approach with O ( Td √ N ) regret in the regime where T = (cid:101) Ω( dk 2 ) .
• We provide simulations and an experiment on MNIST dataset to illustrate the effectiveness of our algorithms and the beneﬁts of representation learning.
Organization This paper is organized as follows.
In Section 2, we discuss related work.
In Section 3, we introduce necessary notation, formally set up our problem, and describe our assumptions.
In Section 4, we present our main algorithm for the ﬁnite-action setting and its performance guarantee.
In Section 5, we describe our algorithm and its theoretical guarantee for the inﬁnite-action setting.
In Section 6, we provide simulation studies and real-world experiments to validate the effectiveness of our approach.
We conclude in Section 7 and defer all proofs to the Appendix.
R ELATED W ORK
Here we mainly focus on related theoretical results.
We refer readers to Bengio et al.
for empirical results of using representation learning.
For supervised learning, there is a long line of works on multi-task learning and representation learning with various assumptions (Baxter, 2000; Ando & Zhang, 2005; Ben-David & Schuller, 2003;
Maurer, 2006; Cavallanti et al., 2010; Maurer et al., 2016; Du et al., 2020; Tripuraneni et al., 2020).
All these results assumed the existence of a common representation shared among all tasks.
However, this assumption alone is not sufﬁcient.
For example, Maurer et al.
(2016) further assumed every task is i.i.d.
drawn from an underlying distribution.
Recently, Du et al.
(2020) replaced the i.i.d.
assumption with a deterministic assumption on the input distribution.
Finally, it is worth mentioning that Tripuraneni et al.
(2020) gave the method-of-moments estimator and built the conﬁdence ball for the feature extractor, which inspired our algorithm for the inﬁnite-action setting.
The beneﬁt of representation learning has been studied in sequential decision-making problems, especially in reinforcement learning domains.
D’Eramo et al.
(2020) showed that representation learning can improve the rate of approximate value iteration algorithm.
Arora et al.
(2020) proved that representation learning can reduce the sample complexity of imitation learning.
Both works require a probabilistic assumption similar to that in (Maurer et al., 2016) and the statistical rates are of similar forms as those in (Maurer et al., 2016).
We remark that representation learning is also closely connected to meta-learning (Schaul &
Schmidhuber, 2010).
Raghu et al.
(2019) empirically suggested that the effectiveness of metalearning is due to its ability to learn a useful representation.
There is a line of works that analyzed the theoretical properties of meta-learning (Denevi et al., 2019;
Finn et al., 2019; Khodak et al., 2019; Lee et al., 2019; Bertinetto et al., 2018).
We also note that there are analyses for other representation learning schemes (Arora et al., 2019; McNamara & Balcan, 2017; Galanti et al., 2016;
Alquier et al., 2016; Denevi et al., 2018).
Linear bandits (stochastic linear bandits / linearly parameterized bandits / contextual linear bandits) have been studied in recent years (Auer, 2002; Dani et al., 2008; Rusmevichientong & Tsitsiklis, 2010; Abbasi-Yadkori et al., 2011;
Chu et al., 2011; Li et al., 2019a;b).
The studies are divided into two branches according to whether the action set is ﬁnite or inﬁnite.
For the ﬁnite-action setting, (cid:101) Θ( √ dN ) has been shown to be the near-optimal regret bound (Chu et al., 2011; Li et al., 2019a), and for the inﬁnite-action setting
, (cid:101) Θ( d √ N ) regret bound has been shown to be near-optimal (Dani et al., 2008; Rusmevichientong & Tsitsiklis, 2010; Li et al., 2019b). Some previous work studied the impact of low-rank structure in linear bandit. Lale et al. (2019) studied a setting where the context vectors share a low-rank structure.
Speciﬁcally, in their setting, the context vectors consist of two parts, i.e. ˆ x = x + ψ , so that x is from a hidden low-rank subspace and ψ is i.i.d.
drawn from an isotropic distribution.
Jun et al.
(2019) and Lu et al. (2020) studied the bilinear bandits with low-rank structure.
In their setting, the player chooses two actions x , y and receives the stochastic reward with mean x (cid:62) Θ y , where Θ is an unknown low-rank bilinear form.
The algorithms proposed in the aforementioned papers share some similarities with our Algorithmfor our inﬁnite-action setting, in that both used Davis-Kahan theorem to recover and exploit the low-rank structure.
Some previous work proposed multi-task bandits with different settings.
Deshmukh et al.
(2017) proposed a setting under the contextual bandit framework.
They assumed similarities among arms.
Bastani et al.
(2019) studied a setting where the coefﬁcients of the tasks were drawn from a gaussian distribution ﬁxed across tasks and proposed an algorithm based on Thompson sampling.
Soare et al.
(2018) proposed a setting where tasks were played one by one sequentially and the coefﬁcients of the tasks were near in (cid:96) 2 distance.
In our setting, the tasks are played simultaneously and the coefﬁcients share a common linear feature extractor.
P RELIMINARIES
Notation.
We use bold lowercases for vectors and bold uppercases for matrices.
For any positive integer n , we use [ n ] to denote the set of integers { 1 , 2 , . . . , n } .
For any vector x , we use (cid:107) x
(cid:107) to denote its (cid:96) 2 norm.
For a matrix A , we use (cid:107) A (cid:107) to denote the 2 -norm of A , (cid:107) A (cid:107) F to denote the Frobenius norm, and (cid:107) A (cid:107) max = max i,j |
A ij | to denote the max-norm.
For two expressions α, β > 0 , we denote α (cid:46) β if there is a numerical constant c > 0 such that α ≤ cβ .
We denote α (cid:38) β if β (cid:46) α .
Problem Setup.
Let d be the ambient dimension and k ( ≤ d ) be the representation dimension.
In total, we have T tasks and we play each task concurrently for N rounds.
Each task t ∈ [ T ] has an unknown vector θ t ∈ R d .
At each round n ∈ [ N ] , the player chooses action a n,t ∈
A n,t for each task t ∈
[ T ] where A n,t is the action set at round n for the task t .
After the player commits to a batch of actions { a n,t } t ∈ [ T ] , it receives a batch of rewards { r n,t } t ∈
[ T ] , where we assume r n,t = (cid:104) a n,t , θ t (cid:105) + ε n,t .
Here we assume the noise ε n,t are independent 1 -subGaussian random variables, which is a standard assumption in the literature.
We use the total expected regret to measure the performance of our algorithm.
When we have N rounds and T tasks, it is deﬁned as R N,T = (cid:80)
Nn =1 (cid:80)
Tt =1 max a ∈A n,t (cid:104) a , θ t (cid:105) − (cid:104) a n,t , θ t (cid:105) .
When the
1: Let M = (cid:100) log 2 log 2 N (cid:101) , G 0 = 0 , G M = N, G m = N 1 − 2 − m for 1 ≤ m ≤ M − 1 , let (cid:98) θ 0 ,t ← 0 ; 2: for m ← 1 , . . .
, M do 3: for n ← G m − 1 + 1 , . . .
, G m do 4:
For each task t ∈ [ T ] : choose action a n,t = arg max a ∈ [ K ] x (cid:62) n,t,a (cid:98) θ m − 1 ,t ; 5:
Compute (cid:98) B (cid:99) W ← arg min B ∈ R
d × k , W ∈ R k × T (cid:80) G m n =
G m − 1 +1 (cid:80) Tt =
[ x (cid:62) n,t,a n,t Bw t − r n,t ] 2 ; 6:
For each task t ∈ [ T ] : let (cid:98) θ m,t = (cid:98) B (cid:98) w t ; action set is ﬁnite, we assume that all A n,t have the same size K , i.e. |A n,t | ≡ K . Furthermore, we write A n,t = { x n,t, 1 , . . . , x n,t,K } . Besides, we interchangeably use the number a n,t ∈
[ K ] and the vector a n,t = x n,t,a n,t ∈ R d to refer to the same action.
Assumptions.
Our main assumption is the existence of a common linear feature extractor. Assumption 1 (Common Feature Extractor) . There exists a linear feature extractor B ∈ R d × k and a set of linear coefﬁcients { w t } T t =1 such that the expected reward of the t -th task at the n -th round satisﬁes E [ r t,n ] = (cid:104) w t , B (cid:62) x n,t,a n,t (cid:105) . For simplicity, we let W =
[ w 1 , . . . , w T ] .
Assumption 1 implies that Θ (cid:44) [ θ 1 , . .
, θ T ] =
BW .
Note this assumption is in a sense necessary to guarantee the effectiveness of representation learning because without it one cannot hope that representation learning helps.
In this paper, we mostly focus on the ﬁnite-action setting.
We put the following assumption on action sets.
Assumption 2.
Marginally, for every n ∈ [ N ] , t ∈ [ T ] , a ∈ [ K ] , the contexts satisfy x n,t,a ∼
N ( 0 , Σ t ) such that λ max (Σ t ) ≤ O (1 /d ) and λ min (Σ t ) ≥ Ω(1 /d ) .
With this assumption, we have an unknown covariance matrix Σ t for each task.
At each round, the actions of the t -th task are sampled from a Gaussian distribution with covariance Σ t .
This is a prototypical setting for theoretical development on linear bandits with ﬁnite actions (See e.g., Han et al.
At a population level, each one of the K actions is equally good but being able to select different actions based on the realized contexts allows the player to gain more reward.
We will also study the inﬁnite-action setting.
We ﬁrst state our assumption about the action sets.
Assumption 3 (Ellipsoid Action Set) .
We assume A n,t =
A t = { x (cid:62) Q − 1 t x ≤ 1 : x ∈ R d } is an ellipsoid with λ min ( Q t ) ≥ λ 0 = Ω(1) .
The ﬁrst assumption states that each action set is an ellipsoid that covers all directions.
This is a standard assumption, e.g., see Rusmevichientong & Tsitsiklis (2010).
In this setting, we will also need to put some additional assumptions on the underlying parameters
B and W .
Assumption 4 (Diverse Source Tasks) .
We assume that λ min ( 1 T W W (cid:62) ) ≥ νk , where ν = Ω(1) .
This assumption roughly states that the underlying linear coefﬁcients { w t } Tt =1 equally spans all directions in R k .
This is a common assumption in representation learning literature that enables us to learn the linear feature extractor (Du et al., 2020; Tripuraneni et al., 2020).
For example, the assumption holds with high probability when w i is uniformly chosen from the sphere S k − 1 .
Assumption 5.
We assume (cid:107) w t (cid:107) ≥ ω
= Ω(1) .
This is a normalization assumption on the linear coefﬁcients.
M AIN R ESULTS FOR F INITE -A CTION S ETTING
In this section focus on the ﬁnite-action setting.
The pseudo-code is listed in Algorithm 1.
Our algorithm uses a doubling schedule rule (Gao et al., 2019; Simchi-Levi & Xu, 2019; Han et al., 2020;
Ruan et al., 2020).
We only update our estimation of θ after an epoch is ﬁnished, and we only use samples collected within the epoch.
In Line 5, we solve an empirical (cid:96) 2 -risk minimization problem on the data collected in the last epoch to estimate the feature extractor B and linear predictors
W , similar to Du et al.
Given estimated feature extractor (cid:98) B and linear predictors (cid:99) W , we compute our estimated linear coefﬁcients of task t as (cid:98) θ t (cid:44) (cid:98) B (cid:98) w t in Line 6.
For choosing actions, for each task, we use a greedy rule, i.e., we choose the action that maximizes the inner product with our estimated θ (cf.
Line 4).
The following theorem gives an upper bound on the regret of Algorithm 1.
Theorem 1 (Regret of Algorithm 1) .
Suppose K, T ≤ poly( d ) and N ≥ d 2 .
Under Assumptionand Assumption 2, the expected regret of Algorithm 1 is upper bounded by
[Equation]
There are two terms in Theorem 1, and we interpret them separately.
The ﬁrst term (cid:101) O ( T √ kN ) represents the regret for playing T independent linear bandits with dimension k for N rounds.
This is the regret we need to pay even if we know the optimal feature extractor, with which we can reduce the original problem to playing T independent linear bandits with dimension k (recall w t are different for different tasks).
The second term (cid:101) O ( √ dkNT ) represents the price we need to pay to learn the feature extractor B .
Notably, this term shows we are using data across all tasks to learn B as this term scales with √ NT .
Now comparing with the naive strategy that plays T independent d -dimensional linear bandits with regret (cid:101)
O ( T √ dN ) , our upper bound is smaller as long as T
= Ω( k ) .
Furthermore, when T is large, our bound is signiﬁcantly stronger than (cid:101)
O
( T √ dN ) , especially when k (cid:28) d .
To our knowledge, this is the ﬁrst formal theoretical result showing the advantage of representation learning for bandit problems.
We remark that requiring T = Ω( k ) is necessary.
One needs at least k tasks to recover the span of W , so only in this regime representation learning can help.
Our result also puts a technical requirement on the scaling K, T ≤ poly( d ) and N ≥ d 2 .
These are conditions that are often required in linear bandits literature.
The ﬁrst condition ensures that K and
T are not too large, so we need not characterize log( KT ) factors in regret bound.
If they are too large, e.g. K, T ≥ Ω( e d ) , then we would have log KT =
O ( d ) and we could no longer omit the logarithmic factors in regret bounds.
The second condition ensures one can at least learn the linear coefﬁcients up to a constant error.
See more discussions in (Han et al., 2020, Section 2.5).
While Algorithm 1 is a straightforward algorithm, the proof of Theorem 1 requires a combination of representation learning and linear bandit techniques.
First, we prove the in-sample guarantee of representation learning, as done in Lemma 2.
Second, we exploit Assumption 2 to show that the learned parameters could extrapolate well on new contexts, as shown in Lemma 4.
The regret analysis then follows naturally.
We defer the proof of Theorem 1 to Appendix A.
The following theorem shows that Algorithm 1 is minimax optimal up to logarithmic factors.
Theorem 2 (Lower Bound for Finite-Action Setting) .
Let A denote an algorithm
and I denote a ﬁnite-actioned multi-task linear bandit instance that satisﬁes Assumption 1 and Assumption 2.
Then for any N, T, d, k ∈ Z + with k ≤ d , k ≤ T , we have
[Equation]
Theorem 2 has the exactly same two terms as in Theorem 1.
This conﬁrms our intuition that the two prices to pay are real: 1) playing T independent k -dimensional linear bandits and 2) learning the d × k -dimensional feature extractor.
We defer the proof of Theorem 2 to Appendix B. At a high level, we separately prove the two terms in the lower bound.
The ﬁrst term is established by the straightforward observation that our multi-task linear bandit problem is at least as hard as solving
T independent k -dimensional linear bandits.
The second term is established by the observation that multi-task linear bandit can be seen as solving k independent d -dimensional linear bandits, each has NT/k rounds.
Note that the observation would directly imply the regret lower bound k (cid:112) d ( NT/k ) = √ dkNT , which is exactly the second term.
To our knowledge, this lower bound is also the ﬁrst one of its kind for multi-task sequential decision-making problems.
We believe our proof framework can be used in proving lower bounds for other related problems.
4: Compute
←
N N
=
Tt =1 r 2 n t x n t x n t ;
: Let do 8:
Play
9: for ∀ do 10:
Estimate
← min
N x
Let
; 13: for do 14:
Play r n,t ;
E XTENSION TO I NFINITE -A CTION S ETTING
In this section we present and analyze an algorithm for the inﬁnite-action setting.
Pseudo-code is listed in Algorithm 2.
Our algorithm has three stages, and we explain each step below.
Stage 1: Estimating Linear Feature Extractor with Method-of-Moments. The goal of the ﬁrst stage is to estimate the linear feature extractor B . Our main idea to view this problem as a lowrank estimation problem for which we use a method-of-moments estimator. In more detail, we ﬁrst sample each x n,t ∼ Unif[ λ 0 · S d ] for N 1 times.
We can use this sampling scheme because the action set is an ellipsoid.
Note this sampling scheme has a sufﬁcient coverage on all directions, which help us estimate B .
Next, we compute the empirical weighted covariance matrix (cid:99) M =
N 1 T (cid:80) N 1 n =1 (cid:80) Tt =1 r 2 n,t x n,t x (cid:62) n,t .
To proceed, we compute the singular value decomposition of (cid:99) M and keep its top- k column space as our estimated linear feature extractor (cid:98) B , which is a sufﬁciently accurate estimator (cf.
Theorem 5).
Stage 2: Estimating Optimal Actions on Low-Dimensional Space.
In the second stage, we use our estimated linear feature extractor to reﬁne our search space for the optimal actions.
Speciﬁcally, we denote (cid:98) B
= [ (cid:98) b 1 , . . .
, (cid:98) b k ] and for i ∈
[ k ] and t ∈ [ T ] , we let v t,i = √ λ 0 (cid:98) b i .
Under Assumption 3, we know v t,i ∈
A t for all i ∈ [ k ] .
Therefore, we can choose v t,i to explore.
Technically, our choice of actions v t,i also guarantees a sufﬁcient cover in the sense that λ min (cid:16)(cid:80) ki =1 (cid:98) B (cid:62) v t,i v (cid:62) t,i (cid:98) B (cid:17) ≥ λ 0 .
In particular, this coverage is on a low-dimensional space instead of the original ambient space.
The second stage has N 2 rounds and on the t -th task, we just play each v t,i for N 2 /k rounds.
After that, we use linear regression to estimate w t for each task.
Given the estimation (cid:98) w t , we can obtain an estimation to the true linear coefﬁcient (cid:98) θ t (cid:44) (cid:98) B (cid:98) w t .
Stage 3: Committing to Near-Optimal Actions.
After the second stage, we have an estimation (cid:98) θ t for each task.
For the remaining ( N − N 1 − N 2 ) rounds, we just commit to the optimal action indicated by our estimations.
Speciﬁcally, we play the action a n,t ← arg max a ∈A t (cid:104) a t , (cid:98) θ t (cid:105) for round n = N 1 + N 2 + 1 , . . .
, N .
The following theorem characterizes the regret of our algorithm.
Theorem 3 (Regret of Algorithm 2) .
If we choose N 1 =
c 1 d 1 .
5 k (cid:113) NT and N
= c 2 k √ N for some constants
c 1 , c 2 > 0 .
The regret of Algorithm 2 is upper bounded by
[Equation]
[Figure]
e g r e t p e r
T a s k k =     k
=
MLinGreedy
Naive     k =
[Figure]
[Figure] [Figure] [Figure] The ﬁrst term represents the regret incurred by estimating the linear feature extractor.
Notably, the term scales with √ TN , which means we utilize all TN data here.
The ﬁrst term scales with d 1 .
5 k , which we conjecture is unavoidable at least by our algorithm.
The second term represents playing
T independent k -dimensional inﬁnite-action linear bandits.
Notice that if one uses a standard algorithm, e.g. the PEGE algorithm (Rusmevichientong & Tsitsiklis, 2010), to play T tasks independently, one can achieve an (cid:101) O ( dT √ N ) regret.
Comparing with this bound, our bound’s second term is always smaller and the ﬁrst is smaller when T = (cid:101) Ω( dk 2 ) .
This demonstrates that more tasks indeed help us learn the representation and reduce the regret.
We complement our upper bound with a lower bound below.
This theorem suggests our second term is tight but there is still a gap in the ﬁrst term.
We leave it as an open problem to design new algorithm to match the lower bound or to prove a stronger lower bound.
Theorem 4 (Lower Bound for Inﬁnite-Action Setting) .
Let A denote an algorithm
and I denote an inﬁnite-action multi-task linear bandit instance that satisﬁes Assumption 1, Assumption 3, Assumption 4, Assumption 5.
Then for any N, T, d, k ∈ Z + with k ≤ d , k ≤ T , we have
[Equation]
E XPERIMENTS
In this section, we use synthetic data and MNIST data to illustrate our theoretical ﬁndings and demonstrate the effectiveness of our Algorithm for the ﬁnite-action setting.
We also have simulation studies for the inﬁnite-action setting, which we defer to Appendix F.
The baseline is the naive algorithm which plays T tasks independently, and for each task, thie algorithm uses linear regression to estimate θ t and choose the action greedily according to the estimated θ t . {+× 
 R e g r e t × 10k =      × 10k = MLinGreedy+}
Naive      × 10k =
[Figure]
{+× 
+}
R e g r e t × 10k =      × 10k =
MLinGreedy
Naive      × 10k =
N : Time Horizon
6. S YNTHETIC D ATA
Setup.
The linear feature extractor B is uniformly drawn from the set of d × k matrices with orthonormal columns.
Each linear coefﬁcient w t is uniformly chosen from the k -dimensional sphere.
The noises are i.i.d.
Gaussian: ε n,t,a = N (0 , 1) for every n ∈ [ N ] , t ∈ [ T ] , a ∈ [ K ] .
We ﬁx
K = 5 and N = 10000 for all simulations on ﬁnite-action setting.
We vary k , d and T to compare
Algorithm 1 and the naive algorithm.
Results and Discussions.
We present the simulation results in Figure 1 and Figure 2.
We emphasize that the y -axis in our ﬁgures corresponds to the regret per task, which is deﬁned as R N,T /T .
We ﬁx
K = 5 , N = 10000 .
These simulations verify our theoretical ﬁndings.
First, as the number of tasks [-× 
 R e g r e t × 10k =      × 10k = MLinGreedy-]increases, the advantage of our algorithm increases compared to the naive algorithm.
[-Naive      × 
-]
[-× 
-]
Secondly, we notice that as k becomes larger (relative to d ), the advantage of our algorithm becomes smaller.
This can be explained by our theorem that as k increases, our algorithm pays more regret, whereas the naive algorithm’s regret does not depend on k .
6. F INITE -A CTION L INEAR B ANDITS FOR MNIST
Setup.
We create a linear bandits problem on MNIST data (LeCun et al., 2010) to illustrate the effectiveness of our algorithm on real-world data.
We ﬁx K = 2 and create T = (cid:0) 102 (cid:1) tasks and each task is parameterized by a pair ( i, j ) , where 0 ≤ i < j ≤ 9 .
We use D i to denote the set of MNIST images with digit i .
At each round n ∈ [ N ] , for each task ( i, j ) , we randomly choose one picture from D i and one from D j , then we present those two pictures to the algorithm and assign the picture with larger digit with reward 1 and the other with reward 0 .
The algorithm is now required to select an image (action).
We again compare our algorithm with the naive algorithm.
Results and Discussions.
The experimental results are displayed in Figure 3 for T = 10 (done by constructing tasks with ﬁrst ﬁve digits) and Figure 4 for T = 45 .
We observe for both T = 10 and T = 45 , our algorithm signiﬁcantly outperforms the naive algorithm for all k .
Interestingly, unlike our simulations, we ﬁnd the advantage of our algorithm does not decrease as we increase k .
We believe the reason is the optimal predictor is not exactly linear, and we need to develop an agnostic theory to explain this phenomenon, which we leave as a future work.
C ONCLUSION
We initiate the study on the beneﬁts of representation learning in bandits.
We proposed new algorithms and demonstrated that in the multi-task linear bandits, if all tasks share a common linear feature extractor, then representation learning provably reduces the regret.
We demonstrated empirical results to corroborate our theory.
An interesting future direction is to generalize our results to general reward function classes (Li et al., 2017; Agrawal et al., 2019).
In the following, we discuss some future directions.
Adversarial Contexts For the ﬁnite-action setting, we assumed the context are i.i.d.
sampled from a Gaussian distribution.
In the bandit literature, there is a large body on developing lowregret algorithms for the adversarial contexts setting. We leave it as an open problem to develop a algorithm with an (cid:101) O ( T √ kN + √ dkNT ) upper bound or show this bound is not possible in the adversarial contexts setting. One central challenge for the upper bound is that existing analyses for multi-task representation learning requires i.i.d.
inputs even in the supervised learning setting.
Another challenge is how to develop a conﬁdence interval for an unseen input in the multi-task linear bandits setting.
This conﬁdence interval should utilize the common feature extractor and is tighter than the standard conﬁdence interval for linear bandits, e.g. LinUCB.
Robust Algorithm
The current approach is tailored to the assumption that there exists a common feature extractor.
One interesting direction is to develop a robust algorithm.
For example, consider the scenario where whether there exists a common feature extractor is unknown.
We want to develop an algorithm with regret bound as in this paper when the common feature extractor exists and gracefully degrades to the regret of T independent linear bandits when the common feature extractor does not exist.
General Function Approximation
In this paper, we focus on linear bandits.
In the bandits literature, sublinear regret guarantees have been proved for more general reward function classes beyond the linear one (Li et al., 2017; Agrawal et al., 2019).
Similarly, in the supervised representation learning literature, general representation function classes have also been studied (Maurer et al., 2016; Du et al., 2020).
An interesting future direction is to merge these two lines of research by developing provably efﬁcient algorithms for multi-task bandits problems where general function class is used for representation
A P ROOF OF T HEOREM
Lemma 1 (General Hoeffding’s inequality, Vershynin (2018), Theorem 2.6.2) .
Let X 1 , . . .
, X n be independent random variables such that E
[ X i ] =
and X i is σ i -sub-Gaussian.
Then there exists a constant c > 0 , such that for any δ > 0 , we have
[Equation]
Lemma 2.
Let T be the number of tasks and N 0 be the number of samples.
For every ( n, t ) ∈ [ N 0 ] × [ T ] , let x n,
t ∈ R d be ﬁxed vectors and let y n,t = x (cid:62) n,t θ t + ε n,t , where θ t ∈ R
d is a vector and ε n,t is an independent 1-sub-Gaussian variable.
Let
[Equation]
where (cid:99) W =
( (cid:98) w· · · (cid:98) w T ) .
Then with probability 1 − δ , we have
[Equation]
Proof.
By (4), we have
[Equation]
Since y n,t = x (cid:62) n,t
Bw t + ε n,t , we have
[Equation]
which implies
[Equation]
Next we bound the right-hand side of (5) via a uniform concentration argument.
We let
[Equation]
For any ﬁxed matrices B (cid:48) ∈ B , W
[Equation]
∈ W , we write W (cid:48) = ( w (cid:48)· · · w
[Equation]
) and deﬁne
[Equation]
Note that η n,t ( B (cid:48) , W (cid:48) ) is an independent sub-Gaussian variable with sub-Gaussian norm 2 x (cid:62) n,t ( B (cid:48) w (cid:48)
t − Bw t ) .
By the general Hoeffding’s inequality (Lemma 1), with probability 1 − δ , we have
[Equation]
6) where f ( B (cid:48) , W (cid:48) )
=
N (cid:88) n =
T (cid:88) t =
[ x (cid:62) n,t ( B (cid:48) w (cid:48) t − Bw t )] 2 .
Next we apply the (cid:15) -net.
Let B (cid:48) =
N ( B , (cid:107)·(cid:107) max , (cid:15) ) , W (cid:48) =
N ( W , (cid:107)·(cid:107) max , (cid:15) ) .
Applying an union bound over B (cid:48)
× W (cid:48) for (6), we have
Pr (cid:34) ∀ ( B (cid:48) , W (cid:48) )
∈ B (cid:48)
× W (cid:48) :
N 0 (cid:88) n = T (cid:88) t = η n,t (cid:46) (cid:112) f ( B (cid:48) , W (cid:48) ) log(1 /δ )
(cid:35) ≥ 1 − δ |B (cid:48) × W (cid:48) | . Since f ( B (cid:48) , W (cid:48) ) is (3 NTdk ) -Lipschitz with respect to (cid:107)·(cid:107) max , we have
Pr (cid:34) ∀ ( B (cid:48) , W (cid:48) )
[Equation]
B × W :
[Equation]
[Equation]
(cid:35)
[Equation]
(cid:48) × W (cid:48) | .
Note that |B (cid:48)
| =
O
((1 /(cid:15) )
dk ) and |W (cid:48)
| =
O
((1 /(cid:15) ) kT ) .
Let δ =
δ 0 / |B (cid:48) × W (cid:48) | and (cid:15) =
(24 NTdk ) − 1 .
We have
Pr (cid:34) ∀ ( B (cid:48) , W (cid:48) ) [Equation] B [Equation]
W :
[Equation]
(cid:88)
[Equation] [Equation] [Equation]
] (cid:35)
[Equation]
Now we assume that the above event holds.
Combining (5) and (7), we have
Pr (cid:20)(cid:113)
[Equation]
( (cid:98) B , (cid:99) W ) (cid:46) (cid:112) ( dk + kT ) log( NTdk )
+ log(1 /δ 0 ) (cid:21) =
Pr (cid:20) f
( (cid:98) B , (cid:99) W
[Equation]
which proves our lemma.
Lemma 3.
With probability 1 − O (( NT ) − 2 ) , for every m ∈
[ M ] , t ∈ [ T ] , we have
[Equation]
Proof.
The proof can be done by following the proof of Lemma 4 in Han et al.
Lemma 4.
For each epoch m ∈ [ M ] , with probability 1 − O (( NT − 2 )) , we have
[Equation]
where (cid:98) B , (cid:99) W are computed at Line 5 in Algorithm 1.
Proof.
{+Placing N 
+}
= G m − G m − 1 in Lemma 2, with probability 1 − O ( NT ) − 2 ) , we have
[Equation]
By Lemma 3, with probability 1 − O (( NT ) − 2 ) , we have
[Equation]
[Equation]
Let
[Equation]
be the regret incurred in the m -th epoch.
We have the following lemma.
Lemma 5.
We have
[Equation]
Proof.
At round n that belongs to epoch m , for task t , we have
[Equation]
By Assumption 2 and a union bound over K actions, T tasks
[Equation]
and ( G m − G m − 1 ) rounds, we have with probability 1 − ( NKT ) − 2 , for every n ∈ ( G m − 1 , G m ] and t ∈ [ T ] ,
[Equation]
[Equation]
[Equation]
Using a union bound over (10) and Lemma
[Equation]
with probability 1 − O (( NT ) − 2 )
[Equation]
the regret incurred in the m -th epoch is
[Equation]
where we denote (cid:98) Θ = (cid:0)(cid:98) θ m − 1 ,· · · (cid:98) θ m − 1 ,T (cid:1) in (11) and the second inequality in (11) uses Cauchy, (12) uses Lemma 4, (13) uses G m − 1 (cid:46) G m − 1 − G m − 2 and
G m / G m − 1 (cid:46) √ N .
Since that the above bound holds with probability 1 − O (( NT ) − 2 ) and that the regret is bounded by NT , we prove the lemma.
Proof of Theorem 1.
The regret is bounded by
[Equation]
B P ROOF OF T HEOREM
In this appendix, we assume that Σ t
= I in Assumption 2 and that all noises are gaussian, i.e. ε n,t ∼ N (0 , 1) for all n ∈ [ N ] , t ∈ [ T ] .
For each task t ∈ [ T ] , we denote the regret incurred on task t as
[Equation]
We divide Theorem 2 into the following two lemmas.
Lemma 6.
Under the setting of Theorem 2, we have inf A sup I E [ R N,T A , I ] ≥ Ω( T √ kN ) .
Lemma 7.
Under the setting of Theorem 2, we have inf A sup I E [ R N,T A , I ] ≥ Ω( √ dkNT ) .
Proof of Theorem 2.
We combine Lemma 6 and Lemma 7.
Our proofs to the lemmas will be based on the lower bounds for the (single-task) linear bandit setting, which corresponds to the T = 1 case in our multi-task setting.
For this single-task setting, we assume k = d and B = I d .
We write the regret as R N = R N, 1 and call algorithms for the single-task setting as single-task algorithms.
Lemma 8 (Han et al.
(2020), Theorem 2) .
Assume N ≥ d 2 and d ≥ 2 .
Let N ( µ, Σ) be the multivariate normal distribution with mean µ and covariance matrix Σ .
There is a constant C > 0 , such that for any single-task algorithm S , we have
[Equation]
where I is the instance with hidden linear coefﬁcients w .
Next we use it to prove Lemma 6 and Lemma 7.
The main idea to prove Lemma 6 is to note that we can treat our setting as T independent k -dimensional linear bandits.
Proof of Lemma 6.
Suppose there is an algorithm A that achieves sup I E
[ R N,T A , I ] ≤ CT √ kN .
Then we have
[Equation]
Therefore, there exists t ∈ [ T ] such that
[Equation]
which contradicts Lemma 8.
Proof of Lemma 7.
Suppose there is an algorithm A that achieves sup I E
[ R N,T A , I ] ≤ C √ dkNT .
We complete the proof separately, based on whether k ≤ d 2 or not.
Note that when k > d 2 , the lower bound in Lemma 6 becomes Ω( T √ kN )
=
Ω( T √ dN ) .
Since T ≥ k , we have √ dkTN (cid:46) T √ dN .
Thus we conclude by Lemma 6.
In the remaining, we assume k ≤ d 2 .
Without loss of generality, we assume that d is even and that 2 k divides T .
For i = 1 , . . .
, k , we denote the regret of group i as
[Equation]
We consider instances such that tasks from the same group share the same hidden linear coefﬁcients θ t .
Since there are k groups, we have
[Equation]
Therefore, there exists a group i ∈
[ k ] such that
[Equation]
[Equation]
which means that the regret incurred in group i is less than C (cid:112) dNT/k .
Since the tasks in group i share the same hidden linear coefﬁcients, they could be regarded as one large linear bandit problem.
Since there are T/k tasks in group i , the large linear bandit is played for N · T/k rounds.
By
Lemma 8, the algorithm A must have incurred C (cid:112) dNT/k regret on group i , which contradicts with (15).
C M ETHOD - OF -M OMENTS E STIMATOR UNDER B ANDIT S ETTING
The following theorem shows the guarantee of the method-of-moments (MoM) estimator we used to ﬁnd the linear feature extractor B .
In this appendix, for a matrix B with orthogonal columns, we write B ⊥ to denote its orthogonal complement matrix (a matrix whose columns are the orthogonal complement of those of B ).
Theorem 5 (MoM Estimator) .
Assume N 1 T (cid:38) polylog( N 1 , T ) · d 1 .
5 k λ 0 ν .
We have with probability at least 1 − ( N 1 T ) − 100 ,
[Equation]
The theorem guarantees that our estimated (cid:98) B is close to the underlying B in the operator norm so long as the values
N 1 and T are sufﬁciently large.
We add a remark that our theorem is similar to
Theorem 3 in Tripuraneni et al.
The key differences are: (i) we use a uniform distribution to ﬁnd the feature extractor, while they assumed the input distribution is standard d -dimensional
Gaussian; (ii) the SNR (signal-to-noise ratio) in our linear bandit setting is worse than that in their supervised learning setting, and thus we get an extra d factor in our theorem.
In the sequel, we prove the theorem.
Lemma 9 (Hoeffding) .
Let ε 1 , . . .
, ε
N be i.i.d.
1 -sub-Gaussian random variables.
We have
[Equation]
Lemma 10 (Matrix Bernstein’s inequality, Vershynin (2018), Theorem 5.4.1) .
Let X 1 , . . .
X m ∈ R
d × d be independent, mean zero, symmetric random matrices that (cid:107)
X i (cid:107) ≤ M almost surely for all i ∈ [ m ] .
Let σ 2 = (cid:13) (cid:13) (cid:80) m i =1 E X[-
-]
i (cid:13) (cid:13) .
We have
[Equation]
m (cid:88) i [Equation] Equivalently, with probability at least 1 − δ , we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
X i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:46) (cid:112)
[Equation]
Lemma 11 (Moments of Uniform Distribution on Sphere) .
Let x ∼ Unif( S d − 1 ) be a uniformly chosen unit vector.
We have E x 6 1 = 15 d ( d +2)
( d +4) ,
E x 4 1 = 3 d ( d +2) , E x 2 1 = 1 d .
Moreover, we have
E x 41 x 22 = 3 d ( d +2)
( d +4) .
Proof.
We need to recall the fact that when x ∼ Unif( S d − 1 ) , its coordinate x i follows the Beta distribution: x 1 + ∼ Beta( d − 12 , d − 12 ) .
Then we prove the lemma by noting the moments of the
Beta distribution (Fang, 2018).
Corollary 6 (Uniform Distribution on Sphere) .
Let x ∼ Unif( S d − 1 ) be a uniformly chosen unit vector.
We have the following statements.
(a) E (cid:104) x , θ (cid:105) 2 xx (cid:62) [Equation] 2 θθ (cid:62) + I d ( d +2) .
(b) E (cid:104) x , θ (cid:105) 4 xx (cid:62)
= 12 θθ (cid:62) +3 I d ( d +2)
( d +4) .
Proof.
Let e 1 = (1 , 0 , . .
. , 0) ∈
R d be the unit vector.
For (a), note that
[Equation]
ij = E [Equation] [Equation]
i x [Equation] =   0 , i, j (cid:54) = 1 and i (cid:54) = j, 0 , i = 1 (cid:54) =
j or j = 1 (cid:54)
=
i, 1 d ( d +2) , i = j (cid:54) = 1 ,d ( d +2) , i = j = 1 .
Therefore, we have E [ (cid:104) x , e 1 (cid:105) 2 xx (cid:62) ]
= 1 d ( d +2) (2 e 1 e (cid:62) 1 + I ) .
By the isotropy (rotation invariance) of uniform distribution, we have E
[Equation]
[Equation]
For (b), note that
[Equation]
  0 , i, j (cid:54) = [-
-]1 and i (cid:54) =
j, 0 , i = 1 (cid:54) =
j or j = 1 (cid:54)
=
i,
[Equation]
d
[Equation]
+
[Equation]
+
[Equation]
j = 1 .
Therefore, we have E (cid:104) x , e 1 (cid:105) 4 xx (cid:62) = 1 d ( d +2)( d +4) (12 e 1 e (cid:62) 1 + 3 I ) .
By the isotropy, we have
E (cid:104) x ,
[Equation]
Let A n,t = r 2 n,t x n,t x (cid:62) n,t , M = 1 N 1 T (cid:80) N 1 n =1 (cid:80) Tt =
1 A n,t .
We decompose M into three terms M =
M 1 + M 2 + M 3 , where
[Equation]
n = [Equation] T (cid:88) t
= (cid:104)
[Equation]
n
[Equation]
n
[Equation]
(cid:62)
[Equation]
M
[Equation]
=
[Equation]
N
[Equation]
N
[Equation]
(cid:88) n =
[Equation]
(cid:88)
[Equation]
=
[Equation]
ε n
[Equation]
(cid:104)
[Equation]
n
[Equation]
(cid:105)
[Equation]
n
[Equation]
(cid:62) n
[Equation]
M 3 =
N
[Equation]
T
N 1 (cid:88) n =
T (cid:88) t =
[Equation]
ε 2 n
[Equation]
t x n [Equation] t x (cid:62) n,t . Let A 1 nt = 1 N 1 T (cid:104) x n,t , θ t (cid:105) 2 x n,t x (cid:62) n,t .
Next we analyze each error (cid:107) M i − E M i (cid:107) .
Lemma 12.
With probability at least 1 − [Equation] we have
[Equation]
Proof.
We have
[Equation]
Using Lemma 10 with m = N 1 T, M = 1 N 1 T , σ 2 (cid:46) m · 1 d 3 N 21 T 2 = 1 d 3 N 1 T , we have with probability at least 1 − 1 N
[Equation]
[Equation]
Next we analyze the errors of M 2 and M 3 .
Since these errors contain the unbounded sub-Gaussian terms ε n,t , we need to cut their tails before applying the matrix Bernstein inequality.
Deﬁne ε (cid:48) n,t = ε n,t I {| ε n,t | ≤ R } .
We have
[Equation]
Let ε (cid:48)(cid:48) n,t = ε 2 n,t
I {| ε n,t | ≤ R } .
We have
[Equation]
Deﬁne
[Equation]
n =
[Equation]
n
[Equation]
x
[Equation]
Lemma 13.
With probability at least 1 − ( N 1 T ) − 3 , we have
[Equation]
Proof.
Let A 2 nt
= 1 N 1 T 2 ε (cid:48) n,t (cid:104) x n,t , θ t (cid:105) x n,t x (cid:62) n,t .
We ﬁnd that
[Equation]
We conclude by using Lemma 10 with m = N 1 T, M = R N 1 T , σ 2 (cid:46) m · [Equation]
Lemma 14.
With probability at least 1 − ( N 1 T ) − 3 , we have
[Equation]
− E M
[Equation]
[Equation]
Proof.
Let A 3 nt = 1 N 1 T ε (cid:48)(cid:48) n,t x n,t x (cid:62) n,t .
We ﬁnd that
[Equation]
We conclude by using Lemma 10 with m = N 1 T, M = R 2 N 1 T , σ 2 (cid:46) m · 1 dN
[Equation]
T 2 = 1 dN 1 T .
Lemma 15.
With probability at least 1 − ( N 1 T ) − 2 , we have
[Equation]
Proof.
Let R =
(cid:112) 8 log( N 1 T ) .
With probability at least 1 − ( N 1 T ) − 3 , we have | ε n,t | ≤ R for every n ∈ [ N 1 ] and t ∈ [ T ] .
Note that in this case, we have M 2 = M (cid:48)
and M 3 =
M (cid:48)(cid:48) 3 .
Using a union bound over Lemma 12, Lemma [Equation] and Lemma 14, we have with probability at least 1 − ( N 1 T ) − 2 ,
[Equation]
Proof of Theorem 5.
We note that σ k +1 ( M ) − σ k +1 ( E M ) ≤ (cid:107) E (cid:107) .
Under Assumption 3, we have
[Equation]
where c 1 , c 3 ∈ R are constants.
Since
[Equation]
we have
[Equation]
Assume N 1 T (cid:38) polylog( N 1 , T ) · dk so that (cid:107) E (cid:107) ≤ λ 0 ν d 2 k .
Together with Davis-Kahan sin θ theorem, we have with probability at least 1 − ( N 1 T ) − 100 ,
[Equation]
where (17) uses the Davis-Kahan sin θ theorem (Bhatia, 2013, Section VII.3).
D P ROOF OF T HEOREM
Our proof is similar to the proof of Theorem 3.1 of Rusmevichientong & Tsitsiklis (2010).
Lemma 16 (Rusmevichientong & Tsitsiklis (2010), Lemma 3.5) .
For two vectors u , v ∈ R d , we have (cid:13)(cid:13)(cid:13) (cid:13) u (cid:107) u
(cid:107) − v (cid:107) v (cid:107) (cid:13)(cid:13)(cid:13) (cid:13) ≤ 2 (cid:107) u − v (cid:107) max {(cid:107) u (cid:107) , (cid:107) v (cid:107)} .
Lemma 17.
Let x t = arg max x ∈A t (cid:104) x
, ˆ θ t (cid:105) .
We have
[Equation]
Proof.
For θ ∈ R d [Equation] we deﬁne f t ( θ ) = max a ∈A t (cid:104) a , θ (cid:105) .
Let x ∗ t = arg max x ∈A t (cid:104) x
θ t [Equation]
Then we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] where the ﬁrst inequality in (18) uses Cauchy and (19) uses Lemma 16.
Lemma 18.
For each task t ∈ [ T ] , we have E (cid:107) (cid:98) θ t [Equation] t [Equation]
[Equation] 21 N [Equation] +
[Equation] B (cid:107) 2 .
Proof.
We deﬁne θ (cid:48) t
= (cid:98) B (cid:98) B (cid:62) θ t .
Note that θ t = θ (cid:48) t + (cid:98) B ⊥ (cid:98) B (cid:62)⊥ .
We have
[Equation]
Note that (cid:98) B is perpendicular to (cid:98) B ⊥ and that (cid:107) (cid:98) B (cid:107)
= (cid:107) (cid:98) B ⊥ (cid:107) = 1 . We have
[Equation]
B ⊥
[Equation]
B [Equation] B
[Equation]
B [Equation] Let v i = √ λ
ˆ b i .
The OLS estimator is given by
[Equation]
Write A = (cid:80) N 1 + N 2 +1 n = N 1 +
[Equation]
B
[Equation]
x n
[Equation]
t
[Equation]
n
[Equation]
t
[Equation]
B
[Equation]
E
[Equation]
w t
[Equation]
t
[Equation]
Putting together
[Equation]
we have
[Equation]
Proof of Theorem 3.
Note that J
= O (1) under our assumptions, as indicated by Rusmevichientong & Tsitsiklis (2010).
So we have
[Equation]
E P ROOF OF T HEOREM
In this appendix, we assume all action sets are spherical, i.e. A n,t ≡ S d − 1 for n ∈ [ N ] , t ∈
[ T ] .
Note that these action sets meet Assumption 3.
For each task t ∈ [ T ] , we use R N, ( t ) =
(cid:80) Nn =1 [max x ∈A n,t (cid:104)
x , θ t (cid:105)− E (cid:104) x n,t , θ t (cid:105) ] to denote the regret incurred on task t .
Lemma 19.
Under the setting of Theorem 4, we have inf A sup I E [ R N,T A , I ] ≥ CkT √ N, where
C = 0 . 0001 .
Lemma 20.
Under the setting of Theorem 4, we have inf A sup I E [ R N,T A , I ] ≥ Cd √ kTN, where
C = 0 . 0001 .
Proof of Theorem 4.
We combine Lemma 19 and Lemma 20.
The proofs in this appendix would largely follow the proofs in Appendix B, yet this appendix is much longer, because we need to construct instances that satisﬁes Assumption 3, Assumption 4, and
Assumption 5.
Our proofs to the lemmas are based on the lower bounds for the (single-task) linearly parameterized bandit setting, which corresponds to the T = 1 case in our setting.
For this single-task setting, we assume k = d and B = I d .
(This setting need not meet Assumption 4 and Assumption 5).
We write the regret as R N = R N, 1 and call algorithms for the single-task setting as single-task algorithms.
We invoke the following lower bound for the single-task setting.
Lemma 21.
Assume N ≥ d 2 and d ≥ 2 .
Let N ( µ, Σ) be the multivariate normal distribution with mean µ and covariance matrix Σ .
For any single-task algorithm, we have
[Equation]
The lemma can be proved by following the proof of Theorem 2.1 of Rusmevichientong & Tsitsiklis (2010).
Next we use it to prove Lemma 19 and Lemma 20.
The main idea to prove Lemma 19 is to note that we can treat our setting as T independent k -dimensional linear bandits.
Let µ d be the conditional probability measure whose density function is given by
[Equation]
where X ∼ N (0 , I d /d ) is the multivariate gaussian vector and g ( x ) is the probability density function of X .
Then (20) implies
[Equation]
Proof of Lemma 19.
Without loss of generality, we assume 2 k divides T .
Suppose, for contradiction, that there is an algorithm A (cid:48) , such that for every instance I , it incurs regret E [ R N,T A (cid:48) , I ] ≤
CTk √ N .
We replace the condition (cid:107) w t (cid:107) ≤ 1 by (cid:107) w t (cid:107) ≤ 3 in our setting.
Note that A (cid:48) implies an algorithm A that incurs regret E [ R N,T A , I ] ≤ 3 CTk √ N .
We construct the following instances I =
( B , W ) , where B =
( I k 0) (cid:62) and W = ( w· · · w T ) as follows.
Let
[Equation]
where e 1 , . .
, e k ∈ R k is the standard basis.
Let
[Equation]
be i.i.d.
drawn.
Thanks to the ﬁrst T 2 tasks, the instance I always satisﬁes Assumption 4.
Note that (cid:107) w t (cid:107) ≥ 0 . 09 , so the instance also satisﬁes Assumption 5.
Then we have
[Equation]
Thus
[Equation]
Therefore, we can ﬁnd t ∈
[ T/ 2 + 1 , T ] such that
[Equation]
We note that the expectation operator E w T/ 2+1 ,...,w T ∼ µ d is over all randomness on tasks τ (cid:54) =
t and its parameter w τ .
So there is a realization of the randomness on other tasks τ (cid:54) =
t that satisﬁes
[Equation]
Based on the realization w τ , ε τ , we design a single-task algorithm S , which plays task t and simulates other tasks τ (cid:54)
=
t with w τ , ε τ .
The algorithm achieves
[Equation]
which contradicts to (21) because C = 0 . 0001 .
Proof of Lemma 20.
Without loss of generality, we assume that d is even and that 2 k divides T .
Suppose, for contradiction, that there is an algorithm A (cid:48) , such that for every instance I , it incurs regret E [ R N,T A (cid:48) , I ] ≤ Cd √ kTN .
We replace the condition (cid:107) w t (cid:107) ≤ 1 by (cid:107) w t (cid:107) ≤ 3 in our setting.
Note that A (cid:48) implies an algorithm A that incurs regret E [ R N,T A , I ] ≤ 3 Cd √ kTN .
We prove the lemma separately, based on whether k ≥ d 2 or not.
Consider k ≤ d 2 .
We generate k vectors ψ 1 , . . .
, ψ k so that ψ i ∼ µ d − i +1 .
For every dimension i , we consider a map
c
i : R
[Equation]
→ R d × ( d − i +1) , ( x 1 , . .
, x i ) (cid:55)→ ( y
[Equation] , y [Equation] − i +1 ) , so that when { x 1 , . . .
, x i } are orthogonal, the set { y 1 , . .
, y d − i +1 } is the orthonormal basis of the orthogonal complement span { x 1 , . . .
, x i } ⊥ of R d .
Note that c
i can be computed efﬁciently, e.g. by
Gram-Schmidt process.
We deﬁne φ 1 = ψ 1 and φ i
=
c i − 1 ( φ 1 , . . .
, φ i − 1 ) · ψ i for i ≥ 2 .
We observe that { φ 1 , . . .
, φ k } are orthogonal.
Next we deﬁne our instance I =
( B , W ) , where B = ( b· · · b k ) and b i =
φ i (cid:107) φ i (cid:107) .
For W = ( w· · · w T ) , we let
[Equation]
where { e 1 , . .
, e k } ⊆
R k is the standard basis.
Note that the instance I always satisﬁes Assumption 4 and Assumption 5 by our choice of w t .
Note that we have divided the tasks into k groups, so that each group share the same vector w t .
For i = 1 , . . .
, k , we denote the regret of group i as
[Equation]
N, (
[Equation]
For the algorithm A , it incurs regret
[Equation]
So there is a group i ∈
[ k ] , such that
[Equation]
Similar to the proof of Lemma 19, we can ﬁx the randomness for groups j (cid:54) =
i to obtain a realization, such that
[Equation]
Here we note that φ i could depend on φ ι for ι ≥ i + 1 .
Now we let ψ (cid:48)
i ∼ µ d − k
+
and let φ (cid:48) i
[Equation]
Note that φ (cid:48)
i and φ i | φ j ( j (cid:54) = i ) are identical, so we have
[Equation]
We complete the proof by showing that (23) implies a single-task algorithm S that plays a ( d − k +1) - dimensional linear bandit for NT k times.
Let w = ψ (cid:48)
i .
Then w is independently drawn from µ d − k +1 .
Next we design the algorithm S , which runs A by playing the tasks t ∈ T =
{ ( i − 1) T/k + 1 , . .
, iT/k } and simulates other tasks τ / ∈ T .
Note that playing the task t ∈ T is the same as playing the single-task bandit deﬁned by w , because we have θ t
= B w t =
φ i
=
Aψ (cid:48)
i and the matrix A is known (as in (22))
after we ﬁx the randomness on other tasks.
Since |T | =
Tk and A is played for N times, S can play the single-task bandit speciﬁed by w for NTk times.
As a result, we have
[Equation]
φ i = φ (cid:48) i [ R N, (( i ))
A , I | φ j , ε j , j (cid:54) = i ] ≤ [Equation] Cd (cid:114)
NT k [Equation]
[Equation] ) (cid:114) NT k , which contradicts to (21) because C = 0 . 0001 .
Since T ≥ k , we have d √ kTN (cid:46) Td √ N .
Thus we conclude by Lemma 19.
[Figure]
[Figure] [Figure] [Figure] R e g r e t p e r
T a s k × 10c =   c
= 1 .
E 2 TC
PEGE   c =
[Figure]
[Figure]
R e g r e t p e r
T a s k
[Figure]
c = 1 .
E 2 TC
PEGE   c =
[Figure]
F E XPERIMENTS FOR I NFINITE -A RM S ETTING
Setup In all experiments, we set d = 10 , N = 10 4 and the action A t = S
d − 1 .
The parameters are generated as follows.
We consider k = 2 , 3 in our experiments.
The noise ε n,t ∼ N (0 , 1) are i.i.d.
Gaussian random variables.
To verify our theoretical results, we consider a hyper-parameter c ∈ { 0 . 5 , 1 , 1 . 5 , 2 } .
For each c , we run E 2 TC with N 1 =
d c k (cid:113) NT and N 2 = k √ N .
Results and Discussions We present the simulation results in Figure 5 and Figure 6.
We emphasize that the y -axis in our ﬁgures corresponds to the regret per task, which is deﬁned as Reg N,T T .
Our main observation is that only when the number of tasks T is large and we choose the right scaling N 1 = d 1 .
5 k (cid:113) NT , our method can outperform the PEGE algorithm.
We discuss several implications of our results.
First, representation learning does help, especially when there are many tasks available for us to learn the representation, as we see in all ﬁgures that the regret per task of
E 2 TC decreases as T increases.
Second, the help of representation learning is bounded.
In the ﬁgures, we see that the curves of E 2 TC bends to a horizontal line as T increases, which suggests a lower bound on the regret per task.
Meanwhile, we also proved an Ω( k √ N ) lower bound on the regret per task in Theorem 4.
Third, representation learning may have adverse effect without enough task.
In our ﬁgures, this was established by noting that our algorithm cannot outperform PEGE when T is small.
This corresponds to the “negative transfer” phenomenon observed in previous work (Wang et al., 2019).
Fourth, the correct hyper-parameter c = 1 . 5 is crucial.
For hyperparameter other than c = 1 . 5 , the ﬁgures show that our algorithm would require much more tasks to outperform PEGE .
Lastly, by comparing the two ﬁgures, we notice that our algorithm has bigger advantage when k is smaller, which corroborates the scaling with respect to k in our regret upper bound.
In contrast, PEGE does not beneﬁt from a smaller k .