N EURAL C ONTEXTUAL B ANDITS WITH D EEP R EPRESENTATION AND S HALLOW E XPLORATION
A BSTRACT
We study neural contextual bandits, a general class of contextual bandits, where each context-action pair is associated with a raw feature vector, but the speciﬁc reward generating function is unknown.
We propose a novel learning algorithm that transforms the raw feature vector using the last hidden layer of a deep ReLU neural network (deep representation learning), and uses an upper conﬁdence bound (UCB) approach to explore in the last linear layer (shallow exploration).
We prove that under standard assumptions, our proposed algorithm achieves (cid:101) O ( √ T ) ﬁnitetime regret, where T is the learning time horizon.
Compared with existing neural contextual bandit algorithms, our approach is computationally much more efﬁcient since it only needs to explore in the last layer of the deep neural network.
I NTRODUCTION
Multi-armed bandits (MAB) (Auer et al., 2002;
Audibert et al., 2009; Lattimore & Szepesv ´ ari, 2020) are a class of online decision-making problems where an agent needs to learn to maximize its expected cumulative reward by repeatedly interacting with a partially known environment.
Following a bandit algorithm (also called a strategy or policy), in each round, the agent adaptively chooses an arm, and then receives a reward associated with that arm.
Since only the reward of the chosen arm will be observed (bandit information feedback), a good bandit algorithm has to deal with the exploration-exploitation dilemma: trade-off between pulling the best arm based on existing knowledge/history data (exploitation) and trying the arms that have not been fully explored (exploration).
In many real-world applications, the agent will also be able to access detailed contexts associated with the arms.
For example, when a company wants to choose an advertisement to present to a user, the recommendation will be much more accurate if the company takes into consideration the contents, speciﬁcations, and other features of the advertisements in the arm set as well as the proﬁle of the user.
To encode the contextual information, contextual bandit models and algorithms have been developed, and widely studied both in theory and in practice (Dani et al., 2008; Rusmevichientong &
Tsitsiklis, 2010; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011).
Most existing contextual bandit algorithms assume that the expected reward of an arm at a context is a linear function in a known context-action feature vector, which leads to many useful algorithms such as LinUCB (Chu et al., 2011), OFUL (Abbasi-Yadkori et al., 2011), etc.
The representation power of the linear model can be limited in applications such as marketing, social networking, clinical studies, etc., where the rewards are usually counts or binary variables.
The linear contextual bandit problem has also been extended to richer classes of parametric bandits such as the generalized linear bandits (Filippi et al., 2010; Li et al., 2017) and kernelised bandits (Valko et al., 2013; Chowdhury & Gopalan, 2017).
With the prevalence of deep neural networks (DNNs) and their phenomenal performances in many machine learning tasks (LeCun et al., 2015; Goodfellow et al., 2016), there has emerged a line of work that employs DNNs to increase the representation power of contextual bandit algorithms (Allesiardo et al., 2014; Riquelme et al., 2018; Collier & Llorens, 2018; Zahavy & Mannor, 2019;
Zhou et al., 2020; Deshmukh et al., 2020; Zhang et al., 2020).
The problems they solve are usually referred to as neural contextual bandits .
For example, Zhou et al.
(2020) developed the NeuralUCB algorithm, which can be viewed as a natural extension of LinUCB (Chu et al., 2011; Abbasi-Yadkori et al., 2011), where they use the output of a deep neural network with the feature vector as input to approximate the reward.
Zhang et al.
(2020) adapted neural networks in Thompson Sampling (Thompson, 1933; Chapelle & Li, 2011; Russo et al., 2018) for both exploration and exploitation and proposed NeuralTS .
For a ﬁxed time horizon T , it has been proved that both NeuralUCB and NeuralTS achieve a O ( (cid:101) d √ T ) regret bound, where (cid:101) d is the effective dimension of a neural tangent kernel matrix which can potentially scale with O ( TK ) for K -armed bandits.
This high complexity is mainly due to that the exploration is performed over the entire huge neural network parameter space, which is inefﬁcient and even infeasible when the number of neurons is large.
A more realistic and efﬁcient way of learning neural contextual bandits may be to just explore different arms using the last layer as the exploration parameter.
More speciﬁcally, Riquelme et al.
(2018) provided an extensive empirical study of benchmark algorithms for contextual-bandits through the lens of Thompson Sampling, which suggests decoupling representation learning and uncertainty estimation improves performance.
In this paper, we show that the decoupling of representation learning and the exploration can be theoretically validated.
We study a new neural contextual bandit algorithm, which learns a mapping to transform the raw features associated with each context-action pair using a deep neural network ( deep representation ), and then performs an upper conﬁdence bound (UCB)-type exploration over the linear output layer of the network ( shallow exploration ).
We prove a sublinear regret of the proposed algorithm by exploiting the UCB exploration techniques in linear contextual bandits (Abbasi-Yadkori et al., 2011) and the analysis of deep overparameterized neural networks using neural tangent kernels (Jacot et al., 2018).
Our theory conﬁrms the empirically observed effectiveness of decoupling the deep representation learning and the UCB exploration in contextual bandits (Riquelme et al., 2018; Zahavy & Mannor, 2019).
Contributions we summarize the main contributions of this paper as follows
• We propose a contextual bandit algorithm, Neural-LinUCB, for solving a general class of contextual bandit problems without knowing the speciﬁc reward generating function.
The proposed algorithm learns a deep representation to transform the raw feature vectors and performs UCBtype exploration in the last layer of the neural network, which we refer to as deep representation and shallow exploration.
Compared with LinUCB (Li et al., 2010; Chu et al., 2011) and neural bandits such as NeuralUCB (Zhou et al., 2020) and NeuralTS (Zhang et al., 2020), our algorithm enjoys the best of two worlds: strong expressiveness due to the deep representation and computational efﬁciency due to the shallow exploration.
• Despite the usage of a DNN as the feature mapping, we prove a (cid:101) O ( √ T ) regret for the proposed Neural-LinUCB algorithm, which matches the regret bound of linear contextual bandits (Chu et al., 2011; Abbasi-Yadkori et al., 2011).
To the best of our knowledge, this is the ﬁrst work that theoretically shows the convergence of bandits algorithms under the scheme of deep representation and shallow exploration.
It is notable that a similar scheme called Neural-Linear was proposed by Riquelme et al.
(2018) for Thompson sampling algorithms, and
they empirically showed that decoupling representation learning and uncertainty estimation improves the performance.
Our work conﬁrms this observation from a theoretical perspective.
• We conduct experiments on contextual bandit problems based on real-world datasets, demonstrating a better performance and computational efﬁciency of Neural-LinUCB over LinUCB and existing neural bandits algorithms such as NeuralUCB, which well aligns with our theory.
1. A DDITIONAL RELATED WORK
There is a line of related work to ours on the recent advance in the optimization and generalization analysis of deep neural networks.
In particular, Jacot et al.
ﬁrst introduced the neural tangent kernel (NTK) to characterize the training dynamics of network outputs in the inﬁnite width limit.
From the notion of NTK, a fruitful line of research emerged and showed that loss functions of deep neural networks trained by (stochastic) gradient descent can converge to the global minimum (Du et al., 2019b; Allen-Zhu et al., 2019b; Du et al., 2019a;
Zou et al., 2018; Zou & Gu, 2019).
The generalization bounds for overparameterized deep neural networks are also established in Arora et al.
(2019a;b); Allen-Zhu et al. (2019a); Cao & Gu (2019a;b).
Recently, the NTK based analysis is also extended to the study of sequential decision problems including bandits (Zhou et al., 2020;
Zhang et al., 2020), and reinforcement learning algorithms (Cai et al., 2019;
Liu et al., 2019; Wang et al., 2020; Xu & Gu, 2020).
Our algorithm is also different from Langford & Zhang (2008); Agarwal et al.
(2014) which reduce the bandit problem to supervised learning.
Moreover, their algorithms need to access an oracle that returns the optimal policy in a policy class given a sequence of context and reward vectors, whose regret depends on the VC-dimension of the policy class.
Notation We use [ k ] to denote a set { 1 , . . .
, k } , k ∈ N + .
(cid:107) x (cid:107) 2 = √ x
(cid:62) x is the Euclidean norm of a vector x ∈ R d .
For a matrix W ∈ R m × n , we denote by (cid:107) W (cid:107) 2 and (cid:107) W (cid:107) F its operator norm and Frobenius norm respectively.
For a semi-deﬁnite matrix A ∈ R d × d and a vector x ∈ R d , we denote the Mahalanobis norm as (cid:107) x (cid:107) A = √ x (cid:62) Ax .
Throughout this paper, we reserve the notations { C i } i =0 , 1 ,... to represent absolute positive constants that are independent of problem parameters such as dimension, sample size, iteration number, step size, network length and so on.
The speciﬁc values of { C i } i =0 , 1 ,... can be different in different context.
For a parameter of interest T and a function f ( T ) , we use notations such as O ( f ( T )) and Ω( f ( T )) to hide constant factors and (cid:101)
O ( f ( T )) to hide constant and logarithmic dependence of T .
P RELIMINARIES
In this section, we provide the background of contextual bandits and deep neural networks.
2. L INEAR CONTEXTUAL BANDITS
A contextual bandit is characterized by a tuple ( S , A , r ) , where S is the context (state) space, A is the arm (action) space, and r encodes the unknown reward generating function at all context-arm pairs.
A learning agent, who knows S and A but does not know the true reward r (values bounded in (0 , 1) for simplicity), needs to interact with the contextual bandit for T rounds.
At each round t = 1 , . . .
, T , the agent ﬁrst observes a context s t ∈ S chosen by the environment; then it needs to adaptively select an arm a t ∈
A based on its past observations; ﬁnally it receives a reward
[Equation]
1) where x s,a ∈ R d is a known feature vector for context-arm pair ( s, a ) ∈ S × A , and ξ t is a random noise with zero mean.
The agent’s objective is to maximize its expected total reward over these T rounds, which is equivalent to minimizing the pseudo regret (Audibert et al., 2009):
[Equation]
where a ∗ t ∈ argmax a ∈A { r ( x s t ,a ) =
E [ (cid:98) r ( x s t ,a )] } .
To simplify the exposition, we use x t,a to denote x s t ,a since it only depends on the round index t in most bandit problems, and we assume A =
[ K ] .
In linear contextual bandits, the reward function in (2.1) is assumed to have a linear structure r ( x s,a ) =
x (cid:62) s,a θ ∗ for some unknown weight vector θ ∗ ∈ R d .
One provably sample efﬁcient algorithm for linear contextual bandits is Linear Upper Conﬁdence Bound (LinUCB)
(Chu et al., 2011) or Optimism in the Face of Uncertainty Linear bandit algorithm (OFUL) (Abbasi-Yadkori et al., 2011).
Speciﬁcally, at each round t , LinUCB chooses the action a t = argmax a ∈ [ K ] { x (cid:62) t,a θ t + α t (cid:107) x t,a (cid:107) A − 1 t } , where θ t is a point estimate of θ ∗ , A t =
λ I + (cid:80) ti =
1 x i,a i x (cid:62) i,a i with some λ >is a matrix deﬁned based on the historical context-arm pairs, and α t > 0 is a tuning parameter that controls the exploration rate in LinUCB.
2. D EEP NEURAL NETWORKS
In this paper, we use f ( x ) to denote a neural network with input data x ∈ R d .
Let L be the number of hidden layers and W l ∈ R m l × m l − 1 be the weight matrices in the l -th layer, where l = 1 , . . .
, L , m 1 = . . .
= m
L − 1 = m and m 0 = m L = d .
Then a L -hidden layer neural network is deﬁned as
[Equation]
where σ l is an activation function and θ ∗ ∈ R
d is the weight of the output layer.
To simplify the presentation, we will assume σ 1 = σ 2 = . . .
=
σ L = σ is the ReLU activation function, i.e., σ ( x ) = max { 0 , x } for x ∈ R .
We denote w = ( vec ( W 1 ) (cid:62) , . .
, vec ( W L ) (cid:62) )
(cid:62) , which is the concatenation of the vectorized weight parameters of all hidden layers of the neural network.
We also write f ( x ; θ ∗ , w ) =
f ( x ) in order to explicitly specify the weight parameters of neural network f .
It is easy to show that the dimension p of vector w satisﬁes p =
( L − 2) m
+ 2 md .
To simplify the notation, we deﬁne φ ( x ; w ) as the output of the L -th hidden layer of neural network f .
[Equation]
Note that φ ( x ; w ) itself can also be viewed as a neural network with vector-valued outputs.
D EEP R EPRESENTATION AND S HALLOW E XPLORATION 3. H IGH - LEVEL I DEA OF THE P ROPOSED A LGORITHM
The linear parametric form in linear contextual bandits might produce biased estimates of the reward due to the lack of representation power (Snoek et al., 2015;
Riquelme et al., 2018).
In contrast, it is well known that deep neural networks are powerful enough to approximate an arbitrary function (Cybenko, 1989).
Therefore, a natural extension of linear contextual bandits is to use a deep neural network to approximate the reward generating function r ( · ) .
Nonetheless, DNNs usually have a prohibitively large dimension for weight parameters, which makes the exploration in neural networks based UCB algorithm inefﬁcient (Kveton et al., 2020;
Zhou et al., 2020).
In this work, we study a neural contextual bandit algorithm, where the hidden layers of a deep neural network are used to represent the features and the exploration is only performed in the last layer of the neural network.
In particular, for any arm feature vector x , we use (cid:104) θ ∗ , φ ( x ; w ) (cid:105) to approximate the unknown reward function r ( x ) , where φ ( x ; w ) deﬁned as in (2.4) is a neural network with weight w , and θ ∗ is a unknown weight parameter.
Note that we can also view (cid:104) θ ∗ , φ ( x ; w ) (cid:105) as a neural network with φ ( x ; w ) being the output of the last hidden layer and θ ∗ the weight parameter of the last (linear) layer.
Different from existing neural bandit algorithms, we only add a UCB bonus term involving the last layer instead of all the weight parameter of this large neural network.
This decoupling of the representation and the exploration achieves the best of both worlds: efﬁcient exploration of shallow (linear) models and high expressive power of deep models.
In what follows, we will describe a neural contextual bandit algorithm that uses the output of the last hidden layer of a neural network to transform the raw feature vectors ( deep representation ) and performs UCB-type exploration in the last layer of the neural network ( shallow exploration ).
Since the exploration is performed only in the last linear layer, we call this procedure Neural-LinUCB, which is displayed in Algorithm 1.
3. D ETAILED I MPLEMENTATION OF D EEP R EPRESENTATION AND S HALLOW E XPLORATION
Now we describe the details of Algorithm 1.
In round t , the agent receives an action set with raw features X t = { x t, 1 , . . .
, x t,K } .
Then the agent chooses an arm a t that maximizes the following upper conﬁdence bound:
[Equation]
where θ t − 1 is a point estimate of the unknown weight in the last layer, φ ( x ; w ) is deﬁned as in (2.4), w t − 1 is an estimate of all the weight parameters in the hidden layers of the neural network, α t >is the algorithmic parameter controlling the exploration, and A t deﬁned as follows.
[Equation]
and λ > 0 .
After pulling arm a t , the agent will observe a noisy reward (cid:98) r t := (cid:98) r ( x t,a t ) =
r ( x t,k )+ ξ t , where ξ t is an independent ν -subGaussian random noise for some ν > 0 and r ( · ) is an unknown reward function.
In this paper, we will interchangeably use notation (cid:98) r t to denote the reward received at the t -th step and an equivalent notation (cid:98) r ( x ) to express its dependence on the feature vector x .
Upon receiving the reward (cid:98) r t , the agent updates its estimate θ t of the output layer weight by using the same (cid:96) 2 -regularized least-squares estimate in linear contextual bandits (Abbasi-Yadkori et al., 2011).
In particular, we have θ t =
A − 1 t b t , where b t =
(cid:80) ti =1 (cid:98)
r i φ ( x i,a i ; w i − 1 ) .
To save the computation, the neural network φ ( · ; w t ) will be updated once every H steps.
Therefore, we have w ( q − 1) H
+1 = . . .
= w qH for q = 1 , 2 , . . . .
We call the time steps { ( q − 1) H +1 , . .
. , qH } an epoch with length H .
At time step t =
Hq , we will retrain the neural network based on all the historical data via Algorithm 2, which minimizes the following empirical loss function:
[Equation]
In practice, one can further save computational cost by only feeding data { x i,a i , (cid:98) r i , θ i } qHi =( q − 1) H + from the q -th epoch into Algorithm 2 to update the parameter w t , which does not hurt the performance since the historical information has been encoded into the estimate of θ i .
In this paper, we will perform the following gradient descent step w ( s ) q = w ( s − 1) q −
[Equation]
L q ( w ( s − 1) ) , for s = 1 , . . . , n , where w (0) q = w (0) is chosen as the same random initialization point.
We will discuss more about the initial point w (0) in the next paragraph.
Then Algorithm 2 outputs w ( n ) q
and we set it as the updated weight parameter w Hq +1 in Algorithm 1.
In the next round, the agent will receive another action set X t +1 with raw feature vectors and repeat the above steps to choose the sub-optimal arm and update estimation for contextual parameters.
Initialization: Recall that w is the collection of all hidden layer weight parameters of the neural network.
We will follow the same initialization scheme as used in Zhou et al.
(2020), where each entry of the weight matrices follows some Gaussian distribution.
Speciﬁcally, for any l ∈ { 1 , . . .
, L − 1 } , we set W l = (cid:20) W
W (cid:21) , where each entry of W follows distribution N (0 , 4 /m ) independently; for
W L , we set it as [ V − V ] , where each entry of V follows distribution N (0 , 2 /m ) independently.
Comparison with LinUCB and NeuralUCB:
Compared with linear contextual bandits in Section 2.1, Algorithm 1 has a distinct feature that it learns a deep neural network to obtain a deep representation of the raw data vectors and then performs UCB exploration.
This deep representation allows our algorithm to characterize more intrinsic and latent information about the raw data { x t,k } t ∈ [ T ] ,k ∈ [ K ] ⊂ R d .
However, the increased complexity of the feature mapping φ ( · ; w ) also introduces great hardness in training.
For instance, a recent work by Zhou et al.
(2020) also studied the neural contextual bandit problem, but different from (3.1), their algorithm (NeuralUCB) performs the UCB exploration on the entire network parameter space, which is R (cid:101) p + d , where (cid:101)
p
= m + md +
( L − 1) m 2 .
Note that in Zhou et al.
(2020), they need to compute the inverse of a matrix Z t ∈ R ( (cid:101) p + d ) × ( (cid:101) p + d ) , which is deﬁned in a similar way to the matrix
A t in our paper except that Z t is deﬁned based on the gradient of the network instead of the output of the last hidden layer as in (3.2).
In sharp contrast, A t in our paper is only of size d × d and thus is much more efﬁcient and practical in implementation, which will be seen from our experiments in later sections.
We note that there is also a similar algorithm to our Neural-LinUCB presented in Deshmukh et al.
(2020), where they studied the self-supervised learning loss in contextual bandits with neural network representation for computer vision problems.
However, no regret analysis has been provided.
When the feature mapping φ ( · ; w ) is an identity function, the problem reduces to linear contextual bandits where we directly use x t as the feature vector.
In this case, it is easy to see that Algorithmreduces to LinUCB (Chu et al., 2011) since we do not need to learn the representation parameter w .
Comparison with Neural-Linear: The high-level idea of decoupling the representation and exploration in our algorithm is also similar to that of the Neural-Linear algorithm (Riquelme et al., 2018;
Zahavy & Mannor, 2019), which trains a deep neural network to learn a representation of the raw feature vectors, and then uses a Bayesian linear regression to estimate the uncertainty in the bandit problem.
However, these two algorithms are signiﬁcantly different since Neural-Linear
(Riquelme et al., 2018) is a Thompson sampling
based algorithm that uses posterior sampling to estimate the weight parameter θ ∗ via Bayesian linear regression, whereas Neural-LinUCB adopts upper conﬁdence bound based techniques to estimate the weight θ ∗ .
Nevertheless, both algorithms share the same idea of deep representation and shallow exploration, and we view our Neural-LinUCB algorithm as one instantiation of the Neural-Linear scheme.
4: receive feature vectors
5: choose arm obtain
11: else 12:
13: end
14: end for 15: Output w
[Table]
M AIN R ESULTS
To analyze the regret bound of Algorithm 1, we ﬁrst lay down some important assumptions on the neural contextual bandit model.
Assumption 4.1.
For all i ≥ 1 and k ∈ [ K ] , we assume that (cid:107) x i,k (cid:107) 2 =
and its entries satisfy [ x i,k ] j
=
[ x i,k ] j + d/ 2 .
The assumption that (cid:107) x
i,k (cid:107) 2 = 1 is not essential and is only imposed for simplicity, which is also used in Zou & Gu (2019);
Zhou et al.
The condition on the entries of x i,k is also mild since otherwise we could always construct x (cid:48)
i,k =
[ x (cid:62) i,k , x (cid:62) i,k ] (cid:62) / √ 2 to replace it.
An implication of
Assumption 4.1 is that the initialization scheme in Algorithm 1 results in φ ( x i,k ; w (0) )
=
for all i ∈ [ T ] and k ∈ [ K ] .
We assume the following stability condition on the spectral norm of the neural network gradient:
Assumption 4.2.
There is a constant (cid:96) Lip > 0 such that it holds (cid:13)(cid:13) ∂ φ ∂ w ( x ; w 0 ) − ∂ φ ∂ w ( x (cid:48) ; w 0 ) (cid:13)(cid:13) 2 ≤ (cid:96) Lip (cid:107) x − x (cid:48)
(cid:107) 2 for all x , x (cid:48) ∈ { x i,k } i ∈
[ T ] ,k ∈
[ K ] .
The inequality in Assumption 4.2 resembles the Lipschitz condition on the gradient of the neural network.
However, it is essentially different from the smoothness condition since here the gradient is taken with respect to the neural network weights while the Lipschitz condition is imposed on the feature parameter x .
Similar conditions are widely made in nonconvex optimization (Wang et al., 2014; Balakrishnan et al., 2017; Xu et al., 2017), in the name of ﬁrst-order stability, which is essential to derive the convergence of alternating optimization algorithms.
Furthermore, Assumption 4.2 is only required on the TK training data points and a speciﬁc weight parameter w 0 .
Therefore, the condition will hold if the raw feature data lie in a certain subspace of R d .
We provided some further discussions in the supplementary material about this assumption for interested readers.
In order to analyze the regret bound of Algorithm 1, we need to characterize the properties of the deep neural network in (2.3) that is used to represent the feature vectors.
Following a recent line of research (Jacot et al., 2018; Cao & Gu, 2019a; Arora et al., 2019b; Zhou et al., 2020), we deﬁne the covariance between two data point x , y ∈ R d as follows.
[Equation]
where ( u, v ) ∼ N ( 0 , Λ ( l − 1) ( x , y )) , and ˙ σ ( · ) is the derivative of activation function.
We denote the neural tangent kernel (NTK) matrix H ∈ R TK × TK based on all feature vectors { x t,k } t ∈ [ T ] ,k ∈
[ K ] .
Renumbering { x t,k } t ∈
[ T ] ,k ∈
[ K ] as { x i } i =1 ,...,TK , then each entry H ij is deﬁned as
[Equation]
for all i, j ∈ [ TK ] .
Based on the above deﬁnition, we impose the following assumption on H .
Assumption 4.3.
The neural tangent kernel deﬁned in (4.2) is positive deﬁnite, i.e., λ min ( H ) ≥ λfor some constant λ 0 > 0 .
Assumption 4.3 essentially requires the neural tangent kernel matrix H to be non-singular, which is a mild condition and also imposed in other related work (Du et al., 2019a; Arora et al., 2019b; Cao & Gu, 2019a;
Zhou et al., 2020).
Moreover, it is shown that Assumption 4.3 can be easily derived from Assumption 4.1 for two-layer ReLU networks (Oymak & Soltanolkotabi, 2020; Zou & Gu, 2019).
Therefore, Assumption 4.3 is mild or even negligible given the non-degeneration assumption on the feature vectors.
Also note that matrix H is only deﬁned based on layers l = 1 , . . .
, L of the neural network, and does not depend on the output layer θ .
It is easy to extend the deﬁnition of H to the NTK matrix deﬁned on all layers including the output layer θ , which would also be positive deﬁnite by Assumption 4.3 and the recursion in (4.2).
Before we present the regret analysis of the neural contextual bandit, we need to modify the regret deﬁned in (2.2) to account for the randomness of the neural network initialization.
For a ﬁxed time horizon T , we deﬁne the regret of Algorithm 1 as follows.
[Equation]
where the expectation is taken over the randomness of the reward noise.
Note that R T deﬁned in (4.3) is still a random variable since the initialization of Algorithm 2 is randomly generated.
Now we are going to present the regret bound of the proposed algorithm.
Theorem 4.4.
Suppose Assumptions 4.1, 4.2 and 4.3 hold.
Assume that (cid:107) θ ∗ (cid:107) 2 ≤ M for some positive constant M > 0 .
For any δ ∈ (0 , 1) , let us choose α t in Neural-LinUCB as
[Equation]
We choose the step size η q of Algorithm 2 as η q ≤ C 0 (cid:0) d 2 mnT 5 .
5 L 6 log( TK/δ ) (cid:1) − 1 and the width of the neural network satisﬁes m = poly ( L, d, 1 /δ, H, log( TK/δ )) .
With probability at least 1 − δ over the randomness of the initialization of the neural network, it holds that
[Equation]
where constants { C i } i =0 , 1 , 2 are independent of the problem, r = ( r ( x 1 ) , r ( x 2 ) , . .
, r ( x TK )) (cid:62) ∈
R TK and (cid:101) r
=
( f ( x 1 ; θ 0 , w 0 ) , . .
f ( x TK ; θ T − 1 , w T − 1 )) (cid:62) ∈ R TK , and (cid:107) r (cid:107)
A = √ r (cid:62) Ar .
Remark 4.5.
Theorem 4.4 shows that the regret of Algorithm 1 can be bounded by two parts: the ﬁrst part is of order (cid:101) O ( √ T ) , which resembles the regret bound of linear contextual bandits (AbbasiYadkori et al., 2011); the second part is of order (cid:101) O ( m − 1 / 6 T (cid:112) ( r − (cid:101) r ) (cid:62)
H − 1 ( r − (cid:101) r )) , which depends on the estimation error of the neural network f for the reward generating function r and the neural tangent kernel H .
It is worth noting that our theoretical analysis depends on the reward structure assumption that r ( · ) =
[Equation]
, ψ ( · ) (cid:105) .
However, the linear structure between θ ∗ and ψ ( · ) is not essential.
As long as the deep representation of the feature vector and the uncertainty weight parameter can be decoupled,
Algorithm 1 can be easily extended to settings with milder assumptions on the reward structure such as generalized linear models (Sarkar, 1991; Filippi et al., 2010; Li et al., 2017; Kveton et al., 2020).
For more general bandit models where no assumption is imposed to the reward generating function, it is still unclear whether the decoupled deep representation and shallow exploration would work especially in cases a thorough exploration may be needed.
Based on the result in Theorem 4.4, we can easily verify the following conclusion:
Corollary 4.6.
Under the same conditions of Theorem 4.4, if we choose a sufﬁciently overparameterized neural network mapping φ ( · ) such that m ≥ T 3 , then the regret of Algorithm 1 is
[Equation]
Remark 4.7.
For the ease of presentation, let us denote E := (cid:107) r − (cid:101) r (cid:107)
H − 1 .
If we have E = O (1) , the total regret in Theorem 4.4 becomes (cid:101) O ( √ T ) which matches the regret of linear contextual bandits (Abbasi-Yadkori et al., 2011).
We remark that there is a similar assumption in Zhou et al.
(2020) where they assume that r (cid:62) H − 1 r can be upper bounded by a constant.
They show that this term can be bounded by the RKHS norm of r if it belongs to the RKHS induced by the neural tangent kernel (Arora et al., 2019a;b; Lee et al., 2019).
In addition, E here is the difference between the true reward function and the neural network function, which can also be small if the deep neural network function well approximates the reward generating function r ( · ) .
E XPERIMENTS
In this section, we provide empirical evaluations of Neural-LinUCB on real-world datasets.
As we have discussed in Section 3, Neural-LinUCB could be viewed as an instantiation of the NeuralLinear scheme studied in Riquelme et al.
(2018) except that we use the UCB exploration instead of the posterior sampling exploration therein.
Note that there has been an extensive comparison (Riquelme et al., 2018) of the Neural-Linear methods with many other baselines such as greedy algorithms, Variational Inference, Expectation-Propagation, Bayesian Non-parametrics
and so on.
Therefore, we do not seek a thorough empirical comparison of Neural-LinUCB with all existing bandits algorithms.
In this experiment, we only aim to validate the advantages of our algorithm over the following baselines: (1) Neural-Linear (Riquelme et al., 2018);
(2) LinUCB (Chu et al., 2011), which does not have a deep representation of the feature vectors; (3) NeuralUCB (Zhou et al., 2020), and (4) NeuralTS
(Zhang et al., 2020) which perform UCB/TS exploration on all the parameters of the neural network.
All numerical experiments were run on a workstation with Intel(R)
Xeon(R)
CPU E5-2637 v4 @
3.50GHz.
Datasets: we evaluate the performances of all algorithms on bandit problems created from realworld data.
Speciﬁcally, following the experimental setting in Zhou et al.
(2020),we use datasets (Shuttle) Statlog , Magic and Covertype from UCI machine learning repository (Dua & Graff, 2017), and the MINST dataset from LeCun et al.
The details of these datasets are presented in Table 1.
In Table 1, each instance represents a feature vector x ∈ R d that is associated with one of the K arms, and dimension d is the number of attributes in each instance.
[Table]
[Figure]
Implementations: for LinUCB, we follow the setting in Li et al.
(2010) to use disjoint models for different arms.
For neural network based algorithms, we use a ReLU neural network deﬁned as in (2.3) with L = 2 and m = 100 for the UCI datasets ( Statlog , Magic , Covertype ).
Thus the neural network weights are W 1 ∈ R m × d , W 2 ∈ R k × m , and θ ∈ R k respectively, where k = 100 , m = 100 , and d is the dimension of features in the corresponding task.
Since the problem size of the MNIST dataset is larger, inspired by Hinton & Salakhutdinov (2006), we use a deeper NN and set L = 3 , k = 100 and m = 100 , with weights W 1 ∈ R m × d , W 2 ∈ R m × m , W 3 ∈ R k × m , and θ ∈ R k .
We set the time horizon T = 15 , 000 , which is the total number of rounds for each algorithm on each dataset.
We use stochastic gradient decent to optimize the network weights, with a step size η q = 1e-5 and maximum iteration number n = 1 , 000 .
To speed up the training process, the network parameter w is updated every H = 100 rounds starting from round 2000.
We also apply early stopping when the loss difference of two consecutive iterations is smaller than a threshold of 1e-6.
We set λ
= 1 and α t = 0 .
02 for all algorithms, t ∈
[ T ] .
For NeuralUCB and NeuralTS, since it is computationally unaffordable to perform the original UCB exploration as displayed in Zhou et al.
(2020), we follow their experimental setting to replace the matrix
Z t ∈ R ( d + (cid:101) p ) ×
( d + (cid:101) p ) in their papers with its diagonal matrix.
Results: we plot the cumulative regret of all algorithms versus round in Figures 1(a), 1(b) and 1(c) for UCI datasets and in Figure 1(d) for MNIST.
The results are reported based on the average ofrepetitions over different random shufﬂes of the datasets.
It can be seen that algorithms based on neural network representations (NeuralUCB, NeuralTS, Neural-Linear and Neural-LinUCB) consistently outperform the linear contextual bandit method LinUCB, which shows that linear models may lack representation power and ﬁnd biased estimates for the underlying reward generating function.
Furthermore, our proposed Neural-LinUCB achieves a comparable regret with NeuralUCB in all experiments despite the fact that our algorithm only explores in the output layer of the neural network, which is more computationally efﬁcient as we will show in the sequel.
The results in our experiment are well aligned with our theory that deep representation and shallow exploration are sufﬁcient to guarantee a good performance of neural contextual bandit algorithms, which is also consistent with the ﬁndings in existing literature (Riquelme et al., 2018)
that decoupling the representation learning and uncertainty estimation improves the performance.
We also conducted experiments to study the effects of different widths of deep neural networks on the regret performance and to show the computational efﬁciency of Neural-LinUCB compared with existing neural bandit algorithms.
Due to the space limit, we defer the results to Appendix A.
C ONCLUSIONS
In this paper, we propose a new neural contextual bandit algorithm called Neural-LinUCB, which uses the hidden layers of a ReLU neural network as a deep representation of the raw feature vectors and performs UCB type exploration on the last layer of the neural network.
By incorporating techniques in liner contextual bandits and neural tangent kernels, we prove that the proposed algorithm achieves a sublinear regret when the width of the network is sufﬁciently large.
This is the ﬁrst regret analysis of neural contextual bandit algorithms with deep representation and shallow exploration, which have been observed in practice to work well on many benchmark bandit problems (Riquelme et al., 2018).
We also conducted experiments on real-world datasets to demonstrate the advantage of the proposed algorithm over LinUCB and existing neural contextual bandit algorithms. A CKNOWLEDGEMENTS
We thank the anonymous reviewers for their helpful comments.
PX and QG are partially supported by the National Science Foundation CAREER Award 1906169 and IIS-1904183.
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies
A A DDITIONAL E XPERIMENTAL R ESULTS
In this section, we provide more experimental results that are omitted in Section 5 due to space limit.
A. C OMPUTATIONAL E FFICIENCY OF N EURAL -L IN UCB
Throughout the experiments, our Neural-LinUCB algorithm is much more computationally efﬁcient than NeuralUCB since we only perform the UCB exploration on the last layer of the neural network, where the dimension is much lower.
In speciﬁc, on the Statlog dataset, it takes on average 1 .seconds for NeuralUCB to ﬁnish 100 rounds (one epoch in Algorithm 1) and achieve the regret in
Figure 1(a), while it only takes 0 .
58 seconds for Neural-LinUCB to ﬁnish 100 rounds and achieve the comparable or even better regret in Figure 1(a).
On the Magic dataset, the average runtimes for 100 rounds of NeuralUCB and Neural-LinUCB are 1 .
32 seconds and 0 .
81 seconds respectively.
On the Covertype dataset, the runtimes of NeuralUCB and Neural-LinUCB are 1 .
02 seconds and 0 .
66 seconds respectively.
And on the MNIST dataset, the average runtimes for 100 rounds of
NeuralUCB and Neural-LinUCB are 4 .
67 seconds and 1 . 29 seconds respectively.
For practical applications in the real-world with larger problem sizes, we believe that the improvement of our algorithm in terms of the computational efﬁciency will be more pronounced.
As we discussed in Section 5 and in the above paragraph as well, the computational efﬁciency of
Neural-LinUCB mainly stems from the design of shallow exploration.
This is because in UCB based bandit algorithms we need to compute the inverse of matrix A at every time step for arm selection (Line 5 of Algorithm 1).
Due to the large width of the neural network used in practice, the arm selection operation could be rather time consuming.
However, the neural network weight can be updated periodically (i.e., in our paper it is only updated every H steps).
To validate our analysis on computational efﬁciency, we further studied the time proﬁling of the experiments conducted on
MNIST to compared our proposed algorithm with NeuralUCB in more details.
[Table]
In particular, the setting is the same as that in Section 5 for MNIST experiments.
We record the time cost of the most expensive two subroutines: (1) the operation of arm selection (Line 5 in
Algorithm 1); and (2) the operation of updating the neural network weights (Line 9 in Algorithm 1), for H = 100 rounds.
The time cost is presented in Table 2.
For Neural-LinUCB, the arm selection operation takes about 0 .
28 seconds (this is 21 .
71% of the total time cost by the algorithm in these H = 100 rounds), among which the matrix inverse step only takes 0 .
17 seconds.
For
NeuralUCB, the arm selection operation takes 3 .
60 seconds (this is 77 .
19% of the total time time cost by NeuralUCB for H = 100 rounds).
Therefore, the operation of arm selection in NeuralUCB is much (almost 13 times) more time consuming than that in Neural-LinUCB.
Moreover, since the UCB matrix Z t in NeuralUCB is deﬁned as ∇ f ( x ; w ) ∇ f ( x ; w ) (cid:62) , it needs to compute the gradients via back-propagation ( 0 . 93 seconds) and compute the matrix inverse ( 1 . 54 seconds), while our Neural-LinUCB algorithm only needs to compute the matrix inverse of a small matrix ( 0 .seconds).
To summarize, our method is much more computationally efﬁcient.
A. I MPACT OF L ARGE W IDTHS
Note that the requirement of width m in our Theorem 4.4 is extremely high.
On one hand, our theory may be too conservative since the current understanding of deep learning is still very limited in the ﬁeld.
We believe our work is a good starting point towards understanding the behavior of deep bandits algorithms.
On the other hand, we would also like to investigate the impact of mild overparameterization on the regret performance of Neural-LinUCB in practice.
Therefore, we conducted additional experiments on the Statlog dataset with wider neural networks.
In particular, the neural
[Figure]
network parameters are listed as follows
[Equation]
where L is the depth, k = 100 , d is the feature dimensions, and m is the width.
We conducted experiments for the following settings: (1) L = 2 , m = 100 , and thus the hidden layer width is (100) ; (2) L = 2 , m = 1000 , and thus the hidden layer width is (1000) ; (3) L = 2 , m = 10000 , and thus the hidden layer width is (10000) ; (4) L = 2 , and the hidden layer width is (50000) ;
(5) L = 3 , m = 10000 , and thus the hidden layer width is (10000 , 10000) ; and (6) L = 4 , m = 10000 , and thus the hidden layer width is (10000 , 10000 , 10000) .
The results are plotted in Figure A.2.
We observe that the performance of our Neural-LinUCB algorithm is not negatively impacted by the width of the neural network.
In fact, Figure A.2 shows improved performance of Neural-LinUCB when the total number of hidden nodes increases.
This is consistent to the observations in Zhang et al.
(2017) that an overparameterized neural network trained by gradient descent does not necessarily lead to overﬁtting and also aligns with our Theorem 4.4 that the regret bound of Neural-LinUCB decreases as the width m increases.
B M ORE D ISCUSSIONS ON A SSUMPTION 4.
In this section, we are going to show that Assumption 4.2 could be satisﬁed as long as the feature vectors { x } lie in a begin subspace of R d .
Let us start with the case that φ : R d → R m is a two-layer ReLU neural network with vector output.
In particular, we deﬁne φ ( x ; w ) as follows φ ( x ; w )
=
σ ( W 2 σ ( W 1 x )) , where w = ( vec ( W 1 ) , vec ( W 2 )) (cid:62) , W 1 ∈ R m × d , W 2 ∈ R d × m , and σ is the ReLU activation function applied elementwise.
We use u (cid:62) i to denote the i -th row of W 1 and thus W 1 = ( u 1 , . . .
, u m ) (cid:62) , where u i ∈ R d , ∀ i ∈ [ m ] .
Similarly, we have W 2 = ( v 1 , . .
. , v d ) (cid:62) , where v j ∈ R m is the j -th row of W 2 , ∀ j ∈
[ d ] .
Let us denote h as the vector σ ( W 1 x ) .
We thus obtain
[Equation]
We use φ l ( x ; w ) to denote the l -th entry of vector φ ( x ; w ) , for any l ∈ [ d ] .
Then it holds that
[Equation]
m x ≥ 0 } x (cid:62) (cid:3) , for all l ∈ [ d ] , where v i 1 is the i -th element in v 1 , i ∈ [ m ] .
This further implies that
[Equation]
Similarly, we can compute the gradient of φ ( x ; w ) with respect to W 2 .
In particular, we have
[Equation]
Therefore, the gradient of φ ( x ; w ) with respect to W 2 is
[Equation]
Lastly, we have
[Equation]
Therefore, for any two feature vectors x and x (cid:48) from { x i,k } i ∈ [ T ] ,k ∈ [ K ] , if many nodes in the initial neural network φ ( x ; w 0 ) are activated or deactivated at the same time for both x and x (cid:48) , then the spectral norm of the matrix ∂ φ ( x ; w 0 ) ∂ w − ∂ φ
[Equation]
) ∂ w would satisfy the condition in Assumption 4.2.
A more thorough study of this stability condition is out of the scope of this paper, though it would be an interesting open direction in the theory of deep neural networks.
C P ROOF OF THE M AIN R ESULTS
In this section, we provide the proof of the regret bound for Neural-LinUCB.
Recall that in neural contextual bandits, we do not assume a speciﬁc formulation of the underlying reward generating function r ( · ) .
Instead, we use deep neural networks deﬁned in Section 2.2 to approximate r ( · ) .
We will ﬁrst show that the reward generating function r ( · ) can be approximated by the local linearization of the overparameterized neural network near the initialization weight w (0) .
In particular, we denote the gradient of φ ( x ; w ) with respect to w by g ( x ; w ) , namely,
[Equation]
which is a matrix in R d × p .
We deﬁne φ j ( x ; w ) to be the j -th entry of vector φ ( x ; w ) , for any j ∈ [ d ] .
Then, we can prove the following lemma.
Lemma C.1.
Suppose Assumptions 4.3 hold.
Then there exists
w ∗ ∈
R p such that (cid:107) w ∗ − w (0) (cid:107) 2 ≤ 1 / √ m (cid:112) ( r − (cid:101) r ) (cid:62) H − 1 ( r
[Equation]
(cid:101) r ) and it holds that
[Equation]
for all k ∈
[ K ] and t = 1 , . . .
, T .
Lemma C.1 implies that the reward generating function r ( · ) at points { x i,k } i ∈
[ T ] ,k ∈
[ K ] can be approximated by a linear function around the initial point w (0) .
Note that a similar lemma is also proved in Zhou et al.
(2020) for NeuralUCB.
The next lemma shows the upper bounds of the output of the neural network φ and its gradient.
Lemma C.2.
Suppose Assumptions 4.1 and 4.3 hold.
For any round index t ∈ [ T ] , suppose it is in the q -th epoch of Algorithm 2, i.e., t = ( q − 1) H
+ i for some i ∈
[ H ] .
If the step size η q in
Algorithm 2 satisﬁes
[Equation]
and the width of the neural network satisﬁes
[Equation]
then, with probability at least 1 − δ we have
[Equation]
for all t ∈ [ T ] , k ∈ [ K ] , where the neural network φ is deﬁned in (2.4) and its gradient is deﬁned in (C.1).
The next lemma shows that the neural network φ ( x ; w ) is close to a linear function in terms of the weight w parameter around a small neighborhood of the initialization point w (0) .
Lemma C.3 (Theorems 5 in Cao & Gu (2019b)) .
Let w , w (cid:48) be in the neighborhood of w 0 , i.e., w , w (cid:48)
∈ B ( w 0 , ω ) for some ω > 0 .
Consider the neural network deﬁned in (2.4), if the width m and the radius ω of the neighborhood satisfy
[Equation]
then for all x ∈ { x t,k } t ∈ [ T ] ,k ∈
[ K ] , with probability at least 1 − δ it holds that
[Equation]
, where (cid:98) φ j ( x ; w ) is the linearization of φ j ( x ; w ) at w (cid:48) deﬁned as follow:
[Equation]
Similar results on the local linearization of an overparameterized neural network are also presented in Allen-Zhu et al.
(2019b)
; Cao & Gu (2019b).
For the output layer θ ∗ , we perform a UCB type exploration and thus we need to characterize the uncertainty of the estimation.
The next lemma shows the conﬁdence bound of the estimate θ t in
Algorithm 1.
Lemma C.4.
Suppose Assumption and 4.3 hold.
For any δ ∈ (0 , 1) , with probability at least 1 − δ , the distance between the estimated weight vector θ t by Algorithm 1 and θ ∗ can be bounded as follows:
[Equation]
for any t ∈ [ T ] .
Note that the conﬁdence bound in Lemma C.4 is different from the standard result for linear contextual bandits in Abbasi-Yadkori et al.
The additional term on the left hand side of the conﬁdence bound is due to the bias caused by the representation learning using a deep neural network.
To deal with this extra term, we need the following technical lemma.
Lemma C.5.
Assume that A t
= λ
I + (cid:80) ts =1 φ s φ (cid:62) s ,
where φ t ∈ R d and (cid:107) φ t (cid:107) 2 ≤ G for all t
≥and some constants λ, G > 0 .
Let { ζ t } t =1 ,... be a real-value sequence such that | ζ t | ≤ U for some constant U > 0 .
Then we have
[Equation]
The next lemma provides some standard bounds on the feature matrix A t , which is a combination of Lemma 10 and Lemma 11 in Abbasi-Yadkori et al.
Lemma C.6.
Let { x t } ∞ t =1 be a sequence in R d and λ > 0 .
Suppose (cid:107) x t (cid:107) 2 ≤ G and λ ≥ max { 1 , G 2 } for some G > 0 .
Let A t
= λ
I + (cid:80) ts =1 x t x (cid:62) t .
Then we have det( A t ) ≤ ( λ + tG 2 /d )
d , and
[Equation]
Now we are ready to prove the regret bound of Algorithm 1.
Proof of Theorem 4.4.
For a time horizon T , without loss of generality, we assume T = QH for some epoch number Q .
By the deﬁnition of regret in (4.3), we have
[Equation]
Note that for the simplicity of presentation, we omit the conditional expectation notation of w (0) in the rest of the proof when the context is clear.
In the second equation, we rewrite the time index t =
qH + i as the i -th iteration in the q -th epoch.
By the deﬁnition in (2.1), we have E [ (cid:98) r ( x t,k ) | x t,k ]
=
r
( x t,k ) for all t ∈
[ T ] and k ∈ K .
Based on the linearization of reward generating function, we can decompose the instaneous regret into different parts and upper bound them individually.
In particular, by Lemma C.1, there exists a vector w ∗ ∈ R p such that we can write the expectation of the reward generating function as a linear function.
Then it holds that
[Equation]
The ﬁrst term in (C.4) can be easily bounded using the ﬁrst order stability in Assumption 4.2 and the distance between w ∗ and w (0) in Lemma C.1.
The second term in (C.4) is related to the optimistic rule of choosing arms in Line 5 of Algorithm 1, which can be bounded using the same technique for
LinUCB (Abbasi-Yadkori et al., 2011).
For the last term in (C.4), we need to prove that the estimate of weight parameter θ t − 1 lies in a conﬁdence ball centered at θ ∗ .
For the ease of notation, we deﬁne
[Equation]
Then the second term in (C.4) can be bounded in the following way:
[Equation]
C.
[Equation] [Equation]
where in the ﬁrst inequality we used the deﬁnition of upper conﬁdence bound in Algorithm 1 and the second inequality is due to Assumption 4.2.
Recall the linearization of φ j in Lemma C.3, we have
[Equation]
Note that by the initialization, we have φ ( x ; w 0 )
=
for any x ∈ R d .
Thus, it holds that
[Equation]
which immediately implies that
[Equation]
where the second inequality is due to
Lemma C
[Equation]
3 and Assumption
[Equation]
Therefore
[Equation]
the instaneous regret can be further upper bounded as follows.
[Equation]
(C.10)
By Assumption 4.1 we have (cid:107) x t,a ∗ t − x t,a t (cid:107) 2 ≤ 2 .
By Lemma C.1 and Lemma C.2, we have
[Equation]
In addition, since the entries of θ 0 are i.i.d.
generated from N (0 , 1 /d ) , we have (cid:107) θ 0 (cid:107) 2 ≤ 2(2 + (cid:112) d − 1 log(1 /δ )) with probability at least 1 − δ for any δ > 0 .
By Lemma C.2, we have (cid:107) g ( x t,a t ; w (0) )
(cid:107) F ≤ C 1 √ dm .
Therefore,
[Equation]
Then, by the deﬁnition of M t in (C.5) and Lemma C.5, we have
[Equation]
C.
[Equation]
2 back into (C.10) further yields r
[Equation]
Note that we have ω
=
O
[Equation]
m
[Equation]
) by Lemma C.1.
Therefore, the regret of the
Neural-LinUCB is
[Equation]
where the ﬁrst inequality is due to Cauchy’s inequality, the second inequality comes from the upper bound of α t in Lemma C.4 and Lemma C.6.
{ C j } j =0 ,..., 6 are absolute constants that are independent of problem parameters.
Proof of Corollary 4.6.
It directly follows the result in Theorem 4.4.
D P ROOF OF T ECHNICAL L EMMAS
In this section, we provide the proof of technical lemmas used in the regret analysis of Algorithm 1.
D. P ROOF OF L EMMA C.
Before we prove the lemma, we ﬁrst present some notations and a supporting lemma for simpliﬁcation.
Let β =
( θ (cid:62) , w (cid:62) )
(cid:62) ∈ R
d + p be the concatenation of the exploration parameter and the hidden layer parameter of the neural network f
( x ; β ) = θ (cid:62) φ ( x ; w ) .
Note that for any input data vector x ∈ R d , we have
[Equation]
where g ( x ; w ) is the partial gradient of φ ( x ; w ) with respect to w deﬁned in (C.1), which is a matrix in R d × p .
Similar to (4.2), we deﬁne H L +1 to be the neural tangent kernel matrix based on all
L + 1 layers of the neural network f ( x ; β ) .
Note that by the deﬁnition of H in (4.2), we must have
H L +1 =
H + B for some positive deﬁnite matrix B ∈ R TK × TK .
The following lemma shows that the NTK matrix is close to the matrix deﬁned based on the gradients of the neural network on TK data points.
Lemma D.1 (Theorem 3.1 in Arora et al. (2019b)) .
Let (cid:15) > 0 and δ ∈ (0 , 1) .
Suppose the activation function in (2.3) is ReLU, i.e., σ l ( x ) = max(0 , x ) , and the width of the neural network satisﬁes
[Equation] [Equation]
Then for any x , x (cid:48) ∈ R d with (cid:107) x (cid:107) 2 = (cid:107) x (cid:48) (cid:107) 2 = 1 , with probability at least 1 − δ over the randomness of the initialization of the network weight w it holds that
[Equation]
Note that in the above lemma, there is a factor 1 / √ m before the gradient.
This is due to the additional √ m factor in the deﬁnition of the neural network in (2.3), which ensures the value of the neural network function evaluated at the initialization is of the order O (1) .
Proof of Lemma C.1.
Recall that we renumbered the feature vectors { x t,k } t ∈
[ T ] ,k ∈
[ K ] for all arms from round 1 to round T as { x i } i =1 ,...,TK .
By concatenating the gradients at different inputs and the gradient in (D.1), we deﬁne Ψ ∈ R TK × ( d + p ) as follows.
[Equation]
By Applying Lemma D.1, we know with probability at least 1 − δ it holds that
[Equation]
for any (cid:15) > 0 as long as the width m satisﬁes the condition in (D.2).
By applying union bound over all data points { x 1 , . . .
, x t , . . .
, x TK } , we further have
[Equation]
Note that H is the neural tangent kernel (NTK) matrix deﬁned in (4.2) and H L +1 is the NTK matrix deﬁned based on all L + 1 layers.
By Assumption 4.3, H has a minimum eigenvalue λ 0 > 0 , which is deﬁned based on the ﬁrst L layers of f .
Furthermore, by the deﬁnition of NTK matrix in (4.2), we know that H L +
= H + B for some semi-positive deﬁnite matrix B .
Therefore, the
NTK matrix H L +1 deﬁned based on all L + 1 layers is also positive deﬁnite and its minimum eigenvalue is lower bounded by λ 0 .
Let (cid:15) =
λ 0 / (2 TK ) .
By triangle equality we have ΨΨ (cid:62) (cid:31)
H L +1 − (cid:107) ΨΨ
[Equation]
−
[Equation]
−
[Equation]
ΨΨ
[Equation]
− H
[Equation]
(cid:107)
[Equation]
I (cid:31) H L +1 − λ 0 / 2 I (cid:31) 1 / 2 H L +1 , which means that Ψ is semi-deﬁnite positive and thus rank ( Ψ ) =
TK since m > TK .
We assume that Ψ can be decomposed as Ψ = PDQ (cid:62) , where P ∈ R TK × TK is the eigenvectors of ΨΨ (cid:62) and thus PP (cid:62)
=
I TK , D ∈ R TK × TK is a diagonal matrix with the square root of eigenvalues of ΨΨ (cid:62) , and Q (cid:62) ∈ R TK × ( d + p ) is the eigenvectors of Ψ (cid:62) Ψ and thus Q (cid:62) Q
= I TK .
We use Q 1 ∈ R d × TK and Q 2 ∈ R p × TK to denote the two blocks of Q such that Q (cid:62) =
[ Q (cid:62) 1 , Q (cid:62) 2 ] .
By deﬁnition, we have
[Equation]
Note that the minimum singular value of Q 1 ∈ R d × TK is zero since d is a ﬁxed number and
TK > d .
Therefore, it must hold that rank ( Q 2 ) =
TK and thus Q (cid:62) 2 Q 2 is positive deﬁnite.
Let r =
( r ( x 1 ) , . . .
, r ( x i ) , . . .
, r ( x TK )) (cid:62) ∈ R TK denote the vector of all possible rewards.
We further deﬁne G ∈ R TKd × p and Φ ∈ R TKd as follows
[Equation]
and Θ , Θ 0 ∈ R TK × TKd as follows
[Equation]
It can be veriﬁed that Ψ = PD [ Q (cid:62) 1 , Q (cid:62) 2 ] and PDQ (cid:62) 2 =
Θ 0 G .
Note that we have Q (cid:62) 2 Q 2 is positive deﬁnite by Assumption 4.3, which corresponds to the neural tangent kernel matrix deﬁned on the ﬁrst L layers.
Then we can deﬁne w ∗ as follows
[Equation]
We can verify that
[Equation]
On the other hand, we have
[Equation]
which completes the proof.
D. P ROOF OF L EMMA C.
Note that we can view the output of the last hidden layer φ ( x ; w ) deﬁned in (2.4) as a vector-output neural network with weight parameter w .
The following lemma shows that the output of the neural network φ is bounded at the initialization.
Lemma D.2 (Lemma 4.4 in Cao & Gu (2019b)) .
Let δ ∈ (0 , 1) , and the width of the neural network satisfy m ≥ C 0 L log( TKL/δ ) .
Then for all t ∈ [ T ] , k ∈ [ K ] and j ∈ [ d ] , we have | φ j ( x t,k ; w (0) )
| ≤ C 1 (cid:112) log( TK/δ ) with probability at least 1 − δ , where w (0) is the initialization of the neural network.
In addition, in a smaller neighborhood of the initialization, the gradient of the neural network φ is uniformly bounded.
Lemma D.3 (Lemma B.3 in Cao & Gu (2019b)) .
Let ω ≤ C
L − 6 (log m ) − 3 and w ∈ B ( w 0 , ω ) .
Then for all t ∈ [ T ] , k ∈ [ K ] and j ∈ [ d ] , the gradient of the neural network φ deﬁned in (2.4) satisﬁes (cid:107)∇
[Equation]
with probability at least
[Equation]
TKL
[Equation]
The next lemma provides an upper bound on the gradient of the squared loss function deﬁned in (3.3).
Note that our deﬁnition of the loss function is slightly different from that in Allen-Zhu et al.
(2019b)
due to the output layer θ i and thus there is an additional term on the upper bound of (cid:107) θ i (cid:107)for all i ∈
[ T ] .
Lemma D.4 (Theorem 3 in Allen-Zhu et al.
(2019b)) .
Let ω ≤ C 0 δ 3 / 2 / ( T 9 / 2 L 6 log 3 m ) .
For all w ∈ B ( w (0) , ω ) , with probability at least 1 − exp( − C 1 mω 2 / 3 L ) over the randomness of w (0) , it holds that
[Equation]
d
[Equation]
Proof of Lemma C.2.
Fix the epoch number q
and we omit it in the subscripts in the rest of the proof when no confusion arises.
Recall that w ( s ) is the s -th iterate in Algorithm 2.
Let δ > 0 be any constant.
Let ω be deﬁned as follows.
[Equation]
We will prove by induction that with probability at least 1 − δ the following statement holds for all s = 0 , 1 , . . . , n
[Equation]
First note that (D.7) holds trivially when s = 0 due to Lemma D.2.
Now we assume that (D.7) holds for all j = 0 , . . .
, s .
The loss function in (3.3) can be bounded as follows.
[Equation]
By the update rule of θ t , we have
[Equation]
where the inequality is due to Lemma C.
[Equation]
which combined with (D.7) immediately implies
[Equation]
Substituting (D.8) and (D.9) into the inequality in Lemma D.
[Equation]
we also have
[Equation]
Now we consider w ( s +1) .
By triangle inequality we have
[Equation]
where the last inequality is due to (D.10).
If we choose the step size η q in the q -th epoch such that
[Equation]
then we have (cid:107) w ( s +1)
q − w (0) (cid:107) 2 ≤ ω .
Note that the choice of m, ω satisﬁes the condition in
Lemma C.3.
Thus we know φ j ( x ; w ) is almost linear in w , which leads to |
[Equation]
where in the second inequality we used the induction hypothesis (D.7), Cauchy-Schwarz inequality and Lemma D.3, and the third inequality is due to (D.10).
Note that the deﬁnition of ω in (D.6) ensures that ω √ dm < 1 / 2 and ω 4 / 3 L 3 d − 1 / 2 √ m log m ≤ m − 1 / 6 T − 6 L − 5 d − 1 / 2 √ log m ≤ 1 /n as long as m ≥ n 6 .
Plugging these two upper bounds back into (D.13) ﬁnishes the proof of (D.7).
Note that for any t ∈ [ T ] , we have w t = w ( n ) q for some q = 1 , 2 , . . . .
Since we have w t ∈ B ( w , ω ) , the gradient g ( x ; w (0) ) can be directly bounded by Lemma D.3, which implies (cid:107) g ( x ; w (0) )
(cid:107) F ≤
C 6 √ dLm .
Applying (D.7) with s = n , we have the following bound of the neural network function φ ( x ; w ( n ) q ) =
φ ( x ; w t ) for all t in the q -th epoch
[Equation]
which completes the proof.
In this proof, { C j > 0 } j =0 ,..., 6 are constants independent of problem parameters.
D. P ROOF OF L EMMA C.
The following lemma characterizes the concentration property of self-normalized martingales.
Lemma D.5 (Theorem 1 in Abbasi-Yadkori et al.
Let { ξ } ∞ t =1 be a real-valued stochastic process and { x t } ∞ t =1 be a stochastic process in R d .
Let F t = σ ( x 1 , . .
, x t +1 , ξ − 1 , . . . , ξ t ) be a σ -algebra such that x t and ξ t are F t − 1 -measurable.
Let A t
= λ
I + (cid:80) ts =1 x s x (cid:62) s for some constant λ > 0 and S t
= (cid:80) ts =1 ξ s x i .
If we assume ξ t is ν -subGaussian conditional on F t − 1 , then for any η ∈ (0 , 1) , with probability at least 1 − δ , we have
[Equation]
Proof of Lemma C.4.
Let Φ t =
[ φ ( x 1 ,a 1 ; w 0 ) , . . .
, φ ( x t,a t ; w t − 1 )]
∈ R d × t be the collection of feature vectors of the chosen arms up to time t and (cid:98) r t = ( (cid:98) r 1 , . . .
, (cid:98) r t ) (cid:62) be the concatenation of all received rewards.
According to Algorithm 1, we have A t
=
λ
I + Φ t Φ (cid:62) t and thus
[Equation]
By Lemma C.1, the underlying reward generating function r
[Equation]
) | x t,a t ] can be rewritten as
[Equation]
By the deﬁnition of the reward in (2.1) we have (cid:98) r t =
r t + ξ t .
Therefore, it holds that
[Equation]
Note that A t is positive deﬁnite as long as λ > 0 .
Therefore (cid:107) · (cid:107) A t and (cid:107) · (cid:107)
A t are well deﬁned norms.
Then for any δ ∈ (0 , 1) by triangle inequality we have
[Equation]
holds with probability at least 1 − δ , where in the last inequality we used Lemma D.5 and the fact that
[Equation]
λ − 1 /
[Equation]
θ ∗ (cid:107)
[Equation]
λ
[Equation] [Equation]
M by Lemma C.1.
Plugging the deﬁnition of Φ t , Θ t and G t and apply Lemma C.6, we further have
[Equation]
where we used the fact that (cid:107) φ ( x ; w ) (cid:107) 2 ≤ C √ d log HK by Lemma C.2.
D. P ROOF OF L EMMA C.
We now prove the technical lemma that upper bounds (cid:107) A − 1 t
[Equation]
ts =1 φ s ζ s (cid:107) 2 .
Proof of Lemma C.5.
We ﬁrst construct auxiliary vectors (cid:101)
φ t ∈ R d +1 and matrices B t ∈
R ( d +1) ×
( d +1) for all t = 1 , . .
in the following way:
[Equation]
where 0 d ∈ R
d is an all-zero vector.
Then by deﬁnition we immediately have
[Equation]
For all s = 1 , 2 . . .
, let { β s,j } d +1 j =1 be the coefﬁcients of the decomposition of U − 1 ζ s (cid:101) φ s on the natural basis.
Speciﬁcally, let { e 1 , . . .
, e d +1 } be the natural basis of R d +1 such that the entries of e j are all zero except the j -th entry which equals 1 .
Then we have
[Equation]
We can conclude that | β s,j | ≤ 1 since | ζ s | ≤ U and (cid:107) (cid:101) φ s (cid:107) 2 ≤ 1 .
Moreover, it is easy to verify that (cid:107) (cid:101) φ t (cid:107) 2 = 1 for all t ≥ 1 .
Therefore, we have
[Equation]
) where the inequality is due to triangle inequality and the last equation is due to the deﬁnition of (cid:101) φ t and B t in (D.14).
For each j = 1 , . . .
, d + 1 , we have
[Equation]
−
[Equation]
Since we have | β s,j | ≤ 1 , it immediately implies
[Equation]
Further by the fact that (cid:107) A − 1 B (cid:107) 2 ≤ 1 for any A (cid:31) B (cid:23) 0 , combining the above results with (D.18) yields
[Equation]
Finally, substituting the above results into (D.17) and (D.15) we have
[Equation]
which completes the proof.