N EURAL C ONTEXTUAL B ANDITS WITH U PPER C ONFIDENCE B OUND -B ASED E XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the stochastic contextual bandit problem, where the reward is generated from an unknown bounded function with additive noise.
We propose the
NeuralUCB algorithm, which leverages the representation power of deep neural networks and uses a neural network-based random feature mapping to construct an upper conﬁdence bound (UCB) of reward for efﬁcient exploration.
We prove that, under mild assumptions, NeuralUCB achieves (cid:101) O ( √ T ) regret, where T is the number of rounds.
To the best of our knowledge, our algorithm is the ﬁrst neural network-based contextual bandit algorithm with near-optimal regret guarantee.
Preliminary experiment results on synthetic data corroborate our theory and shed light on potential applications of our algorithm to real-world problems
I NTRODUCTION
The stochastic contextual bandit problem has been extensively studied in machine learning (Bubeck and Cesa-Bianchi, 2012; Lattimore and Szepesv ´ ari, 2019): at round t ∈ { 1 , 2 , . . .
, T } , an agent is presented with a set of K actions, each of which is associated with a d -dimensional feature vector.
After choosing an action, the agent will receive a stochastic reward generated from some unknown distribution conditioned on the chosen action’s feature vector.
The goal of the agent is to maximize the expected cumulative rewards over T rounds.
Contextual bandit algorithms have been applied to many real-world applications, such as personalized recommendation, advertising and Web search (e.g., Agarwal et al., 2009; Li et al., 2010).
The most studied model in the literature is linear contextual bandits (Auer, 2002;
Abe et al., 2003;
Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011), which assumes that the expected reward at each round is a linear function of the feature vector.
Linear bandit algorithms have achieved great success in both theory and practice, such as news article recommendation (Li et al., 2010).
However, the linear-reward assumption often fails to hold exactly in practice, which motivates the study of nonlinear contextual bandits (e.g., Filippi et al., 2010; Srinivas et al., 2010; Bubeck et al., 2011; Valko et al., 2013).
However, they still require fairly strong assumptions on the reward function.
For instance, Filippi et al.
(2010) makes a generalized linear model assumption on the reward, Bubeck et al.
(2011) require it to have a
Lipschitz continuous property in a proper metric space, and Valko et al.
(2013) assume the reward function belongs to some Reproducing Kernel Hilbert Space (RKHS).
In order to overcome the above shortcomings, deep neural networks (DNNs) (Goodfellow et al., 2016) have been introduced to learn the underlying reward function in contextual bandit problem, thanks to their strong representation power.
Given the fact that DNNs enable the agent to make use of nonlinear models with less domain knowledge, existing work (Riquelme et al., 2018; Zahavy and
Mannor, 2019) focuses on the idea called neural-linear bandit .
More precisely, they use the ﬁrst
L − 1 layers of a DNN as a feature map, which transforms contexts from the raw input space to a low-dimensional space, usually with better representation and less frequent update.
Then they learn a linear exploration policy on top of the last hidden layer of the DNN with a more frequent update.
These attempts have achieved great empirical success.
However, none of these work provides a theoretical guarantee on the regret of the algorithms.
In this paper, we take the ﬁrst step towards provable efﬁcient contextual bandit algorithms based on deep neural networks.
Speciﬁcally, we propose a new algorithm, NeuralUCB, which uses a deep neural network to learn the underlying reward function.
At the core of the algorithm is an upper conﬁdence bound constructed by deep neural network-based random feature mappings.
Our regret analysis of NeuralUCB is built on recent results on optimization and generalization of deep neural networks (Jacot et al., 2018; Arora et al., 2019; Cao and Gu, 2019a).
While the main focus of our paper is mostly theoretical, we also carry out proof-of-concept experiments on synthetic data to validate the effectiveness of our proposed algorithm.
Our contributions are summarized as follows:
• We prove that, under mild assumptions, our algorithm is able to achieve a (cid:101)
O ( (cid:101) d √ T ) regret, where (cid:101) d is the effective dimension of a neural tangent kernel matrix and T is the number of rounds.
Our regret bound recovers the (cid:101) O ( d √ T ) regret for linear contextual bandit as a special case (AbbasiYadkori et al., 2011), where d is the dimension of context.
• We propose a neural contextual bandit algorithm using neural network-based exploration.
It can be regarded as an extension of existing linear bandit algorithms (Li et al., 2010; Abbasi-Yadkori et al., 2011), from linear reward functions to any bounded reward functions.
• We provide empirical evidence in several proof-of-concept experiments to demonstrate potential applications of our algorithm to real-world problems.
Notation: Scalars are denoted by lower case letters, vectors by lower case bold face letters, and matrices by upper case bold face letters.
For a positive integer k , [ k ] denotes { 1 , . . .
, k } .
For a vector θ ∈ R d , we denote its (cid:96) 2 norm by (cid:107) θ (cid:107) 2 = (cid:113)(cid:80) di
=1 θ 2 i and its j -th coordinate by [ θ ] j .
For a matrix A ∈ R d × d , we denote its spectral norm, Frobenius norm, and ( i, j ) -th entry by (cid:107) A (cid:107) 2 , (cid:107) A (cid:107) F , and [ A ] i,j , respectively.
We denote a sequence of vectors by { θ j } tj =1 , and similarly for matrices.
For two sequences { a n } and { b n } , we use a n = O ( b n ) to denote that there exists some constant
C > 0 such that a n ≤ Cb n , a n = Ω( b n ) to denote that there exists some constant C (cid:48) > 0 such that a n ≥ C (cid:48) b n .
In addition, we use (cid:101) O ( · ) to hide logarithmic factors.
We say a random variable
X is ν -sub-Gaussian if E exp( λ ( X − E X ))
≤ exp( λ 2 ν 2 / 2) for any λ > 0 .
R ELATED W ORK 2. C ONTEXTUAL B ANDITS
There is a line of extensive work on linear bandits (e.g., Auer, 2002; Abe et al., 2003; Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010; Li et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011).
For the setting with ﬁnitely many arms, Abe et al.
(2003) formalized the linear bandit setting and analyzed some of the earliest algorithms.
Auer (2002) proposed SupLinRel algorithm that achieves (cid:101) O √ dT regret.
Chu et al.
(2011) obtained the same regret with SupLinUCB that is based on
LinUCB (Li et al., 2010)
; the authors also provided a lower bound of Ω( √ dT ) .
For the other setting with inﬁnitely many arms, a few authors (Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010;
Abbasi-Yadkori et al., 2011) proposed algorithms that achieve (cid:101)
O
( d √ T ) regret, whereas an Ω( d √ T ) lower bound is given by Dani et al.
While most algorithms above are based on the idea of upper conﬁdence bounding, it is also possible to use proper randomization to achieve strong regret guarantees, such as Thompson sampling and reward perturbation (Thompson, 1933; Chapelle and Li, 2011; Agrawal and Goyal, 2013; Russo and
Van Roy, 2014; 2016; Kveton et al., 2019).
To deal with nonlinearity, generalized linear bandit has been considered, which assumes that the reward function can be written as a composition of a linear function and a link function.
In particular,
Filippi et al.
(2010) proposed a GLM-UCB algorithm, which attains (cid:101) O
( d √ T ) regret.
Li et al.
(2017) proposed SupCB-GLM for generalized contextual bandit problems and showed a (cid:101) O ( √ dT ) regret that matches the lower bound.
Jun et al.
(2017) studied how to scale up algorithms for GLM bandits.
A few authors have also explored more general nonlinear bandits without making strong modeling assumptions.
One line of work is variants of expert learning algorithms (Auer et al., 2002), which typically have a time complexity linear in the number of experts (which in many cases can be exponential in the number of parameters).
Another approach is to reduce a bandit problem to supervised learning, such as the epoch-greedy algorithm (Langford and Zhang, 2008) that has an O ( T 2 / 3 ) regret.
Later, Agarwal et al.
(2014) develop an algorithm that yields a near-optimal regret bound, but relies on an optimization oracle that can be expensive.
A third approach uses nonparametric modeling, such as Gaussian processes and kernels (Srinivas et al., 2010; Krause and Ong, 2011; Valko et al., 2013).
More speciﬁcally, Srinivas et al.
(2010) assumed that the reward function is generated from a Gaussian process with known mean and covariance functions.
They proposed a GP-UCB algorithm which achieves (cid:101) O ( √ Tγ T ) regret, where γ T is the maximum information gain.
Krause and Ong (2011) assumed the reward function is deﬁned over the context-arm joint space and proposed a Contextual GP-UCB.
Valko et al.
(2013) assumed that the reward function lies in a RKHS with bounded RKHS norm.
They proposed a SupKernelUCB algorithm and showed an (cid:101) O ( (cid:112)(cid:101) dT ) regret, where (cid:101) d is effective dimension of the kernel that can be seen as a generalized notion of the dimension of contexts.
There is also work focusing on bandit problems in general metric space with
Lipschitz continuous property on the context (Kleinberg et al., 2008; Bubeck et al., 2011).
2. N EURAL N ETWORKS
Different lines of research have been done to provide theoretical understandings of DNNs from different aspects.
For example, to understand how the expressive power of DNNs are related to their architecture, Telgarsky (2015; 2016); Liang and Srikant (2016);
Yarotsky (2017; 2018); Hanin (2017) showed that deep neural networks can express more function classes than shallow networks.
Lu et al.
Hanin and Sellke (2017) suggested that the width of neural networks is crucial to improve the expressive power of neural networks.
For the optimization of DNNs, a series of work have been proposed to show that (stochastic) gradient descent can ﬁnd the global minima of training loss (Li and Liang, 2018; Du et al., 2019b; AllenZhu et al., 2019; Du et al., 2019a;
Zou et al., 2019; Zou and Gu, 2019).
For the generalization of
DNNs, a series of work (Daniely, 2017; Cao and Gu, 2019a;b; Arora et al., 2019) shows that by using (stochastic) gradient descent, the parameters of a DNN are located in a particular regime and the generalization bound of DNNs can be characterized by the best function in the corresponding neural tangent kernel space (Jacot et al., 2018).
P ROBLEM S ETTING
We consider the stochastic K -armed contextual bandit problem, where the total number of rounds
T is known.
At round t ∈ [ T ] , the agent observes the t th context consisting of K feature vectors: { x t,a ∈ R d | a ∈ [ K ] } .
The agent selects an action a t and receive a reward r t,a t .
For simplicity, we denote { x i } TKi =1 as the collection of { x 1 , 1 , x 1 , 2 , . . .
, x T,K } .
Our goal is to maximize the following pseudo regret (or regret for short):
[Equation]
where a ∗ t is the optimal action at round t that maximizes the expected reward, i.e., a ∗ t = argmax a ∈ [ K ] E [ r t,a ] .
This work makes the following assumption on reward generation: for any round t ,
[Equation]
where h is some unknown function satisfying 0 ≤ h ( x ) ≤ 1 for any x , and ξ t is ν -sub-Gaussian noise conditioned on x 1 ,a 1 , . .
, x t − 1 ,a t − 1 .
Note that the ν -sub-Gaussian noise assumption for ξ t is a standard assumption in stochastic bandit literature (e.g., Abbasi-Yadkori et al., 2011); in particular, any bounded noise satisﬁes such an assumption.
Our reward function class contains linear functions, generalized linear functions, Gaussian processes, and kernel functions with bounded RKHS norm over a bounded domain.
In order to learn the reward function h in (3.2), we propose to use a fully connected deep neural networks with depth L ≥ 2 :
[Equation]
where σ ( x ) = max { x, 0 } is the rectiﬁed linear unit (ReLU) activation function, W 1 ∈ R m × d , W i ∈
R m × m , 2 ≤ i ≤ L − 1 , W L ∈ R m × 1 , and θ = [ vec ( W 1 ) (cid:62) , . .
, vec ( W L ) (cid:62) ] (cid:62) ∈
R p with p = m
+ md + m
( L − 1) .
Without loss of generality, we assume that the width of each hidden layer is the same (i.e., m ) for convenience in analysis.
We denote the gradient of the neural network function by g ( x ; θ )
=
∇ θ f
( x ; θ ) ∈ R p .
T HE N EURAL UCB A LGORITHM
We present in Algorithm 1 our algorithm, NeuralUCB.
The key idea is to use a deep neural network f ( x ; θ ) to predict the reward of context x , and upper conﬁdence bound-based exploration (Auer, 2002).
In particular, Algorithm 1 ﬁrst initializes the network by randomly generating each entry of θ from an appropriate Gaussian distribution.
More speciﬁcally, for each 1 ≤ l ≤ L − 1 , we initialize W l as (cid:18)
[Equation]
, where each entry of W is generated independently from N (0 , 4 /m ) .
For W L , we initialize W L as ( w (cid:62) , − w (cid:62) ) , where each entry of w is generated independently from N (0 , 2 /m ) .
At round t , Algorithm 1 observes the contexts for all actions, { x t,a } Ka =1 .
First, it computes the upper conﬁdence bound, U t,a , based on context x t,a , the current neural network parameter θ t − 1 , and a scaling factor γ t − 1 .
It then chooses action a t with the largest U t,a , and receives the corresponding reward r t,a t .
At the end of round t , Algorithm 1 updates θ t by applying Algorithm 2 to (approximately) minimize L ( θ ) using gradient descent, and updates γ t .
We choose gradient descent in Algorithm 2 for the simplicity of analysis, although the training method can be replaced by more efﬁcient algorithms like stochastic gradient descent with a more involved analysis (AllenZhu et al., 2019; Zou et al., 2019) .
do 7:
Compute
[Equation]
θ
[Equation]
γ
[Equation]
x
[Equation] [Equation]
t
[Equation]
Z
[Equation]
[Equation] [Equation]
) /m 8:
Let a t = argmax a ∈ [ K ] U t,a 9: for 10:
Play a t and observe reward r t,a t 11:
Let θ t
= TrainNN ( λ, η, J, m, { x i,a i } ti =1 , { r i,a i } ti =1 , θ 0 ) 12:
Compute Z t =
Z t − 1 + g ( x t,a t ; θ
[Equation] [Equation]
x t,a
[Equation]
; θ t ) (cid:62) /m 13:
Compute
[Equation] [Equation]
14: end for
Comparison with Existing Algorithms Here, we compare NeuralUCB with other neural network based contextual bandit algorithms.
Allesiardo et al.
(2014) proposed NeuralBandit which consists of K neural networks.
It uses a committee of networks to compute the score of each action and chooses the action by (cid:15) -greedy policy.
In contrast, our NeuralUCB uses upper conﬁdence bound based exploration, which is more effective than (cid:15) -greedy.
In addition, our algorithm only uses one neural network instead of K neural networks, thus can be computationally more efﬁcient.
width m
2: Deﬁne L θ
= ti f θ
− r i a i ) 2 / 2 + mλ (cid:107) θ − θ (0) (cid:107) 22 / 2 .
1 do 4:
∇
Lipton et al.
(2018) used Thompson sampling on deep neural networks (through variational inference) in reinforcement learning.
A variant is proposed by Azizzadenesheli et al.
(2018) that works well on a set of Atari benchmarks.
Riquelme et al.
(2018) proposed NeuralLinear, which uses the ﬁrst L − 1 layers of a L -layer DNN to learn a representation, then applies Thompson sampling on the last layer to choose action.
Zahavy and Mannor (2019) proposed a NeuralLinear with limited memory (NeuralLinearLM), which also uses the ﬁrst L − 1 layers of a L -layer DNN to learn a representation and applies Thompson sampling on the last layer.
Instead of computing the exact mean and variance in Thompson sampling, NeuralLinearLM only computes their approximation.
Unlike
NeuralLinear and NeuralLinearLM, NeuralUCB uses the entire DNN to learn the representation and constructs the upper conﬁdence bound based on the random feature mapping deﬁned by the neural network gradient.
Variant of NeuralUCB We provide a variant of NeuralUCB called NeuralUCB 0 in Appendix E.
NeuralUCB 0 can be regarded as a simpliﬁed version of NeuralUCB where the neural network parameter vector θ t is not updated at each round.
In this sense, NeuralUCB 0 can be seen as KernelUCB (Valko et al., 2013) specialized to the Neural Tangent Kernel (Jacot et al., 2018), or LinUCB (Li et al., 2010) with Neural Tangent Random Features (Cao and Gu, 2019a).
We will include both NeuralUCB and NeuralUCB 0 in the experiments.
Efﬁcient Implementation Algorithm 1 can be implemented efﬁciently using iterative ﬁrst-order algorithms, as in linear bandit literature (Li et al., 2010; Valko et al., 2013; Agarwal et al., 2014).
More details are given here for the sake of completeness.
At round t , we deﬁne d t,a =
Z − 1 t − 1 g ( x t,a ; θ t ) .
Based on d t,a , we update det( Z t ) by matrix determinant lemma (Golub and
Van Loan, 1996):
[Equation]
Note that (4.1) only requires to compute vector inner product between g ( x t,a t ; θ t ) and d t,a , which only requires O ( p ) time.
Now we show how to compute d t,a efﬁciently.
By the deﬁnition of d t,a , we have
[Equation]
Thus, d t,a is the global minimizer of the following convex optimization problem:
[Equation]
and we can use (stochastic) gradient descent to ﬁnd d t,a efﬁciently, whose update formula can be written as follows:
[Equation]
which again only requires to compute vector inner products.
R EGRET A NALYSIS
In this section, we present a regret analysis for Algorithm 1.
Recall that { x i } TKi =1 is the collection of all { x t,a } .
Since our regret analysis is built upon the recently proposed neural tangent kernel matrix (Jacot et al., 2018), we start with its formal deﬁnition.
Deﬁnition 5.1 (Jacot et al.
; Cao and Gu (2019a)) .
For a set of contexts { x i } TKi =1 , deﬁne
[Equation]
Then, H =
( (cid:101) H ( L ) + Σ ( L ) ) / 2 is called the neural tangent kernel (NTK) matrix on the context set.
As shown in Deﬁnition 5.1, the Gram matrix of the NTK on the contexts { x i } TKi =1 for L -layer neural networks is deﬁned in a recursive way from the input layer all the way to the output layer of the neural network.
For more details about the derivation of Deﬁnition 5.1, please refer to Jacot et al.
Based on Deﬁnition 5.1, we ﬁrst lay out the assumption on the contexts { x i } TKi =1 .
Assumption 5.2.
H (cid:23) λ
I .
Moreover, for any 1 ≤ i ≤ TK , (cid:107) x i (cid:107) 2 = 1 and [ x i ] j
=
[ x i ] j + d/ 2 .
The ﬁrst part of the assumption says that the neural tangent kernel matrix is non-singular, a mild assumption commonly made in the related literature (Du et al., 2019a; Arora et al., 2019; Cao and
Gu, 2019a).
It can be satisﬁed as long as no two contexts in { x i } TKi =1 are parallel.
The second part is also mild: for any context x , (cid:107) x (cid:107) 2 = 1 , we can always construct a new context x (cid:48) =
[ x (cid:62) , x (cid:62) ] (cid:62) /
√to satisfy Assumption 5.2.
It can be veriﬁed that if θ 0 is generated by the random initialization scheme in Algorithm 1, then f ( x i ; θ 0 )
=
for any i ∈ [ TK ] .
Next we deﬁne the effective dimension (cid:101) d of the neural tangent kernel matrix on contexts { x i } TKi =1 .
Deﬁnition 5.3.
The effective dimension (cid:101) d of the neural tangent kernel matrix on contexts { x i } TKi = is deﬁned as
[Equation]
Remark 5.4.
The notion of effective dimension was introduced by Valko et al.
for analyzing kernel contextual bandits, which was deﬁned by the eigenvalues of any kernel matrix restricted on the given contexts.
We adapt a similar but different deﬁnition of Yang and Wang (2019), which was used for the analysis of kernel-based Q-learning.
Suppose the effective dimension of the reproducing kernel Hilbert space induced by the given kernel is (cid:98) d and the feature mapping ψ induced by the given kernel satisﬁes (cid:107) ψ ( x ) (cid:107) 2 ≤ 1 for any x ∈ R d .
Then, it is easy to verify that we always have (cid:101) d ≤ (cid:98) d ; see Appendix A.1 for details.
Intuitively, (cid:101)
d measures how quickly the eigenvalues of H decay, and it only depends on T logarithmically in certain speciﬁc cases (Valko et al., 2013).
Now we are ready to present the main result, which provides the regret bound R T of Algorithm 1.
Theorem 5.5.
Let (cid:101) d be the effective dimension deﬁned in Deﬁnition 5.3.
Let h =
[ h ( x i )] TKi =1 ∈ R TK .
Suppose there are constants C 1 , C 2 > 0 , such that for any δ ∈ (0 , 1) , if
[Equation] [Equation]
λ ≥ max { 1 , S − 2 } , and S ≥ √ 2 h (cid:62) H − 1 h .
Then, with probability at least 1 − δ over the random initialization of θ 0 , the regret of Algorithm 1 satisﬁes
[Equation]
Remark 5.6.
It is worth noting that, simply applying results for linear bandit to our algorithm would lead to a linear dependence of p or √ p in the regret.
Such a bound is vacuous since in our setting p would be very large compared with the number of rounds T and the input context dimension d .
In contrast, our regret bound only depends on (cid:101) d , which is much smaller than p .
Remark 5.7.
Our regret bound (5.2) has a term (1 − λ/ ( TL ))
J (cid:112) T/λ , which characterizes the optimization error of Algorithm 2.
More speciﬁcally, it sufﬁces to set J = log( λS/ √ T ) /
( λ/ ( TL ))
= (cid:101)
O
( TL/λ ) (independent of m ) such that (1 − λ/ ( TL ))
J (cid:112) T/λ ≤ √ λS , and therefore the optimization error is dominated by √ λS .
Remark 5.8.
Treating ν and λ as constants, and taking S
[Equation]
log( λS/ √ T ) /
( λ/ ( TL )) , the regret bound (5.2) becomes R T
= (cid:101) O
(cid:16)(cid:112)(cid:101) dT (cid:113)
[Equation] { (cid:101)
d [Equation] Speciﬁcally, suppose h belongs to a RKHS H induced by NTK and it can be represented by h ( x )
=
(cid:104) ψ ( x )
, β ∗ (cid:105) , where ψ ( x ) is the feature mapping induced by NTK, for some β ∗ in H (Valko et al., 2013). Then we can show that (cid:107) h (cid:107) H = (cid:107) β ∗ (cid:107) 2 ≥ √ h (cid:62) H − 1 h ; see Appendix A.2 for more details.
Thus our regret bound can be further written as R T = (cid:101) O (cid:16)(cid:112)(cid:101) dT (cid:113) max { (cid:101) d, (cid:107) h (cid:107) H } (cid:17) .
The high-probability result in Theorem 5.5 can be used to obtain a bound on the expected regret.
Corollary 5.9.
Under the same conditions in Theorem 5.5, we have
[Equation]
P ROOF OF M AIN R ESULTS
This section provides the proof of Theorem 5.5.
We ﬁrst point out several technical challenges in this proof:
• We do not make parametric assumptions on the reward function as some previous work (Filippi et al., 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011).
Furthermore, unlike the ﬁxed feature mapping used in Valko et al. (2013), NeuralUCB uses neural network f ( x ; θ t ) and its gradient g ( x ; θ t ) as a dynamic feature mapping depending on θ t .
These differences make the regret analysis of NeuralUCB more difﬁcult.
• In practice, the neural network is often overparametrized, which implies m (and thus p ) is very large.
Thus, we need to make sure the regret bound is independent of m .
The two challenges above are addressed by the following technical lemmas.
Their proofs are gathered in the appendix.
Lemma 6.1.
There exists some constant ¯ C > 0 such that for any δ ∈ (0 , 1) , if m ≥ ¯
CT 4 K 4 L 6 log( T 2 K 2 L/δ ) /λ 40 , then with probability at least 1 − δ over the random initialization of θ 0 , there is a θ ∗ ∈ R p such that
[Equation]
for all i ∈ [ TK ] .
Lemma 6.1 suggests that with high probability, the reward function restricted on { x i } TKi =1 can be regarded as a linear function of g ( x i ; θ 0 ) parameterized by θ ∗ − θ 0 , where θ ∗ lies in a ball centered at θ 0 .
Note that here θ ∗ is not a ground truth parameter for the reward function.
Instead, it is introduced only for the sake of analysis.
Equipped with Lemma 6.1, we can utilize existing results on linear bandits (Abbasi-Yadkori et al., 2011) to show that with high probability, θ ∗ lies in the sequence of conﬁdence sets.
Lemma 6.2.
There exist constants { ¯ C i } 2 i =1 > 0 such that for any δ ∈ (0 , 1) , if η ≤ ¯ C 1 ( TmL + mλ ) − 1 and m ≥ ¯ C 2 max (cid:8) T 7 λ − 7 L 21 (log m ) 3 , λ − 1 / 2 L − 3 / 2 (log( TKL 2 /δ )) 3 / 2 (cid:9) , then with probability at least 1 − δ over the random initialization of θ 0 , we have (cid:107) θ t − θ
(cid:107) 2 ≤ 2 (cid:112) t/ ( mλ ) and (cid:107) θ ∗ − θ t (cid:107) Z t ≤ γ t / √ m for all t ∈
[ T ] .
Lemma 6.3.
Denote a ∗ t = argmax a ∈ [ K ] h ( x t,a ) .
There exist constants { ¯ C i } 3 i = > 0 such that for any δ
[Equation] [Equation]
, if η
[Equation]
≥ ¯ C 2 max (cid:8) T 7 λ − 7 L 21 (log m ) 3 , λ − 1 / 2 L − 3 / 2 (log( TKL 2 /δ )) 3 / 2 (cid:9) , then with probability at least 1 − δ over the random initialization of θ 0 , we have
[Equation]
Lemma 6.3 gives upper bounds for h (cid:0) x t,a
[Equation]
t (cid:1) − h
[Equation]
x t,a t (cid:1) , which helps us to bound the regret R T .
It is worth noting that γ t has a term log det Z t .
A trivial upper bound of log det Z t would result in an dependence on the neural network width m , since the dimension of Z t is p
= md + m
( L − 2)+ m .
The next lemma establishes an upper bound which is independent of m , and is only related to effective dimension (cid:101) d .
The dependence on (cid:101) d is similar to Lemma 4 of Valko et al.
, but the proof is different as we use a different notion of effective dimension.
Lemma 6.4.
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ ∈ (0 , 1) , if η ≤ ¯ C 1 ( TmL + mλ ) − 1 , m ≥ ¯ C 2 max (cid:8) T 7 λ − 7 L 21 (log m ) 3 , T 6 K 6 L 6 (log( TKL 2 /δ )) 3 / 2 (cid:9) , then with probability at least 1 − δ over the random initialization of θ 0 , we have
[Equation]
With above lemmas, we begin to prove Theorem 5.5.
Proof of Theorem 5.5.
The total regret R T can be bounded as follows:
[Equation]
−
[Equation]
(cid:0) x
[Equation]
a
[Equation]
(cid:1)(cid:3) ≤
[Equation]
(cid:88) t = γ
[Equation]
−
[Equation]
min
[Equation] [Equation]
[Equation]
t
[Equation]
min
[Equation]
−
[Equation]
Sm
[Equation] [Equation]
mT
[Equation] [Equation]
[Equation] [Equation]
mT
[Equation] [Equation]
where C 1 , C 2 > 0 are constants, the ﬁrst inequality holds due to Lemma 6.3, the second inequality holds Cauchy-Schwarz inequality, the third inequality holds due to Lemma 6.4 and choosing large enough m .
This completes our proof.
[Figure]
E XPERIMENTS
While the focus of this work is mostly on theoretical analysis of regret, we present results in proofof-concept experiments in simulated problems.
We compare it with four representative baselines: (1) LinUCB; (2) Neural (cid:15) -Greedy, which replaces the UCB based exploration in Algorithm 1 by (cid:15) -greedy
based exploration; (3) NeuralUCB 0 ; and (4) Neural (cid:15) -Greedy 0 which replaces the UCB based exploration in NeuralUCB 0 by (cid:15) -greedy
based exploration.
We use the cumulative regret as the performance metric.
In our simulation, we use contextual bandit problems with context dimension d = 20 , the number of actions K = 4 and the number of rounds T = 10 000 .
The contextual vectors { x 1 , 1 , . . .
, x T,K } are randomly chosen from N ( 0 , I ) and then normalized to have unit norm, i.e., (cid:107) x t,a (cid:107) 2 = 1 .
For the reward function h , we investigate the following nonlinear functions:
[Equation]
where A ∈ R d × d and each entry of A is randomly generated from N (0 , 1) , a is randomly chosen from N ( 0 , I ) and normalized to have (cid:107) a (cid:107) 2 = 1 .
For each h i ( · ) , the reward at round t for action a is generated by r t,a = h i ( x t,a ) + ξ t , where ξ t is independently drawn from N (0 , 1) .
For LinUCB, we follow Li et al.
(2010) to implement it with a constant radius α .
We do a grid search for α over { 0 . 01 , 0 . 1 , 1 , 10 } and choose the best α for comparison.
For NeuralUCB
NeuralUCB 0 , Neural (cid:15) -Greedy and Neural (cid:15) -Greedy 0 , we choose a two-layer neural network f ( x ; θ ) = √ m W 2 σ ( W 1 x ) with network width m = 20 , where θ = [ vec ( W 1 ) (cid:62) , vec ( W 2 ) (cid:62) ]
∈ R p and p = md + m .
We remark that the bound on the required network width m is likely not tight.
Therefore, in experiments we choose m to be relatively large (but not as large as theory suggests).
For γ t in NeuralUCB, we choose γ t = γ in the experiment.
We do a grid search over { 0 . 01 , 0 . 1 , 1 , 10 } and choose the best γ for comparison.
For NeuralUCB 0 , we choose ν = 1 , λ = 1 , δ = 0 .
1 , and we do a grid search over { 0 . 01 , 0 . 1 , 1 , 10 } for hyper-parameter S to choose the best S for comparison.
For Neural (cid:15) -Greedy and Neural (cid:15) -Greedy 0 , we do a grid search for (cid:15) over { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 2 } and choose the best (cid:15) for comparison.
For all the algorithms, we repeat the experiment for 10 runs and report the averaged results for comparison.
For TrainNN, we choose the step size η = 0 .
To accelerate the training process, we update the parameter θ t by
TrainNN every 50 rounds.
We use stochastic gradient descent with batch size 50 and set J = t at t -th round of NeuralUCB and Neural (cid:15) -Greedy.
We plot the cumulative regret of compared algorithms in Figure 1, for reward function h ∈ { h 1 , h 2 , h 3 } .
We can see that due to the nonlinearity of reward function h , LinUCB fails to learn the true reward function and hence achieve an almost linear regret, as expected.
In contrast, thanks to the neural network representation and efﬁcient exploration, NeuralUCB achieves a sublinear regret which is much lower than that of LinUCB.
The performance of Neural (cid:15) -Greedy is in-between.
This suggests that while Neural (cid:15) -greedy can capture the nonlinearity of the underlying reward function, (cid:15) -Greedy
based exploration is not as effective as UCB based exploration.
This conﬁrms the effectiveness of NeuralUCB for contextual bandit problems with any bounded (nonlinear) reward function.
Meanwhile, it is worth noting that NeuralUCB and Neural (cid:15) -Greedy outperform NeuralUCB 0 and
Neural (cid:15) -Greedy 0 , which suggests that using deep neural networks to predict the reward function is better than using a ﬁxed feature mapping associated with NTK.
Another observation is that although the network width m in the experiment is not as large as our theory suggests, NeuralUCB still achieves a sublinear regret for nonlinear reward functions.
We leave it as a future work to investigate the impact of m on regret.
C ONCLUSIONS AND F UTURE W ORK
In this work we proposed a new algorithm NeuralUCB for stochastic contextual bandit problems based on neural networks.
We show that for arbitrary bounded reward function, our algorithm achieves (cid:101) O
( (cid:101) d √ T ) regret bound.
Our preliminary experiment results on synthetic data corroborate our theoretical ﬁndings.
In the future, we are interested in a systematic empirical evaluation of
NeuralUCB on real world datasets, and compare it with the state-of-the-art neural network based contextual bandit algorithms (without provable guarantee in regret) (Riquelme et al., 2018; Zahavy and Mannor, 2019).
Another interesting direction is provably efﬁcient exploration with neural network using other strategies like Thompson sampling
A P ROOF OF A DDITIONAL R ESULTS IN S ECTIONA. V ERIFICATION OF R EMARK 5.
Suppose there exists a mapping ψ : R d → R (cid:98) d satisfying (cid:107) ψ ( x ) (cid:107) 2 ≤ 1 which maps any context x ∈ R d to the Hilbert space H associated with the Gram matrix H ∈ R TK × TK over contexts { x i } TKi =1 .
Then H =
Ψ (cid:62) Ψ , where Ψ =
[ ψ ( x 1 ) , . .
, ψ ( x TK )] ∈ R (cid:98) d × TK .
Thus, we can bound the effective dimension (cid:101) d as follows
[Equation]
λ [Equation] where the second equality holds due to the fact that det( I + A (cid:62) A /λ )
= det( I + AA (cid:62) /λ ) holds for any matrix A , and the inequality holds since det A ≤ (cid:107) A (cid:107) (cid:98)
d 2 for any A ∈ R (cid:98) d × (cid:98) d .
Clearly, (cid:101) d ≤ (cid:98) d as long as (cid:13)(cid:13)
I + [Equation] (cid:62) /λ
(cid:13)(cid:13) 2 ≤ 1 + TK/λ .
Indeed,
[Equation]
where the ﬁrst inequality is due to triangle inequality and the fact λ ≥ 1 , the second inequality holds due to the deﬁnition of Ψ and triangle inequality, and the last inequality is by (cid:107) ψ ( x i ) (cid:107) 2 ≤ 1 for any 1 ≤ i ≤ TK .
A. V ERIFICATION OF R EMARK 5.
Let K ( · , · ) be the NTK kernel, then for any i, j ∈ [ TK ] , we have H i,j = K ( x i , x j ) . We ﬁrst prove that (cid:107) h (cid:107) H =
(cid:107) β ∗ (cid:107) 2 .
Since h ∈ H , we know that h ( x
[Equation]
i = α i K ( x i , x ) = n (cid:88) i = α
[Equation]
ψ ( x i ) , ψ ( x ) (cid:105) =
(cid:68) n (cid:88) i
= α i ψ ( x i ) , ψ ( x ) (cid:69) , for some { x i } ni =1 , { α i } ni =1 . Thus, we have β ∗ = (cid:80) ni
{+=
+}
α i ψ ( x i ) .
Therefore, we have (cid:107) h (cid:107) 2 H = n (cid:88) i,j = α i α j K ( x i , x j ) =
n (cid:88) i,j = α
i α j (cid:104) ψ ( x i ) , ψ ( x j ) (cid:105) =
(cid:68) n (cid:88) i
= α
i ψ ( x i ) , n (cid:88)
j = α j ψ ( x j ) (cid:69)
[Equation]
which immediately suggests that (cid:107) h (cid:107) H = (cid:107) β ∗ (cid:107) 2 .
Next we prove that (cid:107) h (cid:107) H ≥ √ h (cid:62) H − 1 h .
First, h can be decomposed as h =
h H + h ⊥ , where h H ( x ) =
(cid:80) TKi =1 (cid:101)
α i K ( x , x i ) is the projection of h to the function space spanned by { K ( x , x i ) } TKi =
and h ⊥ is the orthogonal part.
By deﬁnition we have h ( x i ) =
h H ( x i ) for i ∈
[ TK ] , thus
[Equation]
which implies that (cid:101) α = H − 1 h .
Thus, we have
[Equation]
Proof of Corollary 5.9. Notice that R T ≤ T since 0 ≤ h ( x ) ≤ 1 . Thus, with the fact that with probability at least 1 − δ , (5.2) holds, we can bound E [ R T ] as
[Equation]
Taking δ = 1 /T completes the proof.
B P ROOF OF L EMMAS IN S ECTIONB. P ROOF OF L EMMA 6.
We start with the following lemma:
Lemma B.1.
Let G =
[ g ( x 1 ; θ 0 ) , . . .
, g ( x TK ; θ 0 )] / √ m ∈ R p × ( TK ) .
We denote the neural tangent kernel matrix H the same as Deﬁnition 5.1.
For any δ ∈ (0 , 1) , if m = [Equation]
log [Equation] then with probability at least 1 − δ over the random initialization of θ 0 , we have [Equation]
G
[Equation]
≤ TK(cid:15).
We begin to prove Lemma 6.1.
Proof of Lemma 6.1.
By Assumption 5.2, we know that λ 0 > 0 .
Then with probability at least 1 − δ , we have
[Equation]
(B [Equation] 1) where the second inequality holds due to Lemma B.1 with the choice of m , the third and fourth inequality holds due to H (cid:23) λ
I (cid:31) 0 .
Thus, suppose the singular value decomposition of G is G =
PAQ (cid:62) , P ∈ R p × TK , A ∈ R TK × TK , Q ∈ R TK × TK , we have A (cid:31)
and θ ∗ =
θ 0 + PA − 1 Q (cid:62) h / √ m satisﬁes (6.1).
To validate that θ ∗ satisﬁes (6.1), ﬁrst we have
[Equation]
which suggests that for any i , (cid:104) g ( x i ; θ 0 ) , θ ∗ − θ 0 (cid:105)
=
h ( x i ) .
We also have
[Equation]
h , where the last inequality holds due to (B.1).
Thus, our proof ﬁnishes.
B. P ROOF OF L EMMA 6.
In this section we prove Lemma 6.2.
For the simplicity, we denote ¯ Z t , ¯ b t , ¯ γ t as follows:
[Equation]
We need the following lemmas.
The ﬁrst lemma shows that the network parameter at each round θ t can be well approximated by θ
+ ¯ Z − 1 t ¯ b t / √ m .
Lemma B.2.
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ > 0 , if η, m satisfy that for all t ∈ [ T ] ,
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for any t ∈ [ T ] , we have that (cid:107) θ t − θ[-
-]
[Equation]
2 ≤
[Equation]
t/ ( mλ ) and
[Equation]
Next lemma shows the error bounds for ¯ Z t and Z t .
Lemma B.3.
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ > 0 , if m satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , we have the following inequalities for any t ∈
[ T ] :
[Equation]
With above lemmas, we prove Lemma 6.2 as follows.
Proof of Lemma 6.2.
By Lemma B.2 we know that (cid:107) θ t − θ 0 (cid:107) 2 ≤ 2 (cid:112) t/ ( mλ ) .
By Lemma 6.1, with probability at least 1 − δ , there exists θ ∗ such that for any 1 ≤ t ≤ T ,
[Equation]
where the second inequality holds due to the choice of S .
Thus, by Theorem 2 in Abbasi-Yadkori et al.
(2011), with probability at least 1 − 2 δ , for any 1 ≤ t ≤ T , θ ∗ satisﬁes that
[Equation]
We now prove that (cid:107) θ ∗ − θ t (cid:107) Z t ≤ γ t / √ m .
From the triangle inequality, (cid:107) θ ∗ − θ t [Equation] Z t ≤ (cid:107)
[Equation]
We bound I
and I 2 separately.
For I 1 , we have
[Equation]
where the ﬁrst inequality holds due to the fact that x (cid:62) Ax ≤ x [Equation] ·(cid:107) A
[Equation] 2 /λ min ( B ) for some B (cid:31)and the fact that λ min (¯ Z t ) ≥ λ , the second inequality holds due to (B.2).
By Lemma B.3, we have
[Equation]
and
[Equation]
where C 1 , C 2 > 0 are two constants.
Substituting (B.5) and (B.6) into (B.4), we have
[Equation]
For I 2 , we have
[Equation]
where C 3 , C 4 > 0 are two constants, the ﬁrst inequality holds due to (cid:107) Z t (cid:107) 2 ≤ (cid:107) Z t (cid:107) F ≤ C 3 tL by
Lemma B.3, the second inequality holds due to Lemma B.2.
Substituting (B.7) and (B.8) into (B.3), we have (cid:13)(cid:13)
θ ∗ − θ t (cid:13)(cid:13)
Z t ≤ γ t / √ m .
Our proof is thus completed.
B. P ROOF OF L EMMA 6.
The proof starts with three lemmas that bound the error terms of the function value and gradient of neural networks.
Lemma B.4 (Lemma 4.1, Cao and Gu (2019a)) .
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ > 0 , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for all (cid:101) θ , (cid:98) θ satisfying (cid:107) (cid:101) θ − θ 0 (cid:107) 2 ≤ τ, [Equation] θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have
[Equation]
Lemma B.5 (Theorem 5, Allen-Zhu et al.
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ ∈ (0 , 1) , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for all (cid:107) θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have
[Equation]
Lemma B.6 (Lemma B.3, Cao and Gu (2019a)) .
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ > 0 , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for any (cid:107) θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have (cid:107) g ( x j ; θ ) (cid:107) F ≤ ¯ C 3 √ mL .
Proof of Lemma 6.3.
We follow the regret bound analysis in Abbasi-Yadkori et al.
(2011); Valko et al.
Denote a ∗ t = argmax a ∈ [ K ] h ( x t,a ) and C t = { θ : (cid:107) θ − θ t (cid:107) Z t ≤ γ t / √ m } .
By
Lemma 6.2, for all 1 ≤ t ≤ T , we have (cid:107) θ t − θ
(cid:107) 2 ≤ 2 (cid:112) t/ ( mλ ) and θ ∗ ∈ C t .
By the choice of m , Lemma B.4, B.5 and B.6 hold.
Thus, h ( x t,a ∗ t ) − h ( x t,a t ) can be bounded as follows:
[Equation]
where the ﬁrst inequality holds due to triangle inequality
[Equation]
the second inequality holds due to Lemma 6.
[Equation]
Lemma B.5 and B. [Equation] the third inequality holds due to θ ∗ ∈ C t − 1 .
Denote
[Equation]
then we have (cid:101) U t,a = max θ
[Equation]
g ( x t,a ; θ
[Equation]
1 ) , θ − θ 0 (cid:105) with the fact that
[Equation]
We also have
[Equation]
where C 2 > 0 is a constant, the inequality holds due to Lemma B.4 with the fact (cid:107) θ t − 1 − θ 0 (cid:107) 2 ≤ 2 (cid:112) t/ ( mλ )) and the fact f ( x j ; θ 0 )
= 0 .
Since θ ∗ ∈ C t − 1 , then (B.9) can be bounded as
[Equation]
where the ﬁrst inequality holds due to (B
[Equation]
[Equation] the second inequality holds since a t = argmax a U t,a , the third inequality holds due to (B.10).
Furthermore,
[Equation]
where the ﬁrst inequality holds due to Cauchy-Schwarz inequality, the second inequality holds due to Lemma 6.2.
Combining (B.9), (B.11) and (B.12), we have
[Equation]
where the second inequality holds due to the fact that 0 ≤ h ( x t,a ∗ t ) − h ( x t,a t ) ≤ 1 , the third inequality holds due to the fact that min { a + b, 1 } ≤ min { a, 1 } + b , the fourth inequality holds due to the fact γ t − 1 ≥ √ λS ≥ 1 .
Finally, by the fact that √ 2 hH − 1 h ≤ S , we ﬁnish the proof.
B. P ROOF OF L EMMA 6.
In this section we prove Lemma 6.4, we need the following lemma from Abbasi-Yadkori et al.
Lemma B.7 (Lemma 11, Abbasi-Yadkori et al.
We have the following inequality:
[Equation]
Proof of Lemma 6.4.
First by the deﬁnition of γ t , we know that γ t is a monotonic function w.r.t.
det Z t .
By the deﬁnition of Z t , we know that Z T (cid:23) Z t , which implies that det Z t ≤ det Z T .
Thus, γ t ≤ γ T .
Second, by Lemma B.7 we know that
[Equation]
where the second inequality holds due to Lemma B.3.
Next we are going to bound log det ¯ Z T .
Denote G =
[ g ( x 1 ; θ 0 ) / √ m, . . .
, g ( x TK ; θ 0 ) / √ m ] ∈ R p × ( TK ) , then we have [Equation]
[Equation]
where the inequality holds naively, the third equality holds since for any matrix A ∈ R p × TK , we have det( I + AA (cid:62) )
=
det( I + A (cid:62) A ) .
We can can be further bound (B.15) by the follows:
[Equation]
where the ﬁrst inequality holds due to the concavity of log det( · ) , the second inequality holds due to the fact that (cid:104) A , B (cid:105) ≤ (cid:107) A (cid:107) F (cid:107) B (cid:107) F , the third inequality holds due to the facts that I + H /λ (cid:23)
I , λ ≥ 1 and (cid:107) A (cid:107) F ≤ √ TK (cid:107) A (cid:107) 2 for any A ∈ R TK × TK , the fourth inequality holds by Lemma B. with the choice of m , the ﬁfth inequality holds by the deﬁnition of effective dimension in Deﬁnition 5. [Equation]
and the last inequality holds due to the choice of λ .
We now bound γ T , which is
[Equation]
where the inequality holds due to Lemma B.3.
Finally, substituting (B.15), (B.16) into (B.14), using (B.14) and (B.17)
[Equation]
we have
[Equation]
+ C 1 m − 1 / 6 (cid:112) log mL 4 T 7 / 6 λ − 7 /· (cid:18) ν (cid:113)(cid:101) d log(1 + TK/λ ) + 1 + C 2 m − 1 / 6 (cid:112) log mL 4 T 5 / 3 λ − 1 / 6 − 2 log δ + √ λS (cid:19) + C 3 (cid:104) (1 − ηmλ ) J (cid:112)
T/ ( mλ ) + m − 3 / 2 (cid:112) log mL 7 / 2 T 5 / 3 λ − 5 / 3 (1 + (cid:112) T/λ ) (cid:105)(cid:21) .
C P ROOFS OF T ECHNICAL L EMMAS IN A PPENDIX B C. P ROOF OF L EMMA B.
In this section we prove Lemma B.1, we need the following lemma from Arora et al.
Lemma C.1 (Theorem 3.1, Arora et al.
Fix (cid:15) > 0 and δ ∈ (0 , 1) .
Suppose that
[Equation]
then for any i, j ∈ [ TK ] , with probability at least 1 − δ over random initialization of θ 0 , we have [Equation] (C.
[Equation]
Proof of Lemma B.1.
Taking union bound over i, j ∈ [ TK ] , we have that if
[Equation]
, then with probability at least 1 − δ , (C.1) holds for all ( i,
j ) ∈
[ TK ] ×
[ TK ] .
Therefore, we have
[Equation]
[Table]
C. P ROOF OF L EMMA B.
In this section we prove Lemma B.2.
During the proof, for the simplicity, we omit the subscript t by default.
We deﬁne the following quantities:
[Equation]
Then the update rule of θ ( j ) can be written as follows:
[Equation]
We also deﬁne the following auxiliary sequence { (cid:101) θ ( k ) } during the proof:
[Equation]
Next lemma provides perturbation bounds for J ( j ) , H ( j ) and (cid:107)
f ( j +1)
− f ( j ) − [ J ( j ) ] (cid:62) ( θ ( j +1) − θ ( j ) )
(cid:107) 2 .
Lemma C.2.
There exist constants { ¯ C i } 6 i =1 > 0 such that for any δ > 0 , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ (0) , if for any j ∈ [ J ] , (cid:107) θ ( j ) − θ (0) (cid:107) 2 ≤ τ , we have the following inequalities for any j ∈ [ J ] ,
[Equation]
Next lemma gives an upper bound for (cid:107) f
( j ) − v (cid:107) 2 .
Lemma C.3.
There exist constants { ¯ C i } 4 i =1 > 0 such that for any δ > 0 , if τ, η satisfy that
[Equation]
then with probability at least 1 − δ over the random initialization of θ (0) , if for any j ∈ [ J ] , (cid:107) θ ( j ) − θ (0) (cid:107) 2 ≤ τ , we have that for any j ∈ [ J ] , (cid:107) f
( j ) − v (cid:107) 2 ≤ 2 √ t .
Next lemma gives an upper bound of the distance between auxiliary sequence (cid:107) (cid:101) θ ( j ) − (cid:101) θ (0) (cid:107) 2 .
Lemma C.4.
There exist constants { ¯ C i }
i
=1 > 0 such that for any δ ∈ (0 , 1) , if τ, η satisfy that
[Equation]
then with probability at least 1 − δ , we have that for any j ∈ [ J ] ,
[Equation]
With above lemmas, we prove Lemma B.2 as follows.
Proof of Lemma B.2.
Set τ
= 2 (cid:112) t/ ( mλ ) .
First we assume that (cid:107) θ ( j ) − θ (0) (cid:107)
≤
τ for all 0 ≤ j ≤
J .
Then with this assumption and the choice of m, τ , we have that Lemma C.2, C.3 and C.4 hold.
Then we have
[Equation]
where the ﬁrst inequality holds due to triangle inequality, the second inequality holds due to the fact that η H (0) =
η [ J (0) ] (cid:62) J (0) (cid:22) C 1 tmLη I ≤ I with (C.3) in Lemma C.2 for some C 1 > 0 .
We now bound I 1 , I 2 separately.
For I 1 , we have
[Equation]
where C 2 > 0 is a constant, the ﬁrst inequality holds due to matrix spectral norm and the second inequality holds due to (C.4) in Lemma C.2 and Lemma C.3.
For I 2 , we have
[Equation] (cid:13)(cid:13)
[Equation]
where C 3 > 0 , the ﬁrst inequality holds due to matrix spectral norm, the second inequality holds due to (C.3) and (C.5) in Lemma C.2 and the fact that f (0) = 0 by random initialization over θ (0) .
Substituting (C.8) and (C.9) into (C.7), we have
[Equation]
where C 4 > 0 is a constant.
Expanding (C.10) for k times
[Equation]
we have
[Equation]
where C 5 > 0 is a constant, the equality holds by the deﬁnition of τ , the last inequality holds due to the choice of m , where
[Equation]
C 6 > 0 is a constant.
Thus, for any j ∈ [ J ] , we have
[Equation]
) where the ﬁrst inequality holds due to triangle inequality, the second inequality holds due to Lemma
C.4.
(C.13) suggests that our assumption (cid:107) θ ( j ) − θ (0) (cid:107) 2 ≤ τ holds for any u .
Note that we have the following inequality by Lemma C.4:
[Equation]
Using (C.11) and (C.13), we have
[Equation]
This completes the proof.
C. P ROOF OF L EMMA B.
In this section we prove Lemma B.3.
Proof of Lemma B.3.
Set τ
= 2 (cid:112) t/ ( mλ ) .
By Lemma 6.2, we have that (cid:107) θ i − θ 0 (cid:107) 2 ≤ τ for i ∈ [ t ] .
(cid:107) Z t (cid:107) F can be bounded as follows.
[Equation]
where C 0 > 0 is a constant, the ﬁrst inequality holds due to the fact that (cid:107) aa (cid:62) (cid:107) F = (cid:107) a (cid:107) 22 , the second inequality holds due to Lemma B.6 with the fact that (cid:107) θ i − θ 0 (cid:107) 2 ≤ τ .
We bound (cid:107) Z t − ¯ Z t (cid:107)as follows.
We have
[Equation]
where the ﬁrst inequality holds due to triangle inequality
[Equation]
the second inequality holds the fact that
[Equation]
for any vectors a , b .
To bound (C.14) [Equation] we have
[Equation]
where C 1 > 0 is a constant [Equation] the inequality holds due to Lemma B.6 with the fact that (cid:107) θ i − θ 0 (cid:107) 2 ≤ τ .
We also have
[Equation]
[Equation]
where C 2 , C 3 > 0 are constants, the ﬁrst inequality holds due to Lemma B.5 with the fact that (cid:107) θ i − θ 0 (cid:107) 2 ≤ τ , the second inequality holds due to Lemma B.6.
Substituting (C.15) and (C.16) into (C.14), we have
[Equation]
where C 4 > 0 is a constant.
We now bound log det ¯ Z t − log det Z t .
It is easy to verify that ¯
[Equation]
¯ [Equation]
[¯ J ] (cid:62) , Z t =
λ
I + JJ (cid:62) , where
[Equation]
We have the following inequalities:
[Equation]
J (cid:105) ≤ (cid:107) ( I + [ J ] (cid:62) J /λ ) − 1 (cid:107) F (cid:107) [¯ J ] (cid:62) ¯ J − [ J ] (cid:62) J (cid:107) F ≤ [Equation] (cid:107) ( I + [ J ] (cid:62) J /λ )
[Equation]
(cid:107)
[Equation] (cid:107) [¯ J ] (cid:62) ¯ [Equation] J ] (cid:62) J (cid:107) F
[Equation]
J ] (cid:62) [Equation] J −
[Equation]
J ] (cid:62) [Equation] (cid:107) F , (C.17) where the second equality holds due to the fact
that det( I + AA (cid:62) ) = det( I + A (cid:62) A ) , the ﬁrst inequality holds due to the fact that log det function is convex, the second inequality hold due to the fact that (cid:104) A , B (cid:105) ≤ (cid:107) A (cid:107) F (cid:107) B (cid:107) F , the third inequality holds since I + [ J ] (cid:62) J /λ is a t -dimension matrix, the fourth inequality holds since I + [ J ] (cid:62) J /λ (cid:23) I .
We have
[Equation]
where C 5 > 0 is a constant, the ﬁrst inequality holds due to the fact that (cid:107) A (cid:107) F ≤ t max |
A i,j | for any A ∈ R t × t , the second inequality holds due to the fact | a (cid:62) a (cid:48) − b (cid:62) b (cid:48) | ≤ (cid:107) a − b {+(cid:107) 2 (cid:107) b (cid:48) (cid:107) 
+}
+ (cid:107) a (cid:48)
− b (cid:48) (cid:107) 2 (cid:107) a (cid:107) 2 , the third inequality holds due to (C.15) and (C.16).
Substituting (C.18) into (C.17), we have
[Equation]
Using the same method, we also have
[Equation]
This completes our proof.
D P ROOFS OF L EMMAS IN A PPENDIX C D. P ROOF OF L EMMA C.
In this section we give the proof of Lemma C.2.
Proof of Lemma C.2.
For any j ∈ [ J ] , the following inequalities hold.
We ﬁrst have
[Equation]
where C 1 > 0 is a constant, the ﬁrst inequality holds due to the deﬁnition of J ( j ) , the second inequality holds due to Lemma B.6.
We also have
[Equation]
where C 2 , C 3 > 0 are constants, the ﬁrst inequality holds due to Lemma B.5 with the assumption that (cid:107) θ ( j ) − θ (0) (cid:107) 2 ≤ τ , the second inequality holds due to (D.1).
We also have
[Equation]
where C 4 > 0 is a constant, the ﬁrst inequality holds due to the the fact that (cid:107) x (cid:107) 2 ≤ √ t max | x i | for any x ∈ R t , the second inequality holds due to Lemma B.4 with the assumption that (cid:107) θ ( j ) − θ (0) (cid:107) 2 ≤ τ, (cid:107) θ ( j +1) − θ (0) (cid:107) 2 ≤ τ .
For (cid:107) v (cid:107) 2 , we have (cid:107) v (cid:107) 2 ≤ √ t max 1 ≤ i ≤ t | r
( x i,a i ) | ≤ √ t .
This completes our proof.
D. P ROOF OF L EMMA C.
Proof of Lemma C.3.
Recall that the loss function L is deﬁned as
[Equation]
We deﬁne J ( θ ) and f ( θ ) as follows:
[Equation]
Suppose (cid:107) θ − θ (0) (cid:107) 2 ≤ τ .
Then by the fact that (cid:107) · (cid:107) 22 / 2 is 1 -strongly convex and 1 -smooth
[Equation]
we have the following inequalities:
[Equation]
where e = f ( θ (cid:48) )
− f ( θ ) − J ( θ ) (cid:62) ( θ (cid:48) − θ ) .
I 1 can be bounded as follows:
[Equation]
where the ﬁrst inequality holds due to Cauchy-Schwarz inequality [Equation] the second inequality holds due to the fact that (cid:107) J ( θ ) (cid:107) 2 ≤ C 2 √ tmL with (cid:107) θ − θ (0) (cid:107) 2 ≤ τ by (C.3) in Lemma C.2.
Substituting (D.4) into (D.3), we have
[Equation]
Taking θ (cid:48) = θ − η ∇ L ( θ )
[Equation]
then by (D.5) [Equation] we have
[Equation]
By the 1 -strongly convexity of (cid:107) · (cid:107) 22 , we further have
L ( θ (cid:48) ) − L ( θ ) ≥ (cid:104) f ( θ ) − v , f ( θ (cid:48) )
− f ( θ ) (cid:105) + mλ (cid:104) θ − θ (0) , θ (cid:48) − θ (cid:105) + mλ 2 (cid:13)(cid:13) θ (cid:48) − θ (cid:13)(cid:13) 2= (cid:104) f ( θ ) − v , [ J ( θ )] (cid:62) ( θ (cid:48) − θ ) + e (cid:105) + mλ (cid:104) θ − θ (0) , θ (cid:48) − θ (cid:105) +
mλ
(cid:13)(cid:13)
θ (cid:48) − θ (cid:13)(cid:13) 2=
{+(cid:104)∇ L ( θ ) , θ (cid:48) − θ (cid:105) + mλ 2 (cid:13)(cid:13) θ (cid:48) − θ (cid:13) (cid:13) 2 
 + (cid:104) f ( θ ) − v , e (cid:105) ≥ (cid:104)∇ L ( θ ) , θ (cid:48) − θ (cid:105) + mλ 2 (cid:13)(cid:13) θ (cid:48)+}
− θ (cid:13)(cid:13) 2 2 − (cid:107) f ( θ ) − v (cid:107) 2 (cid:107) e (cid:107)≥ −(cid:107)∇ L ( θ ) (cid:107)2 mλ − (cid:107)
f ( θ ) − v (cid:107) 2 (cid:107)
e (cid:107) 2 , (D.7) where the second inequality holds due to Cauchy-Schwarz inequality, the last inequality holds due to the fact that (cid:104) a , x (cid:105) + c (cid:107) x (cid:107) 22 ≥ −(cid:107) a (cid:107) 22 / (4 c ) for any vectors
a , x and c > 0 .
Substituting (D.7) into (D.6), we have
[Table]
[Equation]
[Table]
[Equation]
where the second inequality holds due to the choice of η , third inequality holds due to Young’s inequality, fourth inequality holds due to the fact that (cid:107) f ( θ ) − v (cid:107) 22 ≤ 2 L ( θ ) .
Now taking θ = θ ( j ) and θ (cid:48)
= θ (0) , rearranging (D.8), with the fact that θ ( j +1)
= θ ( j ) − η ∇ L
θ ( j ) ) , we have
[Equation]
where the second inequality holds due to the fact that L ( θ (0)
[Equation]
=
[Equation]
and
[Equation]
where the ﬁrst inequality holds due to (C.5) in Lemma C.2, the second inequality holds due to the choice of τ .
Expanding (D.9) u times, we have
[Equation]
which implies that (cid:107) f
( j +1)
− v (cid:107) 2 ≤ 2 √ t .
This completes our proof.
D. P ROOF OF L EMMA C.
In this section we prove Lemma C.4.
Proof of Lemma C.4.
It is worth noting that (cid:101) θ ( j ) is the sequence generated by applying gradient descent on the following problem:
[Equation]
Then (cid:107) θ (0) − (cid:101) θ ( j ) (cid:107) 2 can be bounded as
[Equation]
where the ﬁrst inequality holds trivially, the second inequality holds due to the monotonic decreasing property brought by gradient descent, the third inequality holds due to (C.6) in Lemma C.2.
It is easy to verify that (cid:101) L is a λ -strongly convex and function and C 1 ( tmL + mλ ) -smooth function, since
[Equation]
where the ﬁrst inequality holds due to the deﬁnition of (cid:101) L , the second inequality holds due to (C.3) in Lemma C.2.
Since we choose η ≤ C 2 ( tmL + mλ ) − 1 for some small enough C 2 > 0 , then by standard result of gradient descent on ridge linear regression, (cid:101) θ ( j ) converges to θ (0) + (¯ Z ) − 1 ¯ b / √ m with the convergence rate
[Equation]
E A V ARIANT OF N EURAL UCB
In this section, we present a variant of NeuralUCB called NeuralUCB 0 .
Compared with Algorithm 1, the main differences between NeuralUCB and NeuralUCB 0 are as follows: NeuralUCB uses gradient descent to train a deep neural network to learn the reward function h ( x ) based on observed contexts and rewards.
In contrast, NeuralUCB 0 uses matrix inversions to obtain parameters in closed forms.
At each round, NeuralUCB uses the current DNN parameters ( θ t ) to compute an upper conﬁdence bound.
In contrast, NeuralUCB 0 computes the UCB using the initial parameters ( θ 0 ).
1: Input
2: Initialization: Generate each entry of W l independently from N (0 , 2 /m ) for 1 ≤ l ≤ L − 1 , and each entry of W L independently from N (0 , 1 /m ) .
Deﬁne φ ( x ) =
g ( x ; θ 0 ) / √ m , where
Observe { t a }
[Equation]
(cid:105) (E.1) 6:
Play a t and receive reward r t,a t 7:
Compute
[Equation]
Compute θ t
=
Z − 1 t b t + θ 0 ∈ R p 9:
Construct C t as
C t = { θ : (cid:107) θ t − θ (cid:107) Z t ≤ γ t } , where γ t = ν (cid:114) log det Z t det
λ
I − 2 log δ + √ λS (E.2) 10: end for