N EURAL T HOMPSON S AMPLING
A BSTRACT
Thompson Sampling (TS) is one of the most effective algorithms for solving contextual multi-armed bandit problems.
In this paper, we propose a new algorithm, called Neural Thompson Sampling, which adapts deep neural networks for both exploration and exploitation.
At the core of our algorithm is a novel posterior distribution of the reward, where its mean is the neural network approximator, and its variance is built upon the neural tangent features of the corresponding neural network.
We prove that, provided the underlying reward function is bounded, the proposed algorithm is guaranteed to achieve a cumulative regret of O ( T 1 / 2 ) , which matches the regret of other contextual bandit algorithms in terms of total round number T .
Experimental comparisons with other benchmark bandit algorithms on various data sets corroborate our theory.
I NTRODUCTION
The stochastic multi-armed bandit (Bubeck & Cesa-Bianchi, 2012; Lattimore & Szepesv ´ ari, 2020) has been extensively studied, as an important model to optimize the trade-off between exploration and exploitation in sequential decision making.
Among its many variants, the contextual bandit is widely used in real-world applications such as recommendation (Li et al., 2010), advertising (Graepel et al., 2010), robotic control (Mahler et al., 2016), and healthcare (Greenewald et al., 2017).
In each round of a contextual bandit, the agent observes a feature vector (the “context”) for each of the K arms, pulls one of them, and in return receives a scalar reward.
The goal is to maximize the cumulative reward, or minimize the regret (to be deﬁned later), in a total of T rounds.
To do so, the agent must ﬁnd a trade-off between exploration and exploitation.
One of the most effective and widely used techniques is Thompson Sampling , or TS (Thompson, 1933).
The basic idea is to compute the posterior distribution of each arm being optimal for the present context, and sample an arm from this distribution.
TS is often easy to implement, and has found great success in practice (Chapelle & Li, 2011; Graepel et al., 2010; Kawale et al., 2015; Russo et al., 2017).
Recently, a series of work has applied TS or its variants to explore in contextual bandits with neural network models (Blundell et al., 2015; Kveton et al., 2020; Lu & Van Roy, 2017; Riquelme et al., 2018).
Riquelme et al.
(2018) proposed NeuralLinear, which maintains a neural network and chooses the best arm in each round according to a Bayesian linear regression on top of the last network layer.
Kveton et al.
(2020) proposed DeepFPL, which trains a neural network based on perturbed training data and chooses the best arm in each round based on the neural network output.
Similar approaches have also been used in more general reinforcement learning problem (e.g.,
Azizzadenesheli et al., 2018;
Fortunato et al., 2018; Lipton et al., 2018; Osband et al., 2016a).
Despite the reported empirical success, strong regret guarantees for TS remain limited to relatively simple models, under fairly restrictive assumptions on the reward function.
Examples are linear functions (Abeille & Lazaric, 2017; Agrawal & Goyal, 2013; Koc´ak et al., 2014; Russo & Van Roy, 2014), generalized linear functions (Kveton et al., 2020; Russo & Van Roy, 2014), or functions with small RKHS norm induced by a properly selected kernel (Chowdhury & Gopalan, 2017).
In this paper, we provide, to the best of our knowledge, the ﬁrst near-optimal regret bound for neural network-based Thompson Sampling.
Our contributions are threefold.
First, we propose a new algorithm, Neural Thompson Sampling (NeuralTS) , to incorporate TS exploration with neural networks.
It differs from NeuralLinear (Riquelme et al., 2018) by considering weight uncertainty in all layers, and from other neural network-based TS implementations (Blundell et al., 2015; Kveton et al., 2020) by sampling the estimated reward from the posterior (as opposed to sampling parameters).
Second, we give a regret analysis for the algorithm, and obtain an (cid:101) O
( (cid:101) d √ T ) regret, where (cid:101) d is the effective dimension and T is the number of rounds.
This result is comparable to previous bounds when specialized to the simpler, linear setting where the effective dimension coincides with the feature dimension (Agrawal & Goyal, 2013; Chowdhury & Gopalan, 2017).
Finally, we corroborate the analysis with an empirical evaluation of the algorithm on several benchmarks.
Experiments show that NeuralTS yields competitive performance, in comparison with stateof-the-art baselines, thus suggest its practical value in addition to strong theoretical guarantees.
Notation: Scalars and constants are denoted by lower and upper case letters, respectively.
Vectors are denoted by lower case bold face letters x , and matrices by upper case bold face letters A .
We denote by [ k ] the set { 1 , 2 , · · · , k } for positive integers k .
For two non-negative sequence { a n } , { b n } , a n = O ( b n ) means that there exists a positive constant C such that a n ≤ Cb n , and we use (cid:101) O ( · ) to hide the log factor in O ( · ) .
We denote by (cid:107) · (cid:107) 2 the Euclidean norm of vectors and the spectral norm of matrices, and by (cid:107) · (cid:107) F the Frobenius norm of a matrix.
P ROBLEM S ETTING AND P ROPOSED A LGORITHM
In this work, we consider contextual K -armed bandits, where the total number of rounds T is known.
At round t ∈ [ T ] , the agent observes K contextual vectors { x t,k ∈ R d
| k ∈
[ K ] } .
Then the agent selects an arm a t and receives a reward r t,a t .
Our goal is to minimize the following pseudo regret:
[Equation]
where a ∗ t is the optimal arm at round t that has the maximum expected reward: a ∗ t = argmax a ∈ [ K ] E [ r t,a ] .
To estimate the unknown reward given a contextual vector x , we use a fully connected neural network f ( x ; θ ) of depth L ≥ 2 , deﬁned recursively by
[Equation]
where ReLU( x ) :
= max { x, 0 } , m is the width of neural network, W 1 ∈ R m × d , W l ∈ R m × m , 2 ≤ l < L , W L ∈ R 1 × m , θ = (vec( W 1 ); · · · ; vec( W L ))
∈ R p is the collection of parameters of the neural network, p = dm + m 2 ( L − 2) + m , and g ( x ; θ ) = ∇ θ f ( x ; θ ) is the gradient of f ( x ; θ ) w.r.t.
θ .
Our Neural Thompson Sampling is given in Algorithm 1.
It maintains a Gaussian distribution for each arm’s reward.
When selecting an arm, it samples the reward of each arm from the reward’s posterior distribution, and then pulls the greedy arm (lines 4–8).
Once the reward is observed, it updates the posterior (lines 9 & 10).
The mean of the posterior distribution is set to the output of the neural network, whose parameter is the solution to the following minimization problem:
[Equation]
We can see that (2.3) is an (cid:96) 2 -regularized square loss minimization problem, where the regularization term centers at the randomly initialized network parameter θ 0 .
We adapt gradient descent to solve (2.3) with step size η and total number of iterations J .
A few observations about our algorithm are in place.
First, compared to typical ways of implementing Thompson Sampling with neural networks, NeuralTS samples from the posterior distribution of the scalar reward , instead of the network parameters.
It is therefore simpler and more efﬁcient, as the number of parameters in practice can be large.
Second, the algorithm maintains the posterior distributions related to parameters of all layers of the network, as opposed to the last layer only (Riquelme et al., 2018).
This difference is crucial in our regret analysis.
It allows us to build a connection between Algorithm 1 and recent work about deep learning theory (Allen-Zhu et al., 2018; Cao & Gu, 2019), in order to obtain theoretical guarantees as will be shown in the next section.
Third, different from linear or kernelized TS (Agrawal & Goyal, 2013; Chowdhury & Gopalan, 2017), whose posterior can be computed in closed forms, NeuralTS solves a non-convex optimization problem (2.3) by gradient descent.
This difference requires additional techniques in the regret analysis.
Moreover, stochastic gradient descent can be used to solve the optimization problem with a similar theoretical guarantee (Allen-Zhu et al., 2018; Du et al., 2018;
Zou et al., 2019).
For simplicity of exposition, we will focus on the exact gradient descent approach.
R EGRET A NALYSIS
In this section, we provide a regret analysis of NeuralTS.
We assume that there exists an unknown reward function h such that for any 1 ≤ t ≤ T and 1 ≤ k ≤ K ,
[Equation]
t, [Equation] with | h ( x t,k ) | ≤where { ξ t,k } forms an R -sub-Gaussian martingale difference sequence with constant R > 0 , i.e., E [exp( λξ t,k ) | ξ 1: t − 1 ,k , x 1: t,k ] ≤ exp( λ 2 R 2 ) for all λ ∈ R .
Such an assumption on the noise sequence is widely adapted in contextual bandit literature (Agrawal & Goyal, 2013; Bubeck & CesaBianchi, 2012; Chowdhury & Gopalan, 2017; Chu et al., 2011; Lattimore & Szepesv´ari, 2020; Valko et al., 2013).
Next, we provide necessary background on the neural tangent kernel (NTK) theory (Jacot et al., 2018), which plays a crucial role in our analysis.
In the analysis, we denote by { x i } TKi =1 the set of observed contexts of all arms and all rounds: { x t,k } 1 ≤ t ≤ T, 1 ≤ k ≤ K where i =
K
( t − 1)
+ k .
Deﬁnition 3.1 (Jacot et al.
Deﬁne
[Equation]
Then, H =
( (cid:101) H ( L ) + Σ ( L ) ) / 2 is called the neural tangent kernel matrix on the context set.
The NTK technique builds a connection between deep neural networks and kernel methods.
It enables us to adapt some complexity measures for kernel methods to describe the complexity of the neural network, as given by the following deﬁnition.
Deﬁnition 3.2.
The effective dimension (cid:101) d of matrix H with regularization parameter λ is deﬁned as
[Equation]
Remark 3.3.
The effective dimension is a metric to describe the actual underlying dimension in the set of observed contexts, and has been used by Valko et al.
for the analysis of kernel UCB.
Our deﬁnition here is adapted from Yang & Wang (2019), which also considers UCB-based exploration.
Compared with the maximum information gain γ t used in Chowdhury & Gopalan (2017), one can verify that their Lemma 3 shows that γ t ≥ log det( I + H /λ ) / 2 .
Therefore, γ t and (cid:101) d are of the same order up to a ratio of 1 / (2 log(1 + TK/λ )) .
Furthermore, (cid:101) d can be upper bounded if all contexts x i are nearly on some low-dimensional subspace of the RKHS space spanned by NTK (Appendix D).
We will make a regularity assumption on the contexts and the corresponding NTK matrix H .
Assumption 3.4.
Let H be deﬁned in Deﬁnition 3.1.
There exists λ 0 > 0 , such that H (cid:23) λ 0 I .
In addition, for any t ∈ [ T ] , k ∈ [ K ] , (cid:107) x t,k (cid:107) 2 = 1 and [ x t,k ] j
=
[ x t,k ] j + d/ 2 .
The assumption that the NTK matrix is positive deﬁnite has been considered in prior work on NTK (Arora et al., 2019; Du et al., 2018).
The assumption on context x t,a ensures that the initial output of neural network f ( x ; θ 0 ) is 0 with the random initialization suggested in Algorithm 1.
The condition on x is easy to satisfy, since for any context x , one can always construct a new context (cid:101) x as [ x / ( √ 2 (cid:107) x (cid:107) 2 ) , x / ( √ 2 (cid:107) x (cid:107) 2 )] (cid:62) .
We are now ready to present the main result of the paper:
Theorem 3.5.
Under Assumption 3.4, set the parameters in Algorithm 1 as λ = 1 + 1 /T , ν =
B + R (cid:113)(cid:101) d log(1 + TK/λ ) + 2 + 2 log(1 /δ )
where B = max (cid:110) 1 / (22e √ π ) , √ 2 h (cid:62) H − 1 h (cid:111) with h =
( h ( x 1 ) , . . .
, h ( x TK )) (cid:62) , and R is the sub-Gaussian parameter.
In line 9 of Algorithm 1, set η = C 1 ( mλ + mLT ) − 1 and J =
(1 + LT/λ )
( C 2 + log
( T 3 Lλ − 1 log(1 /δ ))) /C
1 for some positive constants C 1 , C 2 .
If the network width m satisﬁes:
[Equation]
then, with probability at least 1 − δ , the regret of Algorithm 1 is bounded as
[Equation]
where C 3 , C 4 are some positive absolute constants, and c T = √ 4 log T + 2 log K .
Remark 3.6.
The deﬁnition B in Theorem 3.5 is inspired by the RKHS norm of the reward function deﬁned in Chowdhury & Gopalan (2017).
It can be veriﬁed that when the reward function h belongs to the function space induced by NTK, i.e., (cid:107) h (cid:107) H < ∞ , we have √ h (cid:62) H − 1 h ≤ (cid:107) h (cid:107) H according to
Zhou et al.
(2019), which suggests that B ≤ max { 1 / (22 e √ π ) , √ 2 (cid:107) h (cid:107) H } .
Remark 3.7.
Theorem 3.5 implies the regret of NeuralTS is on the order of (cid:101) O
( (cid:101) dT 1 / 2 ) .
This result matches the state-of-the-art regret bound in Chowdhury & Gopalan (2017)
; Agrawal & Goyal (2013); Zhou et al.
; Kveton et al.
Remark 3.8.
In Theorem 3.5, the requirement of m is speciﬁed in Condition 4.1 and the proof of Theorem 3.5, which is a high-degree polynomial in the time horizon T , number of layers L and number of actions K .
However, in our experiments, we can choose reasonably small m (e.g., m = 100 ) to obtain good performance of NeuralTS.
See Appendix A.1 for more details.
This discrepancy between theory and practice is due to the limitation of current NTK theory (Du et al., 2018; Allen-Zhu et al., 2018; Zou et al., 2019).
Closing the gap is a venue for future work.
Remark 3.9.
Theorem 3.5 suggests that we need to know T before we run the algorithm in order to set m .
When T is unknown, we can use the standard doubling trick (See e.g., Cesa-Bianchi &
Lugosi (2006)) to set m adaptively.
In detail, we decompose the time interval (0 , + ∞ ) as a union of non-overlapping intervals [2 s , 2 s +1 ) .
When 2 s ≤ t < 2 s +1 , we restart NeuralTS with the input
T = 2 s +1 .
It can be veriﬁed that similar (cid:101) O
( (cid:101) d √ T
P ROOF OF THE M AIN T HEOREM
This section sketches the proof of Theorem 3.5, with supporting lemmas and technical details provided in Appendix B. While the proof roadmap is similar to previous work on Thompson Sampling (e.g., Agrawal & Goyal, 2013; Chowdhury & Gopalan, 2017; Koc´ak et al., 2014; Kveton et al., 2020), our proof needs to carefully track the approximation error of neural networks for approximating the reward function.
To control the approximation error, the following condition on the neural network width is required in several technical lemmas.
Condition 4.1.
The network width m satisﬁes
[Equation]
where C is a positive absolute constant.
For any t , we deﬁne an event E σt as follows
[Equation]
where c t = √ 4 log t + 2 log K .
Under event E σt , the difference between the sampled reward (cid:101) r t,k and the estimated mean reward f ( x t,k ; θ t − 1 ) can be controlled by the reward’s posterior variance.
We also deﬁne an event E µt as follows
[Equation]
where (cid:15) ( m ) is deﬁned as
[Equation]
and { C (cid:15),i } 4 i =1 are some positive absolute constants.
Under event E µt , the estimated mean reward f ( x t,k ; θ t − 1 ) based on the neural network is similar to the true expected reward h ( x t,k ) .
Note that the additional term (cid:15) ( m ) is the approximate error of the neural networks for approximating the true reward function.
This is a key difference in our proof from previous regret analysis of Thompson
Sampling Agrawal & Goyal (2013); Chowdhury & Gopalan (2017), where there is no approximation error.
The following two lemmas show that both events E σt and E µt happen with high probability.
Lemma 4.2.
For any t ∈ [ T ] , Pr (cid:0) E σt (cid:12)(cid:12) F t (cid:1) ≥ 1 − t − 2 .
Lemma 4.3.
Suppose the width of the neural network m satisﬁes Condition 4.1.
Set η = C
( mλ + mLT ) − 1 , then we have Pr (cid:0) ∀ t ∈
[ T ] , E µt (cid:1) ≥ 1 − δ , where C is an positive absolute constant.
The next lemma gives a lower bound of the probability that the sampled reward (cid:101) r is larger than true reward up to the approximation error (cid:15) ( m ) .
Lemma 4.4.
For any t ∈ [ T ] , k ∈
[ K ] , we have Pr (cid:0)(cid:101) r t,k + (cid:15) ( m ) >
h ( x t,k ) (cid:12)(cid:12) F t , E µt (cid:1) ≥ (4e √ π ) − 1 .
Following Agrawal & Goyal (2013), for any time t , we divide the arms into two groups: saturated and unsaturated arms, based on whether the standard deviation of the estimates for an arm is smaller than the standard deviation for the optimal arm or not.
Note that the optimal arm is included in the group of unsaturated arms.
More speciﬁcally, we deﬁne the set of saturated arms S t as follows
[Equation]
Note that we have taken the approximate error (cid:15) ( m ) into consideration when deﬁning saturated arms, which differs from the Thompson Sampling literature (Agrawal & Goyal, 2013; Chowdhury & Gopalan, 2017).
It is now easy to show that the immediate regret of playing an unsaturated arm can be bounded by the standard deviation plus the approximation error (cid:15) ( m ) .
The following lemma shows that the probability of pulling a saturated arm is small in Algorithm 1.
Lemma 4.5.
Let a t be the arm pulled at round t ∈ [ T ] .
Then, Pr (cid:0) a t / ∈ S t |F t , E µt (cid:1) ≥ 1 4e √ π [Equation]
[Equation]
The next lemma bounds the expectation of the regret at each round conditioned on E µt .
Lemma 4.6.
Suppose the width of the neural network m satisﬁes Condition 4.1.
Set η = C 1 ( mλ + mLT ) − 1 , then with probability at least 1 − δ , we have for all t ∈
[ T ] that
[Equation]
where C 1 , C 2 are some positive absolute constants.
Based on Lemma 4.6, we deﬁne ¯∆ t :=
( h ( x t,a ∗ t ) − h ( x t,a t [Equation]
[Equation] µt ) , and
[Equation]
where C ∆ is the same with constant C in Lemma 4.6.
By Lemma 4.6, we can verify that with probability at least 1 − δ , { Y t } forms a super martingale sequence since E ( Y t − Y t − 1 ) =
E X t ≤ 0 .
By Azuma-Hoeffding inequality (Hoeffding, 1963), we can prove the following lemma.
Lemma 4.7.
Suppose the width of the neural network m satisﬁes Condition 4.1.
Then set η =
C 1 ( mλ + mLT ) − 1 , we have, with probability at least 1 − δ , that
[Equation]
where C 1 , C 2 , C 3 are some positive absolute constants.
The last lemma is used to control (cid:80)
Ti =1 min { σ t,a t , 1 } in Lemma 4.7.
Lemma 4.8.
Suppose the width of the neural network m satisﬁes Condition 4.1.
Then set η =
C 1 ( mλ + mLT ) − 1 , we have, with probability at least 1 − δ , it holds that
[Equation]
where C 1 , C 2 are some positive absolute constants.
With all the above lemmas, we are ready to prove Theorem 3.5.
Proof of Theorem 3.5.
By Lemma 4.3, E µt holds for all t ∈
[ T ] with probability at least 1 − δ .
Therefore, with probability at least 1 − δ , we have
[Equation]
) ν √ L (cid:18)(cid:113) 2 λT ( (cid:101)
d log(1 + TK ) + 1)
+ ¯ C 3 T 13 / 6 (cid:112) log mm − 1 / 6 λ − 2 / 3 L 9 / 2 (cid:19) + π3 + 4 T(cid:15)
( m ) + 4 (cid:15) ( m ) (cid:112) 2 log(1 /δ )
T + (4 + ¯ C 2 (1 + c T ) νL ) (cid:112) 2 log(1 /δ )
T, = ¯ C 1 (1 + c T ) ν √ L (cid:18) (cid:113) 2 λT ( (cid:101)
d log(1 + TK ) + 1)
+ ¯ C 3 T 13 / 6 (cid:112) log mm − 1 / 6 λ − 2 / 3 L 9 / 2 (cid:19)
+ π3 + (cid:15) p ( m )(4 T + (cid:112)
2 log(1 /δ ) T ) + (4 + ¯ C 2 (1 + c T ) νL ) (cid:112) 2 log(1 /δ )
T + C (cid:15), 1 (1 − ηmλ )
J (cid:112) TL/λ (4 T
+ (cid:112) 2 log(1 /δ ) T ) , where ¯ C 1 , ¯ C 2 , ¯ C 3 are some positive absolute constants , the ﬁrst inequality is due to Lemma 4.7, and the second inequality is due to Lemma 4.8.
The third equation is from (4.3).
By setting η = ¯ C 4 ( mλ + mLT ) − 1 and J =
(1 + LT/λ )
(log(24 C (cid:15), 1 ) + log( T 3 Lλ − 1 log(1 /δ )))
/ ¯ C 4 , we have
[Equation]
[Equation]
J
[Equation]
Then choosing m such that
[Equation]
R T can be further bounded by
[Equation]
Taking union bound over Lemmas 4.3, 4.7 and 4.8, the above inequality holds with probability 1 − 3 δ .
By replacing δ with δ/ 3 and rearranging terms, we complete the proof.
E XPERIMENTS
This section gives an empirical evaluation of our algorithm in several public benchmark datasets, including adult , covertype , magic telescope , mushroom and shuttle , all from
UCI (Dua & Graff, 2017), as well as MNIST (LeCun et al., 2010).
The algorithm is compared to several typical baselines: linear and kernelized Thompson Sampling (Agrawal & Goyal, 2013;
Chowdhury & Gopalan, 2017), linear and kernelized UCB (Chu et al., 2011; Valko et al., 2013),
BootstrapNN (Osband et al., 2016b; Riquelme et al., 2018), and (cid:15) -greedy for neural networks.
BootstrapNN trains multiple neural networks with subsampled data, and at each step pulls the greedy action based on a randomly selected network.
It has been proposed as a way to approximate Thompson
Sampling (Osband & Van Roy, 2015; Osband et al., 2016b).
5. E XPERIMENT SETUP
To transform these classiﬁcation problems into multi-armed bandits, we adapt the disjoint models (Li et al., 2010) to build a context feature vector for each arm: given an input feature x ∈ R d of a k -class classiﬁcation problem, we build the context feature vector with dimension kd as: x 1 = (cid:0) x ; 0 ; · · · ; 0 (cid:1) , x 2 = (cid:0) 0 ; x ; · · · ; 0 (cid:1) , · · · , x k = (cid:0) 0 ; 0 ; · · · ; x (cid:1) .
Then, the algorithm generates a set of predicted reward following Algorithm 1 and pulls the greedy arm.
For these classiﬁcation problems, if the algorithm selects a correct class by pulling the corresponding arm, it will receive a reward as 1 , otherwise 0 .
The cumulative regret over time horizon T is measured by the total mistakes made by the algorithm.
All experiments are repeated 8 times with reshufﬂed data.
We set the time horizon of our algorithm to 10 000 for all data sets, except for mushroom which contains only 8 124 data.
In order to speed up training for the NeuralUCB and Neural Thompson
Sampling, we use the inverse of the diagonal elements of U as an approximation of U − 1 .
Also, since calculating the kernel matrix is expensive, we stop training at t = 1000 and keep evaluating the performance for the rest of the time, similar to previous work (Riquelme et al., 2018;
Zhou et al., 2019).
Due to space limit, we defer the results on adult , covertype and magic telescope , as well as further experiment details, to Appendix A. In this section, we only show the results on mushroom , shuttle and MNIST .
5. E XPERIMENT I: P ERFORMANCE OF N EURAL T HOMPSON S AMPLING
The experiment results of Neural Thompson Sampling and other benchmark algorithms are shown in Figure 1.
A few observations are in place.
First, Neural Thompson Sampling’s performance is among the best in 6 datasets and is signiﬁcantly better than all other baselines in 2 of them.
Second, the function class used by an algorithm is important.
Those with linear representations tend to perform worse due to the nonlinearity of rewards in the data.
Third, Thompson Sampling is competitive with, and sometimes better than, other exploration strategies with the same function class, in particular when neural networks are used.
[Figure]
5. E XPERIMENT II: R OBUSTNESS TO R EWARD D ELAY
This experiment is inspired by practical scenarios where reward signals are delayed, due to various constraints, as described by Chapelle & Li (2011).
We study how robust the two most competitive methods from Experiment I, Neural UCB and Neural Thompson Sampling
[Figure]
are when rewards are delayed.
More speciﬁcally, the reward after taking an action is not revealed immediately
[Figure]
but arrive in batches when the algorithms will update their models.
The experiment setup is otherwise identical to Experiment I.
Here, we vary the batch size (i
[Figure]
e., the amount of reward delay)
[Figure]
and Figure 2 shows the corresponding total regret.
Clearly
[Figure]
we recover the result in Experiment I when the delay is 0 .
Consistent with previous ﬁndings (Chapelle & Li, 2011)
[Figure]
Neural TS degrades much more gracefully than Neural UCB when the reward delay increases.
The beneﬁt may be explained by the algorithm’s randomized exploration nature that encourages exploration between batches.
We, therefore, expect wider applicability of Neural TS in practical applications
[Figure]
R e g r e t
[Figure]
T o t a l R e g r e t
[Figure]
R ELATED W ORK
Thompson Sampling was proposed as an exploration heuristic almost nine decades ago (Thompson, 1933), and has received signiﬁcant interest in the last decade.
Previous works related to the present paper are discussed in the introduction, and are not repeated here.
Upper conﬁdence bound or UCB (Agrawal, 1995; Auer et al., 2002; Lai & Robbins, 1985) is a widely used alternative to Thompson Sampling for exploration.
This strategy is shown to achieve near-optimal regrets in a range of settings, such as linear bandits (Abbasi-Yadkori et al., 2011; Auer, 2002;
Chu et al., 2011), generalized linear bandits (Filippi et al., 2010;
Jun et al., 2017; Li et al., 2017), and kernelized contextual bandits (Valko et al., 2013).
Neural networks are increasingly used in contextual bandits.
In addition to those mentioned earlier (Blundell et al., 2015; Kveton et al., 2020; Lu & Van Roy, 2017; Riquelme et al., 2018), Zahavy
& Mannor (2019) used a deep neural network to provide a feature mapping and explored only at the last layer.
Schwenk & Bengio (2000) proposed an algorithm by boosting the estimation of multiple deep neural networks.
While these methods all show promise empirically, no regret guarantees are known.
Recently, Foster & Rakhlin (2020) proposed a special regression oracle and randomized exploration for contextual bandits with a general function class (including neural networks) along with theoretical analysis.
Zhou et al.
(2019) proposed a neural UCB algorithm with near-optimal regret based on UCB exploration, while this paper focuses on Thompson Sampling.
C ONCLUSIONS
In this paper, we adapt Thompson Sampling to neural networks.
Building on recent advances in deep learning theory, we are able to show that the proposed algorithm, NeuralTS, enjoys a (cid:101) O
( (cid:101) dT 1 / 2 ) regret bound.
We also show the algorithm works well empirically on benchmark problems, in comparison with multiple strong baselines.
The promising results suggest a few interesting directions for future research.
First, our analysis needs NeuralTS to perform multiple gradient descent steps to train the neural network in each round.
It is interesting to analyze the case where NeuralTS only performs one gradient descent step in each round, and in particular, the trade-off between optimization precision and regret minimization.
Second, when the number of arms is ﬁnite, (cid:101) O ( √ dT ) regret has been established for parametric bandits with linear and generalized linear reward functions.
It is an open problem how to adapt
NeuralTS to achieve the same rate.
Third, Allen-Zhu & Li (2019) suggested that neural networks may behave differently from a neural tangent kernel under some parameter regimes.
It is interesting to investigate whether similar results hold for neural contextual bandit algorithms like NeuralTS.
A CKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their helpful comments.
WZ, DZ and QG are partially supported by the National Science Foundation CAREER Award 1906169 and IIS-1904183.
The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
A F URTHER D ETAIL OF THE E XPERIMENTS IN S ECTIONA. P ARAMETER T UNING
In the experiments, we shufﬂe all datasets randomly, and normalize the features so that their (cid:96) 2norm is unity.
One-hidden-layer neural networks with 100 neurons are used.
Note that we do not choose m as suggested by theory, and such a disconnection has its root in the current deep learning theory based on neural tangent kernel, which is not speciﬁc in this work.
During posterior updating, gradient descent is run for 100 iterations with learning rate 0 .
For BootstrapNN, we useidentical networks, and to train each network, data point at each round has probability 0 .
8 to be included for training ( p = 10 , q = 0 .
8 in the original paper (Schwenk & Bengio, 2000))
For (cid:15) -
Greedy, we tune (cid:15) with a grid search on { 0 . 01 , 0 . 05 , 0 . 1 } .
For ( λ, ν ) used in linear and kernel
UCB / Thompson Sampling, we set λ
= 1 following previous works (Agrawal & Goyal, 2013;
Chowdhury & Gopalan, 2017), and do a grid search of ν ∈ { 1 , 0 . 1 , 0 . 01 } to select the parameter with best performance.
For the Neural UCB / Thompson Sampling methods, we use a grid search on λ ∈ { 1 , 10 − 1 , 10 − 2 , 10 − 3 } and ν ∈ { 10 − 1 , 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 } .
All experiments are repeated 20 times, and the average and standard error are reported.
A. D ETAILED R ESULTS
Table 1 summarizes the total regrets measured at the last round on different data sets, with mean and standard deviation error computed based on 20 independent runs.
The Bold Faced data is the top performance over 8 experiments.
Table 2 shows the number of times the algorithm in that row signiﬁcantly outperforms, ties, or signiﬁcantly underperforms, compared with other algorithm with t -test at 90% signiﬁcance level.
Figure 3 shows the performance of Neural Thompson Sampling compared with other baseline method.
Figure 4 shows the comparison between Neural Thompson
Sampling and Neural UCB in delay reward settings
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Figure] [Figure]
[Figure]
A. R UN TIME ANALYSIS
We compare the run time of the four algorithms based on neural networks: BootstrapNN, (cid:15) -greedy for neural networks, NeuralUCB
[Figure]
and NeuralTS.
The comparison is shown in Figure 5.
We can see that NeuralTS and NeuralUCB are about 2 to 3 times slower than (cid:15) -greedy, which is due to the extra calculation of the neural network gradient for each input context.
BootstrapNN is often more thantimes slower than (cid:15) -greedy because it has to train several neural networks at each round
[Figure]
B P ROOF OF L EMMAS IN S ECTION
Under Condition 4.1, we can show that the following inequalities hold.
[Equation]
where { C m, 1 , C m, 2 , . . .
, C m, 4 } are some positive absolute constants.
B. P ROOF OF L EMMA 4.
The following concentration bound on Gaussian distributions will be useful in our proof.
Lemma B.1 (Hoffman et al.
Consider a normally distributed random variable X ∼
N ( µ, σ 2 ) and β ≥ 0 .
The probability that X is within a radius of βσ from its mean can then be written as
[Equation]
Proof of Lemma 4.2.
Since the estimated reward (cid:101) r t,k is sampled from N ( f ( x t,k ; θ t − 1 ) , ν 2 σ 2 t,k ) if given ﬁltration F t , Lemma B.1 implies that, conditioned on F t and given t, k ,
[Equation]
Taking a union bound over K arms, we have that for any t
[Equation]
[Table]
Finally, choose c t = √ 4 log t + 2 log K as deﬁned in (4.1), we get the bound that
[Equation]
B. P ROOF OF L EMMA 4.
Before going into the proof, some notation is needed about linear and kernelized models.
Deﬁnition B.2.
Deﬁne ¯ U t
=
λ
I + (cid:80) ti =1 g ( x i,a i ; θ 0 )
g
( x t,a t ; θ 0 ) (cid:62) /m and based on ¯ U t , we further deﬁne ¯ σ 2 t,k = λ g (cid:62) ( x t,k ; θ 0 )
¯ U − 1 t − 1 g ( x t,k ; θ 0 ) /m .
Furthermore, for convenience we deﬁne
[Equation]
where (cid:15) t is the reward noise.
We can verify that U t
= λ
I + J t J (cid:62) t /m , ¯ U t
= λ
I + ¯ J t ¯ J (cid:62) t /m .
We further deﬁne K t
= ¯ J (cid:62)
[Equation]
The ﬁrst lemma shows that the target function is well-approximated by the linearized neural network if the network width m is large enough.
Lemma B.3 (Lemma 5.1, Zhou et al.
There exists some constant C > 0 such that for any δ ∈ (0 , 1) , if
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , there exists a θ ∗ ∈ R p such that
[Equation]
for all i ∈ [ TK ] , where B is deﬁned in Theorem 3.5.
From Lemma B.3, it is easy to show that under this initialization parameter θ 0 , we have that h t = ¯
J (cid:62) t ( θ ∗ − θ 0 )
The next lemma bounds the difference between the ¯ σ t,k from the linearized model and the σ t,k actually used in the algorithm.
Its proof, together with other technical lemmas’, will be given in the next section.
Lemma B.4.
Suppose the network size m satisﬁes Condition 4.1.
Set η = C 1 ( mλ + mLT ) − 1 , then with probability at least 1 − δ ,
[Equation]
where C 1 , C 2 are two positive constants.
We next bound the difference between the outputs of the neural network and the linearized model.
Lemma B.5.
Suppose the network width m satisﬁes Condition 4.1.
Then, set η = C 1 ( mλ + mLT ) − 1 , with probability at least 1 − δ over the random initialization of θ 0 , we have (cid:12)(cid:12)
[Equation]
where { C i } 4 i =1 are positive constants.
The next lemma, due to Chowdhury & Gopalan (2017), controls the quadratic value generated by an R -sub-Gaussian random vector (cid:15) :
Lemma B.6 (Theorem 1, Chowdhury & Gopalan (2017)) .
Let { (cid:15) t } ∞ t =1 be a real-valued stochastic process such that for some R ≥ 0 and for all t ≥ 1 , (cid:15) t is F t -measurable and R -sub-Gaussian conditioned on F t , Recall K t deﬁned in Deﬁnition B.2.
With probability 0 < δ < 1 and for a given η > 0 , with probability 1 − δ , the following holds for all t ,
[Equation]
Finally, the following lemma shows the linearized kernel and the neural tangent kernel are closed:
Lemma B.7.
For all t ∈ [ T ] , there exists a positive constants C such that the following holds: if the network width m satisﬁes
[Equation]
then with probability at least 1 − δ ,
[Equation]
We are now ready to prove Lemma 4.3.
Proof of Lemma 4.3.
First of all, since m satisﬁes Condition 4.1, then with the choice of η ,the condition required in Lemmas B.3–B.7 are satisﬁed.
Thus, taking a union bound, we have with probability at least 1 − 5 δ , that the bounds provided by these lemmas hold.
Then for any t ∈ [ T ] , we will ﬁrst provide the difference between the target function and the linear function (cid:10) g
[Equation]
t −
[Equation]
θ ∗
[Equation]
[Equation]
≤
[Equation]
where the ﬁrst inequality uses triangle inequality and the fact that r t − 1 =
h t − 1 + (cid:15) t − 1 ; the ﬁrst equality is from Lemma B.3 and the second equality uses the fact that ¯ J t − 1 ¯ J (cid:62) t − 1 = m
( ¯ U t − 1 − λ I ) which can be veriﬁed using Deﬁnition B.2; the second inequality is from the fact that | α (cid:62) A β | ≤ √ α (cid:62)
A α (cid:112) β (cid:62) A β .
Since U − 1 t − 1 (cid:22) 1 λ I and ¯ σ t,k deﬁned in Deﬁnition B.2, we obtain the last inequality.
Furthermore, by obtaining
[Equation]
where the ﬁrst equality is from the Sherman-Morrison formula
[Equation]
and the second equality uses Deﬁnition B.2 and the fact that ( λ I + K t − 1 ) − 1 K t −
=
I − λ
( λ I + K t − 1 ) − 1 which could be veriﬁed by multiplying the LHS and RHS together
[Equation]
we have that
[Equation]
where the second inequality is because λ = 1 + 1 /T ≥ 1 set in Theorem 3.5.
Based on (B.2) and (B.3), by utilizing the bound on (cid:107) θ ∗ − θ (cid:107) 2 provided in Lemma B.3, as well as the bound given in Lemma B.6, and λ ≥ 1 , we have
[Equation]
since it is obvious that
[Equation]
where the ﬁrst equality moves the λ outside the log det , the ﬁrst inequality is due to log λ ≤ λ − 1 , and the second inequality is from Lemma B.7 and the fact that λ
= 1+1 /T (as set in Theorem 3.5).
Thus, we have
[Equation]
where we set ν = B + R (cid:112) log det( I + H /λ )
+ 2 + 2 log(1 /δ ) .
Then, by combining this bound with Lemma B.5, we conclude that there exist positive constants ¯ C 1 , ¯ C 2 , ¯ C 3 so that
[Equation]
Finally
[Equation]
by utilizing the bound of | ¯ σ t
[Equation]
k − σ t
[Equation]
k | provided in Lemma B
[Equation]
[Equation]
we conclude that
[Equation]
where (cid:15) ( m ) is deﬁned by adding all of the additional terms and taking t
= T :
[Equation]
where is exactly the same form deﬁned in (4.3).
By setting δ to δ/ 5 (required by the union bound discussed at the beginning of the proof), we get the result presented in Lemma 4.3.
B. P ROOF OF L EMMA 4.
Our proof requires an anti-concentration bound for Gaussian distribution, as stated below:
Lemma B.8 (Gaussian anti-concentration) .
For a Gaussian random variable X with mean µ and standard deviation σ , for any β > 0 ,
[Equation]
Proof of Lemma 4.4.
Since (cid:101) r t,k ∼ N ( f ( x t,k ; θ t − 1 ) , ν 2 t σ 2 t,k ) conditioned on F t , we have
[Equation]
where the ﬁrst inequality is due to | x | ≥ x , and the second inequality follows from event E µt ,
i
[Equation]
e.,
[Equation]
B. P ROOF OF L EMMA 4.
Proof of Lemma
[Equation]
5. Consider the following two events at round t :
[Equation]
Clearly, A implies B , since a t = argmax k (cid:101) r t,k .
Therefore,
[Equation]
Suppose E µ also holds
[Equation]
then it is easy to show that ∀ k ∈
[ K ] ,
[Equation]
Hence [Equation] for all k ∈ S t
[Equation]
we have that
[Equation]
where we used the deﬁnitions of saturated arms in Deﬁnition 4.4, and of E µt and E σt in (4.1).
Consider the following event
[Equation]
Since E σt implies h ( x t,a ∗ t ) − (cid:15) ( m ) ≥ (cid:101) r t,k , we have that if C , E σt holds, then A holds, i.e. E σt ∩C ⊆
A .
Taking union with ¯ E σt we have that C =
¯ E σt ∪ E σt ∩ C ⊆
A ∪ ¯ E σt , which implies
[Equation]
[Equation] [Equation]
Then, (B.5) implies that
[Equation]
where the ﬁrst inequality is from a ∗ t is a special case of ∀ k ∈ [ K ] , the second inequality is from Lemmas 4.2 and 4.4.
B. P ROOF OF L EMMA 4.
To prove Lemma 4.6, we will need an upper bound bound on δ t,k .
Lemma B.9.
For any time t ∈ [ T ] , k ∈
[ K ] , and δ ∈ (0 , 1) , if the network width m satisﬁes
Condition 4.1, we have, with probability at least 1 − δ , that
[Equation]
where C is a positive constant.
Proof of Lemma 4.6.
Recall that given F t and E µt , the only randomness comes from sampling (cid:101) r t,k for k ∈
[ K ] .
Let ¯ k t be the unsaturated arm with the smallest σ t, · , i.e.
[Equation]
then we have that
[Equation]
where the ﬁrst inequality ignores the case when a t ∈ S t , and the second inequality is from
Lemma 4.5 and the deﬁnition of ¯ k t mentioned above.
If both E σt and E µt hold, then
[Equation]
as proved in equation (B.4).
Thus,
[Equation]
where the ﬁrst inequality is from Deﬁnition 4.4 and ¯ k t / ∈ S t , and the second inequality comes from equation (B.7).
Since a trivial bound on h ( x t,a ∗ t ) − h ( x t,a t ) could be get by h ( x t,a ∗ t ) − h ( x t,a t ) ≤ | h ( x t,a ∗ t ) | + | h ( x t,a t ) | ≤ 2 , then we have
[Equation]
where the inequality on the second line uses the bound provide in (B.8) and the trivial bound of h ( x t,a ∗ t ) − h ( x t,a t ) for the second term plus Lemma 4.2, the inequality on the third line uses the bound of σ t, ¯ k t provide in (B.6), inequality on the forth line is directly calculated by 1 ≤ 4e √ π and
[Equation]
which trivially holds since LHS is negative when t ≤ 4 and when t = 5 , the LHS reach its maximum as ≈
11 < 96 .
36 ≈ RHS .
Noticing that | h ( x ) | ≤ 1 , it is trivial to further extend the bound as
[Equation]
and since we have 1 + c t ≥ 1 and ν =
B + R (cid:112) log det( I + H /λ )
+ 2 + 2 log(1 /δ )
≥ B , recall 22e √ πB ≥ 1 , it is easy to verify the following inequality also holds:
[Equation]
where we use the fact that there exists a constant C 1 such that σ t,a t is bounded by C 1 √ L with probability 1 − δ provided by Lemma B.9.
Merging the positive constant C 1 with 44e √ π , we get the statement in Lemma 4.6.
B. P ROOF OF L EMMA 4.
We start with introducing the Azuma-Hoeffding inequality for super-martingale:
Lemma B.10 (Azuma-Hoeffding Inequality for Super Martingale) .
If a super-martingale Y t , corresponding to ﬁltration F t satisﬁes that | Y t − Y t − 1 | ≤ B t , then for any δ ∈ (0 , 1) , w.p. 1 − δ , we have
[Equation]
Proof of Lemma 4.7.
From Lemma B.9, we have that there exists a positive constant C 1 such that
X t deﬁned in (4.5) is bounded with probability 1 − δ by
[Equation]
where the ﬁrst inequality uses the fact that | a − b | ≤ | a | + | b | ; the second inequality is from
Lemma B.9 and the fact that h ≤ 1 , where C 2 is a positive constant used in Lemma B.9; the third inequality uses the fact that t − 2 ≤ 1 .
Noticing the fact that c t ≤ c T , and from Lemma 4.6, we know that with probability at least 1 − δ , Y t is a super martingale.
From Lemma B.10, we have
[Equation]
Considering the deﬁnition of Y T in (4.5), (B.9) is equivalent to
[Equation]
then by utilizing (cid:80) ∞ i =1 t − 2 =
π 2 / 6 , and merge the constant C 1 with 44e √ π , taking union bound of the probability bound of Lemma 4.6, B.10, B.9, we have the inequality above hold with probability at least 1 − 3 δ .
Re-scaling δ to δ/ 3 and merging the product of C 1 C 2 as a new positive constant leads to the desired result.
B. P ROOF OF L EMMA
We ﬁrst state a technical lemma that will be useful:
Lemma B.11 (Lemma 11, Abbasi-Yadkori et al.
Let { v t } ∞ t =1 be a sequence in R d , and deﬁne V t
= λ
I + (cid:80) ti =
v i v (cid:62) i .
If λ ≥ 1 , then
T (cid:88) i
= min { v (cid:62) t V − 1 t − 1 v t − 1 , 1 } ≤ 2 log det (cid:18)
I + λ −t (cid:88) i
= v i v (cid:62) i (cid:19) .
Proof of Lemma 4.8.
First, recall ¯ σ t,k deﬁned in Deﬁnition B.2 and the bound of ¯ σ t,k − σ t,k provided in Lemma B.4.
We have that there exists a positive constants C 1 such that
[Equation]
where the ﬁrst term in the inequality on the second line is from Cauchy-Schwartz inequality, and the second term is from Lemma B.4.
From Deﬁnition B.2, we have
[Equation]
where the ﬁrst inequality moves the positive parameter λ outside the min operator and uses the deﬁnition of ¯ σ t,k in Deﬁnition B.2, then the second inequality utilizes Lemma B.11, the ﬁrst equality use the deﬁnition of ¯ J t in Deﬁnition B.2, the second equality is from the fact
that det( I + AA (cid:62) ) = det( I + A (cid:62) A ) , and the last equality uses the deﬁnition of K t in Deﬁnition B.2.
From Lemma B.7, we have that log [Equation]
under condition on m and η presented in Theorem 3.5.
By taking a union bound we have, with probability 1 − 2 δ , that
[Equation]
where we use the deﬁnition of (cid:101) d in Deﬁnition 3.2.
Replacing δ with δ/ 2 completes the proof.
C P ROOF OF A UXILIARY L EMMAS IN A PPENDIX B
In this section, we are about to show the proof of the Lemmas used in Appendix B, we will start with the following NTK Lemmas.
Among them, the ﬁrst is to control the difference between the parameter learned via Gradient Descent and the theoretical optimal solution to linearized network.
Lemma C.1 (Lemma B.2, Zhou et al.
There exist constants { C i }
i
=1 > 0 such that for any δ > 0 , if η, m satisfy that for all t ∈ [ T ] ,
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for any t ∈ [ T ] , we have that (cid:107) θ t − 1 − θ
(cid:107) 2 ≤ 2 (cid:112) t/ ( mλ ) and (cid:107) θ [Equation] − [Equation]
−
[Equation]
¯
[Equation]
t − 1 r t −
[Equation]
m (cid:107)≤
[Equation]
1 −
[Equation]
J
[Equation]
mλ
[Equation]
log mL 7 / 2 t 5 / 3 λ − 5 /
[Equation]
1 + (cid:112) t/λ ) .
And the next lemma, controls the difference between the function value of neural network and the linearized model:
Lemma C.2 (Lemma 4.1, Cao & Gu (2019)) .
There exist constants { C i }
i
=1 > 0 such that for any δ > 0 , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for all (cid:101) θ , (cid:98) θ satisfying (cid:107) (cid:101) θ − θ 0 (cid:107) 2 ≤ τ, (cid:107) (cid:98) θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have
[Equation]
Furthermore, to continue with, next lemma is proposed to control the difference between the gradient and the gradient on the initial point.
Lemma C.3 (Theorem 5, Allen-Zhu et al.
There exist constants { C i }
i
=1 > 0 such that for any δ ∈ (0 , 1) , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for all (cid:107) θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have
[Equation]
Also, we need the next lemma to control the gradient norm of the neural network with the help of
NTK.
Lemma C.4 (Lemma B.3, Cao & Gu (2019)) .
There exist constants { C i }
i
=1 > 0 such that for any δ > 0 , if τ satisﬁes that
[Equation]
then with probability at least 1 − δ over the random initialization of θ 0 , for any (cid:107) θ − θ 0 (cid:107) 2 ≤ τ and j ∈ [ TK ] we have (cid:107) g ( x j ; θ ) (cid:107) F ≤ C 3 √ mL .
Finally, as literally shows, we can also provide bounds on the kernel provided by the linearized model and the NTK kernel if the network is width enough.
Lemma C.5 (Lemma B.1, Zhou et al.
Set K = (cid:80) Tt =1 (cid:80)
Kk =1 g ( x t,k ; θ 0 ) g
( x t,k ; θ 0 ) /m , recall the deﬁnition of H in Deﬁnition 3.1,then there exists a constant C 1 such that
[Equation]
we could get that (cid:107) K − H (cid:107) F ≤ TK(cid:15) .
Equipped with these lemmas, we could continue for our proof.
C. P ROOF OF L EMMA B.
Proof of Lemma B.4.
Firstly, set τ = 2 (cid:112) t/ ( mλ ) , then we have the condition on the network m and learning rate η satisfy all of the condition need from Lemma C.1 to Lemma C.5.
Thus from
Lemma C.1, we have that there exists (cid:107) θ t − 1 − θ 0 (cid:107) 2 ≤ τ , thus from Lemma C.4, we have that there exists positive constant ¯ C 1 such that (cid:107) g ( x ; θ t − 1 ) (cid:107) 2 ≤ ¯ C 1 √ mL , (cid:107) g ( x ; θ 0 ) (cid:107) 2 ≤ ¯ C 1 √ mL , consider the function deﬁned as
[Equation]
it is then easy to verify that
[Equation]
then we obtain that the function ψ is deﬁned under the domain (cid:107) a (cid:107) 2 ≤ ¯ C 1 √ L, (cid:107) a i (cid:107) 2 ≤ ¯ C 1 √ L then by taking the derivation w.r.t.
ψ 2 , we have that
[Equation]
a , by taking trace with both side and utilizing tr( AB ) =
tr( BA ) and tr( α (cid:62) β ) = tr( αβ (cid:62) ) , we have that
[Equation]
thus by setting C =
[Equation]
1 for simplicity and decompose C = Q (cid:62) DQ , b =
Qa where D = diag( (cid:37) 1 , · · · , (cid:37) p ) as the eigen-value of C
[Equation]
we have that
[Equation]
where the last inequality is from the fact that C (cid:22) 1 /λ
I
[Equation]
which indicates that all eigen-value (cid:37) i ≤ 1 /λ , for the same reason [Equation] we have
[Equation]
Thus under the domain that (cid:107)
[Equation]
L,
[Equation]
i
[Equation]
¯ C 1 √ L , we have that
[Equation]
Then, Lipschitz continuity implies
[Equation]
m (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(C.1)
By Lemma C.3 with τ = 2 (cid:112) t/mλ , there exist positive constants ¯ C 2 and ¯ C 3 so that each gradient difference in (C.1) is bounded by
[Equation]
Thus, since we obtain that there exists constant C 5 such that
[Equation]
where we use the fact that C 1 = max { ¯ C 3 , ¯ C 3 ¯ C 21 } and L ≥ 1 to merge the ﬁrst term into the summation.
This inequality is based on Lemma C.1, Lemma C.3 and Lemma C.4, thus it holds with probability at least 1 − 3 δ .
Replacing δ with δ/ 3 completes the proof.
C. P ROOF OF L EMMA B.
Proof of Lemma B.5.
Setting τ = 2 (cid:112) t/mλ , we have the condition on the network m and learning rate η satisfy all of the condition needed by Lemmas C.1 to C.5.
From Lemma C.1 we have (cid:107) θ t − 1 − θ
(cid:107) 2 ≤ τ .
Then, by Lemma C.2, there exists a constant C 1 such that
[Equation]
Using the bound on θ t − 1 − θ 0 − ¯ U − 1 t − 1 ¯ J t − 1 r t − 1 /m provided in Lemma C.1 and the norm of gradient bound given in Lemma C.4, we have that there exist positive constants ¯ C 1 , ¯ C 2 such that
[Equation]
x t
[Equation]
k
[Equation]
where C 2 = ¯ C 1 , C
= ¯ C 1 ¯ C 2 .
Combining (C.2) and (C.3)
[Equation]
we have
[Equation]
which holds with probability 1 − 3 δ with a union bound (Lemma C.4, Lemma C.
[Equation]
and Lemma C.2).
Replacing δ with δ/ 3 completes the proof
[Equation]
C. P ROOF OF L EMMA B.
Proof of Lemma B.7.
From the deﬁnition of K t , we have that
[Equation]
where the the ﬁrst inequality is because the double summation on the second line contains more elements than the summation on the ﬁrst line.
The second inequality utilizes the deﬁnition of K in
Lemma C.5 and H in Deﬁnition 3.1, the third inequality is from the convexity of log det( · ) function, and the forth inequality is from the fact that (cid:104) A , B (cid:105) ≤ (cid:107) A (cid:107) F (cid:107) B (cid:107) F .
Then the ﬁfth inequality is from the fact that (cid:107) A (cid:107) F ≤ √ TK (cid:107) A (cid:107) 2 if A ∈ R TK × TK and λ ≥ 0 .
Finally, the sixth inequality utilizes
Lemma C.5 by setting (cid:15) =
( TK ) − 3 / 2 with m ≥ C 1 L 6 T 6 K 6 log( TKL/δ ) , where we conclude our proof.
C. P ROOF OF L EMMA B.
Proof of Lemma B.9.
Set τ in Lemma C.4 as 2 (cid:112) t/ ( mλ ) .
Then the network width m and learning rate η satisfy all of the condition needed by Lemma C.1 to C.5.
Hence, there exists C 1 such that (cid:107) g
( x ; θ ) (cid:107) 2 ≤ (cid:107) g
( x ; θ ) (cid:107) F ≤ C 1 √ mL for all x , since it is easy to verify that U − 1 t
[Equation]
λ − 1 I .
Thus we have that for all t ∈
[ T ] , k ∈
[ K ] ,
[Equation]
Therefore, we could get that σ t,k ≤ C 1 √ L , with probability 1 − 2 δ by taking a union bound (Lemmas C.1 and C.4).
Replacing δ with δ/ 2 completes the proof.
D A N U PPER B OUND OF E FFECTIVE D IMENSION (cid:101) d
We now provide an example, showing when all contexts x i concentrate on a d (cid:48)
-dimensional nonlinear subspace of the RKHS space spanned by NTK, the effective dimension (cid:101) d is bounded by d (cid:48) .
We consider the case when λ
= 1 , L = 2 .
Suppose that there exists a constant d (cid:48) such that for any i > d (cid:48) , 0 <
λ i ( H ) ≤ 1 / ( TK ) .
Then the effective dimension (cid:101) d can be bounded as
[Equation]
i =
[Equation]
i
[Equation]
d (cid:48)
[Equation]
i
[Equation]
(cid:124) (cid:123)(cid:122)
(cid:125) I+
TK (cid:88) i =
d (cid:48)
[Equation]
λ i ( H ) (cid:124) (cid:123)(cid:122) (cid:125) I.
For I 1 and I 2 we have
[Equation]
Therefore, the effective dimension satisﬁes that (cid:101) d ≤ d (cid:48) + 1 .
To show how to satisfy the requirement, we ﬁrst give a charcterization of the RKHS space spanned by NTK.
By Bietti & Mairal (2019); Cao et al.
we know that each entry of H has the following formula:
[Equation]
where Y k,j for j = 1 , . .
, N ( d, k ) are linearly independent spherical harmonics of degree k in d variables, d is the input dimension, N ( d, k ) =
(2 k + d − 2) /k
· C d − 2 k + d − 3 , µ k = Θ(max { k − d , ( d − 1) − k +1 } ) .
In that case, the feature mapping ( √ µ k Y k,j ( x )) k,j maps any context x from R d to a
RKHS space R corresponding to H .
Let y i ∈
R denote the mapping for x i .
Then if there exists a d (cid:48) -dimension subspace R (cid:48) such that for all i , (cid:107) y i − z i (cid:107) (cid:28) 1 where z i is the projection of y i onto
R d (cid:48) , the requirement for λ
i ( H ) holds.