O PTIMISTIC P OLICY O PTIMIZATION WITH G ENERAL F UNCTION A PPROXIMATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Although policy optimization with neural networks has a track record of achieving state-of-the-art results in reinforcement learning on various domains, the theoretical understanding of the computational and sample efﬁciency of policy optimization remains restricted to linear function approximations with ﬁnite-dimensional feature representations, which hinders the design of principled, effective, and efﬁcient algorithms.
To this end, we propose an optimistic model-based policy optimization algorithm, which allows general function approximations while incorporating exploration.
In the episodic setting, we establish a √ T -regret that scales polynomially in the eluder dimension of the general model class.
Here T is the number of steps taken by the agent.
In particular, we specialize such a regret to handle two nonparametric model classes; one based on reproducing kernel Hilbert spaces and another based on overparameterized neural networks.
I NTRODUCTION
Reinforcement learning with neural networks achieved impressive empirical breakthroughs (Mnih et al., 2015; Silver et al., 2016; 2017; Berner et al., 2019; Vinyals et al., 2019).
These algorithms are often based on policy optimization (Williams, 1992; Baxter & Bartlett, 2000; Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015; 2017).
Compared with value-based approaches, which iteratively estimate the optimal value function, policy-based approaches directly optimize the expected total reward, which leads to more steady policy improvement.
In particular, as shown in this paper, policy optimization generates steadily improving stochastic policies and consequently allow adversarial environments.
On the other hand, policy optimization often suffers from a lack of computational and statistical efﬁciency in practice, which calls for the principled design of efﬁcient algorithms.
Speciﬁcally, in terms of computational efﬁciency, the recent progress (Abbasi-Yadkori et al., 2019a;b; Bhandari &
Russo, 2019; Liu et al., 2019; Agarwal et al., 2019; Wang et al., 2019) establishes the convergence of policy optimization to a globally optimal policy given sufﬁciently many data points, even in the presence of neural networks.
However, in terms of sample efﬁciency, it remains less understood how to sequentially acquire the data points used in policy optimization while balancing exploration and exploitation, especially in the presence of neural networks, despite the recent progress (Cai et al., 2019; Agarwal et al., 2020).
In particular, such a lack of sample efﬁciency prohibits the principled applications of policy optimization in critical domains, e.g., autonomous driving and dynamic treatment, where data acquisition is expensive.
In this paper, we aim to provably achieve sample efﬁciency in model-based policy optimization, which is quantiﬁed via the lens of regret.
In particular, we focus on the episodic setting with general function approximations on the transition kernel.
Such a setting is studied by Russo & Van Roy (2013; 2014); Osband & Van Roy (2014); Ayoub et al.
; Wang et al.
(2020), which however focus on value iteration.
In contrast, policy optimization remains less understood, despite its critical role in practice.
To this end, we propose an optimistic policy optimization algorithm, which achieves exploration by incorporating optimism into policy evaluation and propagating it through policy improvement.
In particular, we establish a κ ( P ) · √ H 3 T -regret of the proposed algorithm, which matches that of existing value iteration algorithms but additionally allow the reward function to adversarially vary across each episode.
Here T is the number of steps, H is the length of each episode, and κ ( P ) is the model capacity, which is deﬁned based on the eluder dimension.
Moreover, we instantiate the proposed algorithm for the special cases of reproducing kernel Hilbert spaces and overparameterized neural networks, both of which are inﬁnite-dimensional model classes.
Our work is related to the study on computational efﬁciency of policy optimization (Fazel et al., 2018; Yang et al., 2019;
Abbasi-Yadkori et al., 2019a;b; Bhandari & Russo, 2019; Liu et al., 2019;
Agarwal et al., 2019; Wang et al., 2019).
These works assume either the transition model is known or there exists a well-explored behavior policy such that the policy update direction can be estimated accurately.
With such assumptions, the tradeoff between exploration and exploitation is absent and their focus is solely on the computational aspect.
In addition, our work is related to the works on adversarial MDP (Even-Dar et al., 2009; Yu et al., 2009; Neu et al., 2010a;b; Zimin & Neu, 2013; Neu et al., 2012; Rosenberg & Mansour, 2019b;a).
The algorithm in these work directly estimates the visitation measure and their algorithm utilize mirror descent to handle adversarial reward functions.
Furthermore, our work is closely related the recent work on the sample complexity of policy optimization methods Cai et al.
(2019), which only focus on the tabular and linear settings.
In contrast, our work consider general function approximation setting, which is signiﬁcantly more general.
Moreover, the construction of optimistic policy evaluation is related to Ayoub et al.
(2020), where the similar approach is incorporated in estimating the optimal value function.
The theoretical foundation of such a type of optimistic estimation is innovated by Russo & Van Roy (2014) in the bandit problem.
In particular, to characterizing the optimism and accuracy of the optimistic evaluation, we rely on the notion of the eluder dimension proposed by Russo & Van Roy (2014), which is further instantiated by this paper to the cases of kernel and neural function approximations.
1. N OTATIONS
We denote by (cid:107) · (cid:107) p the (cid:96) p -norm of a vector when p ∈ N or the spectral norm of a matrix when p = 2 .
For any two distributions p 1 , p 2 over a discrete set A , we denote by D KL ( p 1 (cid:107) p 2 ) the KL-divergence
D KL ( p 1 (cid:107) p 2 ) =
(cid:88) a ∈A p 1 ( a ) log p 1 ( a ) p 2 ( a ) .
For any a, b, x ∈ R , we deﬁne the clamp function clamp( x, a, b ) =
  b, if x > b, x, if a ≤ x ≤ b, a, if x < a.
P RELIMINARIES 2. O NLINE R EINFORCEMENT L EARNING WITH A DVERSARIAL R EWARDS
We consider an episodic MDP ( S , A , H, { P h } Hh =1 , { r h } Hh =1 ) , where S is a continuous state space, A is a discrete action space, H is the number of steps in each episode, { P h } Hh =1 represent the unknown transition model, and { r h } Hh =1 represent the reward function.
In particular, for any h ∈ [ H ] , P h represents the transition kernel from a state-action pair ( s h , a h ) at the h -th step to the next state s h +1 , while r h represents the reward function at the h -th step that maps a state-action pair to a deterministic reward.
Moreover, we allow the reward function to vary across episodes and denote by r kh the reward function at the h -th step of the k -th episode.
In particular, r kh depends on the trajectories before the k -th episode begins, possibly in an adversarial manner, and remains unobservable until the k -th episode ends.
Without loss of generality, we assume each episode starts from a ﬁxed state s 1 and all rewards fall in the interval [0 , 1] .
For any h ∈ [ H ] , a policy π h represents the conditional distribution of the action given the state at the h -th step.
We drop the subscript h to represent the collection of policies at all steps and still refer to such a collection as a policy when it is clear from the context.
For any ( k, h ) ∈ N ×
[ H ] , given a policy π and reward functions { r k h } H h
=1 , the value function and Q -function at the h -th step of the k -th episode are deﬁned by
[Equation]
for any ( s, a ) ∈ S × A .
Here the subscript π in the expectation E π [ · ] denotes all actions are taken according to the policy π except for the one given in the condition.
An online algorithm aims to construct and execute a sequence of policies { π k } k ≥ 1 and minimize the regret
[Equation]
where K is the number of episodes and T
=
KH is the number of steps taken by the algorithm.
2. R EPRODUCING K ERNEL H ILBERT S PACE
We say H is a reproducing kernel Hilbert space (RKHS) on a set Y with the reproducing kernel
K : Y ×
Y
→ R if there exists an inner product (cid:104)· , ·(cid:105) H such that, for any f ∈ H and x ∈ Y , we have f ( x ) =
(cid:104) f, K x (cid:105) H .
Here K x represents the function K ( x, · ) , which is the Riesz representation of the evaluation functional at x (Sch ¨ olkopf et al., 2002).
When the reproducing kernel K is continuous, symmetric, and positive deﬁnite, Mercer’s theorem (Steinwart & Christmann, 2008) says K has the representation
[Equation]
j =
[Equation] λ [Equation] j ( x ) φ j ( y ) , for any x, y ∈ Y , (2.2) where { φ j } ∞ j =1 is an orthonormal basis of L 2 ( Y ) and λ 1 ≥ λ 2 ≥ · · · ≥ 0 .
See more details on
RKHS in Section A.
A LGORITHM
Framework: Before the k -th episode begins, we construct the policy π k based on π k − 1 and { Q k − 1 h } Hh =1 , which are the policy in the ( k − 1) -th episode and estimators of { Q π k − 1 ,k − 1 h } Hh =1 , respectively.
Then, we execute the policy π k in the k -th episode and correspondingly update the
Q -function estimators { Q kh } Hh =1 using the reward function { r kh } Hh =1 , which is observed after the k -th episode ends.
Policy Improvement:
For any ( k, h ) ∈ [ K ] × [ H ] , we parametrize π kh by [Equation]
[Equation]
Here E kh is the potential function, which is initialized as the zero function and updated by
[Equation]
Here α > 0 is the stepsize of policy improvement.
Equivalently, we have
[Equation]
for any s ∈ S .
To see (3.2) is a policy improvement step, note that π kh is the maximizer of
[Equation]
(cid:104) Q k
[Equation]
h (
[Equation]
[Equation]
[Equation] [Equation] [Equation] This is the same as the update in Politex (Abbasi-Yadkori et al., 2019a), which originates at MDP-E (Even-Dar et al., 2009).
It is also very close to a slightly changed variant of the one-step objective in the proximal policy optimization (PPO) algorithm (Schulman et al., 2017; 2015).
Policy Evaluation: Let P be a known class of transition models such that P h ∈ P for any h ∈
[ H ] , which is speciﬁed in Section 4.
Also, for any P ∈ P , s ∈ S , a ∈ A , and V : S → [0 , H ] , we deﬁne
[Equation]
For any ( k, h ) ∈ [ K ] × [ H ] , we construct a conﬁdence set of the transition model P h and correspondingly the optimistic Q -function estimator Q kh using the data collected before the k -th episode begins.
Note that we do not use the data collected from the k -th episode although they are available, which, however, is only used to simplify the analysis.
Let V kH
+
be the zero function.
Inspired by
Ayoub et al.
(2020), given the optimistic value function estimators { V τh +1 } k − 1 τ =1 from the ﬁrst ( k − 1) episodes, we construct the conﬁdence set P kh of P h by
[Equation]
where P kh = argmin P ∈P
[Equation]
for a threshold β > 0 , which represents the degree of optimism.
Then, for any ( s, a ) ∈ S × A , given the optimistic value function estimator V kh +1 , we deﬁne the optimistic Q -function estimator Q kh by
[Equation]
and correspondingly update the optimistic value function estimator by V kh ( s )
= (cid:104) Q kh ( s, · ) , π kh ( · | s ) (cid:105)
A for any s ∈ S .
We apply the clamp function deﬁned in (1.1) to the second term on the right-hand side of (3.5) to ensure it falls in the range [0 , H − h ] , which is due to the assumption that all rewards fall in the range [0 , 1] .
Implementation: The full algorithm is presented in Algorithm 1.
Given a parametrization of the model class P , we can apply the projected stochastic gradient descent (PSGD) algorithm to solve the constrained minimization problem in Line 11 of Algorithm 1.
In particular, for kernel function approximations in Section 4.1, it reduces to a convex optimization problem, which allows the PSGD algorithm to converge to a global minimizer.
Meanwhile, for neural function approximations, it reduces to an approximately convex problem in the overparametrized regime (Arora et al., 2019), which leads to the same global convergence guarantee.
Also, to implement Lines 12 and 13 of
Algorithm 1, it sufﬁces to solve a constrained maximization problem (Feng et al., 2020), where the constraint is deﬁned in Line 12.
The Lagrangian relaxation of such a constrained maximization problem can be solved by the PSGD algorithm in the same manner of Line 11.
In addition, to instantiate the update of Q kh in Line 13, it sufﬁces to solve a least-squares regression problem.
In summary, we can instantiate the aforementioned steps through supervised learning oracles, which can be implemented in a computationally efﬁcient manner.
T HEORY
We analyze the regret of Algorithm 1, which is deﬁned in (2.1).
In Sections 4.1 and 4.2, we characterize the regret with speciﬁc choices of the model class P , while in Section 4.3, we characterize the regret for a general P , which serves as a meta result.
An informal version of the results is given in the following theorem.
1: Input: number of episodes K , model class P , stepsize α , threshold β 2: Initialize π 0 as the uniformly random policy 3:
For k = 1 to K do 4:
Start the k -th episode and receive the initial state s k5:
For step h = 1 to H do (policy improvement) 6:
Update the policy by π kh ( · | s ) ∝ π k − 1 h ( · | s ) exp { αQ k − 1 h
( s, · ) } for any s ∈ S 7:
Take the action a kh ∼ π kh ( · | s kh ) and receive the next state s kh + 8:
Observe the reward function { r kh ( · , · ) } Hh = 9:
Initialize V kH +1 ( · ) as the zero function 10:
For step h = H to 1 do (policy evaluation) 11:
[Table]
[Table]
P ∈ P :
[Table]
[Table]
, 0 , H − h ) 14:
V kh ( · ) ← (cid:104) Q kh ( · , · ) , π kh ( · | · ) (cid:105)
A
Theorem 4.1 (Informal Version of Theorems 4.3, 4.7, and 4.12) .
With proper choices of α and β , the regret of Algorithm 1 satisﬁes
[Equation]
with high probability.
Here (cid:101)
O omits absolute constants and logarithmic factors of H , T , and |A| , while κ ( P ) denotes the model capacity of P , which is speciﬁed in Sections 4.1, 4.2, and 4.3.
Theorem 4.1 indicates that, compared with the optimal policy in hindsight, namely
[Equation]
( s 1 ) , the average regret of Algorithm 1, namely Regret ( T ) /T , converges to zero at a sublinear rate.
In other words, at least one of the K policies attained by Algorithm 1 achieves a vanishing optimality gap with respect to the varying reward function across the K episodes.
The model capacity κ ( P ) is speciﬁed in Sections 4.1 and 4.2 for kernel and neural function approximations, respectively.
To establish such speciﬁc results, we characterize κ ( P ) using the eluder dimension for general function approximations in Section 4.3, which serves as the uniﬁed analysis.
4. K ERNEL F UNCTION A PPROXIMATIONS
Let P be a subset of an RKHS H with the reproducing kernel K , which has the representation in (2.2).
In detail, let S be a measurable set with |S| ≤ 1 , where | · | denotes the Lebesgue measure.
With a slight abuse of notation, we denote by A the embedding of the action space into a Euclidean space with the dimension |A| , where |A| denotes the number of actions.
Meanwhile, let Y be a d Y -dimensional set such that S
×
A × S ⊂ Y .
We assume there exists R ≥ 2 such that P ⊂ H R , where H R is the RKHS ball over Y with the radius R .
Assumption 4.2.
We assume K satisﬁes the following regularity conditions.
(i).
It holds that |K ( x, y ) | ≤ 1 , | φ j ( x ) | ≤ 1 , and λ j ≤ 1 for any x, y ∈ Y and j ∈ N .
(ii).
There exist a threshold γ ∈ (0 , 1 / 2) and absolute constants C 1 , C 2 > 0 such that λ j ≤ C 1 · exp( − C 2 j γ ) for any j ∈ N .
Note that we can replace the 1 ’s in the upper bounds of Assumption 4.2 with any absolute constant, which is reﬂected in the H -norm of any function in H .
Meanwhile, we can relax | φ j ( x ) | ≤ 1 into | λ τj · φ j ( x ) | ≤ 1 for any absolute constant τ ∈ [0 , 1 / 2) , which leads to the same regret.
We have the following result on the regret of Algorithm 1.
Theorem 4.3.
Suppose Assumption 4.2 holds and P ⊂ H R .
There exist absolute constants C 3 , C 4 > 0 such that, for any p ∈ (0 , 1) , if we set
[Equation]
in Algorithm 1, then it holds that
[Equation]
with probability at least (1 − p ) .
Proof.
See Section C for a detailed proof.
Theorem 4.3 indicates that log 1+1 /γ
( |A| RT/p ) · log 2 (1 /γ ) /γ serves as the model capacity κ ( P ) in
Theorem 4.1 for kernel function approximations.
In particular, we can obtain γ for a broad range of reproducing kernels K (Srinivas et al., 2009).
Meanwhile, we can scale R to control the model capacity κ ( P ) (Sch¨olkopf et al., 2002).
4. N EURAL F UNCTION A PPROXIMATIONS
Let P be a set of overparametrized neural networks.
In detail, we denote by NN a neural network with its weights collected in a vector w ∈ R m .
Let w 0 be the random initial weights.
For a radius
R ≥ 2 , we deﬁne
P =
{ P : ∃ w ∈ B R , s.t.
P
( s (cid:48) | s, a ) =
NN( x ; w ) , for any x = ( s, a, s (cid:48) )
∈ S ×
A × S ⊂ Y} , where B R = { w ∈ R m : (cid:107)
w − w 0 (cid:107) 2 ≤ R } .
[Equation] [Equation]
Without loss of generality, we assume NN( x ; w 0 )
=
for any x ∈ Y , which can be achieved by a symmetric initialization scheme.
See Section E for a detailed explanation.
To connect with the result for kernel function approximations in Section 4.1, we deﬁne the following condition.
Condition 4.4 (Implicit Linearization) .
It holds that
[Equation]
Condition 4.4 indicates that NN( x ; w ) is uniformly close to the linear function ∇ w NN( x ; w 0 ) (cid:62) ( w − w 0 ) of w .
In particular, the linearization error ξ m is negligible compared with the dominating terms in the regret.
The following lemma ensures Condition 4.4 holds for two-layer neural networks when m is sufﬁciently large.
Lemma 4.5 (Overparametrization) .
Suppose NN is a two-layer neural network, where the activation function is 1 -smooth, and it holds that (cid:107) x (cid:107) 2 ≤ 1 for any x ∈ Y .
Then, Condition 4.4 holds when m ≥ d Y R 4 K 3 H 2 .
Proof.
See Section E for a detailed proof.
Note that the analogous of Lemma 4.5 also applies to nonsmooth activation functions, for example, the rectiﬁed linear unit (ReLU), and multilayer neural networks (Allen-Zhu et al., 2019; Du et al., 2019; Zou et al., 2020; Gao et al., 2019), which ensures Condition 4.4 holds.
The linear function of w in Condition 4.4 induces an RKHS H with the reproducing kernel
K
[Equation] [Equation]
y
[Equation]
) , for any x, y ∈ Y , (4.2) which is known as the neural tangent kernel (NTK) (Jacot et al., 2018).
Assumption 4.6.
We assume K NTK satisﬁes the regularity conditions in Assumption 4.2.
Note that the NTK deﬁned in (4.2) depends on the randomness of w 0 .
When m goes to inﬁnity, such an empirical NTK converges to its expectation, which gives the population NTK.
It is shown in Yang
& Salman (2019) that Assumption 4.2 holds for the population NTK, which implies it also holds for the empirical NTK with high probability when m is sufﬁciently large.
We have the following result on the regret of Algorithm 1.
Theorem 4.7.
Suppose Assumption 4.6 and Condition 4.4 hold and P has the representation in (4.1).
There exist absolute constants C 5 , C 6 > 0 such that for any p ∈ (0 , 1) , if we set
[Equation]
in Algorithm 1, then it holds that
[Equation]
with probability at least (1 − p ) .
Proof.
See Section D for a detailed proof.
In parallel with Theorem 4.3, Theorem 4.7 indicates that log 1+1 /γ
( |A| RT/p ) · log 2 (1 /γ ) /γ serves as the model capacity κ ( P ) in Theorem 4.1 for neural function approximations, which can be controlled by scaling R (Arora et al., 2019).
4. G ENERAL F UNCTION A PPROXIMATIONS
Let P be a general model class, whose model capacity is characterized by the eluder dimension (Russo & Van Roy, 2014; Osband & Van Roy, 2014; Ayoub et al., 2020) deﬁned as follows.
Deﬁnition 4.8 (Eluder Dimension) .
Let Z be a set of real-valued functions on the domain X .
For any ε > 0 and τ ∈ N , we say x τ ∈ X is ( Z , ε ) -independent of x 1 , . . .
, x τ − 1 ∈ X if there exist f 1 , f 2 ∈ Z such that
[Equation]
The eluder dimension of Z at scale ε , which is denoted by dim E ( Z , ε ) , is the length of the longest sequence x 1 , . . .
, x τ ∈ X such that, for any j ∈ [ τ ] , x j is ( Z , ε (cid:48) ) -independent of x 1 , . . .
, x j − 1 for some ε (cid:48) ≥ ε .
The following lemma decomposes the regret of Algorithm 1 into errors that arise from policy improvement and policy evaluation, respectively.
Lemma 4.9 (Regret Decomposition) .
For any k ∈ [ K ] , it holds that
[Equation]
Proof.
See Lemma 4.2 of Cai et al.
for a detailed proof.
The following lemma characterizes the error that arises from policy improvement.
Lemma 4.10 (Policy Improvement) .
It we set α = (cid:112) 2 log |A| /
( KH 2 ) in Algorithm 1, then it holds that
[Equation]
| .
Proof.
See Section B.2 for a detailed proof.
Recall that for any P ∈ P , z P is deﬁned in (3.3).
Also, let Z P =
{ z P : P ∈ P} .
For any (cid:15) > 0 , we denote by N (cid:15)
( P , (cid:107) · (cid:107) ∞ , 1 ) the (cid:15) -covering number of P with respect to the (cid:96) ∞ , 1 -norm distance, which is deﬁned by
[Equation]
The following lemma characterizes the error that arises from policy evaluation.
Lemma 4.11 (Policy Evaluation) .
For any p ∈ (0 , 1) , if we set
[Equation]
in Algorithm 1, then the following results hold with probability at least (1 − p ) .
• (Optimism) For any ( k, h ) ∈
[ K ] ×
[ H ] and ( s, a ) ∈ S × A , it holds that
[Equation]
• (Accuracy) Let d = K ∧ dim E ( Z P , 1 /K ) .
It holds that
[Equation]
Proof.
See Section B.1 for a detailed proof.
Recall that T = KH .
The following theorem characterizes the regret of Algorithm 1 when P is a general model class, which serves as a meta result.
Theorem 4.12.
In Algorithm 1, if we set α as in Lemma 4.10 and β as in Lemma 4.11, then it holds that
[Equation]
with probability at least (1 − p ) , where d = K ∧ dim E ( Z P , 1 /K ) .
Proof.
The proof follows from combining Lemmas 4.9, 4.11, and 4.10.
Theorem 4.12 indicates that
[Equation]
serves as the model capacity κ ( P ) in Theorem 4.1.
The regret upper bound in Theorem 4.12 is similar to that in Ayoub et al.
(2020) when P is a general model class, whose model capacity is characterized by the eluder dimension.
In contrast, our algorithm additionally handles adversarial rewards, which is a beneﬁt of the policy optimization approach.
To establish the regret upper bounds in Sections 4.1 and 4.2, it remains to characterize the corresponding eluder dimension and log-covering number, respectively.
See Sections C and D for details.
As a special case, Theorem 4.12 also applies to the case where P is a set of d -dimensional linear models with a ﬁnite d , which is studied in Cai et al.
In particular, the eluder dimension and log-covering number in (4.4) are both (cid:101)
O
( d ) (Ayoub et al., 2020), which leads to the (cid:112) d 2 H 3 T -regret in Cai et al.
In contrast, Theorem 4.
additionally handles the case where d is inﬁnite as in kernel and neural function approximations.
A M ORE D ETAILS ON RKHS
An example of the RKHS is the linear model class.
In particular, let ψ be a d -dimensional feature vector
[Equation]
where ψ 1 , . .
. , ψ d are linearly independent.
Then, the linear span of { ψ j } ∞ j =1 forms an RKHS H ψ with the reproducing kernel K ψ ( x, y ) =
ψ ( x ) (cid:62) ψ ( y ) for any x, y ∈ Y , and the corresponding inner product (cid:104)· , ·(cid:105) H ψ is deﬁned by the Euclidean inner product
[Equation]
for any c 1 , c 2 ∈ R d .
The above example can be naturally generalized to the case where d = ∞ , that is, the feature vector can be inﬁnite-dimensional.
Moreover, recall that (2.2) says the reproducing kernel K has the representation
[Equation]
for any x, y ∈ Y , where { φ j } ∞ j =1 is an orthonormal basis of L 2 ( Y ) and λ 1 ≥ λ 2 ≥ · · · ≥ 0 .
We refer to { φ j } ∞ j =1 as the eigenfunctions of K with the corresponding eigenvalues { λ j } ∞ j =1 .
Such a representation gives the feature vector
[Equation]
The linear span of { (cid:112) λ j · φ j } ∞ j =1 recovers the RKHS H with the reproducing kernel K and inner product (cid:104)· , ·(cid:105) H .
When the reproducing kernel K is inﬁnite-dimensional, that is, K has an inﬁnite number of non zero eigenvalues, (cid:101) φ is an inﬁnite-dimensional vector.
It is known that RKHSs of various inﬁnite-dimensional reproducing kernels, for example, the Gaussian radius basis function kernel (Steinwart & Christmann, 2008), are rich function model classes in the sense that they are dense in the class of continuous and bounded functions.
B P ROOFS FOR S ECTION 4.
In this section, we provide the detailed proofs of the result in Section 4.3.
For notational simplicity, we denote by P h the operator that takes the conditional expectation with respect to the transition kernel P h for any h ∈
[ H ] .
B. P ROOF OF L EMMA 4.
Proof.
We deﬁne the event E that
[Equation]
By our choice of β and Lemma F.4 with δ =
p/ 2 , it holds that E occurs with probability at least (1 − p/ 2) .
Optimism:
For any ( k, h ) ∈ [ K ] × [ H ] , P h ∈ P kh implies
[Equation]
When h = H , the right-hand side of (B.2) is zero since V kH +1 ( · ) = 0 .
When h < H , by the construction of Q kh +1 in (3.5) and the assumption that r kh +1 ( · ) ∈ [0 , 1] , we have
Q kh +1 ( · , · ) ∈ [0 , H − h ] ,
[Equation]
The right-hand side of (B.2) is also zero, which implies
[Equation]
Thus, the optimism result holds under the event E .
Accuracy: We invoke Lemma F.1 and obtain
K (cid:88)
[Equation]
where | D kh, 1 | ≤ 2 H , | D kh, 2 | ≤ 2 H , D kH, 2 = 0 for any ( k, h ) ∈
[ K ] ×
[ H ] , and
D 11 , 1 , D 11 , 2 + D 12 , 1 , D 12 , 2 + D 13 , 1 , . . .
, D 1 H − 1 , 2 + D 1 H, 1 ,
D 21 , 1 , D 21 , 2 + D 22 , 1 , D 22 , 2 + D 23 , 1 , . . .
, D 2 H − 1 , 2 + D 2 H, 1 , . . . . .
. is a martingale difference sequence.
The Azuma-Hoeffding inequality (Azuma, 1967) implies
K (cid:88)
k
[Equation]
B
[Equation] [Equation]
with probability at least (1 − p/ 2) .
It remains to upper bound the second term on the right-hand side of (B.3).
For any ( k, h ) ∈ [ K ] × [ H ] , P h ∈ P kh implies
Q k h ( s k h , a k h ) − (cid:0)
[Equation]
Applying Lemma F.5, for any h ∈
[ H ] , we have
[Equation]
where d = K ∧ dim E ( Z P , 1 /K ) .
Combining (B.3)-(B.5)
[Equation]
we have
[Equation]
In summary
[Equation]
when the event E and (B
[Equation]
4) hold
[Equation]
which occur with probability at least (1 − p )
[Equation]
we have
Q
[Equation]
for any
[Equation]
k
[Equation]
h
[Equation]
∈ [ K ] ×
[ H ]
[Equation]
Therefore, we conclude the proof of Lemma 4.11.
B. P ROOF OF
L EMMA 4.
Proof.
By Lemma 3.3 of Cai et al.
for any ( k, h ) ∈
[ K ] ×
[ H ] and s ∈ S , we have
[Equation]
which implies
[Equation]
Plugging α = (cid:112) 2 log |A| / ( HT ) into the right-hand side of (B.7), we conclude the proof of Lemma 4.10.
C P ROOF OF T HEOREM 4.
In this section
[Equation]
we prove Theorem 4.3.
By Theorem 4.12, it sufﬁces to upper bound the eluder dimension of Z P = { z P : P ∈ P} and log-covering number of H R , which are characterized by the following two lemmas, respectively.
Lemma C.1.
Under Assumption 4.2, there exists an absolute constant C 7 > 0 such that
[Equation]
Proof.
Let t =
K ∧ dim E ( Z P , 1 /K ) .
By Deﬁnition 4.8, there exists a sequence x 1 , . . .
, x t ∈
S ×
A × [0 , H ] S such that x τ is ( Z P , 1 /K ) -independent of x 1 , . . .
, x τ − 1 for any τ ∈ [ t ] .
Here the independency scale is assumed to be 1 /K without loss of generality, which can be changed to any value larger than 1 /K .
In other words, for any τ ∈ [ t ] , there exist P 1 , P 2 ∈ P such that
[Equation]
Note that for any x = ( s, a, V ) ∈ S ×
A × [0 , H ] S and P ∈ P , by the reproducing property of the
RKHS H , we can write
[Equation]
The representation of K in (2.2) implies
[Equation]
For any d 0 such that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 where γ and C 2 are deﬁned in Assumption 4.2, we deﬁne
[Equation]
for any x = ( s, a, V ) ∈ S ×
A × [0 , H ] S and P ∈ P .
Then, we have
[Equation]
where |S| is the Lebesgue measure of S and |S| ≤ 1 .
Here the ﬁrst inequality follows from the
Cauchy-Schwarz inequality and (cid:107) (cid:112)
λ j · φ j (cid:107) H = 1 for any j ∈ N , since { (cid:112)
λ j · φ j } ∞
j =1 is an orthonormal basis of H .
By Assumption 4.2, we obtain
[Equation]
where the last inequality follows from Lemma F.6.
For notational simplicity, let
[Equation]
By (C.1), it holds that
[Equation]
where the last inequality uses τ ≤ t .
For any τ ∈ [ t ]
[Equation]
we deﬁne
[Equation]
Then
[Equation]
it holds that
[Equation]
C
[Equation]
9) where the inequality uses (C.5) and
[Equation]
Here we uses (cid:107) P (cid:107) H ≤ R for any P ∈ P and (cid:107)√ λ i · φ i (cid:107) H = 1 for any j ∈ [ d 0 ] .
In the sequel, we establish an upper bound of | (cid:101)
z P 1 ( x τ ) − (cid:101)
z P 2 ( x τ ) | .
By the deﬁnitions of y in (C.6) and v τ in (C.7), we can write (cid:101)
z P 1 ( x τ ) − (cid:101)
z P 2 ( x τ ) =
(cid:104) y,
v τ (cid:105) .
By (C.9), | (cid:101)
z P 1 ( z τ ) − (cid:101)
z P 2 ( z τ ) | is upper bounded by
[Equation]
The maximizer of such a quadratic program is
[Equation]
Therefore, we obtain
[Equation]
On the other hand, by (C.2)-(C.4), we have
[Equation]
≥
[Equation]
≥ | z P 1 ( x τ ) − z P 2 ( x τ ) |
[Equation]
2Γ( d
[Equation]
> 1 /K − 2Γ( d 0 )
[Equation]
We set d 0 = (cid:100) (cid:101) C
· log(1 /γ )
/γ · log 1 /γ (4 tRKH ) (cid:101) , where (cid:101) C is deﬁned in Lemma F.7.
Then, by
Lemma F.7, it holds that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 and
[Equation]
≤
[Equation] [Equation]
tK
[Equation] [Equation]
Combining (C.10), (C.11), and (C.12), we obtain
[Equation]
Therefore, by (C.13), we have
[Equation]
where the last inequality uses the elliptical potential lemma (Abbasi-Yadkori et al., 2011).
Here, similar to (C.8), Λ t +1 is deﬁned by
[Equation]
Note that
[Equation]
for any τ ∈ [ t ] .
Thus, setting λ = 1 / ( d 0 K 2 R 2 ) , we have
[Equation]
where the last inequality follows from t
= K ∧ dim E ( Z P , 1 /K ) .
Combining (C.15) and (C.16), we have
[Equation]
Combining (C.14) and (C.17), we obtain
[Equation]
Moreover
[Equation]
by d 0 = (cid:100) (cid:101) C · log(1 /γ ) /γ
· log 1 /γ (4 tRKH ) (cid:101) and (C.18)
[Equation]
there exists an absolute constant
C 8 > 0 such that
[Equation]
γ ) /γ · log 1+1 /γ ( tKRH ) .
(C.19)
Recall that t =
K ∧ dim E ( Z P , 1 /K ) .
Thus, by (C.19), we obtain
[Equation]
(C.20) for an absolute constant C 7 > 0 .
Thus, we conclude the proof of Lemma C.1.
Lemma C.2.
Under Assumption 4.2, there exists an absolute constant C 9 > 0 such that
[Equation]
for any R ≥ 2 .
Proof.
Recall that we deﬁne Γ( d 0 ) in (C.4).
By choosing t = 2 KH in Lemma F.7, there exists
[Equation]
where (cid:101) C is deﬁned in Lemma F.7, such that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 and [Equation]
Here γ and C 2 are deﬁned in Assumption 4.2.
Let C be a minimal 1 / (2 d 1 / 2 0 KH )
-covering of the Euclidean ball { v ∈ R d 0 : (cid:107) v (cid:107) 2 ≤ R } with respect to the (cid:96) 2 -norm distance.
For any P ∈ P , we deﬁne (cid:101) P ∈ P by (cid:101) P
[Equation]
P [Equation] (cid:105) H · φ j ( x ) , for any x ∈ Y .
Also, we write
[Equation]
Since { (cid:112) λ j · φ j } ∞ j =1 is an orthonormal basis of H and P ∈ H R , we have
[Equation]
Thus, by the deﬁnition of C
[Equation]
there exists v ∗ ∈ C with (cid:107) v − v ∗ (cid:107) 2 ≤ 1 / (2 d 1 / 2 0 KH ) .
We deﬁne P ∗ ∈ P by
[Equation]
for any x ∈ Y .
Then, by Assumption 4.
[Equation]
for any x ∈ Y
[Equation]
we have
[Equation]
Also, for any x ∈ Y , we have
[Equation]
where the ﬁrst inequality uses (cid:107) P (cid:107) H ≤ R and (cid:107) (cid:112) λ j · φ j (cid:107) H = 1 , the second inequality follows from
Assumption 4.2, and the last inequality follows from the same argument in (C.3) and H ≥ 1 .
Thus, for any x ∈ Y , it holds that
[Equation]
We deﬁne
[Equation]
Then, by (C.24), P C is a 1 / ( KH ) -covering of P with respect to the (cid:96) ∞ -norm distance.
Therefore, we have
[Equation]
where C 12 , C 9 > 0 are absolute constants.
Here the ﬁrst inequality is because (cid:107) · (cid:107) ∞ , 1 ≤ (cid:107) · (cid:107) ∞ given |S| ≤ 1 , the third inequality follows from Corollary 4.2.13 of Vershynin (2018), and the last inequality follows from the same argument in (C.19).
Thus, we conclude the proof of Lemma
C.2.
We note that our proofs of Lemmas C.1 and C.2 for the exponential decay case can be made general. For a ﬁnite-rank kernel, we can let d 0 be the rank of the kernel, i.e., the number of nonzero eigenvalues. Then, the same proof still holds and we can show that the eluder dimension is upper bounded by O ( d 0 log( RKH )) .
Moreover, for a kernel with polynomially decaying eigenvalues, that is, we have λ j ≤ C poly · j − γ for some constants C poly , γ > 0 and any j ≥ 1 , we can still truncate at dimension d 0 and calculate Γ( d 0 ) in (C.4) using the polynomial eigenvalue decay condition.
It can be shown that Γ( d 0 ) decays polynomially in d 0 .
Then, we can ﬁnd a proper d 0 by solving (C.12) and (C.21) and follow the same proof afterwards.
Proof of Theorem
Proof.
Recall that d = K ∧ dim E ( Z P , 1 /K ) .
By Theorem 4.12, it holds that
Regret (
[Equation] 26) with probability at least (1 − p ) , where we set α = (cid:112) 2 log |A| / ( KH 2 ) and β ≥ 2 H [Equation]
· log
[Equation]
(C.27)
By Lemma C.2, we have
[Equation]
which implies there exists an absolute constant C 3 > 0 such that
[Equation]
≤ C 3 H 2 · log 2 (1 /γ )
/γ · log 1+1 /γ
( RT/p ) .
In other words, (C.27) holds if we set
[Equation]
On the other hand, by Lemma C.1, we have
[Equation]
Plugging (C.28) and (C.29) into (C.26), we obtain
[Equation]
with probability at least (1 − p ) , where C 4 > 0 is an absolute constant.
Thus, we conclude the proof of Theorem 4.3.
D P ROOF OF T HEOREM 4.
In this section, we prove Theorem 4.7.
Similar to the proof of Theorem 4.3, we upper bound the eluder dimension of Z P and log-covering number of P by the following two lemmas respectively.
Recall that B R is deﬁned in (4.1).
By the deﬁnition of K NTK in (4.2), we have
[Equation]
Without loss of generality, we assume entries of ∇ w NN( · ; w 0 ) are linearly independent, which happens with probability one for most neural networks with nonlinear activation functions and random initialization.
Lemma D.1.
Under Assumptions 4.6 and 4.4, there exists an absolute constant C 10 such that
[Equation]
Proof.
Let t =
K ∧ dim E ( Z P , 1 /K ) .
By Deﬁnition 4.8, there exists a sequence x 1 , . . .
, x t ∈
S ×
A × [0 , H ] S such that x τ is ( Z P , 1 /K ) -independent of x 1 , . . .
, x τ − 1 for any τ ∈ [ t ] .
In other words, for any τ ∈ [ t ] , there exist P 1 , P 2 ∈ P such that
[Equation]
For any x = ( s, a, V ) ∈ S ×
A × [0 , H ] S and P ∈ P , we deﬁne
[Equation]
Using the reproducing property of H and the representation of K in (2.2), we have
[Equation]
For any d 0 such that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 , we deﬁne
[Equation]
for any x = ( s, a, V ) ∈ S ×
A × [0 , H ] S and P ∈ P .
And we have
[Equation]
where the second inequality uses Assumption 4.2, (cid:107) P (cid:107) H ≤ R ,
[Equation]
(cid:112) λ
[Equation]
φ j
[Equation]
H = 1 , and the deﬁnition of ξ m in Condition 4.4, the third inequality follows from Lemma F.6, and Γ( d 0 ) is deﬁned in (C.4).
Then, using the triangle inequality, we have
[Equation]
where the last inequality is by (D.1)
[Equation]
In the sequel [Equation] we establish an upper bound of | (cid:101)
z P 1 ( x τ ) − (cid:101)
z P 2 ( x τ ) | .
For any τ ∈ [ t ] , similar to (C.6)-(C.8), we denote
[Equation]
Then
[Equation]
we have
[Equation]
Here the inequality uses (D.4), Condition 4.4, and
[Equation]
= 4 R
[Equation]
which follows from the fact (cid:107) P 1 (cid:107) H , (cid:107) P 2 (cid:107) H ≤ R and (cid:107)√ λ i · φ i (cid:107) H = 1 for any j ∈ [ d 0 ] .
By the deﬁnitions of y in (D.5) and v τ in (D.6), we can write (cid:101) z P 1 ( x τ ) − (cid:101) z P 2 ( x τ ) =
(cid:104) y, v τ (cid:105) , which by (D.8) is upper bounded by
[Equation]
The maximizer of such a quadratic program is
[Equation]
Therefore, we have
[Equation]
On the other hand, by (D.2), (D.3), and Condition 4.4 we have
[Equation]
We set d 0 = (cid:100) (cid:101) C
· log(1 /γ )
/γ · log 1 /γ (8 tKRH ) (cid:101) , where (cid:101) C is deﬁned in Lemma F.7.
Then, by
Lemma F.7 it holds that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 and Γ( d 0 ) ≤ 1 / (8 tK ) .
(D.11)
Combining (D.9)-(D.11), we obtain
[Equation]
Following the same argument in (C.14)-(C.20) in the proof of Lemma C.1, we obtain t ≤ C 10 · log 2 (1 /γ ) /γ
· log 1+1 /γ
( KRH ) for an absolute constant C 10 > 0 .
Thus, we conclude the proof of Lemma D.1.
Lemma D.2.
Under Assumption 4.6 and 4.4, there exists an absolute constant C 11 such that we have
[Equation]
for any R ≥ 2 .
Proof.
By choosing t = 4 KH in Lemma F.7, there exists d 0 = (cid:100)
(cid:101) C
· log(1 /γ )
/γ · log 1 /γ (4 KRH 2 ) (cid:101) , where (cid:101) C is deﬁned in Lemma F.7, such that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 and Γ( d 0 ) ≤ 1 / (4 KH ) .
(D.12)
Here Γ( d 0 ) is deﬁned in (C.4), γ and C 2 are deﬁned in Assumption 4.6.
Let C be a minimal 1 / (4 d 1 / 2 0 KH ) -covering of the Euclidean ball { v ∈ R d 0 : (cid:107) v (cid:107) 2 ≤ R } with respect to the (cid:96) 2 -norm distance.
For any P ∈ P with parameter w , we deﬁne ¯ P and (cid:101) P by
[Equation]
for any x ∈ Y .
Also, we write
[Equation]
Note that, because { (cid:112) λ j · φ j } ∞ j =1 is an orthonormal basis in H and w ∈ B R , we have
[Equation]
Thus, there exists v ∗ ∈ C with (cid:107) v − v ∗ (cid:107) 2 ≤ 1 / (4 d 1 / 2 0 KH ) .
We deﬁne P ∗ ∈ H R by
[Equation]
By the same arguments in (C.22) and (C.23), for any x ∈ Y , we have
[Equation]
Then, by (D.12), (D.13), and Condition 4.4, for any x ∈ Y , it holds that
[Equation]
Moreover, since P ∗ ∈ H R , there exists a w v ∈ B R such that
[Equation]
for any x ∈ Y .
By Condition 4.4, we have
[Equation]
Combining (D.14) and (D.15), we have
[Equation]
Note that because entries of ∇ w NN( · ; w 0 ) are linearly independent, w v is unique for any v : (cid:107) v (cid:107) 2 ≤
R .
We deﬁne
[Equation]
for any [Equation] Then, by (D.16), P C is a 1 / ( KH ) -covering of P with respect to the (cid:96) ∞ -norm distance.
Following the same argument of (C.25), we have
[Equation]
where C 13 , C 11 > 0 are absolute constants.
Thus, we conclude the proof of Lemma D.2.
Proof of Theorem 4.
Proof.
Recall that d = K ∧ dim E ( Z P , 1 /K ) .
By the result of a general P in Theorem 4.12, we have
Regret ( T ) ≤ (cid:112) 2 H 3 T · log |A| + (cid:112) 32 H 2 T · log( p/ 2)
+ H ( dH + 1) + 4 (cid:112) dβHT, (D.17) with probability at least (1 − p ) , when we set α = (cid:112) 2 log |A| /
( KH 2 ) and β ≥ 2 H 2 · log (cid:0) N 1 /T
( P , (cid:107) · (cid:107) ∞ , 1 ) · 2 H/p (cid:1) + 4 (cid:0) H + (cid:112)
H 2 / 4 · log(8 K 2 H/p ) (cid:1) (D.18)
By Lemma D.2, we have
[Equation]
which implies that there exists an absolute constant C 5 such that
[Equation]
In other words, (D.18) holds if we set
[Equation]
D.19)
On the other hand, by Lemma D.1, we have
[Equation]
Plugging (D.19) and (D.20) into (D.17), we obtain
Regret ( T ) ≤
[Equation]
with probability at least (1 − p ) , where C 6 > 0 is an absolute constant.
Thus, we conclude the proof of Theorem 4.7.
E I MPLICIT L INEARIZATION
Two-layer fully-connected neural networks: A two-layer fully-connected neural network is deﬁned by
[Equation]
where, without loss of generality, we assume m is integer times of 2 d Y .
Here σ is the activation function.
The weight vectors w and b corresponding to the ﬁrst layer and second layer, respectively, take the form
[Equation]
respectively.
During the training, we only tune the weights in w .
Symmetric Initialization: When initializing the neural network, we generate the initial weight vectors w 0 and b by
[Equation]
for any j ∈ [ m/ (2 d Y )] .
As a result of such initialization, we have NN( · ; w 0 ) =
and we can generalize the result to multilayer neural networks by setting the last two layers in this manner.
Proof of Lemma 4.
Proof.
Let NN be a two-layer fully-connected neural network in the form (E.1).
The activation function σ is 1 -smooth and the second layer weights satisﬁes b j ∈ {− 1 , 1] for any j ∈ [ m/d Y ] .
For any w ∈ R m such that (cid:107) w − w 0 (cid:107) 2 ≤ R , we have
[Equation]
Note that, by the assumption that σ is 1 -smooth, we have
[Equation]
Thus, we have
[Equation]
By the mean value theorem, for any w : (cid:107) w − w 0 (cid:107) 2 ≤ R , there exists w † : (cid:107) w † − w 0 (cid:107) 2 ≤ R such that
[Equation]
, combining which with (E.2) we obtain
[Equation]
Therefore, we have ξ m ≤ d Y 1 / 2 R 2 m − 1 / 2 .
Then, Condition 4.4 holds when m ≥ d Y R 4 K 3 H 2 .
Thus, we conclude the proof of Lemma 4.5.
F S UPPORTING L EMMAS F. D ECOMPOSITION
For notational simplicity, we deﬁne the linear operator J kh for ( k, h ) ∈
[ K ] ×
[ H ] by ( J kh f )( s ) =
E [ f ( s, a ) | a ∼ π kh ( · | s )] , for any s ∈ S , f ∈ [0 , H ] S×A .
Lemma F.1 (Martingale Decomposition) .
For any k ∈ [ K ] , we have
V k 1 ( s k 1 ) − V π k 1 ( s k 1 ) =
H (cid:88) h
= ( D kh,
[Equation]
where D kh, 1 and D kh, 2 take the forms
[Equation]
Moreover, we have D kH, 2 = 0 for any k ∈ [ K ] , and the sequence
[Equation]
is a martingale difference sequence with respect to the ﬁltration { (cid:101) F t } t ≥ 1 in Deﬁnition F.
and each term is bounded by 4 H .
Proof.
For any ( k, h ) ∈ [ K ] × [ H ] , by the deﬁnition of the operator J kh , we have
[Equation]
where we denote the second term on the right-hand side by D kh, 1 .
Also, we have
[Equation]
where we denote the third term on the right-hand side by D k h, 2 .
Combining (F.2) and (F.3) we obtain
[Equation]
Note that V kH +1 ( · ) = 0 for any k ∈ [ K ] .
Using the identity (F.4) for h ∈
[ H ] we have
[Equation]
In the sequel, we show that the sequence in (F.1) is a bounded martingale difference sequence with respect to the ﬁltration { (cid:101) F t } t ≥ 1 .
For any ( k, h ) ∈ [ K ] × [ H ] , by the deﬁnitions of D kh, 1 and D kh, 2 , we have
[Equation]
When h = 1 , we have
[Equation]
Here the second equality is because the only randomness conditional on (cid:101) F i
[Equation]
k − 1 ,H ) is a k 1 ∼ π k 1 (
[Equation]
) , which is because, by our deﬁnitions of Q π k ,k,
[Equation]
F i
[Equation]
k [Equation] [Equation] , Q k 1 , and π k , we have
[Equation]
Similarly, when h ≥ 2 , we have
[Equation]
which is because the only randomness conditional on (cid:101) F i ( k,h − 1) is s kh ∼ P h ( · | s kh − 1 , a kh − 1 ) and we have
[Equation]
Combining (F.5) and (F.6)
[Equation]
we obtain
[Equation]
Therefore
[Equation]
we conclude the proof of Lemma F.1.
Deﬁnition F.2 (Filtration) .
We deﬁne the time index map i ( · , · ) by
[Equation]
for any ( k, h ) ∈
[ K ] ×
[ H ] , which is a bijection from [ K ] ×
[ H ] to [ KH ] .
Then, for any ( τ, h ) ∈
[ K ] × [ H ] , we deﬁne (cid:101) F t ( k,h ) as the σ -algebra generated by
[Equation]
a τ 1 , · · · , s τh , a τh ) , when h ≤ H − 1 , which are the reward functions and state-action pairs determined before s τh +1 , and ( r 1 , s 11 , a 11 , · · · , s 1 H , a 1 H , r 2 , s 21 , a 21 , · · · , s τH , a τH , r τ +1 ) , when h = H , which are the reward functions and state-action pairs determined before s τ + .
Then, the sequence { (cid:101) F t } t ≥ 1 forms a ﬁltration.
Note that r τ = { r τh } Hh =1 are determined before the beginning of the τ -th episode although they are revealed to the agent until the τ -th episode ends.
F. C ONCERNTRATION
Let { ( X τ , Y τ ) } τ ≥ 1 be a sequence of random elements in X × R for some measurable set X .
Let Z be a set of [0 , C ] -valued measurable functions with domain X for some C > 0 .
Let F = {F τ } τ ≥be a ﬁltration such that for any τ ≥ 1 , ( X 1 , Y 1 , · · · , X τ − 1 , Y τ − 1 , X τ ) is F τ − 1 -measurable
and there exists z ∗ ∈
Z such that E
[ Y τ | F τ − 1 ]
= z ∗ ( X τ ) holds.
The least-squares predictor (cid:98) z given { ( X τ , Y τ ) } tτ =1 is deﬁned by
[Equation]
We say that η is conditionally σ -sub-Gaussian given F τ ∈ F for any τ ≥ 1 if for any λ ∈ R , log E [exp( λη ) | F τ ] ≤ λ 2 σ 2 / 2 .
For any ε > 0 , we denote by N ε ( Z , (cid:107)·(cid:107) ∞ ) the ε -covering number of Z with respect to the supremum norm distance (cid:107) z 1 − z 2 (cid:107) ∞ =
sup x ∈ R
| z 1 ( x ) − z 2 ( x ) | .
For any β > 0 , we deﬁne
[Equation]
Lemma F.3 (Proposition 6 of Russo & Van Roy (2014)) .
Assume that for any t ∈ N , Y t − z ∗ ( X t ) is conditionally σ -sub-Gaussian given F t − 1 .
Then, for any ε > 0 and δ ∈ [0 , 1] , with probability at least (1 − δ ) , it holds that z ∗ ∈
Z t ( β t ( δ, ε )) for any t ∈ N , where
[Equation]
Lemma F.4.
For any δ ∈ [0 , 1] , if we let
[Equation]
in Algorithm 1, then, with probability at least (1 − δ ) , for any ( k, h ) ∈
[ K ] × [ H ] , we have P h ∈ P kh .
Proof.
Recall that, for any p ∈ P , we deﬁne z P : S ×
A × [0 , H ] S → [0 , H ] by
[Equation]
Let Z = Z P = { z P : P ∈ P} .
For any ( k, h ) ∈ [ K ] × [ H ] , we set Y k
= V kh +
( s kh +1 ) , X k = ( s kh , a kh , V kh +1 ( · )) , and z ∗ =
z P h .
Then, Y τ − z ∗ ( X τ ) is conditionally H/
-sub-Gaussian given (cid:101) F i ( k,h ) deﬁned in Deﬁnition F.2.
Then, by the deﬁnitions of P kh in (3.4) and Z k ( β ) in (F.8), we have
Z k ( β ) =
{ z P : P ∈ P kh } .
By setting
[Equation]
in Algorithm 1, it holds that
[Equation]
for any k ∈ [ K ] , where β k − 1 ( δ/H, 1 /K ) is deﬁned in Lemma F.3.
Applying Lemma F.3 with
C = H , with probability at least (1 − δ/H ) , for any k ∈
[ K ] , we have
[Equation]
which implies P h ∈ P kh .
Using the union bound over all h ∈
[ H ] , with probability at least (1 − δ ) , for any ( k, H ) ∈
[ K ] × [ H ] , we have P h ∈ P k h .
In the sequel, we prove that N ε ( Z , (cid:107) · (cid:107) ∞ ) ≤ N ε/H
( P , (cid:107) · (cid:107) ∞ , 1 ) for any ε > 0 .
Indeed, this is obtained by the observation
that, for any z P , z P (cid:48) ∈
Z with P, P (cid:48) ∈ P , we have
[Equation]
Thus, we conclude the proof of Lemma F.4.
F. E LUDER D IMENSION
Recall that Z is a set of [0 , C ] -valued functions with domain X for some C > 0 .
Meanwhile, Z k ( β ) is deﬁned in (F.8).
Lemma F.5 (Lemma 5 of Russo & Van Roy (2014)) .
For any β > 0 , we have K
[Equation]
dβK, where d = K ∧ dim E ( Z , 1 /K ) .
Proof.
When dim E ( Z , 1 /K ) ≤ K , by Lemma 5 of Russo & Van Roy (2014) we have
[Equation]
When dim E ( Z , 1 /K ) > K , since Z is a set of [0 , C ] -valued functions and Z k ( β ) ⊂ Z for any k and β , we have
[Equation]
Thus, we conclude the proof of Lemma F.5.
F. O THER U SEFUL I NEQUALITIES
Lemma F.6.
For any γ ∈ (0 , 1 / 2) , C 2 > 0 , and any d 0 ∈ N such that
[Equation]
it holds that
[Equation]
Proof.
By basic calculus we have
[Equation]
where the inequality is because 1 − γ ≥ 0 and t − γ ≤ d − γ 0 for t ≥ d 0 .
Thus, we obtain (cid:90) ∞
[Equation]
where the last inequality is because 1 − 2(1 − γ )
d [Equation] 0 / ( γC 2 ) ≥ 1 / 2 by (F.9).
Then, by the fact that
[Equation]
we conclude the proof of Lemma F.6.
Recall that Γ( d 0 ) is deﬁned in (C.4).
Lemma F.7.
Let C 1 and C 2 be the absolute constants in Assumption 4.2.
There exists an absolute constant (cid:101) C such that for any γ ∈ (0 , 1 / 2) , t ≥ 1 , and R ≥ 2 , if we set
[Equation]
then it holds that d γ 0 ≥ 4(1 − γ )( γC 2 ) − 1 and
[Equation]
Proof.
For any y > 0 , we consider the function
[Equation]
[Equation]
> 0 . Taking derivatives, we have
[Equation]
Note that f (cid:48)
( x ) ≥
if and only if x ≤ y , which implies f ( x ) ≥ f
( y ) for any x > 0 .
Reorganizing the inequality, we obtain e x ≥ ( ex/y ) y for any x > 0 and y > 0 .
Applying the above inequality via choosing
[Equation]
we obtain
[Equation]
Thus, to obtain (F.10), it sufﬁces to make the following inequality hold,
[Equation]
which is equivalent to
[Equation]
Since γ ∈ (0 , 1 / 2) and tRH ≥ 2 , there exists an absolute constant (cid:101) C such that
[Equation]
Therefore, by choosing d 0 ≥ ( (cid:101) C log(1 /γ ) /γ · log( tRH ))
1 /γ , we conclude the proof of Lemma
F.7.
G E XAMPLES OF K ERNELS WITH E XPONENTIALLY D ECAYING E IGENVALUES
In this section, we provide examples of kernels that satisﬁes Assumption 4.2.
We let Y =
S
d Y − 1 , which represents the unit sphere in R d Y .
For any kernel K , we deﬁne the integral operator T K :
L 2 ( Y ) →
L 2 ( Y ) by
[Equation]
where µ is the uniform measure on Y .
G. S QUARED E XPONENTIAL K ERNEL
The squared exponential kernel is deﬁned as
[Equation]
} , for any x, y ∈ Y , (G.2) where the constant ι satisﬁes ι 2 ≥ 2 /d Y .
For any u ∈ [ − 1 , 1] , we deﬁne (cid:101) k ( u ) =
[Equation]
{−
[Equation]
/ι 2 } and
[Equation]
where, with a slight abuse of notations, we use Γ( · ) to denote the Gamma function in this section.
Lemma G.1 (Theorem 2 of Minh et al.
For the kernel K se deﬁned in (G.2), the eigenvalues { ρ j } j ≥ 1 (without duplicates) of the corresponding integral operator T K se take the form
[Equation]
and each ρ j has multiplicity
[Equation]
Moreover, when ι 2 ≥ 2 /d Y , we have that { ρ j } j ≥ 1 is in a decreasing order and satisﬁes
[Equation]
for any j ≥ 1 , where the constants A 1 , A 2 only depend on d Y and ι .
By Lemma G.1, the eigenvalues { λ j } j ≥ 1 (with duplicates) of the kernel K se satisfy, for any j ≥ 1 , λ j = ρ t , for t such that
[Equation]
By the deﬁnition of N ( j ) in (G.4) and Stirling’s formula, we have
[Equation]
Here the asymptotic notation (cid:16) omits constant factors that are independent of j .
Combining (G.6) and (G.7), when j is sufﬁciently large, we have λ j
[Equation]
t , for t such that ( t − 1)
d Y − 1 < j ≤ t d Y − 1 .
Then, by (G.5) we obtain
[Equation]
as j → ∞ for an absolute constant c > 0 .
Thus, we know K se satisﬁes the second condition of
Assumption 4.2.
G. NTK OF S INE A CTIVATION
We consider the neural tangent kernel of a two-layer neural network of the form (E.1) where the activation function is the sine function.
In detail, the neural network is parametrized as
[Equation]
x
[Equation]
) for any x ∈ Y .
Here we modify the initial form in (E.1) by adding an intercept term, which is equivalent to adding one more dimension with constant value 1 to the input space.
The initialization of the network weights follows the same symmetric random initialization scheme
[Equation]
[Equation]
for j ∈ [ m/ (2 d Y + 2)] .
Here without loss of generality we assume m/ (2 d Y + 2) ∈ N .
Then, the population NTK of such a parametrization takes the form
[Equation]
for any x, y ∈ Y , which is the limit of the empirical NTK deﬁned in (4.2) as m goes to inﬁnity.
{+Here the second equality is derived in Rahimi & Recht (2007) and the third equality is by the fact that (cid:107) x (cid:107) 
+}
= (cid:107) y (cid:107) 2 = 1 for any x, y ∈ Y =
S d Y − 1 .
For any j ≥ 1 , let Y j be the set of all homogeneous harmonics of degree j on S d Y − 1 , which is a ﬁnite-dimensional subspace of L 2 µ ( S d Y − 1 ) , the space of square-integrable functions on S d Y − 1 with respect to µ .
It can be shown that the dimensionality of Y j is given by N ( j ) .
Lemma G.2 (Funk-Hecke formula (M¨uller (2012),
Let (cid:101) k 2 : [ − 1 , 1] → R be a continuous function, which gives rise to an inner product kernel (cid:101) K on S d Y − 1 × S d Y − 1 with the deﬁnition
[Equation]
x
[Equation]
) , for any x, y ∈ S d
[Equation]
Then, for any j ≥ 1 , f ∈ Y j , x ∈ S d Y − 1 , we have
[Equation]
where (cid:101) P j ( u ) is deﬁned in (G.3).
We let (cid:101) k 2 ( u ) = u · exp { u − 1 } .
Recall the deﬁnition of (cid:101) k in Section G.1, we have (cid:101) k 2 ( u )
=
u · (cid:101) k
( u ) for ι = √ 2 , which satisﬁes the requirement in Lemma G.1.
Lemma G.2 shows that the eigenvalues { (cid:101) ρ j } j ≥ 1 (with duplicates) of T (cid:101) K takes the form
[Equation]
where C ρ = | S d Y − 2 | / | S d Y − 1 | .
Using the relation
[Equation]
which is from the deﬁnition of (cid:101) P j ( u ) , we have
[Equation]
where { ρ j } j ≥ 1 are the eigenvalues of the operator T K se studied in Section G.1 with ι = √ 2 .
Thus, following the same argument of (G.6)-(G.8), we know such an NTK satisﬁes the second condition of
Assumption 4.2.