R EWARD L EARNING AS D OUBLY N ONPARAMETRIC B ANDITS : O PTIMAL D ESIGN AND S CALING L AWS
Anonymous authors
Paper under double-blind review
A BSTRACT
Specifying reward functions for complex tasks like object manipulation or driving is challenging to do by hand.
Reward learning seeks to address this by learning a reward model using human feedback on selected query policies.
This shifts the burden of reward speciﬁcation to the optimal design of the queries.
We propose a theoretical framework for studying reward learning and the associated optimal experiment design problem.
Our framework models rewards and policies as nonparametric functions belonging to subsets of Reproducing Kernel Hilbert Spaces (RKHSs).
The learner receives (noisy) oracle access to a true reward and must output a policy that performs well under the true reward.
For this setting, we ﬁrst derive non-asymptotic excess risk bounds for a simple plug-in estimator based on ridge regression.
We then solve the query design problem by optimizing these risk bounds with respect to the choice of query set and obtain a ﬁnite sample statistical rate, which depends primarily on the eigenvalue spectrum of a certain linear operator on the RKHSs.
Despite the generality of these results, our bounds are stronger than previous bounds developed for more specialized problems.
We speciﬁcally show that the well-studied problem of Gaussian process (GP) bandit optimization is a special case of our framework, and that our bounds either improve or are competitive with known regret guarantees for the Mat ´ ern kernel.
I NTRODUCTION
Specifying the reward function accurately for a desired objective, or reward engineering , is challenging to perform by hand, as the consequences of even small errors can be drastic (Hadﬁeld-Menell et al., 2017).
To address this, reward learning seeks to learn a predictive model of the reward function from data, which is obtained from carefully selected queries to human annotators.
The learned reward model is then used as the optimization objective for policy learning.
Reward learning has achieved signiﬁcant empirical success in domains such as text summarization (Stiennon et al., 2020;
B ¨ ohm et al., 2019), robot locomotion (Daniel et al., 2014), predicting driving styles (Kuderer et al., 2015), and Atari game playing (Christiano et al., 2017).
Despite their success, reward learning methods still lack theoretical grounding.
Moreover, their behavior can be brittle even on simple tasks, due to the difﬁculty of choosing appropriate queries and due to feedback loops from adaptive querying (Freire et al., 2020).
Indeed, an ablation study in Christiano et al.
(2017) suggests that random queries can outperform or be competitive with adaptive query procedures.
To address these issues, we provide a theoretical framework for analyzing reward learning, framing it as a doubly nonparametric experimental design problem.
This framework helps elucidate the role of query selection (Chaloner & Verdinelli, 1995) and also enables us to derive scaling laws—how the sizes of the policy and reward models affect the query complexity—for reward learning (Kaplan et al., 2020).
Proposed framework.
In our framework, we suppose we are given a reward class C r and policy class C π .
Our goal is to ﬁnd a policy ˆ π ∈ C π that performs well according to an unknown true reward r ∗ ∈ C r .
To do this, we query policies π ∈ C π , observing noisy estimates of their true reward, and use this information to choose the eventual policy
ˆ π .
To be compatible with modern nonparametric learning methods (i.e. neural nets), we view C r and
C π as subsets of Reproducing Kernel Hilbert Spaces (RKHS).
The learner therefore optimizes a nonparametric reward function over a nonparametric space of policies, making the task “doubly” non- parametric.
In contrast, previous work considers a nonparametric function class or reward class, but typically not both.
For instance, nonparametric zeroth order or bandit optimization (Srinivas et al., 2010; Mockus, 2012; Wang et al., 2018) considers a nonparametric function on a ﬁnite-dimensional input space.
Conversely, nonparametric supervised learning (Wahba, 1990; Hofmann et al., 2008) minimizes a known loss function over a nonparametric input space.
The doubly nonparametric nature of our task poses new challenges.
The (possibly) inﬁnitedimensional RKHS requires the learner to select which subspace to explore given a ﬁnite number of queries.
Furthermore, the unknown reward function makes it challenging for the learner to reason about the information gained from the selected query policies.
We address these challenges by deriving a risk upper bound for a family of plug-in estimators based on ridge regression, and then optimizing this bound to solve the optimal design task.
In addition to the optimal design problem, our framework allows us to study scaling laws with respect to the reward (or policy) class by varying the rate of decay of their corresponding eigenspectrum.
This decay rate determines the effective dimensionality of a RKHS (Zhang, 2002), and provides a natural proxy for varying the the size of the reward or policy class.
Qualitatively, our main results show that the excess risk asymptotically vanishes as long as the policy class grows at a slower rate relative to the reward class.
Sharpness of analysis.
Our risk bounds apply to reward and policy classes of arbitrary or even inﬁnite dimensionality.
Despite this generality, we show they provide stronger guarantees than previous known bounds for the specialized settings of compact policy sets and kernel multi-armed bandits.
In Section 4.3, we look at a special case of our problem when the policy set C π is a compact subspace and thus has ﬁnite rank.
For these instances, we show that our learning algorithm obtains a better excess risk O ( n − ββ +2 ) versus a rate of O ( n − β − 1 2( β +1) ) obtained by the adaptive GP-UCB algorithm (Srinivas et al., 2010), where β > 0 is a power law decay rate.
In Section 5, we specialize our general results to the well-studied problem of Gaussian process bandit optimization (Williams & Rasmussen, 2006), also known as kernel multi-armed bandit (MAB).
Speciﬁcally, for the class of Mat ´ ern kernels with parameter ν in d dimensions, we show that our algorithm achieves a regret bound of ˜ O
( T 4 ν + d (4 d +6) 6 ν + d (4 d +7) ) which is strictly better than those achieved by the GP-UCB and GP-Thompson Sampling (GP-TS)
(Chowdhury & Gopalan, 2017) algorithms and comparable with π -GP UCB
(Janz et al., 2020) and supKernelUCB (Valko et al., 2013; Vakili et al., 2021); see Table 1 for details.
GP-UCB and GP-TS are only yield sub-linear regret bounds when the smoothness of the kernel ν > d 2 —thus in high dimensions, these bounds essentially become vacuous.
The π -GP UCB algorithm was designed speciﬁcally to overcome this issue.
Our proposed algorithm achieves sublinear regret for all ν > 3 / 2 .
Our Contributions.
We propose doubly-nonparametric bandits as a framework for theoretically studying the reward learning problem.
Within this framework, we obtain ﬁnite sample risk bounds for a ridge regression based plug-in estimator and derive scaling laws for reward learning.
From a technical standpoint, we study the optimal design problem for our estimator to select informative query points by showing that the excess risk depends only on the spectral properties of a certain operator of the two RKHSs and the empirical covariance matrix.
As a corollary of our risk bounds, we provide sharper regret bounds for a class of kernel MAB problems compared to several existing algorithms, showing that the doubly-nonparametric lens of reward learning is fruitful even for “singly-nonparametric” tasks. To obtain these bounds, our reduction carefully constructs two different RKHSs to embed the input space and reward function into a policy and reward class.
F RAMEWORK : D OUBLY NONPARAMETRIC B ANDITS
Our framework considers non-parametric policy learning with non-parametric reward models.
We let π ∈ H π denote an arbitrary policy and r ∈ H r denote an arbitrary reward function, where H π and H r are Reproducing Kernel Hilbert Spaces.
For technical reasons, we assume the corresponding kernel functions K π and K r both satisfy the Hilbert-Schmidt condition (see Appendix A for details).
We let F ( π, r ) ∈ R denote the reward obtained by selecting policy π under reward function r and consider the case where the evaluation functional F is linear in both π and r .
In other words,
F ( π, r ) = (cid:104) r, Mπ (cid:105) H r where M : H π (cid:55)→ H r is a known linear mapping from the policy space to the [Table] reward space.
Since H π and H r may be inﬁnite-dimensional, linearity is only a weak restriction– e.g. the map f
(cid:55)→ f ( x ) is linear in f for any RKHS.
To incorporate problem structure [Table] we let r ∗ denote the true reward function and assume that r ∗ ∈ C r for some known set C r ⊆
H r such that (cid:107) r ∗ (cid:107) H r = 1 .
We further assume that policies π are restricted to lie in some C π which is a subset of the unit ball in H π (for instance, C π might incorporate physical constraints on implementable policies).
Thus, given the true reward r ∗ , the optimal policy (for a compact C π ) is π ∗ ∈ argmax π ∈ C π F ( π, r ∗ ) .
This proposed framework, which allows for inﬁnite-dimensional policy as well as reward classes, allows us to study how both the policy and reward space affect the difﬁculty of learning.
Query access to reward r ∗ .
The true reward function r ∗ is unknown to the learner but is accessible via queries to an oracle (e.g. a human expert), which provide noisy zeroth-order (or bandit) evaluations of the reward r ∗ .
When queried with a policy π ∈ C π , the oracle provides a response
Oracle O r ∗ : π (cid:55)→ F ( π, r ∗ ) + (cid:15) where (cid:15) ∼ N (0 , τ 2 ) , (1) with τ 2 denoting the variance of the response.
There are two possible query models: passive queries (Atkinson, 1996; Sebastiani & Wynn, 2000), where the learner selects all queries at the same time, and active queries (Bubeck et al., 2011; Lattimore & Szepesv ´ ari, 2020), where the learner is allowed to select queries sequentially.
Our focus in this work will be on the passive query model, but in many cases we will outperform existing active query algorithms.
Problem statement.
Given passive access to the oracle O r ∗ , the objective of the learner is to output a policy ˆ π ∈ C π that has small excess risk ∆ , deﬁned as
[Equation]
We think of queries to the oracle as expensive, and are interested in achieving low excess risk with as few queries as possible. This notion of excess risk is also studied by the term simple regret in pure exploration bandit problems (Lattimore & Szepesv´ari, 2020).
Representations in (cid:96) 2 ( N ) .
By Mercer’s theorem, we can represent any RKHS as a subset of (cid:96) 2 ( N ) .
Formally, the policy and the reward spaces are isomorphic to the ellipsoids
H π : = (cid:40) ∞ (cid:88)
j = κ π,j φ π,j (cid:12)(cid:12)(cid:12) ( κ π,j ) ∞
j =1 ∈ (cid:96) 2 ( N ) with ∞ (cid:88)
j = κ 2 π,j µ 2 π,j < ∞ (cid:41) and
H r : = (cid:40) ∞ (cid:88)
j = κ r,j φ r,j (cid:12)(cid:12)(cid:12) ( κ r,j ) ∞
j =1 ∈ (cid:96) 2 ( N ) with ∞ (cid:88)
j = κ 2 r,j µ 2 r,j < ∞ (cid:41) , for appropriately chosen eigenfunctions φ π,j and φ r,j , and corresponding eigenvalues µ π,j and µ r,j (Wainwright, 2019).
These are deﬁned with respect to a base measure P over the input domain; see Appendix A for details.
With a slight abuse of notation, going forward, we will use π and r to denote the corresponding coefﬁcients ( κ π,j ) and ( κ r,j ) in the expansion above.
1 With this, the
Input: Number of queries n , policy set C π , oracle O r ∗
Select n policies Q = { π 1 , . . .
, π n } and receive noisy reward evaluations
y i
=
O r ∗ ( π i ) .
Estimate ˆ r using observed responses { ( π 1 , y 1 ) , . . .
, ( π n , y n ) } using ridge regression (4).
Obtain plug-in policy ˆ π plug ∈ argmax π ∈ C π F ( π, ˆ r ) . Output: Policy ˆ π plug inner products associated with H π and H r simplify (cid:104) [Equation] [Equation]
j = π 1 ,j π 2 ,j µ π,j and (cid:104) r 1 , r 2 (cid:105) H r : = ∞ (cid:88) j = r 1 ,j r 2 ,j µ r,j . Also let S r : = diag ( µ − 1 r,j ) and S π : =
diag ( µ − 1 π,j ) be diagonal matrices comprising the inverse of the eigenvalues of H r and H π .
With this notation, if we view the map M as a (inﬁnite-dimensional) matrix, its Hermitian adjoint 2 is equal to M ∗ = S − 1 π M (cid:62) S r . In order for the evaluation functional F ( π, r ∗ ) to be ﬁnite for all π ∈ H π , the operator norm (cid:107) S 12 r MS − 12 π (cid:107) op must be bounded (see Appendix A).
We will see later that the decay of this operator’s singular values is closely related to the difﬁculty of learning in our setting.
A LGORITHM : P OLICY L EARNING VIA R EWARD L EARNING
Given the setup above, we now describe a meta-algorithm, policy learning via reward learning (Algorithm 1), for the non-parametric policy learning problem.
The algorithm is a three-stage procedure: it (i) selects a subset of policies Q to query for reward feedback, (ii) uses the responses to learn a reward estimate ˆ r , and (iii) optimizes this learnt estimate to output the policy ˆ π plug , that is, ˆ π plug ∈ argmin π ∈ C π (cid:104) ˆ r, Mπ (cid:105) H r .
Such general plug-in procedure have been studied in the statistics (Van der Vaart, 2000) and the machine learning (Devroye et al., 2013) literature.
We analyze the excess risk of this estimator for our doubly-nonparametric setup and use this risk bound to select our query set Q .
We now discuss the two key design choices in our algorithm: the choice of the reward estimation procedure as well as the choice of query set Q .
Reward learning via ridge regression.
We estimate the reward ˆ r via ridge regression in the
RKHS H r (Friedman et al., 2001; Shawe-Taylor et al., 2004).
Suppose that in the ﬁrst step of the algorithm, we have already queried the oracle on n policies and let { ( π i , y i ) }
ni =1 represent the query-response pairs.
For a regularization parameter λ reg > 0 , the ridge regression estimate of the reward function is
[Equation]
The parameter λ reg , which is usually set as a function of n , controls the bias-variance trade-off in estimating r ∗ —smaller values of λ reg reduce bias while larger values help reduce variance.
Excess risk bound for ﬁxed query set.
Observe that the plug-in estimator ˆ π plug ( Q ) is implicitly a function of the query set Q .
Ideally, we want to choose the set Q which minimizes the expected risk of the plugin estimator.
This requires us to solve the optimization problem
[Equation]
However, solving the above precisely requires knowledge about the underlying reward function r ∗ , and the combinatorial nature of the optimization problem makes it hard to ﬁnd an exact solution.
To address this, we ﬁrst upper bound the excess risk of the plug-in policy ˆ π plug in terms of the query set
Q = { π 1 , . .
π n } .
The following theorem 3 bounds the excess risk in terms of the spectrum of the spaces H r and H π , as well as the covariance matrix of the queried policies Σ Q : =
1 n (cid:80) π ∈Q ππ (cid:62) .
Theorem 1 (Excess risk of plug-in) .
For any query set Q consisting of n policies and regularization parameter λ reg > 0 , the excess risk of the plug-in estimator ˆ π plug is upper bounded as
[Equation]
In addition, letting A = M Σ Q M (cid:62) S r + λ reg
I , the expected squared distance is equal to
E [
[Equation]
The proof follows a standard analysis of ridge regression and is deferred to Appendix B. Observe that in the above theorem, the query set π ∈ Q participates in the excess risk only via the covariance Σ Q .
The risk bound is the sum of two term: the ﬁrst corresponding to the bias and the second corresponding to the variance.
In both these terms, Σ Q appears as part of A − 1 —thus query sets Q which induce a larger correlation with the map M will generally have lower excess risk.
Choices of queries which are orthogonal to the right singular vectors of M will have a constant excess risk, since for those directions the matrix
A
≈ λ reg I .
As shown later in the appendix, in the special case when the policy set consists of the entire unit ball C π = { π ∈ H π | (cid:107) π (cid:107) H π ≤ 1 } , the excess risk bound can be improved by a quadratic factor: E [∆(ˆ π plug ; r ∗ )]
≤ O (cid:0) (cid:107) M ∗ ( r ∗ − ˆ r ) (cid:107) 2 H π (cid:1) . Such a phenomenon was ﬁrst observed in the ﬁnitedimensional setup by Rusmevichientong & Tsitsiklis (2010).
Q UERY SELECTION AND STATISTICAL GUARANTEES
We now show how to select the query set Q effectively and study the excess risk of the corresponding plug-in estimator ˆ π plug .
We will start with the special case where the policy set C π is the unit ball in H π and the map M is diagonal, and then generalize to arbitrary policy sets.
In both cases, low excess risk can be achieved by repeatedly querying (approximations of) the projections of top eigenvectors of M ∗ M onto the H π space.
For the special case when the map M is diagonal, this is reduces to querying the top eigenvectors of H π .
The excess risk will ultimately depend on the the eigenspectrum of the operator S − 12 π M (cid:62) S r MS − 12 π , which is similar to the operator M ∗ M .
Additionally, to interpret our results, we instantiate them for a power law spectrum with exponent β > 0 , that is, σ j ( S − 12 π M (cid:62) S r MS − 12 π ) (cid:16) j − β .
where σ j corresponds to the j th singular value of the corresponding operator. Such power law spectra have been observed in a variety of practical settings, for instance, in the Hessian of trained deep neural networks (Ghorbani et al., 2019).
4. W ARM - UP : C π = UNIT BALL , M = DIAGONAL
In order to get some intuition, we study the special case where the policy set C π consists of the entire unit ball in the space H π and the map M is diagonal with M = diag ( ν j ) .
Further, let us denote the operator ˜ M = S 1 / 2 r MS − 1 / 2 π .
For this special case, our sampling algorithm (Algorithm 2) simply selects the top J eigenvectors of the space H π to query, for some value J which depends on the decay exponent β .
To see why, observe that for a diagonal map M , the right singular vectors of the operator
˜ M are the same as the eigenvectors of the policy space H π .
Therefore, the choice of policy π j in our algorithm is simply the scaled eigenfunction √ µ π,j · φ π,j .
Having selected these J queries, the algorithm queries each one of the nJ times and uses this as query set Q .
The intuition for this choice of query set Q is that since we are in the passive setup with no knowledge of r ∗ , any policy π ∈ C π can be an optimal policy.
By querying the top J ones out of these, we can obtain a good enough approximation to the performance of any policy in the unit ball.
The particular choice of the parameter J depends on the number of queries n available.
Since the oracle responses are noisy, to reduce variance in the responses along those directions, our algorithm performs multiple queries along the same direction. If we further consider the special case when the policies and rewards correspond to the unit balls in the ﬁnite dimensional spaces R d π and R d r respectively, our choice of query set queries the directions
{ e i } d π i =1 , each for J = nd π number of times.
Intuitively, this strategy works well because without any prior over the unknown reward function, the optimal strategy in the passive setup is to explore all directions equally and this is precisely our set of chosen queries.
This simple query strategy enjoys the following excess risk bound.
Proposition 1 (Risk bound for C π = unit ball.) .
For any J ≤ n and regularization parameter λ reg > 0 , consider the plug-in estimator obtained via the passive sampling algorithm which explores the ﬁrst J eigenfunctions of H π .
The excess risk satisﬁes
[Equation]
where the quantity ζ j = ν 2 j µ π,j µ r,j and c > 0 is some universal constant. We defer the proof of the above proposition to Appendix B. The choice of the exploration parameter
J allows us to trade-off between the two terms inside the maximum.
Typically, the second term will be maximized at j = J + 1 .
For the ﬁrst term, the supremum depends on the choice of λ reg — for small values of λ reg , the sup is achieved at j = 1 while for larger values, it is achieved at j = J .
In order to gain more intuition about this bound, we instantiate this for the power law decay.
Corollary 1 (Risk bound for power-law decay) .
Suppose that eigenvalues of the police space H π decay as j − β π , reward space H r as j − β r and the singular values of map M as j − β M .
This satisﬁes the power law assumption with exponent β = β π
+ β M − β r .
The plug-in estimator with exploration parameter J = n 1 β +2 and regularization λ reg = n −
[Equation]
+
[Equation]
+2 satisﬁes E [∆(ˆ π
[Equation]
[Equation]
The above bound shows that our algorithm can learn in the framework as long as β > 0 or equivalently β π + β M > β r , with better rates for larger values of β .
Thus, for a ﬁxed size of reward class β r , the learning rate improves as the policy class grows smaller ( β π increases) – this is intuitive since we are required to search over a smaller policy space.
On the other hand, for a ﬁxed policy class β π , our excess risk rate gets better as the reward class grows in size ( β r increases) – this is because a larger set of reward functions have similar optimal policies and hence learning gets easier.
4. G ENERAL POLICY SETS
We now describe our choice of query sets Q for general policy sets C π .
Our strategy, described in
Algorithm 2, differs from the above special case in that we need to take into account the interaction of the policy space H π with the map M .
Speciﬁcally, we show in Appendix B that the upper bound in Theorem 1 can be diagonalized for this general case via a transformation.
Let us denote the operator ˜ M = S 1 / 2 r MS − 1 / 2 π .
{+Our transformation reveals that the relevant directions to query for this general case corresponds to the columns of Φ π S − 1 / 
+}
π Φ (cid:62) π V M where , then V M are the eigenvectors of the self-adjoint operator ˜ M (cid:62)
˜ M – and it is precisely a subset of these directions that our algorithm queries.
In order to be able to query these policies, we require the set C π to contain some policies which align well with them.
We formally state this regularity assumption below.
Assumption 1 (Regularity assumption on C π ) .
For any eigenfunction φ ˜ M,j of the operator ˜ M (cid:62) ˜ M , consider the policy π j =
Φ π S − 1 / 2 π Φ (cid:62) π φ ˜ M,j .
There exists a policy ˜ π j in policy set C π such that for some constant c π > 0 , we have ˜ π j ˜ π (cid:62) j (cid:23) c
π π j π (cid:62) j .
The above assumption requires that for every choice of the policy π j in Algorithm 2, the set C π has the another policy ˜ π j which is collinear with it.
This assumption can be relaxed in various ways (for instance via convexiﬁcation) but we omit this as it is not needed for our results.
Given this assumption, the following theorem, a generalization of Proposition 1, provides a bound on the excess risk for the plug-in estimate for general policy sets C π .
Theorem 2 (Risk bound for general policy sets C π .) .
For any J ≤ n , regularization parameter λ reg > 0 and set C π satisfying Assumption 1, let ˆ π plug be the estimator output by Algorithm 1.
The squared excess risk satisﬁes
[Equation] Input: Number of queries n , map M , policy set C π , exploration parameter J Construct linear map ˜ M = S 12 r MS − 12 π and compute eigenvectors { φ ˜ M,j } j of operator ˜ M (cid:62)
˜ M
Set policy π j = Φ π S − 12 π Φ (cid:62) π φ ˜ M,j for all j ≤ J
Obtain policy ˜ π j ∈ C π such that ˜ π j ˜ π (cid:62) j (cid:23) c
π π j π (cid:62) j
Form query set Q = { ˜ π ( n/J ), . . .
, ˜ π ( n/J ) n α } where a ( b ) =
{ a, . . .
, a }} repeated b times
Output: Query set Q where the values ζ j correspond to the j th eigen values of the operator ˜ M ∗ ˜ M with ˜ M = S 12 r MS 12 π .
We defer the proof of this theorem to
Appendix B.
The proof of this theorem goes via a transformation which diagonalizes the excess risk bound and reduces the problem to a similar setup as that of Proposition 1. Additionally, Assumption 1 allows us to generalize the results to arbitrary policy sets C π .
Note that the above upper bounds the square of the excess risk. As discussed in Section 3, one can obtain a quadratic improvement in this rate if the set C π is the entire unit ball in H π .
We specialize the above bound for the power law decay assumption in the following corollary.
Corollary 2 (Risk bound for power-law decay) .
Suppose that eigenspectrum of the operator S − 12 π M (cid:62) S r MS − 12 π satisfy the power law assumption with exponent β > 0 , that is, σ j ( S − 12 π M (cid:62) S r MS − 12 π ) (cid:16) j − β .
The plug-in estimator ˆ π plug with parameter J = n 1 β +2 and regularization λ reg = n − β +1 β +2 satisﬁes E [∆(ˆ π plug ; r ∗ )
[Equation]
The above bound indicates that for the general case, learning is possible if the spectrum decay has parameter β > 0 . To get such a spectrum decay with the operator deﬁned in the above corollary, one sufﬁcient condition is that the map M does not ﬂip the larger eigenvectors of H π towards the smaller eigenvectors of H r , that is, the map M preserves the ordering of the eigenvectors of H π when transformed to the space H r . Such a misaligned scenario would require learning a very accurate representation of the reward to learn a good policy and will make learning harder. 4. C OMPARISON WITH UCB- STYLE ADAPTIVE ALGORITHMS
We next turn to evaluating the sharpness of Theorem 2. Existing frameworks for studying “singly”- nonparametric setups require the input domain to be compact. In our doubly-nonparametric setup, the input space is the policy set C π which is often non-compact (i.e. the unit ball is not compact in inﬁnite dimensions). We address this for singly-nonparametrics algorithm by taking a ﬁnitedimensional approximation.
Even though our proposed method is passive, it achieves better rates than well-known adaptive sampling algorithms. Speciﬁcally, in the power law setting of Section 4.1, the analysis of GPUCB algorithm (Srinivas et al., 2010) provides a rate of O ( n − β − 1 2( β +1) ) , which is strictly worse than the O ( n − ββ +1 ) obtained by our analysis in Corollary 1. We refer the reader to Proposition 2 in Appendix D for an exact statement.
The proof adapts the analysis from Srinivas et al. (2010), which hinges on a quantity called the information gain, which we bound for our setup. While we are comparing upper bounds for the two algorithms, we believe that our improved bound is due to a better algorithm and not an analysis gap. While we expect adaptive algorithms to perform better than passive ones in general (Lattimore & Hao, 2021), UCB style algorithms require the construction of conﬁdence intervals around input points, which crucially dictate the regret bounds of such algorithms.
In the frequentist setup, the best known such bounds (Vakili et al., 2021) are known to yield suboptimal regret rates and it is an open question as to whether these can be improved.  N EW BOUNDS FOR KERNEL MULTI - ARMED BANDITS In the previous subsection, we saw that our passive sampling algorithm actually outperforms existing adaptive sampling algorithms for the reward learning task we care about. Here we take this a step further—we specialize our algorithm to the case of kernel MABs, and show that it outperforms standard algorithms for that setting and is competitive with a specialized algorithm for Mat´ern kernels.
We consider the task of maximizing an unknown function f ∗ : X (cid:55)→ R over its domain X ⊂ R d . In the kernel multi-armed bandit (MAB) setup, this unknown function f belongs to an RKHS H , equipped with a positive-deﬁnite kernel 4 K , such that (cid:107) f ∗ (cid:107) H = 1 . Let us further restrict our attention to the space of input points X =
{ x ∈ R d | (cid:107) x (cid:107) 2 ≤ 1 } . The learner is allowed to access this function via a noisy zeroth-order oracle O f ∗ : x (cid:55)→ f ∗ ( x ) + η where η ∼ N (0 , τ 2 ) . Going forward we will assume that τ = 1 . The above oracle is similar to the reward oracle O r ∗ , except that the query points x belong to a ﬁnite dimensional space and f ∗ is a non-linear function of the query point x .
The goal in MAB is to minimize the T -step regret [Equation] where x t is the datapoint queried in the t th round. There have been several algorithms proposed to solve this problem including general purpose UCB algorithms (Srinivas et al., 2010; Chowdhury & Gopalan, 2017), Thompson sampling approaches (Chowdhury & Gopalan, 2017), and specialpurpose algorithms for speciﬁc kernels (Janz et al., 2020).
We next show that kernel MAB can be cast as a special case of our non-parametric policy learning framework. The resulting regret bounds, derived from an application of Theorem 3, are better than several general purpose algorithms (GP-UCB, IGP-UCB, GP-TS) and comparable to those specialized for the Mat ´ ern kernel ( π -GP-UCB). In order to reduce kernel MAB to our framework, we need to introduce three elements – the policy space H π , the reward space H r and the map M .
We would like spaces H r and H π such that (1) the resulting objective F ( r, π ) is linear in this space, (2) the resulting rewards and policies have unit norm in their respective space, and (3) we have a good understanding of [-Φ π S − 1 / 
-]the eigenvalues of the resulting operator. This last point ensures that we can employ our upper bounds from Section 4.
Before we deﬁne these, we let C (cid:15) denote an (cid:15) -net of the input space X under the (cid:96) 2 norm and denote its size by N cov ( (cid:15) ) .
We deﬁne the kernel matrix K ∈ R N cov × N cov on points selected in the cover as K ( i, j ) =
K ( x i , x j ) for all ( x i , x j ) ∈ C (cid:15) × C (cid:15) . Reward space H r . Given the RKHS H as well as the elements of the cover C (cid:15) , we view the reward function as a map from C (cid:15) to R , or equivalently as a vector in R N cov ( (cid:15) ) .
More precisely, letting ˜ f = [ f ( x 1 ) , . . . , f ( x N cov )] denote the vector of evaluations of a function f , we deﬁne
H r : = span { ˜ f | f ∈ H } with (cid:104) ˜ f 1 , ˜ f 2 (cid:105) H r : =
˜ f (cid:62) 1 K − 1 ˜ f 2 , . With this notation, we deﬁne the true reward r ∗ : =
˜ f ∗ = [ f ∗ ( x 1 ) , . . . , f ∗ ( x N cov )] . Policy Space H π .
Similarly to rewards, we will embed policies in R N cov . For any point x ∈ C (cid:15) , let k x = [ K ( x, x 1 ) , . . , K ( x, x N cov )] denote the corresponding vector in R N cov obtained by evaluating the kernel K over the cover.
Then, the space
[Equation]
The choice of the above norm ensures that (cid:104) k
i , k j (cid:105)
H π = (cid:104) K − 1 k
i , K − 1 k j (cid:105) = δ i,j for all ( x i , x j ) ∈ C (cid:15) × C (cid:15) .
Thus in particular, H π contains an orthonormal embedding of the set of vectors { k x } x ∈C (cid:15) .
Map M .
Both the reward space H r and policy space H π can be associated with R N cov .
Under this transformation, the evaluation f ∗ ( x ) for any x ∈ C (cid:15) corresponds to the standard inner product with
F ( r ∗ , π x ) = f ∗ ( x ) =
( ˜ f ∗ ) (cid:62) K − 1 k x = (cid:104) r ∗ , k x (cid:105) H r .
This indicates that we should take the map M to be the identity.
Furthermore, as a simple application of Mercer’s theorem it follows that this map
M is a bounded linear operator.
We make an additional assumption on the kernel function K , requiring it to be Lipschitz in its input arguments.
This assumption is often satisﬁed, in particular for the Mat´ern kernel when ν > 3 / 2 .
Assumption 2 (Lipschitz Kernel K ) .
The Kernel K associated with the Hilbert space H is L KLipschitz with respect to the (cid:96) 2 -norm for some L K > 0 : |K ( x, y ) − K ( x, x ) | ≤ L K (cid:107) x − y (cid:107)for all x ∈ X , y ∈ X .
Furthermore, the kernel satisﬁes K ( x, x ) = 1 for all points x ∈ X . [Figure] Applying Theorem 2 under the above assumption, we obtain the following excess risk bound for the plug-in estimator evaluated on the unknown function f ∗ . Theorem 3 (Excess risk for Kernel MAB) .
Suppose that the eigenvalues of a L K -Lipschitz kernel
K satisfy the power-law decay µ j (cid:16) j − β .
Let ˆ x plug be the output of Algorithm 1 using n queries to the oracle O f ∗ .
Then, for any value of β > 1 + 2 d + log( 1 δ ) and (cid:15) ∈ (0 , 1) , the excess risk satisﬁes
[Equation]
[Figure]
[Equation]
with probability at least 1 − δ .
For Mat ´ ern kernels, it is known that the eigenvalues decay with parameter β
= 1 + 2 νd (Janz et al., 2020).
Using this, we can obtain the following corollary.
Corollary 3 (Regret bound for Mat ´ ern Kernel) .
Consider the family of Mat´ern kernels with parameter ν > 32 deﬁned with the Euclidean norm over R d .
The T -step regret of our algorithm is
R
[Equation] [Equation] The above bound is for regret, which is an online notion, while our previous results are ofﬂine notions. We get from one to the other using a standard batch-to-online conversion bound based on an explore-then-commit strategy.
Table 1 compares the above bound to the existing bounds in the literature. While the bounds for GP-UCB and GP-TS become vacuous for ν (cid:46) d 2 , our bound from Corollary 3 is always sublinear in T .  E XPERIMENTAL EVALUATION
We experimentally evaluate our algorithm via a simulation study. We use these experiments to establish the dimension free nature of our results as well as to conjecture optimality of our bounds. Setup. In the simulation study, we work with d dimensional RKHSs H r and H π .
In order to simulate the nonparmeteric regime, we typically use value of n which are less or at most a constant times the dimension d . We set the matrices S π = diag ( j − 1 . 75 ) , S r = diag ( j − 1 ) and the map M = I . With this, the effective decay parameter β = β π − β r = 0 .
We further sampled the oracle noise (cid:15) ∼ N (0 , 0 . 01) .
All plots were averaged over 10 runs.
Observations.
Figure 1(a) shows the variation of excess risk as the number of queries n are varied from 256 to 4096 on a log-log plot.
Our bounds in Corollary 1 for this setup predict that the excess risk should decay at a rate O ( n − 0 . 27 ) .
By ﬁtting a linear line through the plot, we found that observed risk to vary as O ( n − 0 . 28 ) .
This plot is suggestive of the fact that our theoretical upper bounds might be tight in a minimax way over choices of decay parameter β .
In Figure 1(b), we plot the excess risk as we vary the dimension d from 32 to 8192 for four different choices of sample size, again, on a log-log scale.
Increasing the number of queries decreases the excess risk for all dimensions consistently. The risk curves tend to asymptote at different error levels for different values of n . This corroborates our theoretical ﬁndings that our proposed algorithm provides non-vacuous bounds for the doubly-nonparametric setup in the regime d → ∞ . E THICS S TATEMENT
Our main contribution is a theoretical framework to study reward learning and the associated optimal design problem.
Since our contributions are mostly theoretical in nature, we do not anticipate any ethical issues arising in the near future.
R EPRODUCIBILITY S TATEMENT
On the theoretical side, we provide detailed proofs for all our results in the appendix and appropriately reference the intermediate results we might have used in the proofs.
For the experimental aspect, we have attached our matlab code as a supplementary ﬁle and have provided all necessary hyper parameters details required to reproduce the experiments.
A T ECHNICAL DETAILS FOR PROPOSED FRAMEWORK A. RKHS ASSUMPTION
The Hilbert spaces H π and H r are Reproducing Kernel Hilbert Spaces deﬁned by kernel functions
K π , K r : X ×X (cid:55)→ [0 , 1] respectively deﬁned over a compact instance space X .
Further, the kernels
K π and K r satisfy the Hilbert-Schmidt condition
[Equation]
for some distribution P over space X .
Mercer’s theorem (Mercer, 1909) implies that such kernel functions have an associated set of eigenfunctions (with corresponding eigenvalues) that form an orthonormal basis for L 2 ( X , P ) .
We restate a version of this theorem below (Wainwright, 2019).
Theorem 4 (Mercer’s theorem) . Suppose that the space X is compact and the positive semi-deﬁnite kernel K satisﬁes the Hilbert-Schmidt condition (12) . Then there exists a sequence of eigenfunctions ( φ j ) ∞ j =1 that form an orthonormal basis of L 2 ( X , P ) and non-negative eigenvalues ( µ j ) ∞ j =1 such that [Equation]
Furthermore, the kernel function has the expansion
[Equation]
where the convergence of the sequence holds absolutely and uniformly.
A. C ONDITIONS FOR REWARD BOUNDEDNESS
For learning to be feasible in the proposed framework, we would require that the evaluation functional F ( π, r ∗ ) is bounded for any policy π ∈ H π . Using the fact that (cid:107) r ∗ (cid:107) H r ≤ 1 and (cid:107) π (cid:107) H π ≤ 1 , we have [Equation] [Equation]
Thus one sufﬁcient condition for the reward functional to be bounded is to ensure that the operator norm (cid:107) S 12 r MS − 12 π (cid:107) op is ﬁnite.
In the special case when the map is diagonal with M = diag ( ν j ) , the above condition simpliﬁes to
[Equation]
[Equation] A. R EGULARITY ASSUMPTIONS ON MAP M We assume that the map M is a compact bounded operator from the policy space H π to the reward space H r . By Schauder’s theorem, the adjoint M ∗ is also a compact operator.
Thus, the map
M ∗ M : H π → H π is a compact self-adjoint operator.
This allows us to use the spectral theorem for compact self-adjoint operators which guarantees the existence of eignevalues and eignefunctions for the operator M ∗ M and a corresponding singular value decomposition for the map M (Kreyszig, 1978).
A. N ON - ALIGNED RKHS S
As mentioned in the Section 2, if the eigenvectors of the spaces H r and H π are not aligned, one can consider the following simple transformation which resolves this. Let Φ π and Φ r represent the eigenvectors. [Equation] The above transformation implies that (cid:107) ˜ r (cid:107) H r ≤ 1 and (cid:107) ˜
π (cid:107) H π ≤ 1 .
B P ROOF OF MAIN RESULTS
In this section we provide the proofs for the main results of this work.
Appendix D to follow contains the proofs for the other results. B. P ROOF OF T HEOREM We begin by proving the result for the special case when the policy set C π consists of the entire unit ball and then generalize the analysis to arbitrary policy sets. Case 1: C π is unit ball in H π .
For this special case, observe that the the optimal policy π ∗ and the plug-in policy ˆ π plug for any reward estimate ˆ r can be written as [Equation] and [Equation]
where the operator M ∗ is the adjoint of of the map M . To prove a bound on the excess risk using the plug-in estimate, we use the following lemma which bounds this error in terms a deviation of the estimated and true rewards. Lemma 1. Consider any vectors x and y with ﬁnite non-zero norm under some inner product (cid:104)· , ·(cid:105) . Then, we have
[Equation]
[Equation] The proof of the above lemma is presented in Section B.1.1. Taking the above as given, we can upper bound the excess risk [Equation]
Case 2: Arbitrary set C π . For this case, consider the excess risk of plug-in estimator ˆ π plug obtained by maximizing reward estimate ˆ r [Equation]
21) where the ﬁnal inequality follows from the fact that ˆ π plug maximizes F ( π ; ˆ r ) over the set C π .
Thus, we see that for both the cases above, we can upper bound the excess risk of the plug-in estimator in terms of the norm (cid:107) M ∗ ( r ∗ − ˆ r ) (cid:107) H π .
Next, we evaluate this for the ridge regression based reward estimator for any set of n queries Q = { π 1 , . .
π n } with covariance matrix Σ =n (cid:80) i π i π (cid:62) i . For any regularization parameter λ reg > 0 , we have,
[Equation]
M n [Equation] i = (cid:15)
[Equation]
i (cid:33) , (22) where and equality ( i ) follows by substituting the value of y i =
F ( π i , r ∗ ) + (cid:15) i . Let us denote by matrix A = M Σ M (cid:62) S r + λ reg I .
Therefore, the error in reward estimation
[Equation]
23) where the ﬁnal distribution follows from our assumption on the noise variables (cid:15) i ∼ N (0 , τ 2 ) .
Using this above distributional form, we have [Equation] The ﬁnal bound for the general policy set C π follows from using the above bound with a an application of Jensen’s inequality. In order to convert the above bound to a high probability bound, we require an inﬁnite dimensional analog of the Hanson-Wright concentration inequality.
Using
Theorem 2.6 from Chen & Yang (2021) along with equation (23), we obtain
[Equation]
where the covariance matrix Γ = S 12 π M ∗ A − 1 M Σ M (cid:62) A −(cid:62)
( M ∗ ) (cid:62) S 12 π .
B.1. P ROOF OF L EMMA
Let the vector y
= x
+ δ x for some difference vector δ x .
Using this
[Equation]
we have
[Equation]
25) where ( i ) follows from using the inequality √ a 2 + z ≤ a + z 2 a . This establishes the result.
B. P ROOF OF P ROPOSITION
Let us denote the the map M = diag ( ν j ) and the covariance matrix Σ = diag ( σ j ) .
From the upper bound obtained in Theorem 1, we have,
[Equation]
where inequality ( i ) follows from using the fact that (cid:107) r ∗ (cid:107) H r ≤ 1 and inequality ( ii ) uses the diagonal structure of the map M as well as the fact that each policy π i ∈ Q has unit H π -norm. Recall that the choice of querying strategy queries each scaled eigenfunction √ µ π,j φ π,j of the policy space n 1 − α times. Therefore the j th entry of the covariance matrix Σ is given by
[Equation]
Plugging the above value of σ j into equation (26), we obtain [Equation] This concludes the proof of the proposition. B. P ROOF OF C OROLLARY
We now derive explicit ﬁnial sample rates for the case when the spectrum of the map M (cid:62) S r MS − 1 π satisﬁes a power law decay for some parameter β > 0 .
In the notation used in Proposition 1, we have the quantity
[Equation]
Our proof strategy will be to instantiate the bias and variance terms for this setting of ζ j and ﬁnally select a setting for the exploration parameter α and regularization parameter λ reg .
Bounding Bias.
The bias term in the proposition is a max over two terms
[Equation]
We consider the two terms in the analysis here separately.
For the ﬁrst term,
[Equation]
where the ﬁnal inequality follows from using a 2 + b 2 ≥ 2 ab .
For the second term, we have
[Equation]
Bounding Variance.
Recall that the variance term (assuming τ = 1 ) is given by
[Equation]
We again consider both terms of the maximum separately.
For the ﬁrst term,
[Equation]
[Equation]
where the inequality follows from ignoring the term λ 2 reg n 2 α in the denominator.
For the second variance term,
[Equation]
Setting regularization parameter.
By setting λ reg > n − αβ − α , we can have that the bias term is dominated by λ reg n α .
Similarly, the above setting also implies that the variance term is dominated by n 2 α − 1 .
Combing these observations, we have that the expected error is upper bounded by
[Equation]
where
[Equation]
Setting λ reg = n − α ( β +1) and then α = 1 β +2 , we get that
[Equation]
This completes the proof of the corollary.
B. P ROOF OF T HEOREM
In order to prove the general theorem, we exhibit a transformation which allows us to reduce the problem to that with the diagonal structure described in Proposition 1.
We will consider orthogonally diagonalizable matrices S r and S π which represent the eigenvectors and eigenvalues of the Hilbert spaces H r and H π .
Consider the following set of transformations for any reward r ∈ C r and policy π ∈ C π .
[Equation]
With this transformation, we can rewrite the objective function above
[Equation] where the inner product (cid:104)· , ·(cid:105) denotes the standard (cid:96) 2 inner product. Observe that we have overloaded notation to denote by ˜ r ∗ = ˜ r .
Further, using these above transformations, we can rewrite the adjoint operator
[Equation] Recall from Theorem 1, the matrix [Equation] where the covariance matrix ˜Σ = 1 n (cid:80) i ˜
π ˜ π (cid:62) .
We have used the fact here that the matrices S π and
S r are orthogonally diagonalizable and hence symmetric.
Finally, we will denote the singular value decomposition of the compact map M in the matrix form as
[Equation]
The existence of such a decomposition is guaranteed by the regularity assumptions we consider on the map M in Appendix A.
We will now analyze the bias and the variance terms from the upper bound on E [ (cid:107) M ∗ ( r ∗ − ˆ r (cid:107) 2 H π ] from Theorem 1.
Bound on bias.
The squared bias term is given by λ
[Equation]
where we have used the SVD decomposition for the matrix ˜ M in the last step.
Bound on variance.
The variance term is given by
[Equation]
Finally, by making a substitution for reward ˜ r = U (cid:62) ˜ M ˜ r and policy ˜ π
= V (cid:62) ˜ M ˜ π in equations (41) and (42), we recover back the bias variance expressions used in the analysis for Proposition 1.
What remains to be shown is that our particular choice of query policies correspond to basis vectors in this transformed space.
For this, observe that the sampling policies [Equation] for j ≤ n [Equation]
, is such that the transformed policies
[Equation] 43) indeed correspond to the basis vector. This ﬁnishes the proof of the desired claim. B. P ROOF OF C OROLLARY
The proof of this corollary follows similar to that of Corollary 1 in terms of bounding the bias and the variance.
The ﬁnal rate follows by an application of Jensen’s inequality to conclude
[Equation] The ﬁnal rate that we get in this case is thus upper bounded by the square root of the rate observed in Corollary 1. This concludes the proof. C G AUSSIAN PROCESS BANDIT OPTIMIZATION
In this section, we discuss in detail the application of our framework to the problem of frequentist Gaussian process bandit optimization, also known as Kernelized multi-armed bandits (MAB) problem.
Recall the reduction of the Kernel MAB problem to our setup required us to deﬁne three elements. Reward space H r . Given the RKHS H as well as the elements of the cover C (cid:15) , we view the reward function as a map from C (cid:15) to R , or equivalently as a vector in R N cov ( (cid:15) ) .
More precisely, letting ˜ f
= [ f ( x 1 ) , . . .
, f ( x N cov )] denote the vector of evaluations of a function f , we deﬁne H [Equation] : = span
[Equation]
} with [Equation] where (cid:104)· , ·(cid:105) represents the standard (cid:96) 2 inner product. With this notation, let us deﬁne the true reward r ∗ : =
˜ f ∗ =
[ f ∗ ( x 1 ) , . . .
, f ∗ ( x N cov )] .
Policy Space H π .
For the policy space H π in our setup, we let
[Equation]
(cid:104) k 1 , k
[Equation]
H
[Equation]
(cid:104)
[Equation]
The choice of the above norm ensures that
[Equation]
For the policy space H π , we have created an orthonormal embedding of the set of vectors { k x } x ∈C .
Observe that this policy set that we construct satisﬁes the regularity Assumption 1 because each vector k is an eigenvector of the space H π .
Map M .
By our assumption that the kernel K is a Mercer’s kernel, we have that H π
⊆
H r , that is, for all x ∈ C , the vector k x ∈ H r . Furthermore, both H r and H π are sub-spaces of R N cov and we can take the map M =
I N cov . With these deﬁnitions, we now explicitly establish a correspondence between our doubly nonparameteric bandit problem and the Kernel MAB problem. C. C ONNECTING THE PROBLEMS Given an RKHS H with an associated Mercer’s kernel K , the objective of the zeroth-order bandit optimization problem is
[Equation]
with access to oracle [Equation] where η ∼ N (0 , τ 2 ) . Equivalently, the objective in our reward learning framework is
[Equation] such that [Equation] with the corresponding spaces and inner products are deﬁned in the previous section.
The oracle required in our setup responds with
[Equation] where η ∼ N (0 , τ 2 ) , for any policy π ∈ H π such that (cid:107) π (cid:107) H π ≤ 1 . Our ﬁrst lemma below states that obtaining such a n oracle is indeed feasible if we are able to restrict our queries π to include only points k x for which the vector k x ∈ C (cid:15) .
Lemma 2. Given access to oracle O f ∗ for a function f ∗ , the corresponding oracle O r ∗ can be implemented when the query set consists of { k x } x ∈C (cid:15) . Proof. For any query point k , the oracle O r ∗ needs to compute the value (cid:104) r ∗ , k (cid:105)
H r = f ∗ ( x ) . Thus, these two oracles on the provided query set are exactly identical. Lemma 3. For any f ∗ ∈ H satisfying (cid:107) f ∗ (cid:107) H ≤ 1 , we have that (cid:107) r ∗ (cid:107) H r ≤ 1 .
Proof. Observe that an alternate way to deﬁne the RKHS norm is given by [Equation] The fact that (cid:107) r ∗ (cid:107) H r is computed on C (cid:15) ⊂ X establishes the desired claim.
Finally, we turn to establishing a relation between the solutions obtained from solving the relaxed problem (P2) as compared to solving the original problem (P1). We denote the corresponding maximizers for both problems [Equation]
The following lemma now relates both these maximizers together. Lemma 4. For an RKHS H with kernel K satisfying Assumption 2 with constant L K > 0 and any function f ∗ ∈ H , let x ∗ ∈ X and x ∗ π ∈ C (cid:15) be the maximizers as deﬁned in equation (47) , we have
[Equation] Proof.
Denote by Π C (cid:15) ( x ∗ ) : = arg min x ∈C (cid:15) (cid:107) x ∗ − x (cid:107) 2 the projection of the point x ∗ onto the set C (cid:15) . Then, we have
[Equation]
This completes the proof of the lemma.
The above lemma shows that solving Problem P2 is equivalent to solving Problem P1 up to an additive factor of √ 2 cL K (cid:15) when we are working with an (cid:15) -cover over the domain space. C. A NALYSIS FOR BANDIT OPTIMIZATION Recall from the previous section that the quantity which determines the rate of decay is the ratio of eigenvalues
[Equation]
where ˆ µ r,j is the j th eigenvalue of the kernel matrix K .
Let us denote by P denote the uniform distribution over the input space X and let us suppose that the cover N cov is formed using random samples from this distribution.
Let us denote by { µ j } the eigenvalues and by φ j the corresponding eigen vectors of the Mercer kernel K .
For every point x ∈ X , let us denote by
[Equation]
the corresponding featurization of the point x .
Then, for S :
= E x ∼ P [Φ( x )Φ( x ) (cid:62) ] , we have
[Equation] Observe that the kernel matrix K and the (scaled) sample covariance matrix N cov · ˆ S = (cid:80) x ∈C Φ( x )Φ( x ) (cid:62) are similar matrices and thus have the same eigenvalues. The following lemma, adapted from Koltchinskii & Lounici (2017, Theorem 9) relates the eigenvalues of the sample covariance matrix ˆ S to those of the underlying kernel K . Lemma 5.
For any λ S > 0 and size of the cover satisfying N [Equation] we have, [Equation]
for all j
[Equation]
with probability at least 1 − δ .
The following corollary of Lemma 5 provides us with a way to control the deviation of the eigenvalues ˆ µ j from the corresponding µ j in a multiplicative manner.
Corollary 4.
For any value of decay parameter β > 1 and γ < β , we have, for all j , the eigenvalues
[Equation]
with high probability.
Proof.
Let us understand the condition N cov ( [Equation] and see what restrictions it puts on the value of the covering number. Lets suppose that the true eigen values µ j (cid:16) j − β and we set the value of λ S (cid:16) N − γ cov .
Therefore, the sum [Equation] Thus, if we set (cid:15) S = 12 , then for any β > 1 and γ < β , the above condition on the covering number will be satisﬁed and we get desired bound on the deviation of the empirical eigenvalues from population eigenvalues. The above corollary is essential to our argument because often times we have a good understanding of the decay of the eigenvalues of the kernel K associated with the RKHS and this allows us to relate the set of empirical eigenvalues to these.
We now present a proof of Theorem 3, restated below, which upper bounds the excess risk for this setup. We will then use a batch to online conversion bound to convert this to a regret bound and specialize to the Mat´ern kernel later. Theorem 5 (Restated Theorem 3) . Suppose that the eigenvalues of a L K -Lipschitz kernel K with respect to a distribution P over X satisfy the power-law decay µ j (cid:16) j − β .
Let ˆ x plug be the output of
Algorithm 1 using n queries to the oracle O f ∗ .
Then, for any value of γ ∈ (1 + 1 d log(1 /(cid:15) ) log( L K /(cid:15) 2 ) , β ) and (cid:15) ∈ (0 , 1) , the excess risk
[Equation]
with high probability. Proof. Our strategy, as before, will be to explore n α directions and assume τ 2 = 1 . Recall, that for symmetric matrices, Theorem 2, the excess error of the plug-in estimator can be upper bounded as
[Equation]
Bounding Bias.
We will split the analysis into two cases. Case 1: j > n α . For this case, we have that σ j = 0 and therefore [Equation]
[Equation]
with the above holding with high probability from an application of Corollary 4 for any 1 < γ < β .
Case 2 : j ≤ n α .
For this case, we have σ j = µ π,j n α .
The bias can then be upper bounded as
[Equation]
[Equation]
where the ﬁnal inequality follows from using a 2 + b 2 ≥ 2 ab .
Bounding variance.
As we did in the section above, let us split the analysis into two cases.
Case 1 : j > n α .
For this case, the variance term simpliﬁes to
[Equation]
Case 2: j ≤ n α .
For the second case, we can upper bound the variance term
[Equation]
where the last inequality follows from ignoring the second term in the denominator.
Setting regularization parameter.
From the analysis in the above paragraphs, we have
Bias
[Equation]
For regularization parameter λ reg > N cov n − αβ − α and γ >
[Equation]
n N cov , we have
Bias
[Equation]
Excess risk bound.
To obtain the ﬁnal excess risk bound, we set α =
[Equation]
where inequality ( i ) follows from our particular choice of α .
Combining the above bound with
Lemma 4 completes the proof
[Equation]
The following corollary instantiates the above theorem for the case when the input space is the unit ball, that is, X =
B d (1) .
Corollary 5.
Let the input space X
= B d (1) and the kernel K satisfy Assumption 2. Then, for any β > 1 + 2 d , we have [Equation] Proof.
From the bound in Theorem
[Equation] we have, [Equation] where inequality ( i ) follows from substituting (cid:15)
→ (cid:15) 2 /L K , ( ii ) follows from the fact that
N cov ( (cid:15) )
(cid:16) (cid:0) 1 (cid:15) (cid:1) d , and ( iii ) follows from using the assumption
that β > γ > 1 + 2 d log(1 /(cid:15) ) log( L K /(cid:15) 2 ) .
Finally, setting (cid:15) (cid:16) L
[Equation]
n
[Equation]
, we get
[Equation] This establishes the desired claim. C. R EGRET BOUND FOR M AT ´ ERN K ERNEL In this section, we specialize the bound from Theorem 3 for the special class of Mat´ern kernels.
Recall that the Matern kernel is a distanced based kernel with K ( x, y ) =
f ( (cid:107) x − y (cid:107) ) .
Denote by r = (cid:107) x − y (cid:107) , the exact form for the kernel is given by
[Equation]
with parameters ν and l and where K ν is the modiﬁed Bessel function of the second kind.
Going forward, lets ﬁx the scale parameter l = 1 without loss of generality.
The following lemma then bounds the Lipschitz constant for this class of kernels when the distance function is the (cid:96) 2 norm.
Lemma 6 (Lipschitz Mat´ern Kernel) .
Consider the Mat´ern kernel with parameter ν > 32 .
The
Lipschitz constant of this kernel is bounded by
[Equation]
Proof.
The approach will be to show that the kernel function K Matern ,ν is a Lipschitz function of the distance r and then cover the (cid:96) 2 ball in the d dimensional space appropriately.
We now look at the derivative of the function K Matern ,ν ( r ) with respect to r .
[Equation]
where ( i ) follows from the identity ∂K
[Equation]
z
[Equation]
=
[Equation]
For any ν >
[Equation]
we have the inequality
[Equation]
for
[Equation]
[Equation]
Instantiating the above with y = 1 and ν > 32 , we have
[Equation]
The Lipschitz constant for this case can now be obtained by taking a sup over r ∈ (0 , 1) .
While our upper bound was in terms of sample complexity, in order to compete with the cumulative regret formulation, we adapt an explore-then-commit strategy.
The following lemma relates the sample complexity bound to a cumulative regret bound.
Lemma 7 (Batch to online conversion) .
Suppose an algorithm has sample complexity O ( n − α ) ) in the passive learning setup, the explore then commit strategy based on this learning algorithm would have regret O
( T 11+ α ) .
Proof.
For some parameter γ > 0 , let the explore then commit algorithm explore for T γ steps and the commit to the strategy obtained post this exploration for the remaining T − T γ time steps.
The cumulative regret for such an algorithm is
[Equation]
Setting γ = 1 1+ α ﬁnishes the proof.
We now proceed to prove Corollary 3 which instantiates the bound in Theorem 3 for the class of Mat´ern kernels. Corollary 6 (Restated Corollary 3) . Consider the family of Mat´ern kernels with parameter ν >deﬁned with the euclidean norm over R d .
The T -step regret of the explore-then-commit algorithm is [Equation]
{+with high probability. Proof. First, observe that excess risk bound in Corollary 5 can be converted to a corresponding T -step regret bound by an application of Lemma 
+}
such that [Equation]
For the class of Mat´ern kernels, the decay parameter β = 1 + 2 νd (Janz et al., 2020, Theorem 9). Using this wit the above regret bound, we get, [Equation] This completes the proof.
In this section, we prove an upper bound on the expected risk of the Gaussian process upper conﬁdence bound algorithm (GP-UCB) algorithm of Srinivas et al. (2010). In order to adapt their algorithm for our setup, consider the function [Equation]
such that D = { x | (cid:107)
[Equation] We have used x to denote policies in this setup to be consistent with the notation in Srinivas et al. Observe that the domain deﬁned above is not compact – a necessary condition for the algorithm to work. One work around this is to truncate the unit ball after a ﬁnite number of dimensions and bound this truncation error.
The excess risk incurred by this truncation can be made arbitrary small.
Going forward, we ignore this truncation. The regret for the UCB algorithm is shown to be upper bounded by ˜ O ( γ T √ T ) where γ T is the information gain with [Equation]
where we have assumed without loss of generality that the noise variance τ = 1 .
For our setup, the kernel function K ( x i , x j ) =
(cid:104)
Mx i , Mx j (cid:105) H r .
We additionally require that the reward function r belongs to the RKHS spanned by the set { Mx | x ∈ D } .
Denote by S = S 12 π M (cid:62) S − 1 r S 12 π and suppose that its eigenvalues satisfy a power law decay with σ j ( S ) =
ζ j = j − β .
The following lemma upper bounds the information gain for this setup in terms of the power law parameter β > 0 .
Lemma 8 (Information Gain.) .
The information gain γ T for the above setup is bounded as
[Equation]
Proof. The quantity of interest here is the information gain [Equation] where the matrix X =
[ x (cid:62) 1 ; . . .
; x (cid:62) T ] and we have assumed that the noise variance is 1 .
From the setup described above, we have that the eigen values of S decay as λ j (cid:16) j − β .
It is easy to see that
[Equation]
is a monotonic sub-modular function.
Thus, the value of γ T can be upper bounded by (1 − 1 /e ) −times the value of the greedy maximization algorithm. The greedy maximization algorithm is equivalent to picking [Equation]
It is easy to see that at each time t , the unit vector x t will be an eigen vector of the matrix S .
Given this observation, we can ﬁnally upper bound the value of the info gain
[Equation]
j ) such that m j ≥ 0 and (cid:88)
[Equation]
T.
Solving the above optimization problem, the optimal choice of the variables
[Equation]
and
[Equation]
Setting λ
= T − ββ +1 ensures that there are T 1 β +1 active directions.
Substituting the above values of m j in the expression for γ T , we get
[Equation]
where ( i ) follows from setting λ
= T − ββ +1 . This establishes the required claim. We are now ready to state this our sample complexity bound for GP-UCB for this subclass of problems. Proposition 2 (Sample complexity for GP-UCB) .
Suppose that the police space H π , reward space H r and the map M satisfy the power law decay assumption with exponent β > 0 . The estimator ˆ π ucb output by the GP-UCB algorithm satisﬁes [Equation] The proof of the sample complexity bound in Proposition 2 now follows the regret bound of ˜
O ( γ T √ T ) along with using the upper bound on the information gain from Lemma 8.
[Equation] More recently, Cai & Scarlett (2021) extended the analysis of Valko et al. to show that the SupKernelUCB algorithm achieves a regret bound ˜ O
( √ γ T T ) . Using this modiﬁed bound, one can improve the above analysis to obtain excess risk
[Equation]
which is still worse than those obtained by the bounds by our proposed ridge regression estimator.
In the simulation study, we work with d dimensional RKHSs H r and H π .
In order to simulate the nonparmeteric regime, we typically use value of n which are less or at most a constant times the dimension d .
We set the matrices S π = diag ( j − 1 . 75 ) , S r = diag ( j − 1 ) and the map M
= I .
This is allowed since the policy space is smaller than the reward space. With this, the effective decay parameter β = β π − β r = 0 .
We sampled the true reward r ∗ uniformly at random from the unit ball in H r .
We further sampled the oracle noise (cid:15) ∼ N (0 , 0 . 01) .
All plots were averaged overruns.