Experimental Design for Linear Functionals in Reproducing Kernel Hilbert Spaces
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Optimal experimental design seeks to determine the most informative allocation ofexperiments to infer an unknown statistical quantity.
In this work, we investigate theoptimal design of experiments for estimation of linear functionals in reproducingkernel Hilbert spaces (RKHSs) .
This problem has been extensively studied in thelinear regression setting under an estimability condition, which allows estimatingparameters without bias.
We generalize this framework to RKHSs, and allowfor the linear functional to be only approximately inferred, i e., with a ﬁxed bias.
This scenario captures many important modern applications, such as estimationof gradient maps, integrals, and solutions to differential equations.
We providealgorithms for constructing bias-aware designs for linear functionals.
We derivenon-asymptotic conﬁdence sets for ﬁxed and adaptive designs under sub-Gaussiannoise, enabling us to certify estimation with bounded error with high probability.
Introduction
Optimal Experimental Design (OED) aims to determine data collection schemes – designs – toefﬁciently estimate unknown quantities of interest given limited resources (Chaloner and Verdinelli,1995).
As common, we model experiments via an oracle that yields a (noisy) response to a giveninput.
A design is usually either an (adaptive) policy for querying the oracle, or a (nonadaptive) ﬁxedallocation of query budget to different oracle inputs.
OED has a rich history and close relations to theﬁeld of bandits (Szepesvari and Lattimore, 2019) and active learning (Settles, 2009).
We consider the regression setting, where observations at a ﬁxed input x ∈ X can be obtained via thenoisy oracle by:
[Equation]
where x ∈ X [Equation] and · (cid:62) · denotes the inner product in a Hilbert space H κ , (cid:15) is independent, sub-Gaussian noise withknown variance proxy σ 2 , and θ is a bounded element from a separable reproducing kernel Hilbertspace H κ from kernel κ , with a bound θ (cid:62) V 0 θ ≤ λ − 1 , where V 0 :
H k
→ H k is a positive deﬁniteoperator.
Depending on whether λ is known or unknown, we will propose different estimators.
Incontrast to the classical ED task of estimating θ (see Fedorov and Hackl, 1997), we are interested inestimating a projection of θ .
Namely, let C : H κ → R p be a known linear operator .
The map C issuch that C (cid:62) C is full rank p , where C (cid:62) denotes the adjoint.
Our goal is to identify an estimate of
C θ efﬁciently, i.e., with low query complexity or with a maximal reduction of uncertainty given aﬁxed query budget T .
The formalism above captures, or occurs as a subroutine in, numerous practical problems.
Forexample, evaluation at speciﬁc target points, integration and differentiation are all linear operators,among many other useful linear functionals.
Other examples include ordinary or partial differential equations operators, spectral transforms, and stability metrics from control engineering.
In Section 7we detail several example applications.
A naive approach would be to ﬁrst obtain an estimate ˆ θ of θ , and compute C ˆ θ .
However, the appealof estimating linear functionals directly is that the number of unknowns of interest may be muchlower than for the original overall unknown element θ (which might even be inﬁnite-dimensional).
Consequently, we would hope that the query complexity of reducing the variance of the estimate
C θ scales in the dimension of the range of C , which is p .
For example, when focusing on ﬁnitedimensional RKHSs, the operator C : R m → R p , where m is the dimension of H κ , becomes amatrix.
In this work, we study the cases where p < m , and show that the estimation error can indeedscale with p , and the geometry of the query set { Φ( x ) | x ∈ X} .
The estimation bias plays a centralrole in this work.
It depends among other things the richness of the query set X , as we will detaillater in Sec. 3.
Sequential Experiment Design Apart from classical experiment design, where we ﬁrst commit towhat design (set of inputs x ) we choose, referred to as a ﬁxed design , we also consider sequentialdesign, where data is acquired depending on past observations.
Suppose for example, we are gatheringdata to test whether the null H 0 : f (cid:62) Φ( x ) ≥ 0 for all x ∈ X or otherwise.
We can incrementallygather evidence and check whether the null hypothesis has already been rejected.
As our data dependson prior evaluation points it forms an adaptive design .
In this work, we develop conﬁdence sets forboth ﬁxed and adaptive designs – which are of paramount importance in the context of sequentialexperiment designs used, e.g., to deﬁne stopping rules of adaptive hypothesis testing problems.
Contributions A)
We consider objectives for experiment design for linear functionals in general
RKHS spaces, which carefully take the bias of the estimator into account.
B)
We provide bounds onthe query complexity required to reach (cid:15) accuracy to estimate linear functionals with high probability.
C)
We construct novel non-asymptotic conﬁdence sets for linear estimators of linear functionals of
RKHS elements, both for ﬁxed and adaptive designs, where queries are independent of previous noiserealizations, and where they are not, respectively.
D)
We demonstrate the improved inference errordue to specially deﬁned designs and new conﬁdence sets on the problems of learning trajectories ofdifferential equations, estimation of gradient maps, statistical contamination, and stability veriﬁcationfor non-linear systems.
Background and Related Work
Linear Estimators Let S ⊂
X be a ﬁnite set of selected evaluations s.t. |S| = n .
We focus onlinear estimators of the form L : R n → R p , where L y is estimating C θ .
An estimator is understoodhere as the algorithm to ﬁnd the random estimate L y .
Notice that given S , L is not a random quantity;the randomness rather comes from the realizations y .
To choose the estimator L , one classicallylooks at the covariance of the residuals C θ − L y , E ( L ) =
E [( C θ − L y )
( C θ − L y ) (cid:62) ] , where theargument signiﬁes how the random variable y is transformed (noise is averaged)
[Equation]
where E ( L ) ∈
R p × p , (2) and the matrix X contains stacked evaluation functionals of the RKHS X i,x
= Φ i ( x ) for x ∈ S ⊆ X .
We will then seek a way to transform y via estimator L such that the covariance of residuals isminimized in certain sense.
Importance of Bias and RKHS Classical experimental design studies estimation of C : R m → R pwhere the RKHS is ﬁnite dimensional (Pukelsheim, 2006).
On top of that, they consider onlyestimators which are unbiased for any θ , in other words, LX = C .
While, with ﬁnite dimensions,this simpliﬁcation might be reasonable, for inﬁnite dimensionial RKHSs, bias is inevitable and mustbe controlled.
Consider the case of estimating the gradient of a continuous function f .
We can nearlynever learn it up to arbitrary precision from noisy point queries.
However, estimation up to a smallerror is always possible and sufﬁcient.
Estimating ∇ x f ( x ) from points close to f ( x ) will incur smallbias but larger variance as the change in f compared noise (cid:15) is small.
Hence balancing these twosources of error is crucial for an informative design in RKHSs.
Experiment Design: Classical Perspective Consider an m dimensional version of the model in(1).
Then, among all unbiased estimators, linear least-squares estimators minimize (2) under the
Löwner order due to the famed Gauss-Markov theorem.
Unbiasedness in this form is synonymouswith estimability , which means that by repeating the evaluation in X , arbitrary precision can bereached (Pukelsheim, 2006).
The covariance of residuals then becomes W − 1 † = CV † C (cid:62) , where· † denotes a generalized pseudo-inverse and V
= X (cid:62) X .
The matrix W is often referred to as the information matrix .
Gaffke (1987) and Krafft (1983) note that estimability implies that W † isnon-singular.
We relax this condition and allow the estimator to have a bias ; this means that wecannot in general reduce the error arbitrarily by repeating the measurements.
In fact, our extensionuses the fact that W † as deﬁned above will not be singular even if estimability condition is notsatisﬁed.
We will show that the matrix W † can still play the role of the information matrix.
Experiment Design: Modern Challenges Mutný et al.
(2020) use experimental design to estimatethe Hessian of an unknown RKHS function, while Kirschner et al.
(2019) use it to estimate the gradientof it for use in Bayesian optimization.
Perhaps most related, Shoham and Avron (2020) study overparametrized experimental design for one-shot active deep learning and analyze the bias in connectionto experiment design, similarly as in the seminal work of Bardow (2008).
They do not consider biasarising from the limited design space nor do they treat linear functionals.
More broadly, the uncertaintypropagation is studied in the Bayesian framework in the ﬁeld of probabilistic numerics for linear andnonlinear operators (see Cockayne et al., 2017; Owhadi and Scovel, 2016, and citations therein).
Conﬁdence sets Unlike classical statistics, our focus is on non-asymptotic conﬁdence sets onregression estimators.
They can be found in, e.g., Draper and Smith (2014) and Abbasi-Yadkoriet al.
(2011), for ﬁxed and adaptive designs, respectively.
Our goal is to deﬁne the conﬁdence setsin the appropriate norm such that their width scales with the dimension of the range of C , i.e., p ,and the geometry of { Φ( x ) | x ∈ X} , but not directly dim( H κ ) nor number of data T .
Mutný et al.(2020) derive non-asymptotic conﬁdence sets that scale in p , but grow with the number of points
Tas they do not use the appropriate norm (see Appendix B.4).
Similarly, Khamaru et al.
studyestimators for adaptively collected data, and propose asymptotic conﬁdence sets for their estimators.
Without a speciﬁc condition, their conﬁdence sets can grow with T , however, they consider moregeneral noise distributions apart from sub-Gaussian as we do here.
Estimation and Bias
In this section, we motivate the linear estimators and identify information matrices .
Informationmatrices are the inverses of covariance matrices of residuals E as in Eq.
They are an importantobject in the analysis of the error of the estimators and their conﬁdence sets.
Also, they depend onthe evaluations that deﬁne the estimator L , X .
Maximizing the information matrices as a function ofthe chosen observations and their proportions X gives rise to optimal experimental designs.
Further, we identify quantities that inﬂuence the bias of the estimators.
We study two estimators: theleast-norm estimator (interpolation), when the bound on (cid:107) θ (cid:107) V 0 is unknown but ﬁnite, or the ridgeregularized least squares estimator, where (cid:107) θ (cid:107) V 0 ≤ λ − 1 is known.
Both estimators are motivated asminimizing the error residuals E in trace norm under these two bound assumptions.
3.1 Estimators
Interpolation As apparent from Eq.
(2), without the knowledge of an explicit bound on the normθ , the worst case over θ causes the optimal estimator L to minimize only the trace of the bias.
Thisleads to minmization of ( C − LX )
V − 1 / 2in Frobenius norm, leading to the familiar interpolationestimator ,
[Equation]
where
[Equation]
X [Equation] The covariance of residuals can then be expressed as
[Equation]
, 127 where P X = V − 1 / [-
 (-]2( V − 1 / 
 XK −
[Equation]
X
[Equation]
[Equation] − I ) V − 1 / 2is a scaled projection matrix.
Unlike as inthe classical ED treatment, the error covariance has two terms: bias and variance.
If the span of thescaled projection operator lies in the null space of C , the bias (classically) vanishes.
To control theerror of estimation, we need to control both terms.
For the special case of interpolation estimator,we will control the second term separately using a bias condition , and the variance term will becontrolled by the information matrix W † as we will see in Section 
 [Equation]
[Equation]
Regularized Regression
The ridge regularized estimator is motivated by using θ (cid:62) V 0 θ ≤ λ − 1 , andminimizing then the trace of this upper bound, leading to an estimator,
[Equation]
Like above, we can give an upper bound on the covariance of the residuals.
Conveniently,this estimator automatically balances the error due to variance and bias in one term:
E ( L λ ) (cid:22)
[Equation]
( I
[Equation]
λ + K )
[Equation]
C (cid:62) .
Using the matrix inversionlemma, we can express the above bound in a more concise form, and subsequently its inversemotivates the deﬁnition of the information matrix for the regularized estimator,
[Equation]
142 3.2 Design objectives: Scalarization
The information matrices such as in (4) and (6) represent the inverse of estimation error, and thegoal of optimal design is to maximize them (hence minimizing the error) with a proper choice of X .
As the Löwner order is not a total order, we need to resort to some scalarization of the informationmatrices, thus we solve max X f ( W ( X )) , where f is the scalarization R p
→ R .
We focus on twocommon forms of scalarization and refer to them as E - and A -design (Pukelsheim, 2006).
• f E ( W ) =
λ min ( W ) – when constructing non-asymptotic high probability conﬁdence sets;
• f A ( W ) =
Tr( W ) – when minimizing mean squared error of estimation.
Other popular criteria include D , V , and G -designs (Chaloner and Verdinelli, 1995), which we donot consider due to space considerations, but can be equivalently used should the experimenter havea reason for it.
If p = 1 , C is an element in H k and the design problem degenerates into a specialcase known as c-optimality due to Elfving (1952) and, as W ∈ R , no scalarizations are needed.
Robust Designs The linear functionals are sometimes unknown, or parametrized by an unknownparameter γ as C γ , where γ belongs to a known set Γ .
If we were to construct a design that has lowestimation error for the worst case selection of γ , we can maximize the information in the followingworst case metric max X inf γ ∈ Γ f (cid:16)(cid:0) C γ V − 1 [-
 X ⊤ K-]0 XK − 2 X (cid:62) V − 1 0 C (cid:62) γ (cid:1) − 1 (cid:17) , for the interpolation estimator(and analogously for ridge regression).
If the original function f ( W ) is concave, then so is thefunction deﬁned as the inﬁmum over the compact index set Γ , which is true for A -and E -designs(Boyd and Vandenberghe, 2004).
Fixed Designs and their Conﬁdence Sets
[Figure]
In RKHS spaces, especially inﬁnite dimensionalones, the estimability without bias is too restrictive.
Given a ﬁnite evaluation budget T , we canonly construct a discrete design X : H κ → 166 R T , and there are many practical examples,where, given a ﬁnite query budget, C θ cannot belearned to arbitrary precision, most prominentlygradients and integrals among many others.
Our goal is to establish a condition onthe design space X such that the estimation withthe interpolation estimator is possible up to a certain bias (cid:15) measured under the Euclidean norm.
We measure the bias scaled by the magnitudeof the Frobenius norm of the estimator and callthis the relative ν -bias.
This condition will allow us to balance the error due to the bias and noise.
Deﬁnition 1 (Relative ν -bias) .
Let C : H κ → R k .
The estimator L † on the design space X is saidto have relative ν
-bias if
[Equation]
The (cid:107)·(cid:107) F corresponds to the Frobenius norm of the maps H κ
→ H κ .
Due to the cyclic property of thetrace, we can take the adjoint of the operator and calculate the quantity by taking a trace of p × p matrix instead.
If ν = 0 and dim( H κ ) = m , as we show in Lemma 1 in the Appendix, this is equivalentto the classical estimability condition due to Pukelsheim (2006).
The left hand side correspondsto the classical bias bias( L † y )
= (cid:107) E
[ L † y ] − C θ (cid:107) 2 of an estimator L † under the Frobenius norm.
This is exactly the quantity that L † minimizes.
The classical bias cannot be improved by repeatedmeasurements or allocations thereof.
Using a relation which we show formally in Proposition 4 in
Appendix B, we can bound the bias( L † y )
= (cid:107) E [Equation] (cid:107)
[Equation] [Equation] / √ λ .
To check whether the condition (7) is satisﬁed given a design space X , one needs to evaluate the trace.
Finding the value of p × p matrix, however, depends strongly on the form of the operator C , and ageneral recipe cannot be provided.
Due to Riesz’s representer theorem, and fact that the range of C isﬁnite-dimensional, this is always possible.
For example, for an integral operator (cid:82) Φ( x ) (cid:62) · q ( x ) dx ,using shorthand v = K − 1 y , it holds that L † y
=
(cid:80) ni
=1 (cid:82) κ ( x, x i ) v
i dx where we used n points.
Notice that we used only evaluations of kernel κ to do this calculation.
Also notice that calculation of(7) reduces to the calculation of the maximum mean discrepancy (Gretton et al., 2005) between theempirical distribution (deﬁned on X ), and q ( x ) for this example.
The general recipe is to (locally)project the Hilbert space on a truncated ﬁnite-dimensional basis
Φ m ( x ) with size m and, in that case,the check involves a solution to a linear system, but this is not always necessary.
4.1 Conﬁdence sets
To construct conﬁdence sets for both estimators considered, we split them into two categories: ﬁxeddesigns , where the evaluation queries do not depend on the observed values y , and adaptive designs ,where the queries Φ( x i ) may depend on prior evaluations y i − 1 , . . .
y 1 .
Proofs are in Appendix B.
Theorem 1 (Interpolation – Fixed Design) .
Under the regression model in Eq.
(1) with T datapoint evaluations, let ˆ θ be the estimate as in (3) .
Let X satisfy the bias from Def. 1 with W † =( C V − 1 /
V † V − 1 /
C (cid:62) ) − 1 .
Then,
[Equation]
where V † =
V − 1 /
X
[Equation]
X
[Equation]
−
[Equation]
− 1 X V − 1 /
[Equation]
, and ξ ( δ )
=
p + 2 (cid:113) p log( 1 δ ) .
Notice that in order to balance the source of the error due to bias and variance with high probability,we need to match σ (cid:112) ξ ( δ )
≈ ν √ λ .
Repeating the queries in X T times reduces σ by 1 / √ T butleaves the bias ν , as well as W † , unchanged.
This is because with the interpolation estimator thenoisy repeated queries are averaged, which can be interpreted as a reduction in variance.
Hence, bybalancing ν/ √ λ with σ (cid:112) ξ ( δ ) / √ T , we balance the bias and variance such that they are of the samemagnitude.
It does not make sense to repeat measurements more times if the bias dominates the errorof estimation.
A detailed example of estimating the gradient with ﬁxed bias is given in Sec. 7.1.
We derive conﬁdence sets for the regularized estimator of Eq.
(5), albeit without the relative bias.
Proposition 2 (Regularized estimate – Fixed Design) .
Under the model in Eq.
(1)with T data point evaluations, let ˆ θ λ be the regularized estimate as in (5) .
Then
[Equation]
(cid:19) ≤ δ, where W λ in Eq.
(6) , and ξ ( δ ) =
p + 2 (cid:113) p log( 1 δ ) .
Notice that, since the regularized estimator is designed to balance the bias and variance automatically,we do not need to speciﬁcally control the bias, which is contained within the information matrix W λ .
Despite this elegant property, the regularized estimator involves a more challenging analysis.
The mainmotivation to study the interpolation estimator is to understand the l 2 error as we show in Section 6.
Adaptive Design and Conﬁdence Sets
To provide conﬁdence sets for adaptively collected data, we need to project the data in X onto C ,where we will denote the projection by Z further on.
With the data projected, we can reason aboutthe reduction of the uncertainty of C θ for each point separately, since to each Φ( x i ) we can associatea unique z i in R p .
Deﬁnition 2 (Projected data) .
Let z ( x ) ∈
R p be a vector ﬁeld s.t.
Φ( x ) V − 1 / 2=
z ( x ) C V − 1 / 2+ j ( x ) ,where x ∈ S ⊆
X and j ( x ) ∈ H κ , |S| = n ,
such that C V − 1 / 2j ( x ) = 0 .
We call this vector ﬁeldprojected data .
Classically, the adaptive conﬁdence sets are understood only for C
=
I , i.e., the identity (AbbasiYadkori et al., 2011).
In fact, we can always derive conﬁdence sets for C θ from conﬁdence sets for θ ,as they only project the ellipsoid to a smaller dimensional space.
However, their resulting size maybe unnecessarily large, as the conﬁdence parameter scales as O (dim( H k )) in general (see Figure 1for a visual example).
The martingale analysis of Abbasi-Yadkori et al.
and related works speciﬁcally assumes thatinformation matrix V t , (where C = I ) can be additively decomposed to information matrices due toa single evaluation V t = (cid:80) ti {+=
+}
Φ( x i )Φ( x i ) (cid:62) .
With the matrix W λ,t , this additive decompositionis not always possible.
Therefore, to utilize the martingale analysis, which requires this additiveproperty, we consider a different information matrix, which upper bounds W λ .
The informationmatrix we use, Ω λ , is constructed from the projections z ( x i ) , which have the additive property.
It gives rise to conﬁdence sets, where, under the ellipsoidal norm, their size scales as Θ( p ) .
Theestimation error depends on Ω λ still, but under this norm, the conﬁdence parameter and informationmatrix are decoupled in a similar way as for the ﬁxed design.
Theorem 3 (Ridge estimate – Adaptive Design) .
Under the regression model in Eq.
(1) with tadaptively collected data points, let ˆ θ λ,t be the regularized estimate as in (5) .
Further, assume that Zis as in Def. 2 where X t =
Z t C + J t V 1 / 2 0 .
Then for all t ≥ 0 ,
[Equation]
det [Equation] det [Equation]
with probability 1 − δ , where Ω λ,t = 1 σ 2 Z (cid:62) t
Z t + λ S and S =
( C V − 1 0 C (cid:62) ) − 1 .
The matrix Z t can be calculated by solving a least-squares problem (projection), whereupon
C V − 1 /
J t = 0 as needed by Deﬁnition 2.
Notice that, on one hand, the above conﬁdence parameter grows only when Z is large, but at the same time, the ellipsoid shrinks only in that caseas well.
Also, the ellipsoid above is necessarily smaller than the one with the information matrix
W λ as W λ (cid:22) Ω λ (Lemma 3 in Appendix).
This means we can use the same conﬁdence parameterto give a bound for (cid:107)·(cid:107) W λ,t .
Notice that the estimator is the same as before, i.e., ˆ θ λ , using X todeﬁne the regression, only the information matrix changes.
We also present a visual comparison in
Figure 1 on a simple example, showing that our ﬁxed and adaptive sets are tighter than projectednon-asymptotic sets.
Using the shorthand β t ( δ ) to deﬁne the conﬁdence parameter in (9), we canin fact bound the error as (cid:107) C ( θ λ,t − θ ) (cid:107) 2 ≤ λ min ( Ω λ,t ) − 1 / 2 β t ( δ ) .
It depends on m = dim( H κ )only via Ω λ,t .
The dependence of the conﬁdence parameter can be at most Θ( (cid:112) log(1 + Tm ) k ) (see
Szepesvari and Lattimore, 2019, Lemma 19.4).
The value of λ min ( Ω λ,t ) − 1 depends on the geometryof the set { Φ( x ) | x ∈ X} and the projection operator C as we show in the next section.
Also notethat the lower bound of Szepesvari and Lattimore (2019)[Ex.
20.2.3] does not apply here, since itmakes a statement about the information matrix W λ .
Convex Relaxations, Geometry and Dimensions
Suppose we are given a candidate set of experiments, i.e., a unique subset of evaluations S ⊂ X , |S| =n and a budget T of total queries.
We seek an allocation X , where the rows of X contain potentiallyrepeated evaluations from S .
How many times should we repeat each experiment in order to ﬁndmax X f ( W ( X )) ?
To address this, experimental design literature relaxes this discrete optimizationproblem, and optimizes over fractional allocation η ∈ ∆ n , where the number of repetitions for Φ( x i )is recovered by rounding (cid:100) η i T (cid:101) .
With this interpretation, the objectives in Sec.
3.2 can be written as
[Equation]
[Equation]
where D ( η ) is the diagonalization operator that produces a diagonal matrix with vector η on thediagonal and X S contains non-repeated elements in S .
We have stated the problem above for theinterpolation estimator and dim( H k ) < ∞ , but it naturally generalizes to kernelized estimators,albeit in a less concise form (see Appendix C).
6.1 Optimizing allocations: experiment design algorithms
Given a subset S , the problem (10) can be approximately solved using either convex optimizationmethods or a greedy algorithm.
A comprehensive review of methods constructing designs η ∗ and rounding techniques to get X is beyond the scope of this work , and not the core issue addressed in thiswork.
We brieﬂy review two versatile approaches for completeness (more details in Appendix C).
Greedy selection Firstly, one can greedily maximize the scalarized information matrix with theupdate rule η t
+1 =
t t +1 η t + 11+ t δ t , δ t = arg max x ∈X f (cid:16) W λ (cid:16) t t +1 η t + 1 t +1 δ x (cid:17)(cid:17) , where f refersto the scalarization and δ x to the discrete measure corresponding to the feature map Φ( x ) .
Due to theform of the update rule, tη t is always an integer.
Convex optimization Alternatively, convex optimization can provably solve the problemto optimality.
Speciﬁcally, we look for an allocation using convex optimization methodsmax η ∈ ∆ n f ( W ◦ ( D ( η ) 1 / 2 X )) where ◦ ∈ {† , λ } .
Care needs to be taken when selecting S , as wediscuss in Appendix C.
The most common algorithms for ODE problems are the Frank-Wolfealgorithm (Todd, 2016) and mirror descent algorithm (Silvey et al., 1978).
Optimal designs needto be rounded in practice.
State-of-art rounding techniques are discussed by Allen-Zhu et al.
(2017)and Camilleri et al.
(2021) for ﬁnite and inﬁnite dimensional spaces, respectively.
6.2 Estimation error and its dimension dependence
If we were to bound the squared error of estimation in high probability, the importance of thescalarization λ min ( W † ) becomes apparent.
Using the Cauchy Schwarz inequality, we get
[Equation]
where the term due to Proposition 1 scales as O ( p ) when properly balanced.
Using the optimal allocation η ∗ , the number of repetitive evaluations is equal to η ∗ T .
Inverting the relation above yields a querycomplexity of the order T ≈ O ( k(cid:15) 2 λ min ( W † ( η ∗ ) − 1 )) .
The optimal value λ min ( W † ( η ∗ )) representsa problem dependent quantity that measures the difﬁculty of estimation and captures the geometry ofthe set { Φ( x ) | x ∈ S ⊂ X} .
It cannot be bounded in general, but it has an elegant geometric interpretation.
In particular, it corresponds to the square inverse of the diameter of the largest inscribed ballin the convex hull of symmetrized { Φ( x ) | x ∈ S} in the range of C (Pukelsheim and Studden, 1993).
At ﬁrst glance, calculating this quantity might seem complicated, but with an example it is apparent.
For example, if Φ( x ) = x s.t. (cid:107) x (cid:107) 22 ≤ 1 (unit l 2 ball in R m ) and C = v , where v is a unit vector( p = 1 ), the inverse diameter of largest ball we can inscribe in direction of v inside (cid:107) x (cid:107) 22 ≤ 1 equatesto 1 , which is independent of d .
This is not surprising, since it represents an “easy” design space,where for any direction v one can ﬁnd an action x aligned with that coincides with the optimaldesign.
This is true even if C has more rows.
On the other hand, if we assume evaluation in thel 1 ball (cid:107) x (cid:107) 1 ≤ 1 and C is a vector of ones (again p = 1 ).
Then the diameter of the inscribed ballis proportional to the inverse square height of a simplex, namely, 1 m , despite p = 1 .
Hence, despitethe conﬁdence parameter being O (1) , the dimension depends primarily on the geometry of this set.
To give a more exotic example, if C = ∇ x Φ( x ) , and the design space are points with ﬁxed lengthsteps in all unit derection { x | Φ( x ± e i h ) } , where h is the stepsize and e i are principal vectors in R d ,we can show that λ min ( W † ) − 1 ≤ dh + O ( h 2 ) , leaving the overall complexity to learn a gradient toscale with d instead of the dimensionality of the RKHS which can be inﬁnite.
All formal proofs andreferences are provided in Appendix D.1.
Applications
We now discuss concrete applications that beneﬁt from our contributions.
Details of the experiments,and further applications, e.g., in statistical contamination, can be found in Appendices E and F.
7.1 Gradient maps
Gradients of any order are linear operators.
We can express the gradient at x as dimension-wiseevaluation of the following operator ∇ x (Φ( x ) (cid:62) θ )
= ( ∇ x Φ( x )) (cid:62) θ
=: C θ.
Clearly, estimability isnearly always impossible, since any evaluation inﬁnitesimally away from x will be insufﬁcient toeliminate bias.
Thus our estimates will invariably be biased for any function with inﬁnite Taylorexpansion.
Yet, while estimating the gradient from evaluations very close to the original x leadsto low bias, it at the same time increases the variance, since the difference in the functional valuebetween the two point evaluations is very small compared to the noise magnitude.
Given a ﬁnitebudget or desired accuracy, the best we can do is to ﬁnd the best design with optimal bias-variancetrade-off given the kernel κ , budget T , and noise variance σ .
We consider a class of parametrized
[Figure]
ﬁnite difference designs { Φ( x ± he i ) } , where h is the stepsize and e i are unit vectors.
In Fig.
2b,we plot the total error with high probability with the budget T as a function of the step-size h .
Notice that the lowest error occurs exactly the with is equal to the relative two lines
7.2 Learning linear ODE solutions and their parameters
A solution to a linear ordinary differential equation (ODE) u satisﬁes ddt u ( t ) =
M u ( t ) + s ( t ) , where
M is a linear operator and s ( t ) is the non-homogeneous term.
Assume that the solution to the equationu ( t ) =
Φ( t ) (cid:62) u is a member of a Hilbert space H κ , then T u :=
(cid:0) ddt − M ( t ) (cid:1) Φ( t ) (cid:62) u
= s ( t )for t ∈ [ t 0 , t 1 ] .
Hence, the differential equation becomes a linear constraint for estimating ufrom samples.
In fact, due to differential equations being fully speciﬁed by initial conditions, theonly unknowns are due to initial conditions.
To reveal the linear functional here, one needs toconsider the solution to the differential equation, which can be written as u =
T † s ( t ) + C (cid:62) v ,where v ∈ H κ \ span( T ) belongs to the null space of T .
In this case, we span it with rows of
C .
Consequently, the unknown element v can be found as C ( u − T † s ) .
Thus, what needs to beestimated from samples, is the linear projection C u , since CT † s is known a priori.
In Appendix
F, we discuss implementation and calculation of the operator C on a discretized domain.
Robust Design As a speciﬁc example, consider an example of a pharmacokinetic model capturingthe concentration of a medication in blood and stomach via differential equations: ( d/dt )
c s = − ac sand ( d/dt ) c b = bc s − dc b for t ∈ [ t 0 , t 1 ] , where c s and c b are the concentration in stomach and bloodrespectively.
The goal of this analysis is to infer γ = ( a, b, d ) from the measurements of the bloodconcentration levels with ﬁxed, but perhaps noisy, initial conditions (Gabrielsson and Weiner, 1995).
To apply the above procedure, we need a ﬁxed differential operator T γ to deﬁne the operator C γ .
However, γ is itself unknown.
Instead, we can give a generally plausible set of γ ∈ Γ , initial conditions of concentration in blood c b (0) = 0 , and norm constraints on the initial stomach concentration( c s (0) − c dose ) 2 ≤ λ − 1 (prior V 0 ) in order to apply our framework.
We use the squared exponentialkernel to embed trajectories and consider robust A -optimal design (as in Sec. 3.2) by maximizing theworst case metric inf γ ∈ Γ
Tr(( C γ V − 1 λ C (cid:62) γ ) − 1 ) with the regularized estimator.
This way, the designis appropriate for any γ .
After estimating the trajectory, we can use the estimated trajectories (givenγ ) to optimize for γ via the maximum likelihood.
Fig.
4a presents the concentrations c b and c s withtheir estimates and uncertainties due to the unknown γ .
We show the equally spaced (black) andoptimized designs (yellow).
The more accurately we can infer the trajectory, the more accurately we can estimate γ , which we quantitatively show in Appendix E in Fig.
4b, where we decrease the
MSE of estimating γ by a factor of 10 in comparison to equally spaced design.
7.3 Sequential Design: Certifying Lyapunov Stability
Consider a non-linear system with x ∈ R d such that ddt x ( t ) =
f ( x ( t ) , t )+ u ( x ( t ) , t ) =
A Φ( x ( t ) , t )+u ( x ( t ) , t ) , where the rows of A i ∈ H κ for i ∈
[ d ] model the system dynamics and Φ( x ( t ) , t ) areknown evaluation functionals of H κ .
We assume that the control laws u ( x ( t ) , t ) can be written asu ( x ( t ) , t ) =
B Φ( x ( t ) , t ) .
We want to understand whether a given a control law B stabilizes theabove system.
A common approach is to create an estimate ˆ A of A from data samples of trajectories,and use the controller B
=
ˆ A − P , where P is often a simple reverting law with a known gain.
Withthis choice of u , the system can be written as ddt x ( t ) =
( A − ˆ A − P )Φ( x ( t ) , t ) =
( ˜ A − P )
Φ( x ( t ) , t )where the ˜ A is uncertain.
We can certify the stability of the resulting system using a known quadratic Lyapunov function, e.g., V ( x, t ) =
( x ( t ) − x ref ( t )) (cid:62) Σ ( x ( t ) − x ref ( t )) .
Classical stability theorydictates that if the total derivative of V ,
[Equation]
(11) is negative for all x ( t ) ∈
O then we can guarantee stability in the operating region of x ∈ O (Khalil,2002).
The above condition deﬁnes a linear operator
C x = ( x − x ref ) (cid:62) Σ · φ ( x ) operating on theunknown ˜ A for each x ∈ O .
The linearity is best seen with vectorization as θ = vec( A ) , usingthe shorthand z ( t ) =
x ( t ) − x ref ( t ) , z (cid:62) ΣA φ ( x ) = vec( Σ zφ ( x ) (cid:62) ) (cid:62) vec( A ) =
C x θ.
The operatoris parametrized by x , C x for each x ∈ O .
Even for continuous domains x ∈ O , C x usually haslow-rank structure, which can be calculated and depends on the size O and the size H κ .
If the rankof the operator is small (e.g., the operating space is small) then we can certify negativity of Eq.
(11)faster than learning the whole A .
In other words, we reduce uncertainty only where we need to as inthe seminal work of Berkenkamp et al.
We can sequentially query data points from x andcheck whether C x θ ≤ 0 for all x .
Consider a two dimensional nonlinear system from Lederer et al.
[Equation]
The reverting controller is P Φ( x ( t ) , t ) =
− K ( x − x ref ( t ))
+ ( d/dt ) x ref , where the referencetrajectory corresponds to a circle x ref ( t ) =
(sin( t ) , cos( t )) .
The Lyapunov function is V =( x
( t ) − x ref ( t )) (cid:62) ( x ( t ) − x ref ( t )) .
We assume that we can set the system to an initial condition x (0) andobserve a noisy observations of the state y
(0 + ∆) at rapid sampling times ∆ .
From these, we createa derivative oracle, ( d/dt ) x
( t ) ≈
( x ( t +∆) − x ( t )) / ∆ , a common approach in nonlinear data-drivencontrol (Umlauft et al., 2018).
We use this example to showcase our adaptive conﬁdence sets.
We follow an adaptive stopping rule, where we query new data points if negativity cannot be certiﬁed.
Dueto this adaptive stopping rule, the data is adaptively collected.
We compare our conﬁdence sets (solid)with the classical conﬁdence sets of Abbasi-Yadkori et al.
(2011) (dashed) and report upper bounds onthe supremum over x ∈ O of Eq.
(11) in Fig.
2c.
The operating region is a “tube” around the circularreference trajectory x ref ( t ) .
We see that our conﬁdence sets shrink much faster than the classicalconﬁdence sets, since we can eliminate redundant information by projecting onto C .
Fig.
2c comparesrandom sampling of datapoints from the whole domain (random) and the operating region (randomref) and sampling according to uncertainty in the dynamics in the whole domain (unc) and within thetube around the reference trajectory (unc-ref).
As expected, focused exploration methods work muchbetter.
However, and more importantly, the tightness of our conﬁdence sets enable much quicker stability certiﬁcation (termination).
Note that the classical conﬁdence sets may even grow in the cases whereredundant information for estimation of C is inserted into the estimation, e.g., with random sampling.
CONCLUSION
[Figure] We considered the problem of learning a linear function of an element in reproducing kernel Hilbertspaces.
We addressed the challenging case where linear estimators incur a non-negligible bias andprovided conﬁdence sets for the two most commonly used linear estimators.
We demonstratedthe generality of our approach and the tightness of our conﬁdence