Experimental Design for Linear Functionals in Reproducing Kernel Hilbert Spaces
Anonymous Author(s)
AffiliationAddress email
Abstract
Optimal experimental design seeks to determine the most informative allocation ofexperiments to infer an unknown statistical quantity.
In this work, we investigate theoptimal design of experiments for estimation of linear functionals in reproducingkernel Hilbert spaces (RKHSs) .
This problem has been extensively studied in thelinear regression setting under an estimability condition, which allows estimatingparameters without bias.
We generalize this framework to RKHSs, and allowfor the linear functional to be only approximately inferred, i e., with a fixed bias.
This scenario captures many important modern applications, such as estimationof gradient maps, integrals, and solutions to differential equations.
We providealgorithms for constructing bias-aware designs for linear functionals.
We derivenon-asymptotic confidence sets for fixed and adaptive designs under sub-Gaussiannoise, enabling us to certify estimation with bounded error with high probability
Introduction
Optimal Experimental Design (OED) aims to determine data collection schemes – designs – toefficiently estimate unknown quantities of interest given limited resources (Chaloner and Verdinelli,1995).
As common, we model experiments via an oracle that yields a (noisy) response to a giveninput.
A design is usually either an (adaptive) policy for querying the oracle, or a (nonadaptive) fixedallocation of query budget to different oracle inputs.
OED has a rich history and close relations to thefield of bandits (Szepesvari and Lattimore, 2020) and active learning (Settles, 2009).
We consider the regression setting, where observations at a fixed input x ∈ X can be obtained via thenoisy oracle by:
[Equation]
+ ϵ where x ∈ X (1) and · ⊤ · denotes the inner product in a Hilbert space H κ , ϵ is independent, sub-Gaussian noise withknown variance proxy σ 2 , and θ is a bounded element from a separable reproducing kernel Hilbertspace H κ from kernel κ , with a bound θ ⊤ V 0 θ ≤ λ − 1 , where V 0 :
H k
→ H k is a positive definiteoperator.
Depending on whether λ is known or unknown, we will propose different estimators.
Incontrast to the classical ED task of estimating θ (see Fedorov and Hackl, 1997), we are interested inestimating a projection of θ .
Namely, let C : H κ → R p be a known linear operator .
The map C issuch that C ⊤ C is full rank p , where C ⊤ denotes the adjoint.
Our goal is to identify an estimate of
C θ efficiently, i.e., with low query complexity or with a maximal reduction of uncertainty given afixed query budget T .
The formalism above captures, or occurs as a subroutine in, numerous practical problems.
Forexample, evaluation at specific target points, integration and differentiation are all linear operators,among many other useful linear functionals.
Other examples include ordinary or partial differential equations operators, spectral transforms, and stability metrics from control engineering.
In Section 7we detail several example applications.
A naive approach would be to first obtain an estimate ˆ θ of θ , and compute C ˆ θ .
However, the appealof estimating linear functionals directly is that the number of unknowns of interest may be muchlower than for the original overall unknown element θ (which might even be infinite-dimensional).
Consequently, we would hope that the query complexity of reducing the variance of the estimate
C θ scales in the dimension of the range of C , which is p .
For example, when focusing on finitedimensional RKHSs, the operator C : R m → R p , where m is the dimension of H κ , becomes amatrix.
In this work, we study the cases where p < m , and show that the estimation error can indeedscale with p , and the geometry of the query set { Φ( x ) | x ∈ X} .
The estimation bias plays a centralrole in this work.
It depends among other things the richness of the query set X , as we will detaillater in Sec. 3.
Sequential Experiment Design Apart from classical experiment design, where we first commit towhat design (set of inputs x ) we choose, referred to as a fixed design , we also consider sequentialdesign, where data is acquired depending on past observations.
Suppose for example, we are gatheringdata to test whether the null H 0 : θ ⊤ Φ( x ) ≥ 0 for all x ∈ X or otherwise.
We can incrementallygather evidence and check whether the null hypothesis has already been rejected.
As our data dependson prior evaluation points it forms an adaptive design .
In this work, we develop confidence sets forboth fixed and adaptive designs – which are of paramount importance in the context of sequentialexperiment designs used, e.g., to define stopping rules of adaptive hypothesis testing problems.
Contributions A)
We consider objectives for experiment design for linear functionals in general
RKHS spaces, which carefully take the bias of the estimator into account.
B)
We provide bounds onthe query complexity required to reach ϵ accuracy to estimate linear functionals with high probability.
C)
We construct novel non-asymptotic confidence sets for linear estimators of linear functionals of
RKHS elements, both for fixed and adaptive designs, where queries are independent of previous noiserealizations, and where they are not, respectively.
D)
We demonstrate the improved inference errordue to specially defined designs and new confidence sets on the problems of learning trajectories ofdifferential equations, estimation of gradient maps, statistical contamination, and stability verificationfor non-linear systems.
Background and Related Work
Linear Estimators Let S ⊂
X be a finite set of selected evaluations s.t. |S| = n , potentiallyrepeated.
We focus on linear estimators of the form L : R n → R p , where L y is estimating C θ .
Anestimator is understood here as the algorithm to find the random estimate L y .
Notice that given S , L isnot a random quantity; the randomness rather comes from the realizations y .
To choose the estimator L , one classically looks at the second moment of the residuals C θ − L y , E ( L ) =
E [( C θ − L y )
( C θ − L y ) ⊤ ] , where the argument signifies how the random variable y is transformed (noise is averaged)
[Equation]
L [Equation] + ( LX − C ) θθ ⊤ ( LX − C ) ⊤ , where E ( L ) ∈
R p × p , (2) and the matrix X contains stacked evaluation functionals of the RKHS X i,x
= Φ i ( x ) for x ∈ S ⊆ X .
We will then seek a way to transform y via estimator L such that the second moment of residualsis minimized in certain sense.
Importance of Bias and RKHS Classical experimental design studies estimation of C : R m → R pwhere the RKHS is finite dimensional (Pukelsheim, 2006) and X ∈ R n × m .
On top of that, theyconsider only estimators which are unbiased for any θ , in other words, LX = C .
While, withfinite dimensions, this simplification might be reasonable, for infinite dimensionial RKHSs, biasis inevitable and must be controlled.
Consider the case of estimating the gradient of a continuousfunction f .
We can nearly never learn it up to arbitrary precision from noisy point queries.
However,estimation up to a small error is always possible and sufficient.
Estimating ∇ x f ( x ) from points closeto f ( x ) will incur small bias but larger variance as the change in f compared noise ϵ is small.
Hencebalancing these two sources of error is crucial for an informative design in RKHSs.
Experiment Design: Classical Perspective Consider an m dimensional version of the model in(1).
Then, among all unbiased estimators, linear least-squares estimators minimize (2) under the
Löwner order due to the famed Gauss-Markov theorem.
Unbiasedness in this form is synonymouswith estimability , which means that by repeating the evaluation in X , arbitrary precision can bereached (Pukelsheim, 2006).
The second moment of residuals then becomes W − 1 † [Equation] where · † denotes a generalized pseudo-inverse and V = X
⊤ X .
The matrix W is often referred toas the information matrix .
Gaffke (1987) and Krafft (1983) note that estimability implies that W †is non-singular.
We relax this condition and allow the estimator to have a bias ; this means that wecannot in general reduce the error arbitrarily by repeating the measurements.
In fact, our extensionuses the fact that W † as defined above will not be singular even if estimability condition is notsatisfied.
We will show that the matrix W † can still play the role of the information matrix.
Mutný et al.
(2020) use experimental design to estimatethe Hessian of an unknown RKHS function, while Kirschner et al.
(2019) use it to estimate the gradientof it for use in Bayesian optimization.
Perhaps most related, Shoham and Avron (2020) study overparametrized experimental design for one-shot active deep learning and analyze the bias in connectionto experiment design, similarly as in the seminal work of Bardow (2008).
They do not consider biasarising from the limited design space nor do they treat linear functionals.
More broadly, the uncertaintypropagation is studied in the Bayesian framework in the field of probabilistic numerics for linear andnonlinear operators (see Cockayne et al., 2017; Owhadi and Scovel, 2016, and citations therein).
Confidence sets Unlike classical statistics, our focus is on non-asymptotic confidence sets onregression estimators.
They can be found in, e.g., Draper and Smith (2014) and Abbasi-Yadkoriet al.
(2011), for fixed and adaptive designs, respectively.
Our goal is to define the confidence setsin the appropriate norm such that their width scales with the dimension of the range of C , i.e., p ,and the geometry of { Φ( x ) | x ∈ X} , but not directly dim( H κ ) nor number of data T .
Mutný et al.(2020) derive non-asymptotic confidence sets that scale in p , but grow with the number of points
Tas they do not use the appropriate norm (see Appendix B.4).
Similarly, Khamaru et al.
studyestimators for adaptively collected data, and propose asymptotic confidence sets for their estimators.
Without a specific condition, their confidence sets can grow with T , however, they consider moregeneral noise distributions apart from sub-Gaussian as we do here.
Estimation and Bias
In this section, we motivate the linear estimators and identify information matrices .
Informationmatrices are the inverses of second moment matrices of residuals E as in Eq.
They are animportant object in the analysis of the error of the estimators and their confidence sets.
Also, theydepend on the evaluations that define the estimator L , X .
Maximizing the information matrices asa function of the chosen observations and their proportions X gives rise to optimal experimentaldesigns.
Further, we identify quantities that influence the bias of the estimators.
We study two estimators: theleast-norm estimator (interpolation), when the bound on ∥ θ ∥ V 0 is unknown but finite, or the ridgeregularized least squares estimator, where ∥ θ ∥ V 0 ≤ λ − 1 is known.
Both estimators are motivated asminimizing the error residuals E in trace norm under these two bound assumptions.
3.1 Estimators
Interpolation As apparent from Eq.
(2), without the knowledge of an explicit bound on the normθ , the worst case over θ causes the optimal estimator L to minimize only the trace of the bias.
Thisleads to minmization of ( C − LX )
V − 1 / 2in Frobenius norm, leading to the familiar interpolationestimator ,
[Equation]
The second moment of residuals can then be expressed as
[Equation]
variance
[Equation]
, 128 where P X = V − 1 / {+
 (+} V − 1 /
[Equation]
− I ) V −
[Equation]
is a scaled projection matrix.
Unlike as inthe classical ED treatment, the second moment has two terms: bias and variance.
If the span of thescaled projection operator lies in the null space of C , the bias (classically) vanishes.
To control theerror of estimation, we need to control both terms.
For the special case of interpolation estimator,we will control the second term separately using a bias condition , and the variance term will becontrolled by the information matrix W † as we will see in Section
[Equation]
Regularized Regression
The ridge regularized estimator is motivated by using θ ⊤ V 0 θ ≤ λ − 1 , andminimizing then the trace of this upper bound, leading to an estimator,
[Equation]
Like above, we can give an upper bound on the second moment of the residuals.
Conveniently, this estimator automatically balances the error due to variance and bias in one term:
E ( L λ ) ⪯ [Equation]
⊤
[Equation]
⊤
[Equation]
X ⊤ ( I
[Equation]
2 λ + K ) − 1 X V − 1 0 C ⊤ .
Using the matrix inversion lemma, wecan express the above bound in a more concise form, and subsequently its inverse motivates thedefinition of the information matrix for the regularized estimator,
[Equation]
143 3.2 Design objectives: Scalarization
The information matrices such as in (4) and (6) represent the inverse of estimation error, and thegoal of optimal design is to maximize them (hence minimizing the error) with a proper choice of X .
As the Löwner order is not a total order, we need to resort to some scalarization of the informationmatrices, thus we solve max X f ( W ( X )) , where f is the scalarization R p
→ R .
We focus on twocommon forms of scalarization and refer to them as E - and A -design (Pukelsheim, 2006).
• f E ( W ) =
λ min ( W ) – when constructing non-asymptotic high probability confidence sets;
• f A ( W )
= 1 / Tr( W − 1 ) – when minimizing mean squared error of estimation.
Other popular criteria include D , V , and G -designs (Chaloner and Verdinelli, 1995), which we donot consider due to space considerations, but can be equivalently used should the experimenter havea reason for it.
If p = 1 , C is an element in H k and the design problem degenerates into a specialcase known as c-optimality due to Elfving (1952) and, as W ∈ R , no scalarizations are needed.
Robust Designs The linear functionals are sometimes unknown, or parametrized by an unknownparameter γ as C γ , where γ belongs to a known set Γ .
If we were to construct a design that has lowestimation error for the worst case selection of γ , we can maximize the information in the followingworst case metric max X inf γ ∈ Γ f (cid:16)(cid:0) C γ V − 1 {+
 X ⊤ K+} − 2 X V − 1 0 C ⊤ γ (cid:1) − 1 (cid:17) , for the interpolation estimator(and analogously for ridge regression).
If the original function f ( W ) is concave, then so is thefunction defined as the infimum over the compact index set Γ , which is true for A -and E -designs(Boyd and Vandenberghe, 2004).
Fixed Designs and their Confidence Sets
In RKHS spaces, especially infinite dimensionalones, the estimability without bias is too restrictive.
Given a finite evaluation budget T , we canonly construct a discrete design X : H κ → 167 R T , and there are many practical examples,where, given a finite query budget, C θ cannot belearned to arbitrary precision, most prominentlygradients and integrals among many others.
Estimation with Bias and Interpolation Estimator Our goal is to establish a conditionon the design space X such that the estimationwith the interpolation estimator is possible up toa certain bias ϵ measured under the Euclideannorm.
We measure the bias scaled by the magnitude of the Frobenius norm of the estimator andcall this the relative ν -bias.
This condition will allow us to balance the error due to the bias and noise.
Definition 1 (Relative ν -bias) .
Let C : H κ → R k .
The estimator L † on the design space X is saidto have relative ν
-bias if
[Equation]
The ∥·∥ F corresponds to the Frobenius norm of the maps H κ
→ H κ .
Due to the cyclic property of thetrace, we can take the adjoint of the operator and calculate the quantity by taking a trace of p × p matrix instead.
If ν = 0 and dim( H κ ) = m , as we show in Lemma 1 in the Appendix, this is equivalentto the classical estimability condition due to Pukelsheim (2006).
The left hand side correspondsto the classical bias bias( L † y ) =
∥
E [ L † y ] − C θ ∥ 2 of an estimator L † under the Frobenius norm.
This is exactly the quantity that L † minimizes.
The classical bias cannot be improved by repeatedmeasurements or allocations thereof.
Using a relation which we show formally in Proposition 4 in
Appendix B, we can bound the bias( L † y ) =
∥
E [ L † y − C θ ] ∥ 2 ≤ λ min ( W † ) − 1 / 2 ν/ √ λ .
To check whether the condition (7) is satisfied given a design space X , one needs to evaluate the trace.
Finding the value of p × p matrix, however, depends strongly on the form of the operator C , and ageneral recipe cannot be provided.
Due to Riesz’s representer theorem, and fact that the range of C isfinite-dimensional, this is always possible.
For example, for an integral operator (cid:82) Φ( x ) ⊤ · q ( x ) dx ,using shorthand v = K − 1 y , it holds that L † y
=
(cid:80) ni
=1 (cid:82) κ ( x, x i ) v
i dx where we used n points.
Notice that we used only evaluations of kernel κ to do this calculation.
Also notice that calculation of(7) reduces to the calculation of the maximum mean discrepancy (Gretton et al., 2005) between theempirical distribution (defined on X ), and q ( x ) for this example.
The general recipe is to (locally)project the Hilbert space on a truncated finite-dimensional basis
Φ m ( x ) with size m and, in that case,the check involves a solution to a linear system, but this is not always necessary.
4.1 Confidence sets
To construct confidence sets for both estimators considered, we split them into two categories: fixeddesigns , where the evaluation queries do not depend on the observed values y , and adaptive designs ,where the queries Φ( x i ) may depend on prior evaluations y i − 1 , . . .
y 1 .
Proofs are in Appendix B.
Theorem 1 (Interpolation – Fixed Design) .
Under the regression model in Eq.
(1) with T datapoint evaluations, let ˆ θ be the estimate as in (3) .
Let X satisfy the bias from Def. 1 with W † =( C V − 1 /
V † V − 1 /
C ⊤ ) − 1 .
Then,
[Equation]
where V † =
V − 1 /
[Equation]
, and ξ ( δ )
=
p + 2 (cid:113) p log( 1 δ ) .
Notice that in order to balance the source of the error due to bias and variance with high probability,we need to match σ (cid:112) ξ ( δ )
≈ ν √ λ .
Repeating the queries in X T times reduces σ by 1 / √ T butleaves the bias ν , as well as W † , unchanged.
This is because with the interpolation estimator thenoisy repeated queries are averaged, which can be interpreted as a reduction in variance.
Hence, bybalancing ν/ √ λ with σ (cid:112) ξ ( δ ) / √ T , we balance the bias and variance such that they are of the samemagnitude.
It does not make sense to repeat measurements more times if the bias dominates the errorof estimation.
A detailed example of estimating the gradient with fixed bias is given in Sec. 7.1.
We derive confidence sets for the regularized estimator of Eq.
(5), albeit without the relative bias.
Proposition 2 (Regularized estimate – Fixed Design) .
Under the model in Eq.
(1)with T data point evaluations, let ˆ θ λ be the regularized estimate as in (5) .
Then
P
[Equation]
(cid:19) ≤ δ, where W λ in Eq.
(6) , and ξ ( δ ) =
p + 2 (cid:113) p log( 1 δ ) .
Notice that, since the regularized estimator is designed to balance the bias and variance automatically,we do not need to specifically control the bias, which is contained within the information matrix W λ .
Despite this elegant property, the regularized estimator involves a more challenging analysis.
The mainmotivation to study the interpolation estimator is to understand the l 2 error as we show in Section 6.
Adaptive Design and Confidence Sets
To provide confidence sets for adaptively collected data, we need to project the data in X onto C ,where we will denote the projection by Z further on.
With the data projected, we can reason aboutthe reduction of the uncertainty of C θ for each point separately, since to each Φ( x i ) we can associatea unique z i in R p .
Definition 2 (Projected data) .
Let z ( x ) ∈
R p be a vector field s.t.
Φ( x ) V − 1 / 2=
z ( x ) C V − 1 / 2+ j ( x ) ,where x ∈ S ⊆
X and j ( x ) ∈ H κ , |S| = n ,
such that C V − 1 / 2j ( x ) = 0 .
We call this vector fieldprojected data .
Classically, the adaptive confidence sets are understood only for C
=
I , i.e., the identity (AbbasiYadkori et al., 2011).
In fact, we can always derive confidence sets for C θ from confidence sets for θ ,as they only project the ellipsoid to a smaller dimensional space.
However, their resulting size maybe unnecessarily large, as the confidence parameter scales as O (dim( H k )) in general (see Figure 1for a visual example).
The martingale analysis of Abbasi-Yadkori et al.
and related works specifically assumes thatinformation matrix V t , (where C = I ) can be additively decomposed to information matrices due toa single evaluation V t = (cid:80) ti [-=
-]=
1 Φ( x i )Φ( x i ) ⊤ .
With the matrix W λ,t , this additive decompositionis not always possible.
Therefore, to utilize the martingale analysis, which requires this additiveproperty, we consider a different information matrix, which upper bounds W λ .
The informationmatrix we use, Ω λ , is constructed from the projections z ( x i ) , which have the additive property.
It gives rise to confidence sets, where, under the ellipsoidal norm, their size scales as Θ( p ) .
Theestimation error depends on Ω λ still, but under this norm, the confidence parameter and informationmatrix are decoupled in a similar way as for the fixed design.
Theorem 3 (Ridge estimate – Adaptive Design) .
Under the regression model in Eq.
(1) with tadaptively collected data points, let ˆ θ λ,t be the regularized estimate as in (5) .
Further, assume that Zis as in Def. 2 where X t =
Z t C + J t V 1 / 2 0 .
Then for all t ≥ 0 ,
[Equation]
(cid:19)
+(9) with probability 1 − δ , where Ω λ,t = 1 σ 2 Z ⊤ t
Z t + λ S and S =
( C V − 1 0 C ⊤ ) − 1 .
The matrix Z t can be calculated by solving a least-squares problem (projection), whereupon
C V − 1 /
J t = 0 as needed by Definition 2.
Notice that, on one hand, the above confidence parameter grows only when Z is large, but at the same time, the ellipsoid shrinks only in that caseas well.
Also, the ellipsoid above is necessarily smaller than the one with the information matrix
W λ as W λ ⪯ Ω λ (Lemma 3 in Appendix).
This means we can use the same confidence parameterto give a bound for ∥·∥ W λ,t .
Notice that the estimator is the same as before, i.e., ˆ θ λ , using X todefine the regression, only the information matrix changes.
We also present a visual comparison in
Figure 1 on a simple example, showing that our fixed and adaptive sets are tighter than projectednon-asymptotic sets.
Using the shorthand β t ( δ ) to define the confidence parameter in (9), we canin fact bound the error as ∥ C ( θ λ,t − θ ) ∥ 2 ≤ λ min ( Ω λ,t ) − 1 / 2 β t ( δ ) .
It depends on m = dim( H κ )only via Ω λ,t .
The dependence of the confidence parameter can be at most Θ( (cid:112) log(1 + Tm ) k ) (see
Szepesvari and Lattimore, 2020, Lemma 19.4).
The value of λ min ( Ω λ,t ) − 1 depends on the geometryof the set { Φ( x ) | x ∈ X} and the projection operator C as we show in the next section.
Also notethat the lower bound of Szepesvari and Lattimore (2020)[Ex.
20.2.3] does not apply here, since itmakes a statement about the information matrix W λ .
Convex Relaxations, Geometry and Dimensions
Suppose we are given a candidate set of experiments, i.e., a unique subset of evaluations S ⊂ X , |S| =n and a budget T of total queries.
We seek an allocation X , where the rows of X contain potentiallyrepeated evaluations from S .
How many times should we repeat each experiment in order to findmax X f ( W ( X )) ?
To address this, experimental design literature relaxes this discrete optimizationproblem, and optimizes over fractional allocation η ∈ ∆ n , where the number of repetitions for Φ( x i )is recovered by rounding ⌈ η i T ⌉ .
With this interpretation, the objectives in Sec.
3.2 can be written as
[Equation]
where D ( η ) is the diagonalization operator that produces a diagonal matrix with vector η on thediagonal and X S contains non-repeated elements in S .
We have stated the problem above for theinterpolation estimator and dim( H k ) < ∞ , but it naturally generalizes to kernelized estimators,albeit in a less concise form (see Appendix C).
6.1 Optimizing allocations: experiment design algorithms
Given a subset S , the problem (10) can be approximately solved using either convex optimizationmethods or a greedy algorithm.
A comprehensive review of methods constructing designs η ∗ and rounding techniques to get X is beyond the scope of this work , and not the core issue addressed in thiswork.
We briefly review two versatile approaches for completeness (more details in Appendix C).
Greedy selection Firstly, one can greedily maximize the scalarized information matrix with theupdate rule η t
+1 =
t t +1 η t + 11+ t δ t , δ t = arg max x ∈X f (cid:16) W λ (cid:16) t t +1 η t + 1 t +1 δ x (cid:17)(cid:17) , where f refersto the scalarization and δ x to the discrete measure corresponding to the feature map Φ( x ) .
Due to theform of the update rule, tη t is always an integer.
Convex optimization Alternatively, convex optimization can provably solve the problemto optimality.
Specifically, we look for an allocation using convex optimization methodsmax η ∈ ∆ n f ( W ◦ ( D ( η ) 1 / 2 X )) where ◦ ∈ {† , λ } .
Care needs to be taken when selecting S , as wediscuss in Appendix C.
The most common algorithms for ODE problems are the Frank-Wolfealgorithm (Todd, 2016) and mirror descent algorithm (Silvey et al., 1978).
Optimal designs needto be rounded in practice.
State-of-art rounding techniques are discussed by Allen-Zhu et al.
(2017)and Camilleri et al.
(2021) for finite and infinite dimensional spaces, respectively.
6.2 Estimation error and its dimension dependence
If we were to bound the squared error of estimation in high probability, the importance of thescalarization λ min ( W † ) becomes apparent.
Using the Cauchy Schwarz inequality, we get
[Equation]
where the term due to Proposition 1 scales as O ( p ) when properly balanced.
Using the optimal allocation η ∗ , the number of repetitive evaluations is equal to η ∗ T .
Inverting the relation above yields a querycomplexity of the order T ≈ O ( kϵ 2 λ min ( W † ( η ∗ ) − 1 )) .
The optimal value λ min ( W † ( η ∗ )) representsa problem dependent quantity that measures the difficulty of estimation and captures the geometry ofthe set { Φ( x ) | x ∈ S ⊂ X} .
It cannot be bounded in general, but it has an elegant geometric interpretation.
In particular, it corresponds to the square inverse of the diameter of the largest inscribed ballin the convex hull of symmetrized { Φ( x ) | x ∈ S} in the range of C (Pukelsheim and Studden, 1993).
At first glance, calculating this quantity might seem complicated, but with an example it is apparent.
For example, if Φ( x ) = x s.t. ∥ x ∥ 22 ≤ 1 (unit l 2 ball in R m ) and C = v , where v is a unit vector( p = 1 ), the inverse diameter of largest ball we can inscribe in direction of v inside ∥ x ∥ 22 ≤ 1 equatesto 1 , which is independent of d .
This is not surprising, since it represents an “easy” design space,where for any direction v one can find an action x aligned with that coincides with the optimaldesign.
This is true even if C has more rows.
On the other hand, if we assume evaluation in thel 1 ball ∥ x ∥ 1 ≤ 1 and C is a vector of ones (again p = 1 ).
Then the diameter of the inscribed ballis proportional to the inverse square height of a simplex, namely, 1 m , despite p = 1 .
Hence, despitethe confidence parameter being O (1) , the dimension depends primarily on the geometry of this set.
To give a more exotic example, if C = ∇ x Φ( x ) , and the design space are points with fixed lengthsteps in all unit derection { x | Φ( x ± e i h ) } , where h is the stepsize and e i are principal vectors in R d ,we can show that λ min ( W † ) − 1 ≤ dh + O ( h 2 ) , leaving the overall complexity to learn a gradient toscale with d instead of the dimensionality of the RKHS which can be infinite.
All formal proofs andreferences are provided in Appendix D.1.
Applications
We now discuss concrete applications that benefit from our contributions.
Details of the experiments,and further applications, e.g., in statistical contamination, can be found in Appendices E and F.
7.1 Gradient maps
Gradients of any order are linear operators.
We can express the gradient at x as dimension-wiseevaluation of the following operator ∇ x (Φ( x ) ⊤ θ ) =
( ∇ x Φ( x )) ⊤ θ
=: C θ.
Clearly, estimability isnearly always impossible, since any evaluation infinitesimally away from x will be insufficient toeliminate bias.
Thus our estimates will invariably be biased for any function with infinite Taylorexpansion.
Yet, while estimating the gradient from evaluations very close to the original x leadsto low bias, it at the same time increases the variance, since the difference in the functional valuebetween the two point evaluations is very small compared to the noise magnitude.
Given a finitebudget or desired accuracy, the best we can do is to find the best design with optimal bias-variancetrade-off given the kernel κ , budget T , and noise variance σ .
We consider a class of parametrized
[Figure]
finite difference designs { Φ( x ± he i ) } , where h is the stepsize and e i are unit vectors.
In Fig.
2b,we plot the total error with high probability with the budget T as a function of the step-size h .
Notice that the lowest error occurs exactly when the observations scaled with is equal to the relative (two lines
7.2 Learning linear ODE solutions and their parameters
A solution to a linear ordinary differential equation (ODE) u satisfies ddt u ( t ) =
M u ( t ) + s ( t ) , where
M is a linear operator and s ( t ) is the non-homogeneous term.
Assume that the solution to the equationu ( t ) =
Φ( t ) ⊤ u is a member of a Hilbert space H κ , then T u : =
(cid:0) ddt − M ( t ) (cid:1) Φ( t ) ⊤ u
= s ( t )for t ∈ [ t 0 , t 1 ] .
Hence, the differential equation becomes a linear constraint for estimating ufrom samples.
In fact, due to differential equations being fully specified by initial conditions, theonly unknowns are due to initial conditions.
To reveal the linear functional here, one needs toconsider the solution to the differential equation, which can be written as u =
T † s ( t ) + C ⊤ v ,where v ∈ H κ \ span( T ) belongs to the null space of T .
In this case, we span it with rows of
C .
Consequently, the unknown element v can be found as C ( u − T † s ) .
Thus, what needs to beestimated from samples, is the linear projection C u , since CT † s is known a priori.
In Appendix
F, we discuss implementation and calculation of the operator C on a discretized domain.
Robust Design As a specific example, consider an example of a pharmacokinetic model capturingthe concentration of a medication in blood and stomach via differential equations: ( d/dt )
c s = − ac sand ( d/dt ) c b = bc s − dc b for t ∈ [ t 0 , t 1 ] , where c s and c b are the concentration in stomach and bloodrespectively.
The goal of this analysis is to infer γ = ( a, b, d ) from the measurements of the bloodconcentration levels with fixed, but perhaps noisy, initial conditions (Gabrielsson and Weiner, 1995).
To apply the above procedure, we need a fixed differential operator T γ to define the operator C γ .
However, γ is itself unknown.
Instead, we can give a generally plausible set of γ ∈ Γ , initial conditions of concentration in blood c b (0) = 0 , and norm constraints on the initial stomach concentration( c s (0) − c dose ) 2 ≤ λ − 1 (prior V 0 ) in order to apply our framework.
We use the squared exponentialkernel to embed trajectories and consider robust A -optimal design (as in Sec. 3.2) by maximizing theworst case metric inf γ ∈ Γ
Tr(( C γ V − 1 λ C ⊤ γ ) − 1 ) with the regularized estimator.
This way, the designis appropriate for any γ .
After estimating the trajectory, we can use the estimated trajectories (givenγ ) to optimize for γ via the maximum likelihood.
Fig.
4a presents the concentrations c b and c s withtheir estimates and uncertainties due to the unknown γ .
We show the equally spaced (black) andoptimized designs (yellow).
The more accurately we can infer the trajectory, the more accurately we can estimate γ , which we quantitatively show in Appendix E in Fig.
4b, where we decrease the
MSE of estimating γ by a factor of 10 in comparison to equally spaced design.
7.3 Sequential Design: Certifying Lyapunov Stability
Consider a non-linear system with x ∈ R d such that ddt x ( t ) =
f ( x ( t ) , t )+ u ( x ( t ) , t ) =
A Φ( x ( t ) , t )+u ( x ( t ) , t ) , where the rows of A i ∈ H κ for i ∈
[ d ] model the system dynamics and Φ( x ( t ) , t ) areknown evaluation functionals of H κ .
We assume that the control laws u ( x ( t ) , t ) can be writtenas u ( x ( t ) , t ) =
B Φ( x ( t ) , t ) .
We want to understand whether a given a control law B stabilizesthe above system.
A common approach is to create an estimate ˆ A of A from data samples oftrajectories, and use the controller B
=
ˆ A − P , where P is often a simple reverting law witha known gain, see below for an example.
With this choice of u , the system can be written asddt x ( t ) =
( A − ˆ A − P )Φ( x ( t ) , t ) =
( ˜ A − P )
Φ( x ( t ) , t ) where the ˜ A is the residual.
We can certifythe stability of the resulting system using a known quadratic Lyapunov function, e.g., V ( x, t ) =(
x ( t ) − x ref ( t )) ⊤ Σ ( x ( t ) − x ref ( t )) .
Classical stability theory dictates that if the total derivative of V ,
[Equation]
(11) is negative for all x ( t ) ∈
O then we can guarantee stability in the operating region of x ∈ O (Khalil,2002).
The above condition defines a linear operator
C x = ( x − x ref ) ⊤ Σ · ϕ ( x ) operating on theunknown ˜ A for each x ∈ O .
The linearity is best seen with vectorization as θ = vec( A ) , usingthe shorthand z ( t ) =
x ( t ) − x ref ( t ) , z ⊤ ΣA ϕ ( x ) = vec( Σ zϕ ( x ) ⊤ ) ⊤ vec( A ) =
C x θ.
The operatoris parametrized by x , C x for each x ∈ O .
Even for continuous domains x ∈ O , C x usually haslow-rank structure, which can be calculated and depends on the size O and the size H κ .
If the rank ofthe operator is small (e.g., the operating space is small) then we can certify negativity of Eq.
(11)faster than learning the whole A .
In other words, we reduce uncertainty only where we need to as inthe seminal work of Berkenkamp et al.
We can sequentially query data points from x andcheck whether C x θ ≤ 0 for all x .
Consider a two dimensional nonlinear system from Lederer et al.
[Equation]
The reverting controller is P Φ( x ( t ) , t ) =
− K ( x − x ref ( t ))
+ ( d/dt ) x ref , where the referencetrajectory corresponds to a circle x ref ( t ) =
(sin( t ) , cos( t )) .
The Lyapunov function is V =( x
( t ) − x ref ( t )) ⊤ ( x ( t ) − x ref ( t )) .
We assume that we can set the system to an initial condition x (0) andobserve a noisy observations of the state y
(0 + ∆) at rapid sampling times ∆ .
From these, we createa derivative oracle, ( d/dt ) x
( t ) ≈
( x ( t +∆) − x ( t )) / ∆ , a common approach in nonlinear data-drivencontrol (Umlauft et al., 2018).
We use this example to showcase our adaptive confidence sets.
We follow an adaptive stopping rule, where we query new data points if negativity cannot be certified.
Dueto this adaptive stopping rule, the data is adaptively collected.
We compare our confidence sets (solid)with the classical confidence sets of Abbasi-Yadkori et al.
(2011) (dashed) and report upper bounds onthe supremum over x ∈ O of Eq.
(11) in Fig.
2c.
The operating region is a “tube” around the circularreference trajectory x ref ( t ) .
We see that our confidence sets shrink much faster than the classicalconfidence sets, since we can eliminate redundant information by projecting onto C .
Fig.
2c comparesrandom sampling of datapoints from the whole domain (random) and the operating region (randomref) and sampling according to uncertainty in the dynamics in the whole domain (unc) and within thetube around the reference trajectory (unc-ref).
As expected, focused exploration methods work muchbetter.
However, and more importantly, the tightness of our confidence sets enable much quicker stability certification (termination).
Note that the classical confidence sets may even grow in the cases whereredundant information for estimation of C is inserted into the estimation, e.g., with random sampling.
CONCLUSION
We considered the problem of learning a linear function of an element in reproducing kernel Hilbertspaces.
We addressed the challenging case where linear estimators incur a non-negligible bias andprovided confidence sets for the two most commonly used linear estimators.
We demonstratedthe generality of our approach and the tightness of our confidence sets on several challengingapplications.
We believe our results lay important foundations for principled and efficient experimentdesign in complex real-world settings.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] No. Theorypaper.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
It’s insignificant.
I just used a
personal machine.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]