Experimental Design for Linear Functionals in Reproducing Kernel Hilbert Spaces
Abstract
Optimal experimental design seeks to determine the most informative allocation of experiments to infer an unknown statistical quantity.
In this work, we investigate the optimal design of experiments for estimation of linear functionals in reproducing kernel Hilbert spaces (RKHSs) .
This problem has been extensively studied in the linear regression setting under an estimability condition, which allows estimating parameters without bias.
We generalize this framework to RKHSs, and allow for the linear functional to be only approximately inferred, i.e., with a ﬁxed bias.
This scenario captures many important modern applications, such as estimation of gradient maps, integrals, and solutions to differential equations.
We provide algorithms for constructing bias-aware designs for linear functionals.
We derive non-asymptotic conﬁdence sets for ﬁxed and adaptive designs under sub-Gaussian noise, enabling us to certify estimation with bounded error with high probability.
Introduction
Optimal Experimental Design (OED) aims to determine data collection schemes – designs – to efﬁciently estimate unknown quantities of interest given limited resources (Chaloner and Verdinelli, 1995).
As common, we model experiments via an oracle that yields a (noisy) response to a given input.
A design is usually either an (adaptive) policy for querying the oracle or a (nonadaptive) ﬁxed allocation of query budget to different oracle inputs.
OED has a rich history and close relations to the ﬁeld of bandits (Szepesvari and Lattimore, 2020) and active learning (Settles, 2009).
We consider the regression setting, where observations at a ﬁxed input x ∈ X can be obtained via the noisy oracle by:
[Equation]
x ∈ X (1) and · (cid:62) · denotes the inner product in a Hilbert space H κ , (cid:15) is independent, sub-Gaussian noise with known variance proxy σ 2 , and θ is a bounded element from a separable reproducing kernel Hilbert space H κ from kernel κ , with a bound θ (cid:62) V 0 θ ≤ λ − 1 , where V 0 :
H k
→ H k is a positive deﬁnite operator.
Depending on whether λ is known or unknown, we will propose different estimators.
The norm constraint (under V 0 ) can be interpreted as a bound on the total effect and its parameters if the Hilbert space is ﬁnite dimensional.
For inﬁnite dimensional spaces this assumption can model the additional constraint on regularity.
For example, for kernels that induce Sobolev spaces, it explicitly bounds the total squared derivatives of functions – quantities known to govern complexity of approximation difﬁculty.
The exact bound depends on the speciﬁc deﬁnition of κ , but if two spaces are isometrically isomorphic, the difference can be absorbed to V 0 to induce the desired prior regularity, e.g., the regularity of the derivatives.
Here we assume that kernel κ and the operator Vare known due to prior analysis and/or ﬁrst principles modeling of the system.
In contrast to the classical ED task of estimating θ (see Fedorov and Hackl, 1997), we are interested in estimating a projection of θ .
Namely, let C : H κ → R p be a known linear operator .
The map C is such that C (cid:62) C is full rank p , where C (cid:62) denotes the adjoint.
Our goal is to identify an estimate of
C θ efﬁciently, i.e., with low query complexity or with a maximal reduction of uncertainty given a ﬁxed query budget T .
The formalism above captures, or occurs as a subroutine in, numerous practical problems.
For example, evaluation at speciﬁc target points, integration and differentiation are all linear operators, among many other useful linear functionals.
Other examples include ordinary or partial differential equations operators, spectral transforms, and stability metrics from control engineering.
In Sectionwe detail several example applications.
A naive approach would be to ﬁrst obtain an estimate ˆ θ of θ , and compute C ˆ θ .
However, the appeal of estimating linear functionals directly is that the number of unknowns of interest may be much lower than for the original overall unknown element θ (which might even be inﬁnite-dimensional).
Consequently, we would hope that the query complexity of reducing the variance of the estimate
C θ scales in the dimension of the range of C , which is p .
For example, when focusing on ﬁnitedimensional RKHSs, the operator C : R m → R p , where m is the dimension of H κ , becomes a matrix.
In this work, we study the cases where p < m , and show that the estimation error can indeed scale with p , and the geometry of the query set { Φ( x ) | x ∈ X} .
The estimation bias plays a central role in this work as for very large m estimating C θ might not be possible up to any precision.
The difﬁculty depends among other things the richness of the query set X , as we will detail later in Sec. 3.
Sequential Experiment Design Apart from classical experiment design, where we ﬁrst commit to what set of queries x we choose, referred to as a ﬁxed design , we also consider sequential design, where the selected queries depend on past observations.
Suppose for example, we are gathering data to test whether the null H 0 : θ (cid:62) Φ( x ) ≥ 0 for all x ∈ X or otherwise.
We can incrementally gather evidence and check whether the null hypothesis has already been rejected.
As our data depends on prior evaluation points it forms an adaptive design .
In this work, we develop conﬁdence sets for both ﬁxed and adaptive designs – which are of paramount importance in the context of sequential experiment designs, e.g., to deﬁne stopping rules of adaptive hypothesis testing problems.
Contributions A)
We consider objectives for experiment design for linear functionals in general
RKHS spaces, which carefully take the bias of the estimator into account.
B)
We provide bounds on the query complexity required to reach (cid:15) accuracy to estimate linear functionals with high probability.
C)
We construct novel non-asymptotic conﬁdence sets for linear estimators of linear functionals of
RKHS elements, both for ﬁxed and adaptive designs, where queries are independent of previous noise realizations, and where they are not, respectively.
D)
We demonstrate the improved inference error due to specially deﬁned designs and new conﬁdence sets on the problems of learning differential equations, linear bandits, gradient maps estimation, and stability veriﬁcation of non-linear systems.
Background and Related Work
Linear Estimators Let S ⊂
X be a ﬁnite set of selected evaluations s.t. |S| = n , potentially repeated.
We focus on linear estimators of the form L : R n → R p , where L y is estimating C θ .
An estimator is understood here as the algorithm to ﬁnd the random estimate L y .
Notice that given S , L is not a random quantity; the randomness rather comes from the realizations y .
To choose the estimator
L , one classically looks at the second moment of the residuals C θ − L y , E ( L ) =
E [( C θ − L y )
( C θ −
L y ) (cid:62) ] , where the argument signiﬁes how the random variable y is transformed (noise is averaged)
[Equation]
where E ( L
[Equation]
and the matrix X contains stacked evaluation functionals of the RKHS X i,x
= Φ i ( x ) for x ∈ S ⊆ X .
We will then seek a way to transform y via estimator L such that the second moment of residuals is minimized in certain sense.
Importance of Bias and RKHS Classical experimental design studies estimation of C : R m → R p where the RKHS is ﬁnite dimensional (Pukelsheim, 2006) and X ∈ R n × m .
On top of that, they consider only estimators which are unbiased for any θ , in other words, LX = C .
While, with ﬁnite dimensions, this simpliﬁcation might be reasonable, for inﬁnite dimensionial RKHSs, bias is inevitable and must be controlled.
Consider the case of estimating the gradient of a continuous function f .
We can nearly never learn it up to arbitrary precision from noisy point queries.
However, estimation up to a small error is always possible and sufﬁcient.
Estimating ∇ x f ( x ) from points close to f ( x ) will incur small bias but larger variance as the change in f compared noise (cid:15) is small.
Hence balancing these two sources of error is crucial for an informative design in RKHSs.
Experiment Design: Classical Perspective Consider an m dimensional version of the model in (1).
Then, among all unbiased estimators, linear least-squares estimators minimize (2) under the
Löwner order due to the famed Gauss-Markov theorem.
Unbiasedness in this form is synonymous with estimability , which means that by repeating the evaluation in X , arbitrary precision can be reached (Pukelsheim, 2006).
The second moment of residuals then becomes W − 1 † = CV † C (cid:62) , where · † denotes a generalized pseudo-inverse and V
= X (cid:62) X .
The matrix W is often referred to as the information matrix .
Gaffke (1987) and Krafft (1983) note that estimability implies that W † is non-singular.
We relax this condition and allow the estimator to have a bias ; this means that we cannot in general reduce the error arbitrarily by repeating the measurements.
In fact, our extension uses the fact that W † as deﬁned above will not be singular even if estimability condition is not satisﬁed.
We will show that the matrix W † can still play the role of the information matrix.
Experiment Design: Modern Challenges Mutný et al.
(2020) use experimental design to estimate the Hessian of an unknown RKHS function, while Kirschner et al.
(2019) use it to estimate the gradient of it for use in Bayesian optimization.
Perhaps most related, Shoham and Avron (2020) study overparametrized experimental design for one-shot active deep learning and analyze the bias in connection to experiment design, similarly as in the seminal work of Bardow (2008).
They do not consider bias arising from the limited design space nor do they treat linear functionals.
More broadly, the uncertainty propagation is studied in the Bayesian framework in the ﬁeld of probabilistic numerics for linear and nonlinear operators (see Cockayne et al., 2017; Owhadi and Scovel, 2016, and citations therein).
Conﬁdence sets Unlike classical statistics, our focus is on non-asymptotic conﬁdence sets on regression estimators.
They can be found in, e.g., Draper and Smith (2014) and Abbasi-Yadkori et al.
(2011), for ﬁxed and adaptive designs, respectively.
Our goal is to deﬁne the conﬁdence sets in the appropriate norm such that their width scales with the dimension of the range of C , i.e., p , and the geometry of { Φ( x ) | x ∈ X} , but not directly dim( H κ ) nor number of data T .
Mutný et al.
(2020) derive non-asymptotic conﬁdence sets that scale in p , but grow with the number of points T as they do not use the appropriate norm (see Appendix B.4).
Similarly, Khamaru et al.
(2021) study estimators for adaptively collected data, and propose asymptotic conﬁdence sets for their estimators.
Without a speciﬁc condition, their conﬁdence sets can grow with T , however, they consider more general noise distributions apart from sub-Gaussian as we do here.
Estimation and Bias
In this section, we motivate the linear estimators and identify information matrices .
Information matrices are the inverses of second moment matrices of residuals E as in Eq.
They are an important object in the analysis of the error of the estimators and their conﬁdence sets.
Also, they depend on the evaluations that deﬁne the estimator L , X .
Maximizing the information matrices as a function of the chosen observations and their proportions X gives rise to optimal experimental designs.
Further, we identify quantities that inﬂuence the bias of the estimators.
We study two estimators: the least-norm estimator (interpolation), when the bound on (cid:107) θ (cid:107) V 0 is unknown but ﬁnite, or the ridge regularized least squares estimator, where (cid:107) θ (cid:107) V 0 ≤ λ − 1 is known.
Both estimators are motivated as minimizing the error residuals E in trace norm under these two bound assumptions.
3.1 Estimators
Interpolation As apparent from Eq.
(2), without the knowledge of an explicit bound on the norm θ , the worst case over θ causes the optimal estimator L to minimize only the trace of the bias.
This leads to minmization of ( C − LX )
V − 1 / 2in Frobenius norm, leading to the familiar interpolation estimator ,
[Equation]
where K =
X V − 1 0 X (cid:62) (3)
The second moment of residuals can then be expressed as
[Equation]
, where P X
= V − 1 /
( V − 1 /
X (cid:62)
[Equation]
− I )
V − 1 / 2is a scaled projection matrix.
Unlike as in the classical ED treatment, the second moment has two terms: bias and variance.
If the span of the scaled projection operator lies in the null space of C , the bias (classically) vanishes.
To control the error of estimation, we need to control both terms.
For the special case of interpolation estimator, we will control the second term separately using a bias condition , and the variance term will be controlled by the information matrix W † as we will see in Section
[Equation]
Regularized Regression
The ridge regularized estimator is motivated by using θ (cid:62) V 0 θ ≤ λ − 1 , and minimizing then the trace of this upper bound, leading to an estimator,
[Equation]
Like above, we can give an upper bound on the second moment of the residuals.
Conveniently, this estimator automatically balances the error due to variance and bias in one term:
E ( L λ ) (cid:22) C θθ (cid:62)
[Equation]
(cid:62) −
[Equation]
+ K ) − 1 X V
[Equation]
C
[Equation]
Using the matrix inversion lemma, we can express the above bound in a more concise form, and subsequently its inverse motivates the deﬁnition of the information matrix for the regularized estimator,
[Equation]
3.2 Design objectives: Scalarization
The information matrices such as in (4) and (6) represent the inverse of estimation error, and the goal of optimal design is to maximize them (hence minimizing the error) with a proper choice of X .
As the Löwner order is not a total order, we need to resort to some scalarization of the information matrices, thus we solve max X f ( W ( X )) , where f is the scalarization R p
→ R .
We focus on two common forms of scalarization and refer to them as E - and A -design (Pukelsheim, 2006).
• f E ( W ) =
λ min ( W ) – when constructing non-asymptotic high probability conﬁdence sets; • f A ( W ) = 1 /
Tr( W − 1 ) – when minimizing mean squared error of estimation.
Other popular criteria include D , V , and G -designs (Chaloner and Verdinelli, 1995), which we do not consider due to space considerations, but can be equivalently used should the experimenter have a reason for it.
If p = 1 , C is an element in H k and the design problem degenerates into a special case known as c-optimality due to Elfving (1952) and, as W ∈ R , no scalarizations are needed.
Robust Designs The linear functionals are sometimes unknown, or parametrized by an unknown parameter γ as C γ , where γ belongs to a known set Γ .
If we were to construct a design that has low estimation error for the worst case selection of γ , we can maximize the information in the following worst
[Equation]
, for the interpolation estimator (and analogously for ridge regression).
If the original function f ( W ) is concave, then so is the function deﬁned as the inﬁmum over the compact index set Γ , which is true for the design criteria considered in this work (Boyd and Vandenberghe, 2004).
Fixed Designs and their Conﬁdence Sets
In RKHS spaces, especially inﬁnite dimensional ones, the estimability without bias is too restrictive.
Given a ﬁnite evaluation budget T , we can only construct a discrete design X : H κ → R T , and there are many practical examples, where, given a ﬁnite query budget, C θ cannot be learned to arbitrary precision, most prominently gradients and integrals among many others.
Estimation with Bias and Interpolation Esti-
mator
Our goal is to establish a condition on the design space X such that the estimation with the interpolation estimator is possible up to a certain bias (cid:15) measured under the Euclidean norm.
We measure the bias scaled by the magnitude of the Frobenius norm of the estimator and call this the relative ν -bias.
This condition will allow us to balance the error due to the bias and noise.
Deﬁnition 1 (Relative ν -bias) .
Let C : H κ → R k .
The estimator L † on the design space X is said to have relative ν -bias if
[Equation]
The (cid:107)·(cid:107) F corresponds to the Frobenius norm of the maps H κ
→ H κ .
Due to the cyclic property of the trace, we can take the adjoint of the operator and calculate the quantity by taking a trace of p × p matrix instead.
If ν = 0 and dim( H κ ) = m , as we show in Lemma 1 in the Appendix, this is equivalent to the classical estimability condition due to Pukelsheim (2006).
The left hand side corresponds to the classical bias bias( L † y )
= (cid:107) E
[ L † y ] − C θ (cid:107) 2 of an estimator L † under the Frobenius norm.
This is exactly the quantity that L † minimizes.
The classical bias cannot be improved by repeated measurements or allocations thereof.
Using a relation which we show formally in Proposition 5 in
Appendix B, we can bound the bias( L † y )
= (cid:107)
[Equation]
To check whether the condition (7) is satisﬁed given a design space X , one needs to evaluate the trace.
Finding the value of p × p matrix, however, depends strongly on the form of the operator C , and a general recipe cannot be provided.
Due to Riesz’s representer theorem, and fact that the range of C is ﬁnite-dimensional, this is always possible.
For example, for an integral operator (cid:82) Φ( x ) (cid:62) · q ( x ) dx , using shorthand v = K − 1 y , it holds that L † y
=
(cid:80) ni
=1 (cid:82) κ ( x, x i ) v
i dx where we used n points.
Notice that we used only evaluations of kernel κ to do this calculation.
Also notice that calculation of (7) reduces to the calculation of the maximum mean discrepancy (Gretton et al., 2005) between the empirical distribution (deﬁned on X ), and q ( x ) for this example.
The general recipe is to (locally) project the Hilbert space on a truncated ﬁnite-dimensional basis
Φ m ( x ) with size m and, in that case, the check involves a solution to a linear system, but this is not always necessary.
4.1 Conﬁdence sets
To construct conﬁdence sets for both estimators considered, we split them into two categories: ﬁxed designs , where the evaluation queries do not depend on the observed values y , and adaptive designs , where the queries Φ( x i ) may depend on prior evaluations y i − 1 , . . .
y 1 .
Proofs are in Appendix B.
Theorem 1 (Interpolation – Fixed Design) .
Under the regression model in Eq.
(1) with T data point evaluations, let ˆ θ be the estimate as in (3) .
Let X satisfy the bias from Def. 1 with W † =
( C V − 1 /
[Equation]
Then,
[Equation]
where V † =
V − 1 /
[Equation]
and ξ
[Equation]
δ )
= p +
[Equation]
log
[Equation]
Notice that in order to balance the source of the error due to bias and variance with high probability, we need to match σ (cid:112) ξ ( δ )
≈ ν √ λ .
Repeating the queries in X T times reduces σ by 1 / √ T but leaves the bias ν , as well as W † , unchanged.
This is because with the interpolation estimator the noisy repeated queries are averaged, which can be interpreted as a reduction in variance.
Hence, by balancing ν/ √ λ with σ (cid:112) ξ ( δ ) / √ T , we balance the bias and variance such that they are of the same magnitude.
It does not make sense to repeat measurements more times if the bias dominates the error of estimation.
A detailed example of estimating the gradient with ﬁxed bias is given in Sec. 7.2.
We derive conﬁdence sets for the regularized estimator of Eq.
(5), albeit without the relative bias.
Proposition 2 (Regularized estimate – Fixed Design) .
Under the model in Eq.
(1) with T data point evaluations, let ˆ θ λ be the regularized estimate as in (5) .
Then
[Equation]
δ, where W λ in Eq.
(6) , and ξ ( δ ) =
p + 2 (cid:113) p log( 1 δ ) .
Notice that, since the regularized estimator is designed to balance the bias and variance automatically, we do not need to speciﬁcally control the bias, which is contained within the information matrix W λ .
Despite this elegant property, the regularized estimator involves a more challenging analysis.
The main motivation to study the interpolation estimator is to understand the l 2 error as we show in Section 6.
Adaptive Design and Conﬁdence Sets
To provide conﬁdence sets for adaptively collected data, we need to project the data in X onto C , where we will denote the projection by Z further on.
With the data projected, we can reason about the reduction of the uncertainty of C θ for each point separately, since to each Φ( x i ) we can associate a unique z i in R p .
Deﬁnition 2 (Projected data) .
Let z ( x ) ∈ R p , projected data , be a vector ﬁeld s.t.
Φ( x ) V − 1 / 2=
z ( x ) C V − 1 / 2+ j ( x ) , where x ∈ S ⊆
X and j ( x ) ∈ H κ , |S| = n ,
such that C V − 1 / 2j ( x ) = 0 .
Classically, the adaptive conﬁdence sets are understood only for C
=
I , i.e., the identity (AbbasiYadkori et al., 2011).
In fact, we can always derive conﬁdence sets for C θ from conﬁdence sets for θ , as they only project the ellipsoid to a smaller dimensional space.
However, their resulting size may be unnecessarily large, as the conﬁdence parameter scales as O (dim( H k )) in general (see Figurefor a visual example).
The martingale analysis of Abbasi-Yadkori et al.
(2011) and de la Peña et al.
(2009) speciﬁcally assumes that information matrix V t , (where C = I ) can be additively decomposed to information matrices due to a single evaluation V t = (cid:80) ti =
Φ( x i )Φ( x i ) (cid:62) each at a different time.
With the matrix
W λ,t , this additive decomposition is not always possible.
Therefore, to utilize the martingale analysis, which requires this additive property, we consider a different information matrix, which upper bounds
W λ .
The information matrix we use, Ω λ , is constructed from the projections z ( x i ) , which have the necessary additive property.
It gives rise to conﬁdence sets, where, under the ellipsoidal norm, their size scales as Θ( p ) .
The estimation error depends on Ω λ still, but under this norm, the conﬁdence parameter and information matrix are decoupled in a similar way as for the ﬁxed design.
Theorem 3 (Ridge estimate – Adaptive Design) .
Under the regression model in Eq.
(1) with t adaptively collected data points, let ˆ θ λ,t be the regularized estimate as in (5) .
Further, assume that Z is as in Def. 2 where X t =
Z t C + J t V 1 / 2 0 .
Then for all t ≥ 0 ,
[Equation]
with probability 1 − δ , where Ω λ,t = 1 σ 2 Z (cid:62) t
Z t + λ S and S =
( C V − 1 0 C (cid:62) ) − 1 .
The matrix Z t can be calculated by solving a least-squares problem (projection), whereupon
C V − 1 /
J t = 0 as needed by Deﬁnition 2.
Notice that, on one hand, the above conﬁdence parameter grows only when Z is large, but at the same time, the ellipsoid shrinks only in that case as well.
Also, the ellipsoid above is necessarily smaller than the one with the information matrix
W λ as W λ (cid:22) Ω λ (Lemma 3 in Appendix).
This means we can use the same conﬁdence parameter to give a bound for (cid:107)·(cid:107) W λ,t .
Notice that the estimator is the same as before, i.e., ˆ θ λ , using X to deﬁne the regression, only the information matrix changes.
We also present a visual comparison in
Figure 1 on a simple example, showing that our ﬁxed and adaptive sets are tighter than projected non-asymptotic sets.
Using the shorthand β t ( δ ) to deﬁne the conﬁdence parameter in (9), we can in fact bound the error as (cid:107) C ( θ λ,t − θ ) (cid:107) 2 ≤ λ min ( Ω λ,t ) − 1 / 2 β t ( δ ) .
It depends on m = dim( H κ ) only via Ω λ,t .
The dependence of the conﬁdence parameter can be at most Θ( (cid:112) log(1 + T ) p ) , see Lemma 4 in Appendix B.
The value of λ min ( Ω λ,t ) − 1 depends on the geometry of the set { Φ( x ) | x ∈ X} and the projection operator C as we show in Section 6.3.
Note that the lower bound of (Szepesvari and Lattimore, 2020, Ex. 20.2.3) does not apply here, since it makes a statement only about the information matrix W λ .
Convex Relaxations, Geometry and Dimensions
Suppose we are given a candidate set of experiments, i.e., a unique subset of evaluations S ⊂ X , |S| = n and a budget T of total queries.
We seek an allocation X , where the rows of X contain potentially repeated evaluations from S .
How many times should we repeat each experiment in order to ﬁnd max X f ( W ( X )) ?
To address this, experimental design literature relaxes this discrete optimization problem, and optimizes over fractional allocation η ∈ ∆ n , where the number of repetitions for Φ( x i ) is recovered by rounding (cid:100)
η i T (cid:101) .
With this interpretation, the objectives in Sec.
3.2 can be written as
[Equation]
(10) where D ( η ) is the diagonalization operator that produces a diagonal matrix with vector η on the diagonal and X S contains non-repeated elements in S .
We have stated the problem above for the interpolation estimator and dim( H k ) < ∞ , but it naturally generalizes to kernelized estimators, albeit in a less concise form (see Appendix C).
6.1 Optimizing allocations: experiment design algorithms
Given a subset S , the problem (10) can be approximately solved using either convex optimization methods or a greedy algorithm.
A comprehensive review of methods constructing designs η ∗ and rounding techniques to get X is beyond the scope of this work , and not the core issue addressed in this work.
We brieﬂy review two versatile approaches for completeness (more details in Appendix C).
Greedy selection Firstly, one can greedily maximize the scalarized information matrix with the update rule η t
+1 =
t t +1 η t + 11+ t δ t , δ t = arg max x ∈X f (cid:16) W λ (cid:16) t t +1 η t + 1 t +1 δ x (cid:17)(cid:17) , where f refers to the scalarization and δ x to the discrete measure corresponding to the feature map Φ( x ) .
Due to the form of the update rule, tη t is always an integer.
6.2 Convex optimization
Alternatively, convex optimization can provably solve the problem to optimality.
Speciﬁcally, we look for an allocation using convex optimization methods max η ∈ ∆ n f ( W ◦ ( D ( η ) 1 / 2 X )) where ◦ ∈ {† , λ } .
Care needs to be taken when selecting S , as we discuss in Appendix C. The most common algorithms for ODE problems are the Frank-Wolfe algorithm (Todd, 2016) and mirror descent algorithm (Silvey et al., 1978).
Additionally, semi-deﬁnite reformulation of the above optimization problems can be given as we shown in Appendix C.3.
Optimal designs found via convex optimization need to be rounded in practice.
State-of-art rounding techniques are discussed by Allen-Zhu et al.
(2017) and Camilleri et al.
(2021) for ﬁnite and inﬁnite dimensional spaces, respectively.
The exhaustive cover of rounding techniques is out-of-scope of the current work.
6.3 Estimation error and its dimension dependence
If we were to bound the squared error of estimation in high probability, the importance of the scalarization λ min ( W † ) becomes apparent.
Using the Cauchy Schwarz inequality, we get (cid:13)(cid:13)(cid:13)
[Table]
[Equation]
[Table]
[Equation]
[Table]
where the term due to Proposition 1 scales as O ( p ) when properly balanced.
Using the optimal allocation η ∗ , the number of repetitive evaluations is equal to η ∗ T .
Inverting the relation above yields a query complexity of the order T ≈ O ( p(cid:15) 2 λ min ( W † ( η ∗ ) − 1 )) .
The optimal value λ min ( W † ( η ∗ )) represents a problem dependent quantity that measures the difﬁculty of estimation and captures the geometry of the set { Φ( x ) | x ∈ S ⊂ X} .
It cannot be bounded in general, but it has an elegant geometric interpretation.
In particular, it corresponds to the square inverse of the diameter of the largest inscribed ball in the convex hull of symmetrized { Φ( x ) | x ∈ S} in the range of C (Pukelsheim and Studden, 1993).
At ﬁrst glance, calculating this quantity might seem complicated, but with an example it is apparent.
For example, if Φ( x ) = x s.t. (cid:107) x (cid:107) 22 ≤ 1 (unit l 2 ball in R m ) and C = v , where v is a unit vector ( p = 1 ), the inverse diameter of largest ball we can inscribe in direction of v inside (cid:107) x (cid:107) 22 ≤ 1 equates to 1 , which is independent of d .
This is not surprising, since it represents an “easy” design space, where for any direction v one can ﬁnd an action x aligned with that coincides with the optimal design.
This is true even if C has more rows.
On the other hand, if we assume evaluation in the l 1 ball (cid:107) x
1 ≤ 1 and C is a vector of ones (again p = 1 ).
Then the diameter of the inscribed ball is proportional to the inverse square height of a simplex, namely, 1 m , despite p = 1 .
Hence, despite the conﬁdence parameter being O (1) , the complexity depends primarily on the geometry of this set.
To give a more exotic example, if C = ∇ x Φ( x ) , and the design space are points with ﬁxed length steps in all unit derection { x | Φ( x ± e i h ) } , where h is the stepsize and e i are principal vectors in R d , we can show that λ min ( W † ) − 1 ≤ dh + O ( h 2 ) , leaving the overall complexity to learn a gradient to scale with d instead of the dimensionality of the RKHS which can be inﬁnite.
All formal proofs and references are provided in Appendix D.1.
Applications
We now discuss concrete applications that beneﬁt from our contributions.
Details of the experiments, and further applications, e.g., in statistical contamination, can be found in Appendices E and F.
We provide information about optimization and rationale in choosing the κ , and V 0 in C and F respectively.
[Figure]
7.1 Linear bandits with ﬁnite number of arms
A special but important theoretical consequence of our adaptive conﬁdence set is an improved regret bound for the upper conﬁdence bound (UCB) algorithm (Auer, 2002) for linear bandits when the set of queries X is ﬁnite.
The queries are known as actions in the bandit literature (Szepesvari and Lattimore, 2020).
The UCB algorithm is a procedure which iteratively queries the action according to x t = arg max x ∈X max ϑ ∈ C t − 1 ϑ (cid:62) Φ( x ) , where C t is conﬁdence set for θ in round t .
Different to this, in our analysis of the UCB algorithm, we use the conﬁdence sets due to Theorem 3, which are conﬁdence sets for the projections Φ( x ) (cid:62) θ directly.
We construct |X| conﬁdence sets, one for each of the linear functionals C x = Φ( x ) , and denote them by cf t ( x ) .
The
UCB algorithm then just becomes x t = arg max x ∈X max y ∈ cf t ( x ) y .
We now analyze its cumulative regret R T :=
(cid:80) Tt =
(Φ( x ∗ ) − Φ( x t )) (cid:62) θ , where x ∗ = arg max x ∈X Φ( x ) (cid:62) θ , and θ is the true unknown pay-off vector of the bandit game.
Theorem 4.
Let θ ∈ R
d be an unknown pay-off vector, and a set of actions x ∈ X such that |X| < ∞ .
Then the cumulative regret of the UCB algorithm is bounded by
[Equation]
Thus, our adaptive conﬁdence sets for linear functionals yield a regret bound that is a factor of √ d tighter than the standard bound.
Note that this does not break the known lower bound O ( d √ T ) for
X being the unit ball in R d from Rusmevichientong and Tsitsiklis (2010), as we gain a logarithmic dependence on |X| .
For an (cid:15) covering of the extreme points of the unit ball in R d , one needs |X| ∝ (1 /(cid:15) )
d points, leading to a regret scaling as O ( d √ T log(1 /(cid:15) )) .
Choosing (cid:15) = 1 /T
would, for a ﬁxed T , recover the lower bound up to logarithmic terms.
Using standard doubling-trick techniques, one can provide anytime result.
However, for different query sets (such as the l 1 ball), whose extreme points (i.e., number of vertices) are ﬁnite (equal to d ), these results can vastly improve the regret bounds.
There are alternative algorithms based on arm-elimination, which achieve this regret bound, e.g., the algorithm of Szepesvari and Lattimore (2020), the SupLinRel algorithm of Auer (2002), and the algorithm of Valko et al.
The proof of Theorem 4 is deferred to Appendix G, but it is a trivial application of previous results.
The above technique can be extended to inﬁnite dimensional RKHS, where the role of dimension is played by, γ T called maximum information gain (Srinivas et al., 2010), demonstrated in Appendix
G. The remaining challenge is to bound the (cid:15) -covering of extreme points of the set { Φ( x ) | x ∈ [ − 1 , 1] d } ⊂ H k , where [ − 1 , 1] d is the continuous action set.
If all evaluation operators have (cid:107) Φ( x ) (cid:107) k = 1 , for all x ∈ [ − 1 , 1] , such as for Matérn kernels (Mutný and Krause, 2018), we can use covering numbers of unit balls in these spaces as the upper bound on this number.
These spaces are known to be isometrically isomorphic to Sobolev spaces (Wendland, 2004), and the bounds on covering numbers of unit balls are known to be of order ( 1 (cid:15) )
2 d/s , where s ∈ N , s > d/ 2 , refers to regularity of the Sobolev space (Cucker and Smale, 2002).
This yields a bound on the cumulative regret of the order O ( (cid:112) C ( d ) γ T T log T ) , where C ( d ) can depend on dimension.
The results in this paper, and the above technique provide an important tool for resolving the open problem about regret rates of kernelized bandit algorithms (Vakili et al., 2021), and suggest that, at least for Sobolev spaces, the bounds can be improved from the worst case O ( γ T √ T ) to one scaling with O ( (cid:112) C ( d ) γ T T ) .
7.2 Gradient maps
Gradients of any order are linear operators.
We can express the gradient at x as dimension-wise evaluation of the following operator ∇ x (Φ( x ) (cid:62) θ )
= ( ∇ x Φ( x )) (cid:62) θ
=: C θ.
Clearly, estimability is nearly always impossible, since any evaluation inﬁnitesimally away from x will be insufﬁcient to eliminate bias.
Thus our estimates will invariably be biased for any function with inﬁnite Taylor expansion.
Yet, while estimating the gradient from evaluations very close to the original x leads to low bias, it at the same time increases the variance, since the difference in the functional value between the two point evaluations is very small compared to the noise magnitude.
Given a ﬁnite budget or desired accuracy, the best we can do is to ﬁnd the best design with optimal bias-variance trade-off given the kernel κ , budget T , and noise variance σ .
We consider a class of parametrized ﬁnite difference designs { Φ( x ± he i ) } , where h is the stepsize and e i are unit vectors.
In Fig.
2b, we plot the total error with high probability with the budget T as a function of the step-size h .
Notice that the lowest error occurs exactly when the variance of observations scaled with the conﬁdence parameter is equal to the relative ν (two lines cross).
7.3 Learning linear ODE solutions and their parameters
A solution to a linear ordinary differential equation (ODE) u satisﬁes ddt u ( t ) =
M u ( t ) + s ( t ) , where
M is a linear operator and s ( t ) is the non-homogeneous term.
Assume that the solution to the equation u ( t ) =
Φ( t ) (cid:62) u is a member of a Hilbert space H κ , then T u :=
(cid:0) ddt − M ( t ) (cid:1) Φ( t ) (cid:62) u
= s ( t ) for t ∈ [ t 0 , t 1 ] .
Hence, the differential equation becomes a linear constraint for estimating u from samples.
In fact, due to differential equations being fully speciﬁed by initial conditions, the only unknowns are due to initial conditions.
To reveal the linear functional here, one needs to consider the solution to the differential equation, which can be written as u
= T † s ( t )+ C (cid:62) v , where v ∈ H κ \ span( T ) belongs to the null space of T .
In this case, we span it with rows of C .
Consequently, the unknown element v can be found as C ( u − T † s ) .
Thus, what needs to be estimated from samples, is the linear projection C u , since CT † s is known a priori.
In Appendix F, we discuss implementation and calculation of the operator C on a discretized domain.
Also note that, Z itself induces a Hilbert space H z containing the solutions u (González et al., 2014).
However, as we will see, for robust designs it is sometimes convenient to pick a larger Hilbert space H κ s.t.
H z ⊂ H κ
and then apply the differential equation constraint.
Robust Design As a speciﬁc example, consider an example of a pharmacokinetic model capturing the concentration of a medication in blood and stomach via differential equations: ( d/dt )
c s = − ac s and ( d/dt )
c b = bc s − dc b for t ∈ [ t 0 , t 1 ] , where c s and c b are the concentration in stomach and blood respectively.
The goal of this analysis is to infer γ = ( a, b, d ) from the measurements of the blood concentration levels with ﬁxed, but perhaps noisy, initial conditions (Gabrielsson and Weiner, 1995).
To apply the above procedure, we need a ﬁxed differential operator T γ to deﬁne the operator C γ .
However, γ is itself unknown.
Instead, we can give a generally plausible set of γ ∈ Γ , initial conditions of concentration in blood c b (0) = 0 , and norm constraints on the initial stomach concentration ( c s (0) − c dose ) 2 ≤ λ − 1 (prior V 0 ) in order to apply our framework.
We use the squared exponential kernel to embed trajectories and consider robust A -optimal design (as in Sec. 3.2) by maximizing the worst case metric inf γ ∈ Γ 1 / Tr( C γ V − 1 λ C (cid:62) γ ) with the regularized estimator.
This way, the design is appropriate for any γ .
After estimating the trajectory, we can use the estimated trajectories (given γ ) to optimize for γ via the maximum likelihood.
Fig.
4a presents the concentrations c b and c s with their estimates and uncertainties due to the unknown γ .
We show the equally spaced (black) and optimized designs (yellow).
The more accurately we can infer the trajectory, the more accurately we can estimate γ , which we quantitatively show in Appendix E in Fig.
4b, where we decrease the
MSE of estimating γ by a factor of 10 in comparison to equally spaced design.
7.4 Sequential Design: Certifying Lyapunov Stability
Consider a non-linear system with x ∈ R d such that ddt x ( t ) =
f ( x ( t ) , t )+ u ( x ( t ) , t ) =
A Φ( x ( t ) , t )+ u ( x ( t ) , t ) , where the rows of A i ∈ H κ for i ∈
[ d ] model the system dynamics and Φ( x ( t ) , t ) are known evaluation functionals of H κ .
We assume that the control laws u ( x ( t ) , t ) can be written as u ( x ( t ) , t ) =
B Φ( x ( t ) , t ) .
We want to understand whether a given a control law B stabilizes the above system.
A common approach is to create an estimate ˆ A of A from data samples of trajectories, and use the controller B
=
ˆ A − P , where P is often a simple reverting law with a known gain that compensates the imprecise estimation of A with ˆ A , see below for an example.
With this choice of u , the system can be written as ddt x ( t ) =
( A − ˆ A − P )Φ( x ( t ) , t ) =
( ˜ A − P )
Φ( x ( t ) , t ) , where the ˜ A
=
A − ˆ A is the residual error in estimating A .
We can certify the stability of the resulting system using a known quadratic Lyapunov function as commonly done in control theory, in this case, using
V ( x, t ) =
( x ( t ) − x ref ( t ))
[Equation]
Σ ( x ( t ) − x ref ( t )) .
Classical stability theory dictates that if the total time derivative of V ,
[Equation]
is negative for all x ( t ) ∈
O then we can guarantee stability in the operating region of x ∈ O (Khalil, 2002).
The above condition deﬁnes a linear operator
C x = ( x − x ref ) (cid:62) Σ · φ ( x ) operating on the unknown ˜ A for each x ∈ O .
The linearity is best seen with vectorization as θ = vec( A ) , using the shorthand z ( t ) =
x ( t ) − x ref ( t ) , z (cid:62) ΣA φ ( x ) = vec( Σ zφ ( x ) (cid:62) ) (cid:62) vec( A ) =
C x θ.
The operator is parametrized by x , C x for each x ∈ O .
Even for continuous domains x ∈ O , C x usually has low-rank structure, which can be calculated and depends on the size O and the size H κ .
If the rank of the operator is small (e.g., the operating space is small) then we can certify negativity of Eq.
(11) faster than learning the whole A .
In other words, we reduce uncertainty only where we need to as in the seminal work of Berkenkamp et al.
We can sequentially query data points from x and check whether C x θ ≤ 0 for all x .
Consider a two dimensional nonlinear system from Lederer et al.
, dx
[Equation]
The reverting controller is P Φ( x ( t ) , t ) =
− K ( x − x ref ( t ))
+ ( d/dt ) x ref , where the reference trajectory corresponds to a circle x ref ( t ) =
(sin( t ) , cos( t )) .
The Lyapunov function is
V =
( x ( t ) − x ref ( t )) (cid:62) ( x ( t ) − x ref ( t )) .
We assume that we can set the system to an initial condition x (0) and observe a noisy observations of the state y (0 + ∆) at rapid sampling times ∆ .
From these, we create a derivative oracle, ( d/dt ) x
( t ) ≈
( x ( t + ∆) − x ( t )) / ∆ , a common approach in nonlinear data-driven control (Umlauft et al., 2018).
We use this example to showcase our adaptive conﬁdence sets.
We follow an adaptive stopping rule, where we query new data points if negativity cannot be certiﬁed.
Due to this adaptive stopping rule, the data is adaptively collected.
We compare our conﬁdence sets (solid) with the classical conﬁdence sets of Abbasi-Yadkori et al.
(2011) (dashed) and report upper bounds on the supremum over x ∈ O of Eq.
(11) in Fig. 2c.
The operating region is a “tube” around the circular reference trajectory x ref ( t ) .
We see that our conﬁdence sets shrink much faster than the classical conﬁdence sets, since we can eliminate redundant information by projecting onto C .
Fig.
2c compares random sampling of datapoints from the whole domain (random) and the operating region (random-ref) and sampling according to uncertainty in the dynamics in the whole domain (unc) and within the tube around the reference trajectory (unc-ref).
As expected, focused exploration methods work much better.
However, and more importantly, the tightness of our conﬁdence sets enable much quicker stability certiﬁcation (termination).
Note that the classical conﬁdence sets may even grow in the cases where redundant information for estimation of C is inserted into the estimation, e.g., with random sampling.
CONCLUSION
We considered the problem of learning a linear function of an element in a reproducing kernel
Hilbert space.
We addressed the challenging case where linear estimators incur a non-negligible bias and provided conﬁdence sets for the two most commonly used linear estimators.
We demonstrated the generality of our approach and the tightness of our conﬁdence sets on several challenging applications.
We believe our results lay important foundations for principled and efﬁcient experiment design in complex real-world settings.
Acknowledgement
This project has received funding Swiss National Science Foundation through NFP75 and this publication was created as part of NCCR Catalysis (grant number 180544), a National Centre of
Competence in Research funded by the Swiss National Science Foundation.
A Estimability results
In the following section, we either describe proof of for implication or equivalences of certain conditions studied in this work.
In A.1, we show consequence of Def. 1 which is used in the proofs of conﬁdence sets.
In the subsection following it, we establish relationship to classical ODE as in (Pukelsheim, 2006) showing that our bias condition generalizes notion stemming from there.
A.1 Equivalence of bias conditions
Proposition 5.
Let L † be the interpolation estimator and W † it associated information matrix then, then
[Equation]
Proof.
Note that V † =
V − 1 /
X
[Equation] [Equation]
where K =
X V − 1 0 X (cid:62) .
(LHS
[Equation]
Now let ˜ X
=
X V − 1 / 2and ˜ C
= C V − 1 / 2.
Also notice that, ( ˜ X (cid:62) ˜ X ) † = V − 1 /
[Equation] [Equation]
V †
We can apply Theorem 19, to get
[Equation]
Using the above result, we show that
[Equation]
where we have used the fact that I − P is projection matrix with orthogonal span to
V − 1 /
X
[Equation]
Proposition 6.
Bias (cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) depends only on its support not the allocations.
Proof.
Let L η be the estimator with design η , where
[Equation]
But at the same time D ( η ) 1 / 2 X , hence these two cancel each other.
A.2 Basic equivalences and relations
To relate the estimability to terms used in the ED literature, we state the deﬁnition of the feasibility cone as in Pukelsheim (2006).
We show that our condition in Def. 1 and Pukelsheims and estimability are equivalent under classical assumptions.
We use a different formulation of the relative-bias condition due to Proposition 5.
Deﬁnition 3 ((Pukelsheim, 2006)) .
Let C : H κ → R k , denote A ( C ) =
{ A ∈ P ( H κ ) : range ( C (cid:62) )
⊆ range ( A ) } , where P ( H κ ) is the space of positive-deﬁnite operators on H κ .
We call A ( C ) the feasiblity cone of K .
This deﬁnition is sometimes used as restatement of the estimability property.
Lemma 1 (Equivalence in ν = 0 ) .
Let C ∈ R k × d full rank k .
Let X ∈ R n × d , then if M = X (cid:62) X , the following are equivalent,
Estimability: There exists L ∈ R k × n s.t. C = LX .
Feasibility cone:
M ∈ A ( C )
Proof.
• (2) =
⇒ (1) :
There exists B s.t.
C = BM , and C = BX (cid:62) X , hence we can deﬁne L = BX .
• (1) =
⇒ (2) :
As such C = LX implies that range ( C ) ⊆
range ( X (cid:62) )
=
range ( X (cid:62) X ) .
Deﬁnition 4 (Projected data) .
Let z ( x ) ∈
R p be a vector ﬁeld s.t.
Φ( x ) V − 1 / 2=
z ( x ) C V − 1 / 2+ j ( x ) , where x ∈ S ⊆
X and j ( x ) ∈ H κ , |S| = n ,
such that C V − 1 / 2j ( x ) = 0 .
We call this vector ﬁeld projected data .
If in addition if { z ( x ) : x ∈ S} spans R p it is said to be approximately low-rank .
Lemma 2.
The assumption in Deﬁnition 4 implies the assumption in Deﬁnition 1 with ν = (cid:107) J (cid:107)
k
(cid:107) Z † (cid:107) F (cid:107) L † (cid:107) F .
Proof.
Since Z spans whole R k , there exists a unique left pseudo-inverse.
Z † X V −
[Equation] [Equation]
Z † J .
[Equation]
Now taking the Frobenius norm,
[Equation]
So,
[Equation]
Now, since pseudo-inverse minimizes the LHS, we know that,
[Equation]
B Conﬁdence Sets: Proofs
This section includes proofs for the concentration results presented in the main text.
In Section, B. we restate and prove results from Mutný et al.
(2020) in the current notation for easier comparison.
B.1 Fixed Design with Interpolator
The following Theorem tries to qualify whether a design contains sufﬁcient information to reduce conﬁdence on a speciﬁc subspace C to the desired error.
Theorem 7 (Interpolation (Fixed Design)) .
Under regression model in Eq.
(1) with T data point evaluations, let ˆ θ be the estimate as in (3) .
Further let X be s.t.
satisfy the approximate estimability condition of Def. 1 with T ≥ k .
Then,
[Equation]
where W =
( C V − 1 /
V
[Equation]
V −
[Equation]
C (cid:62)
[Equation]
− 1 , V † =
V −
[Equation] [Equation]
, and
[Equation]
δ
[Equation]
= p + 2 (cid:113) p log( 1 δ ) .
Proof.
The pseudo inverse estimate of C θ is C ˆ θ =
C V − 1 0 X (cid:62) ( X V − 1 0 X (cid:62) ) − 1 y , where X ∈ R T × m .
[Equation]
where the second to last line we used the relative-bias assumption, according to the Proposition 5, where we use the Def. 1 and T ≥ k .
The operator I is identify operator on H k .
If (cid:15) G were Gaussially distributed then, C V − 1 0 X (cid:62) ( X V − 1 0 X (cid:62) ) − 1 (cid:15) G is distributed as N (0 , CV † C (cid:62) ) .
Likewise, q G = W 1 / 2 C V − 1 /
X (cid:62) ( X V − 1 0 X (cid:62) )
− 1 (cid:15) G is distributed as N (0 , I k ) .
Since (cid:15) is subGaussian q needs to have tails of distribution which are below the tails of k -dimensional gaussian.
In other words P ( | q i | ≤ t ) ≤ P ( | q G i | ≤ t ) .
As such, P ( q 2 i ≤ t 2 ) ≤ P ( | q G i | 2 ≤ t ) , where q 2 G i is chi-squared distributed.
Using concentration for chi-squared random variables, P (cid:0) (cid:107) z (cid:107) 2 2 ≥ σ 2 ( p + 2 √ px ) (cid:1) ≤ exp( − x ) as in Laurent and Massart (2000).
Rearranging the expression leads to the result with ξ ( δ ) =
( p + 2 (cid:113) p log( 1 δ )) .
The last ﬁnal inequality in the statement of the probability follows from taking a square root and triangle inequality, which ﬁnishes the proof.
B.2 Fixed Design with Regularized Estimator
Proposition 8 (Fixed Design Ridge Regression) .
Under regression model in Eq.
(1) with T data point evaluations, let ˆ θ λ be the regularized estimate as in (5) .
Then,
[Equation]
where W λ = σ − 2 ( C V − 1 λ C (cid:62) ) − 1 , V − 1 λ =
( X (cid:62) X + λσ 2 V 0 ) − 1 .
In addition, ξ ( δ ) = p +2 (cid:113) p log( 1 δ ) .
Proof.
Notice that invertibility of W is guaranteed by full rank C and invertibility of V λ .
Using shorthand V =
X
[Equation]
X
[Equation]
The second term,
[Equation]
Let us analyze the ﬁrst term,
[Equation]
The distribution of C V − 1 λ X (cid:62) (cid:15) is N (0 , σ 2 C V − 1 λ V V − 1 λ C (cid:62) ) .
Further, ( σ C V − 1 λ C (cid:62) ) − 1 / 2 C V − 1 λ
X
(cid:62) (cid:15) is distributed as z ∼ N (0 , ( C V − 1 λ C (cid:62) )
− 1 / 2 C V
[Equation]
λ C
[Equation]
( C V
[Equation]
1 λ C
[Equation] [Equation]
Let us call the covariance matrix Σ .
It is easy to see that Σ (cid:22) P , an projection matrix with k unit eigenvalues.
The random variable z can be generated as z = q (cid:62) Σ q , where q ∼ N (0 , I n ) .
Our goal is to bound, P ( (cid:107) z (cid:107) 22 ≥ t ) .
Using eigenvalue decomposition of Σ = QΛQ (cid:62) , we can show
[Equation]
where ˜ q is p -dimensional standard normal.
In order to bound the resulting chi-squared, we use concentration for chi-squared random variables, P (cid:0) (cid:107) ˜ q (cid:107) 22 ≥
( p + 2 √ px ) (cid:1) ≤ exp( − x ) as in Laurent and Massart (2000).
Rearranging the expression leads to the result with ξ ( δ ) =
( p + 2 (cid:113) p log( 1 δ )) .
To deal with sub-Gaussianity we apply the same trick as in the previous Theorem.
B.3 Adaptive design and Regularized Estimator
Theorem 9 (Adaptive Design Regularized Regression) .
Under the regression model in Eq.
(1) with t adaptively collected data points, let ˆ θ λ,t be the regularized estimate as in (5) .
Further, assume that Z is as in Def. 2 where X t =
Z t C + J t V 1 / 2 0 .
Then for all t ≥ 0 ,
[Equation]
with probability 1 − δ , where Ω λ,t = 1 σ 2 Z (cid:62) t
Z t + λ S and S =
( C V − 1 0 C (cid:62) ) − 1 .
Proof.
Notice that the theorem is stated with S and S − 1 having inverted roles in the proof, however the result above follows by swapping the two.
Note that J V − 1 /
C (cid:62) = 0 as well as C V − 1 /
J (cid:62) =
due to the assumption.
Also, notice that
X = ZC + J V 1 / 2 0 .
We drop the t subscript for brevity.
[Equation]
Now we analyze the two terms separately, The ﬁrst term,
[Equation]
Let us deﬁne shorthand S − 1 =
C V − 1 0 C (cid:62) which is R k × k p.s.d.
matrix.
[Equation]
The term above is so called self-normalized noise, which can be handled by techniques of de la Peña et al.
(2009) popularized by Abbasi-Yadkori et al.
From now on the proof is generic.
Let us deﬁne, the noise process S t = (cid:80) ti =
z
i (cid:15) i σ 2 and variance V t =
z i z (cid:62) i σ 2 .
Also, let
M t ( x ) = exp( S (cid:62) t x − 12 x (cid:62) V t x ) be a process with index t .
Due to sub-gaussianity of (cid:15) i , M t ( x ) is a super-martingale under ﬁltration that includes all x t − 1 , (cid:15) t − 1 , . . .
x 1 , (cid:15) 1 .
Now it is worth noting that this all has been introduces since exp( (cid:13)(cid:13)
Z (cid:62) (cid:15) (cid:13)(cid:13) 2 Ω − 1 λ ) =
sup x M t ( x ) .
This relation allow us to see that if we can upper bound sup x M t ( x ) we get what we want.
We proceed by pseudo-maximization.
We pick a ﬁxed probability distribution h ( x ) and deﬁne a process ¯ M t = (cid:82) M t ( x ) h ( x ) dx .
Since, h ( x ) is ﬁxed and normalized ¯ M t is also a super-martingale.
It turn out that h ( x ) ∼ N (0 , λ − 1 S − 1 ) will achieve the desired result.
[Equation]
where we have applied standard rules for Gaussian integrals.
Now, we use Ville’s martingale inequality to bound,
[Equation]
which ﬁnishes bounding the ﬁrst term by 2 log
[Equation]
Now we turn to the second term,
[Equation]
This proves the result.
Lemma 3 (Ordering) .
[Equation]
Proof.
Consider series of psd.
manipulations,
[Equation]
where we use the fact that C V − 1 /
J (cid:62) = 0 .
Lemma 4 (conﬁdence parameter size) .
Under assumptions of Thm.
[Equation]
where (cid:107) Φ( x ) (cid:107) 2 k ≤ L 2 , V
= I , (cid:115) 2 log
[Equation]
δ det
[Equation]
det
[Equation]
Further, if (cid:107) Φ( x ) (cid:107) k
≈ ( √ m ) grows with m , e.g. Φ( x ) = 1 m (vector of ones) then
[Equation]
Proof.
First let us determine the absolute bound on all (cid:107)
z i (cid:107) 22 .
Due to projected data,
[Equation]
where the last step follows from properties of projection.
Notice also that
[Equation]
Using arithmetic-geometric mean inequality as in (Szepesvari and Lattimore
[Equation]
Lemma 19.
[Equation]
we can show that
[Equation]
The last line of the Lemma follows from noting that L 2 =
O ( m ) .
B.4 Adaptive design and Modiﬁed Regularized Estimator: Relation to prior work
With the adaptive estimator, we could alternatively directly deﬁne an estimator for C θ as C ˆ ϑ using only the projected values Z as done by Mutný et al.
In this case, however, additional bias growing in time t enters into the conﬁdence parameter as (9) as 1 λ (cid:80) ti =
[Equation]
j i (cid:107) V − 1 0 , as we show in
Theorem 10 below.
Theorem 10 (Adaptive Design Regularized Regression - biased) .
Let Z t be s.t.
X i
= Z i C
+ J i , where J i is minimal as measured by squared norm, then the estimator
[Equation]
(35) has anytime conﬁdence sets for all t ≥
[Equation] [Equation]
where δ ∈ (0 , 1) , and Ω t,λ = Z (cid:62) t
Z t σ 2 + λ S , where S =
( C V 0 C (cid:62) ) −
Notice that if J is small in the Frobenius norm, the conﬁdence sets improve.
In fact, they depend only on the norm.
Proof.
[Equation]
Let us now analyze each term separately.
The self-normalized terms can be analyzed using a classical technique with a proper choice of mixture distribution as we show in the proof of the Theorem 3.
In this case it is a normal distribution N (0 , ( λ S ) − 1 ) .
The ﬁrst term can be shown to be bounded by,
[Equation]
They can be analyzed in the same spirit as in Mutný et al.
(2020) novel term
[Equation]
C Algorithms
Now we will brieﬂy discuss algorithms that construct designs X , and allocations over them η , which lead to low error either in expectation or with high probability - using A or E design, respectively.
Depending on the aim and estimator, different algorithms might be preferable.
For comprehensive reviews please refer to Todd (2016), Pukelsheim (2006),(Fedorov and Hackl, 1997) for optimization algorithms, and (Allen-Zhu et al., 2017), (Camilleri et al., 2021) for rounding techniques.
C.1 Greedy selection
A ﬁrst idea is to greedily maximize the scalarized information matrix with the update rule η t +1 =
t t +1 η t + 11+ t δ t , δ t = arg max x ∈X f (cid:18)
W λ (cid:18) t t + 1 η t + t + 1 δ x (cid:19)(cid:19) where f refers to the scalarization and δ x to the indicator of the discrete measure corresponding to the feature Φ( x ) .
Notice that while stated in form of allocations, due to the form of the update rule tη t is always an integer.
Surprisingly, this algorithm can fail with the interpolation estimator, where adding a new row to X , which does not lie in the kernel of C increases the variance.
This is an artifact of the pseudo-inverse, but it demonstrates that the algorithm is not universal – hence we suggest using it with the regularized estimator only.
C.2 Convex optimization
Alternatively, one can ﬁrst select a design space X supported on ﬁnitely many queries such that if the optimal design is supported on it, it leads to a desired low bias.
After that, we optimize the allocation η ∈ ∆ n over it.
This has the advantage that a) we can bound the query complexity to reach (cid:15) of learning as in Proposition 14, and b) we can provably achieve optimality with convex optimization, in contrast to the greedy heuristic.
We look for an allocation using convex optimization methods as in (10) or more generally, max η ∈ ∆ n f ( W ◦ ( D ( η ) 1 / 2 X )) where ◦ ∈ {† , λ } .
There are three possible ways to initialize the algorithm with X : make an ansatz, greedily reduce bias ﬁrst, or use a modiﬁcation of the random projection initialization of Betke and Henk (1992) described below.
Mirror-descent and Regularity
The objective above (or (10))
would be classically solved via
Frank-Wolf algorithm (Todd, 2016), or a mirror descent algorithm (Beck and Teboulle, 2003) 2 , which starts with the whole support, and reduces the weight of some of the queries η i , as
[Equation]
where s t is the stepsize; if s t ∝ √ t − 1 , the convergence is guaranteed.
Both E and A -design objectives are concave as they are related by linear transform C from the classical objectives, which are known to be concave (Pukelsheim, 2006).
paragraphInitialization via random projections This algorithm is inspired by volume algorithm of
Betke and Henk (1992).
The algorithm proceeds by picking a random vector c 0 in the span of C (cid:62) and picking two points ¯ z, z = arg max x c (cid:62) x, arg min x c (cid:62) x .
Subsequently, we pick another c 1 which is orthogonal to z − ¯ z and still in span of C (cid:62) .
We repeat this procedure k times.
Should this not generate a sufﬁciently accurate starting point with desired bias (as measured by Def. 1), we repeat this procedure until such design is obtained.
C.3 SDP reformulations of the objectives
We will now show that both A and E designs have an associated semi-deﬁnite reformulations which are possible if dim( H κ )
= m 3 .
One should bear in mind that these are possible only if the C θ is estimable in the sense of Def. 3, Hence their might not be utilizable in many instances that this paper studies, and rather apply in cases classical experiment design.
Nevertheless they do not appear in literature to the best of our knowledge.
On top of that these problems are more difﬁcult to solve that lets say the alternative algorithm with mirror descent, however with the use of off-the shelf solvers for SDP problems such as cvxpy, they can be conveniently implemented.
These refomrmulations are inspired by classical reformulations due to Boyd and Vandenberghe (2004), which show these for estimating θ directly.
The key ingredient is to make use of generalized Shur-complement theorem of Zhang (2011).
A-design objective is max η ∈ ∆ n 1 / Tr( W ( η ) − 1 ) can be represented as semi-deﬁnite program:
[Equation]
subject to
[Equation]
i ∈ [ k ]
E-design objective is max η ∈ ∆ n λ min ( W ( η )) , can be again solved using semi-deﬁnite programming as:
[Equation]
to
[Equation]
We will show how to apply the generalized for the ﬁrst objective (A-design) only.
The second reformulation follows analogously.
Notice that the objective max η 1 / Tr( W − 1 ) is equivalent to min η
Tr( W − 1 ) , and W − 1 = CV ( η ) † C (cid:62) .
Generalized Shur-complement lemma of Zhang (2011) states that CV ( η ) † C (cid:62) (cid:22) Λ with the projection operator lying in the null space of C , i.e. ( I −
V ( η ) † V ( η )
[Equation]
C (cid:62) = 0 , is equivalent to
[Equation]
The second condition is satisﬁed as long as the problem is estimable, in other words, there exists
L s.t.
C = LX .
Hence minimizing the trace of Λ , we can minimize the trace of the information matrix.
As the trace depends only on the diagonal elements of Λ , we can choose it to be diagonal, i.e., Λ =
Diag( u ) .
C. Query complexity and Optimization
The objective in (10) was stated in the kernelized form which is only valid if the RKHS is ﬁnitedimensional.
We state here for completeness the version which operates in the kernelized setting where all the operators can be inverted (and calculated explicitly using only the kernel κ evaluation).
[Equation]
D Geometry
We saw in Section 6 that properties of { Φ( x ) : x ∈ S} enter into consideration when we want to bound the total number of queries required to reach (cid:15) accuracy with high probability such that we can learn C θ .
The dependence enters as the smallest eigenvalue λ min of the information matrix W .
In this section, we relate the eigenvalue λ min ( W ) directly to the geometry of the above set using seminal results about E-experimental design from Pukelsheim and Studden (1993).
D.1 Set Width
The geometrical property we are interested in width of a set that we deﬁne below.
However, ﬁrst, we want to make a general note that the formal results of our theorems hold for symmetrized sets { Φ( x ) | x ∈ S} .
This is without the loss of generality, since, an optimal design on symmetrized set and a non-symmetrized set is equivalent.
This can be seen by considering W 1 ( η ) deﬁned on symmetrized set equates the one of non-symmetrized set W 2 ( η ) due to the fact evaluations Φ( x ) enter only as outer products W 1 ( η ) =
r ( (cid:80) n i =1 η i (Φ( x i )Φ( x i ) (cid:62) )) , where the sign clearly does not change anything in terms of the information matrix, and r correspond to further operations to deﬁne
W properly (i.e. projection and inverse).
Deﬁnition 5 (Width) .
Let D be a convex set in H , then the width of it is deﬁned as,
[Equation]
47) where θ is the unit norm.
Width in the span of C is deﬁned to be,
[Equation] [Equation]
where α is the unit norm.
Deﬁnition 6 (Gauge norm) .
Let D be a set of points in H , then the gauge norm of D is,
[Equation] [Equation]
Theorem 11 (Thm. 2.2 in (Pukelsheim and Studden, 1993)) .
Let W be any the information matrix for C θ , and z ∈ R k non-zero then
[Equation]
It holds with equality if W has smallest eigenvalue of multiplicity one and z is the associated eigenvector.
The treatment without multiplicity is somewhat more complicated and can be found in.
Proposition 12.
Under assumption of Thm. 11, no multiplicty, z being eigenvector of W , let
C : R m → R p , D be a symmetric convex set in H , then
[Equation]
Proof.
Due to homogenity of the gauge norm, ρ ( C
[Equation]
z
[Equation]
=
[Equation]
z
[Equation]
( ρ C (cid:62) ˆ z ) , where ˆ z is a unit vector.
Thus, the optimization can be restated as,
[Equation]
where in the second to last step we use the fact that the set is symmetric and ˆ z is unit eigenvector.
The last inequality follows by minimizing over all possible unit direction as int the deﬁnition of width.
The set ˜ D
⊂ H s.t.
D = { f ∈ H k
| f
= C (cid:62) ˆ z } .
Notice that this set lies in 1 -dimension subspace, and is directly proportional to the gauge-norm.
Given the above theorem, we can calculate the worst-case value of parameter
[Equation]
that bounds the complexity if the geometry of the set can be understood easily.
Alternatively, we can resort to direct calculation.
For example consider the gradient design problem, in which we know that for ﬁnite difference problem can be bounded by dh + O ( h ) , where h is the stepsize.
Proposition 13.
Under the assumption of the model (1) , if C = ∇ x Φ( x ) at x ∈ R d , Φ( x ) is the evaluation functional, and the design space X contains rows { Φ( x ± he i } di =1 where e i corresponds to the unit vectors in R d and h is the step-size, then
[Equation]
where η corresponds to the design supported on all points with equal weight.
Proof.
The inverse of the information matrix is,
[Equation]
In the ﬁrst step we used Taylor’s theorem and keep track of the order components.
In the second we used that Φ( x )Φ( x ) (cid:62) is psd, later we used the deﬁnition of the design and the properties of the projection matrix.
Lastly,
[Equation]
which ﬁnishes the proof.
Corollary 14 (Query complexity) .
Under assumption of Prop. 1 where η ∗ is the optimum to Eq. 10, it holds that
[Equation]
with probability 1 − δ .
Proof.
[Equation]
Now the W † nor ν 2 do not depend on σ 2 , however evaluating multiple times the same measurements
Namely T times, reduces the variance σ 2 by 1 /T .
After taking this into consideration and the square root it ﬁnishes the proof.
E Extra Examples
In this section, we provide additional applications of our framework that deserve a mention, but before we do, we want to bring attention to Figure 3, where we show a comparison of our conﬁdence sets with prior work and projected ones, similar to Figure 1.
As a refresher, recall that
[Equation]
E.1 Integral maps
Quadrature An integral is a special linear operator on the function space,
[Equation]
where p = 1 .
The approximate estimability condition can be written in a concise form
[Equation]
where l ( y ) =
L y , and y ∈ S which itself is equal to well-known integral distance
MMD H κ ( q, (cid:80) y ∈D
l ( y ) δ ( · − y )) referred to as Maximum Mean Discrepancy (MMD) multiplied by
B .
It measures the worst case integration error on the class of functions θ ∈ H κ s.t.
(cid:107) θ (cid:107) k ≤ 1 with nodes x ∈ S and weights l
x ) .
The method which minimizes this quantity is sometimes referred to as Bayesian Quadrature (Huszár and Duvenaud, 2012).
Note that minimization of this quantity would correspond to the interpolation estimator.
Fourier Spectra: Low-pass Filters Suppose we are interested in approximately learning the
Fourier spectrum of θ .
This might be especially interesting if θ is composed of two signals one with speciﬁc low-frequency bands and the rest occurs only elsewhere and is of no interest.
Let { ω i } pi =1 be the frequencies of interest, then the rows of C are C j = (cid:82) ∞ ∞ exp( iω j x )Φ( x ) (cid:62) · dx , which can be discretized or evaluated in closed form depending on the kernelized space.
Transductive Risk If the dimension of the response vector is very large much larger than the number of elements in the unlabeled set, which is a common appearance in deep learning or high dimensional statistics; there is still hope to have provable bounded error on selected important candidates.
As opossed to study the full risk E (cid:15)
[ E x ∼ ρ [( x (cid:62) ( θ − ˆ θ )) 2 ]] with data distribution ρ , we can optimize the risk on a set of candidates E (cid:15) [ (cid:80) pi
[( q (cid:62) i ( θ − ˆ θ )) 2 ]] , which becomes Tr( QV † Q ) , where
Q contains the rows of k are the candidates.
E.2 Solution to Contamination (Low pass ﬁlter)
Often in estimation, the signal of interested is corrupted by another signal that we are not interested in.
This is sometimes referred to as contamination (Fedorov and Hackl, 1997).
In particular, consider the following model E
[ y ] = ( α β ) (cid:62) ( φ ( x ) ψ ( x )) , where we want to infer α only.
Note that this is a special case of our framework, where the linear functional C
= I S zeros out the variables β in θ =
( α, β ) .
This problem is well-deﬁned only if the spans of { φ ( x ) } x ∈S and { ψ ( x ) } x ∈S are not contained in each other, which we neglect for our purposes here.
For illustration, consider a linear trend contaminated with function f , which has major frequency components that are known ,
i.e., y = α (cid:62) x
+ f ( x )+ (cid:15) .
We can stack these frequencies in an evaluation functional ψ ( x, { ω } mi =1 ) , and express the problem in the form above.
In Fig.
5, we report the MSE for estimation of α with f which has frequency components { πl, πel | l ∈
[16] } weighted with 1 /l 2 .
The contamination-aware design tries to query points that eliminate the effect of f and leads to lower MSE for the same number of observations and performs much better in expectation than either random or full designs – an optimal design that learns both α and β .
For this design, we chose to sample greedily ﬁrst to reduce bias and then optimized the weights of each data point to achieve optimality.
E.3 Learning ODE solutions
Consider an example of damped harmonic oscillator, u ( t ) (cid:48)(cid:48)
+ κu ( t ) (cid:48)
+ u ( t ) =
for t ∈ [ t 0 , t 1 ] .
Adopting the above procedure and using the squared exponential kernel to embed trajectories 4 , we show in Figure 4a inference with the shape constraint using A -optimal design as well as equally space design, we see the resultant conﬁdence bands and ﬁt are much better for optimized design.
We assume that κ ∈ ( κ − , κ + ) and give a union of conﬁdence sets as in the robust design case.
The optimal design then corresponds to the unknown null subspace C and its variance part is equal to sup κ ∈ [ κ − ,κ + ]
Tr( C κ V † C (cid:62) κ ) .
E.4 Learning PDE solutions
In a similar fashion as the linear ODE feature, a linear constraint so do linear PDEs.
We demonstrate this on a sensor placement problem, where consider heat equation in two dimensions with u ( t, x )
[Equation]
We assume that the point value of initial conditions u (0 , x ) can be measured by placing sensors along x .
The goal is to place these sensors such that after elapsed time T , u ( T ) can be inferred with the lowest error.
This is the forward formulation of the inverse problem in Leykekhman et al.
We assume that θ is a bounded member of RKHS due to the squared exponential kernel.
In this case we are not interested in the full nullspace of N as in the previous example, instead we focus on the range space of C =
Φ( x, T )( N (cid:62) N ) corresponding to information at Φ( x, T ) .
E.5 Sequential Design: Estimating CVar
Often the objective we are trying to estimate is inherently stochastic such as the following model:
[Equation]
In particular one can be interested in x s.t. arg
[Equation]
z ) (cid:62) θ ] .
This is analyzed in sequential experiment design optimization with risk measures, where ρ is the risk measure (see (Cakmak et al., 2020) and (Agrell and Dahl, 2020)).
It is often assumed that W cannot be evaluated explicitly, only sampled from with the evaluation oracle y i =
Φ( x i , z i ) (cid:62) + (cid:15) i .
We focus on a problem where we want to learn the risk of the actions q ∈ Q, | Q | =
k , E z ∼ W [ ρ (Φ( q, z ) (cid:62) θ )] instead of minimization and ρ is CVar risk measure.
With a ﬁxed value θ , this is a linear operator C θ .
Since θ is unknown, we adopt a sequential procedure, where we solve for the risk C ˆ θ , where ˆ θ is sampled uniformly from its conﬁdence set.
F Experiments: Details F.1 Gradient Estimation: Details
With gradient estimation we are interested in ∇ x Φ( x ) of an element θ ∈ H κ .
The example generated in Fig.
2b is in 2 dimensions, where we approximate Φ( x ) of squared exponential kernel with lengthscale l = 0 .
1 with Quadrature Fourier Features of Mutný and Krause (2018) for the convenience of optimization.
We consider a slightly off-set ﬁnite difference design, with the following elements, { Φ( x ) , Φ( x + he 1 ) , Φ( x + 2 he 1 ) , Φ( x − he 1 ) , Φ( x − he 2 ) } .
We calculate the relative-bias ν due to the Def. 1 and balance it with the budget (as in the Figure).
This way we can identify the optimal h ∗ before even solving for the optimal design.
Given the identiﬁed h ∗ , we calculate the optimal design with mirror descent solving the objective (10), to get η ∗ = (0 . 37 , 0 . 09 , 0 .
[Figure]
which is not obvious without understanding the geometry of the problem.
The value of σ was set to be 0 .
The total error in Fig.
2b is the combination of bias and variance which was calculated using the largest eigenvalue of E ( L † ) as in (2).
This is the error we can guarantee with high probability (not the actual error) which is somewhat lower.
F.2 The solution to Contamination (Statistics): Details
For this example, consider a linear trend contaminated with f ( x ) which has known major frequency components, i.e., y
=
α (cid:62) x
+ f ( x ) + (cid:15) .
We can concatenate these frequencies, and construct an evaluation operator where Ψ( x ) i = cos( ω i x ) , and Ψ( x ) i +
= sin( ω i x ) where ω i is one of these frequencies.
Then, we express the problem as y =
θ (cid:62) Φ( x ) =
( α, β ) (cid:62) ( x, Ψ( x, { ω i } mi =1 ) + (cid:15) .
Since we are interested only in α , the operator C = (1 , 0 ) selects it.
Hence we look for a design (subset of points) which only minimizes the residuals due to estimation of α .
In fact, such design might indirectly minimize the other residuals as well, but only if this is necessary to reduce the residuals due to α .
The example depicted in Fig.
5 is created using the frequencies { πl | l ∈ [16] } ∪ { πel | l ∈ [16] weighted with 1 /l 2 .
Such weighting of high frequencies creates contaminating signals that are similar to functions from Matérn kernel spaces (Rasmussen and Williams, 2006).
The contamination-aware design performs much better.
The values of σ = 0 .
5 (relatively high).
We compare with full design – one that minimizes the overall estimation error or in other words, where C is the identity.
We ﬁrst use the greedy algorithm to minimize the regularized estimator to obtain initial design space.
Then we use convex optimization to ﬁnd a proper allocation of a ﬁxed budget (varying in the x-axis) to these queries.
We again optimize with mirror descent.
F.3 Pharmacokinetics: Details
The pharmacokinetic two-compartment model models two organs.
In this case the stomach and blood and the transfer of the medication between them.
The model can be mathematically described as
[Equation]
where c s ( t ) represent concentration of the medication in the stomach, and c b ( t ) represents the concentration in the blood.
The goal of pharmacokinetic studies is to identify ( a, b, c ) from draws of the blood of subjects to e.g. properly assign dosage.
In comparison to the example of the damped harmonic oscillator, the initial conditions are known – albeit imprecisely – what is the true unknown is γ = ( a, b, c ) .
We make the following consideration, in order to infer γ precisely we need to have a precise estimation of the trajectory, hence we inject a slight perturbation to the initial condition and then search for the optimal design that is good for any of the models γ : robust design.
This way with any of the models from γ ∈ Γ we will have a good estimation of trajectories and hence a good estimation of γ using MLE as we report in Fig.
4b.
The speciﬁc parameter used was σ = 0 .
01 which corrupted the observations of c b ( t ) at chosen times and the space which Embeds the trajectories was chosen to be that of the squared exponential kernel with lengthscale l = 0 .
The true parameters were set to be (5 , 10 , 10) and the robust set of Γ =
((4 , 6) , (9 , 11) , (9 , 11)) for each parameter.
We use the greedy algorithm with regularized estimator and λ = 1 / 2 to optimize the robust A -optimal metric.
The resultant set is visualized in
Fig.
2a.
F.4 Control: Details
In the last experiment, we essentially model the same problem as in Lederer et al.
(2021) as we explain.
The only difference in our setup is the use of a gain K = 200 instead of K = 15 .
With a low gain of K = 15 one cannot provably stabilize with the given Lyapunov function.
With K = 200 we were able to quickly stabilize the system once enough data points were obtained.
Notice the reference trajectory and trajectory with "bad" and "good" controllers in Fig.
The driving non-linear dynamics are visualized likewise.
The experiment is designed to showcase the merit of the newly designed adaptive conﬁdence sets with the stopping rule that stops once the controller has been veriﬁed to be stable.
We repeated each experiment 10 times and reported the standard quantiles of the total derivative of the Lyapunov function in Fig. 2c.
We always started with the initial set of 10 data points and then proceeded with exploration (adding speciﬁc data points) if the supermum of the total derivative was not negative.
When it was we stopped.
The stopping corresponds to the line going into negative values in the log plot in Fig.
2c.
We considered these four strategies:
• random - randomly sampling data point from the whole domain [-1.5,1.5].
• random-ref - randomly sampling data point around the operating region.
• unc - sampling (greedily) the most uncertain query from the whole domain [-1.5,1.5].
• unc-ref - sampling (greedily) the most uncertain query from the operating region.
Berkenkamp et al.
(2016) provides a better solution than uncertainty sampling, which is a special case of the linear functional studies in this work.
In this example, this closely follows the unc-ref baseline and would create additional clutter.
The inclusion of this algorithm would not further validate the beneﬁt of the tighter conﬁdence sets.
The operating region is chosen to be a tube around the reference trajectory of width h = 0 .
The dynamics is modeled using Nystöm features ( m = 400 ) of the squared exponential kernel with lengthscale 0 .
25 and regularized estimator with λ estimated experimentally.
We selected this with visual inspection since the example is known, but one can adopt the procedure in Umlauft et al.
The noise std.
σ = 0 .
05 and number of queries is T = 500 .
G Improved regret of linear bandits: Proofs
To prove the following theorem we require the famed potential lemma which we state for completeness,
[Equation]
A reference for this result can be found in (Szepesvari and Lattimore, 2020).
Note that a bounded pay-off vector assumption has been used in the derivation.
In general this result can be extended to kernelized bandits by noting that
[Equation]
where γ T = log (cid:16) det( V t ) det( I ) (cid:17) , due to matrix inversion lemma, this can be also expressed in kernelized form, as is usually referred to as information gain (Srinivas et al., 2010).
[Figure]
Theorem 15.
Let θ ∈ R
d be an unknown pay-off vector, and a set of actions x ∈ X such that |X|∞ , then the regret of UCB algorithm is no more than
[Equation]
54) with probability 1 − δ .
Proof of Theorem 15.
We drop Φ and use x instead only.
Note that due to Lemma 3, we have that
[Equation]
where β t is deﬁned for the information matrix Ω t,k in Theorem 3.
The subscript k designates the dependence on x k as we will need to consider all x k ∈ X , where we index X using k ∈ N .
Using this and δ k,t indicator which is zero or one depending on whether action k was played at time t .
Note that,
R T ≤
[Equation]
=
[Equation]
=
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
which ﬁnishes the proof
The ˜ θ t is the vector that corresponds to the UCB action played by the algorithm.
We took the union bound for the adaptive conﬁdence bounds for each x ∈ X , and hence the logarithmic dependence on |X| .
Deﬁnition 7 (Matrix slices and weighted slices) .
[Equation]
and
[Equation]
and
[Equation]
Lemma 5 (Zhang (2011)) .
Let M be a positive deﬁnite matrix, and S be a subset of [ n ] , then ( M S
[Equation]
† (cid:22) ( M
[Equation]
S
[Equation]
Lemma 6 (Zhang (2011)) .
If A (cid:23) B where A , B ∈ R l × l then for any X ∈ R d × l , XAX (cid:62) (cid:23)
XBX (cid:62) .
Lemma 7.
Let A ∈ R k × m , then the matrix P =
A (cid:62) ( AA (cid:62) ) − 1 A is projection matrix.
Proof.
Its easy to check P (cid:62) =
P and P 2 = P .
Lemma 8.
Let A ∈ R d × d full rank, and C ∈ R k × d with rank k .
Then CAC (cid:62) has rank k .
Proposition 16.
Let X ∈ R m × n s.t. min
( m, n ) ≥ | η | , where X is full rank,
[Equation]
Also,
[Equation]
If n = | η | , then
[Equation] [Equation]
Proof.
Tedious veriﬁcation of four pseudo-inverse criteria.
H.1 Auxiliary
Lemma 9.
Let A ∈ R l × l (cid:31) 0 , and X ∈ R d × l diagonal where d ≤ l , then
[Equation]
Proof.
Let S be a set that selects exactly the non-zero elements of matrix X and further suppose w.l.o.g.
the matrix is in such permutations that this block is right-upper most.
By Lemma 5, we know that ( A SS ) − 1 (cid:22) ( A − 1 ) SS .
Due to the monotonicity of the operation (Lemma 6) X SS ( A SS ) − 1 X (cid:62) SS (cid:22) X SS
[Equation]
A
[Equation] [Equation]
SS X
[Equation]
SS .
Inverse operator reverse the above inequality, hence,
[Equation]
where deﬁnite following matrix Y s.t.
Y SS
=
( X SS ) − 1 and zero-otherwise.
Lastly, we know that A S has eigenvalues strictly smaller than A , and also note that XA − 1 X (cid:62) =
X SS ( A − 1 ) X (cid:62) SS .
Thus applying Lemma 6 we can lift the expression to l × l matrices,
( A ) (cid:23) I : S ( A ) SS I S :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Theorem 17.
Let A ∈ R l × l (cid:31) 0 , and M ∈ R d × l , where d ≤ l , and rank( M ) =
d then
[Equation]
Proof.
Let M = USV (cid:62) , where U ∈ R d × d and V ∈ R l × l , be an SVD decomposition of M and
A = R (cid:62) ΛR , where R ∈ R l × l . be eigendecomposition of A .
We perform the set of following equivalent operations to reduce the problem to diagonal case,
[Equation]
where V = R (cid:62) G is a new rotation matrix, and B = G (cid:62) ΛG =
V (cid:62) R (cid:62)
ΛRV = V (cid:62) AV .
The rest follows from Lemma 9.
Proposition 18.
Let
X ∈ R n × m , and M ∈ R k × m , where k ≤ m , and rank( M ) =
k then if ∃ A s.t.
M = AX , where A ∈ k × n and k ≤ n , the following holds,
[Equation]
Proof.
The pseudo-inverse of ( X (cid:62) X ) + = X (cid:62) ( XX (cid:62) ) − 2 X .
Consequently,
[Equation]
where the last line follows from the properties of a projection.
H.2 Main Linear Algebra Theorem
The previous proposition has rather restrictive conditions.
Namely, it requires the existence of a solution to k linear systems that are overdetermined.
In particular m equations for n unknowns.
A consequence of an approximate solution to such a system with a bounded error matrix is the content of the following theorem.
Theorem 19.
Let
X ∈ R n × m , and C ∈ R k × m , where k ≤ m , and rank( C ) =
k then if the least squares solution A ∈ R k × n satisﬁes (cid:107) C − AX
[Equation]
2 F ≤ δ
[Equation]
A
[Equation]
2 F and is full rank, the following holds,
[Equation] [Equation]
Proof.
Without loss of generality (as n ≥ k ), we assume that C − AX = AΣ , where Σ ∈ R n × m .
The condition in the statement is implied when (cid:107) Σ (cid:107) 2 F ≤ δ 2 .
Due to A , being OLS estimate, A = CX (cid:62) ( XX (cid:62) ) − 1 .
To select Σ we need to solve,
[Equation]
A speciﬁc least Frobenious norm choice for Σ leads to
[Equation]
CV
[Equation]
) −
[Equation]
X ) , for which XΣ (cid:62) = 0 n × n and ΣX (cid:62) = 0 n × n .
In the above, the pseudo-inverse of V † =
( X (cid:62) X ) † =
X (cid:62) ( XX (cid:62) ) − 2 X .
Consequently using C
=
A ( X + Σ ) ,
[Equation]
where S = X (cid:62) ( XX (cid:62) ) − 1 .
Due to properties above
[Equation]
this simpliﬁes to
[Equation]
The last step in the theorem follows by following claim.
We look for a speciﬁc Σ s.t. norm constraint is satisﬁed as well as the expression is maximized in terms of Löwner ordering.
Without loss of generality, there exists B ∈ R n × n s.t. Σ = BX , thus
[Equation]
where max is understood in Löwner ordering
[Equation]
We relax the constraint of the last objective,
[Equation]