Positively Weighted Kernel Quadrature via Subsampling
Anonymous Author(s)
AffiliationAddress
Abstract
We study kernel quadrature rules with convex weights.
Our approach combines thespectral properties of the kernel with recombination results about point measures.
This results in effective algorithms that construct convex quadrature rules usingonly access to i.i.d.
samples from the underlying measure and evaluation of thekernel and that result in a small worst-case error.
In addition to our theoreticalresults and the benefits resulting from convex weights, our experiments indicate thatthis construction can compete with the optimal bounds in well-known examples
Introduction
The goal of numerical quadrature is to provide, for a given probability measure µ on a space X , a setof points x 1 , . . .
, x n ∈ X and weights w 1 , . .
w n ∈ R such that
[Equation]
holds for a large class of functions f : X → R .
Kernel quadrature focuses on the case when thefunction class forms a reproducing kernel Hilbert space (RKHS).
What makes kernel quadratureattractive, is that the kernel choice provides a simple and flexible way to encode the regularityproperties of a function class.
Exploiting such regularity properties is essential when the integrationdomain X is high-dimensional or the function class is large.
Additionally, the domain X does nothave to be Euclidean, but can be any topological space that carries a positive semi-definite kernel.
More formally, given a set Q := { ( w i , x i ) :
i = 1 , . .
. , n } ⊂ R ×
X denote with µ Q := (cid:80) ni =1 w i
δ x ithe resulting measure on X .
We refer to Q [resp.
µ Q ] as a quadrature [resp.
quadrature measure], tothe points x 1 , . . .
, x n as the support of Q [resp.
µ Q ].
The aim of kernel quadrature is to constructquadrature measures µ Q that have a small worst-case error
[Equation]
where H k denotes the RKHS associated with a positive semi-definite kernel k .
If the weights arepositive and sum up to one, w i > 0 , (cid:80) ni
=
w i = 1 , then we refer to Q as a convex quadrature rule.
Contribution.
The primary contribution of this article is to leverage recombination (a consequenceof Carathéodory’s Theorem) with spectral analysis of kernels to construct convex kernel quadraturerules and derive convergence rates.
We also provide efficient algorithms that compute these quadraturerules; they only need access to i.i.d.
samples from µ and the evaluation of the kernel k .
See Table 1for a comparision with other kernel quadrature constructions.
Method
Bound of squared wce
Computational complexity
C
M
E
Herding [9, 3] 1 /n n · ( n : global optimization ) ✓ ✓
SBQ [20]
Not found n · ( n 2 : global optimization ) ✓
Leveraged [2] σ m , m = O ( n log n )
Unavailable
DPP [6, 5] r n + n 3 · ( rejection sampling )
CVS [7] σ n +
Unavailable ✓
Ours:
[Equation]
N. + empirical ‡
[Equation]
✓ ✓ ✓
Why Convex Weights?
There are several reasons why convex weights are preferable: (i) Positive Integral Operator: Kernel quadrature provides an approximation of the integration operatorf (cid:55)→ I ( f )
= (cid:82) f
( x ) d µ ( x ) .
Hence, a natural requirement is to preserve basic properties of thisoperator and positive weights preserve the positivity of this operator.
(ii) Uniform estimates and
Robustness: In applications, the RKHS H k may be mis-specified if a quadrature rule with negativeweights is applied to a function f / ∈ H k , the approximation error (1) can get arbitrary bad; in contrast,a simple estimate shows that convex weights give uniform bounds, see Appendix B.3.
(iii) Iteration: Consider the m -fold product of quadrature formulas for approximating µ ⊗ m on X m .
This isa common construction for a multidimensional quadrature formulas (e.g., for polynomials) fromone-dimensional formulas [52] or numerics for stochastic differential equations [35].
In doing so,working with a probability measure is strongly preferred, since otherwise the total variation of theirm -fold product gets exponentially large as m increases.
Related Literature.
Roughly speaking, there are two approaches to kernel-based quadratureformulas: kernel herding and random sampling.
In kernel herding or its variants, the points ( x i ) ni =1are found iteratively, typically based on the Frank–Wolfe gradient descent algorithm [9, 3, 20].
In the random sampling approach, ( x i ) ni =1 are sampled and subsequently the weights are optimized.
Generically, this results only in a signed measure µ Q but not a probability measure.
Bach [2] and
Belhadji et al.
[6] use the eigenvalues and the eigenfunctions of the integral operator K : f (cid:55)→ 45 (cid:82)
X k ( · , y ) f
( y ) d µ ( y ) to obtain a Mercer-type decomposition of k [50].
Bach [2] then uses theeigenvalues and eigenfunctions of K to define an optimized measure from which the points ( x i ) arei.i.d.
sampled.
This achieves a near optimal rate, but the exact sampling from this measure is usuallyunavailable, although for special cases, it can be done efficiently.
In contrast, Belhadji et al.
[6]proposes non-independent sampling based on the determinantal point process [DPP; 19].
These twopapers also treat the more general quadrature problem that includes a weight function g ∈ L 2 ( µ ) , i.e.,approximating (cid:82) X f ( x ) g ( x ) d µ ( x ) for f ∈ H k ⊂ L 2 ( µ ) , which we do not discuss in this paper.
Finally, we emphasize that the literature on kernel quadrature is vast, and the above distinction is onlya rough dichotomy: in addition to kernel herding or sample optimization, the literature includes hybridapproaches [8], kernel quadrature combined with control variate and/or Sard’s method [40, 24, 48], as well as convergence analysis for the case when the RKHS is mis-specified [22, 23]. Table 1 providesa comparison on existing kernel quadrature methods and our contribution.
Outline.
Section 2 contains our main theoretical and methodological contribution.
Section 3provides numerical experiments on common benchmarks.
The Appendix contains several extensionsof our main result, proofs, and further experiments and benchmarks.
Main Result
Assume we are given a set 1 of n − 1 functions φ 1 , . .
φ n − 1 : X → R such that their linearcombinations well approximate functions in H k .
Then our kernel quadrature problem essentiallyreduces to the construction of an n -point discrete probability measure µ Q n = (cid:80) ni
=1 w i
δ x i such that
[Equation]
A simple way to approximately construct this µ Q n is to first, sample N ≫ n points, ( y i )
Ni =1 ,from µ such that their empirical measure, (cid:101) µ N = 1 N (cid:80) N i =
δ y i , is a good approximation to µ in thesense that (cid:82) φ i d (cid:101) µ N ≈ (cid:82) φ i d µ for i = 1 , . . .
, n − 1 , and secondly, apply a so-called recombinationalgorithm (Remark 1) that takes as input ( y i )
Ni =1 and n functions φ 1 , . .
. , φ n − 1 and outputs a measureµ Q n = (cid:80) w i δ x i by selecting a subset ( x i )
Ni
=1 of the points ( y i )
Ni =1 and giving them weights ( w i ) ni
=1such that µ Q n is a probability measure that satisfies the equation (3) with µ replaced by (cid:101) µ N .
The challenging parts of this approach are (i) to construct functions φ 1 , . . .
, φ n − 1 that approximatelyspan the RKHS H k for a small n ; (ii) to arrive at good quantitative bounds despite the (probabilistic)sampling error resulting from the use of the empirical measure (cid:101) µ N , and the function approximationerror via φ 1 , . .
φ n − 1 .
To address (i) we look for functions such that
[Equation]
with some c i ≥ 0 .
Two classic ways to do this are the Mercer and Nyström approximations.
Theremaining, item (ii) is our main contribution.
Theorem 1 shows that the worst-case error, (5), iscontrolled by the sum of two terms: the first term stems from the kernel approximation (4), the secondterm stems from the sample error.
Theorem 1.
Let µ be a Borel probability measure on X and k a positive semi-definite kernel on Xsuch that (cid:82) X k ( x, x ) d µ ( x ) < ∞ .
Further, let n be a positive integer and assume k 0 is a positivesemi-definite kernel on X such that1.
k − k 0 is a positive semi-definite kernel on X [Equation] and
[Equation]
dim H k 0 < n
[Equation]
There exists a function KQuad such that if D N is a set of N i.i.d.
samples from µ , then Q n =
KQuad( D N ) is a random n -point convex quadrature that satisfies
[Equation]
(5) where c k,µ := (cid:82) X k ( x, x ) d µ ( x ) − (cid:82)(cid:82) X×X
k
( x, y ) d µ ( x ) d µ ( y ) .
Moreover, the support of Q n is a subset of D N and given functions φ 1 , . .
φ n − 1 ∈ L 1 ( µ )with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } , Q n =
KQuad( D N ) can be computed with Algorithm 1 in
O (cid:0) nN + n 3 log( N/n ) (cid:1) computational steps.
The function KQuad is deterministic but since D N is random, the resulting quadrature rule Q n israndom, hence also the resulting worst case error wce( Q ; H k , µ ) and the expectation in (5) denotesthe expectation over the N samples in D N .
The theoretical part of Theorem 1 follows from more general results that we present and prove in Appendix (Theorem 6 for inequality, essentially done bycomparing H k with H k 0 , and 7 for existence) while the algorithmic part of Theorem 1 is discussed inthe following section.
This result covers our two main examples for the construction of k 0 , resp.
thechoice of φ 1 , . .
φ n − 1 , that we discuss below and for which the error estimate gets quite explicit:the Mercer approximation and the Nyström approximation.
The former requires some knowledgeabout the spectrum of the kernel which is, however, known for many popular kernels; the latter worksin full generality but yields worse convergence rates.
Finally, we emphasize that N and n in Theorem1 can be chosen independently and we will see that from a computational point the choice N ∼ n 2 ispreferable in which case (5) is faster rate than Monte Carlo, see also Table 1.
2.1 Algorithm
Suppose we are given k 0 and φ 1 , . .
φ n − 1 ∈ L 1 ( µ ) with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } , and also
N independent samples µ denoted by D N =
( y 1 , . . .
, y N ) .
From Theorem 6 in Appendix, if weconstruct a convex quadrature Q n = ( w i , x i ) ni
=1 satisfying
[Equation]
) where φ = ( φ 1 , . .
φ n − 1 ) ⊤ and k 1 , diag ( x ) =
k ( x, x ) − k 0 ( x, x ) , it satisfies the bound (5).
For thisproblem, we can use the so-called recombination algorithms:
Remark 1 (Recombination) .
Given d − 1 functions (called test functions) and a probability measuresupported on N > d points, there exists a probability measure supported on a subset of d points thatgives the same mean to these d − 1 functions.
This follows from Carathéodory’s theorem and is knownas recombination.
Efficient deterministic [32, 36, 52] as well as randomized [10] algorithms exist tocompute the new probability measure supported on d points; e.g. deterministic algorithms performthe recombination, step 1, in O (cid:0) c φ N + d 3 log( N/d ) (cid:1) time, where c φ is the cost of computing all thetest functions at one sample.
If each function evaluation is in constant time, c φ
= O ( d ) .
Although the recombination introduced here only treats the equality constraints in (6) we cansatisfy the remaining constraints just with n points by modifying it.
This is done in Algorithm 1.
What the algorithm is doing is as follows.
First, via recombination, find an ( n + 1) -point convexquadrature R n +1 =
( v i , y i ) n +1 i =1 that exactly integrate functions φ 1 , . .
. , φ n − 1 , k 1 , diag with regardto the empirical measure 1 N (cid:80) Ni =
δ y i .
Second, to reduce one point, find a direction ( − u in thealgorithm) in the space of weights on ( (cid:101) x i ) n +1 i =1 that does not change the integrals of φ 1 , . .
. , φ n − 1 andthe constant function 1 , and does not increase the integral of k 1 , diag .
Finally, move the weight fromv to the above direction until an entry becomes zero, at v − α u .
Such an α ≥ 0 exists, as u musthave a positive entry since it is a nonzero vector whose entries sum up to one.
Now we have a convexweight vector with at most n nonzero entries, so it outputs the desired quadrature satisfying (6).
Input: A positive semi-definite kernel k on X , a probability µ on X , integers N ≥ n ≥
1 , another kernel k 0 , functions φ 1 , . .
φ n − 1 on X with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } and a set D N of N i.i.d.
samples from µ .
{+Output: A set Q n := { ( w i , x i ) | i = 1 , . . . , n } ⊂ R × X with w i ≥ 0 , (cid:80) ni =
 w i =1: Apply a Recombination Algorithm (Remark 1) with ψ = ( φ 1 , . . . , φ n − 1 , k 1 , diag ) ⊤ , to the empirical measure 1 N (cid:80)+}
y ∈ D N δ y to obtain points { (cid:101)
x 1 , . .
, (cid:101) x n +1 } ⊂ D N and weights v = ( v 1 , . . .
, v n +1 )
⊤ ≥ 0 that satisfy 1 ⊤ v = 1 and ψ ( (cid:101) x ) v = 1 N (cid:80)
Ni =1 ψ ( (cid:101) x i ) , where ψ ( (cid:101) x ) =
[ ψ ( (cid:101) x 1 ) , . . .
, ψ ( (cid:101) x n +1 )]
∈ R n × ( n +1) .
2: Apply SVD with the matrix A
= [ φ i − 1 ( y j )] ij ∈ R n × ( n +1) with φ 0 = 1 to find a nonzero vector u ∈ R n +1 such that A u = 0 and k 1 , diag ( (cid:101) x ) ⊤ u ≥3: Compute the smallest α ≥ 0 such that v − α u ≥ 0 and v j − αu j = 0 for some j 4: Return ( w i ) ni
=1 ← ( v k − αu k ) k
∈
I and ( x i ) ni
=1 ← ( (cid:101) x k ) k
∈
I , where I = { 1 , . .
. , n + 1 } \ { j }
2.2 Mercer Approximation
In this section and Section 2.3, we assume that k has a pointwise convergent Mercer decompositionk ( x, y )
= (cid:80) ∞
m =1 σ m e m ( x ) e m ( y ) with σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 ⊂ L 2 ( µ ) being orthonormal [50].
If we let K be the integral operator L 2 ( µ ) →
L 2 ( µ ) given by f (cid:55)→ (cid:82) X k ( · , y ) f
( y ) d µ ( y ) ,then ( σ m , e m ) ∞ m =1 are the eigenpairs of this operator.
The first choice of the approximate kernel k 0 is just the trucation of Mercer decomposition.
Corollary 2.
Theorem 1 applied with k 0 ( x, y )
= (cid:80) n − [-1 m =
-][Equation]
( x ) e m ( y ) yields a random convexquadrature rule Q n such that
[Equation]
Proof.
It suffices to prove the result under the assumption (cid:82)
X k ( x, x ) d µ ( x ) = (cid:80) ∞ m =1 σ m < ∞ , asotherwise the right-hand side of (7) is infinity.
For k 1 := k − k 0 , we have that k 1 ( x, y )
=
(cid:80) ∞
m = n σ m e m ( x ) e m ( y ) and it is the inner product ofΦ( x ) :
= ( √ σ m e m ( x )) ∞ m = n and Φ( y ) in ℓ 2 ( { n, n + 1 , . . . } ) and so positive semi-definite.
Thus kand k 0 satisfies the assumption of Theorem 1, and (cid:82) X k 1 ( x, x ) d µ ( x ) = (cid:80) ∞ m = n σ m applied to (5)yields the desired inequality.
2.3 Nyström Approximation
Although the Nystöm method [58, 11, 28] is primarily used for approximating a large Gram matrixby a low rank matrix, it can also be used for directly approximating the kernel function itself whichis how we use it.
Given a set of ℓ points Z
=
( z i ) ℓi =1 ⊂ X , the vanilla Nyström approximation ofk ( x, y ) is given by
[Equation]
where P Z : H k → H k is a projection operator onto span { k ( · , z i ) } ℓi =1 .
In matrix notation, we have
[Equation]
where W =
( k ( z i , z j )) ℓi,j =1 is the Gram matrix for Z and W + denotes its Moore–Penrose inverse.
For completeness, we explain the equivalence between (8) and (9) in Appendix B.4.
As k Z isan ℓ -dimensional kernel, there exists an ( ℓ + 1) -point quadrature formula that exactly integratesfunctions in H k Z .
For a quadrature formula, exactly integrating all the functions in H k Z is indeedequivalent to exactly integrating k ( z i , · ) for all 1 ≤ i ≤ ℓ , as long as the Gram matrix k
( Z, Z ) isnonsingular.
Proposition 1 in the Appendix provides bound for the associated worst case error.
Fromthis viewpoint, the Nyström approximation offers a natural set of test functions.
The Nyström method has a further generalization with a low-rank approximation of k ( Z, Z ) .
Preciselyspeaking, by letting W s be the best rank- s approximation of W =
k
( Z, Z ) (given by eigendecomposition), we define the following s -dimensional kernel:
[Equation]
Let W =
U Λ U ⊤ be the eigendecomposition of W , where U =
[ u 1 , . . . , u ℓ ] ∈ R ℓ × ℓ is a realorthogonal matrix and Λ = diag
( λ 1 , . . .
, λ ℓ ) with λ 1 ≥ · · · ≥ λ ℓ ≥ 0 .
Then, if λ s > 0 we have
[Equation]
So we can use functions u ⊤ i k
( Z, · ) ( i = 1 , . . .
, s )
as test functions, which is chosen from a largerdimensional space span { k ( z i , · ) } ℓi =1 .
Although closer to the original usage of the Nystöm method is to obtain u ⊤ i k
( Z, · ) as an approximation of i -th eigenfunction of the integral operator K with Zappropriately chosen with respect to µ , we have adopted an explanation suitable for the machinelearning literature [11, 28].
The following is a continuous analogue of Kumar et al.
[28, Theorem 2] showing the effectiveness ofthe Nyström method.
See also Jin et al.
[21] for an analysis specific to the case s = ℓ .
Theorem 3.
Let s ≤ ℓ be positive integers and δ > 0 .
Let Z be an ℓ -point independent sample fromµ .
If we define the integral operator K Zs : L 2 ( µ ) → L 2 ( µ ) by f (cid:55)→ (cid:82) X k Zs ( · , y ) f ( y ) d µ ( y ) , then wehave, with probability at least 1 − δ , in terms of the operator norm,
[Equation]
The proof is given in Appendix C.5.
By using this estimate, we obtain the following guarantee for therandom convex quadrature given by Algorithm 1 and the Nyström approximation.
Corollary 4.
Let D N be N -point independent sample from µ and let Z be an ℓ -point independent sample from µ .
Theorem 1 applied with the Nyström approximation k [-
-]0 =
k Z n − 1 yields an random n -pointconvex quadrature rule Q n such that, with probability at least 1 − δ and k max := sup x ∈X k ( x, x ) ,
[Equation]
Proof.
From (11), k Z ( x, y ) − k Zn − 1 ( x, y )
= (cid:80) ℓi
= n λ − 1 i ( u ⊤ i k
( Z, x ))( u ⊤ i k
( Z, y ))
(ignore the termswith λ i = 0 if necessary), and it is thus positive semi-definite.
If we define P ⊥ Z : H k → H k as theprojection operator onto the orthogonal complement of span { k ( · , z i ) } ℓi =1 , then, from (8), we alsohave k ( x, y ) − k Z ( x, y )
= (cid:10) P ⊥ Z k ( · , x ) , P ⊥ Z k ( · , y ) (cid:11)
H k , so k − k Z is also positive semi-definite.
Inparticular, k − k
Zn − 1 =
( k − k Z )
+ ( k Z − k Zn − 1 ) is positive semi-definite.
Also, it suffices to provethe result when (cid:80) ∞ m =1 σ m < ∞ , so we can now apply Theorem 1.
For k 1 := k − k
Zn − 1 , we can prove that [Equation] and the desired inequality follows by combining Theorem 1 and Theorem 3 (i.e., (5) and (12)). For amore detailed derivation of (13), see (30) in Appendix D.2.
Remark 2.
Algorithm 1 with the Nystöm approximation can be decomposed into two parts: (a)
Nystöm approximation by truncated singular value decomposition (SVD) (the first n − 1 eigenvectorsfrom an ℓ -point sample), (b) Recombination from an N -point empirical measure.
The complexityof (a) is O (cid:0) nℓ 2 (cid:1) , and it can also be approximated by randomized SVD in O (cid:0) n 2 ℓ + ℓ 2 log n (cid:1) [15].
The cost of part (b) is O (cid:0) nℓN + n 3 log( N/n ) (cid:1) , where nℓN stems from the evaluation of k 1 , diagfor all N sampling points.
If we do not impose the inequality constraint regarding k 1 , diag , whichstill works well in practice, the cost of part (b) becomes O (cid:0) ℓN + n 2 ℓ log( N/n ) (cid:1) , by using the trick1 N (cid:80) Ni
=1 U ⊤ n − 1 k ( Z, y i ) =
U ⊤ n − 1 1 N (cid:80) Ni
=1 k ( Z, y i ) , where U n − 1 =
[ u 1 , . . .
, u n − 1 ] ∈ R ℓ × ( n − 1) is atruncation of the matrix that appears in the Nyström approxiamtion (10,11).
So the overall complexityis O (cid:0) nℓN + nℓ 2 + n 3 log( N/n ) (cid:1) while an approximate algorithm (randomized SVD, without theinequality constraint) runs in O (cid:0) ℓN + ℓ 2 log n + n 2 ℓ log( N/n ) (cid:1) .
Numerical Experiments
In this section, we compare our methods with several existing methods.
In all the experiments, weused the setting where we can compute (cid:82) X k ( x, y )
d µ ( y ) for x ∈ X and (cid:82)(cid:82) X×X
k
( x, y ) d µ ( x ) d µ ( y ) since then we can evaluate the worst-case error of quadrature formulas explicitly.
Indeed, if aquadrature formula Q n is given by points X
= ( x i ) ni
=1 and weights w = ( w i ) ni
=1 , then we havewce( Q n ; H k , µ ) 2 = w ⊤ [Equation] X, X [Equation]
y [
[Equation]
X
[Equation]
y )] [Equation] (14) for independent y, y ′ ∼ µ under (cid:82) X (cid:112) k ( x, x ) d µ ( x ) < ∞ , which is a well-known formula forthe worst-case error [14, 49].
An essential remark shown in Huszár and Duvenaud [20] is that the
Bayesian quadrature [41] with covariance kernel k given observation at points ( x i ) ni
=1 (automatically)estimates the integral as (cid:80) ni
=
w i f ( x i ) with ( w i ) ni
=1 minimizing the above expression.
Once givenpoints ( x i ) ni
=1 and additional knowledge of expectations, we can compute the optimal weights( w i ) ni
=1 by solving a convex quadratic programming (CQP), either without any restrictions or withthe condition that ( w i ) ni
=1 is convex.
Although the former can be solved by matrix inversion, we haveused the optimizer Gurobi 2 for both CQPs to avoid numerical instability.
For the recombination part,we have modified the Python library by Cosentino et al.
[10] implementing the algorithm of [52].
Our theoretical bounds are close to optimal in classic examples and we see that the algorithm evenoutperforms the theory in practice especially in Section 3.1.
We also execute a measure reduction ofa large discrete measure in terms of Gaussian RKHS and our methods shows a fast convergence ratein two ML datasets in Section 3.2.
3.1 Periodic Sobolev Spaces with Uniform Measure
For a positive integer r , consider the Sobolev space of functions on [0 , 1] endowed with the norm∥ f ∥
= ( (cid:82) 1 0 f ( x ) d x ) 2 +(2 π ) 2 r (cid:82) 1 0 f ( r ) ( x ) 2 d x , where f and its derivatives f (1) , . . .
f ( r ) are periodic(i.e., f (0)
=
f (1) and so forth).
This function space can be identifies as the RKHS of the kernel
[Equation]
for x, y ∈ [0 , 1] , where B 2 r is the 2 r -th Bernoulli polynomial [56, 2].
If we let µ be the uniform measure on [0 , 1] , the normalized eigenfunctions (of the integral operator) are 1 , c m ( · ) = √ 2 cos(2 πm · )and s m ( · ) = √ 2 sin(2 πm · ) for m = 1 , 2 , . . .
, and the corresponding eigenvalues are 1 and m − 2 r(both for c m and s m ).
Although the rectangle formula f (cid:55)→ n − 1 (cid:80) ni =1 f ( i/n ) (a.k.a.
Uniform
Grid below) is known to be optimal for this kernel [59, 39] in the sense of worst-case error, this
RKHS is commonly used for testing the efficiency of general kernel quadrature methods [2, 6, 23].
We also consider its multivariate extension on [0 , 1] d , i.e., the RKHS given by the product kernelk ⊗ d r ( x , y ) :
= (cid:81) di =1 k r ( x i , y i ) for x = ( x 1 , . . .
, x d ) , y =
( y 1 , . .
. , y d ) ∈ [0 , 1] d .
We carried out the experiment for ( d, r ) =
For each ( d, r ) , we comparedthe following algorithms for n -point quadrature rules with n ∈ { 5 , 10 , 15 , 20 , 30 , 40 , 50 , 65 , 80 } .
N. + emp, N. + emp + opt: We used the functions u ⊤ i k
( Z, · ) ( i = 1 , . .
. , n − 1 ) given by the Nyström approximation (11) with s = n − 1 as test functions φ 1 , . .
φ n − 1 in Algorithm 1.
The set Z was given as an ( ℓ =)10 n -point independent sample from µ .
We used N = n 2samples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (14)
M. + emp, M. + emp + opt ( d = 1 ): We used the first n − 1 functions of the sequence of eigenfunctions 1 , c 1 , s 1 , c 2 , s 2 , . . .
as test functions φ 1 , . . .
, φ n − 1 in Algorithm 1.
We used N = n 2samples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (14).
Monte Carlo, iid Bayes: With an n -point independent sample ( x i ) ni =1 from µ , we used uniformweights 1 /n in Monte Carlo and the weights optimized using (14) in iid Bayes .
Uniform Grid ( d = 1 ): We used the rectangle formula f
(cid:55)→ n − 1 (cid:80) n i =1 f
( i/n ) .
This is known tobe optimal [not just up to constant, but exactly; 59, 39], and thus equivalent to the Bayesianquadrature on the uniform grid, i.e., the weights are already optimized.
All done
MacBook
Halton, Halton + opt ( d ≥ 2 ): For an n -point sequence given by the Halton sequence with Owenscrambling [16, 42], the uniform weights w i = 1 /n is adopted in Halton and the weightsare additionally optimized using (14) in Halton + opt .
[Figure]
[Figure]
The results are given in Figure 1.
In d = 1 , the optimal rate given by Uniform Grid is known to be O (cid:0) n − 2 r (cid:1) .
As the uniform sampling is equal to the optimized distribution of Bach [2] in this case, iid Bayes also achieves this rate up to log factors.
Although our theoretical guarantee for M. + emp is
O (cid:0) n 1 − 2 r + N − 1 (cid:1) with N = n 2 (Corollary 2), in the case ( d, r ) =
(1 , 1) , we can observe that in theexperiment it is better than iid Bayes and close to the optimal error of Uniform Grid .
Moreover, N. +emp , which does not use the information of spectral decomposition, is remarkably almost as accurateas M. + emp in d = 1 .
Furthermore, if we additionally use the knowledge of expectations, which iid Bayes is already doing, M./N. + emp + opt become surprisingly accurate even with N = n 2 .
Thissuggests that our method proposes a good alternative for selecting points in Bayesian quadrature.
The accuracy of N. + emp + opt becomes more remarkable in multivariate cases.
It behaves almostthe same as Halton + opt in d = 2 and clearly beats it in d = 3 .
Also, the sudden jump of ourmethods around n = 30 in ( d, r ) =
(3 , 3) seems to be caused by the jump of eigenvalues.
Indeed,for the integral operator given by k ⊗ 3 3 with uniform measure, the eigenspace of the largest eigenvalue1 is of dimension 27 , and the next largest eigenvalue is 1 / 64 .
Note also that we can compare Figure 1 with Belhadji et al.
[6, Figure 1] which includes some othermethods such as DPPs, herding and sequential Bayesian quadrature, as we did experiments underalmost the same setting.
In particular, in the case ( d, r ) = (1 , 3) where the eigenvalue decay is fast,we see that our method substantially outperforms the sequential Bayesian quadrature.
3.2 Measure Reduction in Machine Learning Datasets
We used two datasets from UCI Machine Learning Repository 4 .
We set µ as the equally weightedmeasure over (a subset of) the data points X =
( X (1) , . . .
, X ( d ) ) ⊤ ( d = 3 , 5 , respectively), whereeach entry is centered and normalized.
We considered the Gaussian kernel exp( −∥ x − y ∥ (2 λ 2 ))
whose hyperparameter λ is determined by median heuristics [12], and compared the performance of N.+ emp , N. + emp + opt (with ℓ = 10 n , N = n 2 ), Monte Carlo , iid Bayes , and Herding , an equallyweighted greedy algorithm with global optimization [9], for n ∈ { 5 , 10 , 20 , 40 , 60 , 80 , 120 , 160 } .
The first is 3D Road Network Data Set [25].
The original dataset is 3 -dimensional real vectorsat 434874 points.
To be able to compute the worst-case error (14) efficiently to evaluate eachkernel quadrature, we used a random subset X of size 43487 = ⌊ 434874 / 10 ⌋ (fixed throughoutthe experiment) and defined µ as the uniform measure on it.
We determined λ with the medianheuristic by using a random subset of X with size 10000 and used the same X and λ throughout theexperiment.
The second is Combined Cycle Power Plant Data Set [26, 54].
The original dataset is5 -dimensional real vectors at 9568 points.
We set the whole data as X and defined µ as the uniformmeasure on it.
We determined λ with median heuristics by using the whole X .
Figure 2 shows the results.
We can observe that in both experiments N. + emp + opt successfullyexploits the fast spectral decay of Gaussian kernel and significantly outperforms other methods.
Also,even without using the knowledge of any expectations, N. + emp shows a decent convergence ratecomparable to Herding or iid Bayes .
Concluding Remarks
We leveraged a classical measure reduction tool, recombination, with spectral properties of kernelsto construct kernel quadrature rules with positive weights.
The resulting algorithms show strongbenchmark performance despite their restriction to convex weights.
Although our method is applicable to fairly general situation, the usage or performance can be limitedwhen it is difficult or inefficient to directly sample from the target measure µ .
So there are interestingfollow up questions; for example, one could study if one can replace the i.i.d.
samples with smartersampling (DPP, importance sampling, etc) before the recombination is carried out.
Also, the currenttheory does not fully explain the empirical superiority, and needs further investigation in future work.
Checklist
For all authors...
(a Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[No] This is
an improvement on a specific numerical method and we believe it does not have a
forseeable negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2and Appendix.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 2 andAppendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Included in the
supplementary material as a Zip file.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 3.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Figures in Section 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the footnote 3 in page 7 for theresource.
We also included the computational time in each Figure (for the largest case)
as well as the supplementary material (for all cases).
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3 and the
README file in the supplementary material.
(b) Did you mention the license of the assets?
[Yes] See the README file in the supple-
mentary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the README file in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
Did you describe any potential participant risks, with links to Institutional Review
[Table]
approvals, applicable
N
[Table]
you include the estimated hourly wage paid participants and the total amount
[Table]
spent participant compensation
N
[Table]
A Outline of the Appendix
Appendix B contains general results from which the results presented in the main text, in particular
Theorem 1, follow as special cases.
Appendix C contains the proofs of these theoretical results andneeded technical lemmas.
Appendix D shows that if the expectations (cid:82) φ i ( x ) d µ ( x ) are known,then this knowledge can be used to further improve the theoretical bounds; it also gives a simplemodification of Algorithm 1 doing this efficiently.
Appendix E provides additional numericalexperiments and benchmarks.
B Theoretical Results
In this section, we present theoretical results that include our main results as a special case.
Theproofs are given in Section C.
Notation.
For simplicity, for a quadrature Q n given by points ( x i ) ni
=1 and weights ( w i ) ni
=1 and aprobability measure µ , we denote the integration of an integrable function f on X with respect tothese measures by
[Equation]
respectively.
We also write the inner-product and norm of an RKHS H k by ⟨· , ·⟩ H k and ∥·∥ H k .
Furthermore, we use the probability simplex ∆ n and convex hull conv A of a set A ⊂ R d in theproofs:
[Equation]
B.1 Quantitative Results
We work under the following setting as in the assumption of Theorem 1.
Assumption A. µ is a Borel probability measure on X , and k is a positive semi-definite kernel on
X such that (cid:82) X
k
( x, x ) d µ ( x ) < ∞ .
Further, k 0 is a positive semi-definite kernel on X such thatk 1 :=
k − k 0 is a positive semi-definite kernel on X .
The following is a general result regarding a quadrature formula exactly integrating functions in H k 0 .
Theorem 5.
Under Assumption A, if an n -point convex quadrature Q n on X satisfies Q n ( f ) = µ ( f )for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g is the function given by g ( x ) = (cid:112) k 1 ( x, x ) .
In particular, the following assertions hold forsuch a quadrature Q n :
(a) We have wce( Q n ; H k , µ ) ≤ 2 ∥ g ∥ ∞
= 2 sup x ∈X (cid:112) k 1 ( x, x ) .(b)
If we additionally have Q n ( g ) ≤ µ ( g ) , then we have wce( Q n ; H k , µ ) ≤ 2 µ ( g ) .(c)
If we additionally have Q n ( g 2 ) ≤ µ ( g 2 ) instead of (b), we still have
[Equation]
Remark 3.
For a Borel probability measure ν on X and a nonnegative function h : X → R ≥ 0 ,we have an inequality (cid:82) X (cid:112) h ( x ) d ν ( x ) ≤ (cid:0)(cid:82)
X h ( x ) d ν ( x ) (cid:1) 1 / 2 , so the above µ ( g ) can be upperbounded by (cid:82) X k ( x, x ) d µ ( x ) , to the of the integral operator given by k
We can combine Theorem 5 with an empirical approximation of µ to obtain the following result,which essentially implies Theorem 1.
Theorem 6.
Under Assumption A, let D N be a set of N independent samples from µ , and (cid:101) µ N beits empirical measure, i.e., (cid:101) µ N = 1 N (cid:80)
y ∈ D N δ y .
Then, if an n -point convex quadrature Q n on Xsatisfies Q n ( f ) = (cid:101) µ N ( f ) for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g ( x ) := (cid:112) k 1 ( x, x ) and c k,µ :
= (cid:82) X
k
( x, x ) d µ ( x ) − (cid:82)(cid:82)
X×X k ( x, y ) d µ ( x ) d µ ( y ) .
In particular, the following assertions hold for such a quadrature Q n :
[Equation]
We
[Equation]
(b) If we additionally always require Q n ( g ) ≤ (cid:101) µ N ( g ) , then we have
[Equation]
The requirement Q n ( g ) ≤ (cid:101) µ N ( g ) can be replaced by Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 )
Although we have assumed k − k 0 is positive semi-definite in the previous assertions, the uniformbound works without the assumption as follows.
Proposition 1.
Let µ be a Borel probability measure on X .
Let k and k 0 be positive semi-definitekernels on X satisfying (cid:82) X (cid:112) k
( x, x ) d µ ( x ) , (cid:82) X (cid:112) k
( x, x ) d µ ( x ) < ∞ .
If an n -point convex quadrature Q n on X satisfies Q n ( f ) = µ ( f ) for any k 0 ( · , x ) with x ∈ X , we have
[Equation]
Furthermore, if dim H k 0 < n , there exists an n -point convex quadrature Q n satisfying Q n ( f ) = µ ( f )for each f = k 0 ( · , x ) .
In this paper, we focus on the cases where k 0 is either given by the truncated Mercer decomposition or
Nyström approximation.
For many important kernels, however, we may also use the random Fourierfeatures [45] or its periodic version [53] which can easily be combined with Proposition 1, but it isbeyond the scope of this paper to choose its appropriate variant for each kernel [see 33, for a list ofvariants].
B.2 Existence Results
The existence of quadrature formulas satisfying the estimate of Theorem 5 or Theorem 6 is guaranteedwhen dim H k 0 < n .
Theorem 7.
Under Assumption A, if dim H k 0 < n , there exists an n -point convex quadrature Q nsatisfying Q n ( f ) = µ ( f ) for each f = k 0 ( · , x ) .
This still holds even if we additionally require
Q n ( g ) ≤ µ ( g ) or Q n ( g 2 ) ≤ µ ( g 2 ) for g ( x ) =
(cid:112) k 1 ( x, x ) .
Remark 4.
This also implies the existence result of Q n satisfying the condition in Theorem 6 if wereplace µ by (cid:101) µ N .
The algorithm for constructing a kernel quadrature with Theorem 6 is given in the main body,see Algorithm 1.
The ones with Theorem 5 requires further knowledge of the expectation of testfunctions, i.e., the values of (cid:82) X φ i ( x ) d µ ( x ) with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } .
Under this additionalinformation, we have an algorithm (Algorithm 2) based on random sampling given in the followingsection.
B.3 Uniform Robustness
In applications, the RKHS H k may be mis-specified and the quadrature rule µ Q when computed forthe mis-specified function class H k but applied to a function f / ∈
H
k leads only to the attainable bound
[Equation]
via triangle equality and standard integral estimates.
Note that | · | TV denotes the total variation normand the above applies to any (cid:101)
f ∈ H k ; in particular, to the best approximation in uniform norm to fin H k .
Since µ is a probability measure, | µ | TV = 1 but if µ Q is a signed measure with non-convexweights, its total variation | µ Q | TV can be large, resulting in arbitrary large integration errors.
B.4 Equivalence between the projection/matrix Nyström approximations
Let k be a positive semi-definite kernel on X , Z = ( z i ) ℓi =1 ⊂ X .
Let P Z : H k
→ H k be theprojection operator onto span { k ( · , z i ) | i = 1 , . .
. , ℓ } .
For arbitrary x, y ∈ X , we can write
P Z k ( · , x ) = ℓ (cid:88) i = a i k ( · , z i ) ,
P Z k ( · , y ) =
ℓ (cid:88) i =
b i k ( · , z i ) , where a = ( a i ) ℓi =1 , b = ( b i ) ℓi =1 ∈ R ℓ .
From the properties of projection, we have
[Equation]
, P Z k
[Equation]
· , x
[Equation] (cid:88) i =
[Equation] i ) .
In matrix notation, we have k ( Z, x ) =
k ( Z, Z ) a , and k ( Z, y )
=
k ( Z, Z ) b from the same argument.
Thus, by conbining it with the property of Moore–Penrose inverse, we have
[Equation]
This shows the desired equivalence.
C Proofs
C.1 Proof of Theorem 5
Before proceeding to the proof of the theorem, we prepare a couple of assertions.
The following is awell-known estimate proven by using the Cauchy–Schwarz inequality [see e.g., 38, Lemma 3.1 andits proof].
Proposition 2.
Let k be a positive semi-definite kernel on X , and ν be a Borel probability measurewith (cid:82) X (cid:112)
k
( x, x ) d ν ( x ) < ∞ .
Then, for each f ∈ H k , we have
[Equation]
By using the proposition, we obtain the following technical lemma.
Lemma 1.
Let k and k 1 be a positive semi-definite kernels on X such that k − k 1 is also positivesemi-definite.
Let ν be a Borel probability measure on X .
Then, for any n ≥ 1 , a 1 , . . .
, a n ∈ R ,x 1 , . . .
, x n ∈ X , if we let f = (cid:80) ni
=1 a i k ( · , x i ) and f 1 = (cid:80) ni
=1 a i k 1 ( · , x i ) , then we have(cid:12)(cid:12)(cid:12)(cid:12)(cid:90) [Equation] f
[Equation]
x
[Equation]
k (cid:90)
X (cid:112) k
[Equation]
( x, x ) d ν ( x ) .
Proof.
From the positive semi-definiteness of k 0 :
= k − k 1 , we have
[Equation]
a i a j k
[Equation]
i
[Equation]
x
[Equation]
n (cid:88)
i,j = a i a j k 1 ( x i , x j ) + n (cid:88) i,j = a i a j k 0 ( x i , x j ) =
n (cid:88) i,j = a i a j
[Equation]
[Equation]
i
[Equation]
∥ f [Equation] 2 H k . Hence it suffices to prove | ν ( f 1 ) | ≤ ∥ f 1 ∥
H k 1 ν ( g ) for g ( x ) := (cid:112) k 1 ( x, x ) , but it directly followsfrom Proposition 2.
Proof of Theorem 5.
Note first that, for each f ∈ H k 0 , f is integrable with respect to µ .
Indeed, wehave
[Equation]
and it is integrable from assumption, so the equality Q n ( f )
= µ ( f ) with f = k [Equation] x ) is attained at afinite value.
Once we establish (15), the item (b) is clear, and (a) follows from the fact that Q n ( g ) and µ ( g ) areboth integrals of the function g with respect to a probability measure.
Also, (c) is justified as follows:
[Equation]
where µ ( Q n ) 2 ≤ µ ( Q 2 n ) and µ ( g ) 2 ≤ µ ( g 2 ) follows from the Cauchy–Schwarz.
To prove (15),we first prove
[Equation]
for any f of the form f = (cid:80) ni =1 a i k ( · , x i ) with n ≥ 0 and a 1 , . .
. , a n ∈ R .
Given such anf , we have Q n ( f 0 ) =
µ ( f 0 ) for f 0 :=
(cid:80) ni
=1 a i k 0 ( · , x i ) from the assumption.
Thus, by lettingf [Equation] f[-− f 
-]
[Equation] 0 =
[Equation] ni =1 a i k [Equation] · , x i ) , we have
[Equation]
As we have | ν ( f 1 ) | ≤ ∥ f
∥ H k ν
( g ) for ν = Q n , µ from Lemma 1, we obtain | Q n ( f 1 ) − µ ( f 1 ) | ≤∥ f ∥ H k ( Q n ( g ) + µ ( g )) , and so (17) is shown for f of the form f = (cid:80) ni
=1 a i k ( · , x i ) .
Finally, we generalize (17) to any f ∈ H k .
Let (cid:101) f
∈
H k can be written in the form (cid:80) ni
=1 a i k ( · , x i ) .
If we let h ( x ) = (cid:112) k
( x, x ) , from Proposition 2, we have
[Equation]
( h ) . Note that µ ( h ) < ∞ follows from the integrability of k ( x, x ) in Assumption A.
Therefore, we have
[Equation]
As we can make ∥ f − (cid:101) f ∥ H k arbitrarily small from the definition of H k , the proof of (15) is completedby taking the limit.
C.2 Proof of Theorem 6
Proof.
Denote D N = { y 1 , . . .
, y N } and note that the result follows from (16) and
[Equation]
where the first inequality is given by the Cauchy–Schwarz.
Indeed, (a) is an immediate consequence of (16) and Q n and (cid:101) µ N making a probability measure, and(b) is obtained as 2 E (cid:2)
[Equation]
) by using (18) andthe requirement Q n ( g ) ≤ (cid:101) µ N ( g ) .
When the requirement is Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 ) , as we have Q n ( g ) 2 ≤ Q n ( g 2 ) and (cid:101) µ N ( g ) 2 ≤ (cid:101) µ N ( g 2 )by the Cauchy–Schwarz, we also have by the AM–GM,
[Equation]
For showing (16), we remark that we always have
[Equation]
19) by applying Theorem 5 with (cid:101) µ N instead of µ .
Let h ( µ ) , h ( (cid:101) µ N ) , h ( Q n ) ∈
H
k be the kernel mean embeddings of µ , (cid:101) µ N and µ Q n , i.e.,
[Equation]
where ( w i ) ni
=1 and ( x i ) ni
=1 are weights and points defining the quadrature Q n .
Remark that h ( µ ) iswell-defined as (cid:82) X k ( x, x ) d µ ( x ) < ∞ [38, Lemma 3.1].
As we can rewrite the worst-case error as
[Equation]
by triangle inequality and the AM–GM, we obtain
[Equation]
where we have used (19) in the last inequality.
It now suffices to prove E (cid:2) ∥ h ( µ ) − h ( [Equation] µ N ) ∥ 2 H k (cid:3) =
c k,µ /N for showing (16).
Indeed, we have
[Equation]
since (cid:82)(cid:82) X×X k
( x, y ) d µ ( x ) d µ ( y ) =
(cid:82) X E [ k ( x, y i )]
d µ ( x ) =
E [ k ( y i , y j )] holds for i ̸
= j .
Thus,the proof is completed.
C.3 Proof of Proposition
Proof.
As Q n exactly integrates the functions in H k 0 , we have wce( Q n ; H k 0 , µ ) = 0 .
So, if we set
Q n ( f ) =
(cid:80) ni
=
w i f ( x i ) , then we have, from (14) with kernel k 0 ,
[Equation]
If we extract this from the formula (14) for the kernel k , we have, by letting k 1 :=
k − k 0 ,
[Equation]
So, if we define M := sup x ∈X | k 1 ( x, y ) | = sup x ∈X | k ( x, y ) − k 0 ( x, y ) | , we have
[Equation]
as Q n is a convex quadrature.
The existence follows from almost the same proof as in the proof of
Theorem 5, but in this case it directly follows from Tchakaloff’s thorem [51, 4].
C.4 Proof of Theorem 7
Proof.
We prove the existence of the version Q n ( g ) ≤ µ ( g ) .
The other follows just by replacingevery g in the proof below by g 2 .
Let φ 1 , . .
φ n − 1 ∈ H k 0 satisfy H k
= span { φ 1 , . .
φ n − 1 } .
Also, let y, y 1 , y 2 , . .
. be independent samples from µ .
Now, consider the vector-valued function ψ = ( φ 1 , . .
φ n − 1 , g ) ⊤ ∈ R n .
Note that E [ ∥ ψ ( y ) ∥ ] < ∞ follows from the integrability of elements in H k 0 and g with respectto µ .
Therefore, by [17, Theorem 11], with probability 1 , there exists an N such that E [ ψ ( y )]
∈conv { ψ ( y 1 ) , . . .
, ψ ( y N ) } .
So, in particular, there exist deterministic points x 1 , . . .
, x N ∈ X satisfying E [ ψ ( y )]
∈ conv { ψ ( x 1 ) , . .
, ψ ( x N ) } .
For such ( x i ) Ni =1 , consider an optimal solution that isalso a basic feasible solution of the following linear programming problem:minimize
[Equation]
where φ = (1 , φ 1 , . .
. , φ n − 1 ) ⊤ ∈ R n is another vector-valued function (note that its first coordinateis constant so that any feasible solution of (21) sums up to one).
Such a basic solution w has at mostn nonzero entries, say ( w i 1 , . .
, w i n ) ∈ ∆ n with 1 ≤ i 1 < · · · < i n ≤ N .
Then, the quadrature Q ngiven by weights ( w i j ) nj =1 and points ( x i j ) nj =1 satisfies Q n ( φ ) = µ ( φ ) and Q n ( g ) ≤ µ ( g ) .
Thelatter follows from the optimality of w and the fact that E [ ψ ( y )] ∈ conv { ψ ( y 1 ) , . . .
, ψ ( y N ) } leadsto a feasible solution with the objective E [ g ( y )]
= µ ( g ) ).
C.5 Proof of Theorem 3
We prove the theorem by using an existing bound regarding the Nystöm approximation for matrices,which is more common in the machine learning literature.
Let A =
( A ij ) Ni,j =1 ∈ R N × N be a symmetric positive semi-definite matrix.
Let us denote it as
A =
[ a 1 , . .
. , a N ] by using a 1 , . .
. , a N ∈ R N .
Then, we independently sample i 1 , . .
, i ℓ from{ 1 , . . .
, N } uniformly, and construct a submatrix B
=
( a i j i k ) ℓj,k =1 .
If we let B s be the best rank- sapproximation of B and B + s its pseudoinverse, the matrix
[Equation]
works as a rank- s approximation of A .
We use the following result on this matrix version:
Proposition 3 ([28, Theorem 2]) .
For a positive semi-definite matrix A , the rank- s approximation (cid:101)
Agiven above satisfies, with probability at least 1 − δ , the following:
[Equation]
where β ( ℓ, N ) = 1 − 1 2max
[Equation]
N
[Equation]
ℓ } , A max = max i A ii , D A max = max i,j ( A ii + A jj − 2 A ij ) and A sis the best rank- s approximation of A . As D A max ≤ 2 A max , if we have N ≥ 2 ℓ , it holds that
[Equation]
and we can just state
[Equation]
We show the following lemma as a consequence of this proposition.
Lemma 2.
Let s ≤ ℓ be positive integers and δ > 0 .
Let k : X
×
X be a symmetric and positivesemi-definite kernel and y 1 , y 2 , . .
. be i.i.d.
random variables taking values in X .
For each N , definethe N × N matrices K ( N ) , K s ( N ) , K Zs ( N ) by
[Equation]
where Z = ( y 1 , . .
, y ℓ ) .
Then, there exists a sequence ε N → 0 such that
[Equation]
is met with probability at least 1 − δ − ε N .
Proof.
We assume N ≥ 2 ℓ .
Let i 1 , . .
, i ℓ be independent uniform samples from { 1 , . . .
, N } .
Consider the event E N that i 1 , . .
, i ℓ are all different.
Then, P ( E N ) = (cid:81) [-(cid:96)i =
-]ℓi
=1 N +1 − i N convergesto 1 as N → ∞ , and let ε N = 1 − P
( E N ) .
By using Proposition 3, (24) and max i K ( N ) ii ≤
N − 1 sup x k ( x, x ) , we have that the probability
P
[Equation]
is at least (1 − δ − ε N ) / P ( E N ) ≥
1 − δ − ε N , where (cid:101) K s ( N ) is the rank- s Nyström approximationof the matrix K ( N ) by using indices i 1 , . .
, i ℓ .
From (22), if we take (cid:102) W = k
( y i j , y i k ) ℓj,k =1 and (cid:102) W sits best rank- s approximation, it actually satisfies
[Equation]
where D =
( y i 1 , . . .
, y i ℓ ) and k D s is the Nyström approximation given in the main body.
As y 1 , . . .
, y N are i.i.d.
samples, we can see that ( Z, ( y i ) Ni =1 ) (without any conditioning) and( D, ( y i )
Ni =1 ) conditioned on E N actually have the same distribution, so we are done.
We finally prove the result for the Nyström approximation of integral operators.
Proof of Theorem 3.
Take a sufficiently large N and let us use K ( N ) , K s ( N ) , K Zs ( N ) defined inthe previous lemma with y 1 , y 2 , . . .
independently sampled from µ .
It suffices to consider the case C k := sup x ∈X k ( x, x ) < ∞ .
It is clear that K s ( N ) ii ≤ K ( N ) ii ≤
C k /N , and from (10), we also have
[Equation]
and so K Zs ( N ) ii ≤ C k /N
[Equation]
For a matrix A ( N ) ∈ R N × N defined by A ( N ) ij = (1 − δ ij )( K ( N ) − K Zs ( N )) , i.e., the matrixgiven by deleting the diagonal, we have ∥ A ( N ) ∥ 2 → ∥K Zs − K∥ as N → ∞ almost surely [27,
Theorem 3.1].
Since we have observed that ∥ K ( N ) − K Zs ( N ) − A ( N ) ∥ 2 ≤ C k /N , we have
[Equation]
almost surely.
The same argument yields ∥ K ( N ) − K s ( N ) ∥ 2 → σ s +1 , as it converges to the normof the integral operator given by the kernel (cid:80) m ≥ s [-+
-]+1 σ m e m ( x ) e m ( y ) .
Now, by letting A N be the event that (24) holds (so P ( A N ) ≥ 1 − δ − ε N ), the desired inequality(12) almost surely holds under the event lim sup A N = (cid:84)
N>ℓ (cid:83) M ≥ N A M .
Indeed, under this eventwe can just take the limit of both sides of (24) for an appropriate subsequence of (2 ℓ, 2 ℓ + 1 , . . . ) .
Aswe have
[Equation]
the proof is completed.
D Kernel Quadrature when Expectations are Known
When we use an approximate kernel k 0 and know exact expectation of test functions φ 1 , . .
φ n − 1with H k 0 ⊂ span { φ 1 , . .
. , φ n − 1 } , we can obtain an n -point kernel quadrature that exactly integratesφ 1 , . .
φ n − 1 by Algorithm 2.
Input: A positive semi-definite kernel k on X , a probability measure µ on X , integers N ≥ n ≥ 1 , another kernel k 0 and functions φ 1 , . .
φ n − 1 on X with H k 0 ⊂ span { φ 1 , . . . , φ n − 1 } Output: With some probability, returns Q n = w i , x i ) |
i . , n } ⊂ R × X with ( w i ) ∈ ∆ n 1: Calculate the expectations (cid:82) X φ 1 ( x ) d µ ( x ) , . . . , (cid:82) X φ n − 1 ( x ) d µ ( x ) 2: Sample y 1 , . . .
, y N independently from µ 3: For a vector-valued function φ = ( φ 1 , . . . , φ n − 1 ) ⊤ and k 1 , diag ( x ) = k ( x, x ) − k 0 ( x, x ) , solve the linear programming problem ( |·| 0 denotes the number of nonzero entries)
minimize w ⊤ k 1 , diag ( x ) subject to [ φ ( y 1 ) · · · φ ( y N )] w = (cid:90) X φ ( x ) d µ ( x ) , w ≥ 0 ,
[Equation] to obtain points { x 1 , . . , x n } ⊂ { y 1 , . . . , y N } and weights ( w i ) ∈ ∆ n satisfying n
[Equation] if (25) is feasible.
We make several remarks on this algorithm.
First, the problem (25) is, strictly speaking, not a linearprogramming (LP), as it includes the sparsity constraint | w | 0 ≤ n .
However, as it only contains nequality constraints, its basic feasible solution always satisfies | w | 0 ≤ n and the simplex algorithm automatically gives such a sparse (and optimal) solution even if we do not explicitly impose thisconstraint, so we call it an LP for simplicity.
Second, this algorithm occasionally fails to output Q nas, with some probability, the LP has no feasible solution.
Although we can repeat the algorithm untilwe succeed, the number N should be chosen appropriately.
See Remark 5 for this point.
Finally,our algorithm has possibly related approaches such as sparse optimization and Sard’s method, see
Remark 6 and 7.
Remark 5.
A simple approach for constructing a quadrature formula [17] was recently proposed:randomly sample candidate points and find a solution by using a linear programming (LP) solver.
Indeed, for an independent sample y 1 , . . .
, y N ∼ µ , we can construct a quadrature formula withconvex weights exactly integrating the functions in F = span { φ 1 , . . .
, φ n − 1 } using a subset of thesepoints if and only if we have
[Equation]
where φ = ( φ 1 , . .
. , φ n − 1 ) ⊤ : X → R n − 1 and conv A denotes the convex hull of A .
Several sharpestimates for the probability of the event (26) are available in Hayakawa et al.
Under the event(26) , we can find a desired rule by using the simplex method for the LP problem (25) .
Remark 6.
From the viewpoint of subsampling, a direct way to obtain quadrature formulas withconvex weights supported on a small number of points, is to first sample N candidate points
D N = ( x 1 , . .
, x N ) and then solve the following sparse optimization problem:
[Equation]
where k ( D N , D N ) is the corresponding N × N Gram matrix.
Unfortunately, exactly solving thisproblem is computationally challenging, in particular in contrast to our approach that exploitsthe spectral properties of k and µ .
Nevertheless, one could use sparse optimization to obtainan approximate solution of (27) : although the simplex constraint ( w ≥ 0 , 1 ⊤ w = 1 ) makesit impossible to exploit the classical ℓ 1 regulatization, there are possible alternatives under thisconstraint [44, 29, 31] or use the DC (difference of convex functions) algorithm to incorporate thesparsity constraint to find a local minima [13].
This is a promising research direction, and ourgeneral sample estimates might provide a first step towards this direction.
Remark 7.
Sard’s method [46, 30] for constructing numerical integration rules uses the n degreeof freedom (of choosing weights in our setting) separately; m ( ≤ n ) for exactness over a certainm -dimensional space of test functions, and the remaining n − m for minimizing an error criterionsuch as the worst-case error.
In the context of kernel quadrature, one way to use Sard’s method withexactness over F (an m -dimensional space of test functions) is as follows [24, 48]:
[Equation]
subject
[Equation]
(28) where D n = ( x 1 , . . .
, x n ) ⊤ , f ( D n ) =
( f ( x 1 ) , . . .
, f ( x n )) ⊤ .
This amounts to solving a convexquadratic programming for w in an ( n − m ) -dimensional subspace of R n (without constraint).
Thisis similar to our approach in that it enforces exactness in a certain finite-dimensional space of testfunctions.
One key difference is that Sard’s approach aims for a quadrature formula on a given set ofpoints, whereas our method determines also the points themelves.
Hence, the combination of thesetwo approaches seems to be an interesting future research topic.
Computational complexity.
A tricky part of this approach, essentially based on random convex hulls, is that the algorithm possibly does not output a quadrature formula.
Hence, the following quantity plays an important role to estimate the essential complexity of the algorithm:
[Equation]
where y, y 1 , y 2 , . .
. are independent samples from µ .
This value is known to be finite and estimatedunder a variety of conditions on φ ( y ) [55, 18].
If we have some knowledge of µ , we can justkeep trying the algorithm with N = N φ until it succeeds, and its expected computational time is
O ( nN φ + C ( n, N φ )) , where C ( a, b ) is the (expected) cost of solving an a × b LP with a simplexmethod.
Note that, though the worst-case computational time of the simplex method is exponential, itis empirically O ( ab min { a, b } ) in practice [43, 47].
In addition, N φ = O ( n ) holds in examples withsome symmetry [57, 17], so in that case we have a heuristic complexity estimate of O (cid:0) n 3 (cid:1) .
Choice of approximate kernels.
Similarly to the empirical version discussed in the main text, weprove quantitative estimates when k 0 is given by the Mercer approximation or Nyström approximation.
Remark that the necessary information for using these methods is different.
Whereas using the Mercerapproximation requires the knowledge of Mercer decomposition k
( x, y ) = (cid:80) ∞
m =1 σ m e m ( x ) e m ( y )and their exact integration (cid:82)
X
e m ( x ) d µ ( x ) , the Nyström approximation only requires the exactintegral values of kernel, (cid:82) X
k ( x, y ) d µ ( y ) , and so is more generally applicable.
See the followingsections for details.
In the following, we assume that the kernel attains the Mercer decomposition k
( x, y ) = 727 (cid:80) ∞ m =1 σ m e m ( x ) e m ( y ) , where σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 is an orthonormal set of L 2 ( µ ) .
D.1 Algorithm 2 with Mercer Approximation
If we use the truncated Mercer decomposition as an approximate kernel, we have the following result.
Theorem 8.
If Algorithm 2 with k 0 :
= (cid:80) n − 1 m =1 σ m e m ( x ) e m ( y ) and φ i =
e i successfully outputs aconvex quadrature Q n , then it satisfies the following:
(a) If C := sup m ≥ 1 ∥ e m ∥ ∞ < ∞ , we have wce( Q n ; H k , µ ) 2 ≤ 4 C 2 (cid:80) ∞ m = n σ m .(b)
As N in Algorithm 2 tends to infinity, we have
[Equation]
Proof.
As in the proof of Corollary 2, k − k 0 is positive semi-definite.
Thus, when (cid:80) ∞ m = n σ m < ∞ ,the kernel and the measure µ satisfies Assumption A.
So Thorem 5(a) implies (a) of this theorem,since k 1 ( x, x ) = (cid:80) ∞ m = n σ m e m ( x ) 2 ≤ C 2 (cid:80) ∞ m = n σ m .
For (b), if we have Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) , then Theorem 5 implies
[Equation]
So it suffices to prove P ( Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ))
→ 1 as N → ∞ , and it is shown by consideringthe optimal basic feasible solution of the LP (25) and the following fact [18, Proposition 4]:
[Equation]
y N ) } ) → 1 , N → ∞ , where ψ = ( φ 1 , . .
φ n − 1 , k 1 , diag ) ⊤ .
Indeed, under the event ψ ( y ) ∈ conv { ψ ( y 1 ) , . .
. , ψ ( y N ) } ,the LP becomes feasible and Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) follows from the optimality.
See the proof of
Theorem 7 (Section C.4) for a more detailed explanation if necessary.
Note that the boundedness of C is a typical assumption [see 34, Assumption 3.2 and referencestherein], while it does not necessarily hold [37, Section 3].
Under some assumptions, we can quantifythe probability that the LP (25) becomes feasible.
Sampling bound.
Suppose 1 is an eigenfunction of K .
This is satisfied, e.g., in the following cases:
• µ is a Haar measure on a compact group and k is shift-invariant.
• k is a kernel based on Stein’s identity [40, 48, 1] with respect to µ .
In this case, we have a theoretical bound of the required N in Algorithm 2 as follows.
Theorem 9.
Suppose 1 is an eigenfunction of K , i.e., (cid:82) X k ( · , y )
d µ ( y ) is a constant function.
Then,for each n ≥ 2 and N ≥ 6( n − 1) sup x ∈X (cid:80) n − 1 m =1 e m ( x ) 2 , Algorithm 2 returns a feasible quadraturewith probability at least 1 − 2 1 − n , i.e., for an independent sample y 1 , . . .
, y N from µ , we have
P
[Equation]
where φ = ( e 1 , . .
, e n − 1 ) ⊤ .
If the value C = sup m ≥ 1 ∥ e m ∥ ∞ is finite, N ≥ 6 C ( n − 1) 2 is alsosufficient for the above estimate.
Proof.
This follows from the existing results [18, Theorem 14 and Proposition 17].
D.2 Algorithm 2 with Nyström Approximation
Although the method discussed in the previous section requires the knowledge of Mercer decomposition, if we make use of the Nyström approximation, we only require the values of (cid:82) X k ( x, y )
d µ ( y )for x ∈ X .
Recall that k Zs ( x, y ) is the rank- s Nyström approximation of the kernel k based on the point set
Z = ( z 1 , . . .
, z ℓ ) .
From (11), we can use φ Zi := u ⊤ i k
( Z, · ) as test functions.
Theorem 10.
Let n ≤ ℓ and δ > 0 , and let Z be an ℓ -point independent sample from µ .
If Algorithm 2with k 0 = k Zn − 1 and φ i = φ Zi successfully outputs a convex quadrature Q n , then with probability atleast 1 − δ , we have
[Equation]
Proof.
As in the proof of Corollary 4, k − k Zn − 1 is positive semi-definite.
Also, we can assume 766 (cid:80) ∞ m =1 σ m < ∞ , as otherwise the right-hand side is infinity.
Thus k and k 0 =
k Zn − 1 satisfy
Assumption A.
Note that for a function of the form c ( x, y )
= a · b ( x ) b ( y ) with a ∈ R and b ∈ L 2 ( µ ) , and anorthonomal set ( f i )
i ∈ I ⊂
L 2 ( µ ) of L 2 ( µ ) with b ∈ span { f i | i ∈ I } , we have
[Equation]
If we here use the orthonormal set ( e m ) ∞ m =1 that appears in the Mercer decomposition, by lettingk 1 :
= k − k Zn − 1 and using the linear extension of (29), we have
[Equation]
Then, by combining this with Theorem 3 and Theorem 5(c), we obtain the desired estimate.
Remark 8.
If we denote by N φ the required number of samples, the computational complexity ofthe above algorithm becomes O (cid:0) nℓN φ + nℓ 2 + C ( n, N φ ) (cid:1) , including the cost of computing the
Nyström approximation as well as test functions at N φ samples (see also Remark 2).
E Additional Numerical Experiments
In this section, we provide additional experiments on Algorithm 2 using random convex hulls, aswell as the approximated version of the N. + emp described in Remark 2.
Section E.1 showsthe comparison of Algorithm 2 (with Mercer/Nyström approximation) with some of the methodsmentioned in the main text under the periodic Sobolev spaces with uniform measure.
Section E.2investigates Algorithm 2 (with Nystöm approximation) as well as the approximate but fast algorithmfor N. + emp , under the setting of empirical measure reduction.
E.1 Periodic Sobolev Spaces with Uniform Measure
We conducted experiments under the same setting as in Section 3.1, except that we additionally havethe following methods:
Nyström, Nyström + opt: We used the same test functions as N. + emp with the random set Z ofsize ℓ = 10 n , but for Algorithm 2.
We used N = 10 n samples for the LP (25).
In Nyström + opt we additionally optimized the convex weights using (14). Mercer, Mercer + opt ( d = 1 ): We used the same test functions as M. + emp , but for Algorithm 2.We used N = 10 n samples for the LP (25).
In Mercer + opt we additionally optimized theconvex weights using (14).
The results are given in Figure 3.
The weights of Nyström and Mercer are already almost optimizedas they exactly integrate a certain family of functions, so the additional CQP (14) does not change theerror so much.
Surprising is that N. + emp + opt is almost as good as Nyström + opt or even better.
This implies that the recombination points with respect to a moderately large ( N = n 2 in this case)empirical measure can provide a good convergence rate in Bayesian quadrature [20], even though the(equally weighted) empirical measure itself is not that close to the true measure.
Odd behavior of ‘Mercer’.
As we can see in Figure 3(a,b), the methods based on the exact Mercerdecomposition becomes very close to optimal when n = 15 , 65 .
As it seemed to be caused by theparity of n , we carried out another experiment for n ∈ { 5 , 9 , 15 , 19 , 29 , 39 , 49 , 65 , 79 } (Figure 4),then Mercer and its optimization clearly became the best methods except the exact optimal Unifrom
Grid .
It might be related to the structure of the periodic Sobolev space, that has, for each eigenvalueexcept for 1 , two-dimensional eigenspace ( cos and sin ), but needs further investigation.
Also, in thecase ( d, r ) =
(1 , 3) , we see ‘ + opt ’ make the quadrature less accurate for a big n , but it is theoreticallyalmost impossible, so it seems to be caused by numerical accuracy of the CQP solver.
E.2 Measure Reduction in Machine Learning Datasets
We conducted experiments under the same setting as in Section 3.2.
We additionally adopted
Nyström, Nyström + opt (with N = 20 n ), and FNE, FNE + opt , where FNE (stands for ‘fast N.+ emp’) is the approximate algorithm for N. + emp by omitting the inequality in (6) and using therandomized SVD [15] (see Remark 2).
[Figure]
The worst computational time per one trial was 5 seconds of N. + emp and N. + emp + opt in ( d, r, n ) =
(3 , 3 , 80) , while Nyström and Nyström + opt ran in 0.9 seconds under the same setting.
There were no infeasible LPs
[Figure]
[Figure]
log 10 n         l o g
[Figure] [Figure] c e ) [Figure]
FNE
[Figure]
[Figure]
[Figure]
[Figure]
The results are given in Figure 5.
N. + emp + opt and FNE + opt show almost the same convergence.
While in the largest case n = 160 , the average runtime of ( N. + emp + opt , FNE + opt )was (13 . 0 , 2 . 07) seconds in 3D Road Network data and (12 . 8 , 2 .
) seconds in Power Plant data,respectively.
Although our theoretical guarantee no longer holds for FNE , it accelerates the algorithmwhile surprisingly maintaining the accuracy.
Nystöm or Nyström + opt behave much better than iid
Bayes , but are slightly less accurate than N. + emp + opt and FNE + opt , whereas they have goodtheoretical guarantees (Theorem 10).
Their computational time was basically between that of FNE +opt and N. + emp + opt .