Positively Weighted Kernel Quadrature via Subsampling
Abstract
We study kernel quadrature rules with convex weights.
Our approach combines the spectral properties of the kernel with recombination results about point measures.
This results in effective algorithms that construct convex quadrature rules using only access to i.i d. samples from the underlying measure and evaluation of the kernel and that result in a small worst-case error.
In addition to our theoretical results and the benefits resulting from convex weights, our experiments indicate that this construction can compete with the optimal bounds in well-known examples.
Introduction
The goal of numerical quadrature is to provide, for a given probability measure µ on a space X , a set of points x 1 , . . .
, x n ∈ X and weights w 1 , . .
w n ∈ R such that
[Equation]
holds for a large class of functions f : X → R .
Kernel quadrature focuses on the case when the function class forms a reproducing kernel Hilbert space (RKHS).
What makes kernel quadrature attractive, is that the kernel choice provides a simple and flexible way to encode the regularity properties of a function class.
Exploiting such regularity properties is essential when the integration domain X is high-dimensional or the function class is large.
Additionally, the domain X does not have to be Euclidean, but can be any topological space that carries a positive semi-definite kernel.
More formally, given a set Q := { ( w i , x i ) :
i = 1 , . .
. , n } ⊂ R ×
X denote with µ Q := (cid:80) ni =1 w i
δ x i the resulting measure on X .
We refer to Q [resp.
µ Q ] as a quadrature [resp.
quadrature measure], to the points x 1 , . . .
, x n as the support of Q [resp.
µ Q ].
The aim of kernel quadrature is to construct quadrature measures µ Q that have a small worst-case error
[Equation]
[Equation]
where H k denotes the RKHS associated with a positive semi-definite kernel k .
If the weights are positive and sum up to one, w i > 0 , (cid:80) ni
=
w i = 1 , then we refer to Q as a convex quadrature rule.
Contribution.
The primary contribution of this article is to leverage recombination (a consequence of Carathéodory’s Theorem) with spectral analysis of kernels to construct convex kernel quadrature rules and derive convergence rates.
We also provide efficient algorithms that compute these quadrature rules; they only need access to i.i.d.
samples from µ and the evaluation of the kernel k .
See Tablefor a comparision with other kernel quadrature constructions.
The table is written by using σ n and r n , which represent a sort of decay of the kernel with respect to µ .
Typical regimes are σ n ∼ n − β
(e.g. Sobolev) or σ n ∼ exp( − γn ) (e.g. Gaussian) depending on the ‘smoothness’ of the kernel [e.g. 15, 2] (see also Section B.3), and in such regimes (with β ≥ 2 or γ > 0 ), σ n or r n ( ≲ nσ n ) provide faster rates than wce 2 ∼ 1 /n of the usual Monte Carlo rate. For more examples including multivariate Sobolev spaces, see Bach [2, Section 2.3].
Limitation.
Our proposed methods are based on either Mercer or Nyström approximation.
Though our Mercer-based methods have strong theoretical bound, they require the knowledge of Mercer decomposition like [2, 6, 7], which is not available for general ( k, µ ) .
Our Nyström-based methods are practical and outperform existing methods in experiments, but the n/ √ ℓ term makes their theoretical bound far from competitive.
So further study is needed for truly bridging theory and practice.
[Table]
C
M
[Table] ) ✓
[Table]
[Table]
KT++ [ [Table]
[Table] 55] (1 /n [Table]
1 /N )
[Table] N ) N log 3 N ✓ ✓ ✓
[Table]
Mercer † r n nN φ +
[Equation]
n
[Equation]
N φ ) ✓ M. + empirical ‡
[Equation]
N nN
[Equation]
n ) ✓ ✓
Nyström † nσ n
[Equation]
n [Equation] ℓ nℓN [Equation]
✓ ✓
[Equation]
✓ ✓ ✓
Why Convex Weights?
There are several reasons why convex weights are preferable: (i) Positive Integral Operator: Kernel quadrature provides an approximation of the integration operator f (cid:55)→ I ( f ) =
(cid:82) f
( x ) d µ ( x ) .
Hence, a natural requirement is to preserve basic properties of this operator and positive weights preserve the positivity of this operator.
(ii) Uniform estimates and
Robustness: In applications, the RKHS H k may be mis-specified if a quadrature rule with negative weights is applied to a function f / ∈ H k , the approximation error (1) can get arbitrary bad; in contrast, a simple estimate shows that convex weights give uniform bounds, see Appendix B.4.
(iii) Iteration: Consider the m -fold product of quadrature formulas for approximating µ ⊗ m on X m .
This is a common construction for a multidimensional quadrature formulas (e.g., for polynomials) from one-dimensional formulas [60] or numerics for stochastic differential equations [41].
In doing so, working with a probability measure is strongly preferred, since otherwise the total variation of their m -fold product gets exponentially large as m increases ( ∥ µ ⊗ m ∥ TV =
∥ µ ∥ m TV ).
Related Literature.
Roughly speaking, there have been two approaches to kernel-based quadrature formulas: kernel herding and random sampling.
In kernel herding or its variants, the points ( x i ) ni = are found iteratively, typically based on the Frank–Wolfe gradient descent algorithm [9, 3, 24].
In the random sampling approach, ( x i ) ni =1 are sampled and subsequently the weights are optimized.
Generically, this results only in a signed measure µ Q but not a probability measure.
Bach [2] and
Belhadji et al.
[6] use the eigenvalues and the eigenfunctions of the integral operator K : f (cid:55)→
(cid:82) X k ( · , y ) f ( y ) d µ ( y ) to obtain a Mercer-type decomposition of k [58].
Bach [2] then uses the eigenvalues and eigenfunctions of K to define an optimized measure from which the points ( x i ) are i.i.d.
sampled.
This achieves a near optimal rate, but the exact sampling from this measure is usually unavailable, although for special cases, it can be done efficiently.
In contrast, Belhadji et al.
[6] proposes non-independent sampling based on the determinantal point process [DPP; 23].
These two papers also treat the more general quadrature problem that includes a weight function g ∈ L 2 ( µ ) , i.e., approximating (cid:82) X f ( x ) g ( x ) d µ ( x ) for f ∈ H k ⊂ L 2 ( µ ) , which we do not discuss in this paper.
There is another recently proposed practical method called kernel thinning [13, 14], which aims at efficient compression of empirical measures like our ‘+ empirical’ methods.
Its acceleration [55] makes it a competitive candidate in terms of compressing N ∼ n 2 points (‘KT++’ in Table 1).
We emphasize that the literature on kernel quadrature is vast, and the above distinction is only a rough dichotomy, see e.g. [11, 39, 8, 26, 47, 28, 27, 56].
Finally, [63] essentially performs recombination based on Fourier features for kernel density estimation, and our methods can algorithmically contribute, while the problem is different from kernel quadrature and the theory differs accordingly.
Outline.
Section 2 contains our main theoretical and methodological contribution.
Sectionprovides numerical experiments on common benchmarks.
The Appendix contains several extensions of our main result, proofs, and further experiments and benchmarks.
Main Result
Assume we are given a set 2 of n − 1 functions φ 1 , . .
φ n − 1 : X → R such that their linear combinations well approximate functions in H k .
Then our kernel quadrature problem essentially reduces to the construction of an n -point discrete probability measure µ Q n = (cid:80) ni
=1 w i
δ x i such that
[Equation]
A simple way to approximately construct this µ Q n is to first, sample N ≫ n points, ( y i )
Ni =1 , from µ such that their empirical measure, (cid:101) µ N = 1 N (cid:80) Ni =
δ y i , is a good approximation to µ in the sense that (cid:82) φ i d (cid:101) µ N ≈ (cid:82) φ i d µ for i = 1 , . . .
, n − 1 , and secondly, apply a so-called recombination algorithm (Remark 1) that takes as input ( y i )
Ni =1 and n functions φ 1 , . .
. , φ n − 1 and outputs a measure µ Q n = (cid:80) w i δ x i by selecting a subset ( x i ) ni
=1 of the points ( y i )
Ni =1 and giving them weights ( w i ) ni
= such that µ Q n is a probability measure that satisfies the equation (3) with µ replaced by (cid:101) µ N .
The challenging parts of this approach are (i) to construct functions φ 1 , . . .
, φ n − 1 that approximately span the RKHS H k for a small n ; (ii) to arrive at good quantitative bounds despite the (probabilistic) sampling error resulting from the use of the empirical measure (cid:101) µ N , and the function approximation error via φ 1 , . .
φ n − 1 .
To address (i) we look for functions such that
[Equation]
with some c i ≥ 0 .
Two classic ways to do this are the Mercer and Nyström approximations.
The remaining, item (ii) is our main contribution.
Theorem 1 shows that the worst-case error, (5), is controlled by the sum of two terms: the first term stems from the kernel approximation (4), the second term stems from the sample error.
Theorem 1.
Let µ be a Borel probability measure on X and k a positive semi-definite kernel on X such that (cid:82) X k ( x, x ) d µ ( x ) < ∞ .
Further, let n be a positive integer and assume k 0 is a positive semi-definite kernel on X such that
[Equation]
There exists a function KQuad such that if D N is a set of N i.i.d.
samples from µ , then Q n =
KQuad( D N ) is a random n -point convex quadrature that satisfies
[Equation]
where c k,µ := (cid:82) X
k
( x, x ) d µ ( x ) − (cid:82)(cid:82)
X×X k ( x, y ) d µ ( x ) d µ ( y ) .
Moreover, the support of Q n is a subset of D N and given functions φ 1 , . .
φ n − 1 ∈ L 1 ( µ ) with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } , Q n =
KQuad( D N ) can be computed with Algorithm 1 in
O (cid:0) nN + n 3 log( N/n ) (cid:1) computational steps.
The function KQuad is deterministic but since D N is random, the resulting quadrature rule Q n is random, hence also the resulting worst case error wce( Q ; H k , µ ) and the expectation in (5) denotes the expectation over the N samples in D N .
The theoretical part of Theorem 1 follows from more general results that we present and prove in Appendix (Theorem 6 for inequality, essentially done by comparing H k with H k 0 , and 7 for existence) while the algorithmic part of Theorem 1 is discussed in the following section.
This result covers our two main examples for the construction of k 0 , resp.
the choice of φ 1 , . .
φ n − 1 , that we discuss below and for which the error estimate gets quite explicit: the Mercer approximation and the Nyström approximation.
The former requires some knowledge about the spectrum of the kernel which is, however, known for many popular kernels; the latter works in full generality but yields worse convergence rates.
Finally, we emphasize that N and n in Theorem 1 can be chosen independently and we will see that from a computational point the choice N ∼ n 2 is preferable in which case (5) is faster rate than Monte Carlo, see also Table 1.
2.1 Algorithm
Input: A positive semi-definite kernel k on X , a probability measure µ on X , integers N ≥ n ≥ 1 , another kernel k 0 , functions φ 1 , . .
φ n − 1 on X with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } and a set D N of N i.i.d.
samples from µ .
Output:
ni
1 w i
1: Apply a Recombination Algorithm (Remark 1) with ψ = ( φ 1 , . .
. , φ n − 1 , k 1 , diag ) ⊤ , to the empirical measure 1 N (cid:80)
y ∈ D N δ y to obtain points { (cid:101) x 1 , . .
, (cid:101) x n +1 } ⊂ D N and weights v = ( v 1 , . . .
, v n +1 ) ⊤ ≥ 0 that satisfy 1 ⊤ v = 1 and ψ ( (cid:101) x ) v = 1 N (cid:80) Ni =1 ψ ( (cid:101) x i ) , where ψ ( (cid:101) x ) =
[ ψ ( (cid:101) x 1 ) , . . .
, ψ ( (cid:101) x n +1 )]
∈ R n × ( n +1) .
2: Apply SVD with the matrix
[ φ i − 1 ( y j )
ij ∈ R n × ( n +1) with φ 0 = 1 to find a nonzero vector u ∈ R n +1 such that A u = 0 and k 1 , diag ( (cid:101) x ) ⊤ u ≥3: Compute the smallest α ≥ 0 such that v − α u ≥ 0 and v j − αu j = 0 for some j 4: Return
ni
( v k − αu k ) k ∈
I and ( x i ) ni
=1 ← ( (cid:101) x k ) k
∈
I , where I = { 1 , . . . , n + 1 } \ { j }
Suppose we are given k 0 and φ 1 , . .
φ n − 1 ∈ L 1 ( µ ) with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } , and also
N independent samples µ denoted by D N =
( y 1 , . . .
, y N ) .
From Theorem 6 in Appendix, if we construct a convex quadrature Q n = ( w i , x i ) ni
=1 satisfying [Equation]
[Equation]
where φ = ( φ 1 , . .
φ n − 1 ) ⊤ and k 1 , diag ( x ) =
k ( x, x ) − k 0 ( x, x ) , it satisfies the bound (5).
For this problem, we can use the so-called recombination algorithms:
Remark 1 (Recombination) .
Given d − 1 functions (called test functions) and a probability measure supported on N > d points, there exists a probability measure supported on a subset of d points that gives the same mean to these d − 1 functions.
This follows from Carathéodory’s theorem and is known as recombination.
Efficient deterministic [37, 42, 60] as well as randomized [10] algorithms exist to compute the new probability measure supported on d points; e.g. deterministic algorithms perform the recombination, step 1, in O (cid:0) c φ N + d 3 log( N/d ) (cid:1) time, where c φ is the cost of computing all the test functions at one sample.
If each function evaluation is in constant time, c φ
= O ( d ) .
Let us briefly provide the intuition behind the deterministic recombination algorithms.
We can solve a problem of ‘reducing (weighted) 2 d points to d points in R d while keeping the barycenter’ by using a linear programming or its variant.
If we apply this to 2 d points each given by a barycenter of approximately N 2 d points, we can reduce the original problem (of size N ) to a problem of size d · N 2 d = N/ 2 .
So we can just continue the same procedure log 2 ( N/d ) times to obtain the desired measure.
Although the recombination introduced here only treats the equality constraints in (6) we can satisfy the remaining constraints just with n points by modifying it.
This is done in Algorithm 1.
What the algorithm is doing is as follows.
First, via recombination, find an ( n + 1) -point convex quadrature R n +1 =
( v i , y i ) n +1 i =1 that exactly integrate functions φ 1 , . .
. , φ n − 1 , k 1 , diag with regard to the empirical measure 1 N (cid:80) Ni =
δ y i .
Second, to reduce one point, find a direction ( − u in the algorithm) in the space of weights on ( (cid:101) x i ) n +1 i =1 that does not change the integrals of φ 1 , . .
φ n − 1 and the constant function 1 , and does not increase the integral of k 1 , diag .
Finally, move the weight from v to the above direction until an entry becomes zero, at v − α u .
Such an α ≥ 0 exists, as u must have a positive entry since it is a nonzero vector whose entries sum up to one.
Now we have a convex weight vector with at most n nonzero entries, so it outputs the desired quadrature satisfying (6).
2.2 Mercer Approximation
In this section and Section 2.3, we assume that k has a pointwise convergent Mercer decomposition k
( x, y ) = (cid:80) ∞
m =1 σ m e m ( x ) e m ( y ) with σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 ⊂ L 2 ( µ ) being orthonormal [58].
If we let K be the integral operator L 2 ( µ ) →
L 2 ( µ ) given by f (cid:55)→ (cid:82) X k ( · , y ) f
( y ) d µ ( y ) , then ( σ m , e m ) ∞ m =1 are the eigenpairs of this operator.
The first choice of the approximate kernel k 0 is just the trucation of Mercer decomposition.
Corollary 2.
Theorem 1 applied with k 0 ( x, y )
= (cid:80) n − 1 m =
σ m e m ( x ) e m ( y ) yields a random convex quadrature rule Q n such that
[Equation]
Proof.
It suffices to prove the result under the assumption (cid:82)
X k ( x, x ) d µ ( x ) = (cid:80) ∞ m =1 σ m < ∞ , as otherwise the right-hand side of (7) is infinity.
For k 1 := k − k 0 , we have that k 1 ( x, y )
=
(cid:80) ∞
m = n σ m e m ( x ) e m ( y ) and it is the inner product of Φ( x ) :
= ( √ σ m e m ( x )) ∞ m = n and Φ( y ) in ℓ 2 ( { n, n + 1 , . . . } ) and so positive semi-definite.
Thus k and k 0 satisfies the assumption of Theorem 1, and (cid:82) X k 1 ( x, x ) d µ ( x ) = (cid:80) ∞ m = n σ m applied to (5) yields the desired inequality.
2.3 Nyström Approximation
Although the Nyström method [67, 12, 33] is primarily used for approximating a large Gram matrix by a low rank matrix, it can also be used for directly approximating the kernel function itself which is how we use it.
Given a set of ℓ points Z
=
( z i ) ℓi =1 ⊂ X , the vanilla Nyström approximation of k ( x, y ) is given by
[Equation]
where P Z : H k → H k is a projection operator onto span { k ( · , z i ) } ℓi =1 .
In matrix notation, we have
[Equation]
where
W =
( k ( z i , z j )) ℓi,j =1 is the Gram matrix for Z and W + denotes its Moore–Penrose inverse.
For completeness, we explain the equivalence between (8) and (9) in Appendix B.5.
As k Z is an ℓ -dimensional kernel, there exists an ( ℓ + 1) -point quadrature formula that exactly integrates functions in H k Z .
For a quadrature formula, exactly integrating all the functions in H k Z is indeed equivalent to exactly integrating k ( z i , · ) for all 1 ≤ i ≤ ℓ , as long as the Gram matrix k
( Z, Z ) is nonsingular.
Proposition 1 in the Appendix provides bound for the associated worst case error.
From this viewpoint, the Nyström approximation offers a natural set of test functions.
The Nyström method has a further generalization with a low-rank approximation of k ( Z, Z ) .
Precisely speaking, by letting W s be the best rank- s approximation of W =
k ( Z, Z ) (given by eigendecomposition), we define the following s -dimensional kernel:
[Equation]
Let W =
U Λ U ⊤ be the eigendecomposition of W , where U =
[ u 1 , . . . , u ℓ ] ∈ R ℓ × ℓ is a real orthogonal matrix and Λ = diag
( λ 1 , . . .
, λ ℓ ) with λ 1 ≥ · · · ≥ λ ℓ ≥ 0 .
Then, if λ s > 0 we have
[Equation]
So we can use functions u ⊤ i k
( Z, · ) ( i = 1 , . . .
, s )
as test functions, which is chosen from a larger dimensional space span { k ( z i , · ) } ℓi =1 .
Although closer to the original usage of the Nystöm method is to obtain u ⊤ i k
( Z, · ) as an approximation of i -th eigenfunction of the integral operator K with Z appropriately chosen with respect to µ , we have adopted an explanation suitable for the machine learning literature [12, 33].
The following is a continuous analogue of Kumar et al.
[33, Theorem 2] showing the effectiveness of the Nyström method.
See also Jin et al.
[25] for an analysis specific to the case s = ℓ .
Theorem 3.
Let s ≤ ℓ be positive integers and δ > 0 .
Let Z be an ℓ -point independent sample from µ .
If we define the integral operator K Zs : L 2 ( µ ) → L 2 ( µ ) by f (cid:55)→ (cid:82) X k Zs ( · , y ) f ( y ) d µ ( y ) , then we have, with probability at least 1 − δ , in terms of the operator norm, ∥K
[Equation]
The proof is given in Appendix C.5.
By using this estimate, we obtain the following guarantee for the random convex quadrature given by Algorithm 1 and the Nyström approximation.
Corollary 4.
Let D N be N -point independent sample from µ and let Z be an ℓ -point independent sample from µ .
Theorem 1 applied with the Nyström approximation k
= k
Zn − 1 yields an random n -point convex quadrature rule Q n such that, with probability at least 1 − δ and k max := sup x ∈X k ( x, x ) ,
[Equation]
Proof.
From (11), k Z ( x, y ) − k Zn − 1 ( x, y )
= (cid:80) ℓi
= n λ − 1 i ( u ⊤ i k
( Z, x ))( u ⊤ i k
( Z, y ))
(ignore the terms with λ i = 0 if necessary), and it is thus positive semi-definite.
If we define P ⊥ Z : H k → H k as the projection operator onto the orthogonal complement of span { k ( · , z i ) } ℓi =1 , then, from (8), we also have k ( x, y ) − k Z ( x, y )
= (cid:10) P ⊥ Z k ( · , x ) , P ⊥ Z k ( · , y ) (cid:11)
H k , so k − k Z is also positive semi-definite.
In particular, k − k
Zn − 1 =
( k − k Z )
+ ( k Z − k Zn − 1 ) is positive semi-definite.
Also, it suffices to prove the result when (cid:80) ∞ m =1 σ m < ∞ , so we can now apply Theorem 1.
For k 1 := k − k
Zn − 1 , we prove the inequality (cid:82) X k 1 ( x, x ) d µ ( x ) = (cid:80) ∞ m =1 (cid:10) e m , ( K − K Zs ) e m (cid:11) L 2 ≤ ( n − 1) ∥K − K Zs ∥ + (cid:80) m ≥ n σ m (see (30) in Appendix D.2 for details), and the desired inequality follows by combining Theorem 1 and Theorem 3 (i.e., (5) and (12)).
Remark 2.
Algorithm 1 with the Nystöm approximation can be decomposed into two parts: (a)
Nystöm approximation by truncated singular value decomposition (SVD) (the first n − 1 eigenvectors from an ℓ -point sample), (b) Recombination from an N -point empirical measure.
The complexity of (a) is O (cid:0) nℓ 2 (cid:1) , and it can also be approximated by randomized SVD in O (cid:0) n 2 ℓ + ℓ 2 log n (cid:1)
The cost of part (b) is O (cid:0) nℓN + n 3 log( N/n ) (cid:1) , where nℓN stems from the evaluation of k 1 , diag for all N sampling points.
If we do not impose the inequality constraint regarding k 1 , diag , which still works well in practice, the cost of part (b) becomes O (cid:0) ℓN + n 2 ℓ log( N/n ) (cid:1) , by using the trick 1 N (cid:80) Ni = [Equation]
U ⊤ n − [Equation] k ( Z, y i ) =
U ⊤ n − 1 1 N (cid:80) Ni
=1 k ( Z, y i ) , where U n − 1 =
[ u 1 , . . .
, u n − 1 ] ∈ R ℓ × ( n − 1) is a truncation of the matrix that appears in the Nyström approxiamtion (10,11).
So the overall complexity is O (cid:0) nℓN + nℓ 2 + n 3 log( N/n ) (cid:1) while an approximate algorithm (randomized SVD, without the inequality constraint) runs in O (cid:0) ℓN + ℓ 2 log n + n 2 ℓ log( N/n ) (cid:1) .
Numerical Experiments
In this section, we compare our methods with several existing methods.
In all the experiments, we used the setting where we can compute (cid:82) X k ( x, y )
d µ ( y ) for x ∈ X and (cid:82)(cid:82) X×X
k
( x, y ) d µ ( x ) d µ ( y ) since then we can evaluate the worst-case error of quadrature formulas explicitly.
Indeed, if a quadrature formula Q n is given by points X
= ( x i ) ni
=1 and weights w = ( w i ) ni
=1 , then we have wce( Q n ; H k , µ ) 2 = w ⊤ k ( X, X ) w − 2 E y [ w ⊤ k ( X, y )] + E y,y ′
[ k ( y, y ′ )] (13) for independent y, y ′ ∼ µ under (cid:82) X (cid:112) k
( x, x ) d µ ( x ) < ∞ , which is a well-known formula for the worst-case error [18, 57].
An essential remark shown in Huszár and Duvenaud [24] is that the
Bayesian quadrature [48] with covariance kernel k given observation at points ( x i ) ni
=1 (automatically) estimates the integral as (cid:80) ni
=
w i f ( x i ) with ( w i ) ni
=1 minimizing the above expression.
Once given points ( x i ) ni
=1 and additional knowledge of expectations, we can compute the optimal weights ( w i ) ni
=1 by solving a convex quadratic programming (CQP), either without any restrictions or with the condition that ( w i ) ni
=1 is convex.
Although the former can be solved by matrix inversion, we have used the optimizer Gurobi 3 for both CQPs to avoid numerical instability.
For the recombination part, we have modified the Python library by Cosentino et al.
[10] implementing the algorithm of [60].
Our theoretical bounds are close to optimal in classic examples and we see that the algorithm even outperforms the theory in practice especially in Section 3.1.
We also execute a measure reduction of a large discrete measure in terms of Gaussian RKHS and our methods shows a fast convergence rate in two ML datasets in Section 3.2.
3.1 Periodic Sobolev Spaces with Uniform Measure
For a positive integer r , consider the Sobolev space of functions on [0 , 1] endowed with the [-norm(cid:107) f (cid:107) 
-]norm ∥
f ∥ 2 = ( (cid:82) 1 0 f ( x ) d x ) 2 +(2 π ) 2 r (cid:82) 1 0 f ( r ) ( x ) 2 d x , where f and its derivatives f (1) , . . .
f ( r ) are periodic (i.e., f (0) =
f (1) and so forth).
This function space can be identifies as the RKHS of the kernel
[Equation]
for x, y ∈ [0 , 1] , where B 2 r is the 2 r -th Bernoulli polynomial [65, 2].
If we let µ be the uniform measure on [0 , 1] , the normalized eigenfunctions (of the integral operator) are 1 , c m ( · ) = √ 2 cos(2 πm · ) and s m ( · ) = √ 2 sin(2 πm · ) for m = 1 , 2 , . . .
, and the corresponding eigenvalues are 1 and m − 2 r (both for c m and s m ).
Although the rectangle formula f (cid:55)→ n − 1 (cid:80) ni =1 f ( i/n ) (a.k.a.
Uniform
Grid below) is known to be optimal for this kernel [68, 46] in the sense of worst-case error, this
RKHS is commonly used for testing the efficiency of general kernel quadrature methods [2, 6, 27].
We also consider its multivariate extension on [0 , 1] d , i.e., the RKHS given by the product kernel k ⊗ d r ( x , y ) :
= (cid:81) di =1 k r ( x i , y i ) for x = ( x 1 , . . .
, x d ) , y =
( y 1 , . .
. , y d ) ∈ [0 , 1] d .
We carried out the experiment for ( d, r ) =
For each ( d, r ) , we compared the following algorithms for n -point quadrature rules with n ∈ { 4 , 8 , 16 , 32 , 64 , 128 } .
N. + emp, N. + emp + opt: We used the functions u ⊤ i k
( Z, · ) ( i = 1 , . .
. , n − 1 ) given by the Nyström approximation (11) with s = n − 1 as test functions φ 1 , . .
φ n − 1 in Algorithm 1.
The set Z was given as an ( ℓ =)10 n -point independent sample from µ .
We used N = nsamples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (13)
M. + emp, M. + emp + opt ( d = 1 ): We used the first n − 1 functions of the sequence of eigenfunctions 1 , c 1 , s 1 , c 2 , s 2 , . . .
as test functions φ 1 , . . .
, φ n − 1 in Algorithm 1.
We used N = nsamples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (13).
Monte Carlo, iid Bayes: With an n -point independent sample ( x i ) ni =1 from µ , we used uniform weights 1 /n in Monte Carlo and the weights optimized using (13) in iid Bayes .
Uniform Grid ( d = 1 ): We used the rectangle formula f
(cid:55)→ n − 1 (cid:80) ni
=1 f ( i/n ) .
This is known to be optimal [not just up to constant, but exactly; 68, 46], and thus equivalent to the Bayesian quadrature on the uniform grid, i.e., the weights are already optimized.
Halton, Halton + opt ( d ≥ 2 ): For an n -point sequence given by the Halton sequence with Owen scrambling [20, 49], the uniform weights w i = 1 /n is adopted in Halton and the weights are additionally optimized using (13) in Halton + opt .
Thinning, Thin + opt:
Given an N -point independent sample ( y i )
Ni
=1 with N = n 2 from µ , an n -point subset ( x i ) ni
=1 taken from a KT++ algorithm (kernel thinning [13, 14] combined with Compress++ algorithm [55] with the oversampling parameter g = min { 4 , log 2 n } , implemented with GoodPoints package: https://github.com/microsoft/goodpoints ) is adopted in Thinning .
In ‘+ opt’ we additionally optimized the convex weights using (13).
[Figure]
The worst computational time per one trial was 57 seconds of Thin + opt in ( d, r, n ) = (3 , 3 , 128) , where Thinning was 56 seconds and N. + emp
[+ opt] was 22 seconds.
The results are given in Figure 1.
In d = 1 , the optimal rate given by Uniform Grid is known to be O (cid:0) n − 2 r (cid:1) .
As the uniform sampling is equal to the optimized distribution of Bach [2] in this case, iid Bayes also achieves this rate up to log factors.
Although our theoretical guarantee for
M. + emp is O (cid:0) n 1 − 2 r + N − 1 (cid:1) with N = n 2 (Corollary 2), in the case ( d, r ) =
(1 , 1) , we can observe that in the experiment it is better than iid Bayes and close to the optimal error of Uniform
Grid , but slightly worse than Thinning .
Moreover, N. + emp , which does not use the information of spectral decomposition, is remarkably almost as accurate as M. + emp in d = 1 .
Furthermore, if we additionally use the knowledge of expectations, which iid Bayes is already doing, M./N. + emp + opt become surprisingly accurate even with N = n 2 .
They are worse than Thinn + opt when r = 1 , but well outperform it when r = 3 .
Nonlineality in the graph of these methods when ( d, r, n ) =
(1 , 3 , 128) should be from numerical accuracy of the CQP solver (see also Section E.1).
The accuracy of N. + emp + opt becomes more remarkable in multivariate cases.
It behaves almost the same as Halton + opt in d = 2 and clearly beats it in d = 3 .
Also, the sudden jump of our methods around n = 30 in ( d, r ) = (3 , 3) seems to be caused by the jump of eigenvalues.
Indeed, for the integral operator given by k ⊗ 3 3 with uniform measure, the eigenspace of the largest eigenvalue 1 is of dimension 27 , and the next largest eigenvalue is 1 / 64 .
Again in the latter case, N. + emp + opt outperforms Thin + opt , and these results suggest that our method works better when there is a strong spectral decay, as is explicitly incorporated in our algorithm.
Note also that we can compare Figure 1 with Belhadji et al.
[6, Figure 1] which includes some other methods such as DPPs, herding and sequential Bayesian quadrature, as we did experiments under almost the same setting.
In particular, in the case ( d, r ) = (1 , 3) where the eigenvalue decay is fast, we see that our method substantially outperforms the sequential Bayesian quadrature.
3.2 Measure Reduction in Machine Learning Datasets
We used two datasets from UCI Machine Learning Repository ( https://archive.ics.uci.edu/ ml/datasets/ ).
We set µ as the equally weighted measure over (a subset of) the data points
X =
( X (1) , . . .
, X ( d ) ) ⊤ ( d = 3 , 5 , respectively), where each entry is centered and normalized.
We considered the Gaussian kernel exp( −∥ x − y ∥ (2 λ 2 ))
whose hyperparameter λ is determined by median heuristics [16], and compared the performance of N. + emp , N. + emp + opt (with ℓ = 10 n ,
N = n 2 ), Monte Carlo , iid Bayes , Thinning , Thin + opt .
We also added Herding , an equally weighted greedy algorithm with global optimization [9], and its weight optimization Herd + opt within convex quadrature given by (13).
We conducted the experiment for n ∈ { 4 , 8 , 16 , 32 , 64 , 128 } .
The first is 3D Road Network Data Set [30].
The original dataset is 3 -dimensional real vectors at 434874 points.
To be able to compute the worst-case error (13) efficiently to evaluate each kernel quadrature, we used a random subset X of size 43487 = ⌊ 434874 / 10 ⌋ (fixed throughout the experiment) and defined µ as the uniform measure on it.
We determined λ with the median heuristic by using a random subset of X with size 10000 and used the same X and λ throughout the experiment.
The second is Combined Cycle Power Plant Data Set [31, 62].
The original dataset is 5 -dimensional real vectors at 9568 points.
We set the whole data as X and defined µ as the uniform measure on it.
We determined λ with median heuristics by using the whole X .
[Figure]
Figure 2 shows the results.
We can observe that in both experiments N. + emp + opt successfully exploits the fast spectral decay of Gaussian kernel and significantly outperforms other methods.
Also, even without using the knowledge of any expectations, N. + emp (and Thinning ) show a decent convergence rate comparable to Herding or iid Bayes , which actually use the additional information. See the
Concluding Remarks
We leveraged a classical measure reduction tool, recombination, with spectral properties of kernels to construct kernel quadrature rules with positive weights.
The resulting algorithms show strong benchmark performance despite their restriction to convex weights.
Although our method is applicable to fairly general situation, the usage or performance can be limited when it is difficult or inefficient to directly sample from the target measure µ .
So there are interesting follow up questions; for example, one could study if one can replace the i i d. samples with smarter sampling (DPP, importance sampling etc) before the recombination is carried out.
Also, the current theory does not fully explain the empirical superiority; especially the 1 / √ ℓ term does not match the experiments.
Nevertheless, we believe our method is the first generally applicable algorithm with a guarantee from the spectral decay, while the issue with ℓ should be addressed in future work.
Acknowledgments and Disclosure of Funding
The authors would like to thank Chris Oates and Toni Karvonen for helpful remarks and discussions.
The authors are also grateful to anonymous reviewers for detailed and constructive discussions that improved the paper.
Harald Oberhauser and Terry Lyons are supported by the DataSıg Program [EP/S026347/1], the Alan Turing Institute [EP/N510129/1], the Oxford-Man Institute, the Hong Kong
Innovation and Technology Commission (InnoHK Project CIMDA) and the Oxford-Man Institute.
A Outline of the Appendix
Appendix B contains general results from which the results presented in the main text, in particular
Theorem 1, follow as special cases.
Appendix C contains the proofs of these theoretical results and needed technical lemmas.
Appendix D shows that if the expectations (cid:82) φ i ( x ) d µ ( x ) are known, then this knowledge can be used to further improve the theoretical bounds; it also gives a simple modification of Algorithm 1 doing this efficiently.
Appendix E provides additional numerical experiments and benchmarks.
B Theoretical Results and Remarks
In this section, we present theoretical results that include our main results as a special case.
The proofs are given in Section C.
Notation.
For simplicity, for a quadrature Q n given by points ( x i ) ni
=1 and weights ( w i ) ni
=1 and a probability measure µ , we denote the integration of an integrable function f on X with respect to these measures by
[Equation]
respectively.
We also write the inner-product and norm of an RKHS [Equation] k by ⟨· , ·⟩ H k and ∥·∥ H k .
Furthermore, we use the probability simplex ∆ n and convex hull conv A of a set A ⊂ R d in the proofs:
[Equation]
B.1 Quantitative Results
We work under the following setting as in the assumption of Theorem 1.
Assumption A. µ is a Borel probability measure on X , and k is a positive semi-definite kernel on
X such that (cid:82) X
k
( x, x ) d µ ( x ) < ∞ .
Further, k 0 is a positive semi-definite kernel on X such that k 1 :
=
k − k 0 is a positive semi-definite kernel on X .
The following is a general result regarding a quadrature formula exactly integrating functions in H k 0 .
Theorem 5.
Under Assumption A, if an n -point convex quadrature Q n on X satisfies Q n ( f ) = µ ( f ) for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g is the function given by g ( x ) = (cid:112) k 1 ( x, x ) .
In particular, the following assertions hold for such a quadrature Q n :
(a) We have wce( Q n ; H k , µ ) ≤ 2 ∥ g ∥ ∞
= 2 sup x ∈X (cid:112) [Equation] 1 ( x, x ) .
(b) If we additionally have Q n ( g ) ≤ µ ( g ) , then we have wce( Q n ; H k , µ ) ≤ 2 µ ( g ) .
(c) If we additionally have Q n ( g 2 ) ≤ µ ( g 2 ) instead of (b), we still have
[Equation]
Remark 3.
For a Borel probability measure ν on X and a nonnegative function h : X → R ≥ 0 , we have an inequality (cid:82) X (cid:112) h ( x ) d ν ( x ) ≤ (cid:0)(cid:82)
X h ( x ) d ν ( x ) (cid:1) 1 / 2 , so the above µ ( g ) can be upper bounded by (cid:82) X k 1 ( x, x ) d µ ( x ) , which is equal to the trace of the integral operator given by k 1 .
Also, the assumption in Theorem 5 can be weakened to the integrability of (cid:112)
We can combine Theorem 5 with an empirical approximation of µ to obtain the following result, which essentially implies Theorem 1.
Theorem 6.
Under Assumption A, let D N be a set of N independent samples from µ , and (cid:101) µ N be its empirical measure, i.e., (cid:101) µ N = 1 N (cid:80)
y ∈ D N δ y .
Then, if an n -point convex quadrature Q n on X satisfies Q n ( f ) = (cid:101) µ N ( f ) for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g ( x ) := (cid:112) k 1 ( x, x ) and c k,µ :
= (cid:82) X
k
( x, x ) d µ ( x ) − (cid:82)(cid:82)
X×X k ( x, y ) d µ ( x ) d µ ( y ) .
In particular, the following assertions hold for such a quadrature Q n :
(a) We have E (cid:2) wce( Q n ; H k , µ ) 2 (cid:3) ≤ 8 sup x ∈X
[Equation]
k
[Equation]
(b) If we additionally always require Q n ( g ) ≤ (cid:101) µ N ( g ) , then we have
[Equation]
) 2 (cid:3) ≤
[Equation]
(cid:90)
[Equation]
The requirement Q
n ( g
) ≤ (cid:101) µ N ( g ) can be replaced by Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 )
Although we have assumed k − k 0 is positive semi-definite in the previous assertions, the uniform bound works without the assumption as follows.
Proposition 1.
Let µ be a Borel probability measure on X .
Let k and k 0 be positive semi-definite kernels on X satisfying (cid:82) X (cid:112) k
( x, x ) d µ ( x ) , (cid:82) X (cid:112) k
( x, x ) d µ ( x ) < ∞ .
If an n -point convex quadrature Q n on X satisfies Q n ( f ) = µ ( f ) for any k 0 ( · , x ) with x ∈ X , we have
[Equation]
Furthermore, if dim H k 0 < n , there exists an n -point convex quadrature Q n satisfying Q n ( f ) = µ ( f ) for each f = k 0 ( · , x ) .
In this paper, we focus on the cases where k 0 is either given by the truncated Mercer decomposition or
Nyström approximation.
For many important kernels, however, we may also use the random Fourier features [52] or its periodic version [61] which can easily be combined with Proposition 1, but it is beyond the scope of this paper to choose its appropriate variant for each kernel [see 38, for a list of variants].
B.2 Existence Results
The existence of quadrature formulas satisfying the estimate of Theorem 5 or Theorem 6 is guaranteed when dim H k 0 < n .
Theorem 7.
Under Assumption A, if dim H k 0 < n , there exists an n -point convex quadrature Q n satisfying Q n ( f ) = µ ( f ) for each f = k 0 ( · , x ) .
This still holds even if we additionally require
Q n ( g ) ≤ µ ( g ) or Q n ( g 2 ) ≤ µ ( g 2 ) for g ( x ) =
(cid:112) k 1 ( x, x ) .
Remark 4.
This also implies the existence result of Q n satisfying the condition in Theorem 6 if we replace µ by (cid:101) µ N .
The algorithm for constructing a kernel quadrature with Theorem 6 is given in the main body, see Algorithm 1.
The ones with Theorem 5 requires further knowledge of the expectation of test functions, i.e., the values of (cid:82) X φ i ( x ) d µ ( x ) with H k 0 ⊂ span { φ 1 , . .
φ n − 1 } .
Under this additional information, we have an algorithm (Algorithm 2) based on random sampling given in the following section.
B.3 Eigenvalue estimate for Gaussian kernels
We provide a proof of a folklore estimate on the eigenvalues of integral operators given by a Gaussian kernel.
Let k ( x, y ) =
exp( − 12 ℓ 2 ( x − y ) 2 ) for an ℓ > 0 and x, y ∈ R .
Then, it has the following expansion [43, 29]:
[Equation]
Let µ be a Borel probability measure supported on a compact domain, i.e., µ ( { x ∈ R | | x | > R } )
=for some R > 0 .
Let us consider the RKHS given by k over X := supp µ .
Recall that σ n is the n -th eigenvalue of the integral operator
[Equation]
From the minimax property of eigenvalues of compact Hermitian operators, we have
[Equation]
where the orgthogonal complement is taken in terms of L 2 ( µ ) -inner product and φ m are functions given in (16).
They are indeed in L 2 ( µ ) as µ is compactly supported.
Now, let k n ( x, y ) :
= (cid:80) ∞
m = n − 1 φ m ( x ) φ m ( y ) .
For an f ∈ L 2 ( µ ) ∩ { φ 0 , . . .
, φ n − 2 } ⊥ , we have ⟨ f,
[Equation]
f ⟩ [Equation] [Equation] =
[Equation]
Therefore
[Equation]
we have the estimate σ n ≤ (cid:82) X k n ( x, x ) d µ ( x ) .
We have
[Equation]
and this can be regarded as the remainder term of the Maclaurin expansion [Equation] so there is a θ ∈ (0 , 1) such that [Equation]
In particular
[Equation]
if we have | x | ≤ R for µ -almost all x , we have a factorial decay σ n ≤ ( R [Equation] [Equation] n
[Equation]
n [Equation]
B.4 Uniform Robustness
In applications [Equation] the RKHS H k may be mis-specified and the quadrature rule µ Q when computed for the mis-specified function class H k but applied to a function f / ∈
H
k leads only to the attainable bound
[Equation]
via triangle equality and standard integral estimates.
Note that | · | TV denotes the total variation norm and the above applies to any (cid:101)
f ∈ H k ; in particular, to the best approximation in uniform norm to f in H k .
Since µ is a probability measure, | µ | TV = 1 but if µ Q is a signed measure with non-convex weights, its total variation | µ Q | TV can be large, resulting in arbitrary large integration errors.
B.5 Equivalence between the
Let k be a positive semi-definite kernel on X , Z = ( z i ) ℓi =1 ⊂ X .
Let P Z : H k
→ H k be the projection operator onto span { k ( · , z i ) | i = 1 , . .
. , ℓ } .
For arbitrary x, y ∈ X , we can write
P Z k ( · , x ) = ℓ (cid:88) i = a i k ( · , z i ) ,
P Z k ( · , y ) =
ℓ (cid:88) i =
b i k ( · , z i ) , where a = ( a i ) ℓi =1 , b = ( b i ) ℓi =1 ∈ R ℓ .
From the properties of projection, we have
[Equation]
In matrix notation, we have k ( Z, x ) =
k ( Z, Z ) a , and k ( Z, y )
=
k ( Z, Z ) b from the same argument.
Thus, by conbining it with the property of Moore–Penrose inverse, we have
[Equation]
This shows the desired equivalence.
C Proofs C.1 Proof of Theorem
Before proceeding to the proof of the theorem, we prepare a couple of assertions.
The following is a well-known estimate proven by using the Cauchy–Schwarz inequality [see e.g., 45, Lemma 3.1 and its proof].
Proposition 2.
Let k be a positive semi-definite kernel on X , and ν be a Borel probability measure with (cid:82) X (cid:112)
k
( x, x ) d ν ( x ) < ∞ .
Then, for each f ∈ H k , we have
[Equation]
By using the proposition, we obtain the following technical lemma.
Lemma 1.
Let k and k 1 be a positive semi-definite kernels on X such that k − k 1 is also positive semi-definite.
Let ν be a Borel probability measure on X .
Then, for any n ≥ 1 , a 1 , . . .
, a n ∈ R , x 1 , . . .
, x n ∈ X , if we let f = (cid:80) ni
=1 a i k ( · , x i ) and f 1 = (cid:80) ni
=1 a i k 1 ( · , x i ) , then we have
[Equation]
Proof.
From the positive semi-definiteness of k 0 :
= k − k 1 , we have
[Equation]
Hence it suffices to prove | ν ( f 1 ) | ≤ ∥ f 1 ∥
H k 1 ν ( g ) for g ( x ) := (cid:112) k 1 ( x, x ) , but it directly follows from Proposition 2.
Proof of Theorem 5.
Note first that, for each f ∈ H k 0 , f is integrable with respect to µ .
Indeed, we have
[Equation]
and it is integrable from assumption, so the equality Q n ( f )
= µ ( f ) with f = k 0 ( · , x ) is attained at a finite value.
Once we establish (14), the item (b) is clear, and (a) follows from the fact that Q n ( g ) and µ ( g ) are both integrals of the function g with respect to a probability measure.
Also, (c) is justified as follows:
[Equation]
where µ ( Q n ) 2 ≤ µ ( Q 2 n ) and µ ( g ) 2 ≤ µ ( g 2 ) follows from the Cauchy–Schwarz.
To prove (14),we first prove
[Equation]
for any f of the form f = (cid:80) ni =1 a i k ( · , x i ) with n ≥ 0 and a 1 , . .
. , a n ∈ R .
Given such an f , we have Q n ( f 0 ) =
µ ( f 0 ) for f 0 :=
(cid:80) ni
=1 a i k 0 ( · , x i ) from the assumption.
Thus, by letting f 1 := f − f [-
-]0 = (cid:80) ni
=1 a i k 1 ( · , x i ) , we have
[Equation]
As we have | ν ( f 1 ) | ≤ ∥ f
∥ H k ν
( g ) for ν = Q n , µ from Lemma 1, we obtain | Q n ( f 1 ) − µ ( f 1 ) | ≤ ∥ f ∥ H k ( Q n ( g ) + µ ( g )) , and so (17) is shown for f of the form f = (cid:80) [Equation]
i k ( · [Equation] x i ) .
Finally, we generalize (17) to any f ∈ H k .
Let (cid:101) f
∈
H k can be written in the form (cid:80) ni
=1 a i k ( · , x i ) .
If we let h ( x ) = (cid:112) k
( x, x ) , from Proposition 2, we have
[Equation]
Note that µ ( h ) < ∞ follows from the integrability of k ( x, x ) in Assumption A.
Therefore, we have
[Equation]
As we can make ∥ f − (cid:101) f ∥ H k arbitrarily small from the definition of H k , the proof of (14) is completed by taking the limit.
C.2 Proof of Theorem
Proof.
Denote D N = { y 1 , . . .
, y N } and note that the result follows from (15) and
[Equation]
where the first inequality is given by the Cauchy–Schwarz.
Indeed, (a) is an immediate consequence of (15) and Q n and (cid:101) µ N making a probability measure, and (b) is obtained as 2 E (cid:2) [Equation] Q n [Equation]
g )
+
[Equation]
N
[Equation]
g )
[Equation]
[Equation]
≤
[Equation] E [Equation] µ N ( g ) 2 (cid:3) ≤ 8 (cid:82) X k 1 ( x, x ) d µ ( x ) by using (18) and the requirement Q n ( g ) ≤ (cid:101) µ N ( g ) .
When the requirement is Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 ) , as we have Q n ( g ) 2 ≤ Q n ( g 2 ) and (cid:101) µ N ( g ) 2 ≤ (cid:101) µ N ( g 2 ) by the Cauchy–Schwarz, we also have by the AM–GM,
[Equation]
For showing (15), we remark that we always have
[Equation]
by applying Theorem 5 with (cid:101) µ N instead of µ .
Let h ( µ ) , h ( (cid:101) µ N ) , h ( Q n ) ∈
H
k be the kernel mean embeddings of µ , (cid:101) µ N and µ Q n , i.e.,
[Equation]
n
[Equation]
i
[Equation]
where ( w i ) ni
=1 and ( x i ) ni
=1 are weights and points defining the quadrature Q n .
Remark that h ( µ ) is well-defined as (cid:82) X k ( x, x ) d µ ( x ) < ∞ [45, Lemma 3.1].
As we can rewrite the worst-case error as wce
[Equation]
by triangle inequality and the AM–GM, we obtain
[Equation]
where we have used (19) in the last inequality.
It now suffices to prove E (cid:2) ∥ h ( µ ) − h ( (cid:101) µ N ) ∥ 2 H k (cid:3) =
c k,µ /N for showing (15).
Indeed, we have
[Equation]
since (cid:82)(cid:82) X×X k
( x, y ) d µ ( x ) d µ ( y ) =
(cid:82) X E [ k ( x, y i )]
d µ ( x ) =
E [ k ( y i , y j )] holds for i ̸
= j .
Thus, the proof is completed.
C.3 Proof of Proposition
Proof.
As Q n exactly integrates the functions in H k 0 , we have wce( Q n ; H k 0 , µ ) = 0 .
So, if we set
Q n ( f ) =
(cid:80) ni
=
w i f ( x i ) , then we have, from (13) with kernel k 0 ,
[Equation]
If we extract this from the formula (13) for the kernel k , we have, by letting k 1 :=
k − k 0 ,
[Equation]
So, if we define M := sup x ∈X | k 1 ( x, y ) | = sup x ∈X | k ( x, y ) − k 0 ( x, y ) | , we have wce( Q n ; H k , µ ) 2 ≤
[Equation]
as Q n is a convex quadrature.
The existence follows from almost the same proof as in the proof of
Theorem 5, but in this case it directly follows from Tchakaloff’s thorem [59, 4].
C.4 Proof of Theorem
Proof.
We prove the existence of the version Q n ( g ) ≤ µ ( g ) .
The other follows just by replacing every g in the proof below by g 2 .
Let φ 1 , . .
φ n − 1 ∈ H k 0 satisfy H k
= span { φ 1 , . .
φ n − 1 } .
Also, let y, y 1 , y 2 , . .
. be independent samples from µ .
Now, consider the vector-valued function ψ = ( φ 1 , . .
φ n − 1 , g ) ⊤ ∈ R n .
Note that E [ ∥ ψ ( y ) ∥ ] < ∞ follows from the integrability of elements in H k 0 and g with respect to µ .
Therefore, by [21, Theorem 11], with probability 1 , there exists an N such that E [ ψ ( y )]
∈ conv { ψ ( y 1 ) , . .
, ψ ( y N ) } .
So, in particular, there exist deterministic points x 1 , . . .
, x N ∈ X satisfying E [ ψ ( y )]
∈ conv { ψ ( x 1 ) , . .
, ψ ( x N ) } .
For such ( x i ) Ni =1 , consider an optimal solution that is also a basic feasible solution of the following linear programming problem:
[Equation]
where φ = (1 , φ 1 , . .
. , φ n − 1 ) ⊤ ∈ R n is another vector-valued function (note that its first coordinate is constant so that any feasible solution of (21) sums up to one).
Such a basic solution w has at most n nonzero entries, say ( w i 1 , . .
, w i n ) ∈ ∆ n with 1 ≤ i 1 < · · · < i n ≤ N .
Then, the quadrature Q n given by weights ( w i j ) nj =1 and points ( x i j ) nj =1 satisfies Q n ( φ ) = µ ( φ ) and Q n ( g ) ≤ µ ( g ) .
The latter follows from the optimality of w and the fact that E [ ψ ( y )] ∈ conv { ψ ( y 1 ) , . . .
, ψ ( y N ) } leads to a feasible solution with the objective E [ g ( y )]
= µ ( g ) ).
C.5 Proof of Theorem
We prove the theorem by using an existing bound regarding the Nystöm approximation for matrices, which is more common in the machine learning literature.
Let A =
( A ij ) Ni,j =1 ∈ R N × N be a symmetric positive semi-definite matrix.
Let us denote it as
A =
[ a 1 , . .
. , a N ] by using a 1 , . .
. , a N ∈ R N .
Then, we independently sample i 1 , . .
, i ℓ from { 1 , . . .
, N } uniformly, and construct a submatrix B
=
( a i j i k ) ℓj,k =1 .
If we let B s be the best rank- s approximation of B and B + s its pseudoinverse, the matrix
[Equation]
works as a rank- s approximation of A .
We use the following result on this matrix version:
Proposition 3 ([33, Theorem 2]) .
For a positive semi-definite matrix A , the rank- s approximation (cid:101)
A given above satisfies, with probability at least 1 − δ , the following:
[Equation]
where β ( ℓ, N ) = 1 −
[Equation]
is the best rank- s approximation of A .
As D A max ≤ 2 A max , if we have N ≥ 2 ℓ , it holds that
[Equation]
and we can just state
[Equation]
We show the following lemma as a consequence of this proposition.
Lemma 2.
Let s ≤ ℓ be positive integers and δ > 0 .
Let k : X
×
X be a symmetric and positive semi-definite kernel and y 1 , y 2 , . .
. be i.i.d.
random variables taking values in X .
For each N , define the N × N matrices K ( N ) , K s ( N ) , K Zs ( N ) by
[Equation]
where Z = ( y 1 , . .
, y ℓ ) .
Then, there exists a sequence ε N → 0 such that
[Equation]
is met with probability at least 1 − δ − ε N .
Proof.
We assume N ≥ 2 ℓ .
Let i 1 , . .
, i ℓ be independent uniform samples from { 1 , . . .
, N } .
Consider the event E N that i 1 , . .
, i ℓ are all different.
Then, P ( E N ) = (cid:81) [-(cid:96)i =
-]ℓi
=1 N +1 − i N converges to 1 as N → ∞ , and let ε N = 1 − P ( E N ) .
By using Proposition 3, (24) and max i K ( N ) ii ≤
N − 1 sup x k ( x, x ) , we have that the probability
[Equation]
E N (cid:33) is at least (1 − δ − ε N )
/ P ( E N ) ≥ 1 − δ − ε N , where (cid:101) K s ( N ) is the rank- s Nyström approximation of the matrix K ( N ) by using indices
i 1 , . . , i ℓ .
From (22), if we take (cid:102) W = k
( y i j , y i k ) ℓj,k =1 and (cid:102) W s its best rank- s approximation, it actually satisfies
[Equation]
where D =
( y i 1 , . . .
, y i ℓ ) and k Ds is the Nyström approximation given in the main body.
As y 1 , . . .
, y N are i.i.d.
samples, we can see that ( Z, ( y i ) Ni =1 ) (without any conditioning) and ( D, ( y i )
Ni =1 ) conditioned on E N actually have the same distribution, so we are done.
We finally prove the result for the Nyström approximation of integral operators.
Proof of Theorem 3.
Take a sufficiently large N and let us use K ( N ) , K s ( N ) , K Zs ( N ) defined in the previous lemma with y 1 , y 2 , . . .
independently sampled from µ .
It suffices to consider the case C k := sup x ∈X k ( x, x ) < ∞ .
It is clear that K s ( N ) ii ≤ K ( N ) ii ≤
C k /N , and from (10), we also have k
[Equation]
[Equation]
[Equation] [Equation] k [Equation]
k
[Equation]
x ) , and so K Z s ( N ) ii ≤ C k /N .
For a matrix A ( N ) ∈ R N × N defined by A ( N ) ij = (1 − δ ij )( K ( N ) − K Zs ( N )) , i.e., the matrix given by deleting the diagonal, we have ∥ A ( N ) ∥ 2 → ∥K Zs − K∥ as N → ∞ almost surely [32,
Theorem 3.1].
Since we have observed that ∥ K ( N ) − K Zs ( N ) − A ( N ) ∥ 2 ≤ C k /N , we have
[Equation]
almost surely.
The same argument yields ∥ K ( N ) − K s ( N ) ∥ 2 → σ s +1 , as it converges to the norm of the integral operator given by the kernel (cid:80) m ≥ s [-+
-]+1 σ m e m ( x ) e m ( y ) .
Now, by letting A N be the event that (24) holds (so P ( A N ) ≥ 1 − δ − ε N ), the desired inequality (12) almost surely holds under the event lim sup A N = (cid:84) N>ℓ (cid:83) M ≥ N A M .
Indeed, under this event we can just take the limit of both sides of (24) for an appropriate subsequence of (2 ℓ, 2 ℓ + 1 , . . . ) .
As we have
[Equation]
the proof is completed.
D Kernel Quadrature when Expectations are Known
When we use an approximate kernel k 0 and know exact expectation of test functions φ 1 , . .
. , φ n −with H k 0 ⊂ span { φ 1 , . .
. , φ n − 1 } , we can obtain an n -point kernel quadrature that exactly integrates φ 1 , . .
φ n − 1 by Algorithm 2.
φ n − ⊂ w ⊤ k 1 , diag ( x ) subject to [ φ ( y 1 ) · · · φ ( y N )] w = (cid:90)
X φ ( x ) d µ ( x ) , w ≥ 0 , 1 ⊤ w = 1 , | w | 0 ≤ n.
(25) to obtain points { x 1 , . . .
, x n } ⊂ { y 1 , . . . , y N } and weights ( w i ) ∈
∆ n satisfying n (cid:88) i = w i φ ( x i ) =
(cid:90)
X φ ( x ) d µ ( x ) if (25) is feasible.
We make several remarks on this algorithm.
First, the problem (25) is, strictly speaking, not a linear programming (LP), as it includes the sparsity constraint | w | 0 ≤ n .
However, as it only contains n equality constraints, its basic feasible solution always satisfies | w | 0 ≤ n and the simplex algorithm automatically gives such a sparse (and optimal) solution even if we do not explicitly impose this constraint, so we call it an LP for simplicity.
Second, this algorithm occasionally fails to output Q n as, with some probability, the LP has no feasible solution.
Although we can repeat the algorithm until we succeed, the number N should be chosen appropriately.
See Remark 5 for this point.
Finally, our algorithm has possibly related approaches such as sparse optimization and Sard’s method, see
Remark 6 and 7.
Remark 5.
A simple approach for constructing a quadrature formula [21] was recently proposed: randomly sample candidate points and find a solution by using a linear programming (LP) solver.
Indeed, for an independent sample y 1 , . . .
, y N ∼ µ , we can construct a quadrature formula with convex weights exactly integrating the functions in F = span { φ 1 , . . .
, φ n − 1 } using a subset of these points if and only if we have
[Equation]
where φ = ( φ 1 , . .
. , φ n − 1 ) ⊤ : X → R n − 1 and conv A denotes the convex hull of A .
Several sharp estimates for the probability of the event (26) are available in Hayakawa et al.
Under the event (26) , we can find a desired rule by using the simplex method for the LP problem (25) .
Remark 6.
From the viewpoint of subsampling, a direct way to obtain quadrature formulas with convex weights supported on a small number of points, is to first sample N candidate points
D N = ( x 1 , . .
, x N ) and then solve the following sparse optimization problem:
[Equation]
27) where k ( D N , D N ) is the corresponding N × N Gram matrix.
Unfortunately, exactly solving this problem is computationally challenging, in particular in contrast to our approach that exploits the spectral properties of k and µ .
Nevertheless, one could use sparse optimization to obtain an approximate solution of (27) : although the simplex constraint ( w ≥ 0 , 1 ⊤ w = 1 ) makes it impossible to exploit the classical ℓ 1 regulatization, there are possible alternatives under this constraint [51, 34, 36] or use the DC (difference of convex functions) algorithm to incorporate the sparsity constraint to find a local minima [17].
This is a promising research direction, and our general sample estimates might provide a first step towards this direction.
Remark 7.
Sard’s method [53, 35] for constructing numerical integration rules uses the n degree of freedom (of choosing weights in our setting) separately; m ( ≤ n ) for exactness over a certain m -dimensional space of test functions, and the remaining n − m for minimizing an error criterion such as the worst-case error.
In the context of kernel quadrature, one way to use Sard’s method with exactness over F (an m -dimensional space of test functions) is as follows [28, 56]:
[Equation]
where D n = ( x 1 , . . . ,
x n ) ⊤ , f ( D n ) =
( f ( x 1 ) , . . .
, f ( x n )) ⊤ .
This amounts to solving a convex quadratic programming for w in an ( n − m ) -dimensional subspace of R n (without constraint).
This is similar to our approach in that it enforces exactness in a certain finite-dimensional space of test functions.
One key difference is that Sard’s approach aims for a quadrature formula on a given set of points, whereas our method determines also the points themelves.
Hence, the combination of these two approaches seems to be an interesting future research topic.
Computational complexity.
A tricky part of this approach, essentially based on random convex hulls, is that the algorithm possibly does not output a quadrature formula.
Hence, the following quantity plays an important role to estimate the essential complexity of the algorithm:
[Equation]
where y, y 1 , y 2 , . .
. are independent samples from µ .
This value is known to be finite and estimated under a variety of conditions on φ ( y ) [64, 22].
If we have some knowledge of µ , we can just keep trying the algorithm with N = N φ until it succeeds, and its expected computational time is
O ( nN φ + C ( n, N φ )) , where C ( a, b ) is the (expected) cost of solving an a × b LP with a simplex method.
Note that, though the worst-case computational time of the simplex method is exponential, it is empirically O ( ab min { a, b } ) in practice [50, 54].
In addition, N φ = O ( n ) holds in examples with some symmetry [66, 21], so in that case we have a heuristic complexity estimate of O (cid:0) n 3 (cid:1) .
Choice of approximate kernels.
Similarly to the empirical version discussed in the main text, we prove quantitative estimates when k 0 is given by the Mercer approximation or Nyström approximation.
Remark that the necessary information for using these methods is different.
Whereas using the Mercer approximation requires the knowledge of Mercer decomposition k
( x, y ) = (cid:80) ∞
m =1 σ m e m ( x ) e m ( y ) and their exact integration (cid:82)
X
e m ( x ) d µ ( x ) , the Nyström approximation only requires the exact integral values of kernel, (cid:82) X
k ( x, y ) d µ ( y ) , and so is more generally applicable.
See the following sections for details.
In the following, we assume that the kernel attains the Mercer decomposition k
( x, y ) = (cid:80) ∞
m =1 σ m e m ( x ) e m ( y ) , where σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 is an orthonormal set of L 2 ( µ ) .
D.1 Algorithm 2 with Mercer Approximation
If we use the truncated Mercer decomposition as an approximate kernel, we have the following result.
Theorem 8.
If Algorithm 2 with k 0 :
= (cid:80) n − 1 m =1 σ m e m ( x ) e m ( y ) and φ i =
e i successfully outputs a convex quadrature Q n , then it satisfies the following:
(a) If C := sup m ≥ 1 ∥ e m ∥ ∞ < ∞ , we have wce( Q n ; H k , µ ) 2 ≤ 4 C 2 (cid:80) ∞ m = n σ m .
(b) As N in Algorithm 2 tends to infinity, we have
[Equation]
Proof.
As in the proof of Corollary 2, k − k 0 is positive semi-definite.
Thus, when (cid:80) ∞ m = n σ m < ∞ , the kernel and the measure µ satisfies Assumption A.
So Thorem 5(a) implies (a) of this theorem, since k 1 ( x, x ) = (cid:80) ∞ m = n σ m e m ( x ) 2 ≤ C 2 (cid:80) ∞ m = n σ m .
For (b), if we have Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) , then Theorem 5 implies
[Equation]
So it suffices to prove P ( Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ))
→ 1 as N → ∞ , and it is shown by considering the optimal basic feasible solution of the LP (25) and the following fact [22, Proposition 4]:
[Equation]
where ψ = ( φ 1 , . .
φ n − 1 , k 1 , diag ) ⊤ .
Indeed, under the event ψ ( y ) ∈ conv { ψ ( y 1 ) , . .
. , ψ ( y N ) } , the LP becomes feasible and Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) follows from the optimality.
See the proof of
Theorem 7 (Section C.4) for a more detailed explanation if necessary.
Note that the boundedness of C is a typical assumption [see 40, Assumption 3.2 and references therein], while it does not necessarily hold [44, Section 3].
Under some assumptions, we can quantify the probability that the LP (25) becomes feasible.
Sampling bound.
Suppose 1 is an eigenfunction of K .
This is satisfied, e.g., in the following cases:
• µ is a Haar measure on a compact group and k is shift-invariant.
• k is a kernel based on Stein’s identity [47, 56, 1] with respect to µ .
In this case, we have a theoretical bound of the required N in Algorithm 2 as follows.
Theorem 9.
Suppose 1 is an eigenfunction of K , i.e., (cid:82) X k ( · , y )
d µ ( y ) is a constant function.
Then, for each n ≥ 2 and N ≥ 6( n − 1) sup x ∈X (cid:80) n − 1 m =1 e m ( x ) 2 , Algorithm 2 returns a feasible quadrature with probability at least 1 − 2 1 − n , i.e., for an independent sample y 1 , . . .
, y N from µ , we have
[Equation]
where φ = ( e 1 , . .
, e n − 1 ) ⊤ .
If the value C = sup m ≥ 1 ∥ e m ∥ ∞ is finite, N ≥ 6 C ( n − 1) 2 is also sufficient for the above estimate.
Proof.
This follows from the existing results [22, Theorem 14 and Proposition 17].
D.2 Algorithm 2 with Nyström Approximation
Although the method discussed in the previous section requires the knowledge of Mercer decomposition, if we make use of the Nyström approximation, we only require the values of (cid:82) X k ( x, y )
d µ ( y ) for x ∈ X .
Recall that k Zs ( x, y ) is the rank- s Nyström approximation of the kernel k based on the point set
Z = ( z 1 , . . .
, z ℓ ) .
From (11), we can use φ Zi := u ⊤ i k
( Z, · ) as test functions.
Theorem 10.
Let n ≤ ℓ and δ > 0 , and let Z be an ℓ -point independent sample from µ .
If Algorithmwith k {+
+} =
k Zn − 1 and φ i =
φ Zi successfully outputs a convex quadrature Q n , then with probability at least 1 − δ , we have
[Equation]
Proof.
As in the proof of Corollary 4, k − k Zn − 1 is positive semi-definite.
Also, we can assume (cid:80) ∞ m =1 σ m < ∞ , as otherwise the right-hand side is infinity.
Thus k and k 0 =
k Zn − 1 satisfy
Assumption A.
Note that for a function of the form c ( x, y )
= a · b ( x ) b ( y ) with a ∈ R and b ∈ L 2 ( µ ) , and an orthonomal set ( f i ) i ∈ I ⊂
L 2 ( µ ) of L 2 ( µ ) with b ∈ span { f i | i ∈ I } , we have
[Equation]
i
[Equation]
If we here use the orthonormal set ( e m ) ∞ m =1 that appears in the Mercer decomposition, by letting k 1 :
= k − k Zn − 1 and using the linear extension of (29), we have
[Equation]
Then, by combining this with Theorem 3 and Theorem 5(c), we obtain the desired estimate.
Remark 8.
If we denote by N φ the required number of samples, the computational complexity of the above algorithm becomes O (cid:0) nℓN φ + nℓ 2 + C ( n, N φ ) (cid:1) , including the cost of computing the
Nyström approximation as well as test functions at N φ samples (see also Remark 2).
E Additional Numerical Experiments
In this section, we provide additional experiments on Algorithm 2 using random convex hulls, as well as the approximated version of the N. + emp described in Remark 2.
Section E.1 shows the comparison of Algorithm 2 (with Mercer/Nyström approximation) with some of the methods mentioned in the main text under the periodic Sobolev spaces with uniform measure.
Section E. investigates Algorithm 2 (with Nystöm approximation) as well as the approximate but fast algorithm for N. + emp , under the setting of empirical measure reduction.
E.1 Periodic Sobolev Spaces with Uniform Measure
We conducted experiments under the same setting as in Section 3.1, except that we additionally have the following methods
: Nyström, Nyström + opt: We used the same test functions as N. + emp with the random set Z of size ℓ = 10 n , but for Algorithm 2.
We used N = 10 n samples for the LP (25).
In Nyström + opt we additionally optimized the convex weights using (13). Mercer, Mercer + opt ( d = 1 ): We used the same test functions as M. + emp , but for Algorithm 2. We used N = 10 n samples for the LP (25).
In Mercer + opt we additionally optimized the convex weights using (13).
The results are given in Figure 3.
The weights of Nyström and Mercer are already almost optimized as they exactly integrate a certain family of functions, so the additional CQP (13) does not change the error so much.
Surprising is that N. + emp + opt is almost as good as Nyström + opt or even better.
This implies that the recombination points with respect to a moderately large ( N = n 2 in this case) empirical measure can provide a good convergence rate in Bayesian quadrature [24], even though the (equally weighted) empirical measure itself is not that close to the true measure.
[Figure]
The worst computational time per one trial was 5 seconds of N. + emp and N. + emp + opt in ( d, r, n ) =
(3 , 3 , 80) , while Nyström and Nyström + opt ran in 0.9 seconds under the same setting.
There were no infeasible LPs
[Figure]
Odd behavior of ‘Mercer’.
As we can see in Figure 3(a,b), the methods based on the exact Mercer decomposition becomes very close to optimal when n = 15 , 65 .
As it seemed to be caused by the parity of n , we carried out another experiment for n ∈ { 5 , 9 , 15 , 19 , 29 , 39 , 49 , 65 , 79 } (Figure 4), then Mercer and its optimization clearly became the best methods except the exact optimal Unifrom
Grid .
It might be related to the structure of the periodic Sobolev space, that has, for each eigenvalue except for 1 , two-dimensional eigenspace ( cos and sin ), but needs further investigation.
Also, in the case ( d, r ) =
(1 , 3) , we see ‘ + opt ’ make the quadrature less accurate for a big n , but it is theoretically almost impossible, so it seems to be caused by numerical accuracy of the CQP solver.
E.2 Measure Reduction in Machine Learning Datasets
We conducted experiments under the same setting as in Section 3.2.
We additionally adopted
Nyström, Nyström + opt (with N = 20 n ), and FNE, FNE + opt , where FNE (stands for ‘fast N. + emp’) is the approximate algorithm for N. + emp by omitting the inequality in (6) and using the randomized SVD [19] (see Remark 2).
[Figure]
The results are given in Figure 5.
N. + emp + opt and FNE + opt show almost the same convergence.
While in the largest case n = 160 , the average runtime of ( N. + emp + opt , FNE + opt ) was (13 . 0 , 2 . 07) seconds in 3D Road Network data and (12 . 8 , 2 .
) seconds in Power Plant data, respectively.
Although our theoretical guarantee no longer holds for FNE , it accelerates the algorithm while surprisingly maintaining the accuracy.
Nystöm or Nyström + opt behave much better than iid
Bayes , but are slightly less accurate than N. + emp + opt and FNE + opt , whereas they have good theoretical guarantees (Theorem 10).
Their computational time was basically between that of FNE + opt and N. + emp + opt . Comparison with another empirical measure. The setting of ‘ML datasets’ treated here is empirical measures given by some real data, so it is also just an approximation of a true distribution from the viewpoint of frequentists. Therefore, if we want to evaluate the performance of measure reduction methods with regard to the true distribution, we should measure the worst-case error using it.
As it is not feasible in reality, we take another empirical measure µ ′ (of the same size as but different from the empirical measure µ , used in the construction of a kernel quadrature rule Q n ), and plot the quantities of wce( Q n ; H k , µ ′ ) to better estimate the actual performance of Q n in this section.
The overall setting is the same as in Section 3.2, except the following points
: • In the 3D Road Network Data Set, we used another random 43487-point subset from the remaining 434874 − 43487 data points to define µ ′ .
• In the Combined Cycle Power Plant Data Set, we used exactly a half of the whole data points to define µ (so the size of supp µ is different from the original experiment) and the other half to define µ ′ .
Note that µ and µ ′ were randomly taken at first and fixed throughout the experiment.
The median heuristics as well as the normalization of the data (for both of the points in µ and µ ′ ) was carried out by using the statistical information solely given by µ .
[Figure]
The results are given in Figure 6.
We can see that, though our methods are still competitive, the error eventually becomes dominated by the (MMD-)distance between µ and µ ′ as n gets larger.
This is inevitable as we are only using the empirical measure µ to construct Q n , so in an application to this kind of setting, we can just pick any method whose error is sufficiently small compared to the ‘inevitable’ error.