Positively Weighted Kernel Quadrature via Subsampling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study kernel quadrature rules with convex weights.
Our approach combines thespectral properties of the kernel with recombination results about point measures.
This results in effective algorithms that construct convex quadrature rules usingonly access to i.i.d.
samples from the underlying measure and evaluation of thekernel and that result in a small worst-case error.
In addition to our theoreticalresults and the beneﬁts resulting from convex weights, our experiments indicate thatthis construction can compete with the optimal bounds in well-known examples.
Introduction
The goal of numerical quadrature is to provide, for a given probability measure µ on a space X , a setof points x 1 , . . .
, x n ∈ X and weights w 1 , . .
w n ∈ R such that
[Equation]
holds for a large class of functions f : X → R .
Kernel quadrature focuses on the case when thefunction class forms a reproducing kernel Hilbert space (RKHS).
What makes kernel quadratureattractive, is that the kernel choice provides a simple and ﬂexible way to encode the regularityproperties of a function class.
Exploiting such regularity properties is essential when the integrationdomain X is high-dimensional or the function class is large.
Additionally, the domain X does nothave to be Euclidean, but can be any topological space that carries a positive semi-deﬁnite kernel.
More formally, given a set Q := { ( w i , x i ) :
i = 1 , . .
. , n } ⊂ R ×
X denote with µ Q := (cid:80) n i =1 w i
δ x ithe resulting measure on X .
We refer to Q [resp.
µ Q ] as a quadrature [resp.
quadrature measure], tothe points x 1 , . . .
, x n as the support of Q [resp.
µ Q ].
The aim of kernel quadrature is to constructquadrature measures µ Q that have a small worst-case error
[Equation]
where H k denotes the RKHS associated with a positive semi-deﬁnite kernel k .
If the weights arepositive and sum up to one, w i > 0 , (cid:80) ni
=
w i = 1 , then we refer to Q as a convex quadrature rule.
Contribution.
The primary contribution of this article is to leverage recombination (a consequenceof Carathéodory’s Theorem) with spectral analysis of kernels to construct convex kernel quadraturerules and derive convergence rates.
We also provide efﬁcient algorithms that compute these quadraturerules; they only need access to i.i.d.
samples from µ and the evaluation of the kernel k .
See Table 1for a comparision with other kernel quadrature constructions.
The table is written by using σ n and r n , which represent a sort of decay of the kernel with respect toµ .
Typical regimes are σ n ∼ n − β
(e.g. Sobolev) or σ n ∼ exp( − γn ) (e.g. Gaussian) depending onthe ‘smoothness’ of the kernel [e.g. 15, 2] (see also Section B.3), and in such regimes (with β ≥ 2 orγ > 0 ), σ n or r n ( (cid:46) nσ n ) provide faster rates than wce 2 ∼ 1 /n of the usual Monte Carlo rate.
Limitation.
Our proposed methods are based on either Mercer or Nyström approximation.
Thoughour Mercer-based methods have strong theoretical bound, they require the knowledge of Mercerdecomposition like [2, 6, 7], which is not available for general ( k, µ ) .
Our Nyström-based methods arepractical and outperform existing methods in experiments, but the n/ √ (cid:96) term makes their theoreticalbound far from competitive.
So further study is needed for truly bridging theory and practice.
Method
Bound of squared wce
Computational complexity
C
M
E
Herding [9, 3] 1 /n n · ( n : global optimization ) (cid:88) (cid:88)
SBQ [24]
Not found n · ( n 2 : global optimization ) (cid:88)
Leveraged [2] σ m , m = O ( n log n )
Unavailable
DPP [6, 5] r n + n 3 · ( rejection sampling )
CVS [7] σ n +
Unavailable (cid:88)
KT++ [13, 14, 55] (1 /n 2 + 1 /N )
polylog( N )
N log 3 N (cid:88) (cid:88) (cid:88)
Ours:
Mercer † r n nN ϕ + C ( n, N ϕ ) (cid:88)
M. + empirical ‡ r n + 1 N nN + n 3 log( N/n ) (cid:88) (cid:88)
Nyström † [Equation] n(cid:96)N
[Equation]
+ n(cid:96)
[Equation]
+ C ( n, N [Equation] ) (cid:88) (cid:88)
N. + empirical ‡ [Equation] n(cid:96)N + n(cid:96) 2 + n 3 log( N/n ) (cid:88) (cid:88) (cid:88)
Why Convex Weights?
There are several reasons why convex weights are preferable: (i) Positive Integral Operator: Kernel quadrature provides an approximation of the integration operatorf (cid:55)→ I ( f )
= (cid:82) f
( x ) d µ ( x ) .
Hence, a natural requirement is to preserve basic properties of thisoperator and positive weights preserve the positivity of this operator.
(ii) Uniform estimates and
Robustness: In applications, the RKHS H k may be mis-speciﬁed if a quadrature rule with negativeweights is applied to a function f / ∈ H k , the approximation error (1) can get arbitrary bad; in contrast,a simple estimate shows that convex weights give uniform bounds, see Appendix B.4.
(iii) Iteration: Consider the m -fold product of quadrature formulas for approximating µ ⊗ m on X m .
This isa common construction for a multidimensional quadrature formulas (e.g., for polynomials) fromone-dimensional formulas [60] or numerics for stochastic differential equations [41].
In doing so,working with a probability measure is strongly preferred, since otherwise the total variation of theirm -fold product gets exponentially large as m increases ( (cid:107) µ ⊗ m (cid:107) TV
= (cid:107) µ (cid:107) m TV ).
Related Literature.
Roughly speaking, there have been two approaches to kernel-based quadratureformulas: kernel herding and random sampling.
In kernel herding or its variants, the points ( x i ) ni =1are found iteratively, typically based on the Frank–Wolfe gradient descent algorithm [9, 3, 24].
In the random sampling approach, ( x i ) ni =1 are sampled and subsequently the weights are optimized.
Generically, this results only in a signed measure µ Q but not a probability measure.
Bach [2] and
Belhadji et al.
[6] use the eigenvalues and the eigenfunctions of the integral operator K : f (cid:55)→ 54 (cid:82)
X k ( · , y ) f
( y ) d µ ( y ) to obtain a Mercer-type decomposition of k [58].
Bach [2] then uses the eigenvalues and eigenfunctions of K to deﬁne an optimized measure from which the points ( x i ) arei.i.d.
sampled.
This achieves a near optimal rate, but the exact sampling from this measure is usuallyunavailable, although for special cases, it can be done efﬁciently.
In contrast, Belhadji et al.
[6]proposes non-independent sampling based on the determinantal point process [DPP; 23].
These twopapers also treat the more general quadrature problem that includes a weight function g ∈ L 2 ( µ ) , i.e.,approximating (cid:82) X f ( x ) g ( x ) d µ ( x ) for f ∈ H k ⊂ L 2 ( µ ) , which we do not discuss in this paper.
There is another recently proposed practical method called kernel thinning [13, 14], which aims atefﬁcient compression of empirical measures like our ‘+ empirical’ methods.
Its acceleration [55]makes it a competitive candidate in terms of compressing N ∼ n 2 points (‘KT++’ in Table 1).
Weemphasize that the literature on kernel quadrature is vast, and the above distinction is only a roughdichotomy, see e.g. [11, 39, 8, 26, 47, 28, 27, 56].
Finally, [63] essentially performs recombination based on Fourier features for kernel density estimation, and our methods can algorithmicallycontribute, while the problem is different from kernel quadrature and the theory differs accordingly.
Outline.
Section 2 contains our main theoretical and methodological contribution.
Section 3provides numerical experiments on common benchmarks.
The Appendix contains several extensionsof our main result, proofs, and further experiments and benchmarks.
Main Result
Assume we are given a set 1 of n − 1 functions ϕ 1 , . . .
, ϕ n − 1 : X → R such that their linearcombinations well approximate functions in H k .
Then our kernel quadrature problem essentiallyreduces to the construction of an n -point discrete probability measure µ Q n = (cid:80) ni
=1 w i
δ x i such that
[Equation]
A simple way to approximately construct this µ Q n is to ﬁrst, sample N (cid:29) n points, ( y i )
Ni =1 ,from µ such that their empirical measure, (cid:101) µ N = 1 N (cid:80) Ni =
δ y i , is a good approximation to µ in thesense that (cid:82) ϕ i d (cid:101) µ N ≈ (cid:82) ϕ
i d µ for i = 1 , . .
. , n − 1 , and secondly, apply a so-called recombinationalgorithm (Remark 1) that takes as input ( y i )
Ni =1 and n functions ϕ 1 , . . .
, ϕ n − 1 and outputs a measureµ Q n = (cid:80) w i
δ x i by selecting a subset ( x i ) ni
=1 of the points ( y i )
Ni =1 and giving them weights ( w i ) ni
=1such that µ Q n is a probability measure that satisﬁes the equation (3) with µ replaced by (cid:101) µ N .
The challenging parts of this approach are (i) to construct functions ϕ 1 , . . .
, ϕ n − 1 that approximatelyspan the RKHS H k for a small n ; (ii) to arrive at good quantitative bounds despite the (probabilistic)sampling error resulting from the use of the empirical measure (cid:101) µ N , and the function approximationerror via ϕ 1 , . . .
, ϕ n − 1 .
To address (i) we look for functions such that
[Equation]
with some c i ≥ 0 .
Two classic ways to do this are the Mercer and Nyström approximations.
Theremaining, item (ii) is our main contribution.
Theorem 1 shows that the worst-case error, (5), iscontrolled by the sum of two terms: the ﬁrst term stems from the kernel approximation (4), the secondterm stems from the sample error.
Theorem 1.
Let µ be a Borel probability measure on X and k a positive semi-deﬁnite kernel on Xsuch that (cid:82) X k ( x, x ) d µ ( x ) < ∞ .
Further, let n be a positive integer and assume k 0 is a positivesemi-deﬁnite kernel on X such that
k − k 0 is a positive semi-deﬁnite kernel on X , and 2.
dim H k 0 < n .
There exists a function KQuad such that if D N is a set of N i.i.d.
samples from µ , then Q n =
KQuad( D N ) is a random n -point convex quadrature that satisﬁes
[Equation]
[Equation] where c k,µ := (cid:82) X
k
( x, x ) d µ ( x ) − (cid:82)(cid:82)
X×X k ( x, y ) d µ ( x ) d µ ( y ) .
Moreover, the support of Q n is a subset of D N and given functions ϕ 1 , . . .
, ϕ n − 1 ∈ L 1 ( µ )with H k 0 ⊂ span { ϕ 1 , . . .
, ϕ n − 1 } , Q n =
KQuad( D N ) can be computed with Algorithm 1 in
O (cid:0) nN + n 3 log( N/n ) (cid:1) computational steps.
The function KQuad is deterministic but since D N is random, the resulting quadrature rule Q n israndom, hence also the resulting worst case error wce( Q ; H k , µ ) and the expectation in (5) denotesthe expectation over the N samples in D N .
The theoretical part of Theorem 1 follows from moregeneral results that we present and prove in Appendix (Theorem 6 for inequality, essentially done bycomparing H k with H k 0 , and 7 for existence) while the algorithmic part of Theorem 1 is discussed inthe following section.
This result covers our two main examples for the construction of k 0 , resp.
thechoice of ϕ 1 , . .
ϕ n − 1 , that we discuss below and for which the error estimate gets quite explicit:the Mercer approximation and the Nyström approximation.
The former requires some knowledgeabout the spectrum of the kernel which is, however, known for many popular kernels; the latter worksin full generality but yields worse convergence rates.
Finally, we emphasize that N and n in Theorem1 can be chosen independently and we will see that from a computational point the choice N ∼ n 2 ispreferable in which case (5) is faster rate than Monte Carlo, see also Table 1.
2.1 Algorithm
Input: A positive semi-deﬁnite kernel k on X , a probability measure µ on X , integers N ≥ n ≥ 1 , another kernel k 0 , functions ϕ 1 , . . .
, ϕ n − 1 on X with H k 0 ⊂ span { ϕ 1 , . . . , ϕ n − 1 } and a set D N of N i.i.d.
samples from µ .
Output:
i
n
w i ≥ 0 , ni 1 w i  1: Apply a Recombination Algorithm (Remark 1) with ψ = ( ϕ 1 , . . .
, ϕ n − 1 , k 1 , diag ) (cid:62) , to the empirical measure 1 N (cid:80) y ∈ D
N δ y to obtain points { (cid:101) x 1 , . .
, (cid:101) x n +1 } ⊂ D N and weights v = ( v 1 , . . .
, v n +1 ) (cid:62) ≥ 0 that satisfy 1 (cid:62) v = 1 and ψ ( (cid:101) x ) v = 1 N (cid:80) Ni =1 ψ ( (cid:101) x i ) , where ψ ( (cid:101) x ) =
[ ψ ( (cid:101) x 1 ) , . . .
, ψ ( (cid:101) x n +1 )]
∈ R n × ( n +1) .
with the i − y j ij ∈ R n ×
n
ϕ 0 = ﬁnd nonzero vector u ∈ R n +1 such that A u = 0 and k 1 , diag ( (cid:101) x ) u ≥
: Compute the smallest α such v − α u ≥ 0 and v j − αu j 0 for some j
k − αu k ) k ∈ I ( x i ) ni = x k )
k ∈ I , I = { 1 , . . . , n + 1 } \ { Suppose we are given k 0 and ϕ 1 , . . .
, ϕ n − 1 ∈ L 1 ( µ ) with H k 0 ⊂ span { ϕ 1 , . . .
, ϕ n − 1 } , and also
N independent samples µ denoted by D N =
( y 1 , . . .
, y N ) .
From Theorem 6 in Appendix, if weconstruct a convex quadrature Q n = ( w i , x i ) ni
=1 satisfyingn (cid:88)
[Equation]
where ϕ
= ( ϕ 1 , . . .
, ϕ n − 1 ) (cid:62) and k 1 , diag ( x ) =
k ( x, x ) − k 0 ( x, x ) , it satisﬁes the bound (5).
For thisproblem, we can use the so-called recombination algorithms (see Section ?? for intuition):
Remark 1 (Recombination) .
Given d − 1 functions (called test functions) and a probability measuresupported on N > d points, there exists a probability measure supported on a subset of d points thatgives the same mean to these d − 1 functions.
This follows from Carathéodory’s theorem and is knownas recombination.
Efﬁcient deterministic [37, 42, 60] as well as randomized [10] algorithms exist tocompute the new probability measure supported on d points; e.g. deterministic algorithms performthe recombination, step 1, in O (cid:0)
c ϕ N + d 3 log( N/d ) (cid:1) time, where c ϕ is the cost of computing all thetest functions at one sample.
If each function evaluation is in constant time, c ϕ
= O ( d ) .
Although the recombination introduced here only treats the equality constraints in (6) we cansatisfy the remaining constraints just with n points by modifying it.
This is done in Algorithm 1.
What the algorithm is doing is as follows.
First, via recombination, ﬁnd an ( n + 1) -point convexquadrature R n +1 =
( v i , y i ) n +1 i =1 that exactly integrate functions ϕ 1 , . . .
, ϕ n − 1 , k 1 , diag with regard to the empirical measure 1 N (cid:80) Ni =
δ y i .
Second, to reduce one point, ﬁnd a direction ( − u in thealgorithm) in the space of weights on ( (cid:101) x i ) n +1 i =1 that does not change the integrals of ϕ 1 , . .
, ϕ n − 1 andthe constant function 1 , and does not increase the integral of k 1 , diag .
Finally, move the weight fromv to the above direction until an entry becomes zero, at v − α u .
Such an α ≥ 0 exists, as u musthave a positive entry since it is a nonzero vector whose entries sum up to one.
Now we have a convexweight vector with at most n nonzero entries, so it outputs the desired quadrature satisfying (6).
2.2 Mercer Approximation
In this section and Section 2.3, we assume that k has a pointwise convergent Mercer decompositionk ( x, y )
= (cid:80) ∞
m =1 σ m e m ( x ) e m ( y ) with σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 ⊂ L 2 ( µ ) being orthonormal [58].
If we let K be the integral operator L 2 ( µ ) →
L 2 ( µ ) given by f (cid:55)→ (cid:82) X k ( · , y ) f
( y ) d µ ( y ) ,then ( σ m , e m ) ∞ m =1 are the eigenpairs of this operator.
The ﬁrst choice of the approximate kernel k 0 is just the trucation of Mercer decomposition.
Corollary 2.
Theorem 1 applied with k 0 ( x, y )
= (cid:80) n − 1 m =
σ m e m ( x ) e m ( y ) yields a random convexquadrature rule Q n such that
[Equation]
Proof.
It sufﬁces to prove the result under the assumption (cid:82)
X k
( x, x ) d µ ( x ) = (cid:80) ∞ m =1 σ m < ∞ , asotherwise the right-hand side of (7) is inﬁnity.
For k 1 := k − k 0 , we have that k 1 ( x, y )
=
(cid:80) ∞
m = n σ m e m ( x ) e m ( y ) and it is the inner product ofΦ( x ) :
= ( √ σ m e m ( x )) ∞ m = n and Φ( y ) in (cid:96) 2 ( { n, n + 1 , . . . } ) and so positive semi-deﬁnite.
Thus kand k 0 satisﬁes the assumption of Theorem 1, and (cid:82) X k 1 ( x, x ) d µ ( x ) = (cid:80) ∞ m = n σ m applied to (5)yields the desired inequality.
2.3 Nyström Approximation
Although the Nyström method [67, 12, 33] is primarily used for approximating a large Gram matrixby a low rank matrix, it can also be used for directly approximating the kernel function itself whichis how we use it.
Given a set of (cid:96) points Z
= ( z i ) (
cid:96)i =1 ⊂ X , the vanilla Nyström approximation ofk ( x, y ) is given by [Equation]
[Equation]
) where P Z : H k → H k is a projection operator onto span { k ( · , z i ) } (cid:96)i =1 .
In matrix notation, we havek Z
[Equation]
x,
[Equation]
[Equation]
[Equation]

[Equation]
[Equation]
) 

[Equation]
where W =
( k ( z i , z j ))
(cid:96) i,j =1 is the Gram matrix for Z and W + denotes its Moore–Penrose inverse.
For completeness, we explain the equivalence between (8) and (9) in Appendix B.5.
As k Z isan (cid:96) -dimensional kernel, there exists an ( (cid:96) + 1) -point quadrature formula that exactly integratesfunctions in H k Z .
For a quadrature formula, exactly integrating all the functions in H k Z is indeedequivalent to exactly integrating k ( z i , · ) for all 1 ≤ i ≤ (cid:96) , as long as the Gram matrix k
( Z, Z ) isnonsingular.
Proposition 1 in the Appendix provides bound for the associated worst case error.
Fromthis viewpoint, the Nyström approximation offers a natural set of test functions.
The Nyström method has a further generalization with a low-rank approximation of k ( Z, Z ) .
Preciselyspeaking, by letting W s be the best rank- s approximation of W =
k
( Z, Z ) (given by eigendecomposition), we deﬁne the following s -dimensional kernel:
[Equation]
[Equation]
Let W =
U Λ U (cid:62) be the eigendecomposition of W , where U =
[ u 1 , . . .
, u (cid:96) ]
∈ R (cid:96) ×
(cid:96) is a realorthogonal matrix and Λ
= diag( λ 1 , . . .
, λ (cid:96) ) with λ 1 ≥ · · · ≥ λ (cid:96) ≥ 0 .
Then, if λ s > 0 we have
[Equation]
[Equation]
So we can use functions u (cid:62) i
k ( Z, · ) ( i = 1 , . . .
, s )
as test functions, which is chosen from a largerdimensional space span { k ( z i , · ) } (cid:96)i =1 .
Although closer to the original usage of the Nystöm methodis to obtain u (cid:62) i k
( Z, · ) as an approximation of i -th eigenfunction of the integral operator K with Zappropriately chosen with respect to µ , we have adopted an explanation suitable for the machinelearning literature [12, 33].
The following is a continuous analogue of Kumar et al.
[33, Theorem 2] showing the effectiveness ofthe Nyström method.
See also Jin et al.
[25] for an analysis speciﬁc to the case s = (cid:96) .
Theorem 3.
Let s ≤ (cid:96) be positive integers and δ > 0 .
Let Z be an (cid:96) -point
independent sample fromµ .
If we deﬁne the integral operator K Zs : L 2 ( µ ) → L 2 ( µ ) by f (cid:55)→ (cid:82) X k Zs ( · , y ) f ( y ) d µ ( y ) , then wehave, with probability at least 1 − δ , in terms of the operator norm,
[Equation]
The proof is given in Appendix C.5.
By using this estimate, we obtain the following guarantee for therandom convex quadrature given by Algorithm 1 and the Nyström approximation.
Corollary 4.
Let D N be N -point independent sample from µ and let Z be an (cid:96) -point
independent sample from µ .
Theorem 1 applied with the Nyström approximation k
= k
Zn − 1 yields an random n -pointconvex quadrature rule Q n such that, with probability at least 1 − δ and k max := sup x ∈X k
[Equation]
x ) ,
[Equation]
N
[Equation]
Proof.
From (11), k Z ( x, y ) − k Zn − 1 ( x, y )
= (cid:80) (
cid:96)i = n λ − 1 i ( u (cid:62) i k
( Z, x ))
( u (cid:62) i k
( Z, y ))
(ignore the termswith λ i = 0 if necessary), and it is thus positive semi-deﬁnite.
If we deﬁne P ⊥ Z : H k
→ H k as theprojection operator onto the orthogonal complement of span { k ( · , z i ) } (cid:96)i =1 , then, from (8), we alsohave k ( x, y )
− k Z ( x, y ) =
(cid:10) P ⊥ Z k ( · , x ) , P ⊥ Z k ( · , y ) (cid:11)
H k , so k − k Z is also positive semi-deﬁnite.
Inparticular, k − k
Zn − 1 =
( k − k Z )
+ ( k Z − k Zn − 1 ) is positive semi-deﬁnite.
Also, it sufﬁces to provethe result when (cid:80) ∞ m =1 σ m < ∞ , so we can now apply Theorem 1.
For k 1 := k − k
Zn − 1 , we prove the inequality (cid:82) X k 1 ( x, x ) d µ ( x ) = (cid:80) ∞ m =1 (cid:10) e m , ( K − K Zs ) e m (cid:11) L 2 ≤( n − 1) (cid:107)K − K Zs (cid:107) + (cid:80) m ≥ n σ m (see (30) in Appendix D.2 for details), and the desired inequalityfollows by combining Theorem 1 and Theorem 3 (i.e., (5) and (12)).
Remark 2.
Algorithm 1 with the Nystöm approximation can be decomposed into two parts: (a)
Nystöm approximation by truncated singular value decomposition (SVD) (the ﬁrst n − 1 eigenvectorsfrom an (cid:96) -point sample), (b) Recombination from an N -point empirical measure.
The complexityof (a) is O (cid:0) n(cid:96) 2 (cid:1) , and it can also be approximated by randomized SVD in O (cid:0) n 2 (cid:96) + (cid:96) 2 log n (cid:1)
The cost of part (b) is O (cid:0) n(cid:96)N
+ n 3 log( N/n ) (cid:1) , where n(cid:96)N stems from the evaluation of k 1 , diagfor all N sampling points.
If we do not impose the inequality constraint regarding k 1 , diag , whichstill works well in practice, the cost of part (b) becomes O (cid:0) (cid:96)N + n 2 (cid:96) log( N/n ) (cid:1) , by using the trick1 N (cid:80) Ni
=1 U (cid:62) n − 1 k ( Z, y i ) =
U (cid:62) n − 1 1 N (cid:80) Ni
=1 k ( Z, y i ) , where U n − 1 =
[ u 1 , . . .
, u n − 1 ] ∈ R (cid:96) × ( n − 1) is atruncation of the matrix that appears in the Nyström approxiamtion (10,11).
So the overall complexityis O (cid:0) n(cid:96)N
+ n(cid:96) 2 + n 3 log( N/n ) (cid:1) while an approximate algorithm (randomized SVD, without theinequality constraint) runs in O (cid:0) (cid:96)N + (cid:96) 2 log n + n 2 (cid:96) log( N/n ) (cid:1) .
Numerical Experiments
In this section, we compare our methods with several existing methods.
In all the experiments, weused the setting where we can compute (cid:82) X k ( x, y )
d µ ( y ) for x ∈ X and (cid:82)(cid:82) X×X
k
( x, y ) d µ ( x ) d µ ( y )since then we can evaluate the worst-case error of quadrature formulas explicitly.
Indeed, if aquadrature formula Q n is given by points X
= ( x i ) ni
=1 and weights w = ( w i ) ni
=1 , then we have
[Equation]
for independent y, y (cid:48) ∼ µ under (cid:82) X (cid:112) k
( x, x ) d µ ( x ) < ∞ , which is a well-known formula forthe worst-case error [18, 57].
An essential remark shown in Huszár and Duvenaud [24] is that the
Bayesian quadrature [48] with covariance kernel k given observation at points ( x i ) ni
=1 (automatically)estimates the integral as (cid:80) ni
=
w i f ( x i ) with ( w i ) ni
=1 minimizing the above expression.
Once givenpoints ( x i ) ni
=1 and additional knowledge of expectations, we can compute the optimal weights( w i ) ni
=1 by solving a convex quadratic programming (CQP), either without any restrictions or withthe condition that ( w i ) ni
=1 is convex.
Although the former can be solved by matrix inversion, we haveused the optimizer Gurobi 2 for both CQPs to avoid numerical instability.
For the recombination part,we have modiﬁed the Python library by Cosentino et al.
[10] implementing the algorithm of [60].
Our theoretical bounds are close to optimal in classic examples and we see that the algorithm evenoutperforms the theory in practice especially in Section 3.1.
We also execute a measure reduction ofa large discrete measure in terms of Gaussian RKHS and our methods shows a fast convergence ratein two ML datasets in Section 3.2.
3.1 Periodic Sobolev Spaces with Uniform Measure
For a positive integer r , consider the Sobolev space of functions on [0 , 1] endowed with the norm(cid:107) f (cid:107)
= ( (cid:82) 1 0 f ( x ) d x ) 2 +(2 π ) 2 r (cid:82) 1 0 f ( r ) ( x ) 2 d x , where f and its derivatives f (1) , . . .
f ( r ) are periodic(i.e., f (0)
=
f (1) and so forth).
This function space can be identiﬁes as the RKHS of the kernel
[Equation]
x − y
[Equation]
for x, y ∈ [0 , 1] , where B 2 r is the 2 r -th Bernoulli polynomial [65, 2].
If we let µ be the uniform measure on [0 , 1] , the normalized eigenfunctions (of the integral operator) are 1 , c m ( · ) = √ 2 cos(2 πm · )and s m ( · ) = √ 2 sin(2 πm · ) for m = 1 , 2 , . . .
, and the corresponding eigenvalues are 1 and m − 2 r(both for c m and s m ).
Although the rectangle formula f (cid:55)→ n − 1 (cid:80) ni =1 f ( i/n ) (a.k.a.
Uniform
Grid below) is known to be optimal for this kernel [68, 46] in the sense of worst-case error, this
RKHS is commonly used for testing the efﬁciency of general kernel quadrature methods [2, 6, 27].
We also consider its multivariate extension on [0 , 1] d , i.e., the RKHS given by the product kernelk ⊗ d r ( x , y ) :
= (cid:81) di =1 k r ( x i , y i ) for x = ( x 1 , . . .
, x d ) , y =
( y 1 , . .
. , y d ) ∈ [0 , 1] d .
We carried out the experiment for ( d, r ) =
For each ( d, r ) , we comparedthe following algorithms for n -point quadrature rules with n ∈ { 4 , 8 , 16 , 32 , 64 , 128 } .
N. + emp, N. + emp + opt: We used the functions u (cid:62) i k
( Z, · ) ( i = 1 , . .
. , n − 1 ) given by the Nyström approximation (11) with s = n − 1 as test functions ϕ 1 , . .
, ϕ n − 1 in Algorithm 1.
The set Z was given as an ( (cid:96) =)10 n -point independent sample from µ .
We used N = n 2samples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (13)
M. + emp, M. + emp + opt ( d = 1 ): We used the ﬁrst n − 1 functions of the sequence of eigenfunctions 1 , c 1 , s 1 , c 2 , s 2 , . . .
as test functions ϕ 1 , . . .
, ϕ n − 1 in Algorithm 1.
We used N = n 2samples from µ .
In ‘ + opt ’ we additionally optimized the convex weights using (13).
Monte Carlo, iid Bayes: With an n -point independent sample ( x i ) ni =1 from µ , we used uniformweights 1 /n in Monte Carlo and the weights optimized using (13) in iid Bayes .
Uniform Grid ( d = 1 ): We used the rectangle formula f
(cid:55)→ n − 1 (cid:80) ni
=1 f ( i/n ) .
This is known tobe optimal [not just up to constant, but exactly; 68, 46], and thus equivalent to the Bayesianquadrature on the uniform grid, i.e., the weights are already optimized.
Halton, Halton + opt ( d ≥ 2 ): For an n -point sequence given by the Halton sequence with Owenscrambling [20, 49], the uniform weights w i = 1 /n is adopted in Halton and the weightsare additionally optimized using (13) in Halton + opt .
Thinning, Thin + opt:
Given an N -point independent sample ( y i )
Ni
=1 with N = n 2 from µ , ann -point subset ( x i ) ni
=1 taken from a KT++ algorithm (kernel thinning [13, 14] combinedwith Compress++ algorithm
[55] with the oversampling parameter g = min { 4 , log 2 n }
, im-
[Figure]
The shaded regions show their standard deviation.
The worst computational time per one trial was 57 seconds of Thin + opt in ( d, r, n ) = (3 , 3 , 128) , where Thinning was 56 seconds and N. + emp
[+ opt] was 22 seconds.
The results are given in Figure 1.
In d = 1 , the optimal rate given by Uniform Grid is known tobe O (cid:0) n − 2 r (cid:1) .
As the uniform sampling is equal to the optimized distribution of Bach [2] in thiscase, iid Bayes also achieves this rate up to log factors.
Although our theoretical guarantee for
M. + emp is O (cid:0) n 1 − 2 r + N − 1 (cid:1) with N = n 2 (Corollary 2), in the case ( d, r ) =
(1 , 1) , we canobserve that in the experiment it is better than iid Bayes and close to the optimal error of Uniform
Grid , but slightly worse than Thinning .
Moreover, N. + emp , which does not use the informationof spectral decomposition, is remarkably almost as accurate as M. + emp in d = 1 .
Furthermore,if we additionally use the knowledge of expectations, which iid Bayes is already doing, M./N. +emp + opt become surprisingly accurate even with N = n 2 .
They are worse than Thinn + optwhen r = 1 , but well outperform it when r = 3 .
Nonlineality in the graph of these methods when( d, r, n ) =
(1 , 3 , 128) should be from numerical accuracy of the CQP solver (see also Section E.1).
The accuracy of N. + emp + opt becomes more remarkable in multivariate cases.
It behaves almostthe same as Halton + opt in d = 2 and clearly beats it in d = 3 .
Also, the sudden jump of ourmethods around n = 30 in ( d, r ) =
(3 , 3) seems to be caused by the jump of eigenvalues.
Indeed,for the integral operator given by k ⊗ 3 3 with uniform measure, the eigenspace of the largest eigenvalue1 is of dimension 27 , and the next largest eigenvalue is 1 / 64 .
Again in the latter case, N. + emp +opt outperforms Thin + opt , and these results suggest that our method works better when there is astrong spectral decay, as is explicitly incorporated in our algorithm.
Note also that we can compare Figure 1 with Belhadji et al.
[6, Figure 1] which includes some othermethods such as DPPs, herding and sequential Bayesian quadrature, as we did experiments underalmost the same setting.
In particular, in the case ( d, r ) = (1 , 3) where the eigenvalue decay is fast,we see that our method substantially outperforms the sequential Bayesian quadrature.
3.2 Measure Reduction in Machine Learning Datasets
We used two datasets from UCI Machine Learning Repository ( https://archive.ics.uci.edu/ml/datasets/ ).
We set µ as the equally weighted measure over (a subset of) the data points
X =
( X (1) , . . .
, X ( d ) ) (cid:62) ( d = 3 , 5 , respectively), where each entry is centered and normalized.
Weconsidered the Gaussian kernel exp( −(cid:107) x − y (cid:107) 2 / (2 λ 2 ))
whose hyperparameter λ is determined bymedian heuristics [16], and compared the performance of N. + emp , N. + emp + opt (with (cid:96) = 10 n ,
N = n 2 ), Monte Carlo , iid Bayes , Thinning , Thin + opt .
We also added Herding , an equallyweighted greedy algorithm with global optimization [9], and its weight optimization Herd + optwithin convex quadrature given by (13).
We conducted the experiment for n ∈ { 4 , 8 , 16 , 32 , 64 , 128 } .
The ﬁrst is 3D Road Network Data Set [30].
The original dataset is 3 -dimensional real vectorsat 434874 points.
To be able to compute the worst-case error (13) efﬁciently to evaluate eachkernel quadrature, we used a random subset X of size 43487 = (cid:98) 434874 / 10 (cid:99) (ﬁxed throughoutthe experiment) and deﬁned µ as the uniform measure on it.
We determined λ with the medianheuristic by using a random subset of X with size 10000 and used the same X and λ throughout theexperiment.
The second is Combined Cycle Power Plant Data Set [31, 62].
The original dataset is5 -dimensional real vectors at 9568 points.
We set the whole data as X and deﬁned µ as the uniformmeasure on it.
We determined λ with median heuristics by using the whole X .0.
[Figure]
Figure 2 shows the results.
We can observe that in both experiments N. + emp + opt successfullyexploits the fast spectral decay of Gaussian kernel and signiﬁcantly outperforms other methods.
Also,even without using the knowledge of any expectations, N. + emp (and Thinning ) show a decentconvergence rate comparable to Herding or iid Bayes , which actually use the additional information.
Concluding Remarks
We leveraged a classical measure reduction tool, recombination, with spectral properties of kernelsto construct kernel quadrature rules with positive weights.
The resulting algorithms show strongbenchmark performance despite their restriction to convex weights.
Although our method is applicable to fairly general situation, the usage or performance can be limitedwhen it is difﬁcult or inefﬁcient to directly sample from the target measure µ .
So there are interestingfollow up questions; for example, one could study if one can replace the i.i.d.
samples with smartersampling (DPP, importance sampling, etc) before the recombination is carried out.
Also, the currenttheory does not fully explain the empirical superiority; especially the 1 / √ (cid:96) term does not match theexperiments.
Nevertheless, we believe our method is the ﬁrst generally applicable algorithm with aguarantee from the spectral decay, while the issue with (cid:96) should be addressed in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 4.(c)
Did you discuss any potential negative societal impacts of your work?
[No] This is
an improvement on a speciﬁc numerical method and we believe it does not have
a
forseeable negative social impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2and Appendix.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Section 2 andAppendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Included in the
supplementary material as a Zip ﬁle.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 3.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figures in Section 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the footnote 3 in page 7 for theresource.
We also included the computational time in each Figure (for the largest case)
as well as the supplementary material (for all cases).4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3 and the
[Table]
README ﬁle in the supplementary material.
[Table]
(b) Did you mention the license of the assets?
[Yes] See the README ﬁle in the supple- [Table]
mentary material.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] [Table] See the README ﬁle in the supplementary material. [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A Outline of the Appendix
Appendix B contains general results from which the results presented in the main text, in particular
Theorem 1, follow as special cases.
Appendix C contains the proofs of these theoretical results andneeded technical lemmas.
Appendix D shows that if the expectations (cid:82) ϕ i ( x ) d µ ( x ) are known,then this knowledge can be used to further improve the theoretical bounds; it also gives a simplemodiﬁcation of Algorithm 1 doing this efﬁciently.
Appendix E provides additional numericalexperiments and benchmarks.
B Theoretical Results and Remarks
In this section, we present theoretical results that include our main results as a special case.
Theproofs are given in Section C.
Notation.
For simplicity, for a quadrature Q n given by points ( x i ) ni
=1 and weights ( w i ) ni
=1 and aprobability measure µ , we denote the integration of an integrable function f on X with respect tothese measures by
[Equation]
respectively.
We also write the inner-product and norm of an RKHS H k by (cid:104)· , ·(cid:105) H k and (cid:107)·(cid:107)
H k .
Furthermore, we use the probability simplex ∆ n and convex hull conv A of a set A ⊂ R d in theproofs:
[Equation]
B.1 Quantitative Results
We work under the following setting as in the assumption of Theorem 1.
Assumption A. µ is a Borel probability measure on X , and k is a positive semi-deﬁnite kernel on
X such that (cid:82) X
k
( x, x ) d µ ( x ) < ∞ .
Further, k 0 is a positive semi-deﬁnite kernel on X such thatk 1 :=
k − k 0 is a positive semi-deﬁnite kernel on X .
The following is a general result regarding a quadrature formula exactly integrating functions in H k 0 .
Theorem 5.
Under Assumption A, if an n -point convex quadrature Q n on X satisﬁes Q n ( f ) = µ ( f )for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g is the function given by g ( x ) = (cid:112) k 1 ( x, x ) .
In particular, the following assertions hold forsuch a quadrature Q n :
(a) We have wce( Q n ; H k , µ ) ≤ 2 (cid:107) g (cid:107) ∞
= 2 sup x ∈X (cid:112) k 1 ( x, x ) .
(b) If we additionally have Q n ( g ) ≤ µ ( g ) , then we have wce( Q n ; H k , µ ) ≤ 2 µ ( g ) .
(c) If we additionally have Q n ( g 2 ) ≤ µ ( g 2 ) instead of (b), we still have
[Equation]
Remark 3.
For a Borel probability measure ν on X and a nonnegative function h : X → R ≥ 0 ,we have an inequality (cid:82) X (cid:112) h ( x ) d ν ( x ) ≤ (cid:0)(cid:82)
X h ( x ) d ν ( x ) (cid:1) 1 / 2 , so the above µ ( g ) can be upperbounded by (cid:82) X k 1 ( x, x ) d µ ( x ) , which is equal to the trace of the integral operator given by k 1 .
Also, the assumption in Theorem 5 can be weakened to the integrability of (cid:112) k ( x, x ) from the sameinequality as you can see in the proof.
We can combine Theorem 5 with an empirical approximation of µ to obtain the following result,which essentially implies Theorem 1.
Theorem 6.
Under Assumption A, let D N be a set of N independent samples from µ , and (cid:101) µ N beits empirical measure, i.e., (cid:101) µ N = 1 N (cid:80)
y ∈ D N δ y .
Then, if an n -point convex quadrature Q n on Xsatisﬁes Q n ( f ) = (cid:101) µ N ( f ) for any f = k 0 ( · , x ) with x ∈ X , we have
[Equation]
where g ( x ) := (cid:112) k 1 ( x, x ) and c k,µ :
= (cid:82) X
k
( x, x ) d µ ( x ) − (cid:82)(cid:82)
X×X k ( x, y ) d µ ( x ) d µ ( y ) .
In particular, the following assertions hold for such a quadrature Q n :
[Equation]
We
[Equation]
(b) If we additionally always require Q n ( g ) ≤ (cid:101) µ N ( g ) , then we have
[Equation]
The requirement Q n ( g ) ≤ (cid:101) µ N ( g ) can be replaced by Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 )
Although we have assumed k − k 0 is positive semi-deﬁnite in the previous assertions, the uniformbound works without the assumption as follows.
Proposition 1.
Let µ be a Borel probability measure on X .
Let k and k 0 be positive semi-deﬁnitekernels on X satisfying (cid:82) X (cid:112) k
( x, x ) d µ ( x ) , (cid:82) X (cid:112) k
( x, x ) d µ ( x ) < ∞ .
If an n -point convex quadrature Q n on X satisﬁes Q n ( f ) = µ ( f ) for any k 0 ( · , x ) with x ∈ X , we have
[Equation]
Furthermore, if dim H k 0 < n , there exists an n -point convex quadrature Q n satisfying Q n ( f ) = µ ( f )for each f = k 0 ( · , x ) .
In this paper, we focus on the cases where k 0 is either given by the truncated Mercer decomposition or
Nyström approximation.
For many important kernels, however, we may also use the random Fourierfeatures [52] or its periodic version [61] which can easily be combined with Proposition 1, but it isbeyond the scope of this paper to choose its appropriate variant for each kernel [see 38, for a list ofvariants].
B.2 Existence Results
The existence of quadrature formulas satisfying the estimate of Theorem 5 or Theorem 6 is guaranteedwhen dim H k 0 < n .
Theorem 7.
Under Assumption A, if dim H k 0 < n , there exists an n -point convex quadrature Q nsatisfying Q n ( f ) = µ ( f ) for each f = k 0 ( · , x ) .
This still holds even if we additionally require
Q n ( g ) ≤ µ ( g ) or Q n ( g 2 ) ≤ µ ( g 2 ) for g ( x ) =
(cid:112) k 1 ( x, x ) .
Remark 4.
This also implies the existence result of Q n satisfying the condition in Theorem 6 if wereplace µ by (cid:101) µ N .
The algorithm for constructing a kernel quadrature with Theorem 6 is given in the main body,see Algorithm 1.
The ones with Theorem 5 requires further knowledge of the expectation of testfunctions, i.e., the values of (cid:82) X ϕ i ( x ) d µ ( x ) with H k 0 ⊂ span { ϕ 1 , . . .
, ϕ n − 1 } .
Under this additionalinformation, we have an algorithm (Algorithm 2) based on random sampling given in the followingsection.
B.3 Eigenvalue estimate for Gaussian kernels
We provide a proof of a folklore estimate on the eigenvalues of integral operators given by a Gaussiankernel.
Let k ( x, y ) =
exp( − 12 (cid:96) 2 ( x − y ) 2 ) for an (cid:96) > 0 and x, y ∈ R .
Then, the it has the followingexpansion [43, 29]: [Equation] Let µ be a Borel probability measure supported on a compact domain, i.e., µ ( { x ∈ R | | x | > R } ) =
0for some R > 0 .
Let us consider the RKHS given by k over X := supp µ .
Recall that σ n is the n -th eigenvalue of the integral operator
[Equation]
From the minimax property of eigenvalues of compact Hermitian operators, we have
[Equation]
where the orgthogonal complement is taken in terms of L 2 ( µ ) -inner product and ϕ m are functionsgiven in (16).
They are indeed in L 2 ( µ ) as µ is compactly supported.
Now, let k n ( x, y ) :
= (cid:80) ∞
m = n − 1 ϕ m ( x ) ϕ m ( y ) .
For an f ∈ L 2 ( µ ) ∩ { ϕ 0 , . . .
, ϕ n − 2 } ⊥ , we have
[Equation]
(positive deﬁniteness
[Equation]
[Equation]
Therefore, we have the estimate σ n ≤ (cid:82) X k n ( x, x ) d µ ( x ) .
We have
[Equation]
and this can be regarded as the remainder term of the Maclaurin expansion, there is a θ ∈ (0 , 1) suchthat
[Equation]
In particular, if we have | x | ≤ R for µ -almost all x , we have a factorial decay σ n ≤ ( R/(cid:96) ) 2( n
[Equation]
1) ( n − 1)! .
B.4 Uniform Robustness
In applications, the RKHS H k may be mis-speciﬁed and the quadrature rule µ Q when computed forthe mis-speciﬁed function class H k but applied to a function f / ∈
H
k leads only to the attainablebound
[Equation]
via triangle equality and standard integral estimates.
Note that | · | TV denotes the total variation normand the above applies to any (cid:101)
f ∈ H k ; in particular, to the best approximation in uniform norm to fin H k .
Since µ is a probability measure, | µ | TV = 1 but if µ Q is a signed measure with non-convexweights, its total variation | µ Q | TV can be large, resulting in arbitrary large integration errors.
B.5 Equivalence between the projection/matrix Nyström approximations
Let k be a positive semi-deﬁnite kernel on X , Z = ( z i ) (cid:96)i =1 ⊂ X .
Let P Z : H k
→ H k be theprojection operator onto span { k ( · , z i ) | i = 1 , . . .
, (cid:96) } .
For arbitrary x, y ∈ X , we can write
P Z k ( · , x ) = (cid:96) (cid:88) i = a i k ( · , z i ) ,
P Z k ( · , y ) =
(cid:96) (cid:88)
i
=
b i k ( · , z i ) , where a = ( a i ) (cid:96)i =1 , b
= ( b i ) (cid:96)i =1 ∈ R (cid:96) .
From the properties of projection, we havek ( z j , x ) =
(cid:104) k ( · , z
[Equation]
) , k ( · , x ) (cid:105) H k
= (cid:104) k ( · , z j ) , P Z k ( · , x ) (cid:105)
H k
= (cid:96) (cid:88)
i = a i k ( z j , z i ) . In matrix notation, we have k ( Z, x ) =
k ( Z, Z ) a , and k ( Z, y )
=
k ( Z, Z ) b from the same argument.
Thus, by conbining it with the property of Moore–Penrose inverse, we have(cid:104) P
[Equation]
k ( · , x ) , P Z k ( · , y ) (cid:105)
[Equation]
= a (cid:62) k
( Z, Z ) b
[Equation]
(Moore–Penrose) =
k
[Equation]
x
[Equation]
Z
[Equation]
k ( Z
[Equation]
y ) .
This shows the desired equivalence.
B.6 Intuition behind Recombination
Let us brieﬂy provide the intuition behind the deterministic recombination algorithms (Remark 1). We can solve a problem of ‘reducing (weighted) 2 d points to d points in R d while keeping thebarycenter’ by using a linear programming or its variant. If we apply this to 2 d points each givenby a barycenter of approximately N 2 d points, we can reduce the original problem (of size N ) to aproblem of size d · N 2 d = N 2 .
So we can just continue the same procedure log 2 ( N/d ) times to obtainthe desired measure.
C ProofsC.1 Proof of Theorem 5
Before proceeding to the proof of the theorem, we prepare a couple of assertions.
The following is awell-known estimate proven by using the Cauchy–Schwarz inequality [see e.g., 45, Lemma 3.1 andits proof].
Proposition 2.
Let k be a positive semi-deﬁnite kernel on X , and ν be a Borel probability measurewith (cid:82) X (cid:112)
k
( x, x ) d ν ( x ) < ∞ .
Then, for each f ∈ H k , we have
[Equation]
By using the proposition, we obtain the following technical lemma.
Lemma 1.
Let k and k 1 be a positive semi-deﬁnite kernels on X such that k − k 1 is also positivesemi-deﬁnite.
Let ν be a Borel probability measure on X .
Then, for any n ≥ 1 , a 1 , . . .
, a n ∈ R ,x 1 , . . .
, x n ∈ X , if we let f = (cid:80) ni
=1 a i k ( · , x i ) and f 1 = (cid:80) ni
=1 a i k 1 ( · , x i ) , then we have [Equation] x
[Equation]
Proof.
From the positive semi-deﬁniteness of k 0 :
= k − k 1 , we have
[Equation]
= n (cid:88) [Equation] Hence it sufﬁces to prove | ν ( f 1 ) | ≤ (cid:107) f 1 (cid:107) H k 1 ν ( g ) for g ( x ) :=
(cid:112) k 1 ( x, x ) , but it directly followsfrom Proposition 2.
Proof of Theorem 5.
Note ﬁrst that, for each f ∈ H k 0 , f is integrable with respect to µ .
Indeed, wehave
[Equation]
and it is integrable from assumption, so the equality Q n ( f )
= µ ( f ) with f = k 0 ( · , x ) is attained at aﬁnite value.
Once we establish (14), the item (b) is clear, and (a) follows from the fact that Q n ( g ) and µ ( g ) areboth integrals of the function g with respect to a probability measure.
Also, (c) is justiﬁed as follows:
[Equation]
where µ ( Q n ) 2 ≤ µ ( Q 2 n ) and µ ( g ) 2 ≤ µ ( g 2 ) follows from the Cauchy–Schwarz.
To prove (14),we ﬁrst prove
[Equation]
17) for any f of the form f = (cid:80) n i =1 a i k ( · , x i ) with n ≥ 0 and a 1 , . .
. , a n ∈ R .
Given such anf , we have Q n ( f 0 ) =
µ ( f 0 ) for f 0 :=
(cid:80) ni
=1 a i k 0 ( · , x i ) from the assumption.
Thus, by lettingf 1 := f − f
= (cid:80) ni
=1 a i k 1 ( · , x i ) , we have
[Equation]
As we have | ν ( f 1 ) | ≤ (cid:107) f
(cid:107) H k ν
( g ) for ν = Q n , µ from Lemma 1, we obtain | Q n ( f 1 ) − µ ( f 1 ) | ≤(cid:107)
f (cid:107) H k
( Q n ( g ) + µ ( g )) , and so (17) is shown for f of the form f = (cid:80) ni
=1 a i k ( · , x i ) .
Finally, we generalize (17) to any f ∈ H k .
Let (cid:101) f
∈
H k can be written in the form (cid:80) ni
=1 a i k ( · , x i ) .
If we let h ( x ) = (cid:112) k
( x, x ) , from Proposition 2, we have [Equation] [Equation] k µ
[Equation]
Note that µ ( h ) < ∞ follows from the integrability of k ( x, x ) in Assumption A.
Therefore, we have
[Equation]
As we can make (cid:107) f − (cid:101) f (cid:107) H k arbitrarily small from the deﬁnition of H k , the proof of (14) is completedby taking the limit.
C.2 Proof of Theorem
Proof.
Denote D N = { y 1 , . . .
, y N } and note that the result follows from (15) and
[Equation]
where the ﬁrst inequality is given by the Cauchy–Schwarz.
Indeed, (a) is an immediate consequence of (15) and Q n and (cid:101) µ N making a probability measure, and(b) is obtained as 2 E
[Equation]
x, x ) d µ ( x ) by using (18) andthe requirement Q n ( g ) ≤ (cid:101) µ N ( g ) .
When the requirement is Q n ( g 2 ) ≤ (cid:101) µ N ( g 2 ) , as we have Q n ( g ) 2 ≤ Q n ( g 2 ) and (cid:101) µ N ( g ) 2 ≤ (cid:101) µ N ( g 2 )by the Cauchy–Schwarz, we also have by the AM–GM,
[Equation]
For showing (15), we remark that we always have
[Equation]
by applying Theorem 5 with (cid:101) µ N instead of µ .
Let h ( µ ) , h ( (cid:101) µ N ) , h ( Q n ) ∈
H
k be the kernel mean embeddings of µ , (cid:101) µ N and µ Q n , i.e.,
[Equation]
where ( w i ) ni
=1 and ( x i ) ni
=1 are weights and points deﬁning the quadrature Q n .
Remark that h ( µ ) iswell-deﬁned as (cid:82) X k ( x, x ) d µ ( x ) < ∞ [45, Lemma 3.1].
As we can rewrite the worst-case error as
[Equation]
by triangle inequality and the AM–GM, we obtain
[Equation]
where we have used (19) in the last inequality.
It now sufﬁces to prove E (cid:2) (cid:107) h ( µ ) − h ( (cid:101) µ N ) (cid:107) 2 H k (cid:3) =
c k,µ /N for showing (15).
Indeed, we have
[Equation]
since (cid:82)(cid:82) X×X k
( x, y ) d µ ( x ) d µ ( y ) =
(cid:82) X E [ k ( x, y i )]
d µ ( x ) =
E [ k ( y i , y j )] holds for i (cid:54) =
j .
Thus,the proof is completed.
C.3 Proof of Proposition
Proof.
As Q n exactly integrates the functions in H k 0 , we have wce( Q n ; H k 0 , µ ) = 0 .
So, if we set
Q n ( f ) =
(cid:80) ni
=
w i f ( x i ) , then we have, from (13) with kernel k 0 [-,
-],0 =
[Equation]
If we extract this from the formula (13) for the kernel k , we have, by letting k 1 :=
k − k 0 ,
[Equation]
So, if we deﬁne M := sup x ∈X | k 1 ( x, y
[Equation]
| = sup x ∈X | k ( x, y ) − k 0 ( x, y ) | , we havewce( Q n ; H k , µ
[Equation]
≤
[Equation]
as Q n is a convex quadrature.
The existence follows from almost the same proof as in the proof of
Theorem 5, but in this case it directly follows from Tchakaloff’s thorem [59, 4].
C.4 Proof of Theorem 7
Proof.
We prove the existence of the version Q n ( g ) ≤ µ ( g ) .
The other follows just by replacingevery g in the proof below by g 2 .
Let ϕ 1 , . . .
, ϕ n − 1 ∈ H k 0 satisfy H k
= span { ϕ 1 , . . .
, ϕ n − 1 } .
Also, let y, y 1 , y 2 , . .
. be independent samples from µ .
Now, consider the vector-valued function ψ = ( ϕ 1 , . . .
, ϕ n − 1 , g ) (cid:62)
∈ R n .
Note that E [ (cid:107) ψ ( y ) (cid:107) ] < ∞ follows from the integrability of elements in H k 0 and g with respectto µ .
Therefore, by [21, Theorem 11], with probability 1 , there exists an N such that E [ ψ ( y )]
∈conv { ψ ( y 1 ) , . . .
, ψ ( y N ) } .
So, in particular, there exist deterministic points x 1 , . . .
, x N ∈ X satisfying E [ ψ ( y )]
∈ conv { ψ ( x 1 ) , . .
, ψ ( x N ) } .
For such ( x i ) Ni =1 , consider an optimal solution that isalso a basic feasible solution of the following linear programming problem:minimize
[Equation]
subject to [
[Equation]
where ϕ
= (1 , ϕ 1 , . . .
, ϕ n − 1 ) (cid:62) ∈ R n is another vector-valued function (note that its ﬁrst coordinateis constant so that any feasible solution of (21) sums up to one).
Such a basic solution w has at mostn nonzero entries, say ( w i 1 , . .
, w i n ) ∈ ∆ n with 1 ≤ i 1 < · · · < i n ≤ N .
Then, the quadrature Q ngiven by weights ( w i j ) nj =1 and points ( x i j ) nj =1 satisﬁes Q n ( ϕ ) =
µ ( ϕ ) and Q n ( g ) ≤ µ ( g ) .
Thelatter follows from the optimality of w and the fact that E [ ψ ( y )] ∈ conv { ψ ( y 1 ) , . . .
, ψ ( y N ) } leadsto a feasible solution with the objective E [ g ( y )]
= µ ( g ) ).
C.5 Proof of Theorem
We prove the theorem by using an existing bound regarding the Nystöm approximation for matrices,which is more common in the machine learning literature.
Let A =
( A ij ) Ni,j =1 ∈ R N × N be a symmetric positive semi-deﬁnite matrix.
Let us denote it as
A =
[ a 1 , . .
. , a N ] by using a 1 , . .
. , a N ∈ R N .
Then, we independently sample i 1 , . .
, i (cid:96) from{ 1 , . . .
, N } uniformly, and construct a submatrix B
=
( a i j i k ) (cid:96)j,k
=1 .
If we let B s be the best rank- sapproximation of B and B + s its pseudoinverse, the matrix(cid:101)
A =
[ a [Equation]
[Equation]
s
[Equation]
(22) works as a rank- s approximation of A .
We use the following result on this matrix version:
Proposition 3 ([33, Theorem 2]) .
For a positive semi-deﬁnite matrix A , the rank- s approximation (cid:101)
Agiven above satisﬁes, with probability at least 1 − δ , the following:
[Equation]
δ
[Equation]
where β ( (cid:96), N ) =
1 − 1 2max { (cid:96),N − (cid:96) } , A max = max i A ii ,
D A max = max i,j ( A ii + A jj − 2 A ij ) and A sis the best rank- s approximation of A .
As D A max ≤ 2 A max , if we have N ≥ 2 (cid:96) , it holds that
[Equation]
A
[Equation]
and we can just state
[Equation]
We show the following lemma as a consequence of this proposition.
Lemma 2.
Let s ≤ (cid:96) be positive integers and δ > 0 .
Let k : X
×
X be a symmetric and positivesemi-deﬁnite kernel and y 1 , y 2 , . .
. be i.i.d.
random variables taking values in X .
For each N , deﬁnethe N × N matrices K ( N ) , K s ( N ) , K Zs ( N ) by
[Equation]
where Z = ( y 1 , . .
, y (cid:96) ) .
Then, there exists a sequence ε N → 0 such that
[Equation]
is met with probability at least 1 − δ − ε N .
Proof.
We assume N ≥ 2 (cid:96) .
Let i 1 , . .
, i (cid:96) be independent uniform samples from { 1 , . .
. , N } .
Consider the event E N that i 1 , . .
, i (cid:96) are all different.
Then, P ( E N ) = (cid:81) (cid:96)i =
N +1 − i N convergesto 1 as N → ∞ , and let ε N = 1 − P
( E N ) .
By using Proposition 3, (24) and max i K ( N ) ii ≤
N − 1 sup x k ( x, x ) , we have that the probability
P (cid:32)
[Equation]
is at least (1 − δ − ε N ) / P ( E N ) ≥
1 − δ − ε N , where (cid:101) K s ( N ) is the rank- s Nyström approximationof the matrix K ( N ) by using indices i 1 , . .
, i (cid:96) .
From (22), if we take (cid:102) W = k
( y i j , y i k )
(cid:96)j,k
=1 and (cid:102) W sits best rank- s approximation, it actually satisﬁes
[Equation]
where D =
( y i 1 , . . .
, y i (cid:96) ) and k Ds is the Nyström approximation given in the main body.
As y 1 , . . .
, y N are i.i.d.
samples, we can see that ( Z, ( y i ) Ni =1 ) (without any conditioning) and( D, ( y i )
Ni =1 ) conditioned on E N actually have the same distribution, so we are done.
We ﬁnally prove the result for the Nyström approximation of integral operators.
Proof of Theorem 3.
Take a sufﬁciently large N and let us use K ( N ) , K s ( N ) , K Zs ( N ) deﬁned inthe previous lemma with y 1 , y 2 , . . .
independently sampled from µ .
It sufﬁces to consider the case C k := sup x ∈X k ( x, x ) < ∞ .
It is clear that K s ( N ) ii ≤ K ( N ) ii ≤
C k /N , and from (10), we also have
[Equation]
=
k
[Equation]
x ) , and so K Z s ( N ) ii ≤ C k /N .
For a matrix A ( N ) ∈ R N × N deﬁned by A ( N ) ij =
(1 − δ ij )( K ( N ) − K Zs ( N )) , i.e., the matrixgiven by deleting the diagonal, we have (cid:107) A ( N ) (cid:107) 2 → (cid:107)K Zs − K(cid:107) as N → ∞ almost surely [32,
Theorem 3.1].
Since we have observed that (cid:107) K ( N ) − K Zs ( N ) − A ( N ) (cid:107) 2 ≤ C k /N , we have
[Equation]
almost surely.
The same argument yields (cid:107) K ( N ) − K s ( N ) (cid:107) 2 → σ s +1 , as it converges to the normof the integral operator given by the kernel (cid:80) m ≥ s +
σ m e m ( x ) e m ( y ) .
Now, by letting A N be the event that (24) holds (so P ( A N ) ≥ 1 − δ − ε N ), the desired inequality(12) almost surely holds under the event lim sup A N = (cid:84) N>(cid:96)
(cid:83) M ≥ N A M .
Indeed, under this eventwe can just take the limit of both sides of (24) for an appropriate subsequence of (2 (cid:96), 2 (cid:96) + 1 , . . . ) .
Aswe have
[Equation]
the proof is completed.
D Kernel Quadrature when Expectations are Known
When we use an approximate kernel k 0 and know exact expectation of test functions ϕ 1 , . . .
, ϕ n − 1with
H k 0 ⊂ span { ϕ 1 , . . .
, ϕ n − 1 } , we can obtain an n -point kernel quadrature that exactly integratesϕ 1 , . . .
, ϕ n − 1 by Algorithm 2.
Algorithm w (cid:62) k 1 , diag ( x ) subject to [ ϕ ( y 1 ) · · · ϕ ( y N )] w = (cid:90)
X ϕ ( x ) d µ ( x ) , w ≥ 0 , 1 (cid:62) w = 1 , | w | 0 ≤ n.
(25) to obtain points { x 1 , . . .
, x n } ⊂ { y 1 , . . . , y N } and weights ( w i ) ∈
∆ n satisfying n (cid:88) i = w i ϕ ( x i ) =
(cid:90)
X ϕ ( x ) d µ ( x ) if (25) is feasible.
We make several remarks on this algorithm.
First, the problem (25) is, strictly speaking, not a linearprogramming (LP), as it includes the sparsity constraint | w | 0 ≤ n .
However, as it only contains nequality constraints, its basic feasible solution always satisﬁes | w | 0 ≤ n and the simplex algorithmautomatically gives such a sparse (and optimal) solution even if we do not explicitly impose thisconstraint, so we call it an LP for simplicity.
Second, this algorithm occasionally fails to output Q nas, with some probability, the LP has no feasible solution.
Although we can repeat the algorithm untilwe succeed, the number N should be chosen appropriately.
See Remark 5 for this point.
Finally,our algorithm has possibly related approaches such as sparse optimization and Sard’s method, see
Remark 6 and 7.
Remark 5.
A simple approach for constructing a quadrature formula [21] was recently proposed:randomly sample candidate points and ﬁnd a solution by using a linear programming (LP) solver.
Indeed, for an independent sample y 1 , . . .
, y N ∼ µ , we can construct a quadrature formula withconvex weights exactly integrating the functions in F = span { ϕ 1 , . . .
, ϕ n − 1 } using a subset of thesepoints if and only if we have
[Equation]
where ϕ
= ( ϕ 1 , . . .
, ϕ n − 1 ) (cid:62) :
X → R n − 1 and conv A denotes the convex hull of A .
Several sharpestimates for the probability of the event (26) are available in Hayakawa et al.
Under the event(26) , we can ﬁnd a desired rule by using the simplex method for the LP problem (25) .
Remark 6.
From the viewpoint of subsampling, a direct way to obtain quadrature formulas withconvex weights supported on a small number of points, is to ﬁrst sample N candidate points
D N = ( x 1 , . .
, x N ) and then solve the following sparse optimization problem:
[Equation]
where k ( D N , D N ) is the corresponding N × N Gram matrix.
Unfortunately, exactly solving thisproblem is computationally challenging, in particular in contrast to our approach that exploitsthe spectral properties of k and µ .
Nevertheless, one could use sparse optimization to obtain an approximate solution of (27) : although the simplex constraint ( w ≥ 0 , 1 (cid:62) w = 1 ) makesit impossible to exploit the classical (cid:96) 1 regulatization, there are possible alternatives under thisconstraint [51, 34, 36] or use the DC (difference of convex functions) algorithm to incorporate thesparsity constraint to ﬁnd a local minima [17].
This is a promising research direction, and ourgeneral sample estimates might provide a ﬁrst step towards this direction.
Remark 7.
Sard’s method [53, 35] for constructing numerical integration rules uses the n degreeof freedom (of choosing weights in our setting) separately; m ( ≤ n ) for exactness over a certainm -dimensional space of test functions, and the remaining n − m for minimizing an error criterionsuch as the worst-case error.
In the context of kernel quadrature, one way to use Sard’s method withexactness over F (an m -dimensional space of test functions) is as follows [28, 56]:
[Equation]
subject to w (cid:62) f ( D
[Equation]
=
[Equation]
F
[Equation]
(28) where D n = ( x 1 , . . .
, x n ) (cid:62) , f ( D n ) =
( f ( x 1 ) , . . .
, f ( x n )) (cid:62) .
This amounts to solving a convexquadratic programming for w in an ( n − m ) -dimensional subspace of R n (without constraint).
Thisis similar to our approach in that it enforces exactness in a certain ﬁnite-dimensional space of testfunctions.
One key difference is that Sard’s approach aims for a quadrature formula on a given set ofpoints, whereas our method determines also the points themelves.
Hence, the combination of thesetwo approaches seems to be an interesting future research topic.
Computational complexity.
A tricky part of this approach, essentially based on random convexhulls, is that the algorithm possibly does not output a quadrature formula.
Hence, the followingquantity plays an important role to estimate the essential complexity of the algorithm:
[Equation]
where y, y 1 , y 2 , . .
. are independent samples from µ .
This value is known to be ﬁnite and estimatedunder a variety of conditions on ϕ ( y ) [64, 22].
If we have some knowledge of µ , we can justkeep trying the algorithm with N = N ϕ until it succeeds, and its expected computational time is
O ( nN ϕ + C ( n, N ϕ )) , where C ( a, b ) is the (expected) cost of solving an a × b LP with a simplexmethod.
Note that, though the worst-case computational time of the simplex method is exponential, itis empirically O ( ab min { a, b } ) in practice [50, 54].
In addition, N ϕ
= O ( n ) holds in examples withsome symmetry [66, 21], so in that case we have a heuristic complexity estimate of O (cid:0) n 3 (cid:1) .
Choice of approximate kernels.
Similarly to the empirical version discussed in the main text, weprove quantitative estimates when k 0 is given by the Mercer approximation or Nyström approximation.
Remark that the necessary information for using these methods is different.
Whereas using the Mercerapproximation requires the knowledge of Mercer decomposition k
( x, y ) = (cid:80) ∞
m =1 σ m e m ( x ) e m ( y )and their exact integration (cid:82)
X
e m ( x ) d µ ( x ) , the Nyström approximation only requires the exactintegral values of kernel, (cid:82) X
k ( x, y ) d µ ( y ) , and so is more generally applicable.
See the followingsections for details.
In the following, we assume that the kernel attains the Mercer decomposition k
( x, y ) = 795 (cid:80) ∞ m =1 σ m e m ( x ) e m ( y ) , where σ 1 ≥ σ 2 ≥ · · · ≥ 0 and ( e m ) ∞ m =1 is an orthonormal set of L 2 ( µ ) .
D.1 Algorithm 2 with Mercer Approximation
If we use the truncated Mercer decomposition as an approximate kernel, we have the following result.
Theorem 8.
If Algorithm 2 with k 0 :
= (cid:80) n − 1 m =1 σ m e m ( x ) e m ( y ) and ϕ i
=
e i successfully outputs aconvex quadrature Q n , then it satisﬁes the following:
(a) If C := sup m ≥ 1 (cid:107) e m (cid:107) ∞ < ∞ , we have wce( Q n ; H k , µ ) 2 ≤ 4 C 2 (cid:80) ∞ m = n σ m
(b) As N in Algorithm 2 tends to inﬁnity, we have
[Equation]
Proof.
As in the proof of Corollary 2, k − k 0 is positive semi-deﬁnite.
Thus, when (cid:80) ∞ m = n σ m < ∞ ,the kernel and the measure µ satisﬁes Assumption A.
So Thorem 5(a) implies (a) of this theorem,since k 1 ( x, x ) = (cid:80) ∞ m = n σ m e m ( x ) 2 ≤ C 2 (cid:80) ∞ m = n σ m .
For (b), if we have Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) , then Theorem 5 implies
[Equation]
So it sufﬁces to prove P ( Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ))
→ 1 as N → ∞ , and it is shown by consideringthe optimal basic feasible solution of the LP (25) and the following fact [22, Proposition 4]:
[Equation]
where ψ = ( ϕ 1 , . .
ϕ n − 1 , k 1 , diag ) (cid:62) .
Indeed, under the event ψ ( y ) ∈ conv { ψ ( y 1 ) , . .
. , ψ ( y N ) } ,the LP becomes feasible and Q n ( k 1 , diag ) ≤ µ ( k 1 , diag ) follows from the optimality.
See the proof of
Theorem 7 (Section C.4) for a more detailed explanation if necessary.
Note that the boundedness of C is a typical assumption [see 40, Assumption 3.2 and referencestherein], while it does not necessarily hold [44, Section 3].
Under some assumptions, we can quantifythe probability that the LP (25) becomes feasible.
Sampling bound.
Suppose 1 is an eigenfunction of K .
This is satisﬁed, e.g., in the following cases:
• µ is a Haar measure on a compact group and k is shift-invariant.
• k is a kernel based on Stein’s identity [47, 56, 1] with respect to µ .
In this case, we have a theoretical bound of the required N in Algorithm 2 as follows.
Theorem 9.
Suppose 1 is an eigenfunction of K , i.e., (cid:82) X k ( · , y )
d µ ( y ) is a constant function.
Then,for each n ≥ 2 and N ≥ 6( n − 1) sup x ∈X (cid:80) n − 1 m =1 e m ( x ) 2 , Algorithm 2 returns a feasible quadraturewith probability at least 1 − 2 1 − n , i.e., for an independent sample y 1 , . . .
, y N from µ , we have
[Equation]
where ϕ
=
( e 1 , . . .
, e n − 1 ) (cid:62) .
If the value C = sup m ≥ 1 (cid:107) e m (cid:107) ∞ is ﬁnite, N ≥ 6 C ( n − 1) 2 is alsosufﬁcient for the above estimate.
Proof.
This follows from the existing results [22, Theorem 14 and Proposition 17].
D.2 Algorithm 2 with Nyström Approximation
Although the method discussed in the previous section requires the knowledge of Mercer decomposition, if we make use of the Nyström approximation, we only require the values of (cid:82) X k ( x, y )
d µ ( y )for x ∈ X .
Recall that k Zs ( x, y ) is the rank- s Nyström approximation of the kernel k based on the point set
Z = ( z 1 , . . .
, z (cid:96) ) .
From (11), we can use ϕ Zi := u (cid:62) i k
( Z, · ) as test functions.
Theorem 10.
Let n ≤ (cid:96) and δ > 0 , and let Z be an (cid:96)
-point
independent sample from µ .
If
Algorithm 2 with k 0 =
k Zn − 1 and ϕ i = ϕ
Zi successfully outputs a convex quadrature Q n , then withprobability at least 1 − δ , we have
[Equation]
Proof.
As in the proof of Corollary 4, k − k Zn − 1 is positive semi-deﬁnite.
Also, we can assume
[Equation]
∞ m =1 σ m < ∞ , as otherwise the right-hand side is inﬁnity.
Thus k and k 0 =
k Zn − 1 satisfy
Assumption A.
Note that for a function of the form c ( x, y )
= a · b ( x ) b ( y ) with a ∈ R and b ∈ L 2 ( µ ) , and anorthonomal set ( f i )
i ∈ I ⊂
L 2 ( µ ) of L 2 ( µ ) with b ∈ span { f i | i ∈ I } , we have
[Equation]
i
[Equation]
If we here use the orthonormal set ( e m ) ∞ m =1 that appears in the Mercer decomposition, by lettingk 1 :
= k − k Zn − 1 and using the linear extension of (29), we have
[Equation]
Then, by combining this with Theorem 3 and Theorem 5(c), we obtain the desired estimate.
Remark 8.
If we denote by N ϕ the required number of samples, the computational complexity ofthe above algorithm becomes O (cid:0) n(cid:96)N
ϕ
+ n(cid:96)
2 + C ( n, N ϕ ) (cid:1) , including the cost of computing
the
Nyström approximation as well as test functions at N ϕ samples (see also Remark 2).
E Additional Numerical Experiments
In this section, we provide additional experiments on Algorithm 2 using random convex hulls, aswell as the approximated version of the N. + emp described in Remark 2.
Section E.1 showsthe comparison of Algorithm 2 (with Mercer/Nyström approximation) with some of the methodsmentioned in the main text under the periodic Sobolev spaces with uniform measure.
Section E.2investigates Algorithm 2 (with Nystöm approximation) as well as the approximate but fast algorithmfor N. + emp , under the setting of empirical measure reduction.
E.1 Periodic Sobolev Spaces with Uniform Measure
We conducted experiments under the same setting as in Section 3.1, except that we additionally havethe following methods:
Nyström, Nyström + opt: We used the same test functions as N. + emp with the random set Z ofsize (cid:96) = 10 n , but for Algorithm 2.
We used N = 10 n samples for the LP (25).
In Nyström+ opt we additionally optimized the convex weights using (13).Mercer, Mercer + opt ( d = 1 ): We used the same test functions as M. + emp , but for Algorithm 2.We used N = 10 n samples for the LP (25).
In Mercer + opt we additionally optimized theconvex weights using (13).
The results are given in Figure 3.
The weights of Nyström and Mercer are already almost optimizedas they exactly integrate a certain family of functions, so the additional CQP (13) does not change theerror so much.
Surprising is that N. + emp + opt is almost as good as Nyström + opt or even better.
This implies that the recombination points with respect to a moderately large ( N = n 2 in this case)empirical measure can provide a good convergence rate in Bayesian quadrature [24], even though the(equally weighted) empirical measure itself is not that close to the true measure.
[Figure]
The worst computational time per one trial was 5 seconds of N. + emp and N. + emp + opt in ( d, r, n ) =
(3 , 3 , 80) , while Nyström and Nyström + opt ran in 0.9 seconds under the same setting.
There were no infeasible LPs
[Figure]
Odd behavior of ‘Mercer’.
As we can see in Figure 3(a,b), the methods based on the exact Mercerdecomposition becomes very close to optimal when n = 15 , 65 .
As it seemed to be caused by theparity of n , we carried out another experiment for n ∈ { 5 , 9 , 15 , 19 , 29 , 39 , 49 , 65 , 79 } (Figure 4),then Mercer and its optimization clearly became the best methods except the exact optimal Unifrom
Grid .
It might be related to the structure of the periodic Sobolev space, that has, for each eigenvalueexcept for 1 , two-dimensional eigenspace ( cos and sin ), but needs further investigation.
Also, inthe case ( d, r ) =
(1 , 3) , we see ‘ + opt ’ make the quadrature less accurate for a big n , but it istheoretically almost impossible, so it seems to be caused by numerical accuracy of the CQP solver.
E.2 Measure Reduction in Machine Learning Datasets
We conducted experiments under the same setting as in Section 3.2.
We additionally adopted
Nyström, Nyström + opt (with N = 20 n ), and FNE, FNE + opt , where FNE (stands for ‘fast N.+ emp’) is the approximate algorithm for N. + emp by omitting the inequality in (6) and using therandomized SVD [19] (see Remark 2).
[Figure]
The results are given in Figure 5.
N. + emp + opt and FNE + opt show almost the same convergence.
While in the largest case n = 160 , the average runtime of ( N. + emp + opt , FNE + opt )was (13 . 0 , 2 . 07) seconds in 3D Road Network data and (12 . 8 , 2 .
) seconds in Power Plant data,respectively.
Although our theoretical guarantee no longer holds for FNE , it accelerates the algorithm while surprisingly maintaining the accuracy.
Nystöm or Nyström + opt behave much better than iid
Bayes , but are slightly less accurate than N
+ emp + opt and FNE + opt , whereas they have goodtheoretical guarantees (Theorem 10).
Their computational time was basically between that of FNE opt