G ENERALIZED K ERNEL T HINNING
A BSTRACT
The kernel thinning (KT) algorithm of Dwivedi and Mackey (2021) compresses a probability distribution more effectively than independent sampling by targeting a reproducing kernel Hilbert space (RKHS) and leveraging a less smooth squareroot kernel.
Here we provide four improvements.
First, we show that KT applied directly to the target RKHS yields tighter, dimension-free guarantees for any kernel, any distribution, and any fixed function in the RKHS.
Second, we show that, for analytic kernels like Gaussian, inverse multiquadric, and sinc, target KT admits maximum mean discrepancy (MMD) guarantees comparable to or better than those of square-root KT without making explicit use of a square-root kernel.
Third, we prove that KT with a fractional power kernel yields better-thanMonte-Carlo MMD guarantees for non-smooth kernels, like Laplace and Mat ´ ern, that do not have square-roots.
Fourth, we establish that KT applied to a sum of the target and power kernels (a procedure we call KT+) simultaneously inherits the improved MMD guarantees of power KT and the tighter individual function guarantees of target KT.
In our experiments with target KT and KT+, we witness significant improvements in integration error even in 100 dimensions and when compressing challenging differential equation posteriors.
I NTRODUCTION
A core task in probabilistic inference is learning a compact representation of a probability distribution P .
This problem is usually solved by sampling points x 1 , . . .
, x n independently from P or, if direct sampling is intractable, generating n points from a Markov chain converging to P .
The benefit of these approaches is that they provide asymptotically exact sample estimates
P in f ≜ 1 n (cid:80) ni =1 f ( x i ) for intractable expectations P f ≜ E X ∼ P [ f ( X )] .
However, they also suffer from a serious drawback: the learned representations are unnecessarily large, requiring n points to achieve | P f − P in f | =
Θ( n − 12 ) integration error.
These inefficient representations quickly become prohibitive for expensive downstream tasks and function evaluations: for example, in computational cardiology, each function evaluation f ( x i ) initiates a heart or tissue simulation that consumes 1000s of CPU hours (Niederer et al., 2011; Augustin et al., 2016; Strocchi et al., 2020).
To reduce the downstream computational burden, a standard practice is to thin the initial sample by discarding every t -th sample point (Owen, 2017).
Unfortunately, standard thinning often results in a substantial loss of accuracy: for example, thinning an i.i.d.
or fast-mixing Markov chain sample from n points to n 12 points increases integration error from Θ( n − 12 ) to Θ( n − 14 ) .
The recent kernel thinning (KT) algorithm of Dwivedi & Mackey (2021) addresses this issue by producing thinned coresets with better-than-i.i.d. integration error in a reproducing kernel Hilbert space (RKHS, Berlinet & Thomas-Agnan, 2011).
Given a target kernel 1 k and a suitable sequence of input points S in = ( x i ) ni
=1 approximating P , KT returns a subsequence S out of √ n points with better-than-i.i.d.
maximum mean discrepancy (MMD, Gretton et al., 2012),
[Equation]
[Equation] [Equation] where ∥·∥ k denotes the norm for the RKHS H associated with k .
That is, the KT output admits o ( n − 14 ) worst-case integration error across the unit ball of H .
KT achieves its improvement with high probability using non-uniform randomness and a less smooth square-root kernel k rt satisfying
[Equation]
When the input points are sampled i.i.d.
or from a fast-mixing Markov chain on R d , Dwivedi &
Mackey prove that the KT output has, with high probability, O d ( n − 12 √ log n ) - MMD k error for
P and k rt with bounded support, O d ( n − 12 (log d +1 n log log n ) 12 ) - MMD k error for P and k rt with light tails, and O d ( n − 12 + d 2 ρ √ log n log log n ) - MMD k error for P and k 2 rt with ρ > 2 d moments.
Meanwhile, an i.i.d.
coreset of the same size suffers Ω( n − 14 ) MMD k .
We refer to the original KT algorithm as ROOT KT hereafter.
Our contributions
In this work, we offer four improvements over the original KT algorithm.
First, we show in Sec.
2.1 that a generalization of KT that uses only the target kernel k provides a tighter O ( n − 12 √ log n ) integration error guarantee for each function f in the RKHS.
This TARGET
KT guarantee (a) applies to any kernel k on any domain (even kernels that do not admit a squareroot and kernels defined on non-Euclidean spaces), (b) applies to any target distribution P (even heavy-tailed P not covered by ROOT KT guarantees), and (c) is dimension-free , eliminating the exponential dimension dependence and (log n ) d/ 2 factors of prior ROOT KT guarantees.
Second, we prove in Sec.
2.2 that, for analytic kernels, like Gaussian, inverse multiquadric (IMQ), and sinc, TARGET KT admits MMD guarantees comparable to or better than those of Dwivedi &
Mackey (2021) without making explicit use of a square-root kernel.
Third, we establish in Sec.that generalized KT with a fractional α -power kernel k
α yields improved MMD guarantees for kernels that do not admit a square-root, like Laplace and non-smooth Mat ´ ern.
Fourth, we show in
Sec. 3 that, remarkably, applying generalized KT to a sum of k and k α —a procedure we call kernel thinning+ (KT+)—simultaneously inherits the improved MMD of POWER KT and the dimensionfree individual function guarantees of TARGET KT.
In Sec. 4, we use our new tools to generate substantially compressed representations of both i.i.d.
samples in dimensions d = 2 through 100 and Markov chain Monte Carlo samples targeting challenging differential equation posteriors.
In line with our theory, we find that TARGET KT and KT+ significantly improve both single function integration error and MMD, even for kernels without fast-decaying square-roots.
[Table]
Related work For bounded k , both i.i.d.
samples (Tolstikhin et al., 2017, Prop. A.1) and thinned geometrically ergodic Markov chains (Dwivedi & Mackey, 2021, Prop. 1) deliver n 12 points with
O ( n − 14 ) MMD with high probability.
The online Haar strategy of Dwivedi et al.
and low discrepancy quasi-Monte Carlo methods (see, e.g., Hickernell, 1998; Novak & Wozniakowski, 2010;
Dick et al., 2013) provide improved O d ( n − 12 log d n ) MMD guarantees but are tailored specifically to the uniform distribution on [0 , 1] d .
Alternative coreset constructions for more general P include kernel herding (Chen et al., 2010), discrepancy herding (Harvey & Samadi, 2014), super-sampling with a reservoir (Paige et al., 2016), support points convex-concave procedures (Mak & Joseph, 2018), greedy sign selection (Karnin & Liberty, 2019, Sec. 3.1), Stein point MCMC (Chen et al., 2019), and Stein thinning (Riabiz et al., 2020a).
While some admit better-than-i.i.d.
MMD guarantees for finite-dimensional kernels on R d (Chen et al., 2010; Harvey & Samadi, 2014), none apart from KT are known to provide better-than-i.i.d.
MMD or integration error for the infinitedimensional kernels covered in this work.
The lower bounds of Phillips & Tai (2020, Thm. 3.1) and
Tolstikhin et al.
(2017, Thm. 1) respectively establish that any procedure outputting n 12 -sized coresets and any procedure estimating P based only on n i.i.d.
sample points must incur Ω( n − 12 ) MMD in the worst case.
Our guarantees in Sec.
2 match these lower bounds up to logarithmic factors.
Notation We define the norm ∥ k ∥ ∞ =
sup x,y | k ( x, y ) | and the shorthand [ n ] ≜ { 1 , . . . , n } , R + ≜ { x ∈ R : x ≥ 0 } , N 0 ≜ N ∪ { 0 } , B k ≜ { f ∈ H : ∥ f ∥ k ≤ 1 } , and B 2 ( r ) ≜ (cid:8)
y ∈ R d : ∥ y ∥ 2 ≤ r (cid:9) .
We write a ≾ b and a ≿ b to mean a = O ( b ) and a = Ω( b ) , use ≾ d when masking constants dependent on d , and write a = O p ( b ) to mean a/b is bounded in probability.
For any distribution Q and point sequences S , S ′ with empirical distributions Q n , Q ′ n , we define MMD k ( Q , S ) ≜
MMD k ( Q , Q n ) and MMD k ( S , S ′ ) ≜ MMD k
( Q n , Q ′ n ) .
G ENERALIZED K ERNEL T HINNING
Our generalized kernel thinning algorithm (Alg. 1) for compressing an input point sequence S in = ( x i ) n i =1 proceeds in two steps: KT - SPLIT and KT - SWAP detailed in App.
A. First, given a thinning parameter m and an auxiliary kernel k split , KT - SPLIT divides the input sequence into 2 m candidate coresets of size n/ 2 m using non-uniform randomness.
Next, given a target kernel k , KT - SWAP selects a candidate coreset with smallest MMD k to S in and iteratively improves that coreset by exchanging coreset points for input points whenever the swap leads to reduced MMD k .
When k split is a square-root kernel k rt (2) of k , generalized KT recovers the original ROOT KT algorithm of
Dwivedi & Mackey.
In this section, we establish performance guarantees for more general k split with special emphasis on the practical choice k split = k .
Like ROOT KT, for any m , generalized
KT has time complexity dominated by O ( n 2 ) evaluations of k split and k and O ( n min( d, n )) space complexity from storing either S in or the kernel matrices ( k split ( x i , x j )) ni,j =1 and ( k ( x i , x j )) ni,j =1 .
k split target kernel sequence in
) ni = ∈ probabilities ( δ i ) ⌊ n/ 2 ⌋ i =
( S ( m,ℓ ) )
2 m ℓ =1 ← KT - SPLIT ( k split , S in , m, ( δ i ) ⌊ n/ 2 ⌋ i =1 ) //
Split S in into 2 m candidate coresets of size ⌊ n 2 m ⌋
S KT ← KT - SWAP ( k , S in , ( S ( m,ℓ ) ) 2 m ℓ =1 ) //
Select best coreset and iteratively refine return coreset S KT of size ⌊ n/ 2 m ⌋
2. S INGLE FUNCTION GUARANTEES FOR KT - SPLIT
We begin by analyzing the quality of the KT - SPLIT coresets.
Our first main result, proved in App.
B, bounds the KT - SPLIT integration error for any fixed function in the RKHS H split generated by k split .
Theorem 1 (Single function guarantees for KT - SPLIT ) Consider KT - SPLIT (Alg.
1a) with oblivious 3 S in and ( δ i )
n/
i
{+=
+}
and δ ⋆ ≜ min i δ i .
If n 2 m ∈ N , then, for any fixed f ∈ H split , index ℓ ∈ [2 m ] , and scalar δ ′ ∈ (0 , 1) , the output coreset S ( m,ℓ ) with P ( ℓ ) split ≜ 1 n/ 2 m (cid:80) x ∈S ( m,ℓ ) δ x satisfies | P in f − P ( ℓ ) split f | ≤ ∥ f
∥ k split · σ m (cid:113) 2 log( 2 δ ′ ) for σ m ≜ 2 √ 3 2 m n (cid:113) ∥
k split ∥ ∞ , in · log( 6 m 2 m δ ⋆ ) with probability at least p sg ≜ 1 − δ ′ − (cid:80) mj
=1 2 j − 1 m (cid:80) n/ 2 j i =
 δ i .
Here, ∥ k split ∥ ∞ , in ≜ max x ∈S in k split ( x,x ) .
Remark 1 (Guarantees for known and oblivious stopping times) By Dwivedi & Mackey (2021,
App. D), the success probability p sg is at least 1 − δ if we set δ ′ =
δ 2 and δ i = δ n for a stopping time n known a priori or δ i =
mδ 2 m +2 ( i +1)log
( i +1) for an arbitrary oblivious stopping time n .
When compressing heavily from n to √ n points, Thm. 1 and Rem.
1 guarantee O ( n − 12 √ log n ) integration error with high probability for any fixed function f ∈ H split .
This represents a near-quadratic improvement over the Ω( n − 14 ) integration error of √ n i.i.d.
points.
Moreover, this guarantee applies to any kernel defined on any space including unbounded kernels on unbounded domains (e.g., energy distance (Sejdinovic et al., 2013) and Stein kernels (Oates et al., 2017; Chwialkowski et al., 2016;
Liu et al., 2016; Gorham & Mackey, 2017)); kernels with slowly decaying square roots (e.g., sinc kernels); and non-smooth kernels without square roots (e.g., Laplace, Mat´ern with γ ∈ ( d 2 , d ] ), and the compactly supported kernels of Wendland (2004) with s < 12 ( d +1) ).
In contrast, the MMD guarantees of Dwivedi & Mackey covered only bounded, smooth k on R d with bounded, Lipschitz, and rapidly-decaying square-roots.
In addition, for ∥ k ∥ ∞ = 1 on R d , the MMD bounds of Dwivedi & Mackey feature exponential dimension dependence of the form c d or (log n ) d/ 2 while the Thm.guarantee is dimension-free and hence practically relevant even when d is large relative to n .
Thm. 1 also guarantees better-than-i.i.d.
integration error for any target distribution with | P f −
P in f | =
o ( n − 14 ) .
In contrast, the MMD improvements of Dwivedi & Mackey (2021, cf.
Tab. 2) applied only to P with at least 2 d moments.
Finally, when KT - SPLIT is applied with a squareroot kernel k split = k rt , Thm. 1 still yields integration error bounds for f ∈ H , as H ⊆
H split .
However, relative to target KT - SPLIT guarantees with k split =
k , the error bounds are inflated by a multiplicative factor of (cid:113) ∥ k rt ∥ ∞ , in ∥ k ∥ ∞ , in ∥ f ∥ k rt ∥ f ∥ k .
In App.
G, we show that this inflation factor is at least 1 for each kernel explicitly analyzed in Dwivedi & Mackey (2021) and grows exponentially in dimension for Gaussian and Mat ´ ern kernels, unlike the dimension-free target KT - SPLIT bounds.
2. MMD GUARANTEE FOR TARGET KT
Our second main result bounds the MMD k (1)—the worst-case integration error across the unit ball of H —for generalized KT applied to the target kernel, i.e., k split = k .
The proof of this result in
App.
C is based on Thm. 1 and an appropriate covering number for the unit ball B k of the k RKHS.
Definition 1 ( k covering number)
For a set A and scalar ε > 0 , we define the k covering number
N k
( A , ε ) with M k
( A , ε ) ≜ log N k
( A , ε ) as the minimum cardinality of a set C ⊂ B k satisfying
[Equation]
Theorem 2 (MMD guarantee for TARGET KT) Consider generalized KT (Alg. 1) with k split =
k , oblivious S in and ( δ i ) ⌊ n/ 2 ⌋ i =1 , and δ ⋆ ≜ min i δ i .
If n 2 m ∈ N , then for any δ ′ ∈ (0 , 1) , the output coreset
S KT is of size n 2 m and satisfies
MMD k ( S in , S KT ) ≤
[Equation]
with probability at least p
sg , where ∥ k ∥ ∞ , in and p sg were defined in Thm. 1.
When compressing heavily from n to √ n points, Thm. 2 and Rem. 1 with ε = (cid:113) ∥ k ∥ ∞ , in n and
A = B 2 ( R in ) for R in ≜ max x ∈S in ∥ x ∥ 2 guarantee
[Equation]
with high probability.
Thus we immediately obtain an MMD guarantee for any kernel k with a covering number bound.
Furthermore, we readily obtain a comparable guarantee for P since
MMD k ( P , S KT ) ≤ MMD k
( P , S in )
+MMD k
( S in , S KT ) .
Any of a variety of existing algorithms can be used to generate an input point sequence S in with MMD k
( P , S in ) no larger than the compression bound (5), including i.i.d.
sampling (Tolstikhin et al., 2017, Thm. A.1), geometric MCMC (Dwivedi & Mackey, 2021, Prop. 1), kernel herding (Lacoste-Julien et al., 2015, Thm. G.1), Stein points (Chen et al., 2018, Thm. 2), Stein point MCMC (Chen et al., 2019, Thm. 1), greedy sign selection (Karnin & Liberty, 2019, Sec. 3.1), and Stein thinning (Riabiz et al., 2020a, Thm. 1).
2. C ONSEQUENCES OF T HM .
Tab. 2 summarizes the MMD guarantees of Thm. 2 under common growth conditions on the log covering number M k and the input point radius R S in ≜ max x ∈S in ∥ x ∥ 2 .
In Props.
2 and 3 of
App.
I, we show that analytic kernels, like Gaussian, inverse multiquadric (IMQ), and sinc, have
L OG G ROWTH M k (i.e., M k ( B 2 ( r ) , ε ) ≾ d r d log ω ( 1 ε ) ) while finitely differentiable kernels (like
Mat´ern and B-spline) have P OLY G ROWTH M k
(i.e., M k ( B 2 ( r ) , ε ) ≾ d r d ε − ω ).
Our conditions on R S in arise from four forms of target distribution tail decay: (1) C OMPACT ( R S in ≾ d 1 ), (2) S UB G AUSS ( R S in ≾ d √ log n ), (3) S UB E XP ( R S in ≾ d log n ), and (4) H EAVY T AIL ( R S in ≾ d n 1 /ρ ) .
The first setting arises with a compactly supported P (e.g., on the unit cube [0 , 1] d ), and the other three settings arise in expectation and with high probability when S in is generated i.i.d.
from P with sub-Gaussian tails, sub-exponential tails, or ρ moments respectively.
Substituting these conditions into (5) yields the eight entries of Tab.
We find that, for L OGG ROWTH M k , TARGET KT MMD is within log factors of the Ω( n − 1 / 2 ) lower bounds of Sec.for light-tailed P and is o ( n − 1 / 4 ) (i.e., better than i.i.d.)
for any distribution with ρ > 4 d moments.
Meanwhile, for P OLY G ROWTH M k , TARGET KT MMD is o ( n − 1 / 4 ) whenever ω < 12 for lighttailed P or whenever P has ρ > 2 d/ ( 12 − ω ) moments.
[Table]
Next, for each of the popular convergence-determining kernels of Tab.
[Table]
we compare the ROOT
KT MMD guarantees of Dwivedi & Mackey (2021) with the TARGET KT guarantees of Thm.combined with covering number bounds derived in Apps.
I and J. We see in Tab.
that Thm.provides better-than-i.i.d.
and better-than- ROOT KT guarantees for kernels with slowly decaying or non-existent square-roots (e
[Table]
g., IMQ with ν < d 2 , sinc, and B-spline) and nearly matches known
ROOT KT guarantees for analytic kernels like Gauss and IMQ with ν ≥ d 2 , even though TARGET
KT makes no explicit use of a square-root kernel.
See App.
J for the proofs related to Tab. 3.
K ERNEL T HINNING +
We next introduce and analyze two new generalized KT variants: (i) POWER KT which leverages a power kernel k α that interpolates between k and k rt to improve upon the MMD guarantees of target
KT even when k rt is not available and (ii) KT+ which uses a sum of k and k α to retain both the improved MMD guarantee of k α and the superior single function guarantees of k .
First, we define a generalization of the square-root kernel (2) for shiftinvariant k .
Let (cid:98) f denote the generalized Fourier transform (GFT, Wendland, 2004, Def. 8.9) of order 0 of f : R d → R .
Definition 2 ( α -power kernel) Define k 1 ≜ k .
We say a kernel k 12 is a 12 -power kernel for k if k ( x, y ) =
(2 π ) − d/ 2 (cid:82) R d k 12 ( x, z ) k
( z, y ) dz .
For α ∈ ( 12 , 1) , a kernel k α ( x, y )
=
κ α ( x − y ) on R d is an α -power kernel for k ( x, y )
=
κ ( x − y ) if (cid:99) κ α = (cid:98) κ α .
By design, k 12 matches k rt (2) up to an immaterial constant rescaling.
Given a power kernel k α we define POWER KT as generalized KT with k split = k α .
Our next result (with proof in App. D) provides an MMD guarantee for POWER KT.
Theorem 3 (MMD guarantee for POWER KT) Consider generalized KT (Alg. 1) with k split =
k α for some α ∈ [ 1 2 , 1] , oblivious sequences S in and ( δ i ) ⌊ n/ 2 ⌋ i =1 , and δ ⋆ ≜ min i δ i .
If n 2 m ∈ N , then for
[Table]
any δ ′ ∈ (0 , 1) , the output coreset S KT is of size n 2 m and satisfies
[Equation]
(6) with probability at least p sg (defined in Thm. 1).
The parameters (cid:102) M α and R max are defined in
App. D and satisfy (cid:102)
M α
= O d ( √ log n ) and R max = O d (1) for compactly supported P and k α and (cid:102)
M α
= O d ( √ log n log log n ) and R max =
O d (log n ) for subexponential P and k α , when δ ⋆ = δ ′ n .
Thm. 3 reproduces the ROOT KT guarantee of Dwivedi & Mackey (2021, Thm. 1) when α = 12 and more generally accommodates any power kernel via an MMD interpolation result (Prop. 1) that may be of independent interest.
This generalization is especially valuable for less-smooth kernels like
L APLACE and M AT ´ ERN ( ν, γ ) with ν ∈ ( d 2 , d ] that have no square-root kernel.
Our TARGET KT
MMD guarantees are no better than i.i.d.
for these kernels, but, as shown in App.
J, these kernels have M AT ´ ERN kernels as α -power kernels, which yield o ( n − 14 ) MMD in conjunction with Thm. 3.
Our final KT variant, kernel thinning+ , runs KT - SPLIT with a scaled sum of the target and power kernels, k † ≜ k / ∥ k ∥ ∞ + k α / ∥
k α ∥ ∞ .
Remarkably, this choice simultaneously provides the improved MMD guarantees of Thm. 3 and the dimension-free single function guarantees of Thm. 1 (see App.
E for the proof).
Theorem 4 (Single function & MMD guarantees for KT+) Consider generalized KT (Alg. 1) with k split = k † , oblivious S in and ( δ i ) ⌊ n/ 2 ⌋ i =1 , δ ⋆ ≜ min i δ i , and n 2 m ∈ N .
For any fixed function f ∈ H , index ℓ ∈ [2 m ] , and scalar δ ′ ∈ (0 , 1) , the KT - SPLIT coreset S ( m,ℓ ) satisfies
[Equation]
with probability at least p
sg
(for p sg and P ( ℓ ) split defined in Thm. 1).
Moreover,
[Equation]
i.i.d.
{+Root KT (Gauss) 17 .17 .
+}
{+Target KT (Gauss) 17 .17 .
+}
{+KT+ (IMQ) 17 .17 .
+}
{+KT+ (Laplace) 
+}
{+KT+ (Bspline) 
+}
[Figure] [Figure] with probability at least p sg , where M targetKT ( k ) denotes the right hand side of (4) with ∥ k ∥ ∞ , in replaced by ∥ k ∥ ∞ , and M powerKT ( k α ) denotes the right hand side of (6) .
As shown in Tab. 3, KT+ provides better-than-i.i.d.
MMD guarantees for every kernel in Tab. 1— even the Laplace, non-smooth Mat ´ ern, and odd B-spline kernels neglected by prior analyses—while matching or improving upon the guarantees of TARGET KT and ROOT KT in each case.
E XPERIMENTS
Dwivedi & Mackey (2021) illustrated the MMD benefits of ROOT KT over i.i.d.
sampling and standard MCMC thinning with a series of vignettes focused on the Gaussian kernel.
We revisit those vignettes with the broader range of kernels covered by generalized KT and demonstrate significant improvements in both MMD and single-function integration error.
We focus on coresets of size √ n produced from n input points with δ i = 12 n , let P out denote the empirical distribution of each output coreset, and report mean error ( ± 1 standard error) over 10 independent replicates of each experiment.
Target distributions and kernel bandwidths We consider three classes of target distributions on
R d : (i) mixture of Gaussians (MoG) P = 1 M (cid:80) Mj
=1 N ( µ j , I 2 ) with M component means µ j ∈ Rdefined in App.
H, (ii) Gaussian P = N (0 , I d ) for d ∈ { 2 , 10 , 20 , 50 , 100 } , and (iii) the posteriors of four distinct coupled ordinary differential equation models: the Goodwin (1965) model of oscillatory enzymatic control ( d = 4 ), the Lotka (1925) model of oscillatory predator-prey evolution ( d = 4 ), the Hinch et al.
(2004) model of calcium signalling in cardiac cells ( d = 38 ), and a tempered Hinch posterior.
For settings (i) and (ii) we use an input sequence S in and kernel bandwidths σ = /γ = √ 2 d
For (iii) we MCMC input sequences S in from 12 posterior inference experiments of Riabiz et al.
2020a)
and set the bandwidths σ 1 /γ as specified by Dwivedi &
Mackey [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
Function testbed To evaluate the ability of generalized KT to improve integration both inside and outside of H , we evaluate integration error for (a) a random element of the target kernel RKHS ( f ( x ) =
k ( X ′ , x ) described in App. H), (b) moments ( f ( x ) = x 1 and f ( x ) = x 21 ), and (c) a standard numerical integration benchmark test function from the continuous integrand family (CIF,
Genz, 1984), f CIF ( x ) = exp( − 1 d (cid:80) dj
=1 | x j − u j | ) for u j drawn i.i.d.
and uniformly from [0 , 1] .
Generalized KT coresets For an 8 -component mixture of Gaussians target P , the top row of Fig.highlights the visual differences between i.i.d.
coresets and coresets generated using generalized
KT.
We consider ROOT KT with G AUSS k , TARGET KT with G AUSS k , and KT+
( α = 0 .
7 ) with
L APLACE k , KT+
( α = 12 ) with IMQ k
( ν = 0 . 5 ), and KT+( α = 2 3 ) with B- SPLINE (5) k , and note that the B- SPLINE (5) and L APLACE k do not admit square-root kernels.
In each case, even for small n , generalized KT provides a more even distribution of points across components with fewer within-component gaps and clumps.
Moreover, as suggested by our theory, TARGET KT and ROOT KT coresets for G AUSS k have similar quality despite TARGET KT making no explicit use of a square-root kernel.
The MMD error plots in the bottom row of Fig.
1 provide a similar conclusion quantitatively, where we observe that for both variants of KT, the MMD error decays as n − 12 , a significant improvement over the n − 14 rate of i.i.d.
sampling.
We also observe that the empirical MMD decay rates are in close agreement with the rates guaranteed by our theory in Tab.
( n − 12 for G AUSS , B- SPLINE , and IMQ and n − 14 α = n − 0 .
36 for L APLACE ).
We provide additional visualizations and results in Figs.
4 and 5 of App. H, including MMD errors for M = 4 and M =component MoG targets.
The conclusions remain consistent with those drawn from Fig.
T ARGET KT vs. i.i.d.
sampling For Gaussian P and Gaussian k , Fig. 2 quantifies the improvements in distributional approximation obtained when using TARGET KT in place of a more typical i.i.d.
summary.
Remarkably, TARGET KT significantly improves the rate of decay and order of magnitude of mean MMD k
( P , P out ) , even in d = 100 dimensions with as few as 4 output points.
Moreover, in line with our theory, TARGET KT MMD closely tracks that of ROOT KT without using k rt .
Finally, TARGET KT delivers improved single-function integration error, both of functions in the RKHS (like k ( X ′ , · ) ) and those outside (like the first moment and CIF benchmark function), even with large d and relatively small sample sizes.
KT+ vs. standard MCMC thinning
For the MCMC targets, we measure error with respect to the input distribution P in (consistent with our guarantees), as exact integration under each posterior P is intractable.
We employ KT+
( α = 0 . 81 ) with L APLACE k for Goodwin and Lotka-Volterra and KT+ ( α = 0 . 5 ) with IMQ k
( ν = 0 .
for Hinch.
Notably, neither kernel has a squareroot with fast-decaying tails.
In Fig.
3, we evaluate thinning results from one chain targeting each of the Goodwin, Lotka-Volterra, and Hinch posteriors and observe that KT+ uniformly improves upon the MMD error of standard thinning (ST), even when ST exhibits better-than-i.i.d. accuracy.
Furthermore, KT+ provides significantly smaller integration error for functions inside of the RKHS (like k ( X ′ , · ) ) and outside of the RKHS (like the first and second moments and the benchmark CIF function) in nearly every setting.
See Fig.
6 of App.
H for plots of the other 9 MCMC settings.
[Table]
[Table]
a A D A - R W
L a p l a c e k
[Table]
[Figure]
D ISCUSSION AND C ONCLUSIONS
In this work, we introduced three new generalizations of the ROOT KT algorithm of Dwivedi &
Mackey (2021) with broader applicability and strengthened guarantees for generating compact representations of a probability distribution.
T ARGET KT - SPLIT provides √ n -point summaries with
O ( (cid:112) log n/n ) integration error guarantees for any kernel, any target distribution, and any function in the RKHS; POWER KT yields improved better-than-i.i.d.
MMD guarantees even when a square-root kernel is unavailable; and KT+ simultaneously inherits the guarantees of TARGET KT and POWER KT.
While we have focused on unweighted coreset quality we highlight that the same MMD guarantees extend to any improved reweighting of the coreset points.
For example, for downstream tasks that support weights, one can optimally reweight P out to approximate P in in O ( n 32 ) time by directly minimizing MMD k .
Finally, one can combine generalized KT with the C OMPRESS ++ meta-algorithm of Shetty et al. (2022) to obtain coresets of comparable quality in near-linear time.
R EPRODUCIBILITY S TATEMENT
See App. H for supplementary experimental details and results and the goodpoints Python package https://github.com/microsoft/goodpoints for Python code reproducing all experiments. A CKNOWLEDGMENTS
RD acknowledges the support by the National Science Foundation under Grant No. DMSfor the Foundations of Data Science Institute (FODSI)
A PPENDIX
A Details of KT - SPLIT and KT - SWAP
B Proof of Thm.
1: Single function guarantees for KT - SPLIT
C Proof of Thm. 2: MMD guarantee for TARGET KT
D Proof of Thm. 3: MMD guarantee for POWER KT
E Proof of Thm. 4: Single function & MMD guarantees for KT+
F Proof of Prop. 1: An interpolation result for MMD
G Sub-optimality of single function guarantees with root KT
H Additional experimental results
I Upper bounds on RKHS covering numbers
J Proof of Tab.
3 results
A D ETAILS OF KT - SPLIT AND KT - SWAP
Algorithm 1a: KT - SPLIT – Divide points into candidate coresets of size ⌊ n/ 2 m ⌋
Input: kernel k split , point sequence S in = ( x i ) ni
=1 , thinning parameter m ∈ N , probabilities ( δ i ) ⌊ n 2 ⌋ i =
S ( j,ℓ ) ← {} for 0 ≤ j ≤ m and 1 ≤ ℓ ≤ 2 j // Empty coresets: S ( j,ℓ ) has size ⌊ i 2 j ⌋ after round i σ j,ℓ ← 0 for 1 ≤ j ≤ m and 1 ≤ ℓ ≤ 2 j −// Swapping parameters for i = 1 , . . . , ⌊ n/ 2 ⌋ do
S (0 , 1) .append
( x i ); S (0 , 1) .append
( x 2 i ) //
Every 2 j rounds, add one point from parent coreset S ( j − 1 ,ℓ ) to each child coreset S ( j, 2 ℓ − 1) , S ( j, 2 ℓ ) for ( j = 1; j ≤ m and i/ 2 j − 1 ∈ N ; j = j + 1 ) do for ℓ = 1 , . .
, 2 j − 1 do ( S , S ′ )
← ( S ( j − 1 ,ℓ ) , S ( j, 2 ℓ − 1) ) ; ( x, ˜ x ) ← get last two points ( S ) // Compute swapping threshold a a , min end end end return function get swap params( σ, b , δ ) : a ← max( b σ √ 2 log(2 /δ ) , b 2 ) σ ← σ + b 2 (1+( b 2 − 2 a ) σ 2 / a 2 ) + return ( a , σ )
Input: kernel k , point sequence S in = ( x i ) ni
=1 , candidate coresets ( S ( m,ℓ ) )
2 m ℓ =
S ( m, 0) ← baseline thinning ( S in , size = ⌊ n/ 2 m ⌋ ) //
Compare to baseline (e.g., standard thinning)
S KT ← S ( m,ℓ ⋆ ) for ℓ ⋆ ← argmin ℓ ∈{ 0 , 1 ,..., 2 m } MMD k
( S in , S ( m,ℓ ) ) //
Select best candidate coreset //
Swap out each point in S KT for best alternative in S in for i = 1 , . . .
, ⌊ n/ 2 m ⌋ do
S KT [ i ] ← argmin z ∈S in MMD k
( S in , S KT with S KT [ i ] =
z ) endreturn S KT , refined coreset of size ⌊ n/ 2 m ⌋
B P ROOF OF T HM . 1: S INGLE FUNCTION GUARANTEES FOR KT - SPLIT
The proof is identical for each index ℓ , so, without loss of generality, we prove the result for the case ℓ = 1 .
Define
[Equation]
Next, we use the results about an intermediate algorithm, kernel halving (Dwivedi & Mackey, 2021,
Alg. 3) that was introduced for the analysis of kernel thinning.
Using the arguments from Dwivedi & Mackey (2021, Sec. 5.2), we conclude that KT - SPLIT with k split set as k and thinning parameter m , is equivalent to repeated kernel halving with kernel k for m rounds (with no Failure in any rounds of kernel halving).
On this event of equivalence, denoted by E equi , Dwivedi & Mackey (2021,
Eqns.
(50, 51)) imply that the function (cid:102) W m ∈ H is equal in distribution to another random function
W m , where W m is unconditionally sub-Gaussian with parameter
[Equation]
that is,
E
[Equation]
for all [Equation] [Equation] 9) where we note that the analysis of Dwivedi & Mackey (2021) remains unaffected when we replace ∥ k ∥ ∞ by ∥ k ∥ ∞ , in in all the arguments.
Applying the sub-Gaussian Hoeffding inequality (Wainwright, 2019, Prop. 2.5) along with (9), we obtain that
[Equation]
Call this event E sg .
As noted above, conditional to the event E equi , we also have
[Equation]
where d = denotes equality in distribution.
Furthermore, Dwivedi & Mackey (2021, Eqn. 48) implies that
[Equation]
Putting the pieces together, we have
[Equation]
as claimed.
The proof is now complete.
C P ROOF OF T HM . 2: MMD GUARANTEE FOR TARGET KT
First, we note that by design, KT - SWAP ensures
[Equation]
where S ( m, 1) denotes the first coreset returned by KT - SPLIT .
Thus it suffices to show that
MMD k ( S in , S ( m, 1) ) is bounded by the term stated on the right hand side of (4).
Let P (1) out ≜ 1 n/ 2 m (cid:80) x ∈S ( m, 1) δ x .
By design of KT - SPLIT , supp( P (1) out )
⊆ supp( P in ) .
Recall the set A is such that supp( P in ) ⊆
A .
Proof of (4) Let C ≜ C k ,ε ( A ) denote the cover of minimum cardinality satisfying (3).
Fix any f ∈ B k .
By the triangle inequality and the covering property (3) of C , we have
[Equation]
Applying Thm.
[Equation]
we have
[Equation]
(11) with probability at least 1 − δ ′ − (cid:80) mj
=1 2 j − 1 m (cid:80) n/ 2 j i =
δ i = p sg − δ ′ .
A standard union bound then yields that
[Equation]
probability at least p sg − δ ′ .
Since f ∈ B k was arbitrary, and C ⊂ B k and thus sup g ∈C ∥ g ∥ k ≤ 1 , we therefore have
[Equation]
with probability at least p sg − δ ′ as claimed.
D P ROOF OF T HM . 3: MMD GUARANTEE FOR POWER KT Definition of (cid:102) M α and R max
Define the k α tail radii,
R † k α ,n ≜ min { r : τ k α ( r ) ≤ ∥ k α ∥ ∞ n } , where τ k α ( R ) ≜ (sup x (cid:82) ∥
y ∥ 2 ≥ R k 2 α ( x, x − y ) dy ) 12 ,
R k α ,n ≜ min { r :sup [Equation] x [Equation]
y
[Equation]
k [Equation] [Equation] and the S in tail radii
[Equation]
Furthermore, define the inflation factor
[Equation]
where L k α denotes a Lipschitz constant satisfying | k α ( x, y ) − k α
( x, z ) | ≤ L k α ∥ y − z
∥ 2 for all x, y, z ∈ R d .
With the notations in place, we can define the quantities appearing in Thm. 3:
[Equation]
≜
[Equation]
k α
[Equation]
n,m, d,
[Equation]
⋆ ,
[Equation]
, R S in , k α ,
[Equation]
and R max ≜ max( R S in , R † k α ,n/ 2 m ) .
[Equation]
[Equation]
The scaling of these two parameters depends on the tail behavior of k α and the growth of the radii
R S in (which in turn would typically depend on the tail behavior of P ).
The scaling of (cid:102)
M α and R max stated in Thm.
3 under the compactly supported or subexponential tail conditions follows directly from Dwivedi & Mackey (2021, Tab. 2, App. I).
Proof of Thm.
The KT - SWAP step ensures that
[Equation]
where S ( m, 1) α denotes the first coreset output by KT - SPLIT with k split = k α .
Next, we state a key interpolation result for MMD k that relates it to the MMD of its power kernels (Def. 2) (see App.
F for the proof).
Proposition 1 (An interpolation result for MMD) Consider a shift-invariant kernel k that admits valid α and 2 α -power kernels
k α and k 2 α respectively for some α ∈ [ 12 , 1] .
Then for any two discrete measures P and Q supported on finitely many points, we have
[Equation]
Given Prop. 1, it remains to establish suitable upper bounds on MMDs of k α and k 2 α .
To this end, first we note that for any reproducing kernel k and any two distributions P and Q , H¨older’s inequality implies that
[Equation]
Now, let P in and P ( m, 1) α denote the empirical distributions of S in and S ( m, 1) α .
Now applying Dwivedi & Mackey (2021, Thm. 4(b)), we find that
[Equation]
[Equation]
with probability p sg − δ ′ , where (cid:102)
M k α was defined in (14).
We note that while Dwivedi & Mackey (2021, Thm. 4(b)) uses ∥ k α ∥ ∞ in their bounds, we can replace it by ∥
k α ∥ ∞ , in , and verifying that all the steps of the proof continue to be valid (noting that ∥ k α ∥ ∞ , in is deterministic given S in ).
Furthermore, Dwivedi & Mackey (2021, Thm. 4(b)) yields that
[Equation]
[Equation] with probability p sg − δ ′ [Equation] where we have once again replaced the term ∥ k α ∥ ∞ with ∥ k α ∥ ∞ , in for the same reasons as stated above.
We note that the two bounds (16) and (17) apply under the same high probability event as noted in Dwivedi & Mackey (2021, proof of Thm. 1, eqn.
Putting together the pieces [Equation] we find that
[Equation]
as claimed.
The proof is now complete
[Equation]
E P ROOF OF T HM . 4: S INGLE FUNCTION & MMD GUARANTEES FOR KT+
Proof of (7) First, we note that the RKHS H of k is contained in the RKHS H † of k † Berlinet & Thomas-Agnan (2011, Thm. 5).
Now, applying Thm. 1 with k split = k † for any fixed function f ∈ H ⊂ H † and δ ′ ∈ (0 , 1) , we obtain that
[Equation]
with probability at least p sg .
Here step (i) follows from the inequality ∥ k † ∥ ∞ ≤ 2 , and step (ii) follows from the inequality ∥
f ∥ k † ≤ (cid:112) ∥ k ∥ ∞ ∥ f ∥ k , which in turn follows from the standard facts that
[Equation]
see, e.g., Zhang & Zhao (2013, Proof of Prop. 2.5) for a proof of step (iii), Berlinet & ThomasAgnan (2011, Thm. 5) for step (iv).
The proof for the bound (7) is now complete.
□
Proof of (8) Repeating the proof of Thm.
2 with the bound (11) replaced by (7) yields that
[Equation]
with probability at least p sg .
Let us denote this event by E 1 .
To establish the other bound, first we note that KT - SWAP step ensures that
[Equation]
where S ( m, 1) KT+ denotes the first coreset output by KT - SPLIT with k split = k † .
We can now repeat the proof of Thm. 3, using the sub-Gaussian tail bound (7), and with a minor substitution, namely, ∥ k α ∥ ∞ , in replaced by 2 ∥ k α ∥ ∞ .
Putting it together with (19), we conclude that
[Equation]
with probability at least p sg .
Let us denote this event by E 2 .
Note that the quantities on the right hand side of the bounds (18) and (20) are deterministic given
S in , and thus can be computed apriori.
Consequently, we apply the high probability bound only for one of the two events E 1 or E 2 depending on which of the two quantities (deterministically) attains the minimum.
Thus, the bound (8) holds with probability at least p sg as claimed.
□
F P ROOF OF P ROP . 1: A N INTERPOLATION RESULT FOR MMD
For two arbitrary distributions P and Q , and any reproducing kernel k , Gretton et al.
(2012, Lem. 4) yields that
[Equation]
Let F denote the generalized Fourier transform (GFT) operator (Wendland (2004, Def. 8.9)).
Since k ( x, y ) = κ ( x − y ) , Wendland (2004, Thm. 10.21) yields that
[Equation]
Let (cid:98) κ ≜ F ( κ ) , and consider a discrete measure D =
(cid:80) ni
=1 w i
δ x i supported on finitely many points, and let D k ( x ) ≜ (cid:80) w i
k ( x, x i ) =
(cid:80) w i κ ( x − x i ) .
Now using the linearity of the GFT operator F , we find that for any ω ∈ R d ,
[Equation]
[Equation]
where we used the time-shifting property of GFT that F ( κ ( ·− x i ))( ω )
=
e −⟨ ω,x i ⟩ (cid:98) κ ( ω ) (proven for completeness in Lem. 1), and used the shorthand (cid:98) D ( ω ) ≜ ( (cid:80) ni
=1 w i e −⟨ ω,x i ⟩ ) in the last step.
Putting together (21) to (23) with D = P − Q , we find that
MMD 2 k ( P , Q ) =
[Equation]
where step (i) makes use of Jensen’s inequality and the fact that the function t (cid:55)→ t 1 − αα for t ≥is concave for α ∈ [ 12 , 1] , and step (ii) follows by applying (24) for kernels k α and k 2 α and noting that by definition F ( k α )
= (cid:98) κ α , and F ( k 2 α )
= (cid:98) κ 2 α .
Noting MMD is a non-negative quantity, and taking square-root establishes the claim (15).
Lemma 1 (Shifting property of the generalized Fourier transform) If (cid:98) κ denotes the generalized
Fourier transform (GFT) (Wendland, 2004, Def. 8.9) of the function κ : R d → R , then e −⟨· ,x i ⟩ (cid:98) κ denotes the GFT of the shifted function κ ( · − x i ) , for any x i ∈ R d .
Proof Note that by definition of the GFT (cid:98) κ (Wendland, 2004, Def. 8.9), we have
[Equation]
(25) for all suitable Schwartz functions γ (Wendland, 2004, Def. 5.17), where (cid:98) γ denotes the Fourier transform (Wendland, 2004, Def. 5.15) of γ since GFT and FT coincide for these functions (as noted in the discussion after Wendland (2004, Def. 8.9)).
Thus to prove the lemma, we need to verify that
[Equation]
(26) for all suitable Schwartz functions γ .
Starting with the right hand side of the display (26), we have [Equation]
e −⟨
[Equation]
x i
[Equation] [Equation] [Equation] where step (i) follows from the shifting property of the FT (Wendland, 2004, Thm. 5.16(4)), and the fact that the GFT condition (25) holds for the shifted function γ ( · + x i ) function as well since it is still a Schwartz function (recall that (cid:98) γ is the FT), and step (ii) follows from a change of variable.
We have thus established (26), and the proof is complete.
□
G S UB - OPTIMALITY OF SINGLE FUNCTION GUARANTEES WITH ROOT KT
Define (cid:101) k rt as the scaled version of k rt , i.e., (cid:101) k rt ≜ k rt / ∥ k rt ∥ ∞ that is bounded by 1 .
Then Zhang &
Zhao (2013, Proof of Prop. 2.3) implies that
[Equation]
And thus we also have H rt
= (cid:101)
H rt where H rt and (cid:101) H rt respectively denote the RKHSs of k rt and (cid:101) k rt .
Next, we note that for any two kernels k 1 and k 2 with corresponding RKHSs H 1 and H 2 with
H 1 ⊂ H 2 , in the convention of Zhang & Zhao (2013, Lem. 2.2, Prop. 2.3), we have
[Equation]
for
[Equation]
Consequently, we have
[Equation]
where in the last step, we have applied the bound (28) with ( k 1 , H 1 ) ← ( k , H ) and ( k 2 , H 2 )
← ( (cid:101) k rt , (cid:101) k rt ) since H
⊂ H rt
= (cid:101) k rt .
Next, we use (29) to the kernels studied in Dwivedi & Mackey (2021) where we note that all the kernels in that work were scaled to ensure ∥ k ∥ ∞ = 1 and in fact satisfied k ( x, x ) = 1 .
Consequently, the multiplicative factor stated in the discussion after Thm. 1, namely, (cid:113) ∥
k rt ∥ ∞ , in ∥ k ∥ ∞ , in ∥ f ∥ k rt ∥
f ∥ k can be bounded by (cid:113) λ
( H , (cid:101) H rt ) given the arguments above.
For k =
Gauss ( σ ) kernels, Zhang & Zhao (2013, Prop. 3.5(1)) yields that
[Equation]
For k = B-spline (2 β + 1 , γ ) with β ∈ 2 N + 1 , Zhang & Zhao (2013, Prop. 3.5(1)) yields that λ ( H , (cid:101) H rt )
= 1 .
For k = Mat´ern ( ν, γ ) with ν > d , some algebra along with Zhang & Zhao (2013, Prop 3.1) yields that
[Equation]
H A DDITIONAL EXPERIMENTAL RESULTS
This section provides additional experimental details and results deferred from Sec. 4.
Common settings and error computation To obtain an output coreset of size n 12 with n input points, we (a) take every n 12 -th point for standard thinning (ST) and (b) run KT with m = 12 log 2 n using an ST coreset as the base coreset in KT - SWAP .
For Gaussian and MoG target we use i.i.d.
points as input, and for MCMC targets we use an ST coreset after burn-in as the input (see App.
H for more details).
We compute errors with respect to P whenever available in closed form and otherwise use P in .
For each input sample size n ∈ (cid:8) 2 4 , 2 6 , . . .
, 2 14 (cid:9) with δ i = 12 n , we report the mean MMD or function integration error ± 1 standard error across 10 independent replications of the experiment (the standard errors are too small to be visible in all experiments).
We also plot the ordinary least squares fit (for log mean error vs log coreset size), with the slope of the fit denoted as the empirical decay rate, e.g., for an OLS fit with slope − 0 .
25 , we display the decay rate of n − 0 .
Details of test functions
We note the following: (a) For Gaussian targets, the error with CIF function and i.i.d.
input is measured across the sample mean over the n input points and √ n output points obtained by standard thinning the input sequence, since P f CIF does not admit a closed form.
(b) To define the function f : x (cid:55)→ k
( X ′ , x ) , first we draw a sample X ∼ P , independent of the input, and then set X ′ = 2 X .
For the MCMC targets, we draw a point uniformly from a held out data point not used as input for KT.
For each target, the sample is drawn exactly once and then fixed throughout all sample sizes and repetions.
H. M IXTURE OF G AUSSIANS E XPERIMENTS
Our mixture of Gaussians target is given by P = 1 M (cid:80)
Mj
=1 N ( µ j , I d ) for M ∈ { 4 , 6 , 8 } where
[Equation]
Two independent replicates of Fig.
1 can be found in Fig.
Finally,we display mean MMD ( ±standard error across ten independent experiment replicates) as a function of coreset size in Fig.for M = 4 , 6 component MoG targets.
The conclusions from Fig.
5 are identical to those from the bottom row of Fig.
1: TARGET KT and ROOT KT provide similar MMD errors with G AUSS k , and all variants of KT provide a significant improvement over i.i.d.
sampling both in terms of magnitude and decay rate with input size.
Morever the observed decay rates for KT+ closely match the rates guaranteed by our theory in Tab. 3.
H. MCMC EXPERIMENTS
Our set-up for MCMC experiments follows closely that of Dwivedi & Mackey (2021).
For complete details on the targets and sampling algorithms we refer the reader to Riabiz et al.
(2020a, Sec. 4).
Goodwin and Lotka-Volterra experiments From Riabiz et al.
(2020b), we use the output of four distinct MCMC procedures targeting each of two d = 4 -dimensional posterior distributions P : (1) a posterior over the parameters of the Goodwin model of oscillatory enzymatic control (Goodwin, 1965) and (2) a posterior over the parameters of the Lotka-Volterra model of oscillatory predatorprey evolution (Lotka, 1925; Volterra, 1926).
For each of these targets, Riabiz et al.
(2020b) provide 2 × 10 6 sample points from the following four MCMC algorithms: Gaussian random walk (RW), adaptive Gaussian random walk (adaRW, Haario et al., 1999), Metropolis-adjusted Langevin algorithm
(MALA, Roberts & Tweedie, 1996), and pre-conditioned MALA (pMALA, Girolami &
Calderhead, 2011).
{+i.i.d. Root KT (Gauss) 17 .17 .
 Target KT (Gauss) 17 .17 .
 KT+ (IMQ) 17 .17 .
+}
{+KT+ (Laplace) 
 KT+ (Bspline) 
    i.i.d. Root KT (Gauss) 17 .17 .
+}
{+Target KT (Gauss) 17 .17 .
 KT+ (IMQ) 17 .17 .
 KT+ (Laplace) 
 KT+ (Bspline) 
+}
Hinch experiments Riabiz et al.
(2020b) also provide the output of two independent Gaussian random walk MCMC chains targeting each of two d = 38 -dimensional posterior distributions P : (1) a posterior over the parameters of the Hinch model of calcium signalling in cardiac cells (Hinch et al., 2004) and (2) a tempered version of the same posterior, as defined by Riabiz et al.
(2020a,
App. S5.4).
Burn-in and standard thinning
We discard the initial burn-in points of each chain using the maximum burn-in period reported in Riabiz et al.
(2020a, Tabs.
S4 & S6, App. S5.4).
Furthermore, we also normalize each Hinch chain by subtracting the post-burn-in sample mean and dividing each coordinate by its post-burn-in sample standard deviation.
To obtain an input sequence S in of length n to be fed into a thinning algorithm, we downsample the remaining even indices of points using standard thinning (odd indices are held out).
When applying standard thinning to any Markov chain output, we adopt the convention of keeping the final sample point.
The selected burn-in periods for the Goodwin task were 820,000 for RW; 824,000 for adaRW; 1,615,000 for MALA; and 1,475,000 for pMALA.
The respective numbers for the Lotka-Volterra task were 1,512,000 for RW; 1,797,000 for adaRW; 1,573,000 for MALA; and 1,251,000 for pMALA.
Additional remarks on Fig. 3 When a Markov chain is fast mixing (as in the Goodwin and LotkaVolterra examples), we expect standard thinning to have Ω( n − 14 ) error. However, when the chain is slow mixing, standard thinning can enjoy a faster rate of decay due to a certain degeneracy of the chain that leads it to lie close to a one-dimensional curve.
In the Hinch figures, we observe these better-than-i.i.d. rates of decay for standard thinning, but, remarkably, KT+ still offers improvements in both MMD and integration error. Moreover, in this setting, every additional point discarded via
[Figure]
improved compression translates into thousands of CPU hours saved in downstream heart-model simulations.
In this section, we state several results on covering bounds for RKHSes for both generic and specific kernels. We then use these bounds with Thm. 2 (or Tab. 2) to establish MMD guarantees for the output of generalized kernel thinning as summarized in Tab. 3.
We first state covering number bounds for RKHS associated with generic kernels, that are either (a) analytic, or (b) finitely many times differentiable.
These results follow essentially from Sun & Zhou (2008); Steinwart & Christmann (2008), but we provide a proof in App.
I.2 for completeness.
Proposition 2 (Covering numbers for analytic and differentiable kernels)
The following results hold true.
(a) Analytic kernels: Suppose that k ( x, y ) =
κ ( ∥ x − y ∥ 22 ) for κ : R + → R real-analytic with convergence radius R κ , that is,
[Equation]
for some constant C κ , where κ ( j )+ denotes the right-sided j -th derivative of κ .
Then for any set A ⊂ R d and any ε ∈ (0 , 12 ) , we have
[Equation]
where r † ≜ min (cid:16) √ R κ 2 d , (cid:112) R κ + D
[Equation]
− D A (cid:17) , and D A ≜ max x,y ∈A ∥ x − y ∥ 2 .
[Equation]
(b) Differentiable kernels: Suppose that for X ⊂ R d , the kernel k : X × X
→ R is s -times continuously differentiable, i. e., all partial derivatives ∂ α,α k : X × X
→ R exist and are continuous for all multi-indices α ∈ N d 0 with | α | ≤ s .
Then, for any closed Euclidean ball [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
ST
[Figure]
¯ B 2 ( r ) contained in X and any ε > 0 , we have
[Equation]
for some constant c s,d, k that depends only on on s, d and k .
Next, we state several explicit bounds on covering numbers for several popular kernels.
See App. I. for the proof.
Proposition 3 (Covering numbers for specific kernels)
The following statements hold true.
(a)
When
k = G AUSS ( σ ) , we
have
[Equation]
[Equation]
where C G AUSS ,d ≜ (cid:0) 4 e + d d (cid:1) e − d ≤ (cid:26) 4 .for
d =0 .
· d 4 e e − d for d ≥≤ 30 for all d ≥ 1 .
(b) When k = M AT ´ ERN ( ν, γ ) , ν ≥ d 2 + 1 , then for some constant C M AT ´ ERN ,ν,γ,d , we have
[Equation]
(c) When k = IMQ ( ν, γ ) , we have
[Equation]
37) where
C IMQ ,ν,γ ≜ 4 log
(cid:16) 16 (2 ν +1)
[Equation]
+
[Equation] [Equation]
+ (cid:17) , and (cid:101) r
≜
min (cid:16)
[Equation]
(d) When k = SINC ( θ ) , then for ε ∈ (0 , 12 ) , we have
[Equation]
(e) When k = B- SPLINE (2 β + 1 , γ ) , then for some universal constant C B- SPLINE , we have
[Equation]
I. A UXILIARY RESULTS ABOUT RKHS AND E UCLIDEAN COVERING NUMBERS
In this section, we collect several results regarding the covering numbers of Euclidean and RKHS spaces that come in handy for our proofs.
These results can also be of independent interest.
We start by defining the notion of restricted kernel and its unit ball (Rudi et al.
(2020, Prop. 8)).
For
X ⊂ R d , let | X denotes the restriction operator.
That is, for any function f : R d → R , we have f | X : X → R such that f | A ( x ) =
f
( x ) for x ∈ X .
Definition 3 (Restricted kernel and its RKHS) Consider a kernel k defined on R d × R d with the corresponding RKHS H , any set X ⊂ R d .
The restricted kernel k |X is defined as k | X : X × X
→ R such that k | X ( x, y ) ≜ k | X×X ( x, y )
=
k ( x, y ) for all x, y ∈ X , and H| X denotes its RKHS.
For f ∈ H| X , the restricted RKHS norm is defined as follows:
[Equation]
= inf h ∈H ∥ h ∥ k such that h | X = f.
Furthermore, we use B k |X ≜ { f ∈ H| X : ∥ f ∥ k
| X ≤ 1 } to denote the unit ball of the RKHS corresponding to this restricted kernel.
In this notation, the unit ball of unrestricted kernel satisfies B k ≜ B k | R d .
Now, recall the RKHS covering number definition from Def. 1.
In the sequel, we also use the covering number of the restricted kernel defined as follows:
[Equation]
that is N † k
( X , ε ) denotes the minimum cardinality over all possible covers C ⊂ B k |X that satisfy
[Equation]
With this notation in place, we now state a result that relates the covering numbers N † (42) and
N Def. 1.
Lemma 2 (Relation between restricted and unrestricted RKHS covering numbers)
We have
[Equation]
Proof Rudi et al.
(2020, Prop. 8(d,f)) imply that there exists a bounded linear extension operator
E : H| X
→ H with operator norm bounded by 1 , which when combined with Steinwart &
Christmann (2008, eqns.
(A.38), (A.39)) yields the claim.
□
Next, we state results that relate RKHS covering numbers for a change of domain for a shift-invariant kernel.
We use B ∥·∥ ( x ; r ) ≜ (cid:8) y ∈ R
d : ∥ x − y ∥ ≤ r (cid:9) to denote the r radius ball in R d defined by the metric induced by a norm ∥·∥ .
Definition 4 (Euclidean covering numbers)
Given a set X ⊂ R d , a norm ∥·∥ , and a scalar ε > 0 , we use N ∥·∥ ( X , ε ) to denote the ε -covering number of X with respect to ∥·∥ -norm.
That is,
N ∥·∥ ( X , ε ) denotes the minimum cardinality over all possible covers C ⊂ X that satisfy
X ⊂ ∪ z ∈C B ∥·∥ ( z ; ε ) .
When ∥·∥ = ∥·∥ q for some q ∈ [1 , ∞ ] , we use the shorthand N q ≜ N ∥·∥ q .
Lemma 3 (Relation between RKHS covering numbers on different domains) Given a shiftinvariant kernel k , a norm ∥·∥ on R d , and any set X ⊂ R d , we have
[Equation]
Proof Let C ⊂ X denote the cover of minimum cardinality such that
[Equation]
We then have
[Equation]
where step (i) follows by applying Steinwart & Fischer (2021, Lem. 3.11), 5 and step (ii) follows by applying Steinwart & Fischer (2021, Lem. 3.10).
The claim follows by noting that C denotes a cover of minimum cardinality, and hence by definition |C| = N ∥·∥ ( X , 1) .
□
Lemma 4 (Covering number for product kernel)
Given X ⊂ R and a reproducing kernel κ :
X × X → R , consider the product kernel k ≜ κ ⊗ d : X ⊗ 2 d → R defined as [Equation] d
[Equation]
Then the covering numbers of the two kernels are related as follows:
[Equation]
Proof Let H denote the RKHS corresponding to κ .
Then the RKHS corresponding to the kernel k is given by the tensor product H k ≜
H × H × . . .
× H Berlinet & Thomas-Agnan (2011, Sec. 4.6), i.e., for any f ∈ H k , there exists f 1 , f 2 , . . .
, f d ∈ H such that
[Equation]
Let C κ ( X , ε ) ⊂ B κ denote an ε -cover of B κ in L ∞ -norm
(Def. 1).
Then for each f i ∈ H , we have (cid:101)
f i ∈ C κ
( X , ε ) such that
[Equation]
Now, we claim that for every f ∈ B k , there exists g ∈ C k ≜ ( C κ ( X , ε )) ⊗ d such that
[Equation]
which immediately implies the claimed bound (43) on the covering number.
We now prove the claim (46).
For any fixed f ∈ H k , let f i , (cid:101)
f i denote the functions satisfying (44) and (45) respectively.
Then, we prove our claim (46) with g = (cid:81) di =1 (cid:101) f i ∈ C k .
Using the convention (cid:81)
k =1 (cid:101)
f k ( x k ) = 1 , we find that
[Equation]
where in the last step we have used the following argument:
[Equation]
The proof is now complete.
□
Lemma 5 (Relation between Euclidean covering numbers)
We have
[Equation]
for all d ≥ 1 .
Proof We apply Wainwright (2019, Lem. 5.7) with B = B 2 ( r ) and B ′ =
B ∞ (1) to conclude that
[Equation]
where Vol ( X ) denotes the d -dimensional Euclidean volume of X ⊂ R d [Equation] and Γ( a ) denotes the
Gamma function.
Next [Equation] we apply the following bounds on the Gamma function from Batir (2017,
Thm. 2.2):
Γ( b + 1) ≥ ( b/e ) b √ 2 πb for any b ≥ 1 , and Γ( b + 1) ≤ ( b/e ) b √ e
b for any b ≥ 1 . 1 .
Thus, we have
[Equation]
as claimed, and we are done.
□
I. P ROOF OF P ROP . 2: C OVERING NUMBERS FOR ANALYTIC AND DIFFERENTIABLE KERNELS
First we apply Lem.
2 so that it remains to establish the stated bounds simply on log N † k
( X , ε ) .
Proof of bound (31) in part (a) The bound (31) for the real-analytic kernel is a restatement of
Sun & Zhou (2008, Thm. 2) in our notation (in particular, after making the following substitutions in their notation: R ← 1 , C 0 ← C κ , r ← R κ , X ← A , (cid:101) r ← r † , η ← ε, D ← D 2 A , n ← d ).
□
Proof of bound (33) for part (b): Under these assumptions, Steinwart & Christmann (2008,
Thm. 6.26) states that the i -th dyadic entropy number Steinwart & Christmann (2008, Def. 6.20) of the identity inclusion mapping from H| ¯ B 2 ( r ) to L ∞ ¯ B 2 ( r ) is bounded by c ′ s,d, k · r s
i − s/d for some constant c ′ s,d, k independent of ε and r .
Given this bound on the entropy number, and applying Steinwart & Christmann (2008, Lem. 6.21), we conclude that the log-covering number log N † k ( ¯ B 2 ( r ) , ε ) is bounded by ln 4 · ( c ′ s,d, k r s /ε )
d/s =
c s,d, k r d · (1 /ε )
d/s as claimed.
□
I. P ROOF OF P ROP . 3: C OVERING NUMBERS FOR SPECIFIC KERNELS
First we apply Lem.
2 so that it remains to establish the stated bounds in each part on the corresponding log N k .
Proof for G AUSS kernel: Part (a)
The bound (34) for the Gaussian kernel follows directly from
Steinwart & Fischer (2021, Eqn. 11) along with the discussion stated just before it.
Furthermore, the bound (35) for C Gauss ,d are established in Steinwart & Fischer (2021, Eqn. 6), and in the discussion around it.
□
Proof for M AT ´ ERN kernel: Part (b) We claim that M AT ´ ERN ( ν, γ ) is ⌊ ν − d 2 ⌋ -times continuously differentiable which immediately implies the bound (36) using Prop. 2(b).
To prove the differentiability, we use Fourier transform of Mat´ern kernels.
For k = M AT ´ ERN ( ν, γ ) , let κ : R d → R denote the function such that noting
that k ( x, y ) = κ ( x − y ) .
Then using the Fourier transform of κ from Wendland (2004, Thm 8.15), and noting that κ is real-valued, we can write
[Equation]
for some constant c k ,d depending only on the kernel parameter, and d (due to the normalization of the kernel, and the Fourier transform convention).
Next, for any multi-index a ∈ N d 0 , we have
[Equation]
where ∂ a,a denotes the partial derivative of order a .
Moreover, we have
[Equation]
where step (i) holds whenever
[Equation]
Then applying Newey & McFadden (1994, Lemma 3.6), we conclude that for all multi-indices a such that (cid:80) dj
=1 a j ≤ ⌊ ν − d 2 ⌋ , the partial derivative ∂ a,a k exists and is given by
[Equation]
and we are done.
□
Proof for IMQ kernel: Part (c)
The bounds (37) and (38) follow from Sun & Zhou (2008,
Ex. 3), and noting that N 2 ( B 2 ( r ) , (cid:101) r/ 2) is bounded by (1 + 4 r (cid:101) r ) d (Wainwright, 2019, Lem. 5.7).
□
Proof for SINC kernel: Part (d) For k = SINC ( θ ) , we can write k ( x, y )
= (cid:81)
d i =1 κ θ ( x i − y i ) for κ θ : R → R defined as κ θ ( t ) = sin( θt ) θt ( i ) =
sin( | θt | ) | θt | , where step (i) follows from the fact that t (cid:55)→ sin t/t is an even function.
Thus, we can apply Lem. 4.
Given the bound (43), and noting that ∥ κ θ ∥ ∞ = 1 , it suffices to establish the univariate version of the bound (39), namely,
[Equation]
To do so, we claim that univariate SINC kernel is an analytic kernel that satisfies the condition (30) of Prop. 2(a) with κ ( t )
=
SINC ( θ √ t ) , R κ = 12 θ 2 , and C κ = 1 ; and thus applying the bounds (31) and (32) from Prop. 2(a) with A = B d 2 ( r ) yields the claimed bound (39) and (40).
To verify the condition (30) with the stated parameters, we note that
[Equation]
which implies
[Equation]
and we are done.
□
Proof for B- SPLINE kernel: Part (e) For k = B-
SPLINE (2 β + 1 , γ ) , we can write k ( x, y )
= (cid:81) di
=1 κ β,γ (( x i − y i )) for κ β : R → R defined as κ β,γ ( t )
=
B − 1 2 β +2 ⊛ 2 β +2 1 [ − 1 2 , 1 2 ] ( γ · t ) , and thus we can apply Lem. 4.
Given the bound (43), and noting that ∥ κ β,γ ∥ ∞ ≤ 1 (Dwivedi & Mackey (2021, Eqn. 107)), it suffices to establish the univariate version of the bound (41).
Abusing notation and using κ β,γ to denote the univariate B- SPLINE (2 β + 1 , γ ) kernel, we find that
[Equation]
where step (i) follows from Steinwart & Fischer (2021, Thm. 2.4, Sec. 3.3), and for step (ii) we use the fact that the unit-covering number of [0 , γ ] is bounded by max( γ, 1) , and apply the covering number bound for the univariate B- SPLINE kernel from Zhou (2003, Ex. 4) (by substituting m = 2 β + 2 in their notation) along with the fact that log N †
[Equation] [Equation]
N
[Equation] [Equation]
) since κ β is shift-invariant.
□
J P ROOF OF T AB . 3 RESULTS
In Tab. 3, the stated results for all the entries in the TARGET KT column follow directly by substituting the covering number bounds from Prop. 3 in the corresponding entry along with the stated radii growth conditions for the target P .
(We substitute m = 12 log 2 n since we thin to √ n output size.)
For the KT+ column, the stated result follows by either taking the minimum of the first two columns (whenever the ROOT KT guarantee applies) or using the POWER KT guarantee.
First we remark how to always ensure a rate of at least O ( n − 14 ) even when the guarantee from our theorems are larger, using a suitable baseline procedure and then proceed with our proofs.
Remark 2 (Improvement over baseline thinning)
First we note that the KT - SWAP step ensures that, deterministically, MMD k ( S in , S KT )
≤ MMD k
( S in , S base ) and MMD k
( P , S KT ) ≤ 2 MMD k
( P , S in )
+ MMD k
( P , S base ) for S base a baseline thinned coreset of size n 2 m and any target P .
For example if the input and baseline coresets are drawn i.i.d.
and k is bounded, then
MMD k ( S in , S KT ) and MMD k ( P , S KT ) are O
( (cid:112)
2 m /n ) with high probability (Tolstikhin et al., 2017, Thm. A.1), even if the guarantee of Thm.
2 is larger.
As a consequence, in all well-defined KT variants, we can guarantee a rate of n − 14 for MMD k
( S in , S KT ) when the output size is √ n simply by using baseline as i.i.d.
thinning in the KT - SWAP step.
G AUSS kernel The TARGET KT guarantee follows by substituting the covering number bound for the Gaussian kernel from Prop. 3(a) in (5), and the ROOT KT guarantee follows directly from
Dwivedi & Mackey (2021, Tab. 2).
Putting the guarantees for the ROOT KT and TARGET KT together (and taking the minimum of the two) yields the guarantee for KT+.
IMQ kernel The TARGET KT guarantee follows by putting together the covering bound Prop. 3(c) and the MMD bounds (5).
For the ROOT KT guarantee, we use a square-root dominating kernel (cid:101) k rt IMQ ( ν ′ , γ ′ )
Dwivedi &
Mackey (2021, Def.2) as suggested by Dwivedi & Mackey (2021).
Dwivedi & Mackey (2021,
Eqn.(117)) shows that (cid:101) k rt is always defined for appropriate choices of ν ′ , γ ′ .
The best ROOT KT guarantees are obtained by choosing largest possible ν ′
(to allow the most rapid decay of tails), and
Dwivedi & Mackey (2021, Eqn.(117)) implies with ν < d 2 , the best possible parameter satisfies ν ′ ≤
d 4 + ν 2 .
For this parameter, some algebra shows that max( R † (cid:101)
k rt ,n R (cid:101) k rt ,n ) ≾ d,ν,γ n 1 / 2 ν , leading to a guarantee worse than n − 14 , so that the guarantee degenerates to n − 14 using Rem. 2 for ROOT
KT.
When ν ≥ d 2 , we can use a M AT ´ ERN kernel as a square-root dominating kernel from Dwivedi & Mackey (2021, Prop. 3), and then applying the bounds for the kernel radii (12), and the inflation factor (14) for a generic Mat´ern kernel from Dwivedi & Mackey (2021, Tab. 3) leads to the entry for the ROOT KT stated in Tab. 2.
The guarantee for KT+ follows by taking the minimum of the two.
M AT ´ ERN kernel
For TARGET KT, substituting the covering number bound from Prop. 3(b) in (5) with R = log n yields the MMD bound of order
[Equation]
[Equation]
which is better than n − 14 only when ν > 3 d/ 2 , and simplified to the entry in the Tab. 3 when we assume ν − d 2 is an integer.
When ν ≤ 3 d/ 2 , we can simply use baseline as i.i.d.
thinning to obtain an order n − 14 MMD error as in Rem. 2.
The ROOT KT (and thereby KT+) guarantees for ν > d follow from Dwivedi & Mackey (2021,
Tab. 2).
When ν ∈ ( d 2 , d ] , we use POWER KT with a suitable α to establish the KT+ guarantee.
For
M AT ´ ERN ( ν, γ ) kernel, the α -power kernel is given by M AT
´ ERN ( αν, γ ) if αν > d 2 (a proof of this follows from Def. 2 and Dwivedi & Mackey (2021, Eqns (71-72))).
Since L APLACE ( σ )
=
M AT ´ ERN ( d +12 , σ − 1 ) , we conclude that its α -power kernel is defined for α > dd +1 .
And using the various tail radii (12), and the inflation factor (14) for a generic Mat´ern kernel from
Dwivedi & Mackey (2021, Tab. 3), we conclude that (cid:102)
M α ≾ d, k α ,δ √ log n log log n , and max( R † k α ,n R k α ,n ) ≾ d, k α log n , so that R max =
O d, k α (log n ) (13) for S UB E XP P setting.
Thus for this case, the MMD guarantee for √ n thinning with POWER KT (tracking only scaling with n ) is
[Equation]
where c n = log log n ; and we thus obtain the corresponding entry (for KT+) stated in Tab. 3.
SINC kernel
The guarantee for TARGET KT follows directly from substituting the covering number bounds from Prop. 3(d) in (5).
For the ROOT KT guarantee, we note that the square-root kernel construction of Dwivedi & Mackey (2021, Prop.2) implies that SINC ( θ ) itself is a square-root of SINC ( θ ) since the Fourier transform of SINC is a rectangle function on a bounded domain.
However, the tail of the SINC kernel does not decay fast enough for the guarantee of Dwivedi & Mackey (2021, Thm. 1) to improve beyond the n − 14 bound of Dwivedi & Mackey (2021, Rem. 2) obtained when running ROOT KT with i.i.d.
baseline thinning.
In this case, TARGET KT and KT+ are identical since k rt = k .
B- SPLINE kernel
The guarantee for TARGET KT follows directly from substituting the covering number bounds from Prop. 3(d) in (5).
For B- SPLINE (2 β + 1 , γ ) kernel, using arguments similar to that in Dwivedi & Mackey (2021,
Tab.4), we conclude that (up to a constant scaling) the α -power kernel is defined to be B- SPLINE ( A + 1 , γ ) whenever A [-(cid:44) 
-]≜ 2 αβ +2 α − 2 ∈
[-+
 α − 2 ∈ 2-] N 0 .
For odd β we can always take α = 12 and B- SPLINE ( β + 1 , γ ) is a valid (up to a constant scaling) square-root kernel (Dwivedi & Mackey, 2021).
For even β , we have to choose α ≜ p +1 β +1 ∈ ( 12 , 1) by taking p ∈ N suitably, and the smallest suitable choice is p =
⌈ β − 1 2 ⌉ = β 2 ∈ N , which is feasible as long as β ≥ 2 .
And, thus B- SPLINE ( β + 1 , γ ) is a suitable k α for B- SPLINE (2 β + 1) for even β ≥ 2 with α = β +2 2 β +2 ∈ ( 12 , 1) .
Whenever the α -power kernel is defined, we can then apply the various tail radii (12), and the inflation factor (14) for the power B- SPLINE kernel from Dwivedi & Mackey (2021, Tab. 3) to obtain the MMD rates for POWER KT from Dwivedi & Mackey (2021, Tab. 2) (which remains the same as ROOT KT upto factors depending on α and β ).
The guarantee for KT+ follows by taking the minimum MMD error for TARGET KT and ROOT KT for even β , and α - POWER KT for odd β .