G ENERALIZED K ERNEL T HINNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The kernel thinning (KT) algorithm of Dwivedi and Mackey (2021) compresses an n point distributional summary into a √ n point summary with better-thanMonte-Carlo maximum mean discrepancy for a target kernel k by leveraging a less smooth square-root kernel.
Here we provide four improvements.
First, we show that KT applied directly to the target kernel yields a tighter O ( (cid:112) log n/n ) integration error bound for each function f in the reproducing kernel Hilbert space. This modiﬁcation extends the reach of KT to any kernel—even non-smooth kernels that do not admit a square-root, demonstrates that KT is suitable even for heavytailed target distributions, and eliminates the exponential dimension-dependence and (log n ) d/ 2 factors of standard square-root KT.
Second, we show that, for analytic kernels, like Gaussian and inverse multiquadric, target kernel KT admits maximum mean discrepancy (MMD) guarantees comparable to square-root KT without the need for an explicit square-root kernel.
Third, we prove KT with a fractional α -power kernel k α for α > 1 / 2 yields better-than-Monte-Carlo MMD guarantees for non-smooth kernels, like Laplace and Mat ´ ern, that do not have square-roots.
Fourth, we establish that KT applied to a sum of k and k α (a procedure we call KT+) simultaneously inherits the improved MMD guarantees of power KT and the tighter individual function guarantees of KT on the target kernel.
Finally, we illustrate the practical beneﬁts of target KT and KT+ for compression after high-dimensional independent sampling and challenging Markov chain Monte Carlo posterior inference
I NTRODUCTION
A core task in probabilistic inference is learning a compact representation of a probability distribution P .
This problem is usually solved by sampling n points independently from P or, if direct sampling is intractable, generating n points from a Markov chain converging to P .
The beneﬁt of these approaches is that they provide asymptotically exact sample estimates P n f (cid:44) 1 n (cid:80) ni
=1 f ( x i ) for intractable expectations P f (cid:44) E X ∼ P [ f ( X )] .
However, they also suffer from a serious drawback: the learned representations are unnecessarily large, requiring n points to achieve | P f − P n f | =
Θ( n − 12 ) integration error.
These inefﬁcient representations quickly become prohibitive for expensive downstream tasks and function evaluations: for example, in computational cardiology, each function evaluation f ( x i ) initiates a heart or tissue simulation that consumes 1000s of CPU hours (Niederer et al., 2011; Augustin et al., 2016; Strocchi et al., 2020).
To reduce the downstream computational burden, a standard practice is to thin the initial sample by discarding every t -th sample point (Owen, 2017).
Unfortunately, standard thinning often results in a substantial loss of accuracy: for example, thinning an i.i.d.
or fast-mixing Markov chain sample from n points to n 12 points increases integration error from Θ( n − 12 ) to Θ( n − 14 ) .
Our goal is to produce more accurate compressed representations with provably better-than-i.i.d. integration error. Problem setup
Given a target distribution P on a domain X , a sequence of input points S in = ( x i ) ni
=1 in X , and a function space H , our ﬁrst goal is to identify a thinned H coreset, a point subsequence S out of length √ n with better-than-i.i.d. integration error, | P f − P out f | = o ( n − 14 ) , for each function f ∈ H and P out (cid:44) 1 √ n (cid:80) x ∈S out δ x .
We will focus our attention on the reproducing kernel Hilbert space (RKHS) H associated with a given kernel k (Berlinet & Thomas-Agnan,
[Table] [Table] [Table]
1 We additionally measure the quality of our thinned representations in terms of the kernel maximum mean discrepancy (MMD, Gretton et al., 2012), deﬁned as the worst-case error between sample and target expectations over the unit ball of H equipped with RKHS norm (cid:107)·(cid:107) k :
[Equation]
Our second goal is to produce thinned MMD coresets S out with MMD k ( P , P out ) = o ( n − 14 ) . For reference, we collect expressions for several commonly used kernels in Tab. 1.
The starting point for our work is the recent kernel thinning (KT) algorithm of Dwivedi & Mackey (2021) which takes as input n points and returns n 12 points with o ( n − 14 ) -MMD error thereby beating the Monte Carlo rate.
To effectively control MMD k , KT uses non-uniform randomness and a less smooth square-root kernel k rt satisfying k
( x, y )
= (2 π ) d · (cid:82) R d k rt ( x, z ) k rt (
z, y ) dz , up to an immaterial constant rescaling.
When the input points are sampled i.i.d.
or from a fast-mixing Markov chain on R d , Dwivedi & Mackey prove that the KT output has, with high probability, O d ( n − 12 √ log n ) - MMD k error for P and k rt with bounded support, O d ( n − 12 (log d +1 n log log n ) 12 ) - MMD k error for P and k rt with light tails, and ( n 12 , O d ( n − 12 + d 2 ρ √ log n log log n )) - MMD k error for P and k 2rt with ρ > 2 d moments.
Our contributions While KT provides a superior to Monte Carlo error rate for a range of settings, the algorithm in its original form, required the knowledge of a square-root kernel k rt to run the algorithm, and establish theoretical guarantees.
Here we provide several generalizations.
First, we directly analyze a variant of kernel thinning which uses only the target kernel k .
To distinguish the two cases, going forward, we use ROOT KT to denote the original kernel thinning algorithm using k rt and TARGET KT to denote kernel thinning just with k .
Furthermore, we introduce and analyze two other KT variants: (i) α - POWER KT, which uses a generalized power kernel k α of kernel k (see Def. 2), and (ii) KT+ , which uses a sum of k α and k . We note that the existence of generalized power k α is a weaker condition than the existence of k rt .
We provide several theoretical and practical improvements with kernel thinning.
First, we prove that the output of target KT provides O ( n − 12 √ log n ) integration error for any ﬁxed function in the RKHS of any kernel k on any domain .
In particular, when the domain is R d , our single function guarantee for target KT has explicit constants and is dimension independent. Second, we show that like root KT, even target KT provides o ( n − 14 ) - MMD k error with √ n points for a wide range of settings.
Third, we show that α -power KT provides o ( n − 1 / 4 ) -MMD errors for non-smooth kernels which do not admit a square-root kernel. Finally, when k rt exists, we show that the KT+ inherits the uniform error guarantees of root KT and the tighter individual function guarantees of target KT.
Related work For bounded k , both i.i.d.
samples (Tolstikhin et al., 2017, Prop. A.1) and thinned geometrically ergodic Markov chains (Dwivedi & Mackey, 2021, Prop. 1) deliver n 12 points with
O ( n − 14 ) MMD with high probability.
The online Haar strategy of Dwivedi et al.
and low discrepancy quasi-Monte Carlo methods (see, e.g., Hickernell, 1998; Novak & Wozniakowski, 2010;
Dick et al., 2013) provide improved O d ( n − 12 log d n ) MMD guarantees but are tailored speciﬁcally to the uniform distribution on [0 , 1] d .
Alternative coreset constructions for more general P include kernel herding (Chen et al., 2010), discrepancy herding (Harvey & Samadi, 2014), super-sampling with a reservoir Paige et al. (2016), support points convex-concave procedures (Mak & Joseph, 2018), greedy sign selection (Karnin & Liberty, 2019, Sec. 3.1), Stein points MCMC (Chen et al., 2019), and Stein thinning ( Riabiz et al., 2020a).
While some admit better-than-i.i.d.
MMD guarantees for ﬁnite-dimensional kernels on R d (Chen et al., 2010; Harvey & Samadi, 2014), apart from KT, none are known to provide better-than-i.i.d.
MMD or integration error for inﬁnite-dimensional kernels, like the convergence-determining Gaussian, Mat´ern, inverse multiquadric, B-spline, and sinc kernels studied in this work.
The minimax lower bounds of Phillips & Tai (2020, Thm. 3.1) and
Tolstikhin et al.
(2017, Thm. 1) respectively establish that any procedure outputting n 12 -sized coresets and any procedure estimating P based only on n i.i.d.
sample points must incur Ω( n − 12 ) MMD in the worst case.
Our guarantees in Sec.
2 match these lower bounds up to logarithmic factors.
Notation We deﬁne the norm (cid:107) k (cid:107) ∞ =
sup x,y | k ( x, y ) | and the shorthand [ n ] (cid:44) { 1 , . . . , n } , P in (cid:44) 1 n (cid:80) ni
{+=
 δ x i+} , R + (cid:44) { x ∈ R : x ≥ 0 } , N 0 = N ∪ { 0 } , B k (cid:44) { f ∈ H : (cid:107) f (cid:107) k ≤ 1 } , and B 2 ( r ) = (cid:8)
y ∈ R
d : (cid:107) y (cid:107) 2 ≤ r (cid:9) .
We write a (cid:45) b and a (cid:37) b to mean a = O ( b ) and a = Ω( b ) and use (cid:45) d when masking constants dependent on d .
For point sequences S , S (cid:48) with empirical distributions Q n , Q (cid:48) n , we deﬁne MMD k
( P , S ) (cid:44) MMD k ( P , Q n ) and MMD k ( S , S (cid:48) ) (cid:44) MMD k
( Q n , Q (cid:48) n ) .
G ENERALIZED K ERNEL T HINNING
Our generalization of kernel thinning in Alg. 1 proceeds in two steps: KT - SPLIT and KT - SWAP .
Given a thinning parameter m and a target kernel k , KT - SPLIT divides the input sequence S in = ( x i ) ni =1 into 2 m candidate coresets of size n/ 2 m in a non-uniform manner using an auxiliary kernel k split .
Next, the KT - SWAP step adds a baseline coreset to the candidate list (for example, one produced by standard thinning or uniform subsampling), selects the candidate closest to S in in terms of MMD k , and reﬁnes the selected coreset by swapping each input point into the coreset if it offers an improvement in MMD k .
The original variant of kernel thinning uses k split = k rt
(and we thus refer to it as root KT), and here we establish theoretical guarantees for more general k split with a notable example being k split = k .
Other than the choice of k split being more general, the KT - SPLIT and
KT - SWAP considered here are identical to Dwivedi & Mackey (2021, Alg 1a & 1b) and reproduced for reference in App. A. Overall, our generalization does not affect the runtime or storage of kernel thinning, which for any m has the time complexity dominated by O ( n 2 ) evaluations of k split and k , while the space complexity is O ( n min( d, n )) , achieved by storing the smaller of the input sequence ( x i ) ni
=1 and the kernel matrices ( k split ( x i , x j ))
ni,j =1 and ( k ( x i , x j ))
ni,j =1 .
[Table]
2. S INGLE FUNCTION GUARANTEES FOR KT - SPLIT
Our ﬁrst main result, proved in App.
B, provides a high probability bound for the integration error of any ﬁxed function in the RKHS H split of kernel k split for the output of KT - SPLIT .
Theorem 1 (Single function guarantees for KT - SPLIT ) Consider KT - SPLIT (Alg.
1a) with oblivious 2 S in and ( δ i )
n/
i
[-=
-]=1 and δ (cid:63) (cid:44) min
i δ i .
If n 2 m ∈ N , then, for any ﬁxed f ∈ H split , index (cid:96) ∈ [2 m ] , and scalar δ (cid:48) ∈ (0 , 1) , the output coreset S ( m,(cid:96) ) with P ( (cid:96) ) split (cid:44) [Table] n
[Table]
m (cid:80) x ∈S [Table] m, [Table]
δ x satisﬁes
[Equation]
P
[Equation]
[Table] [Equation] (cid:107) [Equation]
[Table] σ [Table] (cid:113) 2 log( 2 [-j i =
-]δ (cid:48) )
{+for σ m (cid:44) 2 √ 3 2 m n (cid:113) (cid:107) k split (cid:107) ∞ , in · log( 4 δ (cid:63) ) (1) with probability at least p sg (cid:44) 1 − (cid:80) mj =1 (cid:80) n/ 2 j i =
+} δ i − δ (cid:48) .
Here, (cid:107) k split (cid:107) ∞ , in (cid:44) max x ∈S in k split ( x,
Throughout Remark 1 (Guarantees for known and oblivious stopping times)
To ensure that the bound (1) holds with probability at least 1 − δ , we can set δ (cid:48) =
δ 2 and δ i = δ 2 n for a stopping time n known a priori or δ i
= δ 4( i +1)log
( i +1) for an arbitrary oblivious stopping time n .
Remark 2 (Target KT - SPLIT yields better-than-i.i.d. thinning) For each f ∈ H , KT - SPLIT with k split = k , m = 12 log 2 n , δ i = δ 2 n , and δ (cid:48) =
δ 2 provides O ( n − 12 √ log n ) integration error, [Equation] , with probability at least 1 − δ using only √ n points. When | P f − P in f | =
O ( n − 12 ) , the triangle inequality implies that | P f − P ( (cid:96) ) split f | =
O ( n − 12 √ log n ) , a near-quadratic improvement over the Ω( n − 14 ) integration error of √ n i.i.d.
points.
Remark 3 (Guarantees for any target kernel k ) Thm. 1 applies to any kernel deﬁned on any space including unbounded kernels on unbounded domains (e.g., energy distance and Stein kernels); kernels with slowly decaying square roots (e.g., sinc kernels); and non-smooth kernels without square roots (e.g., Laplace, Mat´ern with γ ∈ ( d 2 , d ] ), and the compactly supported kernels of Wendland (2004) with s < 1 2 ( d + 1) ).
In contrast, the MMD guarantees of Dwivedi & Mackey covered only bounded, smooth k on R d with bounded, Lipschitz, and rapidly-decaying square-roots.
Remark 4 (Guarantees for any target distribution P ) Thm.
1 guarantees better-than-i.i.d.
integration error for any target distribution with | P f − P in f |
= o ( n − 14 ) .
In contrast, the MMD improvements of Dwivedi & Mackey (2021, cf.
Tab. 2) applied only to P with at least d moments.
Remark 5 (Dimension-independent bounds) When (cid:107) k (cid:107) ∞ = 1 , the MMD bounds of Dwivedi &
Mackey feature exponential dimension dependence of the form c d or (log n ) d/ 2 while the Thm.guarantee is dimension-free and hence practically relevant even when d is large relative to n .
Remark 6 (Consequences for KT - SPLIT with square-root kernel k rt )
When a bounded squareroot kernel k rt is available, we can apply Thm.
1 with k split = k rt since any function f in the RKHS of k also lies in the RKHS of k rt : This bound with root KT - SPLIT , when compared to the bound (1) admits a multiplicative factor of (cid:113) (cid:107) k rt (cid:107) ∞ , in (cid:107) k (cid:107) ∞ , in (cid:107) f (cid:107) k rt (cid:107)
f
(cid:107) k . For the kernels explicitly analyzed in Dwivedi &
Mackey (2021), we show in App.
G that this factor grows exponentially in dimension for Gaussian and Mat ´ ern kernels, and is 1 for B-spline kernels so that the single function guarantee from target
KT is better than root KT for all the kernels explicitly considered in the prior work.
2. MMD GUARANTEES FOR TARGET KT
We now establish an MMD guarantee for target KT , our name for generalized KT with k split = k . Deﬁnition 1 (RKHS covering number) Consider a kernel k deﬁned on R d × R d with the corresponding RKHS H , any set X ⊂ R d , and a scalar ε > 0 .
We deﬁne N k
( X , ε ) as the minimum cardinality over all possible covers C ⊂ B k (the unit ball of RKHS) that satisfy
B k ⊆ (cid:83) h ∈C { g ∈ B k : sup x ∈X | h ( x ) − g ( x ) | ≤ ε } , and M k ( X , ε ) (cid:44) log N k
( X , ε ) .
With the deﬁnitions in place, our next result provides an MMD guarantee for target KT in terms of the log-covering number M k .
Its proof based on Thm. 1 is provided in App. C.
Theorem 2 (MMD guarantee for target KT) Consider generalized KT (Alg. 1) with k split =
k , oblivious S in and ( δ i ) (cid:98) n/ 2 (cid:99) i =1 , and δ (cid:63) (cid:44) min i δ i .
If n 2 m ∈ N , then for any δ (cid:48) ∈ (0 , 1) , the output coreset
S tKT is of size n 2 m and satisﬁes MMD k
[Equation]
S in , S tKT
[Equation] ≤ [Equation] ) with probability at least p sg , where (cid:107) k
(cid:107) ∞ , in and p sg were deﬁned in Thm. 1.
Remark 7 (Guarantee for arbitrary target P )
By the triangle inequality, MMD k ( P , S KT ) ≤ MMD k ( P , S in )+MMD k ( S in , S KT ) , yielding a guarantee for any target distribution P . Remark 8 (Improvement over baseline thinning) As in root KT, the target KT - SWAP step ensures that, deterministically, MMD k ( S in , S KT ) ≤ MMD k
( S in , S base ) for S base a baseline thinned coreset of size n 2 m . Therefore, MMD k ( P , S KT ) ≤ 2 MMD k ( P , S in ) + MMD k ( P , S base ) . Remark 9 (MMD guarantee for √ n -thinning) Substituting m = 12 log 2 n , δ i = δ 2 n , δ (cid:48) =
[Equation]
2 , ε = n − 12 , A = B 2 ( R in ) for R in (cid:44) max x ∈S in (cid:107) x (cid:107) 2 , we ﬁnd that with probability at least 1 − δ ,
[Equation]
This bound yields an MMD guarantee for any kernel k with a covering number bound.
In Tab. 2, we summarize the MMD rates for the target KT output for a wide range of settings under different forms of growth conditions on M k and the input point radius R S in (cid:44) max x ∈S in (cid:107) x (cid:107) 2 .
We consider two different growth conditions on M k (i) L OG G ROWTH ( M k ( B 2 ( r ) , ε ) (cid:45) d r d log ω ( 1 ε ) ), and (ii) P OLY G ROWTH ( M k ( B 2 ( r ) , ε ) (cid:45) d r d ε − ω ).
We show in Props.
2 and 3 that the L OG G ROWTH condition applies to inﬁnitely differentiable analytic kernels, including Gaussian, inverse multiquadric, and sinc kernels, and the P OLY G ROWTH condition applies to kernels differentiable ﬁnitely many times, including Mat ´ ern and B-spline.
Our conditions on R S in , arise from four forms of target distribution tail decay: (1) C OMPACT ( R S in (cid:45) d 1 ), (2) S UB G AUSS ( R S in (cid:45) d √ log n ), (3) S UB E XP ( R S in (cid:45) d log n ), and (4)
H EAVYT AIL ( R S in (cid:45) d n 1 /ρ ) .
The ﬁrst setting arises with a compactly supported P (e.g., on the unit cube [0 , 1] d ), and the other three settings arise in expectation and with high probability when P has, respectively, sub-Gaussian tails, sub-exponential tails, or ρ moments and S in is generated i.i.d. from P .
Substituting these conditions in (4) yields the eight entries of Tab.
We, thus, conclude that for a L OG G ROWTH kernel, target KT provides an o ( n − 1 / 4 ) MMD error for all four settings of P ; and for a P OLY G ROWTH kernel, it provides an o ( n − 1 / 4 ) MMD error if ω < 12 for C OMPACT , S UB G AUSS , and S UB E XP P , and if ω + 2 d/ρ < 1 / 2 for H EAVY T AIL P with tail exponent ρ .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
Next in Tab. 3, we summarize the MMD rates for target KT and root KT for all kernels displayed in Tab. 1.
The results for root KT follow from Dwivedi & Mackey (2021), while the results for target KT are obtained by combining Thm. 2 with kernel-speciﬁc covering number bounds derived in Apps.
I and J. Notably, Thm. 2 provides better-than-i.i.d.
and better-than-root KT guarantees for a variety of kernels with slowly decaying or non-existent square-roots (e.g., IMQ with ν < d 2 , sinc, sinc-rad, and B-spline) and nearly matches known root KT guarantees for analytic kernels like Gauss and IMQ with ν ≥ d 2 , even though target KT makes no explicit use of a square-root kernel.
K ERNEL T HINNING +
We next introduce two new instantiations of generalized KT: (i) power KT which leverages a power kernel k α that interpolates between k and k rt to improve upon the MMD guarantees of target KT even when k rt is not available and (ii) KT+ which uses a sum of k and k α to retain both the improved MMD guarantee of k α and the superior single function guarantees of k . [Table]
First, we deﬁne a generalization of the square-root kernel for a shift-invariant [-k using the order 
-]kernel.
Let (cid:98) f denote the generalized Fourier transform (GFT) (Wendland, 2004, Def. 8.9) of f : R d → R d .
Deﬁnition 2 ( α -power kernel)
For α > 0 , a kernel k α ( x, y )
=
κ α ( x − y ) is an α -power kernel for k ( x, y )
=
κ ( x − y ) if (cid:99) κ α = (cid:98) κ α .
By Dwivedi & Mackey (2021, Prop. 2) k 12 = k rt for any shift-invariant k .
Given a power kernel k α , we instantiate generalized KT with k split = k α and call it power KT .
To state an MMD guarantee for power KT, we need additional notation.
Deﬁne the k α and S in tail radii:
[Equation] r ) ≤ (cid:107) k α (cid:107) ∞ n } , where τ k α ( R ) (cid:44) (sup x (cid:82) (cid:107) y (cid:107) 2 ≥ R k 2 α ( x, x − y ) dy ) 12 ,
R k α ,n (cid:44) min { r :sup (cid:107) x − y (cid:107) 2 ≥ r | k α ( x, y ) |≤ (cid:107) k
α (cid:107) ∞ n } , (5) R [Equation] (cid:44) max
[Equation]
in (cid:107) [Equation] , and R S in , k α ,n (cid:44) min (cid:0) R S in , n 1+ 1 d R k α ,n + n 1 d (cid:107) k
α (cid:107) ∞ L k α (cid:1) and the inﬂation factor
[Equation] where L k α denotes the Lipschitz-constant of k α , i.e., | k α ( x, y ) − k α ( x, z ) | ≤ L k α (cid:107) y − z (cid:107) 2 for all x, y, z ∈ R d . Our next result (with proof in App. D) provides an MMD guarantee for power KT. Theorem 3 (MMD guarantee for power KT) Consider generalizd KT (Alg. 1) with k split =
k α for some α ∈ [ 1 2 , 1] , oblivious sequences S in and ( δ i ) (cid:98) n/ 2 (cid:99) i =1 , and δ (cid:63) (cid:44) min i δ i . If n 2 m ∈ N , then for
[Figure]
any δ (cid:48) ∈ (0 , 1) , the output coreset S α KT is of size n 2 m and satisﬁes
[Equation]
with probability at least p sg (deﬁned in Thm. 1) where
[Equation]
This result generalizes the root KT guarantees of Dwivedi & Mackey (2021) to power KT, and is proven via a novel interpolation result (of independent interest) for MMDs, that relates MMD k in terms of MMD k α and MMD k 2 α (Prop. 1).
In fact, we can recover Dwivedi & Mackey (2021, Thm. 1) from Thm. 3 by simply substituting α = 12 .
For less-smooth kernels like L APLACE and
M AT ´ ERN ( ν, γ ) with ν ∈ ( d 2 , d ] , k rt does not exist and our target KT guarantees are no better than the Monte Carlo rate.
However, as shown in App.
J, these kernels have M AT ´ ERN kernels as their α -power kernels, which when substituted appropriately in Thm. 3 yields o ( n − 14 ) MMD error.
3. K ERNEL T HINNING +
Our next result shows that choosing the split kernel as the scaled sum of target and power kernels, k † (cid:44) k / (cid:107)
k (cid:107) ∞
+ k α / (cid:107) k α (cid:107) ∞ . 3 , simultaneously provides improved MMD error guarantees and dimension-free single function guarantees.
[-i.i.d. Root KT (Gauss) 17 .17 .
 Target KT (Gauss) 17 .17 .
 KT+ (IMQ) 17 .17 .
-]See App.
[-KT+ (Laplace) 
 KT+ (Bspline) 
 [Figure] [Figure]-]E for the proof.
Theorem 4 (Single function & MMD guarantees for KT+) Consider kernel thinning (Alg. 1) with k split = k † , oblivious S in and ( δ i ) (cid:98) n/ 2 (cid:99) i
=1 , δ (cid:63) (cid:44) min i δ i , and n 2 m ∈ N .
For any ﬁxed function f ∈ H , index (cid:96) ∈ [2 m ] , and scalar δ (cid:48) ∈ (0 , 1) , the KT - SPLIT coreset S ( m,(cid:96) ) satisﬁes [Equation]
[Equation]
with probability at least p
sg
(for p sg and P ( (cid:96) ) split deﬁned in Thm. 1).
Moreover,
[Equation]
[Equation] with probability at least p sg where M targetKT ( k ) denotes the right hand side of (3) with (cid:107) k (cid:107) ∞ , in replaced by (cid:107) k (cid:107) ∞ , and M powerKT ( k α ) denotes the right hand side of (6) .
/ (cid:107) k α (cid:107) ∞ , in .
E XPERIMENTS
We now evaluate the empirical performance of generalized KT when producing a coreset of size √ n from n input points.
We consider three classes of target distributions: (i) mixture of Gaussians (MoG) P = 1 M (cid:80) Mj
=1 N ( µ j , I 2 ) with M component means µ j ∈ R 2 deﬁned in App.
H, (ii) Gaussian P = N (0 , I d ) for d ∈ { 2 , 10 , 20 , 50 , 100 } , and (iii) MCMC targets based on 12 posterior inference experiments from Riabiz et al. (2020a) targeting the posteriors of four distinct coupled ordinary differential equation models: the Goodwin (1965) model of oscillatory enzymatic control ( d = 4 ), the Lotka (1925) model of oscillatory predator-prey evolution ( d = 4 ), the Hinch et al.
(2004) model of calcium signalling in cardiac cells ( d = 38 ), and a tempered Hinch posterior.
[Figure]
[Table]
[Figure]
To test the ability of generalized KT to improve integration both inside and outside of H , we evaluate integration error for (a) a random element of the target kernel RKHS ( f ( x )
= k ( X (cid:48) , x ) described in App. H), (b) moments ( f ( x ) = x 1 and f ( x ) = x 21 ), and (c) a standard numerical integration benchmark test function from the continuous integrand family (CIF, Genz, 1984), f CIF ( x ) = exp( − (cid:80) di
=1 a i | x i − u i | ) for u i drawn i.i.d.
from uniform distribution on [0 , 1] , and a i = 1 /d .
We let P out denote the empirical distribution of the output points. For the MoG P we visualize in Fig.
1 the coresets from root KT with G AUSS (4), target KT with
G AUSS (4) and B- SPLINE (1), KT+ for L APLACE (2) k and k 0 . 7 , and KT+ with IMQ(0.5, 2) k and k 0 . 5 . Notably, the B- SPLINE (1) and L APLACE kernels do not admit a square-root.
In all cases, the spatial distribution of the KT output is superior to i.i.d.
points and provides a better approximation of the underlying target (also see Fig. 4).
This observation can be quantiﬁed via the MMD error plots in Fig.
5 in App.
H, where we notice that the MMD error for all variants of KT decays more rapidly than i.i.d. with a rate close to n − 12 across all settings.
For the Gaussian targets, we use a Gaussian kernel with squared bandwidth σ 2 = 2 d and an i.i.d. input sequence S in from P .
In line with our theory, we observe from Fig.
2 that the target and root KT MMD errors are nearly identical and that both variants are signiﬁcantly superior to i.i.d.
points even in 50 and 100 dimensions with as few as 8 points.
Moreover, for single function integration error, target KT provides a signiﬁcant improvement over i.i.d.
points in almost all settings.
Since the exact posterior P is unknown for the MCMC targets, we measure error with respect to the input distribution P in (consistent with our guarantees).
We employ KT+ with L APLACE ( σ ) k and k 0 . 81 for Goodwin and Lotka-Volterra and IMQ( 0 .
5 , 1 σ ) k and k 0 .
5 for Hinch.
Here the kernel bandwidth σ is chosen via the popular median heuristic (see, e.g., Garreau et al., 2017): for all n , σ is set to the median distance between all pairs of points in the largest S in tested.
In Fig.
3, we plot one posterior setting from each of the Goodwin, Lotka-Volterra, and Hinch models and observe that that KT+ uniformly improves the MMD error over standard thinning in all settings.
Furthermore, it provides a signiﬁcantly smaller integration error for all four test functions in almost all settings. For the other 9 MCMC settings, we provide the results in Fig.
6 in App. H.‘
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
a
A D A - R
W
L a p l a c
[Table]
k
[Table]
2|( in out ) f
CIF |
ST: n 0.46 KT+: n 0.
C ONCLUSION
We introduced and analyzed generalized kernel thinning, a generalization of the kernel thinning algorithm of Dwivedi & Mackey (2021) for generating better-than-i.i.d.
distributional summaries. We provided the ﬁrst analysis of KT applied directly on the target kernel and thereby extended the reach of KT to kernels without fast-decaying square-roots.
Moreover, we established dimensionindependent single function error guarantees for any kernels.
We also introduced two new variants of KT, namely power KT and KT+ that provide improved MMD guarantees whenever a less-smooth power kernel if available, while simultaneously providing single function guarantees like target KT.
We highlight that while generalized KT provides unweighted coresets with better than Monte Carlo
MMD, its guarantees also extend to any improved reweighting of the output points.
In particular, if the downstream inferential task supports weighted coresets, one can optimally reweight P out to approximate P in in O ( n 3 2 ) time by directly minimizing the MMD.
Finally, we note that like the original root KT algorithm, target KT and KT+ have a runtime of n 2 that might be prohibitive for large values of n . An important open question is whether one can substantially improve the runtime of KT without signiﬁcant degradation of the MMD or single function guarantees.
R EPRODUCIBILITY S TATEMENT
Python code reproducing all experiments is available at redacted , and supplementary experimental details can be found in App. H.
A PPENDIX
A Details of KT - SPLIT and KT - SWAP
B Proof of Thm.
1: Single function guarantees for KT - SPLIT
C Proof of Thm. 2: MMD guarantee for target KT
D Proof of Thm. 3: MMD guarantee for power KT
E Proof of Thm. 4: Single function & MMD guarantees for KT+
F Proof of Prop. 1: An interpolation result for MMD
G Sub-optimality of single function guarantees with root KT
H Additional experimental results
I Upper bounds on RKHS covering numbers
J Proof of Tab.
3 results
A D ETAILS OF KT - SPLIT AND KT - SWAP
Algorithm 1a: size (cid:98) n/ 2 m (cid:99)
Input: kernel k split , point sequence S in = ( x i ) ni
=1 , thinning parameter m ∈ N , probabilities ( δ i ) (cid:98) n/ 2 (cid:99) i =
S ( j,(cid:96) ) ← {} for j = 0 , . . . , m and (cid:96) = 1 , . .
, 2 j // Empty coresets: S ( j,(cid:96) ) has size (cid:98) i/ 2 j (cid:99) after round i σ j,(cid:96) ← 0 for j = 1 , . . . , m and (cid:96) = 1 , . . , 2 j −// Swapping parameters for i = 1 , . . .
, n do
S (0 , 1) .append
( x i ) //
Every 2 j iterations, add one point from each parent coreset S ( j − 1 ,(cid:96) ) to each child S ( j, 2 (cid:96) − 1) , S ( j, 2 (cid:96) ) for ( j = 1; j ≤ m and i/ 2 j ∈ N ; j = j + 1 ) do for (cid:96) = 1 , . .
, 2 j − 1 do ( S , S (cid:48) )
← ( S ( j − 1 ,(cid:96) ) , S ( j, 2 (cid:96) − 1) ) ; ( x, x (cid:48) ) ← get last two points ( S ) ( p swap , σ j,(cid:96) ) ← Update params ( k split , S , S (cid:48) , x, x (cid:48) , σ j,(cid:96) , δ i/ 2 j ) ( x, x (cid:48) )
← ( x (cid:48) , x ) with probability p swap S ( j, 2 (cid:96) − 1) .append ( x ); S ( j, 2 (cid:96) ) .append
( x (cid:48) ) end end end return ( S ( m,(cid:96) ) ) 2 m (cid:96) =1 , candidate coresets of size (cid:98) n/ 2 m (cid:99)
Update params ( k split , S , S (cid:48) , x, x (cid:48) , σ, δ ) :
b 2 ← k split ( x, x )+ k split ( x (cid:48) , x (cid:48) ) − 2 k split ( x, x (cid:48) ) ; a ← max( b σ √ 2 log(4 /δ ) , b 2 ) σ ← σ + b 2 (1+( b 2 − 2 a ) σ 2 / a 2 ) + α ← k ( x (cid:48) ,x (cid:48) ) − k ( x,x )+Σ y ∈S ( k split ( y,x ) − k ( y,x (cid:48) )) − 2Σ z ∈S (cid:48) ( k ( z,x ) − k
( z,x (cid:48) )) p ← min(1 , 12 (1 − α/ a ) + ) return ( p, σ )
Input: kernel k , point sequence S in = ( x i ) ni
=1 , candidate coresets ( S ( m,(cid:96) ) )
2 m (cid:96) =
S ( m, 0) ← baseline thinning ( S in , size = (cid:98) n/ 2 m (cid:99) ) //
Compare to baseline
S KT ← S ( m,(cid:96) (cid:63) ) for (cid:96) (cid:63) ← argmin (cid:96) ∈{ 0 , 1 ,..., 2 m } MMD k
( S in , S ( m,(cid:96) ) ) //
Select best candidate coreset for i = 1 , . . .
, (cid:98) n/ 2 m (cid:99) do
S KT [ i ] ← argmin z ∈S in MMD k
( S in , S KT with S KT [ i ] =
z ) // Swap out each point in S KT for best alternative in S in endreturn S KT , reﬁned coreset of size (cid:98) n/ 2 m (cid:99)
B P ROOF OF T HM . 1: S INGLE FUNCTION GUARANTEES FOR KT - SPLIT
We prove the theorem for the case (cid:96) = 1 , and for other indices, the proof can be derived by repeating the arguments.
Deﬁne
[Equation]
Next, we use the results about an intermediate algorithm, kernel halving (Dwivedi & Mackey, 2021,
Alg. 3) that was introduced for the analysis of kernel thinning.
Using the arguments from Dwivedi & Mackey (2021, Sec. 5.2), we conclude that KT - SPLIT with k split set as k and thinning parameter m , is equivalent to repeated kernel halving with kernel k for m rounds (with no Failure in any rounds of kernel halving).
On this event of equivalence, denoted by E equi , Dwivedi & Mackey (2021, Eqns.
(50, 51)) imply that the function (cid:102) W m ∈ H is equal in distribution to another random function W m , where W m is unconditionally sub-Gaussian with parameter
[Equation]
that is,
[Equation]
where we note that the analysis of Dwivedi & Mackey (2021) remains unaffected when we replace (cid:107) k (cid:107) ∞ by (cid:107) k (cid:107) ∞ , in in all the arguments.
Applying the sub-Gaussian Hoeffding inequality (Wainwright, 2019, Prop. 2.5) along with (10), we obtain that
[Equation]
Call this event E sg .
As noted above, conditional to the event E equi , we also have
[Equation]
where d = denotes equality in distribution.
Furthermore, Dwivedi & Mackey (2021, Eqn. 48) implies that
[Equation]
[Equation]
Putting the pieces together, we have
[Equation]
as claimed.
The proof is now complete.
[-(11) for the extended empirical distributions P ′ in = 1 n (cid:80) ni =
-]
C P ROOF OF T HM . 2: MMD GUARANTEE FOR TARGET KT
First, we note that by design, KT - SWAP ensures
[Equation]
[Equation]
where S ( m, 1) denotes the ﬁrst coreset returned by KT - SPLIT .
Thus it sufﬁces to show that
MMD k ( S in , S ( m, 1) ) is bounded by the term stated on the right hand side of (3).
Let P (1) out (cid:44) 1 n/ 2 m (cid:80) x ∈S ( m, 1) δ x .
By design of KT - SPLIT , supp( P (1) out )
⊆ supp( P in ) .
Recall the set A is such that supp( P in ) ⊆
A .
Proof of (3) Let C (cid:44) C k ,ε ( A ) denote the cover of minimum cardinality satisfying (2).
Fix any f ∈ B k .
By the triangle inequality and the covering property (2) of C , we have
[Equation]
Applying Thm.
[Equation]
we have
[Equation]
[Equation]
with probability at least 1 − δ (cid:48) − (cid:80) mj
=1 (cid:80) n/ 2 j i =
δ i = p sg − δ (cid:48) .
A standard union bound then yields that sup
[Equation]
probability at least p sg − δ (cid:48) .
Since f ∈ B k was arbitrary, and C ⊂ B k and thus sup g ∈C (cid:107) g (cid:107) k ≤ 1 , we therefore have
MMD
[Equation]
with probability at least p sg − δ (cid:48) as claimed.
D P ROOF OF T HM . 3: MMD GUARANTEE FOR POWER KT
KT - SWAP step ensures that
[Equation]
where S ( m, 1) α denotes the ﬁrst coreset output by KT - SPLIT with k split = k α .
Next, we state a key interpolation result for MMD k that relates it to the MMD of its power kernels (Def. 2) (see App.
F for the proof).
Proposition 1 (An interpolation result for MMD) Consider a shift-invariant kernel k that admits valid α and 2 α -power kernels
k α and k 2 α respectively for some α ∈ [ 1 2 , 1] .
Then for any two discrete measures P and Q supported on ﬁnitely many points, we have
[Equation]
Given Prop. 1, it remains to establish suitable upper bounds on MMDs of k α and k 2 α .
To this end, ﬁrst we note that for any reproducing kernel k and any two distributions P and Q , H¨older’s inequality implies that
[Equation]
Now, let P in and P ( m, 1) α denote the empirical distributions of S in and S ( m, 1) α .
Now applying Dwivedi & Mackey (2021, Thm. 4(b)), we ﬁnd that
[Equation]
(14) with probability p sg − δ (cid:48) , where (cid:102)
M k α was deﬁned in (7).
We note that while Dwivedi & Mackey (2021, Thm. 4(b)) uses (cid:107) k α (cid:107) ∞ in their bounds, we can replace it by (cid:107)
k α (cid:107) ∞ , in , and verifying that all the steps of the proof continue to be valid (noting that (cid:107) k α (cid:107) ∞ , in is deterministic given S in ).
Furthermore, Dwivedi & Mackey (2021, Thm. 4(b)) yields that
[Equation]
with probability p sg − δ (cid:48) , where we have once again replaced the term (cid:107) k α (cid:107) ∞ with (cid:107) k α (cid:107) ∞ , in for the same reasons as stated above.
We note that the two bounds (14) and (15) apply under the same high probability event as noted in Dwivedi & Mackey (2021, proof of Thm. 1, eqn.
Putting together the pieces, we ﬁnd that
[Equation]
as claimed.
The proof is now complete.
E P ROOF OF T HM . 4: S INGLE FUNCTION & MMD GUARANTEES FOR KT+
Proof of part (8) First, we note that the RKHS H of k is contained in the RKHS H † of k † Berlinet & Thomas-Agnan (2011, Thm. 5).
Now, applying Thm. 1 with k split = k † for any ﬁxed function f ∈ H ⊂ H † and δ (cid:48) ∈
(0 , 1) , we obtain that
[Equation]
with probability at least p sg − δ (cid:48) .
Here step (i) follows from the inequality (cid:107) k † (cid:107) ∞ ≤ 2 , and step (ii) follows from the inequality (cid:107) f (cid:107) k † ≤ (cid:112) (cid:107) k
(cid:107) ∞ (cid:107) f (cid:107) k , which in turn follows from the standard facts that
[Equation]
λ
[Equation]
see, e.g., Zhang & Zhao (2013, Proof of Prop. 2.5) for a proof of step (i), Berlinet & Thomas-Agnan (2011, Thm. 5) for step (ii).
The proof for the bound (8) is now complete.
Proof of part (9) Repeating the proof of Thm.
2 with the bound (12) replaced by (8) yields that
[Equation]
with probability at least p sg .
Let us denote this event by E 1 .
To establish the other bound, ﬁrst we note that KT - SWAP step ensures that
[Equation]
where S ( m, 1) KT+ denotes the ﬁrst coreset output by KT - SPLIT with k split = k † .
We can now repeat the proof of Thm. 3, using the sub-Gaussian tail bound (8), and with a minor substitution, namely, (cid:107) k α (cid:107) ∞ , in replaced by 2 (cid:107) k α (cid:107) ∞ .
Putting it together with (17), we conclude that
[Equation]
[Equation]
with probability at least p sg .
Let us denote this event by E 2 .
Note that the quantities on the right hand side of the bounds (16) and (18) are deterministic given
S in , and thus can be computed apriori.
Consequently, we apply the high probability bound only for one of the two events E 1 or E 2 depending on which of the two quantities (deterministically) attains the minimum.
Thus, the bound (9) holds with probability at least p sg as claimed, and we are done.
F P ROOF OF P ROP . 1: A N INTERPOLATION RESULT FOR MMD
For two arbitrary distributions P and Q , and any reproducing kernel k , Gretton et al.
(2012, Lem. 4) yields that
[Equation]
Let F denote the generalized Fourier transform (GFT) operator (Wendland (2004, Def. 8.9)).
Since k ( x, y ) = κ ( x − y ) , Wendland (2004, Thm. 10.21) yields that
[Equation]
for f ∈ H .
Let (cid:98) κ (cid:44) F ( κ ) , and consider a discrete measure D =
(cid:80) n i =
w i
δ x i supported on ﬁnitely many points, and let D k ( x ) (cid:44) (cid:80) w i
k ( x i , x ) =
(cid:80) w i κ ( x i − x ) .
Now using the linearity of the GFT operator F , we ﬁnd that for any ω ∈ R d ,
[Equation]
where we used the fact that F ( κ ( x i − · ))( ω )
=
e (cid:104) ω,x i (cid:105) (cid:98) κ ( ω ) , 4 and used the shorthand (cid:98) D ( ω ) (cid:44) ( (cid:80) ni
=1 w i e (cid:104) ω,x i (cid:105) ) in the last step.
Putting together (19) to (21) with D = P − Q , we ﬁnd that
[Equation]
where step (i) makes use of Jensen’s inequality and the fact that the function t (cid:55)→ t 1 − αα for t ≥is concave for α ∈ [ 12 , 1] , and step (ii) follows by applying (22) for kernels k α and k 2 α and noting that by deﬁnition F ( k α )
= (cid:98) κ α , and F ( k 2 α )
= (cid:98) κ 2 α .
Noting MMD is a non-negative quantity, and taking square-root establishes the claim (13).
G S UB - OPTIMALITY OF SINGLE FUNCTION GUARANTEES WITH ROOT KT
Deﬁne (cid:101) k rt as the scaled version of k rt , i.e., (cid:101) k rt (cid:44) k rt / (cid:107) k rt (cid:107) ∞ that is bounded by 1 .
Then Zhang &
Zhao (2013, Proof of Prop. 2.3) implies that
[Equation]
[Equation]
And thus we also have H rt
= (cid:101)
H rt where H rt and (cid:101) H rt respectively denote the RKHSs of k rt and (cid:101) k rt .
Next, we note that for any two kernels k 1 and k 2 with corresponding RKHSs H 1 and H 2 with
H 1 ⊂ H 2 , in the convention of Zhang & Zhao (2013, Lem. 2.2, Prop. 2.3), we have
[Equation]
(cid:107)
[Equation]
Consequently, we have
[Equation]
where in the last step, we have applied the bound (24) with ( k 1 , H 1 ) ← ( k , H ) and ( k 2 , H 2 )
← ( (cid:101) k rt , (cid:101) k rt ) since H
⊂ H rt
= (cid:101) k rt .
Next, we use (25) to the kernels studied in Dwivedi & Mackey (2021) where we note that all the kernels in that work were scaled to ensure (cid:107) k (cid:107) ∞ = 1 and in fact satisﬁed k ( x, x ) = 1 .
Consequently, the multiplicative factor stated in the discussion after Thm. 1, namely, (cid:113)
[Equation] k (cid:107) ∞ , in (cid:107) f (cid:107) [Table]
rt (cid:107) f (cid:107) k can be bounded by (cid:113) λ
( H , (cid:101) H rt ) given the arguments above.
For k =
Gauss ( σ ) kernels, Zhang & Zhao (2013, Prop. 3.5(1)) yields that λ ( H , (cid:101) H rt )
= 2 d/ 2 .
For k = B-spline (2 β + 1) with β ∈ 2 N + 1 , Zhang & Zhao (2013, Prop. 3.5(1)) yields that λ ( H , (cid:101) H rt )
= 1 .
For k = Mat´ern ( ν, γ ) with ν > d , some algebra along with Zhang & Zhao (2013, Prop 3.1) yields that
[Equation]
H A DDITIONAL EXPERIMENTAL RESULTS
This section provides additional experimental details and results deferred from Sec. 4.
Common settings and error computation To obtain an output coreset of size n 12 with n input points, we (a) take every n 12 -th point for standard thinning (ST) and (b) run KT with m = 12 log 2 n using an ST coreset as the base coreset in KT - SWAP .
For Gaussian and MoG target we use i.i.d.
points as input, and for MCMC targets we use an ST coreset after burn-in as the input (see App.
H for more details).
We compute errors with respect to P whenever available in closed form and otherwise use P in .
For each input sample size n ∈ (cid:8) 2 4 , 2 6 , . . .
, 2 14 (cid:9) with δ i = 12 n , we report the mean MMD or function integration error ± 1 standard error across 10 independent replications of the experiment (the standard errors are too small to be visible in all experiments).
We also plot the ordinary least squares ﬁt (for log mean error vs log coreset size), with the slope of the ﬁt denoted as the empirical decay rate, e.g., for an OLS ﬁt with slope − 0 .
25 , we display the decay rate of n − 0 .
Details of test functions
We note the following: (a) For Gaussian targets, the error with CIF function and i.i.d.
input is measured across the sample mean over the n input points and √ n output points obtained by standard thinning the input sequence, since P f CIF does not admit a closed form.
(b) To deﬁne the function f : x (cid:55)→ k
( X (cid:48) , x ) , ﬁrst we draw a sample X ∼ P , independent of the input, and then set X (cid:48) = 2 X .
For the MCMC targets, we draw a point uniformly from a held out data point not used as input for KT.
For each target, the sample is drawn exactly once and then ﬁxed throughout all sample sizes and repetions.
H. M IXTURE OF G AUSSIANS E XPERIMENTS
Our mixture of Gaussians target is given by P = 1 M (cid:80)
Mj
=1 N ( µ j , I d ) for M ∈ { 4 , 6 , 8 } where µ 1 = [ − 3 , 3] (cid:62) , µ 2 = [ − 3 , 3] (cid:62) , µ 3 = [ − 3 , − 3] (cid:62) , µ 4 = [3 , − 3] (cid:62) , µ 5 =
0 , 6] (cid:62) , µ 6 = [ − 6 , 0] (cid:62) , µ 7 = [6 , 0] (cid:62) , µ 8 = [0 , − 6] (cid:62) . We provide visualizations similar to Fig.
1 from two other independent runs of the experiment in
Fig. 4; where we once again notice that the KT output visually provides a signiﬁcantly improved approximation of the underlying 8-MoG target compared to i.i [Figure] d. points.
Finally, we provide the MMD errors for target KT and root KT with G AUSS (4) k , target KT with G AUSS (4) and B- SPLINE (1), KT+ for L APLACE (2) k and k 0 .
[-sampling both in terms of magnitude and decay rate with input size. Morever the observed decay rates for KT+ closely match the rates guaranteed by our theory in Tab. 3. i.i.d. Root KT (Gauss) 17 .17 .
-][Figure]
[-Target KT (Gauss) 17 .17 .
-]
and KT+ [-(IMQ) 17 .17 .
 KT+ (Laplace) 
 KT+ (Bspline) 
    i.i.d.-]with IMQ(0.5, 2) k and k 0 .
[-Root KT (Gauss) 17 .17 .
 Target KT (Gauss) 17 .17 .
 KT+ (IMQ) 17 .17 .
 KT+ (Laplace) 
-]5 in Fig.
[-KT+ (Bspline) 
-][Figure]
H. MCMC EXPERIMENTS
Our set-up for MCMC experiments follows closely that of Dwivedi & Mackey (2021).
For complete details on the targets and sampling algorithms we refer the reader to Riabiz et al.
(2020a, Sec. 4).
Goodwin and Lotka-Volterra experiments From Riabiz et al.
(2020b), we use the output of four distinct MCMC procedures targeting each of two d = 4 -dimensional posterior distributions P : (1) a posterior over the parameters of the Goodwin model of oscillatory enzymatic control (Goodwin, 1965) and (2) a posterior over the parameters of the Lotka-Volterra model of oscillatory predatorprey evolution (Lotka, 1925; Volterra, 1926).
For each of these targets, Riabiz et al.
(2020b) provide 2 × 10 6 sample points from the following four MCMC algorithms: Gaussian random walk (RW), adaptive Gaussian random walk (adaRW, Haario et al., 1999), Metropolis-adjusted Langevin algorithm
(MALA, Roberts & Tweedie, 1996), and pre-conditioned MALA (pMALA, Girolami &
Calderhead, 2011).
Hinch experiments Riabiz et al.
(2020b) also provide the output of two independent Gaussian random walk MCMC chains targeting each of two d = distributions [Table] [Figure] et al., 2004) and (2) a tempered version of the same posterior, as deﬁned by Riabiz et al.
(2020a,
App. S5.4).
Burn-in and standard thinning
We discard the initial burn-in points of each chain using the maximum burn-in period reported in Riabiz et al.
(2020a, Tabs.
S4 & S6, App. S5.4).
Furthermore, we also normalize each Hinch chain by subtracting the post-burn-in sample mean and dividing each coordinate by its post-burn-in sample standard deviation.
To obtain an input sequence S in of length n to be fed into a thinning algorithm, we downsample the remaining even indices of points using standard thinning (odd indices are held out).
When applying standard thinning to any Markov chain output, we adopt the convention of keeping the ﬁnal sample point.
The selected burn-in periods for the Goodwin task were 820,000 for RW; 824,000 for adaRW; 1,615,000 for MALA; and 1,475,000 for pMALA.
The respective numbers for the Lotka-Volterra task were 1,512,000 for RW; 1,797,000 for adaRW; 1,573,000 for MALA; and 1,251,000 for pMALA.
I U PPER BOUNDS ON RKHS COVERING NUMBERS
In this section, we state several results on covering bounds for RKHSs for generic and some speciﬁc kernels.
These bounds can then be used with Thm. 2 (or Tab. 2) to establish MMD guarantees for the output of kernel thinning as summarized in Tab. 3.
We ﬁrst state covering number bounds for RKHS associated with generic kernels, that are either (a) analytic, or (b) ﬁnitely many times differentiable.
These results follow essentially from Sun & Zhou (2008); Steinwart & Christmann (2008), but we provide a proof in App.
I.2 for completeness.
Proposition 2 (Covering numbers for analytic and differentiable kernels)
The following results hold true.
(a) Analytic kernels: Suppose that k ( x, y ) =
κ ( (cid:107) x − y (cid:107) 22 ) for κ : R + → R real-analytic with convergence radius R κ , that is,
[Equation]
for some constant C κ , where κ ( j )+ denotes the right-sided j -th derivative of κ .
Then for any set A ⊂ R d and any ε ∈ (0 , 12 ) , we have
[Equation]
[Equation]
where r † (cid:44) min (cid:16) √ R κ 2 d , (cid:112) R κ + D 2 A − D A (cid:17) , and D A (cid:44) max x,y ∈A (cid:107) x − y (cid:107) 2 .
[Equation]
28) (b) Differentiable kernels: Suppose that for X ⊂ R d , the kernel k : X × X
→ R is s -times continuously differentiable, i.e., all partial derivatives ∂ α,α k : X × X
→ R exist and are continuous for all multi-indices α ∈ N d 0 with | α | ≤ s .
Then, for any closed Euclidean ball ¯ B 2 ( r ) contained in X and any ε > 0 , we have
[Equation]
for some constant c s,d, k that depends only on on s, d and k .
Next, we state several explicit bounds on covering numbers for several popular kernels.
See App. I. for the proof.
Proposition 3 (Covering numbers for speciﬁc kernels)
The following statements hold true.
(a) When k = G AUSS ( σ ) , we have
[Equation]
where C G AUSS ,d (cid:44) (cid:0) 4 e + d d (cid:1) e − d ≤ (cid:26) 4 .
for d =
[Equation] [Equation]
[Table]
i n M A L A
L a
[Table]
l a c e k [Table] L o t k a M A L A
L a p l a c e k
MMD
[Table]
( in out )
[Table]
1 |
[Table]
[Table]
[Table]
[Table]
[Table]
)|
[Table]
[Table]
[Table]
H i n c h
T e m p e r e d
I M Q k
MMD
[Table]
Coreset size n
[Table]
G oo d w i n p M A L A
L a p l a c e k
MMD k ( in
[Table]
[Table]
|(
[Table]
[Table]
|
[Table]
[Table]
a p
M A L A
L a p l a c e
[Table]
H i n c h
T e m p e r e d
I M Q k
MMD k
[Table]
KT+: n 0.
222 7 Coreset size n 222 7 |( in out ) k
( X 0 ) |
ST: n 0.32 KT+: n 0.
222 7 Coreset size n 222|( in out ) f CIF | ST: n 0.41 KT+: n 0. (c) When k = IMQ ( ν, γ ) , we have
[Equation]
where C IMQ ,ν,γ (cid:44) 4 log (cid:16) 16 (2 ν +1)
ν +1 γ 2 ν +1 (cid:17) , and (cid:101) r (cid:44) min (cid:16) γ 2 d , (cid:112)
γ 2 + 4 r 2 − 2 r (cid:17) .
[Equation] (d) When k = SINC - RAD ( θ ) , then for d ∈ { 1 , 2 , 3 } and ε ∈ (0 , 12 ) , we have
[Equation]
where (cid:101) r θ,d (cid:44) min (cid:16) √ 3 | θ | d , (cid:113)
12 θ 2 + 4 r 2 − 2 r (cid:17) .
[Equation]
(e) When k = SINC ( θ ) , then for ε ∈ (0 , 12 ) and (cid:101) r θ, 1 as in (36) , we have
[Equation]
(f) When k = B- SPLINE (2 β + 1) , then for some universal constant C B- SPLINE , we have
[Equation]
I. A UXILIARY RESULTS ABOUT RKHS AND E UCLIDEAN COVERING NUMBERS
In this section, we collect several results regarding the covering numbers of Euclidean and RKHS spaces that come in handy for our proofs.
These results can also be of independent interest.
We start by deﬁning the notion of restricted kernel and its unit ball (Rudi et al.
(2020, Prop. 8)).
For
X ⊂ R d , let | X denotes the restriction operator.
That is, for any function f : R d → R , we have f | X : X → R such that f | A ( x ) =
f
( x ) for x ∈ X .
Deﬁnition 3 (Restricted kernel and its RKHS) Consider a kernel k deﬁned on R d × R d with the corresponding RKHS H , any set X ⊂ R d .
The restricted kernel k |X is deﬁned as k | X : X × X
→ R such that k | X ( x, y ) (cid:44) k | X×X ( x, y )
=
k ( x, y ) for all x, y ∈ X , and H| X denotes its RKHS.
For f ∈ H| X , the restricted RKHS norm is deﬁned as follows:
[Equation]
such that h | X = f. Furthermore, we use B k |X (cid:44) { f ∈ H|
X : (cid:107) f (cid:107) k
| X ≤ 1 } to denote the unit ball of the RKHS corresponding to this restricted kernel.
In this notation, the unit ball of unrestricted kernel satisﬁes B k (cid:44) B k | R d .
Now, recall the RKHS covering number deﬁnition from Def. 1.
In the sequel, we also use the covering number of the restricted kernel deﬁned as follows:
[Equation]
(39) that is N † k
( X , ε ) denotes the minimum cardinality over all possible covers C ⊂ B k |X that satisfy
[Equation]
) |≤ ε (cid:9) .
With this notation in place, we now state a result that relates the covering numbers N † (39) and
N Def. 1.
Lemma 1 (Relation between restricted and unrestricted RKHS covering numbers)
We have
[Equation]
Proof of Lem. Rudi et al.
(2020, Prop. 8(d,f)) imply that there exists a bounded linear extension operator E : H| X
→ H with operator norm bounded by 1 , which when combined with Steinwart &
Christmann (2008, eqns.
(A.38), (A.39)) yields the claim.
Next, we state results that relate RKHS covering numbers for a change of domain for a shift-invariant kernel.
We use B (cid:107)·(cid:107) ( x ; r ) (cid:44) (cid:8) y ∈ R
d : (cid:107) x − y (cid:107) ≤ r (cid:9) to denote the r radius ball in R d deﬁned by the metric induced by a norm (cid:107)·(cid:107) .
Deﬁnition 4 (Euclidean covering numbers)
Given a set X ⊂ R d , a norm (cid:107)·(cid:107) , and a scalar ε > 0 , we use N (cid:107)·(cid:107) ( X , ε ) to denote the ε -covering number of X with respect to (cid:107)·(cid:107) -norm.
That is,
N (cid:107)·(cid:107) ( X , ε ) denotes the minimum cardinality over all possible covers C ⊂ X that satisfy
[Equation]
When (cid:107)·(cid:107) = (cid:107)·(cid:107) q for some q ∈ [1 , ∞ ] , we use the shorthand N q (cid:44) N (cid:107)·(cid:107) q .
Lemma 2 (Relation between RKHS covering numbers on different domains) Given a shiftinvariant kernel k , a norm (cid:107)·(cid:107) on R d , and any set X ⊂ R d , we have
[Equation]
Proof of Lem.
Let C ⊂ X denote the cover of minimum cardinality such that
[Equation]
We then have
[Equation]
where step (i) follows by applying Steinwart & Fischer (2021, Lem. 3.11), 5 and step (ii) follows by applying Steinwart & Fischer (2021, Lem. 3.10).
The claim follows by noting that C denotes a cover of minimum cardinality, and hence by deﬁnition |C| =
N (cid:107)·(cid:107) ( X , 1) .
Lemma 3 (Covering number for product kernel)
Given X ⊂ R and a reproducing kernel κ :
X × X → R , consider the product kernel k (cid:44) κ ⊗ d : X ⊗ 2 d → R deﬁned as
[Equation]
Then the covering numbers of the two kernels are related as follows:
[Equation]
Proof of Lem. Let H denote the RKHS corresponding to κ .
Then the RKHS corresponding to the kernel k is given by the tensor product H k (cid:44)
H × H × . . .
× H Berlinet & Thomas-Agnan (2011, Sec. 4.6), i.e., for any f ∈ H k , there exists f 1 , f 2 , . . .
, f d ∈ H such that
[Equation]
[Equation]
Let C κ ( X , ε ) ⊂ B κ denote an ε -cover of B κ in L ∞ -norm
(Def. 1).
Then for each f i ∈ H , we have (cid:101)
f i ∈ C κ
( X , ε ) such that
[Equation]
Now
[Equation]
we claim that for every f ∈ B k [Equation] there exists g ∈ C k (cid:44) ( C κ ( X , ε )) ⊗ d such that
[Equation]
which immediately implies the claimed bound (40) on the covering number.
We now prove the claim (43).
For any ﬁxed f ∈ H k , let f i , (cid:101)
f i denote the functions satisfying (41) and (42) respectively. Then
[Equation] we prove our claim (43) with g = (cid:81) di =1 (cid:101) f i ∈ C k .
Using the convention (cid:81)
k =1 (cid:101)
f k ( x k ) = 1 , we ﬁnd that
[Equation]
where in the last step we have used the following argument:
[Equation]
[Table]
for any h ∈ B κ
The proof is now complete
Lemma 4 (Relation between Euclidean covering numbers)
We have
[Equation]
for all d ≥ 1 .
Proof of Lem. We apply Wainwright (2019, Lem. 5.7) with B = B 2 ( r ) and B (cid:48) =
B ∞ (1) to conclude that
[Equation]
where Vol( X ) denotes the d -dimensional Euclidean volume of X ⊂ R d , and Γ( a ) denotes the
Gamma function.
Next, we apply the following bounds on the Gamma function from Batir (2017,
Thm. 2.2):
Γ(
[Equation]
+
[Equation] ≥ ( [Equation] e ) b √ 2 πb for any b ≥ 1 , and Γ( b + 1) ≤ ( b/e ) b √ e 2 b for any b ≥ 1 . 1 .
Thus, we have
[Equation]
as claimed, and we are done.
I.
P ROOF OF P ROP . 2: C OVERING NUMBERS FOR ANALYTIC AND DIFFERENTIABLE KERNELS
First we apply Lem.
1 so that it remains to establish the stated bounds simply on log N † k
( X , ε ) .
Proof of bound (27) in part (a) The bound (27) for the real-analytic kernel is a restatement of
Sun & Zhou (2008, Thm. 2) in our notation (in particular, after making the following substitutions in their notation: R ← 1 , C 0 ← C κ , r ← R κ , X ← A , (cid:101) r ← r † , η ← ε, D ← D 2 A , n ← d ).
Proof of bound (29) for part (b): Under these assumptions, Steinwart & Christmann (2008,
Thm. 6.26) states that the i -th dyadic entropy number Steinwart & Christmann (2008, Def. 6.20) of the identity inclusion mapping from H| ¯ B 2 ( r ) to L ∞ ¯ B 2 ( r ) is bounded by c (cid:48)
s,d, k · r s i − s/d for some constant c (cid:48) s,d, k independent of ε and r .
Given this bound on the entropy number, and applying Steinwart & Christmann (2008, Lem. 6.21), we conclude that the log-covering number log N † k ( ¯ B 2 ( r ) , ε ) is bounded by ln 4 · ( c (cid:48) s,d, k r s /ε )
d/s =
c s,d, k r d · (1 /ε )
d/s as claimed.
I. P ROOF OF P ROP . 3: C OVERING NUMBERS FOR SPECIFIC KERNELS
First we apply Lem.
1 so that it remains to establish the stated bounds in each part on the corresponding log N k .
Proof for G AUSS kernel: Part (a)
The bound (30) for the Gaussian kernel follows directly from
Steinwart & Fischer (2021, Eqn. 11) along with the discussion stated just before it.
Furthermore, the bound (31) for C Gauss ,d are established in Steinwart & Fischer (2021, Eqn. 6), and in the discussion around it.
Proof for M AT ´ ERN kernel: Part (b) Since M AT ´ ERN ( ν, γ ) is ν -times continuously differentiable Stein (2012), the bound (32) immediately follows from Prop. 3(b).
Proof for IMQ kernel: Part (c)
The bounds (33) and (34) follow from Sun & Zhou (2008, Ex. 3), and noting that N 2 ( B 2 ( r ) , (cid:101) r/ 2) is bounded by (1 + 4 r (cid:101) r ) d Wainwright (2019, Lem. 5.7).
Proof for SINC - RAD kernel: Part (d) First, we note that the SINC - RAD kernel is positive deﬁnite in d = { 1 , 2 , 3 } Schoenberg (1938), and thus is a reproducing kernel.
Next, we claim that it is an analytic kernel that satisﬁes the condition (26) of Prop. 2(a) with κ ( t ) =
SINC ( θ √ t ) , R κ = 12 θ 2 , and
C κ = 1 ; and thus applying the bounds (27) and (28) from Prop. 2(a) with A = B d 2 ( r ) yields the claimed bound (35) and (36).
To verify the condition (26) with the stated parameters, we note that
[Equation] which implies [Equation] and we are done.
Proof for SINC kernel: Part (e) For k = SINC ( θ ) , we can write k ( x, y )
= (cid:81) di
=1 κ θ ( x i − y i ) for κ θ : R → R deﬁned as κ θ ( t )
= sin( θt ) θt ( i ) =
sin( | θt | ) | θt | , where step (i) follows from the fact that t (cid:55)→ sin t/t is an even function.
Thus, we can apply Lem.
Given the bound (40), and noting that (cid:107) κ θ (cid:107) ∞
= 1 , it sufﬁces to establish the univariate version of the bound (37), namely,
[Equation]
which follows directly by substituting d = 1 in part(d).
Proof for B- SPLINE kernel: Part (f) For k = B-
SPLINE (2 β + 1) , we can write k ( x, y )
= (cid:81) di
=1 κ β ( x i − y i ) for κ β : R → R deﬁned as κ β ( t ) =
B − 1 2 β +2 (cid:126) 2 β +2 1 [ − 12 , 12 ] ( t ) , and thus we can apply Lem. 3.
Given the bound (40), and noting that (cid:107) κ β (cid:107) ∞
= 1 (Dwivedi & Mackey (2021,
Eqn. 107)), it sufﬁces to establish the univariate version of the bound (38), namely,
[Equation]
[Equation]
where we abuse notation and use κ β to denote the univariate B- SPLINE kernel.
The bound (44) follows from Zhou (2003, Ex. 4) and using the fact that log N † κ β ([ − 12 , 12 ] , ε )
=
log N † κ β ([0 , 1] , ε ) since κ β is shift-invariant.
. J P ROOF OF T AB . 3 RESULTS
In Tab. 2, the stated results for all the entries in target KT column follow directly by substituting the covering number bounds from Prop. 3 in the corresponding entry along with the stated radii growth conditions for the target P .
For the KT+ column, results follow from the ﬁrst column, and noting the following root or power KT guarantees:
For IMQ ( ν, γ ) kernel, one can establish a guarantee for root KT using a square-root dominating kernel (cid:101) k rt IMQ ( ν (cid:48) , γ (cid:48) )
Dwivedi & Mackey (2021, Def.2), and Dwivedi & Mackey (2021, Eqn.(117)) shows that they are always deﬁned for appropriate choices of ν (cid:48) , γ (cid:48) .
The best root KT guarantees are obtained by choosing largest possible ν (cid:48)
(to allow the most rapid decay of tails), and Dwivedi &
Mackey (2021, Eqn.(117)) implies with ν < d 2 , the best possible parameter satisﬁes ν (cid:48)
≤ d 4 + ν 2 .
For this parameter, some algebra shows that max( R † (cid:101)
k rt ,n R (cid:101) k rt ,n ) (cid:45) d,ν,γ n 1 / 2 ν , leading to the guarantee stated for root KT in the corresponding row.
When ν ≥ d 2 , we can use a M AT ´ ERN kernel as a square-root dominating kernel from Dwivedi & Mackey (2021, Prop. 3), and then applying the bounds for the kernel radii (5), and the inﬂation factor (7) for a generic Mat´ern kernel from Dwivedi & Mackey (2021, Tab. 3) yields the entry for root KT stated in Tab. 2.
For M AT ´ ERN ( ν, γ ) kernel, the α -power kernel is given by M AT
´ ERN ( aν, γ ) if aν > d 2 (a proof of this follows from Def. 2 and Dwivedi & Mackey (2021, Eqns (71-72))).
Since L APLACE ( σ )
=
M AT ´ ERN ( d +12 , σ − 1 ) , we conclude that its α -power kernel is deﬁned for a > dd +1 .
And using the various tail radii (5), and the inﬂation factor (7) for a generic Mat´ern kernel from Dwivedi & Mackey (2021, Tab. 3), we conclude that the MMD bounds remain same as root KT (upto factors depending on kernel parameters and d ), leading to the entries in the KT+ column for M AT ´ ERN row.
For SINC ( θ ) kernel, the square-root kernel is SINC ( θ ) itself (as can be seen by Dwivedi & Mackey (2021, Prop.2) and the fact that Fourier transform of SINC is a rectangle function on bounded domain).
However, one can directly verify that the tail of SINC kernel does not decay fast enough leading to a vacuous bound of n − 14 for the root KT (with baseline thinning as i.i.d.
thinning as noted in Dwivedi & Mackey (2021, Rem. 2)).
For B- SPLINE (2 β +1) kernel, using arguments similar to that in Dwivedi & Mackey (2021, Tab.4), we conclude that the α -power kernel is deﬁned to be B- SPLINE ( A + 1) whenever A (cid:44) 2 αβ + 2 α − 1 ∈ 2 N 0 +1 .
For odd β we can always take α = 12 and the square-root kernel is well deﬁned Dwivedi & Mackey (2021).
For even β , we have to choose α (cid:44) p +1 β +1 ∈ ( 12 , 1) by taking p ∈ N suitably as long as β ≥ 2 .
Whenever the α -power kernel is deﬁned, we can then apply the various tail radii (5), and the inﬂation factor (7) for the power B- SPLINE kernel from Dwivedi & Mackey (2021, Tab. 3) to obtain the MMD rates for power KT from Dwivedi & Mackey (2021, Tab. 2) (which remains the same as root KT upto factors depending on α and β ), thereby providing the corresponding entry for
KT+ in our Tab. 3.