D ISTRIBUTION C OMPRESSION IN N EAR - LINEAR T IME
A BSTRACT
In distribution compression, one aims to accurately summarize a probability distribution P using a small number of representative points.
Near-optimal thinning procedures achieve this goal by sampling n points from a Markov chain and identifying √ n points with (cid:101)
O
(1 / √ n ) discrepancy to P .
Unfortunately, these algorithms suffer from quadratic or super-quadratic runtime in the sample size n .
To address this deficiency, we introduce Compress++, a simple meta-procedure for speeding up any thinning algorithm while suffering at most a factor of 4 in error.
When combined with the quadratic-time kernel halving and kernel thinning algorithms of Dwivedi and Mackey (2021), Compress++ delivers √ n points with
O ( (cid:112) log n/n ) integration error and better-than-Monte-Carlo maximum mean discrepancy in O ( n log 3 n ) time and O ( √ n log 2 n ) space.
Moreover, Compress++ enjoys the same near-linear runtime given any quadratic-time input and reduces the runtime of super-quadratic algorithms by a square-root factor.
In our benchmarks with high-dimensional Monte Carlo samples and Markov chains targeting challenging differential equation posteriors, Compress++ matches or nearly matches the accuracy of its input algorithm in orders of magnitude less time.
I NTRODUCTION
Distribution compression—constructing a concise summary of a probability distribution—is at the heart of many learning and inference tasks.
For example, in Monte Carlo integration and Bayesian inference, n representative points are sampled i.i.d.
or from a Markov chain to approximate expectations and quantify uncertainty under an intractable (posterior) distribution (Robert & Casella, 1999).
However, these standard sampling strategies are not especially concise.
For instance, the
Monte Carlo estimate P in f ≜ 1 n (cid:80) ni
=1 f ( x i ) of an unknown expectation P f ≜ E X ∼ P [ f ( X )] based on n i.i.d.
points has Θ( n − 12 ) integration error | P f − P in f | , requiring 10000 points for 1% relative error and 10 6 points for 0 .
1% error.
Such bloated sample representations preclude downstream applications with critically expensive function evaluations like computational cardiology, where a 1000-CPU-hour tissue or organ simulation is required for each sample point (Niederer et al., 2011;
Augustin et al., 2016; Strocchi et al., 2020).
To restore the feasibility of such critically expensive tasks, it is common to thin down the initial point sequence to produce a much smaller coreset.
The standard thinning approach, select every t -th point (Owen, 2017), while being simple often leads to an substantial increase in error: e.g., standard thinning n points from a fast-mixing Markov chain yields Ω( n − 14 ) error when n 12 points are returned.
Recently, Dwivedi & Mackey (2021) introduced a more effective alternative, kernel thinning (KT), that provides near optimal (cid:101) O d ( n − 12 ) error when compressing n points in R d down to size n 12 .
While practical for moderate sample sizes, the runtime of this algorithm scales quadratically with the input size n , making its execution prohibitive for very large n .
Our goal is to significantly improve the runtime of such compression algorithms while providing comparable error guarantees.
Problem setup Given a sequence S in of n input points summarizing a target distribution P , our aim is to identify a high quality coreset S out of size √ n in time nearly linear in n .
We measure coreset quality via its integration error | P f − P S out f | ≜ | P f − 1 |
[Equation]
|
[Equation]
f ( x ) | for functions f in the reproducing kernel Hilbert space (RKHS)
H k induced by a given kernel k (Berlinet &
Thomas-Agnan, 2011).
We consider both single function error and kernel maximum mean discrepancy (MMD, Gretton et al., 2012), the worst-case integration error over the unit RKHS norm ball:
[Equation]
Our contributions We introduce a new simple meta procedure—C OMPRESS ++—that significantly speeds up a generic thinning algorithm while simultaneously inheriting the error guarantees of its input up to a factor of 4 .
A direct application of C OMPRESS ++ to KT improves its quadratic Θ( n 2 ) runtime to near linear O
( n log 3 n ) time while preserving its error guarantees.
Since the (cid:101) O d ( n − 12 )
KT MMD guarantees of Dwivedi & Mackey (2021) match the Ω( n − 12 ) minimax lower bounds of
Tolstikhin et al.
; Phillips & Tai (2020) up to factors of √ log n and constants depending on d , KT-C OMPRESS ++ also provides near-optimal MMD compression for a wide range of k and P .
Moreover, the practical gains from applying C OMPRESS ++ are substantial: KT thins 65 , 000 points in 10 dimensions in 20 m, while KT-C OMPRESS ++ needs only 1.5m; KT takes more than a day to thin 250 , 000 points in 100 dimensions, while KT-C OMPRESS ++ takes less than 1hr (a 32 × speedup).
For larger n , the speed-ups are even greater due to the order n log 3 n reduction in runtime.
C OMPRESS ++ can also be directly combined with any thinning algorithm, even those that have suboptimal guarantees but often perform well in practice, like kernel herding (Chen et al., 2010),
MMD-critic (Kim et al., 2016), and Stein thinning (Riabiz et al., 2020a), all of which run in Ω( n 2 ) time.
As a demonstration, we combine C OMPRESS ++ with the popular kernel herding algorithm and observe 45 × speed-ups when compressing 250 , 000 input points.
In all of our experiments,
C OMPRESS ++ leads to minimal loss in accuracy and, surprisingly, even improves upon herding accuracy for lower-dimensional problems.
Most related to our work are the merge-reduce algorithms of Matousek (1995); Chazelle & Matousek (1996); Phillips (2008) which also speed up input thinning algorithms while controlling approximation error.
In our setting, merge-reduce runs in time Ω( n 1 . 5 ) given an n 2 -time input and in time Ω( n ( τ +1) / 2 ) for slower n τ -time inputs (see, e.g., Phillips, 2008, Thm. 3.1).
In contrast, C OMPRESS ++ runs in near-linear O ( n log 3 n ) time for any n 2 -time input and in O ( n τ/ 2 log τ n ) time for slower n τ -time inputs.
After providing formal definitions in Sec. 2, we introduce and analyze C OMPRESS ++ and its primary subroutine C OMPRESS in Secs.
3 and 4, demonstrate the empirical benefits of C OMPRESS ++ in Sec. 5, and present conclusions and opportunities for future work in Sec. 6.
Notation We let P S denote the empirical distribution of S .
For the output coreset S A LG of an algorithm A LG with input coreset S in , we use the simpler notation P A LG ≜ P S A LG and P in ≜ P S in .
We extend our MMD definition to point sequences ( S 1 , S 2 ) via MMD k
( S 1 , S 2 ) ≜
MMD k
( P S 1 , P S 2 ) and MMD k
( P , S 1 ) ≜
MMD k
( P , P S 1 ) .
We use a ≾ b to mean a = O ( b ) , a ≿ b to mean a = Ω( b ) , a = Θ( b ) to mean both a = O ( b ) and a = Ω( b ) , and log to denote the natural logarithm.
T HINNING AND H ALVING A LGORITHMS
We begin by defining the thinning and halving algorithms that our meta-procedures take as input.
Definition 1 (Thinning and halving algorithms)
A thinning algorithm
A LG takes as input a point sequence S in of length n and returns a (possibly random) point sequence S A LG of length n out .
We say A LG is α n -thinning if n out =
⌊ n/α n ⌋ and root-thinning if α n = √ n .
Moreover, we call A LG a halving algorithm if S A LG always contains exactly ⌊ n 2 ⌋ of the input points.
Many thinning algorithms offer high-probability bounds on the integration error | P S in f − P S
A LG f | .
We capture such bounds abstractly using the following definition of a sub-Gaussian thinning
Definition 2 (Sub-Gaussian thinning algorithm)
For a function f , we call a thinning algorithm
A LG f -sub-Gaussian with parameter ν and write A LG ∈ G f ( ν ) if
[Equation]
Def. 2 is equivalent to a sub-Gaussian tail bound for the integration error, and, by Boucheron et al.
(2013, Section 2.3), if A LG ∈ G f ( ν ) then E [ P S A LG f | S in ] =
P S in f and, for all δ ∈ (0 , 1) , | P S in f − P S
A LG f | ≤ ν ( n ) (cid:113) 2 log( 2 δ ) , with probability at least 1 − δ given S in .
Hence the integration error of A LG is dominated by the sub-Gaussian parameter ν ( n ) .
Example 1 ( KT - SPLIT )
Given a kernel k and n input points S in , the KT - SPLIT ( δ ) algorithm 1 of
Dwivedi & Mackey (2022; 2021, Alg. 1a) takes Θ( n 2 ) kernel evaluations to output a coreset of size n out with better-than-i.i.d.
integration error.
Specifically, Dwivedi & Mackey (2022, Thm. 1) prove that, on an event with probability 1 − δ 2 , KT - SPLIT ( δ ) ∈
G f ( ν ) with
[Equation]
for all f with ∥ f ∥ k = 1 .
■
Many algorithms also offer high-probability bounds on the kernel MMD (1), the worst-case integration error across the unit ball of the RKHS.
We again capture these bounds abstractly using the following definition of a k -sub-Gaussian thinning algorithm.
Definition 3 ( k -sub-Gaussian thinning algorithm)
For a kernel k , we call a thinning algorithm
A LG k -sub-Gaussian with parameter v and shift a and write A LG ∈ G k
( v, a ) if
[Equation]
for all t ≥ 0 .
We also call ε k , A LG ( n ) ≜ max( v n , a n ) the k -sub-Gaussian error of A LG .
Example 2 (Kernel thinning)
Given a kernel k and n input points S in , the generalized kernel thinning (KT ( δ ) ) algorithm 1 of Dwivedi & Mackey (2022; 2021, Alg. 1) takes Θ( n 2 ) kernel evaluations to output a coreset of size n out with near-optimal MMD error.
In particular, by leveraging an appropriate auxiliary kernel k split , Dwivedi & Mackey (2022, Thms. 2-4) establish that, on an event with probability 1 − δ 2 , KT ( δ ) ∈
G k ( a, v ) with a n = C a n out (cid:112)
∥ k split ∥ ∞ , and v n = C v n out (cid:113) ∥ k split ∥ ∞ log( 6 n out log 2 ( n/n out ) δ ) M S in , k split , (4) where ∥ k split ∥ ∞ =
sup x k split ( x, x ) , C a and C v are explicit constants, and M S in , k split ≥ 1 is nondecreasing in n and varies based on the tails of k split and the radius of the ball containing S in .
■
C OMPRESS
The core subroutine of C OMPRESS ++ is a new meta-procedure called C OMPRESS that, given a halving algorithm H ALVE , an oversampling parameter g , and n input points, outputs a thinned coreset of size 2 g √ n .
The C OMPRESS algorithm (Alg. 1) is very simple to implement: first, divide the input points into four subsequences of size n 4 (in any manner the user chooses); second, recursively call
C OMPRESS on each subsequence to produce four coresets of size 2 g − 1 √ n ; finally, call H ALVE on the concatenation of those coresets to produce the final output of size 2 g √ n .
As we show in App.
H,
C OMPRESS can also be implemented in a streaming fashion to consume at most O (4 g √ n ) memory.
3. I NTEGRATION ERROR AND RUNTIME GUARANTEES FOR C OMPRESS
Our first result relates the runtime and single-function integration error of C OMPRESS to the runtime and error of H ALVE .
We measure integration error for each function f probabilistically in terms of the sub-Gaussian parameter ν of Def. 2 and measure runtime by the number of dominant operations performed by H ALVE (e.g., the number of kernel evaluations performed by kernel thinning).
Theorem 1 (Runtime and integration error of C OMPRESS )
If H ALVE has runtime r H ( n ) for inputs of size n , then C OMPRESS has runtime
[Equation]
where ℓ n ≜ 2 g +1 √ n (twice the output size of C OMPRESS ), and β n ≜ log 2 ( nℓ n ) = log 4 n − g − 1 .
Furthermore, if, for some function f , H ALVE ∈ G f ( ν H ) , then C OMPRESS ∈
G f ( ν C ) with
[Equation]
[Equation] [Equation]
As we prove in App.
B, the runtime guarantee (5) is immediate once we unroll the C OMPRESS recursion and identify that C OMPRESS makes 4 i calls to H ALVE with input size ℓ n 2 − i .
The error guarantee (6) is more subtle: here, C OMPRESS benefits significantly from random cancellations among the conditionally independent and mean-zero H ALVE errors.
Without these properties, the errors from each H ALVE call could compound without cancellation leading to a significant degradation in C OMPRESS quality.
Let us now unpack the most important implications of Thm. 1.
Remark 1 (Near-linear runtime and quadratic speed-ups for C OMPRESS )
Thm. 1 implies that a quadratic-time H ALVE with r H ( n ) = n 2 yields a near-linear time C OMPRESS with r C ( n ) ≤ 4 g +1 n (log 4 ( n ) − g ) .
If H ALVE instead has super-quadratic runtime r H ( n ) = n τ , C OMPRESS enjoys a quadratic speed-up: r C ( n ) ≤ c ′ τ n τ/ 2 for c ′ τ ≜ 2 τ ( g +2) 2 τ − 4 .
More generally, whenever H ALVE has superlinear runtime r H ( n ) = n τ ρ ( n ) for some τ ≥ 1 and non-decreasing ρ , C OMPRESS satisfies
[Equation]
for τ >where c τ
≜
τ
− 1)(
g
+
Remark 2 (C OMPRESS inflates sub-Gaussian error by at most (cid:112) log 4 n ) Thm. 1 also implies
[Equation]
in the usual case that n ν H ( n ) is non-decreasing in n .
Hence the sub-Gaussian error of C OMPRESS is at most (cid:112) log 4 n larger than that of halving an input of size ℓ n .
This is an especially strong benchmark, as ℓ n is twice the output size of C OMPRESS , and thinning from n to ℓ n 2 points should incur at least as much approximation error as halving from ℓ n to ℓ n 2 points.
Example 3 ( KT - SPLIT -C OMPRESS ) Consider running C OMPRESS with, for each H ALVE input of size ℓ , H ALVE =
KT - SPLIT ( ℓ 2 n 4 g +1 ( β n +1) δ ) from Ex. 1.
Since KT - SPLIT runs in time Θ( n 2 ) ,
C OMPRESS runs in near-linear O ( n log n ) time by Rem.
In addition, as we detail in App. F.1, on an event of probability 1 − δ 2 , every H ALVE call invoked by C OMPRESS is f -sub-Gaussian with ν H ( ℓ ) = 4 ℓ √ 3 (cid:113) log( 12 n 4 g ( β n +1) ℓδ ) ∥ k ∥ ∞ for all f with ∥ f ∥ k = 1 .
Hence, Rem. 2 implies that C OMPRESS is f -sub-Gaussian on the same event with ν C ( n ) ≤ (cid:112) log 4 n − g ν H ( ℓ n ) , a guarantee within (cid:112) log 4 n of the original KT - SPLIT ( δ ) error (2).
■
3. MMD GUARANTEES FOR C OMPRESS
Next, we bound the MMD error of C OMPRESS in terms of the MMD error of H ALVE .
Recall that
MMD k (1) represents the worst-case integration error across the unit ball of the RKHS of k .
Its proof, based on the concentration of subexponential matrix martingales, is provided in App.
C.
Theorem 2 (MMD guarantees for C OMPRESS )
Suppose H ALVE ∈
G k ( a, v ) for n a n and n v n non-decreasing and E (cid:2) P H ALVE k | S in (cid:3) =
P in k .
Then C OMPRESS ∈
G k ( (cid:101) a, (cid:101) v ) with (cid:101)
[Equation]
n
[Equation]
(cid:101)
[Equation]
(cid:112)
[Equation] [Equation]
4 n
[Equation]
, and (cid:101) a n ≜ (cid:101) c n log( n +1)+ (cid:101) v n (cid:112) log( n +1) , (8) where (cid:101) c n ≜ 2( a ℓ n + v ℓ n ) and ℓ n = 2 g +1 √ n as in Thm. 1.
Remark 3 (Symmetrization)
We can convert any halving algorithm into one that satisfies the unbiasedness condition E (cid:2) P H ALVE k | S in (cid:3) =
P in k without impacting integration error by symmetrization , i.e., by returning either the outputted half or its complement with equal probability.
Remark 4 (C OMPRESS inflates MMD guarantee by at most 9 log( n +1) )
Thm. 2 implies that the k -sub-Gaussian error of C OMPRESS is always at most 9 log( n +1) times that of H ALVE with input size ℓ n =2 g +1 √ n since ε
[Equation] [Equation]
As in Rem. 2, H ALVE applied to an input of size ℓ n is a particularly strong benchmark, as thinning from n to ℓ n 2 points should incur at least as much MMD error as halving from ℓ n to ℓ n 2 .
Example 4 (KT-C OMPRESS ) Consider running C OMPRESS with, for each H ALVE input of size ℓ ,
H ALVE =
KT ( ℓ 2 n 4 g +1 ( β n +1) δ ) from Ex. 2 after symmetrizing as in Rem.
Since KT has Θ( n 2 ) runtime, C OMPRESS yields near-linear O
( n log n ) runtime by Rem.
Moreover, as we detail in
App.
F.2, using the notation of Ex. 2, on an event E of probability at least 1 − δ 2 , every H ALVE call invoked by C OMPRESS is k -sub-Gaussian with
[Equation]
and
[Equation]
=
[Equation]
) M S in , k .
Hence, Rem. 4 implies that, on E , KT-C OMPRESS has k -sub-Gaussian error ε k , C OMPRESS ( n ) ≤ 9 log( n +1) · ε k , H ALVE ( ℓ n ) , a guarantee within 9 log( n +1) of the original KT ( δ ) MMD error (4).
■
C OMPRESS ++
To offset any excess error due to C OMPRESS while maintaining its near-linear runtime, we next introduce C OMPRESS
++ (Alg. 2)
, a simple two-stage meta-procedure for faster root-thinning.
C OMPRESS ++ takes as input an oversampling parameter g , a halving algorithm H ALVE , and a 2 g -thinning algorithm T HIN (see Def. 1).
In our applications, H ALVE and T HIN are derived from the same base algorithm
(e.g., from KT with different thinning factors), but this is not required.
C OMPRESS +
+ first runs the faster but slightly more erroneous C OMPRESS ( H ALVE , g ) algorithm to produce an intermediate coreset of size 2 g √ n .
Next, the slower but more accurate T HIN algorithm is run on the greatly compressed intermediate coreset to produce a final output of size √ n .
In the sequel, we demonstrate how to set g to offset error inflation due to C OMPRESS while maintaining its fast runtime.
Algorithm 2: C OMPRESS ++
Input: oversampling sequence
S
C ←
C OMPRESS (
ALVE ,
, S g n
S C++ ←
T HIN ( S C ) //
n return S C++
4. I NTEGRATION ERROR AND RUNTIME GUARANTEES FOR C OMPRESS ++
The following result, proved in App.
D, relates the runtime and single-function integration error of
C OMPRESS ++ to the runtime and error of H ALVE and T HIN .
Theorem 3 (Runtime and integration error of C OMPRESS ++) If H ALVE and T HIN have runtimes r H ( n ) and r T ( n ) respectively for inputs of size n , then C OMPRESS ++ has runtime r C++ ( n ) =
r C ( n ) + r T ( ℓ n / 2) where r C ( n ) (5) =
(cid:80) β n i =0
i · r H ( ℓ n 2 − i ) , (9) ℓ n = 2 g +1 √ n , and β n = log 4 n − g − 1 as in Thm. 1.
Furthermore, if for some function f ,
H ALVE ∈
G f ( ν H ) and T HIN ∈ G f ( ν T ) , then C OMPRESS ++ ∈ G f ( ν C++ ) with ν 2 C++ ( n ) =
ν 2 C ( n ) + ν 2 T ( ℓ n / 2) where ν 2 C ( n ) (6)
= (cid:80) β n i =
4 − i · ν 2 H ( ℓ n 2 − i ) .
Remark 5 (Near-linear runtime and near-quadratic speed-ups for C OMPRESS ++)
When
H ALVE and T HIN have quadratic runtimes with max( r H ( n ) , r T ( n ))
= n 2 , Thm. 3 and Rem.
1 yield that r C++ ( n ) ≤ 4 g +1 n (log 4 ( n ) − g )
+ 4 g n .
Hence, C OMPRESS ++ maintains a near-linear runtime
[Equation]
whenever
[Equation]
= O (
[Equation] [Equation]
If H ALVE and T HIN instead have super-quadratic runtimes with max( r H ( n ) , r T ( n ))
= n τ , then by
Rem.
1 we have r C++ ( n ) ≤ ( 4 τ 2 τ − 4 + 1) 2 g τ n τ/ 2 , so that C OMPRESS ++ provides a near-quadratic speed up r C++ ( n ) =
O ( n τ/ 2 log cτ/ 2( n )) whenever 4 g = O (log c 4 n ) .
Remark 6 (C OMPRESS ++ inflates sub-Gaussian error by at most √ 2 )
In the usual case that n ν H ( n ) is non-decreasing in n , Thm. 3 and Rem.
2 imply that
[Equation]
where we have introduced the rescaled quantities ζ H ( ℓ n ) ≜ ℓ n 2 ν H ( ℓ n ) and ζ T ( ℓ n 2 ) ≜ √ n ν T ( ℓ n 2 ) .
Therefore, C OMPRESS ++ satisfies
[Equation] [Equation]
That is, whenever C OMPRESS ++ is run with an oversampling parameter g satisfying (11) its subGaussian error is never more than √ 2 times the second-stage T HIN error.
Here, T HIN represents a strong baseline for comparison as thinning from ℓ n / 2 to √ n points should incur at least as much error as thinning from n to √ n points.
As we illustrate in the next example, when T HIN and H ALVE are derived from the same thinning algorithm, the ratio ζ H ( ℓ n ) ζ T ( ℓ n / 2) is typically bounded by a constant C so that the choice g = ⌈ log 4 log 4 n + log 2 C ⌉ suffices to simultaneously obtain the √ 2 relative error guarantee (11) of
Rem. 6 and the substantial speed-ups (10) of Rem. 5.
Example 5 ( KT - SPLIT -C OMPRESS ++)
In the notation of Ex. 1, consider running C OMPRESS ++ with H ALVE =
KT - SPLIT ( ℓ 2 4 n 2 g ( g +2 g ( β n +1))
δ ) when applied to an input of size ℓ and T HIN =
KT - SPLIT ( g g +2 g ( β n +1) δ ) .
As detailed in App.
F.3, on an event of probability 1 − δ 2 , all C OMPRESS ++ invocations of H ALVE and T HIN are simultaneously f -sub-Gaussian with parameters satisfying
[Equation]
Since KT - SPLIT runs in Θ( n 2 ) time, Rems. 5 and 6 imply that KT - SPLIT -C OMPRESS ++ with g = ⌈ log 4 log 4 n ⌉ runs in near-linear O ( n log 2 n ) time and inflates sub-Gaussian error by at most √ 2 .
■ 4. MMD GUARANTEES FOR C OMPRESS ++
Next, we bound the MMD error of C OMPRESS ++ in terms of the MMD error of H ALVE and T HIN .
The proof of the following result can be found in App.
E.
Theorem 4 (MMD guarantees for C OMPRESS ++) If T HIN ∈ G k ( a ′ ,v ′ ) , H ALVE ∈
G k ( a,v ) for n a n and n v n non-decreasing, and E (cid:2) P H ALVE k | S in (cid:3) =
P in k , then C OMPRESS ++ ∈ G k ( (cid:98) a, (cid:98) v ) with (cid:98) v n ≜ (cid:101) v n + v ′ ℓ n /and (cid:98) a n ≜ (cid:101) a n + a ′ ℓ n / 2 + (cid:98) v n √ logfor (cid:101) v n and (cid:101) a n defined in Thm. 2 and ℓ n = 2 g +1 √ n as in Thm. 1.
Remark 7 (C OMPRESS ++ inflates MMD guarantee by at most 4 ) Thm. 4 implies that the
C OMPRESS ++ k -sub-Gaussian error ε k , C OMPRESS ++ ( n ) = max( (cid:98) a n , (cid:98) v n ) satisfies
[Equation]
where we have introduced the rescaled quantities (cid:101) ζ H ( ℓ n ) ≜ ℓ n 2 ε k , H ALVE ( ℓ n ) and (cid:101) ζ T ( ℓ n 2 ) ≜ √ n ε k , T HIN ( ℓ n 2 ) .
Therefore, C OMPRESS ++ satisfies
[Equation]
In other words, relative to a strong baseline of thinning from ℓ n 2 to √ n points, C OMPRESS ++ inflates k -sub-Gaussian error by at most a factor of 4 whenever g satisfies (13).
For example, when the ratio (cid:101) ζ H ( ℓ n ) / (cid:101) ζ T ( ℓ n 2 ) is bounded by C , it suffices to choose g = ⌈ log 2 log( n +1)+log 2 (8 C ) ⌉ .
Example 6 (KT-C OMPRESS ++)
In the notation of Ex. 2 and Rem. 3, consider running C OMPRESS ++ with H ALVE =
symmetrized KT ( ℓ 2 4 n 2 g ( g +2 g ( β n +1)) δ ) when applied to an input of size ℓ and T HIN = KT ( g g +2 g ( β n +1) δ ) .
As we detail in App.
F.4, on an event of probability 1 − δ 2 , all
C OMPRESS ++ invocations of H ALVE and T HIN are simultaneously k -sub-Gaussian with
[Equation]
Since KT runs in Θ( n 2 ) time, Rems. 5 and 7 imply that KT-C OMPRESS
++ with g = ⌈ log 2 log n +3 ⌉ runs in near-linear O ( n log 3 n ) time and inflates k -sub-Gaussian error by at most 4 .
■
E XPERIMENTS
We now turn to an empirical evaluation of the speed-ups and error of C OMPRESS ++.
We begin by describing the thinning algorithms, compression tasks, evaluation metrics, and kernels used in our experiments.
Supplementary experimental details and results can be found in App.
G.
Thinning algorithms
Each experiment compares a high-accuracy, quadratic time thinning algorithm—either target kernel thinning (Dwivedi & Mackey, 2022) or kernel herding (Chen et al., 2010)—with our near-linear time C OMPRESS and C OMPRESS ++ variants that use the same input algorithm to H ALVE and T HIN .
In each case, we perform root thinning, compressing n input points down to √ n points, so that C OMPRESS is run with g = 0 .
For C OMPRESS ++, we use g = 4 throughout to satisfy the small relative error criterion (11) in all experiments.
When halving we restrict each input algorithm to return distinct points and symmetrize the output as discussed in Rem. 3.
Compressing i.i.d.
summaries To demonstrate the advantages of C OMPRESS ++ over equal-sized i.i.d.
summaries we compress input point sequences S in drawn i.i.d.
from either (a) Gaussian targets P = N (0 , I d ) with d ∈ { 2 , 4 , 10 , 100 } or (b) M -component mixture of Gaussian targets P = 1 M (cid:80) Mj =
1 N ( µ j , I 2 ) with M ∈ { 4 , 6 , 8 , 32 } and component means µ j ∈ R 2 defined in App.
G.
Compressing MCMC summaries To demonstrate the advantages of C OMPRESS
++ over standard
MCMC thinning, we also compress input point sequences S in generated by a variety of popular
MCMC algorithms (denoted by RW, ADA-RW, MALA, and pMALA) targeting four challenging
Bayesian posterior distributions P .
In particular, we adopt the four posterior targets of Riabiz et al.
(2020a)
based on the Goodwin (1965) model of oscillatory enzymatic control ( d = 4 ), the Lotka (1925); Volterra (1926) model of oscillatory predator-prey evolution ( d = 4 ), the Hinch et al.
(2004) model of calcium signalling in cardiac cells ( d = 38 ), and a tempered Hinch model posterior ( d = 38 ).
Notably, for the Hinch experiments, each summary point discarded via an accurate thinning procedure saves 1000s of downstream CPU hours by avoiding an additional critically expensive whole-heart simulation (Riabiz et al., 2020a).
See App.
G for MCMC algorithm and target details.
Kernel settings Throughout we use a Gaussian kernel k ( x, y ) =
exp( − 12 σ 2 ∥ x − y ∥ 22 ) with σ 2 as specified by Dwivedi & Mackey (2021, Sec.
K.2) for the MCMC targets and σ 2 = 2 d otherwise.
Evaluation metrics For each thinning procedure we report mean runtime across 3 runs and mean
MMD error across 10 independent runs ± 1 standard error (the error bars are often too small to be visible).
All runtimes were measured on a single core of an Intel Xeon CPU.
For the i.i.d.
targets, we report MMD k
( P , P out ) which can be exactly computed in closed-form.
For the MCMC targets, we report the thinning error MMD k
( P in , P out ) analyzed directly by our theory (Thms. 2 and 4).
Kernel thinning results We first apply C OMPRESS
++ to the near-optimal KT algorithm to obtain comparable summaries at a fraction of the cost.
Figs.
1 and 2 reveal that, in line with our guarantees,
[Figure]
KT-C OMPRESS ++ matches or nearly matches the MMD error of KT in all experiments while also substantially reducing runtime.
For example, KT thins 65000 points in 10 dimensions in 20 m, while KT-C OMPRESS ++ needs only 1 . 5 m; KT takes more than a day to thin 250000 points indimensions, while KT-C OMPRESS ++ takes less than an hour (a 32 × speed-up).
For reference we also display the error of standard thinning (ST) to highlight that KT-C OMPRESS ++ significantly improves approximation quality relative to the standard practice of i.i.d.
summarization or standard
MCMC thinning.
See Fig.
4 in App.
G.1 for analogous results with mixture of Gaussian targets.
Kernel herding results
A strength of C OMPRESS ++ is that it can be applied to any thinning algorithm, including those with suboptimal or unknown performance guarantees that often perform well in practical.
In such cases, Rems. 4 and 6 still ensure that C OMPRESS ++ error is never much larger than that of the input algorithm.
As an illustration, we apply C OMPRESS ++ to the popular quadratic-time kernel herding algorithm (Herd).
Fig.
1 shows that Herd-C OMPRESS ++ matches or nearly matches the MMD error of Herd in all experiments while also substantially reducing runtime.
For example, Herd requires more than 11 hours to compress 250000 points in 100 dimensions, while Herd-C OMPRESS
++ takes only 14 minutes (a 45 × speed-up).
Moreover, surprisingly, HerdC OMPRESS ++ is consistently more accurate than the original kernel herding algorithm for lower dimensional problems.
See Fig.
4 in App.
G.1 for comparable results with mixture of Gaussian P .
Visualizing coresets For a 32-component mixture of Gaussians target, Fig.
3 visualizes the coresets produced by i.i.d.
sampling, KT, kernel herding, and their C OMPRESS ++ variants.
The C OMPRESS ++ coresets closely resemble those of their input algorithms and, compared with i.i.d.
sampling, yield visibly improved stratification across the mixture components.
[Table]
[Figure] [Figure]
D ISCUSSION AND C ONCLUSIONS
We introduced a new general meta-procedure, C OMPRESS ++, for speeding up thinning algorithms while preserving their error guarantees up to a factor of 4 .
When combined with the quadratic-time
KT - SPLIT and kernel thinning algorithms of Dwivedi & Mackey (2021; 2022), the result is nearoptimal distribution compression in near-linear time.
Moreover, the same simple approach can be combined with any slow thinning algorithm to obtain comparable summaries in a fraction of the time.
Two open questions recommend themselves for future investigation.
First, why does HerdC OMPRESS ++ improve upon the original kernel herding algorithm in lower dimensions, and can this improvement be extended to higher dimensions and to other algorithms?
Second, is it possible to thin significantly faster than C OMPRESS ++ without significantly sacrificing approximation error?
Lower bounds tracing out the computational-statistical trade-offs in distribution compression would provide a precise benchmark for optimality and point to any remaining opportunities for improvement.
R EPRODUCIBILITY S TATEMENT
See the goodpoints Python package for Python implementations of all methods in this paper and https://github.com/microsoft/goodpoints for code reproducing each experiment.
A CKNOWLEDGMENTS
RD acknowledges the support by the National Science Foundation under Grant No. DMSfor the Foundations of Data Science Institute (FODSI).
Part of this work was done when AS was interning at Microsoft Research New England
A PPENDIX
A Additional Definitions and Notation
B Proof of Thm.
1: Runtime and integration error of C OMPRESS
C Proof of Thm. 2: MMD guarantees for C OMPRESS
D Proof of Thm.
3: Runtime and integration error of C OMPRESS ++
E Proof of Thm. 4: MMD guarantees for C OMPRESS
++
F Proofs of Exs. 3, 4, 5, and
G Supplementary Details for Experiments
H Streaming Version of C OMPRESS
A A DDITIONAL D EFINITIONS AND N OTATION
This section provides additional definitions and notation used throughout the appendices.
We associate with each algorithm A LG and input S in the measure difference
[Equation]
that characterizes how well the output empirical distribution approximates the input.
We will often write ϕ
A LG instead of ϕ A LG ( S in ) for brevity if S in is clear from the context.
We also make use of the following standard definition of a sub-Gaussian random variable (see, e.g.,
Boucheron et al., 2013, Sec. 2.3).
Definition 4 (Sub-Gaussian random variable)
We say that a random variable G is sub-Gaussian with parameter ν and write G ∈ G ( ν ) if
[Equation]
for all λ
[Equation]
R .
Given Def. 4, it follows that A LG ∈ G f ( ν ) for a function f as in Def.
if and only if the random variable ϕ
A LG ( f ) ≜ P S in f − P S A LG f is sub-Gaussian with parameter ν conditional on the input S in .
In our proofs, it is often more convenient to work with an unnormalized measure discrepancy
[Equation]
By definition (15), we have the following useful equivalence:
[Equation]
The following standard lemma establishes that the sub-Gaussian property is closed under scaling and summation.
Lemma 1 (Summation and scaling preserve sub-Gaussianity)
Suppose G 1 ∈ G ( σ 1 ) .
Then, for all β ∈ R , we have β · G 1 ∈ G ( βσ 1 ) .
Furthermore, if G 1 is F -measurable and G 2 ∈ G ( σ 2 ) given
F , then G 1 + G 2 ∈ G ( (cid:112) σ 21 + σ 22 ) .
Proof Fix any β ∈ R .
Since G 1 ∈ G ( σ 1 ) , for each λ ∈ R ,
[Equation]
(cid:16)
[Equation]
(cid:17) , so that βG 1 ∈ G ( βσ 1 ) as advertised.
Furthermore, if G 1 is F -measurable and G 2 ∈ G ( σ 2 ) given F , then, for each λ ∈ R ,
E (cid:104)
[Equation]
λ ·
[Equation]
G
[Equation]
G
[Equation]
, so that G 1 + G 2 ∈ G ( (cid:112) σ 21 + σ 22 ) as claimed.
□
B P ROOF OF T HM . 1: R UNTIME AND INTEGRATION ERROR OF C OMPRESS
First, we bound the running time of C OMPRESS .
By definition, C OMPRESS makes four recursive calls to C OMPRESS on inputs of size n/ 4 .
Then, H ALVE is run on an input of size 2 g +1 √ n .
Thus, r C satisfies the recursion
[Equation]
Since r C (4 g ) = 0 , we may unroll the recursion to find that
[Equation]
as claimed in (5).
Next, we bound the sub-Gaussian error for a fixed function f .
In the measure discrepancy (15) notation of App.
A we have
[Equation]
where S i and (cid:101) S are defined as in Alg. 1.
Unrolling this recursion, we find that running C OMPRESS on an input of size n with oversampling parameter g leads to applying H ALVE on 4 i coresets of size n i =
2 g
+
−
i √ n for 0 ≤ i ≤ β n .
Denoting these H ALVE inputs by ( S in i,j ) j ∈ [4 i ] , we have
[Equation]
Now define σ H ( n ) =
nν H ( n ) .
Since ψ H ( S in i,j )( f ) are σ H ( n i ) sub-Gaussian given ( S in i ′ ,j ′ )
i ′ >
i,j ′ ≥and
( S in i,j ′ )
j ′ ≤
j , Lem.
1 implies that ψ C ( S in )( f ) is σ C sub-Gaussian given S in for σ 2 C ( n ) = n 4 − g − 1 (cid:80) β n i =
σ 2 H ( n i ) .
Recalling the relation (16) between σ and ν from App.
A, we conclude that
[Equation]
as claimed in (6).
C P ROOF OF T HM . 2: MMD GUARANTEES FOR C OMPRESS
Our proof proceeds in several steps.
To control the MMD (1), we need to control the Hilbert norm of the measure discrepancy of C OMPRESS (15), which we first write as a weighted sum of measure discrepancies from different (conditionally independent) runs of H ALVE .
To effectively leverage the MMD tail bound assumption for this weighted sum, we reduce the problem to establishing a concentration inequality for the operator norm of an associated matrix.
We carry out this plan in four steps summarized below.
First, we express the MMD associated with each H ALVE measure discrepancy as the Euclidean norm of a suitable vector (Lem. 2).
Second, we define a matrix dilation operator for a vector that allows us to control vector norms using matrix spectral norms (Lem. 3).
Third, we establish moment bounds for these matrices by leveraging tail bounds for the MMD error (Lem. 5).
Finally, we prove and apply a subexponential matrix Freedman concentration inequality (Lem. 4) to control the MMD error for the C OMPRESS output.
We now begin our formal argument.
We will make use of the unrolled representation (17) for the C OMPRESS measure discrepancy ψ C ( S in ) in terms of the H ALVE inputs ( S in k,j ) j ∈
[4 k ] of size n k = 2 g +1 − k √ n for 0 ≤ k ≤ log 4 n − g − 1 .
For brevity, we will use the shorthand ψ C ≜ ψ C ( S in ) , ψ H k,j ≜ ψ H ( S in k,j ) , and ψ T ≜ ψ T ( S C ) hereafter.
C. R EDUCING MMD TO VECTOR E UCLIDEAN NORM
Number the elements of S in as ( x 1 , . . .
, x n ) , define the n × n kernel matrix K ≜ ( k ( x i , x j )) ni,j =1 , and let K 1 2 denote a matrix square-root such that K = K 12 · K 12 (which exists since K is a positive semidefinite matrix for any kernel k ).
Next, let S out k,j denote the output sequence corresponding to ψ H k,j (i.e., running H ALVE on S in k,j ), and let { e i } ni =1 denote the canonical basis of R n .
The next lemma (with proof in App. C.5) relates the Hilbert norms to Euclidean norms of carefully constructed vectors.
Lemma 2 (MMD as a vector norm) Define the vectors u k,j ≜ K 12 (cid:80) ni
=1 e i (cid:16) 1 ( x i ∈S in k,j ) − 2 · 1 ( x i ∈S out k,j ) (cid:17) , and u C ≜ (cid:80) log 4 n − g − 1 k
= (cid:80) 4 k j =1 w k,j u k,j , (19) where w k,j ≜ √ n 2 g +
+ k .
Then, we have n 2 · MMD 2 k ( S in , S C ) =
∥ u C ∥ 22 , and (20)
E [ u k,j |{ u k ′ ,j ′ , j ′ ∈ [4 k ′ ] , k ′ > k } ] = 0 for k = 0 , . .
. , log 4 n − g − 2 , (21) and u k,j for j ∈ [4 k ] are conditionally independent given { u k ′ ,j ′ , j ′ ∈ [4 k ′ ] , k ′ > k } .
Applying (20), we effectively reduce the task of controlling the MMD errors to controlling the
Euclidean norm of suitably defined vectors.
Next, we reduce the problem to controlling the spectral norm of a suitable matrix.
C. R EDUCING VECTOR E UCLIDEAN NORM TO MATRIX SPECTRAL NORM
To this end, we define a symmetric dilation matrix operator: given a vector u ∈ R n , define the matrix M u as
[Equation]
It is straightforward to see that u (cid:55)→ M u is a linear map.
In addition, the matrix M u also satisfies a few important properties (established in App.
C.6) that we use in our proofs.
Lemma 3 (Properties of the dilation operator)
For any u ∈ R n , the matrix M u (22) satisfies
[Equation]
[Equation]
Define the shorthand M k,j ≜ M w k,j u k,j (defined in Lem. 2).
Applying Lems.
2 and 3, we find that
[Equation]
where step (i) follows from the linearity of the dilation operator.
Thus to control the MMD error, it suffices to control the maximum eigenvalue of the sum of matrices appearing in (24).
C. C ONTROLLING MATRIX SPECTRAL NORM VIA A MATRIX F REEDMAN INEQUALITY
To control the maximum eigenvalue of the matrix M u C , we make use of (24) and the following subexponential generalization of the matrix Freedman inequality of Tropp (2011, Thm. 1.2).
The proof of Lem.
4 can be found in App.
C.7.
For two matrices A and B of the same size, we write
A ⪯ B if B − A is positive semidefinite.
Lemma 4 (Subexponential matrix Freedman inequality)
If a finite sequence { Y i } Ni =1 of symmetric random matrices in R m × m satisfies
[Equation]
[Equation] [Equation]
p
[Equation] [Equation]
[Equation] [Equation]
for some R > 0 , and a deterministic sequence { A i } Ni =1 of symmetric matrices in R m × m , then
[Equation]
all t >and equivalently
[Equation]
To apply Lem.
4 with the matrices (cid:8) M k,j (cid:9) , we need to establish the zero-mean and moment bound conditions for suitable R and A k,j in (25).
C.3. V ERIFYING THE ZERO MEAN CONDITION
To this end, first we note that the conditional independence and zero-mean property of (cid:110) ψ H k,j (cid:111) implies that the random vectors u k,j and the matrices M k,j also satisfy a similar property, and in particular that
[Equation]
E STABLISHING MATRIX MOMENT BOUNDS VIA MMD TAIL BOUNDS FOR H ALVE
To establish the moment bounds on M k,j , note that Lems.
2 and 3 imply that
[Equation] [Equation]
where w k,j was defined in Lem.
Thus it suffices to establish the moment bounds on ∥ u k,j ∥ q k .
To this end, we first state a lemma that converts tail bounds to moment bounds (see App.
C.8 for a proof inspired by Boucheron et al.
(2013, Thm. 2.3)).
Lemma 5 (Tail bounds imply moment bounds)
For a non-negative random variable Z ,
P [ Z >a +
[Equation]
t + ct ] ≤
[Equation] [Equation]
⇒
[Equation]
[ Z
[Equation] [Equation]
(2 a
[Equation]
+
[Equation]
q , ∀ q ∈ N with q ≥ 2 .
To obtain a moment bound for (cid:13)(cid:13) u k,j (cid:13)(cid:13) 2 , we first state some notation.
For each n , define the quantities
[Equation]
(28) where a n and v n are the parameters
such that H ALVE ∈
G k ( a n , v n ) on inputs of size n .
Using an argument similar to Lem.
2, we have
[Equation]
Thereby
[Equation]
using the G k assumption on H ALVE implies that
[Equation]
conditioned on { u k ′ ,j ′ , j ′ ∈ [4 k ′ ] , k ′ > k } .
Combining the bound (29) with Lem.
5 yields that
[Equation]
for all q ∈ N , where ℓ k is defined in (29).
Now, putting together (27) and (30), and using the conditional independence of M k,j , we obtain the following control on the q -th moments of M k,j for q ≥ 2 :
[Equation]
where ℓ k is defined in (29).
C.
P UTTING THE PIECES TOGETHER
Putting (26) and (31) together, we conclude that with a suitable ordering of the indices ( k, j ) , the assumptions of Lem.
4 are satisfied by the random matrices (cid:8)
M k,j , j ∈ [4 k ] , k ∈ { 0 , 1 , . .
. , log 4 n − g − 1 } (cid:9) , the deterministic sequence (cid:8)
A k,j (cid:9) , and any scalar R satisfying R ≥ max k,j R k,j .
Now, since ℓ k = √ n 2 g +1 − k is decreasing in k , w k,j = ℓ k 4 g +1 , and na ′ n and nv ′ n , (28) are assumed non-decreasing in n , we find that R = n · R M , C is a valid choice as
[Equation]
Moreover, since a ′ n and v ′ n , are assumed non-decreasing in n , ℓ k is decreasing in k , and w k,j = √ n 2 g +
+ k , our choice σ 2M , C satisfies n
[Equation]
Finally, applying (24) along with Lem.
[Equation]
we conclude that
[Equation]
≤ e − t for all t > 0 .
C. P ROOF OF L EM . 2: MMD AS A VECTOR NORM
Let v k,j ≜ (cid:80) ni
=1 e i (cid:16) 1 ( x i ∈ S in k,j ) − 2 · 1 ( x i ∈
S out k,j ) (cid:17) .
By the reproducing property of k we have ∥
[Equation] [Equation]
Using (18), (19)
[Equation]
and (22)
[Equation]
and mimicking the derivation above (32), we can also conclude that
[Equation]
Additionally, we note that
[Equation]
Finally the conditional independence and zero mean property (21) follows from (18) by noting that conditioned on ( S in k ′ ,j ′ )
k ′ >k,j ′ ≥ 1 , the sets ( S in k,j ) j ≥ 1 are independent.
C. P ROOF OF L EM . 3: P ROPERTIES OF THE DILATION OPERATOR
For claim (a) in the display (23), we have
[Equation]
where step (i) follows from the standard fact that uu ⊤ ⪯ ∥
u ∥ 22 I n and step (ii) from the facts M 2 u (cid:101) e 1 = ∥
u ∥ 22 (cid:101)
e 1 for (cid:101)
e 1 the first canonical basis vector of R n +1 and ∥ M u ∥ 2op =
(cid:13)(cid:13)
M 2 u (cid:13)(cid:13) op .
Claim (b) follows directly by verifying
that the vector v =
[1 , u ⊤ ∥ u ∥ 2 ] ⊤ is an eigenvector of M u with eigenvalue
∥ u ∥ 2 .
Finally, claim (c) follows directly from the claim (a) and the fact that ∥ M qu ∥ op = ∥ M u ∥ q op for all integers q ≥ 1 .
C. P ROOF OF L EM . 4: S UBEXPONENTIAL MATRIX F REEDMAN INEQUALITY
We first note the following lemmas about the moment generating functions (MGFs) for matrix valued random variables.
Lemma 6 (Matrix tail bounds (Tropp, 2011, Thm 2.3))
Let X k be an adaptive sequence of matrix valued random variables in R m × m (with respect to the filtration F k and let V k be a sequence that is measureable with respect to F k − 1 .
Suppose that, for some θ > 0 ,
[Equation]
Let Y k
= (cid:80) i ≤ k X i and W k = (cid:80) i
≤ k V i .
Then,
P
[Equation]
0 : λ max ( Y k ) ≥ t and λ max ( W k )
[Equation]
(cid:3)
[Equation]
m
[Equation]
Lemma 7 (Supexponential matrix MGF) Let X be a random symmetric matrix such that
E X
=
and E X q ⪯ q !
2 R q − 2 Afor q = 2 , 3 , . . . .
Then,
E exp( θ X ) ⪯ exp (cid:16) θ 2 2(1 − Rθ ) ·
A 2 (cid:17) for all θ ∈ (0 , 1 /R ) .
Proof When R = 1 , the result coincides with Tropp (2012, Lem. 6.8).
To obtain the general result, we apply Tropp (2012, Lem. 6.8) to X ′ =
X /R and A ′ =
A /R .
□
We apply Lem.
7 conditional on { X i } i<k along with the operator monotonicity of log to get log
[Equation]
for all θ ∈
[Equation]
By assumption, we have that λ max ( (cid:80) i
A 2 i ) ≤ σ 2 .
Thus, applying Lem.
6, we get, for all t > 0 ,
[Equation]
where, by Boucheron et al. (2013, Sec. 2.4),
[Equation]
Now, since the inverse of h 1 takes the form h − 1 1 ( u ) =
u + √ 2 u for u > 0 by Boucheron et al.
(2013, Sec. 2.4), we additionally have
[Equation]
as advertised.
C. P ROOF OF L EM . 5: T AIL BOUNDS IMPLY MOMENT BOUNDS
Let ¯ a = a + v/ 2 and ¯ c = c
+ v/ 2 .
For all t > 0 , by the arithmetic-geometric mean inequality,
[Equation]
We begin by bounding the moments of the shifted random variable X =
Z − ¯ a .
First note that
X = X
+ − X − where X ± = max( ± X, 0) and that | X | q =
X q + + X q − .
Furthermore, X q − ≤ ¯ a q by the nonnegativity of Z , so that | X | q ≤ ¯ a q + X q + .
Since P [ X + > u ] = P [ X > u ]
= P [ Z > ¯ a + u ] for any u > 0 we will use the our tail bounds to control the moments of X + .
In particular, we have
[Equation]
where we have applied ( i ) integration by parts, ( ii ) the substitution u
= ¯ ct , ( iii ) the assumed tail bound for Z , and ( iv ) the definition of the Gamma function.
Now since Z = X +¯ a , we have E Z q ≤ 2 q − 1 (¯ a q + E | X | q ) ≤ 2 q − 1 (2¯ a q + E X q + ) from the convexity of t q .
For each q ≥ 2 , our combined results now yield
[Equation]
since x q + y q ≤ ( x + y ) q for all q ∈ N and x, y ≥ 0 .
D P ROOF OF T HM . 3: R UNTIME AND INTEGRATION ERROR OF C OMPRESS ++
First, the runtime bound (9) follows directly by adding the runtime of C OMPRESS ( H ALVE , g ) as given by (5) in Thm.
1 and the runtime of T HIN .
Recalling the notation (14) and (15) from App.
A and noting the definition of the point sequences S C and S C++ in Alg.
2, we obtain the following relationship between the different discrepancy vectors:
[Equation]
and
[Equation]
Noting the G f property of H ALVE and applying Thm.
1, we find that ϕ C ( S in )( f ) is sub-Gaussian with parameter ν C ( n ) defined in (6).
Furthermore, by assumption on T HIN , given S C , the variable ϕ T ( S C )( f ) is ν C ( ℓ n 2 ) sub-Gaussian.
The claim now follows directly from Lem.
E P ROOF OF T HM . 4: MMD GUARANTEES FOR C OMPRESS ++
Noting that MMD is a metric, and applying triangle inequality, we have
[Equation]
Since S C++ is the output of T HIN (2 g ) with S C as the input, applying the MMD tail bound assumption (33) with |S C | = √ n 2 g substituted in place of n , we find that
[Equation]
for all t
[Equation] [Equation]
Recall that ℓ n / 2 = 2 g √ n .
Next, we apply Thm.
2 with H ALVE
to conclude that
[Equation]
for all t ≥ 0 .
Thus, we have
[Equation]
which in turn implies that
[Equation]
thereby yielding the claimed result.
F P ROOFS OF E XS . 3, 4, 5, AND
We begin by defining the notions of sub-Gaussianity and k -sub-Gaussianity on an event.
Definition 5 (Sub-Gaussian on an event)
We say that a random variable G is sub-Gaussian on an event E with parameter σ if
[Equation]
Definition 6 ( k -sub-Gaussian on an event)
For a kernel k , we call a thinning algorithm
A LG ksub-Gaussian on an event E with parameter v and shift a if
[Equation]
[Equation]
We will also make regular use of the unrolled representation (17) for the C OMPRESS measure discrepancy ψ C ( S in ) in terms of the H ALVE inputs ( S in k,j ) j ∈
[4 k ] of size
[Equation]
F. P ROOF OF E X . 3: KT - SPLIT -C OMPRESS
For H ALVE = KT - SPLIT ( ℓ 2 n 4 g +1 ( β n +1) δ ) when applied to an input of size ℓ , the proof of Thm. 1 in
Dwivedi & Mackey (2022) identifies a sequence of events E k,j and random signed measures ˜ ψ k,j such that, for each 0 ≤ k ≤ β n , j ∈ [4 k ] , and f with ∥ f ∥ k = 1 ,
(a) P [ E ck,j ] ( i ) ≤ n 2 k n 4 g
[Equation]
1 ( β n +1)
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
n +1) , (b)
[ E k,j ] ψ H k,j = 1 [ E k,j ] ˜ ψ k,j , and (c) ˜ ψ k,j ( f ) is n k ν H ( n k ) sub-Gaussian (7) given ( ˜ ψ k ′ ,j ′ )
k ′ >k,j ′ ≥ 1 and (
˜
ψ k,j ′ )
j ′ <j ,
where step (ii) follows from substituting the definition n k
= 2 g +1 − k √ n (34).
To establish step (ii) in property (a), we use the definition 1 of KT - SPLIT ( n 2 k n 4 g +1 ( β n +1) δ ) for an input of size n k , which implies that δ i = n k n 4 g +1 ( β n +1) δ in the notation of Dwivedi & Mackey (2022).
The proof of Thm.in Dwivedi & Mackey (2022) then implies that
[Equation]
Hence, on the event E = (cid:84) k,j E k,j , these properties hold simultaneously for all H ALVE calls made by C OMPRESS , and, by the union bound,
[Equation]
Now fix any f with ∥ f ∥ k = 1 .
We invoke the measure discrepancy representation (17), the equivalence of ψ H k,j and ˜ ψ k,j on E , the nonnegativity of the exponential, and Lem.
1 in turn to find
E
[Equation]
) for
[Equation]
n
[Equation]
= (cid:80)
[Equation]
( n
[Equation]
) so that ϕ C ( f ) is ν C sub-Gaussian on E .
F. P ROOF OF E X . 4: KT-C OMPRESS
For H ALVE = symmetrized KT ( ℓ 2 n 4 g +1 ( β n +1) δ ) when applied to an input of size ℓ , the proofs of
Thms.
1–4 in Dwivedi & Mackey (2022) identify a sequence of events E k,j and random signed measures ˜ ψ k,j such that, for each 0 ≤ k ≤ β n and j ∈ [4 k ] ,
(a) P [ E ck,j ] ≤δ 4 k ( β n +1) , (b) 1 [ E k,j ] ψ H k,j =
[ E k,j ] ˜ ψ k,j , (c) P [ 1 n k ∥ ˜ ψ k,j ( k ) ∥
k ≥ a n k + v n k √ t | ( ˜ ψ k ′ ,j ′ )
k ′ >k,j ′ ≥ 1 , ( ˜ ψ k,j ′ )
j ′ <
j ] ≤ e − t for all t ≥ 0 , and (d) E [ ˜ ψ k,j ( k ) | ( ˜ ψ k ′ ,j ′ )
k ′ >k,j ′ ≥ 1 , ( ˜ ψ k,j ′ )
j ′ <
j ] = 0 ,
where n k = 2 g +1 − k √ n was defined in (34).
We derive property (a) exactly as in App. F.1.
Hence, on the event E = (cid:84) k,j E k,j , these properties hold simultaneously for all H ALVE calls made by C OMPRESS , and, by the union bound (35), P [ E c ] ≤ δ 2 .
Furthermore, we may invoke the measure discrepancy representation (17), the equivalence of ψ H k,j and ˜ ψ k,j on E , the nonnegativity of the exponential, and the proof of Thm. 2 in turn to find
[Equation]
for all t ≥ 0 , so that C OMPRESS is k -sub-Gaussian on E with parameters (˜ v, ˜ a ) .
F. P ROOF OF E X . 5: KT - SPLIT -C OMPRESS ++ For T HIN = KT - SPLIT ( g g +2 g
n +
) δ
and H ALVE = KT - SPLIT (
[Equation]
g
[Equation]
g
[Equation]
n
+
[Equation]
) δ
) when applied to an input of size ℓ , the proof of Thm. 1 in Dwivedi & Mackey (2022) identifies a sequence of events
E k,j and E T and random signed measures ˜ ψ k,j and ˜ ψ T such that, for each 0 ≤ k ≤ β n , j ∈ [4 k ] , and f with ∥ f ∥ k = 1 ,
[Equation]
and
, (b) 1 [ E k,j ] ψ H k,j = 1 [ E k,j ] ˜ ψ k,j and 1 [ E T ] ψ T =
[ E T ] ˜ ψ T , and (c) ˜ ψ k,j ( f ) is n k ν H ( n k ) sub-Gaussian (12) given ( ˜ ψ k ′ ,j ′ )
k ′ >k,j ′ ≥ 1 and ( ˜ ψ k,j ′ )
j ′ <
j and ˜ ψ T is ℓ n 2 ν T ( ℓ n 2 ) sub-Gaussian (12) given S C .
Here, step (i) and (ii) follow exactly as in steps (i) and (ii) of property (a) in App. F.1.
For step (iii), we use the definition 1 of KT - SPLIT ( g g +2 g ( β n +1) δ ) for an input of size 2 g √ n , which implies that δ i =
g √ n 2 g ( g +2 g ( β n +1))
δ in the notation of Dwivedi & Mackey (2022).
The proof of Thm. 1 in
Dwivedi & Mackey (2022) then implies that
[Equation]
as claimed.
Hence, on the event E = (cid:84) k,j E k,j ∩ E T , these properties hold simultaneously for all H ALVE calls made by C OMPRESS , and, repeating an argument similar to the union bound (35),
P [ E
[Equation]
] ≤ P [
[Equation]
] +
[Equation]
P [ E ck,j ] ≤
[Equation] [Equation]
Moreover, since ϕ C++ = 1 n ( ψ C + ψ T ) , Lem. 1 and the argument of App. F.1 together imply that ϕ C ( f ) is ν C++ sub-Gaussian on E for each f with ∥ f ∥ k = 1 .
F. P ROOF OF E X . 6: KT-C OMPRESS ++
In the notation of Ex. 2, define
[Equation]
and
[Equation]
Since H ALVE =
symmetrized KT ( ℓ 2 4 n 2 g ( g +2 g ( β n +1)) δ ) for inputs of size ℓ and T HIN =
KT ( g g +2 g ( β n +1) δ ) , the proofs of Thms.
1–4 in Dwivedi & Mackey (2022) identify a sequence of events E k,j and E T and random signed measures ˜ ψ k,j and ˜ ψ T such that, for each 0 ≤ k ≤ β n and j ∈ [4 k ] ,
(a) P
[Equation] [Equation]
and P [ E c T ] ≤
[Equation]
(b)
[ E k,j ] ψ H k,j = 1 [ E k,j ] ˜ ψ k,j and 1 [ E T ] ψ T = 1 [ E T ] ˜ ψ T
[Equation] [Equation]
e
[Equation]
and
[Equation] [Equation]
e − t
for
all
t ≥
0 , and
(d) E [ ˜ ψ k,
[Equation]
( k
[Equation]
( ˜ ψ
[Equation]
k ′
>
k
[Equation]
j
[Equation]
≥ 1 , ( ˜ ψ k,j
[Equation]
j
<
j
] = 0 .
We derive property (a) exactly as in App.
F.3.
Hence, on the event E = (cid:84) k,j E k,j ∩ E T , these properties hold simultaneously for all H ALVE calls made by C OMPRESS and
[Equation]
Moreover, by the union bound (36), P [ E c ] ≤ δ 2 .
Finally, since ϕ C++ = 1 n ( ψ C + ψ T ) and the argument of App.
F.2 implies that C OMPRESS is ksub-Gaussian on E with parameters (˜ v, ˜ a ) , the triangle inequality implies that C OMPRESS
+
+ is k -sub-Gaussian on E with parameters (ˆ v, ˆ a ) as in App.
E.
G S UPPLEMENTARY D ETAILS FOR E XPERIMENTS
In this section, we provide supplementary experiment details deferred from Sec. 5, as well as some additional results.
In the legend of each MMD plot, we display an empirical rate of decay.
In all experiments involving kernel thinning, we set the algorithm failure probability parameter δ = 12 and compare KT ( δ ) to
C OMPRESS and C OMPRESS
++ with H ALVE and T HIN set as in Exs. 4 and 6 respectively.
G. M IXTURE OF G AUSSIAN TARGET DETAILS AND MMD PLOTS
For the target used for coreset visualization in Fig.
3, the mean locations are on two concentric circles of radii 10 and 20 , and are given by µ
[Equation]
(cid:20)
[Equation]
cos( j ) (cid:21) where α j = 10 · 1 ( j ≤ 16) + 20 · 1 ( j > 16) for j = 1 , 2 , . . . , 32 .
Here we also provide additional results with mixture of Gaussian targets given by P =
M (cid:80) Mj
=1 N ( µ j , I d ) for M ∈ { 4 , 6 , 8 } .
The mean locations for these are given by
[Equation]
Fig.
4 plots the MMD errors of KT and herding experiments for the mixture of Gaussians targets with 4 , 6 and 8 centers, and notice again that C OMPRESS ++ provides a competitive performance to the original algorithm, in fact suprisingly, improves upon herding.
G. D ETAILS O F MCMC T ARGETS
Our set-up for the MCMC experiments is identical to that of Dwivedi & Mackey (2021, Sec. 6), except that we use all post-burn-in points to generate our Goodwin and Lotka-Volterra input point sequences S in instead of only the odd indices.
In particular, we use the MCMC output of Riabiz et al.
(2020b) described in (Riabiz et al., 2020a, Sec. 4) and perform thinning experiments after discarding the burn-in points.
To generate an input S in of size n for a thinning algorithm, we downsample the post-burn-in points using standard thinning.
For Hinch, we additionally do coordinate-wise normalization by subtracting the sample mean and dividing by sample standard deviation of the post-burn-in-points.
In Sec. 5, RW and ADA-RW respectively refer to Gaussian random walk and adaptive Gaussian random walk Metropolis algorithms (Haario et al., 1999) and MALA and pMALA respectively refer to the Metropolis-adjusted Langevin algorithm (Roberts & Tweedie, 1996) and pre-conditioned
MALA (Girolami & Calderhead, 2011).
For Hinch experiments, RW 1 and RW 2 refer to two independent runs of Gaussian random walk, and “Tempered” denotes the runs targeting a tempered
Hinch posterior.
For more details on the set-up, we refer the reader to Dwivedi & Mackey (2021,
Sec.
6.3, App.
J.2).
[Figure]
H S TREAMING V ERSION OF C OMPRESS
C OMPRESS can be efficiently implemented in a streaming fashion (Alg. 3) by viewing the recursive steps in Alg.
1 as different levels of processing, with the bottom level denoting the input points and the top level denoting the output points.
The streaming variant of the algorithm efficiently maintains memory at several levels and processes inputs in batches of size 4 g +1 .
At any level i (with i =denoting the level of the input points), whenever there are 2 i 4 g +1 points, the algorithm runs H ALVE on the points in this level, appends the output of size 2 i − 1 4 g +1 to the points at level i +1 , and empties the memory at level i (and thereby level i never stores more than 2 i 4 g +1 points).
In this fashion, just after processing n = 4 k
+ g +1 points, the highest level is k + 1 , which contains a compressed coreset of size 2 k − 1
g +1 =
2 k
+ g +1 2 g = √ n 2 g (outputted by running H ALVE at level k for the first time), which is the desired size for the output of C OMPRESS .
Our next result analyzes the space complexity of the streaming variant (Alg. 3) of C OMPRESS .
The intuition for gains in memory requirements is very similar to that for running time, as we now maintain (and run H ALVE ) on subsets of points with size much smaller than the input sequence.
We count the number of data points stored as our measure of memory.
Proposition 1 (C OMPRESS Streaming Memory Bound) Let H ALVE store s H ( n ) data points on inputs of size n .
Then, after completing iteration t , the streaming implementation of C OMPRESS (Alg. 3) has used at most s C ( t )
= 4 g +3 √ t + s H
(2 g +1 √ t ) data points of memory.
Proof At time t , we would like to estimate the space usage of the algorithm.
At the i th level of memory, we can have at most 2 i +2 4 g data points.
Since we are maintaining a data set of size at most √ t 4 g at time t , there are at most log t 2 levels.
Thus, the maximum number of points stored at time t is bounded by
[Equation]
Furthermore, at any time up to time t , we have run H ALVE on a point sequence of size at most √ t 2 g +1 which requires storing at most s H ( √ t 2 g +1 ) additional points.
□
Input: halving
H ALVE oversampling parameter g stream input points x x
S 0 ← {} // Initialize empty level 0 coreset t = 1 ,
, do
S 0 ← S 0 ∪ ( x j ) t · 4 g +1 j =1+( t − 1) · 4 g + // Process input in batches of size 4 g + if t then
S j +1 ← {} // Initialize level j + 1 coreset after processing 4 j + g +1 input points endfor i = 0 , . .
. , ⌈ log 4 t ⌉ + 1 do if | then
S ← H ALVE ( S i ) //
Halve level i coreset to size 2 i − 1 4 g +
S i +1 ← S
i +1 ∪ S // Update level i + 1 coreset: has size ∈ { 1 , 2 , 3 , 4 } · 2 i − 1 4 g +
S i ← {} // level i end end if then output S j
2 g
≜ t 4 g
4 j j ∈ N end end
Example 7 (KT-C OMPRESS and KT-C OMPRESS ++) First consider the streaming variant of
C OMPRESS with H ALVE = symmetrized KT ( ℓ 2 n − ℓ n δ ) for H ALVE inputs of size ℓ as in Ex. 4.
Since s KT ( n ) ≤ n (Dwivedi & Mackey, 2021, Sec. 3), Prop. 1 implies that s C ( n ) ≤ 4 g +4 √ n .
Next consider C OMPRESS ++ with the streaming variant of C OMPRESS , with H ALVE = symmetrized KT ( ℓ 2 n − ℓ n +2 g √ n δ ) when applied to an input of size ℓ , and T HIN = KT ( g √ n − 2 g + g δ ) as in Ex. 6.
The space complexity s C++ ( n ) =
s C ( n )+ s KT ( ℓ n )=
4 g +4 √ n + ℓ n ≤ 4 g +5 √ n .
Setting g as in Ex. 6, we get s C++ ( n ) =
O ( √ n log 2 n ) .
■