Fast Bayesian Coresets via Subsampling and Quasi-Newton Refinement
Anonymous Author(s)
AffiliationAddress email
Abstract
Bayesian coresets approximate a posterior distribution by building a small weightedsubset of the data points.
Any inference procedure that is too computationallyexpensive to be run on the full posterior can instead be run inexpensively on thecoreset, with results that approximate those on the full data.
However, currentapproaches are limited by either a significant run-time or the need for the user tospecify a low-cost approximation to the full posterior.
We propose a Bayesiancoreset construction algorithm that first selects a uniformly random subset ofdata, and then optimizes the weights using a novel quasi-Newton method.
Ouralgorithm is a simple to implement, black-box method, that does not require theuser to specify a low-cost posterior approximation.
It is the first to come with ageneral high-probability bound on the KL divergence of the output coreset posterior.
Experiments demonstrate that our method provides significant improvements incoreset quality against alternatives with comparable construction times, with farless storage cost and user input required.
Introduction
Bayesian methods are key tools for parameter estimation and uncertainty quantification, but exactinference is rarely possible for complex models.
Currently, the gold standard method for approximateinference is Markov chain Monte Carlo (MCMC) [1; 2; 3, Ch.
11,12], which involves simulating a
Markov chain whose stationary distribution is the Bayesian posterior.
However, modern applicationsare often concerned with very large datasets; in this setting, MCMC methods typically have a Θ( NT )complexity—for T samples and dataset size N —which quickly becomes intractable as N increases.
Motivated by stochastic methods in variational inference [4, 5], this cost can be reduced by involvingonly a random subsample of M ≪
N data points in each Markov chain iteration [6–13].
This reducesthe per-iteration computation time of MCMC, but it can create substantial error in the stationarydistribution of the resulting Markov chain and cause slow mixing [11, 14–17].
Bayesian coresets [18–23] provide an alternative for reducing the cost of MCMC and other inferencemethods.
The key idea is that in a large-scale data setting, much of the data is often redundant.
In particular, there often exists a fixed small, weighted subset of the data—a coreset —that suffices tocapture the dataset as a whole in some sense [21].
Thus, if one can find such a coreset, it can beused in place of the full dataset in the MCMC algorithm, providing the simplicity, generality, andper-iteration speed of data-subsampled MCMC without the statistical drawbacks.
Current state-ofthe-art Bayesian coreset approaches based on sparse variational inference [21] empirically providehigh-quality coresets, but are limited by their significant run-time and lack of theoretical convergenceguarantees.
Methods based on sparse regression [19, 20, 23] are significantly faster and come withsome limited theoretical guarantees, but require a low-cost approximation to the full posterior that is often impractical to find and difficult to tune.
Aside from [21], these methods also require Θ( N ) storage, which is problematic in the large-data setting.
There is a third class of constructions basedon importance sampling [18], but these do not provide reliable posterior approximations in practice.
In this work, we develop a novel coreset construction algorithm that is both faster and easier to tunethan the current state of the art, and has theoretical guarantees on the quality of its output.
The keyinsight in our work is that we can split the construction of a coreset into two stages: first we select thedata in the coreset via uniform subsampling—an idea developed concurrently by [24, 25]—and thenoptimize the weights using a few steps of a novel quasi-Newton method.
Selecting points uniformlyrandomly avoids the slow inner-outer loop optimization of [21], and guarantees that the optimallyweighted coreset has a low KL divergence to the posterior (Theorems 4.1 and 4.2).
Weighting theseuniformly selected points correctly is crucial, and our quasi-Newton method is guaranteed to convergeto a point close to the optimally-weighted coreset (Theorem 4.3) in significantly fewer iterationsthan the method of [21].
Finally, our method is easy to tune and does not require the low-costposterior approximation of [19, 20].
Our experiments show that the algorithm exhibits significantimprovements in coreset quality against alternatives with comparable construction times.
Bayesian coresets
The problem we study is as follows.
Suppose we are given a target probability density π ( θ ) forvariable θ ∈ Θ that is comprised of N potentials ( f n ( θ ))
Nn =1 and base density π 0 ( θ ) ,
[Equation]
where Z is the normalizing constant.
This setup corresponds to a Bayesian statistical model withprior π 0 and i.i.d.
data X n conditioned on θ , where f n ( θ ) = log p ( X n | θ ) .
Computing expectationsunder π exactly is often intractable.
MCMC methods can be employed here, but typically have acomputational complexity of Θ( NT ) to obtain T samples, since (cid:80) n f n ( θ ) needs to be evaluated ateach step.
Bayesian coresets [18] offer an alternative, by constructing a small, weighted subset of thedataset, on which any MCMC algorithm can then be run.
To do this, we find a set of weights w ∈ R N ≥ 0 corresponding to each data point, with the constraintthat only M ≪ N of these are nonzero, i.e. ∥ w ∥ 0 :=
(cid:80) Nn =1 1 w n > 0 ≤ M.
The weighted subset ofpoints corresponding to the strictly positive weights is called a coreset , and we can then create thecoreset posterior approximation,
[Equation]
where Z ( w ) is the new normalizing constant, and
π 1 = π corresponds to the full posterior.
Running
MCMC on this approximation has complexity Θ( MT ) , a considerable speedup if M ≪ N .
Constructing a sparse set of weights w so that π w is as close to π as possible is the key challengehere.
Methods based on importance sampling [18] and sparse regression [19, 20, 23] have beendeveloped, but these provide poor posterior approximations or require first finding a low-cost posteriorapproximation.
The current state-of-the-art approach resolves both of the aforementioned issues byformulating the coreset construction problem as variational inference in the family of coresets [21]:
[Equation]
where D KL ( ·∥· ) denotes the KL divergence.
Noting that coresets are a sparse subset of an exponentialfamily—the weights form the natural parameter w ∈ R N ≥ 0 , the component potentials ( f n ( θ ))
Nn =1form
the sufficient statistic, log Z ( w ) is the log partition function, and π 0 is the base density,
[Equation]
one can obtain a formula for the gradient
[Equation]
where Cov w denotes covariance under π w .
The method of [21] then involves selecting a data pointto add to the coreset, optimizing the coreset weights using stochastic gradient estimates based on
Eq.
(5), and then iterating.
In practice, this method is infeasibly slow outside of small-data problems.
For every data point selected it requires sampling from π w a number of times equal to the number ofoptimization steps (set to 100 in [21]).
This can easily lead to tens of thousands of sampling steps intotal, each of which typically necessitates the use of an MCMC algorithm.
Conversely, our methodrequires tens of sampling steps in total.
Construction via subsampling and quasi-Newton refinement
In this section, we provide a new Bayesian coreset construction algorithm (Algorithm 1) to solve thesparse variational inference problem Eq.
This method involves first uniformly subsampling thedata, and then optimizing the weights on the subsample.
Here, we give a detailed explanation of howthe algorithm works.
We defer to Section 4 the theoretical analysis of the method, which demonstratesthat, with high probability, it finds a near-optimal coreset in a small number of optimization iterations.
3.1 Uniform subsampling
The first key insight to our approach is that we can select the M data points that comprise the coresetby simple uniform subsampling.
From an algorithmic standpoint, the benefit of this approach isclear: it makes the selection of coreset points fast and easy to implement.
It also decouples thesubset selection from the optimizing of the coreset weights, leaving us with a simpler optimizationproblem to solve.
1 Intuitively, this is reasonable from a coreset quality standpoint because a uniformsubsample of the data will, with high probability, create a highly-expressive “basis” of log-likelihoodfunctions ( f n )
Mn =1 with which to approximate the full data log-likelihood.
Of course, we still need tooptimize the coreset weights; but we have not limited the flexibility of the coreset family significantlyby selecting points via uniform subsampling.
Theorem 4.2 provides a precise statement of theseideas, and shows that the optimal weighted coreset posterior built using a uniformly random subset ofdata is a good posterior approximation.
Since the coreset data points are chosen uniformly randomly, we will w.l.o.g. re-label the selectedsubset to have indices 1 , . . .
, M , and the remaining data points to have indices M + 1 , . . .
, N fromnow on.
We can specify a new weight constraint set W ⊂ R N of vectors w with 0 entries for everyindex beyond M , and a further subset W N where the weights sum to N ,
[Equation]
It is also useful to denote g ( θ ) =
( f 1 ( θ ) , . . .
, f M ( θ ))
T ∈ R M as the first M components of f ( θ ) .
3.2 Quasi-Newton optimization
Given the choice of M data points to include in the coreset, it is crucial to optimize their weightscorrectly.
This is the second step of our algorithm.
The KL divergence gradient for the M weights is
[Equation]
[Equation]
To reduce the number of weight optimization iterations —each of which generally involves MCMCsampling from π w , where w is the current set of weights—we develop a second-order optimizationmethod.
Again using the properties of exponential families, we can derive the Hessian of the KLdivergence with respect to the M weights
[Equation]
This matrix is not guaranteed to be positive (semi-)definite; but near the optimum ,where the coresetapproximation is ideally good, i.e., w T f ( θ ) ≈ 1 T f ( θ ) , the first term should dominate the expression.
This motivates the use of
[Equation] [Equation]
to scale gradient steps in the optimization method rather than the true Hessian, thus creating aquasi-Newton method [26].
This heuristic motivates our approach, but it is rigorously justified by
Theorem 4.3, which proves that this scaling results in optimization iterations that are guaranteedto converge exponentially to a coreset which is nearly globally optimal.
Note that when M islarger than the inherent dimension of the space of log-likelihood functions, Cov w [ g, g ] will havezero eigenvalues and be noninvertible.
Therefore we add a regularization τ > 0 prior to inversion.
Theorem 4.3 shows how the regularization influences the optimization, but in general we want τ tobe as small as possible while still ensuring the numerical stability of inverting Cov
w [ g, g ] .
Our optimization method is as follows.
First, we initialize at the uniformly weighted coreset, i.e.,
[Equation]
Next, given a step-size tuning sequence γ k , k ≥ 0 , we update the weights at each iteration k ∈ N ∪{ 0 }using the ( τ > 0) -regularized
quasi-Newton step
[Equation]
Theorem 4.3 suggests that the step size γ k should be constant: γ k = γ ∈ (0 , 1] ; but the analysisin that section assumes that we can compute G ( w ) and H ( w ) exactly, whereas in practice we willhave to estimate them.
Hence we allow γ k to depend on the iteration number in general.
Finally, weproject ˆ w k +1 back onto the constraint set to obtain next iterate w k +1 , i.e. for m = 1 , . . .
, M set
[Equation]
3.3 Algorithm
The pseudocode for the quasi-Newton coreset construction method is shown in Algorithm 1.
Thereare a number of practical considerations needed to use this method; we discuss these here.
The first consideration is that we cannot calculate G ( w ) and H ( w ) in closed form.
Instead, at stepk of the algorithm, we use a Monte Carlo estimate; first taking S samples ( θ s )
Ss =1 i.i.d.
∼ π w k , andcalculating ˆ g s :=
g ( θ s ) − 1 S (cid:80)
Sr =1 g ( θ r ) , so that we may estimate G ( w k ) as
[Equation]
As H ( w k ) is an M × N matrix, estimating it directly will incur a Θ( MN ) storage cost.
Instead, wenote that we only require H ( w k )(1 − w k ) =
Cov w k (cid:2) g, f T (1 − w k ) (cid:3) =
Cov w k [ g, h ] , where h :=
f T 1 − g T w k =
¯ f − g T w k .
We can estimate h using ˆ h s = (cid:80)
Nn =1 (cid:104) f n ( θ s ) − 1 S (cid:80) Sr =1 f n ( θ r ) (cid:105) − ˆ g
Ts w k ,which requires O ( N ) time, but only O ( S ) space.
We can then estimate H ( w k )(1 − w k ) as
[Equation]
Thus, we can find w k +1 from w k by taking the stochastic Newton step
[Equation]
(13) and projecting onto the constraint set: w k + [Equation] +1 ,m ≥ 0 .
We set the regularizationparameter τ by examining the condition number of ˆ G k + τI and keeping it below a reasonable value.
We can tune γ k using a line search method.
As we do not have access to the objective function (i.e.the KL-divergence between the coreset and full posteriors), we use the curvature part of the Wolfeconditions [27] to tune this.
In practice, this line search is expensive.
Thus, we only tune γ k fork ≤ K tune , and leave it as a constant thereafter.
The intuition here is that we may start with quite poorcoreset weights, and so need to choose the initial steps carefully.
Computing ˆ G k + τI involves taking S samples, forming the product ˆ g s ˆ g Ts , and then inverting theresulting matrix.
Its time complexity is thus O ( SM 2 + M 3 ) .
Computing ˆ H ( w k )(1 − w k ) similarly has
Require: ( f n ( θ ))
Nn =1 π 0 , S , K , γ , τ , M .
1: Uniformly sample M data points, w.l.o.g. these correspond to indices 1 , . . . , M . 2: Set w 0 m = NM , m = 1 , . .
. , M. 3: for k = 0 , . . . , K − 1 do 4:
Sample ( θ s ) Ss =1 i.i.d. ∼ π w k ∝ exp (cid:0) w Tk f ( θ ) (cid:1) π 0 ( θ ) 5: Set ˆ g s ← g
θ s ) − 1 S (cid:80) Sr =1 g ( θ r ) and ˆ h s ← (cid:80) Nn =1 (cid:104) f n ( θ s ) − 1 S (cid:80) Sr =1 f n ( θ r ) (cid:105) − ˆ g
Ts w k 6: Set ˆ H k (1 − w k ) ← {+1 S (cid:80) Ss =
 ˆ g s ˆ h s and ˆ G k ← 1 S (cid:80) Ss =
+}
ˆ g s ˆ g
Ts 7: if k ≤ K tune then 8: Choose γ k via line search with starting value γ 9: else 10: Set γ k ← γ 11: Take a quasi-Newton step: ˆ w k +1 ← w k + γ k (cid:16) ˆ G k + τI (cid:17) − 1 ˆ H k
(1 − w k ) 12: Project: for all m ∈ [ M ] , w k +1 ,m ← max ( ˆ w k +1 ,m , 0) 13: return w = w K complexity O
( SM + SN ) , and computing the product with ( ˆ G k + τI )
− 1 has complexity O (cid:0) M 2 (cid:1) .
For K Newton steps in Algorithm 1, the overall time complexity is thus O ( K ( M 3 + SM 2 + SN )) .
The space complexity is O
( M 2 ) (to store ˆ G k ).
This is sublinear in N , unlike in [20, 23].
Finally, we can reduce the time complexity by using a further stochastic estimate of ˆ H k (1 − w k ) .
In particular, we can instead calculate ˆ h s only for indices in I := { 1 , . . .
, M } ∪ T , where T is auniformly selected sample T ⊆ { 1 , . . .
, N } .
Calculating ˆ H ( w k )(1 − w k ) using this subsampledvector ˆ h s, I then has complexity O
( SM + ST ) in the worst case, where T := |T
| .
The overall timecomplexity is then O
( K ( M 3 + SM 2 + ST )) .
Theoretical analysis
In this section we provide a theoretical analysis of the proposed method.
We demonstrate (in
Theorems 4.1 and 4.2) that the optimal coreset posterior built using a uniformly random subset of datais, with high probability, an exact or near-exact approximation of the true posterior.
Second, we show(in Theorem 4.3) that the proposed optimization algorithm converges exponentially quickly to a pointnear the optimal coreset, thus realizing the guarantee in Theorem 4.2 in practice, and theoreticallyjustifying our proposed algorithm.
The assumptions required for our theory are somewhat technical.
We give some intuition here on settings where they hold, but discuss this further in Appendix A.
Proofs of all results may be found in Appendix B.
4.1 Subset selection via uniform subsampling
Theorem 4.1 and Theorem 4.2 provide a theoretical foundation for selection via uniform subsampling.
Both recommend setting M ≳ d log N , where d is essentially the “dimension” of the span of thelog-likelihood functions.
Theorem 4.1 states that if the span of the log-likelihood functions is indeedfinite-dimensional with dimension d —e.g., in a usual exponential family model—then with highprobability, the optimal coreset using a uniformly random selection of M points is exact.
Theorem 4.2extends this to the general setting where Θ ⊆ R D , N is large relative to D , and where we assume theposterior concentrates as N → ∞ .
In particular, Theorem 4.2 states that with high probability, theoptimal coreset using a uniformly random selection of M data points provides low error.
In what follows, we assume that f n ( · ) are i.i.d.
from some true data-generating distribution p , andwe define ¯ f ( · ) =
E p ( f n ( · )) .
We also introduce the new notation f ( · , · ) , where we consider thepotentials f as functions of both x and θ .
This enables us to make the data-dependence explicitwhen needed, i.e., f n ( · ) =
f
( x n , · ) ; for instance, in the standard context of Bayesian inferencef n ( · ) = f
( x n , · ) = log p ( x n |· ) , with x n i.i.d.
∼ p 0 for n ∈ [ N ] .
Theorem 4.1.
Let S 0 be the vector space of functions on Θ spanned by { f ( x, · ); x ∈ X} .
Assumethat S 0 is finite dimensional, with dimension d .
Let µ be any measure on Θ such that S
⊂ L 2 ( µ )and denote r 2 µ = E µ (cid:0) E p ([ f 1 − ¯ f ] 2 ) (cid:1) .
For δ > 0 define
[Equation]
Then for δ > 0 such that J ( δ ) + δ < 1 , a universal constant C 1 , and M ∈ N such that
[Equation]
we have that with probability ≥ 1 − δ − e − MJ (
[Equation] [Equation]
[Equation]
Note that the condition that S 0 has finite dimension d < ∞ is satisfied for all d -dimensionalexponential families when f n ( θ )
= log p ( x n | θ ) .
Further, note that Theorem 4.1 still applies insituations where d increases with N .
To understand the behaviour of J ( δ ) , we consider in Appendix Aa full rank exponential family.
We can show that, under certain conditions, J ( δ ) converges to astrictly positive value as δN → ∞ .
In this case, we can set δ = ω (1 /N ) , such that with probability≥ 1 − ω (1 /N ) , we indeed have that min w ∈W N KL( π w || π ) = 0 for M ≳ d log N .
In the interest of obtaining a more general approximation result, we now consider the case where
S 0 is infinite dimensional, but where we can find a finite-dimensional space S 1 which approximatesf n ( θ ) − ¯ f ( θ ) reasonably well.
A typical setup where this is valid is when Θ ⊆ R D , the posteriorconcentrates (as N → ∞ ) at a point θ 0 ∈ Θ that maximizes ¯ f ( θ ) , and each f n ( · ) is smooth nearθ 0 and globally well-behaved in some sense.
Theorem 4.2 below states, roughly, that for a modelsatisfying these assumptions, min w ∈W N KL( π w || π ) will be small with high probability.
We give theexact conditions in Appendix A. Although they are perhaps strong, the result is still indicative ofwhen a good posterior approximation is obtained by our method.
Intuitively, we require that f ( x, θ ) be differentiable, strongly concave in θ , and “smooth enough”.
Letting θ 0 be the unique value such that ¯ f ( θ ) ≤ ¯ f ( θ 0 ) ∀ θ ∈ Θ , we also require that we can splitf ( x ; θ ) into two parts,
[Equation]
where the span of (cid:8) f (1) ( x, · ) − f (1) ( x, θ 0 ) : x ∈ X (cid:9) is a finite-dimensional vector space S 1 .
Thissplit needs to be made such that f (2) is negligible, in the sense that f ( x, θ ) ≈ f (1) ( x, θ ) locallyaround θ 0 .
A usual case where these conditions hold is when f ( x ; θ ) is C 3 in θ .
In this case, we canobtain f (1) ( x ; θ ) by a second-order Taylor expansion,
[Equation]
which has dimension bounded by 1 + D + D
( D + 1) / 2 .
f (2) ( x ; θ ) is then the remainder term, whichis indeed negligible around θ 0 in the desired sense.
In Appendix A, we provide further discussion onsituations when the necessary conditions for Theorem 4.2 hold.
Recalling that a quantity is o (1) as
N
→ ∞ if and only if it converges to 0, we can then state the result as follows.
Theorem 4.2.
Suppose the assumptions A1-A4 in Appendix A hold, and we set M ≳ D (log N + 1) .
Then as N → ∞ , with probability at least 1 − o (1) , we have that
[Equation]
In the above example, we can examine the proof to obtain a bound on KL( π w || π ) of order 1 / √ Nwith probability converging to 1, showing that this result does indeed lead to a useful bound.
Weight optimization via approximate Newton
Theorems 4.1 and 4.2 guarantee the existence of a high-quality coreset, but do not provide any insightinto whether it is possible to find it tractably.
Theorem 4.3 addresses this remaining gap.
In particular, intuitively, it asserts that as long as (1) the regularization parameter τ > 0 is nonzero but smallenough that it does not interfere with the optimization (in particular, the minimum eigenvalue of
G ( w ) over the space), and (2) the optimal coreset is a good approximation to the full dataset (whichis already guaranteed by Theorem 4.2), then the weights w k at iteration k of the approximate Newtonmethod converge exponentially to a point close to a global optimum.
In the result below, let G ( w )and H ( w ) be defined as in Eq.
Theorem 4.3.
Let W ⊆ W be a closed convex set, W ⋆ ⊆ argmin w ∈ W KL( π w || π ) be a maximalclosed convex subset, and fix the regularization parameter τ > 0 .
Define
[Equation]
Suppose ∃ ϵ ∈ [0 , 1) and δ ≥ 0 such that for all w ∈ W
[Equation]
where w ⋆ = proj W ⋆ ( w ) .
Then the k th iterate w k ∈ W and its projection w ⋆k =
proj W ⋆ ( w k ) of theprojected Newton method defined in Section 3.2 by Eqs.
(8) , (9) and (10) , initialized at w 0 ∈ W witha fixed step size γ ∈ [0 , 1] and regularization τ satisfies
[Equation]
In order to use this result, the minimum positive eigenvalue of G ( w ) for w ∈ W ⊆ W needs to bebounded, as does the error of the optimal coreset weights w ⋆ in the sense of Eq.
A notablecorollary of Theorem 4.3 occurs when there exists a w ⋆ such that∀
[Equation]
i.e., the optimal coreset is identical to the true posterior distribution, as is guaranteed by Theorem 4.1.
For any model where the conditions of Theorem 4.1 hold, such as any exponential family model,
Eq.
(15) holds with ϵ = δ = 0 .
We can thus take γ = 1 to find that
[Equation]
This provides some intuition on Theorem 4.3; as long as the optimal coreset posterior is a reasonableapproximation to π , and we use a small regularization τ > 0 (such that ξ ≈ 1 ), then we generally needonly a small number of optimization steps to find the optimal coreset.
The conditions of Theorem 4.1are sufficient, but not necessary, for Eq.
(15) to hold.
We present further discussion in Appendix A.
Experiments
In this section, we compare our proposed quasi-Newton coreset (QNC) construction against existingconstructions—uniform subsampling (UNIF), greedy iterative geodesic ascent (GIGA)
[20] anditerative hard thresholding (IHT) [23]—as well as the Laplace approximation (LAP), which representswhat one obtains by assuming posterior normality in the large-data setting.
In each case, we use
S = 500 Monte Carlo samples during coreset construction.
We compute error metrics using 1000samples from each method’s approximate posterior.
We also compare to the baseline of samplingfrom the full posterior (FULL) to establish a noise floor for the given comparison sample size of1000.
In the synthetic Gaussian experiment, which is the simplest one we consider, we see that sparsevariational inference (SVI) [21] is prohibitively slow, taking 4 .
2 × 10 3 s to construct a coreset of size1 (and scaling at best linearly with coreset size).
Thus, we do not compare against it.
For GIGA and IHT, we need to supply a low-cost approximation ˆ π to the posterior.
To ensure thesemethods apply as generally as QNC and SVI, we use a uniformly sampled coreset approximation ofsize M with weights N/M (where M is the same as the desired coreset size we are constructing).
We also use these weights for UNIF.
In Appendix C we provide additional results for GIGA and IHTwith a Laplace approximation used for ˆ π .
Experiments are performed in settings standard in the literature, but with larger dataset sizes thanthose that could be considered previously.
For each experiment we plot the approximate reverse KL [Figure] divergence obtained by assuming posterior normality, and the build time.
In Appendix C we providecomparisons on the additional metrics of relative mean and covariance error, forward KL divergenceand per sample time to sample from the respective posteriors.
For the experiments in Sections 5.2and 5.3 with heavy tailed priors, we also compare using the maximum mean discrepancy [28] andkernel Stein discrepancy [29, 30], with inverse multi-quadratic (IMQ) kernel.
Experiments wereperformed on a machine with a 2.6GHz 6-Core Intel Core i7 processor, and 16GB memory.
Throughout the experiments, we see that QNC outperforms the other subsampling methods for mostcoreset sizes we consider.
GIGA and IHT in particular are limited by a fixed choice of ˆ π and finiteprojection dimension defined by our choice of S = 500 —we discuss this further in Appendix C.
Furthermore, outside of the synthetic Gaussian experiment, we see that QNC outperforms LAP forcoreset sizes above certain thresholds, which represent a small fraction of the full dataset.
5.1 Synthetic Gaussian location model
Our first comparison is on a Gaussian location model, with prior θ ∼ N ( µ 0 , σ 0 I ) , and likelihood( X n ) Nn =1 i.i.d.
∼ N ( θ, σI ) in D dimensions.
Here, we take µ 0 = 0 and σ 0 = 1 .
Closed form expressionsare available for the subsampled posterior distributions [21, Appendix B], and we can sample fromthem without Monte Carlo.
We compare the methods on a synthetic dataset with N = 1 , 000 , 000 ,
D = 100 , where we generate the ( X n ) Nn =1 i.i.d.
∼ N ( µ, σI ) and set ( µ i ) Di =1 i.i.d.
∼ N (0 , 100) andσ = 100 .
From Fig.
1a we see that our method outperforms the other subsampling methods for allcoreset sizes.
This example is largely illustrative; we expect LAP to provide the exact posterior hereby design, and this is indeed what we find (the reverse KL plots for LAP and FULL overlap).
5.2 Bayesian sparse linear regression
Next, we study a Bayesian sparse linear regression problem, where the data ( x n , y n ) Nn =1 consists of afeature x n ∈ R D and an outcome y n ∈ R N .
The posterior distribution is that of θ ∈ R D in the model
[Equation]
(16) where ϵ n i.i.d.
∼ N (0 , σ ) , n = 1 , . . .
, N .
We place independent Cauchy priors on the coefficients θ :θ i i.i.d.
∼ Cauchy (0 , τλ i ) , i = 1 , . . .
, D, where the hyperpriors are λ
i i.i.d.
∼ Half-Cauchy (0 , 1) , τ ∼ Half-Cauchy (0 , σ 0 ) , with σ 0 = 2 , andσ ∼ Γ( a 0 , b 0 ) with a 0 = 1 , b 0 = 1 .
We perform posterior inference on the 2 D + 2 − dimensionalset of parameters ( θ 1 , . .
. , θ D , λ 1 , . .
, λ D , σ, τ ) .
Sampling is performed using STAN [31].
Thedataset we study is a flight delays dataset, 2 with N = 100 , 000 and D = 13 (so that the overallinference problem is 28 − dimensional).
The response variable y n is the delay in the departure time ofa flight, and the features are meteorological and flight-specific information.
We see in Fig.
1b that ourmethod outperforms the other subsampling methods for all coreset sizes, and LAP for sizes above500 —representing 0 .
5% of the data.
In Appendix C we see that the target posterior in this case hasheavy tails, which makes the Laplace approximation particularly unsuited to this problem.
We cansee this even more clearly in the additional results presented in Appendix C.
5.3 Heavy-tailed Bayesian logistic regression
For this comparison, we perform Bayesian logistic regression with parameter θ ∈ R D having aheavy-tailed Cauchy prior θ i i.i.d.
∼ Cauchy (0 , σ ) , i = 1 , . . .
, D .
The data ( x n , y n ) Nn =1 consists of afeature x n ∈ R D and a label y n ∈ {− 1 , 1 } .
The relevant posterior distribution is that of θ ∈ R Dwhich governs the generation of y n given x n via [Equation]
[Equation]
Again, sampling is performed using STAN.
As in Section 5.2, we use the flight delays dataset, sowe have that N = 100 , 000 and D = 13 .
The response variables y n are binarized, so that y n = 1 ifa flight was cancelled or delayed by more than an hour, and y n = [-
-]− 1 if it was not.
In Fig.
1c wesee that our method outperforms other subsampling methods for all coreset sizes, and LAP for sizesabove 500 —representing 0 .
5% of the data.
5.4 Bayesian radial basis function regression
Our final comparison is on a Bayesian basis function regression example.
This is a larger version ofthe same experiment performed by [21], with D = 301 as before, but N = 100 , 000 .
Full detailsare given in Appendix C. From Fig.
1d we see that our method provides a significant improvementover LAP for all coreset sizes, which performs particularly poorly in this experiment, despite the
Gaussianity of the target posterior [21, Appendix B].
It also outperforms the other subsamplingmethods for coreset sizes of 1000 and above (representing 1% of the data) but this difference is not asmarked as in the other examples.
The fact that QNC requires a large coreset to start providing a benefitbeyond UNIF stems from the fact that, in this higher dimensional example, there are sometimes datapoints that are very influential for a certain basis, and both UNIF and QNC can miss these.
Thisindicates the need (in some settings) for a secondary method that can check for and include these,without the significant cost that e.g. SVI entails.
We defer this study to future work.
However, we donote that for this example QNC has the lowest build time, even outperforming LAP.
This is becausewe only perform a small number of quasi-Newton steps before no further improvement is possible.
Conclusion
This paper introduces a novel method for data summarization prior to Bayesian inference.
Inparticular, the method first selects a small subset of data uniformly randomly, and then optimizes theweights on those data points using a novel quasi-Newton method.
Theoretical results demonstratethat the method is guaranteed to find a near-optimal coreset, and that the optimal coreset has a low KLdivergence to the posterior with high probability.
Future work includes investigating the performanceof the method in more complex models, and studying conditions under which the method can providecompression in time sublinear in dataset size.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] The claims made in the abstract are supported by thetheoretical and experimental results in Section 4 and Section [-
 (b)-]5(b)
Did you describe the limitations of your work?
[Yes] Methodological limitations arediscussed in Section 3.3 - mainly the fact that we need to used Monte Carlo estimatesof some quantities of interest.
The theoretical results in Section 4 show the settingin which our method can be expected to work.
We give some intuition for whatsettings these are in the main text, with further discussion in Appendix A.
In our fourthexperiment in Section 5.4 we also discuss some of the limitations of our work in highdimensions.
(c)
Did you discuss any potential negative societal impacts of your work?
[No] Our work provides a generic preprocessing algorithm for sampling from probability distributions whose log density involves many summed terms.
Our method is generic andfoundational in nature: it has many downstream applications, including Bayesian inference, which may have a societal impact depending on the particular data and modelunder consideration.
But we do not speculate on the impacts of potential downstreamapplications in this work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The assump-
tions for Theorem 4.1 and Theorem 4.3 are given in the theorem statements.
Beforethe statement of Theorem 4.2 we give intuitive explanations of its assumptions, but the
full assumptions are explicitly laid out in Appendix A. (b) Did you include complete proofs of all theoretical results?
[Yes] Proofs of all theoreticalresults are included in Appendix B.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code,data, and instructions needed to reproduce all experimental results is provided in the
supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Important details are provided in Section 5, and all further details
are included in the code provided as part of the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars representing the 25 th /75 th percentiles over
random trials are presented for each plot of the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] The details of the machine used to
perform all experiments are given in Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Creators of allexisting assets are cited.
(b) Did you mention the license of the assets?
[Yes] All the datasets we use are publicly
available, and some do not include a license.
Where relevant, the license is mentioned.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include new code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The datasets we use are publicly available, and do not pertain to identifiable individuals.
We obey the license terms of the data where applicable, but do
not discuss whether consent was obtained from the data creators.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] None of the datasets we use contain personally
identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of
instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks
, with
links
to Institutional Review Board (IRB) approvals, if applicable?
[N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A