Fast Bayesian Coresets via Subsampling and Quasi-Newton Refinement
Abstract
Bayesian coresets approximate a posterior distribution by building a small weighted subset of the data points.
Any inference procedure that is too computationally expensive to be run on the full posterior can instead be run inexpensively on the coreset, with results that approximate those on the full data.
However, current approaches are limited by either a significant run-time or the need for the user to specify a low-cost approximation to the full posterior.
We propose a Bayesian coreset construction algorithm that first selects a uniformly random subset of data, and then optimizes the weights using a novel quasi-Newton method.
Our algorithm is a simple to implement, black-box method, that does not require the user to specify a low-cost posterior approximation.
It is the first to come with a general high-probability bound on the KL divergence of the output coreset posterior.
Experiments demonstrate that our method provides significant improvements in coreset quality against alternatives with comparable construction times, with far less storage cost and user input required.
Introduction
Bayesian methods are key tools for parameter estimation and uncertainty quantification, but exact inference is rarely possible for complex models.
Currently, the gold standard method for approximate inference is Markov chain Monte Carlo (MCMC) [1; 2; 3, Ch.
11,12], which involves simulating a
Markov chain whose stationary distribution is the Bayesian posterior.
However, modern applications are often concerned with very large datasets; in this setting, MCMC methods typically have a Θ( NT ) complexity—for T samples and dataset size N —which quickly becomes intractable as N increases.
Motivated by stochastic methods in variational inference [4, 5], this cost can be reduced by involving only a random subsample of M ≪
N data points in each Markov chain iteration [6–13].
This reduces the per-iteration computation time of MCMC but it can create substantial error in the stationary distribution of the resulting Markov chain and cause slow mixing [11, 14–17].
Bayesian coresets [18–23] provide an alternative for reducing the cost of MCMC and other inference methods.
The key idea is that in a large-scale data setting, much of the data is often redundant.
In particular, there often exists a fixed small, weighted subset of the data—a coreset —that suffices to capture the dataset as a whole in some sense [21].
Thus, if one can find such a coreset, it can be used in place of the full dataset in the MCMC algorithm, providing the simplicity, generality, and per-iteration speed of data-subsampled MCMC without the statistical drawbacks.
Current state-of- the-art Bayesian coreset approaches based on sparse variational inference [21] empirically provide high-quality coresets, but are limited by their significant run-time and lack of theoretical convergence guarantees.
Methods based on sparse regression [19, 20, 23] are significantly faster and come with some limited theoretical guarantees, but require a low-cost approximation to the full posterior. This approximation is often impractical to find and difficult to tune.
Moreover, it can fundamentally limit the quality of the coreset obtained.
Aside from [21], these methods also require Θ( N ) storage, which is problematic in the large-data setting.
There is a third class of constructions based on importance sampling [18], but these do not provide reliable posterior approximations in practice.
In this work, we develop a novel coreset construction algorithm that is both faster and easier to tune than the current state of the art, and has theoretical guarantees on the quality of its output.
The key insight in our work is that we can split the construction of a coreset into two stages: first we select the data in the coreset via uniform subsampling—an idea developed concurrently by [24, 25]—and then optimize the weights using a few steps of a novel quasi-Newton method.
Selecting points uniformly randomly avoids the slow inner-outer loop optimization of [21], and guarantees that the optimallyweighted coreset has a low KL divergence to the posterior (Theorems 4.1 and 4.2).
Weighting these uniformly selected points correctly is crucial, and our quasi-Newton method is guaranteed to converge to a point close to the optimally-weighted coreset (Theorem 4.3) in significantly fewer iterations than the method of [21].
Finally, our method is easy to tune and does not require the low-cost posterior approximation of [19, 20].
Our experiments show that the algorithm exhibits significant improvements in coreset quality against alternatives with comparable construction times.
Bayesian coresets
The problem we study is as follows.
Suppose we are given a target probability density π ( θ ) for variable θ ∈ Θ that is comprised of N potentials ( f n ( θ ))
Nn =1 and base density π 0 ( θ ) ,
[Equation]
where Z is the normalizing constant.
This setup corresponds to a Bayesian statistical model with prior π 0 and i.i.d.
data X n conditioned on θ , where f n ( θ ) = log p ( X n | θ ) .
Computing expectations under π exactly is often intractable.
MCMC methods can be employed here, but typically have a computational complexity of Θ( NT ) to obtain T samples, since (cid:80) n f n ( θ ) needs to be evaluated at each step.
Bayesian coresets [18] offer an alternative, by constructing a small, weighted subset of the dataset, on which any MCMC algorithm can then be run.
To do this, we find a set of weights w ∈ R N ≥ 0 corresponding to each data point, with the constraint that only M ≪ N of these are nonzero, i.e. ∥ w ∥ 0 : =
(cid:80) Nn =1 1 w n > 0 ≤ M.
The weighted subset of points corresponding to the strictly positive weights is called a coreset , and we can then create the coreset posterior approximation,
[Equation]
where Z ( w ) is the new normalizing constant, and
π 1 = π corresponds to the full posterior.
Running
MCMC on this approximation has complexity Θ( MT ) , a considerable speedup if M ≪ N .
Constructing a sparse set of weights w so that π w is as close to π as possible is the key challenge here.
Methods based on importance sampling [18] and sparse regression [19, 20, 23] have been developed, but these provide poor posterior approximations or require first finding a low-cost posterior approximation.
The current state-of-the-art approach resolves both of the aforementioned issues by formulating the coreset construction problem as variational inference in the family of coresets [21]:
[Equation]
where D KL ( ·∥· ) denotes the KL divergence.
Noting that coresets are a sparse subset of an exponential family—the weights form the natural parameter w ∈ R N ≥ 0 , the component potentials ( f n ( θ ))
Nn = form the sufficient statistic, log Z ( w ) is the log partition function, and π 0 is the base density,
[Equation]
one can obtain a formula for the gradient
[Equation]
where Cov w denotes covariance under π w .
Here, and throughout, we take derivatives with respect to w on the ambient space w ∈ R N .
The method of [21] then involves selecting a data point to add to the coreset, optimizing the coreset weights using stochastic gradient estimates based on Eq.
(5), and then iterating.
In practice, this method is infeasibly slow outside of small-data problems.
For every data point selected it requires sampling from π w a number of times equal to the number of optimization steps (set to 100 in [21]).
This can easily lead to tens of thousands of sampling steps in total, each of which typically necessitates the use of an MCMC algorithm.
Conversely, our method requires tens of sampling steps in total.
Construction via subsampling and quasi-Newton refinement
In this section, we provide a new Bayesian coreset construction algorithm (Algorithm 1) to solve the sparse variational inference problem Eq.
This method involves first uniformly subsampling the data, and then optimizing the weights on the subsample.
Here, we give a detailed explanation of how the algorithm works.
We defer to Section 4 the theoretical analysis of the method, which demonstrates that, with high probability, it finds a near-optimal coreset in a small number of optimization iterations.
3.1 Uniform subsampling
The first key insight to our approach is that we can select the M data points that comprise the coreset by simple uniform subsampling.
From an algorithmic standpoint, the benefit of this approach is clear: it makes the selection of coreset points fast and easy to implement.
It also decouples the subset selection from the optimizing of the coreset weights, leaving us with a simpler optimization problem to solve.
1 Intuitively, this is reasonable from a coreset quality standpoint because a uniform subsample of the data will, with high probability, create a highly-expressive “basis” of log-likelihood functions ( f n )
Mn =1 with which to approximate the full data log-likelihood.
Of course, we still need to optimize the coreset weights; but we have not limited the flexibility of the coreset family significantly by selecting points via uniform subsampling.
Theorem 4.2 provides a precise statement of these ideas, and shows that the optimal weighted coreset posterior built using a uniformly random subset of data is a good posterior approximation.
Since the coreset data points are chosen uniformly randomly, we will w.l.o.g. re-label the selected subset to have indices 1 , . . .
, M , and the remaining data points to have indices M + 1 , . .
, N from now on.
We can specify a new weight constraint set W ⊂ R N of vectors w with 0 entries for every index beyond M , and a further subset W N where the weights sum to N ,
[Equation]
It is also useful to denote g ( θ ) =
( f 1 ( θ ) , . . .
, f M ( θ ))
T ∈ R M as the first M components of f ( θ ) .
3.2 Quasi-Newton optimization
Given the choice of M data points to include in the coreset, it is crucial to optimize their weights correctly.
This is the second step of our algorithm.
The KL divergence gradient for the M weights is
[Equation]
To reduce the number of weight optimization iterations —each of which generally involves MCMC sampling from π w , where w is the current set of weights—we develop a second-order optimization method.
Again using the properties of exponential families, we can derive the Hessian of the KL divergence with respect to the M weights
[Equation]
This matrix is not guaranteed to be positive (semi-)definite. However, writing out the second term explicitly as [Equation] we see that this term will be small near the optimum, where the coreset approximation is ideally good, i.e., w T f ( θ ) ≈ 1 T f ( θ ) .
The first term does not contain this f T (1 − w ) term, and thus should dominate the expression.
This motivates the use of
[Equation]
to scale gradient steps in the optimization method rather than the true Hessian, thus creating a quasi-Newton method [26].
This heuristic motivates our approach, but it is rigorously justified by
Theorem 4.3, which proves that this scaling results in optimization iterations that are guaranteed to converge exponentially to a coreset which is nearly globally optimal.
Note that when M is larger than the inherent dimension of the space of log-likelihood functions, Cov w [ g, g ] will have zero eigenvalues and be noninvertible.
Therefore we add a regularization τ > 0 prior to inversion.
Theorem 4.3 shows how the regularization influences the optimization, but in general we want τ to be as small as possible while still ensuring the numerical stability of inverting Cov
w [ g, g ] .
Our optimization method is as follows.
First, we initialize at the uniformly weighted coreset, i.e.,
[Equation]
Next, given a step-size tuning sequence γ k , k ≥ 0 , we update the weights at each iteration k ∈ N ∪{ 0 } using the ( τ > 0) -regularized
quasi-Newton step
[Equation]
Theorem 4.3 suggests that the step size γ k should be constant: γ k = γ ∈ (0 , 1] ; but the analysis in that section assumes that we can compute G ( w ) and H ( w ) exactly, whereas in practice we will have to estimate them.
Hence we allow γ k to depend on the iteration number in general.
Finally, we project ˆ w k +1 back onto the constraint set to obtain next iterate w k +1 , i.e. for m = 1 , . . .
, M set
[Equation]
3.3 Algorithm
The pseudocode for the quasi-Newton coreset construction method is shown in Algorithm 1.
There are a number of practical considerations needed to use this method; we discuss these here.
The first consideration is that we cannot calculate G ( w ) and H ( w ) in closed form.
Instead, at step k of the algorithm, we use a Monte Carlo estimate; first taking S samples ( θ s )
Ss =1 i.i.d.
∼ π w k , and calculating ˆ g s :=
g ( θ s ) − 1 S (cid:80)
S r =1 g ( θ r ) , so that we may estimate G ( w k ) as
[Equation]
Using a Monte Carlo estimate in this way introduces one source of error.
Furthermore, we often cannot sample exactly from π w k , and instead use MCMC to do so.
This could lead to additional errors, for example if our samples are from unconverged chains.
However, we find that this method works well in practice, for a reasonable number of samples S .
As H ( w k ) is an M × N matrix, estimating it directly will incur a Θ( MN ) storage cost.
Instead, we note that we only require H ( w k )(1 − w k ) =
Cov w k (cid:2) g, f T (1 − w k ) (cid:3) =
Cov w k [ g, h ] , where h :=
f T 1 − g T w k =
¯ f − g T w k .
We can estimate h using ˆ h s = (cid:80) N n =1 (cid:104) f n ( θ s ) − 1 S (cid:80)
S r =1 f n ( θ r ) (cid:105) − ˆ g T s w k , which requires O ( N ) time, but only O ( S ) space.
We can then estimate H ( w k )(1 − w k ) as
[Equation]
Require: ( f n ( θ ))
Nn =1 π 0 , S , K , K tune , γ , τ , M .
1: Uniformly sample M data points, w.l.o.g.
these correspond to indices 1 , . . .
, M .
2: Set w 0 m = NM , m = 1 , . .
. , M.
3: for k = 0 , . . .
, K − 1 do 4:
Sample ( θ s )
Ss =1 i.i.d.
∼ π w k ∝ exp (cid:0) w Tk f ( θ ) (cid:1) π 0 ( θ ) 5:
Set ˆ g s ← g
θ s ) − 1 S (cid:80)
Sr =1 g ( θ r ) and ˆ h s ← (cid:80)
Nn =1 (cid:104) f n ( θ s ) − 1 S (cid:80)
Sr =1 f n ( θ r ) (cid:105) − ˆ g
Ts w k 6:
Set ˆ H k (1 − w k ) ← 1 S (cid:80) Ss =
ˆ g s ˆ h s and ˆ G k ← 1 S (cid:80)
Ss =
ˆ g s ˆ g
Ts 7: if k ≤ K tune then 8:
Choose γ k via line search with starting value γ 9: else 10:
Set γ k ← γ 11:
Take a quasi-Newton step: ˆ w k +1 ← w k + γ k (cid:16) ˆ G k + τI (cid:17) − 1 ˆ H k
(1 − w k ) 12:
Project: for all m ∈ [ M ] , w k +1 ,m ← max ( ˆ w k +1 ,m , 0) 13: return w = w K
Thus, we can find w k +1 from w k by taking the stochastic Newton step
[Equation] (13) and projecting onto the constraint set: w k +1 ,m = ˆ w k +
[Equation] ,m 1 ˆ w k +1 ,m ≥ 0 . We set the regularization parameter τ by examining the condition number of ˆ G k + τI and keeping it below a reasonable value. We can tune γ k using a line search method.
As we do not have access to the objective function (i.e. the KL-divergence between the coreset and full posteriors), we use the curvature part of the Wolfe conditions [27] to tune this.
In practice, this line search is expensive.
Thus, we only tune γ k for k ≤ K tune , and leave it as a constant thereafter.
The intuition here is that we may start with quite poor coreset weights, and so need to choose the initial steps carefully.
In Appendix C, we perform a sensitivity analysis for the parameters S , K tune and τ .
We see that our results are generally not sensitive to the choice of these parameters, within reasonable ranges. Computing ˆ G k + τI involves taking S samples, forming the product ˆ g s ˆ g Ts , and then inverting the resulting matrix. Its time complexity is thus O ( SM 2 + M 3 ) . Computing ˆ H ( w k )(1 − w k ) similarly has complexity O ( SM + SN ) , and computing the product with ( ˆ G k + τI ) − 1 has complexity O (cid:0) M 2 (cid:1) .
For K Newton steps in Algorithm 1, the overall time complexity is thus O ( K ( M 3 + SM 2 + SN )) , which is linear in N .
The space complexity is O
( M 2 ) (to store ˆ G k ), which is sublinear in N , unlike in [20, 23].
In theory, we can reduce the time complexity by using a further stochastic estimate of ˆ H k (1 − w k ) .
In particular, we can instead calculate ˆ h s only for indices in I := { 1 , . . .
, M } ∪ T , where T is a uniformly selected sample T ⊆ { 1 , . . .
, N } .
Calculating ˆ H ( w k )(1 − w k ) using this subsampled vector ˆ h s, I then has complexity O
( SM + ST ) in the worst case, where T := |T
| .
The overall time complexity is then O
( K ( M 3 + SM 2 + ST )) , which is sublinear in N if T = o ( N ) .
This could give significant improvements in coreset construction time in the large-data regime.
However, this approach requires further study, as we find that it leads to a degradation in the performance of our algorithm. We do not use it in our experiments. The final practical consideration in the design of Algorithm 1 is that we cannot actually calculate the objective function that we are trying to minimize, namely the KL divergence D KL ( π w ∥ π 1 ) between the coreset and full posteriors. Thus, it may be hard to tell if the optimization is actually making progress.
In practice, we monitor the norm of the gradient of the KL divergence (which we do have access to, as given by Eq.
We can terminate our algorithm early if we do not see a significant enough decrease in this measure.
Theoretical analysis
In this section we provide a theoretical analysis of the proposed method.
We demonstrate (in
Theorems 4.1 and 4.2) that the optimal coreset posterior built using a uniformly random subset of data is, with high probability, an exact or near-exact approximation of the true posterior.
Second, we show (in Theorem 4.3) that the proposed optimization algorithm converges exponentially quickly to a point near the optimal coreset, thus realizing the guarantee in Theorem 4.2 in practice, and theoretically justifying our proposed algorithm.
The assumptions required for our theory are somewhat technical.
We give some intuition here on settings where they hold, but discuss this further in Appendix A.
Proofs of all results may be found in Appendix B.
4.1 Subset selection via uniform subsampling
Theorem 4.1 and Theorem 4.2 provide a theoretical foundation for selection via uniform subsampling.
Both recommend setting M ≳ d log N , where d is essentially the “dimension” of the span of the log-likelihood functions.
Theorem 4.1 states that if the span of the log-likelihood functions is indeed finite-dimensional with dimension d —e.g., in a usual exponential family model—then with high probability, the optimal coreset using a uniformly random selection of M points is exact.
Theorem 4. extends this to the general setting where Θ ⊆ R D , N is large relative to D , and where we assume the posterior concentrates as N → ∞ .
In particular, Theorem 4.2 states that with high probability, the optimal coreset using a uniformly random selection of M data points provides low error.
In what follows, we assume that f n ( · ) are i.i.d.
from some true data-generating distribution p , and we define ¯ f ( · ) =
E p ( f n ( · )) .
We also introduce the new notation f ( · , · ) , where we consider the potentials f as functions of both x and θ .
This enables us to make the data-dependence explicit when needed, i.e., f n ( · ) =
f
( x n , · ) ; for instance, in the standard context of Bayesian inference f n ( · ) = f
( x n , · ) = log p ( x n |· ) , with x n i.i.d.
∼ p 0 for n ∈ [ N ] .
Theorem 4.1.
Let S 0 be the vector space of functions on Θ spanned by { f ( x, · ); x ∈ X} .
Assume that S 0 is finite dimensional, with dimension d .
Let µ be any measure on Θ such that S
⊂ L 2 ( µ ) and denote r 2 µ = E µ (cid:0) E p ([ f 1 − ¯ f ] 2 ) (cid:1) .
For δ > 0 define
[Equation]
Then for δ > 0 such that J ( δ ) + δ < 1 , a universal constant C 1 , and M ∈ N such that
[Equation]
we have that with probability ≥ 1 − δ − e − MJ (
[Equation]
[Equation]
Note that the condition that S 0 has finite dimension d < ∞ is satisfied for all d -dimensional exponential families when f n ( θ )
= log p ( x n | θ ) .
Further, note that Theorem 4.1 still applies in situations where d increases with N .
To understand the behaviour of J ( δ ) , we consider in Appendix A a full rank exponential family.
We can show that, under certain conditions, J ( δ ) converges to a strictly positive value as δN → ∞ .
In this case, we can set δ = ω (1 /N ) , such that with probability ≥ 1 − ω (1 /N ) , we indeed have that min w ∈W N KL( π w || π ) = 0 for M ≳ d log N .
In the interest of obtaining a more general approximation result, we now consider the case where
S 0 is infinite dimensional, but where we can find a finite-dimensional space S 1 which approximates f n ( θ ) − ¯ f ( θ ) reasonably well.
A typical setup where this is valid is when Θ ⊆ R D , the posterior concentrates (as N → ∞ ) at a point θ 0 ∈ Θ that maximizes ¯ f ( θ ) , and each f n ( · ) is smooth near θ 0 and globally well-behaved in some sense.
Theorem 4.2 below states, roughly, that for a model satisfying these assumptions, min w ∈W N KL( π w || π ) will be small with high probability.
We give the exact conditions in Appendix A. Although they are perhaps strong, the result is still indicative of when a good posterior approximation is obtained by our method.
Intuitively, we require that f ( x, θ ) be differentiable, strongly concave in θ , and “smooth enough”.
Letting θ 0 be the unique value such that ¯ f ( θ ) ≤ ¯ f ( θ 0 ) ∀ θ ∈ Θ , we also require that we can split f ( x ; θ ) into two parts,
[Equation]
where the span of (cid:8) f (1) ( x, · ) − f (1) ( x, θ 0 ) : x ∈ X (cid:9) is a finite-dimensional vector space S 1 .
This split needs to be made such that f (2) is negligible, in the sense that f ( x, θ ) ≈ f (1) ( x, θ ) locally around θ 0 .
A usual case where these conditions hold is when f ( x ; θ ) is C 3 in θ .
In this case, we can obtain f (1) ( x ; θ ) by a second-order Taylor expansion,
[Equation]
which has dimension bounded by 1 + D + D
( D + 1) / 2 .
f (2) ( x ; θ ) is then the remainder term, which is indeed negligible around θ 0 in the desired sense.
In Appendix A, we provide further discussion on the conditions of Theorem 4.2.
Recalling that a quantity is o (1) as N → ∞ if and only if it converges to 0, we can then state the result as follows.
Theorem 4.2.
Suppose the assumptions A1-A4 in Appendix A hold, and we set M ≳ D (log N + 1) .
Then as N → ∞ , with probability at least 1 − o (1) , we have that
[Equation]
In the above example, we can examine the proof to obtain a bound on KL( π w || π ) of order 1 / √ N with probability converging to 1, showing that this result does indeed lead to a useful bound.
4.2 Quasi-Newton weight optimization
Theorems 4.1 and 4.2 guarantee the existence of a high-quality coreset, but do not provide any insight into whether it is possible to find it tractably.
Theorem 4.3 addresses this remaining gap.
In particular, intuitively, it asserts that as long as (1) the regularization parameter τ > 0 is nonzero but small enough that it does not interfere with the optimization (in particular, the minimum eigenvalue of
G ( w ) over the space), and (2) the optimal coreset is a good approximation to the full dataset (which is already guaranteed by Theorem 4.2), then the weights w k at iteration k of the approximate Newton method converge exponentially to a point close to a global optimum.
In the result below, let G ( w ) and H ( w ) be defined as in Eq.
Theorem 4.3.
Let W ⊆ W be a closed convex set, W ⋆ ⊆ argmin w ∈ W KL( π w || π ) be a maximal closed convex subset, and fix the regularization parameter τ > 0 .
Define
[Equation]
Suppose ∃ ϵ ∈ [0 , 1) and δ ≥ 0 such that for all w ∈ W
[Equation]
where w ⋆ = proj W ⋆ ( w ) .
Then the k th iterate w k ∈ W and its projection w ⋆ k = proj
W ⋆ ( w k ) of the projected Newton method defined in Section 3.2 by Eqs.
(8) , (9) and (10) , initialized at w 0 ∈ W with a fixed step size γ ∈ [0 , 1] and regularization τ satisfies
[Equation]
For this result to make sense, we require that η ∈ [0 , 1] .
We have that η ≥ 0 , since γ ∈ [0 , 1] , ϵ ≥and ξ ≤ 1 by definition.
It may be the case that η ≥ 1 , in which case our result still holds, but is vacuous.
In order to use this result, the minimum positive eigenvalue of G ( w ) for w ∈ W ⊆ W needs to be bounded, as does the error of the optimal coreset weights w ⋆ in the sense of Eq.
A notable corollary of Theorem 4.3 occurs when there exists a w ⋆ such that
[Equation]
i.e., the optimal coreset is identical to the true posterior distribution, as is guaranteed by Theorem 4.1.
For any model where the conditions of Theorem 4.1 hold, such as any exponential family model,
Eq.
(15) holds with ϵ = δ = 0 .
We can thus take γ = 1 to find that
[Equation]
This provides some intuition on Theorem 4.3; as long as the optimal coreset posterior is a reasonable approximation to π , and we use a small regularization τ > 0 (such that ξ ≈ 1 ), then we generally need only a small number of optimization steps to find the optimal coreset.
The conditions of Theorem 4. are sufficient, but not necessary, for Eq.
(15) to hold.
We present further discussion in Appendix A.
We note that if ξ = 0 , then there is no convergence, even in the setting where Theorem 4.1 holds.
However, it is important to emphasises that, for any given dataset, we are free to choose τ .
We can (and should) choose τ to be sufficiently small that ξ is roughly 1 .
In particular, this means setting τ significantly smaller than the minimum positive eigenvalue of G ( w ) .
Experiments
In this section, we compare our proposed quasi-Newton coreset (QNC) construction against existing constructions—uniform subsampling (UNIF), greedy iterative geodesic ascent (GIGA)
[20] and iterative hard thresholding (IHT) [23]—as well as the Laplace approximation (LAP), which represents what one obtains by assuming posterior normality in the large-data setting.
Experiments were performed on a machine with a 2.6GHz 6-Core Intel Core i7 processor, and 16GB memory; code is available at https://github.com/trevorcampbell/quasi-newton-coresets-experiments .
In each case, we use {+S = 
+}
Monte Carlo samples during coreset construction.
We compute error metrics using 1000 samples from each method’s approximate posterior.
We also compare to the baseline of sampling from the full posterior (FULL) to establish a noise floor for the given comparison sample size of 1000.
In the synthetic Gaussian experiment, which is the simplest one we consider, we see that sparse variational inference (SVI) [21] is prohibitively slow, taking 4 .
2 × 10 3 s to construct a coreset of size 1 (and scaling at best linearly with coreset size).
Thus, we do not compare against it here.
In Appendix C we run a smaller, lower dimensional, synthetic Gaussian experiment.
Here, we compare against SVI, and confirm that it is prohibitively slow for the size of datasets that we consider.
For GIGA and IHT, we need to supply a low-cost approximation ˆ π to the posterior.
To ensure these methods apply as generally as QNC and SVI, we use a uniformly sampled coreset approximation of size M with weights N/M (where M is the same as the desired coreset size we are constructing).
We also use these weights for UNIF.
In Appendix C we provide additional results for GIGA and IHT with a Laplace approximation used for ˆ π .
This does not lead to a significant improvement, and we provide further discussion there.
Experiments are performed in settings standard in the Bayesian coresets literature, but with larger dataset sizes than those that could be considered previously.
For each experiment we plot the approximate reverse KL divergence obtained by assuming posterior normality, and the build time.
In Appendix C we provide comparisons on the additional metrics of relative mean and covariance error, forward KL divergence and per sample time to sample from the respective posteriors.
For the experiments in Sections 5.2 and 5.3 with heavy tailed priors, we also compare using the maximum mean discrepancy [28] and kernel Stein discrepancy [29, 30], with inverse multi-quadratic (IMQ) kernel.
Throughout the experiments, we see that QNC outperforms the other subsampling methods for most coreset sizes we consider.
GIGA and IHT in particular are limited by a fixed choice of ˆ π and finite projection dimension defined by our choice of S = 500 —we discuss this further in Appendix C.
Furthermore, outside of the synthetic Gaussian experiment, we see that QNC outperforms LAP for coreset sizes above certain thresholds, which represent a small fraction of the full dataset.
In Appendix C, we also perform a sensitivity analysis for the parameters S , K tune and τ that we use in Algorithm 1.
We see that our results are generally not sensitive to the choice of these parameters, within reasonable ranges.
[Figure]
5.1 Synthetic Gaussian location model
Our first comparison is on a Gaussian location model, with prior θ ∼ N ( µ 0 , σ 0 I ) , and likelihood ( X n ) Nn =1 i.i.d.
∼ N ( θ, σI ) in D dimensions.
Here, we take µ 0 = 0 and σ 0 = 1 .
Closed form expressions are available for the subsampled posterior distributions [21, Appendix B], and we can sample from them without MCMC.
We compare the methods on a synthetic dataset with N = 1 , 000 , 000 ,
D = 100 , where we generate the ( X n ) Nn =1 i.i.d.
∼ N ( µ, σI ) and set ( µ i ) Di =1 i.i.d.
∼ N (0 , 100) and σ = 100 .
From Fig.
1a we see that our method outperforms the other subsampling methods for all coreset sizes.
This example is largely illustrative; we expect LAP to provide the exact posterior here by design, and this is indeed what we find (the reverse KL plots for LAP and FULL overlap).
5.2 Bayesian sparse linear regression
Next, we study a Bayesian sparse linear regression problem, where the data ( x n , y n ) Nn =1 consists of a feature x n ∈ R D and an outcome y n ∈ R N .
The posterior distribution is that of θ ∈ R D in the model y n
[Equation]
(16) where ϵ n i.i.d.
∼ N (0 , σ ) , n = 1 , . . .
, N .
We place independent Cauchy priors on the coefficients θ : θ i i.i.d.
∼ Cauchy (0 , τλ i ) , i = 1 , . . .
, D, where the hyperpriors are λ
i i.i.d.
∼ Half-Cauchy (0 , 1) , τ ∼ Half-Cauchy (0 , σ 0 ) , with σ 0 = 2 , and σ ∼ Γ( a 0 , b 0 ) with a 0 = 1 , b 0 = 1 .
We perform posterior inference on the 2 D + 2 − dimensional set of parameters ( θ 1 , . .
. , θ D , λ 1 , . .
, λ D , σ, τ ) .
Sampling is performed using STAN [31].
The dataset we study is a flight delays dataset, 2 with N = 100 , 000 and D = 13 (so that the overall inference problem is 28 − dimensional).
The response variable y n is the delay in the departure time of a flight, and the features are meteorological and flight-specific information.
We see in Fig.
1b that our method outperforms the other subsampling methods for all coreset sizes, and LAP for sizes above 500 —representing 0 .
5% of the data.
In Appendix C we see that the target posterior in this case has heavy tails, which makes the Laplace approximation particularly unsuited to this problem.
We can see the effect this has even more clearly in the additional results presented there.
5.3 Heavy-tailed Bayesian logistic regression
For this comparison, we perform Bayesian logistic regression with parameter θ ∈ R D having a heavy-tailed Cauchy prior θ i i.i.d.
∼ Cauchy (0 , σ ) , i = 1 , . . .
, D .
The data ( x n , y n ) Nn =1 consists of a feature x n ∈ R D and a label y n ∈ {− 1 , 1 } .
The relevant posterior distribution is that of θ ∈ R D which governs the generation of y n given x n via
[Equation]
Again, sampling is performed using STAN.
As in Section 5.2, we use the flight delays dataset, so we have that N = 100 , 000 and D = 13 .
The response variables y n are binarized, so that y n = 1 if a flight was cancelled or delayed by more than an hour, and y n = − 1 if it was not.
In Fig.
1c we see that our method outperforms other subsampling methods for all coreset sizes, and LAP for sizes above 500 —representing 0 .
5% of the data.
5.4 Bayesian radial basis function regression
Our final comparison is on a Bayesian basis function regression example.
This is a larger version of the same experiment performed by [21], with D = 301 as before, but N = 100 , 000 .
Full details are given in Appendix C. From Fig.
1d we see that our method provides a significant improvement over LAP for all coreset sizes, which performs particularly poorly in this experiment, despite the
Gaussianity of the target posterior [21, Appendix B].
It also outperforms the other subsampling methods for coreset sizes of 1000 and above (representing 1% of the data) but this difference is not as marked as in the other examples.
The fact that QNC requires a large coreset to start providing a benefit beyond UNIF stems from the fact that, in this higher dimensional example, there are sometimes data points that are very influential for a certain basis, and both UNIF and QNC can miss these.
This indicates the need (in some settings) for a secondary method that can check for and include these, without the significant cost that e.g. SVI entails.
We defer this study to future work.
However, we do note that for this example QNC has the lowest build time, even outperforming LAP.
This is because we only perform a small number of quasi-Newton steps before no further improvement is possible.
Conclusion
This paper introduces a novel method for data summarization prior to Bayesian inference.
In particular, the method first selects a small subset of data uniformly randomly, and then optimizes the weights on those data points using a novel quasi-Newton method.
Theoretical results demonstrate that the method is guaranteed to find a near-optimal coreset, and that the optimal coreset has a low KL divergence to the posterior with high probability.
Future work includes investigating the performance of the method in more complex models, and studying conditions under which the method can provide compression in time sublinear in dataset size.
Acknowledgments and Disclosure of Funding
T. Campbell was supported by a National Sciences and Engineering Research Council of Canada (NSERC)
Discovery Grant and Discovery Launch Supplement.
C. Naik was supported by the
Engineering and Physical Sciences Research Council and Medical Research Council [award reference 1930478].
J. Rousseau was supported by the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (grant agreement No 834175).