Black-box Coreset Variational Inference
Dionysis Manousakas †∗
Meta dm754@cantab.ac.uk
Hippolyt Ritter ∗
Meta hippolyt@meta.com
Theofanis Karaletsos ‡
Insitro theofanis@insitro.com
Abstract
Recent advances in coreset methods have shown that a selection of representative datapoints can replace massive volumes of data for Bayesian inference, preserving the relevant statistical information and signiﬁcantly accelerating subsequent downstream tasks.
Existing variational coreset constructions rely on either selecting subsets of the observed datapoints, or jointly performing approximate inference and optimizing pseudodata in the observed space akin to inducing points methods in Gaussian Processes.
So far, both approaches are limited by complexities in evaluating their objectives for general purpose models, and require generating samples from a typically intractable posterior over the coreset throughout inference and testing.
In this work, we present a black-box variational inference framework for coresets that overcomes these constraints and enables principled application of variational coresets to intractable models, such as Bayesian neural networks.
We apply our techniques to supervised learning problems, and compare them with existing approaches in the literature for data summarization and inference.
Introduction
Machine learning models are widely trained using mini-batches of data [7, 20], enabling practitioners to leverage large scale data for increasingly accurate models.
However, repeatedly processing these accumulating amounts of data becomes more and more hardware-intensive over time and therefore costly.
Thus, efﬁcient techniques for extracting and representing the relevant information of a dataset for a given model are urgently needed.
Recent work on coresets for probabilistic models has demonstrated that Bayesian posteriors on large scale datasets can be sparsely represented via surrogate densities deﬁned through a weighted subset of the training data (Sparse Variational Inference; Sparse VI ) [9] or a set of learnable weighted pseudo-observations in data space (Pseudodata Sparse Variational Inference; PSVI ) [29].
Similar to inducing points in Gaussian Processes [46], these target to parsimoniously represent the sufﬁcient statistics of the observed dataset that are required for inference in the model.
Replacing the original data via the reduced set of (pseudo-) observations unlocks the potential of scaling up downstream learning tasks, compressing for efﬁcient storage and visualisation, and accelerating model exploration.
Unfortunately, Sparse VI and PSVI rely on access to exact samples from the model posterior and closed form gradients of it to be evaluated and updated.
Yet, the posterior for most common probabilistic models is intractable, limiting their use to the class of tractable models or necessitating the use of heuristics deviating from the core objective.
Dealing with intractable models typically necessitates the use of approximate methods such as variational inference, which has been increasingly popular with the development of black-box techniques for arbitrary models [40].
In this work, we overcome these challenges and introduce a principled framework for performing approximate inference with variational families based on weighted coreset posteriors.
This leads to extensions of Sparse VI and PSVI to black-box models, termed BB Sparse VI and BB PSVI , respectively.
As illustrated in Fig.
1 (and further detailed in the supplement) on a synthetic binary classiﬁcation problem with a Bayesian neural network — an intractable model-class for prior approaches such as PSVI relying on exact posterior samples — BB PSVI is able to learn pseudodata locations and weights inducing a posterior that more faithfully represents the statistical information in the original dataset compared to random selection.
However, we emphasize that the utility of coresets goes beyond applying expensive but accurate inference methods to large-scale datasets, and may drive progress in ﬁelds such as continual learning as we demonstrate later on.
Speciﬁcally, we make the following contributions in this work:
We derive a principled interpretation of coresets as rich variational families, 2.
we modify the objective to obtain a black-box version of it using importance sampling and a nested variational inference step, 3.
utilizing this objective, we propose BB PSVI and BB Sparse VI, two novel suites of variational inference algorithms for black-box probabilistic models, 4.
we empirically compare to prior Bayesian and non-Bayesian coreset techniques, and 5.
we showcase these techniques on previously infeasible models, Bayesian neural networks.
Background 2.1 Variational Inference
Consider a modeling problem where we are given a standard model:
[Equation]
Given a dataset D = { ( x , y ) } ⊆
( X × Y ) N , the task of inference and maximization of the marginal likelihood p ( y | x ) entails estimation of the posterior distribution over model parameters θ , p ( θ | x , y ) .
In general, the posterior p ( θ | x , y ) is not tractable.
A common solution is to resort to approximate inference, for instance via variational inference (VI).
In VI, we assume that we can approximate the true posterior within a variational family q ( θ ; λ ) with free variational parameters λ , by minimizing the Kullback-Leibler (KL) divergence between the approximate and the true posterior
D KL ( q ( θ ; λ ) || p ( θ | x , y )) , or equivalently by maximizing the Evidence Lower Bound (ELBO):
[Equation]
2.2 (Bayesian) coresets and pseudocoresets
Bayesian coreset constructions aim to provide a highly automated data austere approach to the problem of inference, given a dataset D and a probabilistic model Eq.
Agnostic to the particular inference method used, Bayesian coresets shrink in a model-speciﬁc way the original dataset to a much sparser weighted dataset representation { ( v , x , y ) } ⊆
( R ≥
× X × Y ) N , where || v ||
=
M (cid:28) N , that maximizes the parsimony of observations considered over inference, while allowing close approximations of the full data posterior, i.e. p ( θ | x , y , v ) ≈ p ( θ | x , y ) .
Initial Bayesian coreset constructions formulated the problem as yielding a sparsity-constrained approximation of the true data likelihood function uniformly over the parameter space.
This perspective has admitted solutions via convexiﬁcations relying on importance sampling and conditional gradient methods [21, 11], greedy methods for geodesic ascent [10], as well as non-convex approaches employing iterative hard thresholding [52].
Recently, Sparse VI [9] reformulated the problem as performing directly VI within a sparse exponential family of approximate posteriors deﬁned on the coreset datapoints. PSVI
The idea of using a set of learnable weighted pseudodata (or inducing points ) ( v , u , z ) :
= { ( v m , u m , z m )
Mm =1 } ⊆
( R ≥ 0 ×
X × Y )
M to summarize a dataset for the purpose of posterior inference was introduced in [29], as the following objective:
[Equation]
This objective facilitates learning weighted pseudodata such that the exact posterior of model parameters given pseudodata p ( θ | u , z , v ) approximates the exact posterior given the true dataset p ( θ | x , y ) , and was demonstrated on a variety of models as a generally applicable objective.
However, in [29] it was assumed that this objective is tractable, as demonstrated in experiments on models admitting analytical posteriors, or via the use of heuristics to sample from the coreset posterior, as is the case in a logistic regression employing a Laplace approximation.
Sparse VI Sparse VI [9] targets to incrementally minimize the rhs of Eq.
(3) via ﬁxing the coreset points locations and optimizing their weights v .
In its original construction, the coreset points are initialized to the empty set and v to 0 .
In the general case of an intractable model, the coreset posterior is approximately computed via ﬁtting a Laplace approximation q Lap ( θ ; v ) on the weighted datapoints of the coreset ( v , x , y ) .
The next point selection step involves: ( i ) computing approximations of the centered log-likelihood function of each datapoint ( x i , y i ) for model parameters θ s sampled from the current coreset posterior, via an MC estimate ˜f (
[Equation]
( ii ) computing correlations with the residual error of the total data log-likelihood r ( v )
=
f T (1 − v ) , and ( iii ) making a greedy selection of the point that maximizes the correlation between the two: n (cid:63) ∈ arg max n
[Equation]
Corr
[Equation]
, where
[Equation]
Subsequently, the vector v gets optimized via minimizing the KL divergence between the coreset and the true posterior.
For the variational family of coresets, under the assumption that θ is sampled from the true coreset posterior, the formula of the gradient of the objective in Eq.
(3) can be derived, and approximated via resampling from the coreset over the course of v optimization.
At the end of the optimization, the extracted points and weights are given as input to an approximate inference algorithm—commonly a Laplace approximation—which provides the ﬁnal estimate of the posterior.
Black-box coreset Variational Inference
In order to derive a general PSVI objective, we observe that we ﬁrst need to posit a variational family involving the pseudodata appropriately.
Let’s assume the (intractable) variational family q ( θ | u , z ) :
= p ( θ | u , z ) , which has the following particular structure: it is parametrized as the posterior over θ as estimated given inducing points u , z .
In this case, q ( θ | u , z ) can be thought of as a variational program (see [41]) with variational parameters φ = { u , z } and expressed as:
[Equation]
where the marginal likelihood term p ( z | u ) is typically intractable.
Utilizing this for the formulation of a pseudo-coreset variational inference algorithm leads to the following lower bound on the marginal likelihood of the true data:
[Equation]
Performing VI involves maximizing Eq.
(5) with respect to the parameters { u , z } .
Scrutinizing this objective reveals that the posterior q ( θ | u , z ) : = p ( θ | u , z ) is used in two locations:
ﬁrst , as the sampling distribution for θ , second to evaluate log-densities of samples ( scoring ).
This objective can only be evaluated if the posterior p ( θ | u , z ) can be readily evaluated in closed form and sampled from directly.
This causes this expression to remain intractable but for the simplest of cases, and reveals the need to make it computable for any sample θ .
3.1 Bayesian coresets for intractable posteriors
We identiﬁed two key problems to evaluate Eq.
(5) involving q ( θ | u , z ) .
Sampling While we cannot sample from the expectation over q ( θ | u , z ) , we can draw K samples from a tractable parametrized distribution θ k ∼ r ( θ ; ψ ) with variational parameters ψ (i.e. any suitable variational family). However, we cannot plug these samples from r into Eq.
(5) directly, as they do not follow the exact distribution called for.
To overcome this, we can leverage a selfnormalized importance sampling (IS) correction [37] to obtain the desired approximate samples from the coreset posterior.
We thus estimate q ( θ | u , z ) via importance sampling by re-weighting samples θ from the tractable distribution r ( θ ; ψ ) .
We denote the resulting implicit distribution q ( θ | u , z ; ψ ) .
We assign the samples unnormalized weights w k :
[Equation]
and denote the corresponding normalized weights ˜ w
[Equation]
Using q ( θ | u , z ) from Eq.
(4), we rewrite Eq.
(5) as:
[Equation]
Scoring After having tackled sampling, we shift our attention to the second problem, evaluating q ( θ | u , z ) inside the logarithm.
The key problem with scoring log q ( θ | u , z ) as deﬁned in Eq.
(4) is the intractable marginal probability of the inducing points (IP) log p ( z | u ) .
We overcome this by introducing a variational approximation to log q ( θ | u , z ) using the same distribution r ( θ ; ψ ) used above and obtain evidence lower bound ELBO IP of the IP:
[Equation]
Putting it all together We return to Eq. (7) and substitute q ( θ k | u , z ) , per Eq.
(4) to derive the following multi-sample importance sampling lower bound on the evidence of the observed data and expose the term log p ( z | u ) :
[Equation]
By lower bounding log p ( z | u ) with ELBO IP ( ψ ) , we now propose our ﬁnal black box objective ELBO PSVI-IS-BB as a lower bound to ELBO PSVI-IS ( u , z ) , under which we can tractably score u , z :
[Equation]
We now remind the reader out that by introducing ELBO PSVI-IS we lower bounded log p ( y | x ) , and that ELBO PSVI-IS-BB rigorously lower bounds ELBO PSVI-IS , showing that our proposed fully tractable black box objective ELBO PSVI-IS-BB is a rigorous variational objective for approximating log p ( y | x ) :
[Equation]
Maximizing ELBO PSVI-IS-BB leads to optimizing parameters ψ and thus adapting the proposals r for the importance weighting throughout inference, within the conﬁnes of the chosen variational family.
This is a common approach when blending variational inference and Monte Carlo corrections and can be thought of as specifying an implicit variational family.
Depending on the structure of the model one might elect to replace importance sampling with richer Monte Carlo schemes here.
We also note that parameters ψ serve maximization of evidence for inducing points given ﬁxed { u , z } , while parameters φ = { u , z } are adapted to maximize evidence over observed data { x , y } which need to account for during optimization of said parameters in Section 3.4.
Studying this objective further reveals that replacing our pseudodata dependent importance weights with a uniform distribution ˜ w k = 1 /K , k = 1 , . . .
, K , or using a single-sample objective (i.e. in the degenerate case
when
K = 1 ), allows us to recover the classical ELBO of Eq.
(2), cancelling the pseudodata likelihood terms in the variational objective.
The setting where we apply this black-box estimator to the basic
PSVI algorithm for inferring locations φ and model variables ψ will be denoted BB PSVI .
3.2 Variational families: (un)weighted inducing points
When studying the objective in Eq.
(5) it quickly becomes evident that the exact amount of the selected coreset datapoints deﬁnes the available evidence for that particular posterior construction and as such is a quantity of interest.
We will consider this quantity pseudo-evidence .
For notational brevity so far we have considered the variational parameters φ to represent pseudodata directly, while rescaling using the ﬁxed data compression ratio to maintain invariance of the total pseudo-evidence to the coreset size.
We will be calling this setting unweighted pseudodata .
One choice we can make now is to ensure that the available pseudo-evidence is invariant to the chosen size of the coreset support and potentially equal to the regular dataset.
To achieve that, we set:
[Equation]
[Equation]
with p ( z | u ) similarly reweighted via the selected observational data compression ratio N/M , i.e. for continuous parameter spaces p ( z | u ) :
= (cid:82) p ( z | θ , u ) N/M p ( θ ) d θ .
We may also consider a scenario where we have more generally weighted pseudodata , which involves an additional learnable parameter v i ≥ 0 per datapoint
u i and permits the following interpretation: v i ∼ Multi ( v ) .
Sampling repeatedly from this distribution yields a collection of inducing points with their frequencies governed by their respective weights:
[Equation]
Simplifying the notation for weighted log-likelihoods throughout, we denote:
[Equation]
[Equation]
We can now also combine the ideas of learning the weights of pseudodata and controlling the amount of pseudoevidence by posing parametrizations where v i is non-negative and sums to(e.g.
v i := softmax ( β i ) , i = 1 , . . . , M ), and adding global variational parameters to optimize the magnitude of total evidence over the pseudodata (e.g. log p ( z | v , u , θ , α ) :
= α v T log p ( z | u , θ ) ), where α can be set by hand (i.e. to N/M ) or be an extra learnable quantity.
We will explore various such choices for coreset variational family parametrizations φ = { u , z , v , α } in our experiments and the effects they have on learning.
3.3 Incremental and Batch Black-Box Sparse VI
The estimator we derive can also be used to design a black-box version of the Sparse VI algorithm introduced in [9] (see Sec. 2.2), by the insight that it requires only the variational parameters for coreset weights to be updated while using copies of real datapoints as coreset locations. This unlocks various choices to the overall algorithmic ﬂow. The original incremental construction scheme can become black-box via modifying how the approximate posterior on the coreset data gets computed, and introducing a generalized objective for the optimization of model evidence involving the coreset parameters.
In this construction ( BB Sparse VI ), we posit a variational family r ( θ ; ψ ) (e.g. mean-ﬁeld variational distributions) and maximize the corresponding ELBO computed on the weighted datapoints of the coreset.
Using the extracted variational approximation, we draw samples from the coreset and correct for the model.
Next, over the greedy selection step, we use the samples and importance weights to compute the centered log-likelihood vectors ˜f ( x n , y n , θ s ) :
= log p ( y n | x n , θ s ) − (cid:80) s (cid:48)
w s (cid:48) log p ( y n | x n , θ s (cid:48) ) , and select the next datapoint via the greedy correlation maximization criterion.
Subsequently, to reﬁne the weight vector towards the true data posterior, we maximize w.r.t.
v , ψ an extension of the PSVI ELBO per Eq.
(9), where the coreset data log-likelihood is multiplied by the weight vector v as in Eq.
[Equation]
The importance sampling scheme for the posterior is deﬁned identically to Eq.
(6), after substituting the likelihood term with p ( y | v , x , θ k ) for each sample θ k ∼ r ( θ k ; ψ ) .
At each iteration, this variational objective (which, following similar reasoning with the previous section, can be shown to be a lower bound of the evidence), can be maximized wrt the coreset and variational parameters v , ψ .
Moreover, we can omit the incremental inclusion of points to the coreset, and consider a batch version of this construction, where we initialise at a random subset of the original dataset, keep coreset point locations ﬁxed, and optimize only the weights attached to the coreset support using Eq.
(13) with non-negativity constraints.
Finally, we use the variational approximation ﬁt on the coreset to compute predictive posteriors on unseen datapoints correcting for our importance sampling.
We also deﬁne a variant of the algorithm without incremental selection through pruning based on the weighting assigned to the coreset points when optimizing the ELBO of Eq.
We can start from a large coreset size and, after training, reduce the size of the summary by keeping K samples from a multinomial deﬁned on the coreset points via their so far learned weights, and re-initialising.
We provide pseudo-code for the respective methods in Algs.
3 − 5 in Supplement B. 3.4 Nested optimization When using a nested optimizer for the maximization of the ELBO PSVI-IS-BB , rearranging the terms of the objective places our method in the generic cardinality-constrained bilevel optimization framework for data distillation and coreset constructions [24, 5].
VI maximizing the objective of Eq.
(5) inside a parametric family r ( θ ; ψ ) can be reformulated as a bilevel optimization problem as:
[Equation]
where
[Equation]
and
[Equation]
We optimize via iterative differentiation [27], i.e. trace the gradient computation over the optimization of the variational parameters ψ when solving the inner problem, and use this information at the computation of the outer objective gradient wrt the variational coreset parameters φ = { v , u , z , α } .
Experiments
In this section we evaluate the performance of our inference framework in intractable models and compare against standard variational inference methods and earlier Bayesian coreset constructions, as well as black-box extensions of existing variational coresets that rely on our generalized ELBO Eq.
As a running baseline with unrestricted access to the training data, we use the standard mean-ﬁeld
VI with a diagonal covariance matrix.
To capture the implications of enforcing data cardinality constraints, we also construct a random coreset using a randomly selected data subset with ﬁxed data locations and ﬁxed likelihood multiplicities correcting for the dataset size reduction, which we then use to optimize a mean-ﬁeld approximate posterior.
As earlier work relied on Laplace approximations on coreset data as a black-box approach for sampling for intractable coreset posteriors, we also experiment with Laplace approximations on random data subsets as an extra baseline.
We make code available at www.github.com/facebookresearch/Blackbox-Coresets-VI .
4.1 Logistic regression
First, we perform inference on logistic regression ﬁtting 3 publicly available binary classiﬁcation datasets [49, 16] with sizes ranging between 10 k and 150 k datapoints, and 10 and 128 dimensions.
We posit normal priors θ ∼ N (0 , I ) and consider mean-ﬁeld variational approximations with diagonal covariance.
In the following, all presented metrics are averaged across 3 independent trials, and we show the means along with the corresponding standard errors.
Impact of variational parameters We present the predictive metrics of accuracy and log-likelihood on the test set in Tables 1 and 2.
We observe that coreset methods relying on optimizing pseudodata have the capacity to better approximate inference on the full dataset for small coreset sizes regardless of data dimension, reaching the performance of unrestricted mean-ﬁeld VI with the use of less than 50 weighted points.
In contrast, the approximations that rely on existing datapoints are limited by data dimensionality and need to acquire a larger support to reach the performance of full-data VI (with effects being more evident in webspam , which has a data dimensionality of 128).
Including variational parameterisations for optimal scaling of the coreset likelihood aids inference, offering faster convergence.
Omitting to correct for dataset reduction ( v = 1 ) has detrimental effects on predictive performance.
As opposed to earlier constructions limited by heuristics on coreset posterior approximation throughout optimization, our variational framework enables approximating the full data posterior within the assumed variational family.
Evaluation of predictive metrics and computation time requirements across a wider range of coreset sizes is included in the supplement.
Impact of importance weighting In Fig.
2 we plot the normalized effective sample size (ESS) of our importance weighting scheme for 10 Monte Carlo samples computed on test data.
We observe that the posteriors can beneﬁt from IW, achieving non-trivial ESSs, which tend to converge to close values for large coreset sizes.
This ﬁnding applies at the testing phase also on ablations of the PSVI variational training that replace the IW scheme with uniform sampling.
Importance sampling in high dimensions.
Importance sampling can suffer from large variance in high dimensional spaces [37].
Hence, one may be skeptical about the approximation quality the scheme can provide.
To this end, in Supplement G we conduct a complementary experiment with a synthetic logistic regression problem.
As dimensionality increases, we ﬁnd that while ESS goes down, it remains non-trivial.
More involved sampling methods such as [54] are a promising avenue for leveraging the modular nature of BB PSVI to achieve further performance improvements.
4.2 Bayesian Neural Networks
In this section we present inference results on Bayesian neural networks (BNNs), a model class that previous work on Bayesian coresets did not consider due to the absence of a black-box variational estimator.
In the ﬁrst part we perform inference via black-box PSVI on 2-dimensional synthetic data [Table] using single-hidden layer BNNs, while in the latter part we evaluate the performance of our methods on compressing the MNIST dataset using the LeNet architecture [23].
Simulated datasets We generate two synthetic datasets with size 1 k datapoints, corresponding to noisy samples from a half-moon shaped 2-class dataset, and a mixture of 4 unimodal clusters of data each belonging to a different class [22] [Table] and use a layer with 20 and 50 units respectively.
To evaluate the representation ability of the pseudocoresets we consider two initialization schemes: we initialise the pseudo locations on a random subset equally split across categories [Table] and a random initialization using a Gaussian centered on the means of the empirical distributions.
In Fig.
3 we visualize the inferred predictive posterior entropy, along with coreset points locations and relative weights.
Regardless of the initialization, BB PSVI has the capacity to optimize the locations of the coreset support according to the statistics of the true data distribution, yielding correct decision boundaries.
Importantly, the coreset support consists itself of separable summarizing pseudo points,
[Table]
[Table]
while the likelihood reweighting variational parameters enable emphasizing on critical regions of the landscape, assigning larger weights to pseudopoints lying close to the boundaries among the classes.
Moreover, our pruning scheme which makes use of batch BB Sparse VI is able to gradually compress a large coreset of 250 datapoints to a compact performant subset of 20 datapoints arranged in critical locations for learning the Bayesian posterior.
MNIST In this part we assess the approximation quality of large-scale dataset compression for
BNNs via coresets.
We compare the predictive performance of black-box PSVI against standard mean-ﬁeld VI, random coresets and frequentist methods relying on learnable synthetic data, namely dataset distillation w/ and w/o learnable soft labels [50, 45], and data condensation [53].
We can see that BB PSVI consistently outperforms the random coreset baseline for coreset sizes in the orders of few tens or hundreds, however the gap narrows towards 250 coreset points (Table 3).
Moreover, at size 100 our construction provides better test accuracy compared to two frequentist approaches, enjoying the additional advantage of providing uncertainty estimates.
Continual learning with Bayesian coresets As discussed in the introduction, we envision coresets being leveraged as a tool beyond scaling sampling methods to large datasets.
One such example setting is continual learning, where past methods have encoded statistical information in the form of approximate posterior distributions over the weights [34, 43].
While [34] leverages (randomly selected) coresets, these necessarily lose information about the data due to the approximate nature of their closed-form posteriors.
Bayesian coresets, in contrast do not make use of an explicit closedform posterior, as they rely on the implicit posterior induced by the coreset, representing the prior information by mixing the pseudo data at frequency equal to their weight with the new data.
We showcase such a continual learning method on the four-class synthetic problem from above in Fig.with the difference that each class arrives sequentially after the initial two and prior classes are not revisited except for the coresets (details in Supplement D).
Non-Bayesian approaches to continual learning with coresets have been investigated in more depth in [5, 4, 51].
Related work and discussion
Coresets and data pruning beyond Bayesian inference The paradigm of coresets emerged from seminal work in computational geometry [1, 18], and has since found broad use in data-intensive machine learning, including least mean square solvers [26], dimensionality reduction [19], clustering [17, 25, 2] and maximum likelihood estimation [3, 30].
Coreset ideas have recently found applications in active learning [39, 6], continual learning [5, 4] and robustness [31, 28].
Recent works in neural networks have investigated heuristics for pruning datasets, including the misclassiﬁcation events of datapoints over learning [48], or gradient scores in early stages of training [38].
Pseudodata for learning Despite the breakthroughs of classical coreset methods, the common conﬁne of selecting the coresets among the constituent datapoints might lead to large summary sizes in practice.
Exempt from this limitation are methods that synthesize informative pseudo-examples for training that can summarize statistics lying away from the data manifold.
These appear in the literature under various terms, including dataset distillation [27, 50, 24], condensation [53], dataset meta-learning [35] and inducing points [36].
Moreover, recent work [47] has demonstrated that incorporating learnable pseudo-inputs as hyperparameters of the model prior can enhance inference in variational auto-encoders.
Learnable pseudodata that resemble the sufﬁcient statistics of the original dataset have also been used in herding [12] and scalable GP inference methods [44, 46].
Variational inference, importance weighting, and Monte Carlo objectives Importance weighting can provide tighter lower bounds of the data log-likelihood for training variational autoencoders [8], resulting in richer latent representations [13].
In later work, [14] propose an importance weighting VI scheme for general purpose probabilistic inference, while [42, 41] introduce variational programs as programmatic ways to specify a variational distribution.
Higher dimensional proposals are commonly handled via Sequential Monte Carlo techniques [15, 32], while [54] develops a family of methods for learning proposals for importance samplers in nested variational approximations. Two forces are at play when considering our approximate inference scheme: our variational family r and the sampling scheme we use to estimate q ( θ | u , z ; ψ ) . These two interact nontrivially, since r forms the proposal for our importance sampling scheme, and this inference yields gradients for the formation of the coreset. It is fair to assume that for harder inference problems richer inference strategies will yield a useful coreset and posterior belief over the model faster and with more accuracy.
Conclusion
We have developed a framework for novel black-box constructions of variational coresets that can be applied at scale to large datasets as well as richly structured models, and result in effective modelconditional dataset summarization.
We overcome the intractability of previous inference algorithms with our proposed black-box objectives, which we develop for both Sparse VI as well as PSVI, and propose a suite of algorithms utilizing them.
Through experiments on intractable models, we show that the inferred coresets form a useful summary of a dataset with respect to the learning problem at hand, and are capable to efﬁciently compress information for Bayesian posterior approximations.
In future work, we plan to extend this inference toolbox for privacy purposes and non-traditional learning settings, e.g. continual learning, explore alternative objectives, and richer models of coresets.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]