On Divergence Measures for Bayesian Pseudocoresets
Abstract
A Bayesian pseudocoreset is a small synthetic dataset for which the posterior over parameters approximates that of the original dataset.
While promising, the scalability of Bayesian pseudocoresets is not yet validated in realistic problems such as image classification with deep neural networks.
On the other hand, dataset distillation methods similarly construct a small dataset such that the optimization using the synthetic dataset converges to a solution with performance competitive with optimization using full data.
Although dataset distillation has been empirically verified in large-scale settings, the framework is restricted to point estimates, and their adaptation to Bayesian inference has not been explored.
This paper casts two representative dataset distillation algorithms as approximations to methods for constructing pseudocoresets by minimizing specific divergence measures: reverse
KL divergence and Wasserstein distance.
Furthermore, we provide a unifying view of such divergence measures in Bayesian pseudocoreset construction.
Finally, we propose a novel Bayesian pseudocoreset algorithm based on minimizing forward
KL divergence.
Our empirical results demonstrate that the pseudocoresets constructed from these methods reflect the true posterior even in high-dimensional
Bayesian inference problems.
Introduction
One of the main ingredients for modern statistical machine learning is large-scale data, which enables the learning of powerful and flexible models such as deep neural networks when handled correctly [3, 32]. However, training a neural network on larger datasets usually requires many nontrivial choices, ranging from architecture choice, hyperparameter optimization and infrastucture challenges [33].
Aside from requiring enormous computing resources for training, large-scale data often contains sensitive information that must not be shared publicly [11].
This motivates the construction of coresets [29, 14], a sparse subset of a large-scale dataset that summarizes essential features of the original dataset.
In the context of Bayesian inference, the posterior conditioned on an essential coreset should be similar to the exact posterior conditioned on the full dataset.
Coreset construction becomes more difficult with high-dimensional data.
A previous work [19] shows that even the best coreset becomes suboptimal at large scales: the divergence between the posterior conditioned on the optimal coreset and the exact posterior grows with the dimension of the data.
Motivated by this weakness of strict coresets, Manousakas et al.
[19] proposes to construct a Bayesian pseudocoreset : a synthetic dataset represented as a set of learnable parameters and trained to minimize the divergence between the coreset posterior and the full-data posterior.
Compared to coresets, pseudocoresets scale much better with data dimension and achieve significantly lower posterior approximation error.
However, the KL divergence minimization in pseudocoreset learning requires the construction of approximate posteriors on the fly during the learning process, making this approach hard to apply to high-dimensional models such as deep neural networks.
On the other hand, dataset distillation [28] aims to distill a large-scale dataset into a small synthetic dataset such that the test performance of a model trained on the distilled data is comparable to that of a model trained on the original data. Such methods have achieved remarkable performance in real-world problems with high-dimensional data and deep neural networks.
Although dataset distillation methods have a similar motivation to pseudocoresets, the two methods optimize different objectives.
Whereas pseudocoresets minimize the divergence between coreset and full-data posteriors, dataset distillation considers a non-Bayesian setting with heuristically set objective functions such as matching the gradients of loss functions [36] or matching the parameters obtained from optimization trajectories [8].
Such objectives have no direct Bayesian analogue due to their heuristic nature, and more theoretical grounding can advance our understanding of dataset distillation and the empirical performance of Bayesian pseudocoreset methods.
In this paper, we provide a unifying view of Bayesian pseudocoresets and dataset distillation to take advantages of both approaches.
We first study various choices of divergence measures as objectives for learning pseudocoresets.
While Manousakas et al. [19] minimizes the reverse KL divergence between the pseudocoreset posterior distribution and the full data posterior distribution, we show that alternative divergence measures such as forward KL divergence and Wasserstein distance are also effective in practice.
Based on this perspective, we re-interpret existing dataset distillation algorithms [36, 8] approximations to the Bayesian pseudocoresets learned by minimizing reverse KL and Wasserstein distance, with specific choices of variational approximations for coreset posteriors.
This connection justifies the heuristically chosen learning objectives of dataset distillation algorithms, and at the same time, provides a theoretical background for using the distilled datasets obtained from such procedure for Bayesian inference.
Conversely, Bayesian pseudocoresets benefit from this connection by borrowing various ideas and tricks used in the dataset distillation algorithms to make them scale for complex tasks.
For instance, the variational posteriors we identified by recasting dataset distillation algorithms as Bayesian pseudocoreset learning can be used for Bayesian pseudocoresets with various choices of divergence measures.
Also, we can adapt the idea of reusing pre-computed optimization trajectories [8] which have already been shown to work at large scales.
We empirically compare pseudocoreset algorithms based on three different divergence measures on high-dimensional image classification tasks.
Our results demonstrate that we can efficiently and scalably construct Bayesian pseudocoresets with which MCMC algorithms such as Hamiltonian Monte-Carlo (HMC) [10, 22] or Stochastic Gradient Hamiltonian Monte-Carlo (SGHMC) [9] accurately approximate the full posterior.
Background 2.1 Bayesian Pseudocoresets
Denote observed data as x = { x n } Nn =1 .
Given a probabilistic model indexed by a parameter θ with some prior distribution π 0 , we are interested in the posterior conditioned on the full data x ,
[Equation]
where f ( x , θ )
:=
[ f ( x 1 , θ ) , . .
f ( x N , θ )] ⊤ , 1 N is the N -dimensional one vector, and Z ( x ) = (cid:82) exp( 1 ⊤ N f ( x , θ ))
d θ is a partition function.
The posterior π x is usually intractable to compute due to Z ( x ) , so we employ approximate inference algorithms.
Bayesian pseudocoreset methods [19] aim to construct a synthetic dataset called a pseudocoreset u = { u m } Mm =1 with M ≪ N such that the posterior of θ conditioned on it approximates the original posterior π x .
The pseudocoreset posterior is written as 1 ,
[Equation]
where f ( u , θ ) , 1 M , and Z ( u ) are defined similarly.
Manousakas et al.
[19] suggests to learn u by minimizing the KL divergence between π u and π x :
[Equation]
and show that the gradient of this objective is computed as
[Equation]
The expectation over the pseudocoreset posterior π u in the above equation is further approximated with the Monte-Carlo estimation with an approximate posterior q u ≈ π u , for instance, from the
Laplace approximation.
2.2 Dataset Distillation
With a similar motivation as Bayesian pseudocoresets, dataset distillation methods construct a small synthetic dataset u , but from the perspective of matching optimal parameters.
That is, in dataset distillation, we find u such that
[Equation]
with a loss function ℓ ( · , θ ) and a distance function d ( · , · ) comparing two parameters.
In general, optimizing d ( θ ∗ x , θ ∗ u ) is intractable since it is a bilevel optimization problem requiring θ ⋆ x and θ ∗ u , so existing works relax the problem in several ways, such as using kernel-based approaches [23, 24] or solving only for a short horizon in inner optimization [36, 34, 8].
Below, we briefely review two representative methods—Dataset Condensation (DC) [36] and Matching Training Trajectories (MTT) [8].
Dataset Condensation Instead of fully unrolling the inner optimization process, DC instead computes the matching objective at every inner step starting from some random parameter θ (0) .
[Equation]
where d cos denotes the cosine similarity between two gradients.
To implement this objective, we build a trajectory of parameters starting from a random θ (0) with the current distilled dataset u .
We then compute the gradient matching objective for each layer at each step of the trajectory to update u .
Despite its appealing simplicity, this algorithm may suffer from short-horizon bias due to the limited trajectory length.
Matching Training Trajectories MTT extends the single step gradient matching in DC and considers longer inner optimization trajectories.
Starting from a randomly initialized parameter θ (0) ,
MTT takes multiple inner optimization steps with both u and x , and minimize the normalized Ldistance between two parameters at the end of the learning trajectories.
[Equation]
where d 2 ( · , · ) is the normalized L 2 distance between parameters.
Unlike DC, this algorithm starts from a common initial parameter θ (0) , keeps track of two separate optimization trajectories (one with u and another with x ), and the outer-update for u is done only after L x and L u inner steps.
To reduce the heavy computational cost for the inner update with the full dataset x , MTT employs mini-batch
SGD for the inner steps, in addition to saving and re-using precomputed SGD trajectories to train u .
The use of precomputed SGD trajectories is similar to that of replay buffers for reinforcement learning [17, 20, 1], and allows MTT to handle long inner-loop sequences.
While MTT significantly improves over DC, presumably due to its longer inner optimization horizon, the algorithm requires large memory for even a moderate number of inner-steps L u because it requires backpropagation through all inner steps with u .
On Divergence Measures for Bayesian Pseudocoresets
While the Bayesian pseudocoreset is originally obtained by minimizing the reverse KL divergence, in principle we can minimize an arbitrary divergence measure D ( · , · ) to achieve the goal:
[Equation]
In this section, we study three different choices for such divergence measures along with their relation to dataset distillation methods.
3.1 Reverse KL Divergence
Manousakas et al.
[19] proposed to minimize reverse KL divergence with a Laplace approximation to obtain the pseudocoreset posterior π u .
Here, we show that it is possible to recast Dataset Condensation (DC) as a reverse KL divergence minimization with an alternative choice for the variational posterior.
Let θ ( t ) be a parameter in a learning trajectory with a loss function ℓ ( u , θ ) :
= − 1 ⊤ M f ( u , θ ) .
Provided that θ ( t ) is sufficiently close to a local optimum, we can construct a naïve approximation of π u as
[Equation]
Under this setting, we can actually show that one gradient descent step with respect to reverse KL divergence D KL [ π u ∥ π x ] is equivalent to one step of gradient matching in DC when the parameter being optimized is close to a local optima.
We note that while the Gaussian posterior is a simple approximation class, it is commonly used due to its demonstrated usefulness in several literatures [18], such as the Bernstein-von Mises theorem [16, 27].
However, we also note that these Gaussian approximations are not necessary for establishing the connection between the dataset distillation and Bayesian pseudocoresets; Gaussian approximations are just a special case which have the benefit of simplifying the analysis.
Proposition 3.1.
Let q u ( θ ) set as Eq.
Assume θ
( t − 1) is close enough to a local optima, so that we have ∥ θ ( t − 1) − θ ( t ) ∥ ≪ 1 .
Then we have
[Equation]
The proof is given in Appendix A.
In other words, when sufficiently close to a local optima, DC can be interpreted as a special instance of Bayesian pseudocoreset via reverse KL minimization with
Gaussian approximation for the pseudocoreset posterior π u .
Throughout the paper, we will refer to the Bayesian pseudocoreset with reverse KL minimization as BPC-rKL.
To simply implement the
Algorithm 1 in [19] for the image dataset, we approximated π u as a Gaussian distribution.
Please refer to Appendix B.2 for the detailed algorithm.
3.2 Wasserstein Distance
We can instead choose D ( · , · ) to be the Wasserstein distance between π u and π x .
While the Wasserstein distance is intractable in general, by approximating both π u and π x with Gaussian distributions, we can attain a closed-form expression for this distance metric.
Specifically, let θ u
and θ x be two parameters sufficiently close to local optima.
Then we can construct a crude approximation for π u and π x as,
[Equation] [Equation]
Then the Wasserstein distance between these two distributions are computed as
[Equation]
Now consider a single outer-step of the MTT.
Starting from a common initial θ 0 , the algorithm takes the gradient steps both with x and u to get θ ( L x ) x
and θ ( L u ) u .
If we set q u ( θ ) =
N ( θ ; θ ( L u ) u , Σ) and q x ( θ ) =
N ( θ ; θ ( L x ) x , Σ) with a common covariance Σ , then the Wasserstein distance reduces to ∥ θ ( L x ) x − θ ( L u ) u ∥ 22 , which is proportional to the L 2 objective being optimized in the MTT.
Similarly to the reverse KL divergence and BPC-rKL, we refer to the Bayesian pseudocoreset with Wasserstein distance minimization as BPC-W.
3.3 Forward KL Divergence
Finally, we consider an alternative Bayesian pseudocoreset algorithm based on minimizing forward KL divergence.
The forward KL minimization is known to encourage a model to cover the entire target distribution while the reverse KL minimization favors mode capturing models, indicating that the forward KL may be a better choice for the Bayesian pseudocoreset.
Under our setting, the forward
KL is computed as
[Equation]
Then the gradient w.r.t.
the pseudocoreset u is computed as,
[Equation]
Again [Equation] we should approximate the expectation w [Equation]
r [Equation] t. π u and π x to obtain this gradient.
We introduce similar variational posteriors,
[Equation]
The endpoint θ ( L u ) u is a function of u , so in principle, the expectation w.r.t.
q u involves unrolling through the parameter trajectories θ (0)
→ θ (1) u
→ · · · → θ ( L u ) u , as for the BPC-W. We employ truncated backpropagation to reduce memory cost due to this unrolling, i
[Equation] e., we block the gradient flow through θ ( L u ) to obtain
[Equation]
where sg ( · ) denotes the stop-grad operation.
We further approximate this via Monte-Carlo estimation,
[Equation]
with ε (1) x , . . .
, ε ( S ) x , ε (1) u , . .
. , ε ( S ) u i .
i .
d . ∼ N (0 , I ) .
The resulting algorithm starts from a common initial parameter θ (0) , keeps track of two parameter trajectories, and minimizes the difference in the loglikelihood 1 ⊤ M f ( u , θ ) evaluated at the Gaussian neighbors of the endpoints of the two trajectories.
Note that even though we block gradient flow through θ ( L u ) u , the pseudocoreset u still influences each likelihood term f ( u , θ ) , so the truncated gradient includes meaningful learning signals.
In contrast, if we apply a similar trick to the BPC-W, the matching objective would be ∥ sg ( θ ( L u ) u ) − θ ( L x ) x ∥ , which is constant w.r.t.
u .
We can additionally borrow an idea from MTT, where we utilize the set of expert trajectories to efficiently evaluate the parameter trajectory with x .
We call this overall algorithm BPC-fKL, and provide a summary of the overall procedure in Algorithm 1.
Relation to contrastive divergence BPC-fKL is closely related to contrastive divergence [13, 7], a learning algorithm for approximating the maximum likelihood estimator.
Suppose we want to train a model p θ ( x ) =
1 Z ( θ ) exp
( f θ ( x )) by maximizing the log-likelihood function of training examples {
[Equation]
[Equation]
Then the gradient of the log-likelihood is
[Equation]
Contrastive divergence approximates this gradient by approximating the second term with sampling methods such as Langevin dynamics or HMC.
By doing this we can obtain the MLE for log-likelihood without knowing the exact potential function.
Our proposed BPC-fKL performs the same function as the contrastive divergence if the model parameter θ and training data x change to the Bayesian pseudocoresets u and the parameters from the true posterior distribution which are approximated by the points near the expert trajectories.
In this point of view, BPC-fKL is the maximum likelihood estimator that maximizes the log-likelihood of the parameters from the posterior distribution of the original dataset.
Related Works
Bayesian coresets As running algorithms such as MCMC and VI on large datasets is challenging due to expensive computational cost, Bayesian coresets [14], methods that represent the entire dataset as a sparse and weighted sum of subsets, have been studied.
Campbell and Broderick [6] interprets coreset construction as a sparse approximation of vector sums and generalizes it to a sparse regression problem in a Hilbert space.
Campbell and Broderick [5] demonstrates that Hilbert coresets scale the coresets log-likelihood sub-optimally, and presents a modified algorithm.
As Hilbert coresets include the choice of weighted L 2 inner product or finite-dimensional projection and it is difficult to choose them optimally, another method that minimizes the KL divergence between the coreset posterior and true posterior has also been studied recently [4].
Bayesian coresets have the advantages of simple and theoretically guaranteed quality of the posterior approximations.
However, for high-dimensional data, considering only subsets of the dataset fails as even an optimal coreset has a KL divergence that increases with data dimension. Additionally, privacy concerns make subset-based coresets difficult to apply to real-world conditions.
Bayesian pseudocoresets [19], constructed from learned synthetic datapoints, can avoid these shortcomings of coresets, but have been empirically validated only in relatively easier low-dimensional settings such as logistic regression. To our best knowledge, this paper is the first to experimentally demonstrate the viability of Bayesian pseudocoresets for high-dimensional real data.
Dataset distillation The goal of dataset distillation [28] is to create a small synthetic dataset that allows the model to have similar test performance to the original dataset even after training with the smaller synthetic dataset.
Such synthetic datasets have potential applications in many subfields of machine learning such as continual learning and neural architecture search.
However, dataset distillation typically involves a bilevel optimization structure in the learning process, which suffers from high computational costs or training instability.
Existing methods alleviate these challenges by matching one-step gradients between synthetic and real data [36, 34], solving using a closed form solution of kernel ridge regression [23, 24], or reducing the normalized distance between parameters at the end of synthetic and real training trajectories [8].
However, because these methods focus only on high test accuracy, they do not consider uncertainty or the degree to which the posterior distribution on the synthetic dataset matches the true posterior distribution.
While Zhao and Bilen [35] matches the distributions of embedding features for many random networks of synthetic and real data, this [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
1.0 Frequency 0.
L o g a m p li t u d e
BPC-fKL BPC-W
BPC-rKL work similarly does not consider posterior distributions.
This paper re-interprets and extends scalable dataset distillation methods to the problem of matching posterior distribution matching.
Experiments 5.1 Experimental Setup
Datasets and model architecture We use the CIFAR10 dataset [15] to generate Bayesian pseudocoresets, and evaluate on the test split of CIFAR10 in addition to the CIFAR10-C dataset [12], which imposes additional uncertainty through image corruptions.
Following the experimental setup of previous works [36, 34], we use a 3-layer ConvNet as the network architecture.
Evaluation We obtain pseudocoresets from three Bayesian pseudocoreset construction methods: BPC-fKL, BPC-W, and BPC-rKL. We run two Markov chain Monte Carlo methods (HMC [22] and SGHMC [9]) and report the top-1 accuracy and negative log likelihood (NLL) with respect to groundtruth labels.
Note that pseudocoresets are small enough to run full-batch HMC in our experimental setting, and SGHMC which originally subsamples mini-batches from the full data is not a favorable option in our situation.
Instead, at each iteration, while holding the batch (pseudocoreset) fixed, we apply random data augmentation to the batch.
As a result, the gradients computed from the randomly augmented batches differ at each iteration, and we run SGHMC with these stochastic gradients.
To distinguish this setting from the typical mini-batch SGHMC, we denote this as Altered-SGHMC (A-SGHMC).
As our results were not sensitive to the choice of hyperparameters, we used a single set of hyperparameters which performed best in initial experiments.
Please refer to Appendix B for detailed evaluation settings including hyperparameters.
5.2 Main Results
We experimentally evaluate the effectiveness of the three pseudocoreset construction methods previously discussed. We consider three different settings corresponding to different memory budgets for the pseudocoresets: n ∈ { 1 , 10 , 20 } images per class (ipc). In each setting, we consider a random coreset baseline, in which we randomly subsample n images from the original dataset.
To further evaluate the effectiveness of data augmentation in pseudocoreset training, we show performance of each coreset method with and without augmentations.
We additionally examine the role of augmentations at test time through A-SGHMC.
Results in Table 1 show that all Bayesian pseudocoresets have higher accuracy and lower negative log-likelihood compared to random coresets.
BPC-rKL is slightly worse than BPC-W and BPC-fKL, which we interpret as the reverse KL suffering from the property that covers only one major mode, making the BPC-rKL posterior somewhat sub-optimal.
In the 1 ipc (image per class) setting, BPC-W and BPC-fKL have comparable results, and as the pseudocoreset size increases, BPC-fKL is better if there is no augmentation in test time and BPC-W is better if not.
It seems that the strength of BPC-W with higher performances with test time augmentations is due to the training method that exactly matches the pseudocoreset training trajectories and expert training trajectories that also be trained with the data augmentations.
However, because the learning method of A-SGHMC is not accurate Bayesian learning, and it is known that data augmentation causes a cold posterior effect [30], the HMC results without augmentations are preferred.
To qualitatively examine the pseudocoresets trained with each divergence measure, we plot the learned pseudocoreset images in Fig.
We see that BPC-fKL looks the most smooth while BPC-rKL is the most noisy.
The noisiness of the images learned by BPC-rKL may be due to the nature of the reverse KL divergence, which makes it difficult to escape the mode captured by the initial pseudocoreset images.
In contrast, BPC-W and BPC-fKL seem learn better pseudocoreset as desired to include high-level shapes semantic features representative of each class.
As a more quantitative measure of image noisiness, we plot the log amplitude of the diagonal components of the 2D Fourier transform of pseudocoresets in Fig.
2: BPC-rKL has the most high-frequency noise, as reflected in the qualitative examples.
This result is consistent with previous studies showing that CNNs are not robust to high-frequency noise [26, 25].
Please refer to Appendix D for all pseudocoreset images.
5.3 Computational Cost
We show that BPC-fKL, in addition to achieving high performance in terms of accuracy and NLL, requires substantially lower computational cost compared to other pseudocoreset training methods.
We measure the computational costs of each method on CIFAR10 with fixed gradient steps L u = 30 .
We use 32 cores of Intel Xeon CPU Gold 5120 and 4 Tesla V100s.
We note that BPC-W with many pseudocoresets per class cannot be trained on single GPU due the memory limitations.
Therefore, we used a parallel implementation using 4 GPUs in all the methods for a fair comparison.
To compare the training time, we measure the time required for a single iteration by averaging across 100 iterations, and repeat this process three times to select the largest value among them.
Fig.
3a shows that the memory usage is significantly lower for BPC-fKL compared to BPC-W.
This is because unlike BPC-fKL, BPC-W must hold all gradient flows in memory for trajectory matching.
As shown in
[Figure]
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 0.2 0.
A-SGHMC Acc.
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 1.9 2.
A-SGHMC NLL GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 0.2 0.
HMC Acc.
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 1.
HMC NLL Fig. 3b, BPC-fKL also has the advantage of faster training time. Pseudocoreset training time increases roughly linearly with pseudocoreset size, but BPC-W’s rate of increase is much faster than that of
BPC-fKL.
Combining the two results and Table 1, the pseudocoreset training using forward KL divergence is not only comparable in terms of accuracy and NLL but also remarkably efficient in both memory and training time, compared to BPC-W.
5.4 Robustness to Out-of-Distribution Inputs
Since one of the merits of the Bayesian approach is Bayesian model averaging which has been shown to improve robustness to distributional shift and calibration, we evaluate the performance of each pseudocoreset method on an out-of distribution dataset CIFAR10-C [12].
Fig.
4 shows the accuracy and negative log-likelihood of each corrupted dataset for BPC-W and BPC-fKL.
Note that we optimize each pseudocoreset with the clean CIFAR10 dataset and evaluate them on CIFAR10 with 14 corruptions.
For A-SGHMC results, we apply differentiable augmentations to both the original dataset and coreset for training, but we do not use it for running HMC.
As shown in Fig.
4, for both HMC and A-SGHMC, BPC-fKL achieves higher or comparable accuracy and lower negative log-likelihood than the BPC-W for all but one corruption, suggesting that the forward KL divergence is indeed an effective divergence measure for learning Bayesian pseudocoresets.
Conclusion
In this paper, we explored three divergence measures for Bayesian pseudocoresets: reverse KLD,
Wasserstein distance, and forward KLD.
We showed that existing dataset distillation methods can be linked to the Bayesian pseudocoresets with reverse KLD and Wasserstein distance, and further proposed a novel algorithm for learning Bayesian pseudocoresets by minimizing forward KL divergence.
We empirically validated all three methods in terms of their ability to approximate the posterior distribution for real-world image datasets.
Bayesian pseudocoresets with both Wasserstein distance and forward KL divergence can approximate the true posterior well and BPC-fKL is more effective in terms of computational cost and robustness to out-of-distribution data.
Limitation Despite showing promising results for the first time on Bayesian pseudocoresets for real datasets, there still exists a substantial performance gap between stochastig gradient MCMC on a pseudocoreset and the original dataset.
Thus, a promising future direction is to analyze whether such pseudocoresets are useful for stochastic gradient MCMC algorithms when they are used together with mini-batches of the original dataset.
Societal Impacts Our work is hardly likely to bring any negative societal impacts.
Nevertheless, we should be careful while learning pseudocoresets because any bias present in the original dataset can be transferred to the pseudocoresets.
On a positive note, pseudocoresets can alleviate data privacy concerns by eliminating the need for access to the original dataset during downstream task learning. Acknowledgments
This work was partly supported by KAIST-NAVER Hypercreative AI Center, Korea Foundation for Advanced Studies (KFAS), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial
Intelligence Graduate School Program (KAIST), No. 2021-0-02068, Artificial Intelligence Innovation
Hub, No.2022-0-00713), and National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021M3E5D9025030).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include our code in supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5 and Appendix B. (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 5.
We reported standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Proposition A.1. Let q u ( θ ) set as Eq. Assume θ
( t − 1) is close enough to a local optima, so that we have ∥ θ ( t − 1) − θ ( t ) ∥ ≪ 1 . Then we have [Equation] Proof.
For notational simplicity
[Equation]
{+let θ 
+}
= θ ( t − 1) .
We can reparameterize θ ∼ q u as
[Equation]
[Equation]
Assume that η and Σ are chosen such that ∥ η ∇ θ ℓ ( u , θ 0 ) − Σ 1 / 2 ε ∥ ≪ 1 .
Then we have
[Equation]
[Equation]
Similarly,
[Equation]
[Equation]
Note also that
[Equation]
Plugging this into the KL gradient
[Equation]
we get
[Equation]
B Experimental Details
Code is available at https://github.com/balhaekim/BPC-Divergences.
B.1 Hyperparameter settings
Training In Table 2, we enumerate the hyperparameters used for our results in Section 5.
Since we use expert trajectories for all methods to train the Bayesian pseudocoresets, we refer to hyperparameters related to expert trajectories, such as the number of SGD steps or the maximum random starting points, described in [8].
We found that a slightly shorter expert training step is better for BPC-fKL, so we used an expert step 1 epoch shorter than
BPC-W.
Another important hyperparameter for BPC
[Table]
Algorithm
Require:
Number of iteration N , initial sample distribution scale σ θ , initial momentum distribution scale σ r , number of leapfrog step m , step size ε ,
Require: Potential energy function U ( u , θ )
= − 1 ⊤ M f ( u , θ ) + λ
∥ θ ∥ 22 with a dataset u and the weight decay factor λ .
Initialize θ (1) ∼ N (0 , σ 2 θ ) .
for t = 1 , . . .
, N do
Resample momentum r ( t ) ∼ N (0 , σ 2 r ) .
Set ( θ 0 , r 0 )
=
( θ ( t ) , r ( t ) ) , θ ( t +1)
= θ ( t ) .
r 0 ← r 0 − ε 2 ∇ U ( u , θ 0 ) for i = 1 , . . .
, m do θ i ← θ i − 1 + εr i − 1 r i ← r i − 1 − ε ∇ U ( u , θ i ) end for r m ← r m − 1 − ε 2 ∇ U ( u , θ m ) (ˆ θ, ˆ r ) =
( m , r m )
Metropolis-Hastings correction: u ∼ Uniform (0 , 1) ρ =
) −
( t ) )
if u < min(1 , ρ ) then θ ( t +1)
=
ˆ θ end if end for
Evaluation The evaluation methods we used are summarized in Algorithm 2 and Algorithm 3.
We sampled the momentum from a normal distribution with scale σ r only for initialization.
During leapfrog steps, we simulated the Hamiltonian dynamics as if it came from a standard Gaussian.
As mentioned in the main text, the tendency did not significantly change depending on sampling hyperparameters.
Since our focus is on providing a fair comparison between each Bayesian pseudocoreset method rather than raw performance, we used a single set of hyperparameters to generate all results.
We summarize the hyperparmeters used for our evaluations in Table
[Table]
[Table]
[Table]
Require: Number of iteration N , initial sample distribution scale σ θ , initial momentum distribution scale σ r , number of leapfrog step m , step size ε , momentum decay factor α , noise scale T .
Require: Potential energy function U ( u , θ )
= − 1 ⊤ M f ( u , θ ) + λ
∥ θ ∥ 22 with a dataset u and the weight decay factor λ .
Require:
Differentiable augmentation function A used during the pseudocoreset training.
Initialize θ (1) ∼ N (0 , σ 2 θ ) .
Initialize momentum r (1) ∼ N (0 , σ 2 r ) .
for t = 1 , . . .
, N do ( θ 0 , r 0 )
=
( θ ( t ) , r ( t ) ) .
for i = 1 , . . .
, m do θ i ← θ i − 1 + εr i − 1 r i ← (1 − α ) r
i − 1 − ε ∇ U
( A ( u ) , θ i )
+ N (0 , 2 αT ) end for ( θ ( t +1) , r ( t +1) )
=
( θ m , r m ) end for
[Table]
B.2 Implementation details for BPC-rKL
To obtain a Bayesian pseudocoreset with reverse KL divergence by Algorithm 1 in [19]
[Table]
we need to sample from an approximated pseudocoreset posterior at each step through MCMC methods such as Langevin dynamics or HMC.
To simply implement this, we also approximate the pseudocoreset posterior by a Gaussian distribution with mean of the end point of SGD training trajectories like
BPC-W or BPC-fKL.
In initial experiments, we tried using SGD training trajectories starting from an random initial point or a point on the expert trajectory, but we found that using the expert trajectory achieves better performance.
We provide a detailed description of the BPC-rKL algorithm in Algorithm 4.
C Additional Experiments C.1 Extending BPC-W to Gaussians with diagonal covariances
In Section 3.2, we approximated the pseudocoreset posterior and the original posterior to Gaussian distributions with same covariances to obtain
BPC-W.
As an extension, we tried approximating the two distributions using Gaussian distributions with diagonal covariances.
The A-SGHMC results for these pseudocoresets are in Table 4.
We found that the results are comparable, and the additional expressivity of a diagonal covariance did not further increase performance.
It seems to be because both posteriors would be much more complicated to approximate with Gaussians with diagonal covariances.
While using a more complex distribution family might improve performance, we use
Gaussian distributions with the same covariance in all dimensions to obtain BPC-W results throughout this paper.
C.2 Gaussian approximation in BPC-fKL
In this experiment, we investigate the effect of the hyperparameters of the Gaussian approximation to the performance of BPC-fKL.
Firstly, we explore the number of Gaussian samples
S
and variances Σ 1 / 2 u , Σ 1 / 2 x in Eq. 20.
Fig.
5a shows the accuracy of HMC as the function of the number of samples.
Even though the estimation becomes more accurate as the number of samples increases, in Fig.
5a, the number of samples do not significantly improve the performance of pseudocoresets.
Thus, we
Require: Set of expert parameter trajectories { τ } trained with x , each parameter trajectory saves parameters at the end of training epochs.
Require: Number of updates with the pseudocoreset L u , total training steps K , maximum start epoch T + , number of Gaussian samples S , variance Σ u , inner SGD learning rate η , minibatch size B , pseudocoresets learning rate γ .
Require:
Differentiable augmentation function A (Optional).
Initialize the pseudocoreset u by randomly selecting a subset of size M from x . for k = 1 , . .
K do
Sample an expert trajectory τ = { θ ( r ) ∗ } Tr =0 .
Randomly choose an epoch to start r ≤ T + and initialize θ (0) u
= θ ( r ) ∗ .
for t = 1 , . . .
, L u do
Update the network parameter θ ( t ) u ← θ ( t − 1) u + η ∇ 1 ⊤ M f ( A ( u ) , θ ( t − 1) u ) .
end for
Sample random Gaussian noises { ε ( s ) u } Ss =1 i .
i .
d . ∼ N (0 , I ) .
Obtain a minibatch of B datapoints from the original dataset { x 1 , . . .
, x B } ⊂ x . for s = 1 , . . .
, S do g s ← (cid:16) f
( A ( x b ) , θ ( L u ) u
+
[Equation]
˜ g s ← (cid:16) f
( A ( u m ) , θ
[Equation]
L u ) u
[Equation]
for m = 1 , . . .
, M do ˜ h m
[Figure]
end for end for for m = 1 , . . .
, M do ˆ
[Figure]
end for for m = 1 , . . .
, M do u
[Figure]
← u
[Figure]
−
[Figure]
end for end for
[Figure]
use 30 samples for all the experiments.
Fig.
5b, we show the accuracy with varying the variances.
The values of x-axis are Σ 1 / 2 x ’s and Σ 1 / 2 u ’s are presented as colors.
As the graph shows, too small variances are not much different from using variance of 0, and when the both values are 0.01 is the best and performance drops again for the variances larger than that.
So we used both Σ 1 / 2 x and Σ 1 / 2 u of 0.01.
C.3 Additional results on CIFAR
Table 5 shows additional results for the CIFAR10 dataset when the pseudocoreset size is larger.
Even in these cases, BPC-W and BPC-fKL effectively generate Bayesian pseudocoresets.
Moreover, compared to Table 1, the 10-ipc pseudocoreset trained with BPC-fKL outperforms the 100-ipc random
[Table]
[Table]
coreset which has 10 times more images.
The overall BPC-fKL results demonstrate that the forward
KL divergence is effective for constructing the Bayesian pseudocoresets.
For evaluations, we used same hyperparameters as the case of ipc 20 as described in Table 2 and Table 3, except that σ r is 0 .
01 , ε is 0 .
02 and λ is 0 .
Other coreset baselines and evaluation metrics
We compared our results with other coreset baselines and other evaluation metrics for validating the quality of obtained posterior distributions more rigorously.
We added Herding [29] and K-center [31] as another coreset baslines and for other evaluation metrics, we used expected calibration error (ECE) [21] and Brier score [2].
Herding constructs coresets by gathering samples close to the centers of the feature representations for each class and K-center constructs coresets by selecting multiple center points such that the distance between each data point is maximized while the distance between centers is minimized.
To obtain the feature representations for both methods, we use a pre-trained ConvNet as in [13].
On the other hand, the ECE and Brier scores are conventional metrics for evaluating posterior qualities.
Tableshows the HMC results of various metrics for coresets and psuedocoresets with 10 ipc.
As [13] already shows that the coreset baselines underperform pseudocoresets for SGD, Table 6 also shows that pseudocoresets are better for Bayesian inference tasks through various metrics.
C.4 Additional results on other datasets
In the main text, we trained Bayesian pseudocoresets only on the CIFAR10 dataset.
To validate how well each method works on different datasets, we trained the pseudocoresets with a size of 1 image per class on other datasets, CIFAR100 and ImageNet.
We can see that how well each method works when the number of classes is large with the CIFAR100 dataset and when the data dimension is large with the ImageNet dataset.
The CIFAR100 dataset has the same image dimension as CIFAR10 but has 100 classes and ImageNet has 128 × 128 data dimension.
For ImageNet, we use the same existing subset of the entire dataset, ImageNette, which consists of 10 classes and an increased network architecture.
Following previous work [8], we use a depth-5 ConvNet as the model architecture.
As in the results on CIFAR10, Table 7 shows all three pseudocoresets are better than a random coreset.
Moreover, BPC-W and BPC-fKL outperform BPC-rKL, demonstrating that the proposed divergence measures are effective.
[Table]
C.5 Additional results on the synthetic dataset
To validate if the posterior distribution each algorithm learns as intended
[Table]
we trained pseudocoresets on the synthetic dataset of samples from 10-dimensional multivariate Gaussian distribution, whose posterior distribution is tractable.
Given data samples { x 1 , x 2 , ..., x 100 } i .
i .
d . ∼ N ( θ, Σ) , we trained pseudocoresets { u m } Mm =1 of sizes M = { 5 , 20 , 40 , 60 , 80 , 100 } to have similar posterior distribution of θ ∼ N ( θ 0 , Σ 0 ) with the true posterior which is tractable in this setting.
Since we know that the exact posterior distribution is given by N ( θ u , Σ u ) where
[Equation]
[Equation]
we validate each method by directly calculating the divergence measures between pseudocoresets posteriors and true posterior.
Fig.
6 shows that all three methods work well in that all divergences are well reduced even when trained with the algorithm with different divergence measure in this simple synthetic setting.
Also, as expected, Fig.
7 shows divergences decrease as pseudocoreset sizes increase.
D Example images of each Bayesian pseudocoreset
Fig.
8 are the example images of CIFAR10 pseudocoresets of 10 images per class.
[Figure]