On Divergence Measures for Bayesian Pseudocoresets
Anonymous Author(s)
AffiliationAddress email
Abstract
A Bayesian pseudocoreset is a small synthetic dataset for which the posteriorover parameters approximates that of the original dataset.
While promising, thescalability of Bayesian pseudocoresets is not yet validated in realistic problemssuch as image classification with deep neural networks.
On the other hand, datasetdistillation methods similarly construct a small dataset such that the optimizationusing the synthetic dataset converges to a solution with performance competitivewith optimization using full data.
Although dataset distillation has been empiricallyverified in large-scale settings, the framework is restricted to point estimates, andtheir adaptation to Bayesian inference has not been explored.
This paper caststwo representative dataset distillation algorithms as approximations to methods forconstructing pseudocoresets by minimizing specific divergence measures: reverse
KL divergence and Wasserstein distance.
Furthermore, we provide a unifying viewof such divergence measures in Bayesian pseudocoreset construction.
Finally, wepropose a novel Bayesian pseudocoreset algorithm based on minimizing forward
KL divergence.
Our empirical results demonstrate that the pseudocoresets constructed from these methods reflect the true posterior even in high-dimensional
Bayesian inference problems.
Introduction
One of the main ingredients for modern statistical machine learning is large-scale data, whichenable the learning of powerful and flexible models such as deep neural networks when handledcorrectly [3, 32].
However, in general, the larger the data are, the harder to learn with them [33].
Asidefrom requiring enormous computing resources for training, large-scale data often contain sensitiveinformation that must not be shared publicly [11].
This motivates the construction of coresets [29, 14],a sparse subset of a large-scale dataset summarizing essential features of the original dataset.
In thecontext of Bayesian inference, the posterior conditioned on an essential coreset should be similar tothe exact posterior conditioned on the full dataset.
Coreset construction becomes more difficult as data dimensionality increases.
Manousakas et al.[19] show that even the best coreset becomes suboptimal at large scales: the divergence betweenthe posterior conditioned on the optimal coreset and the exact posterior grows with the dimensionof the data.
To this end, Manousakas et al.
[19] propose to construct a Bayesian pseudocoreset , asynthetic dataset represented as a set of learnable parameters and trained to minimize the divergencebetween coreset posteriors and full-data posteriors.
Compared to coresets, pseudocoresets scale muchbetter with data dimension and achieve significantly lower posterior approximation error.
However,the KL divergence minimization in pseudocoreset learning requires the construction of approximateposteriors on the fly during the learning process, which limits its tractability for high-dimensionalmodels such as deep neural networks.
On the other hand, dataset distillation [28] aims to distill a large-scale dataset into a small syntheticdataset such that the test performance of a model trained on the distilled data is comparable to that of a model trained on the original data.
Even though dataset distillation methods have a similar motivationto pseudocoresets, the two methods optimize different objectives.
Whereas pseudocoresets minimizethe divergence between coreset and full-data posteriors, dataset distillation considers a non-Bayesiansetting with heuristically set objective functions, such as matching the gradients of loss functions [36]or matching the parameters obtained from optimization trajectories [8].
Dataset distillation methodshave achieved remarkable performance, showing premises on real-world problems involving highdimensional data and models such as image classification with deep neural networks.
However, sincethe datasets are distilled with heuristical objectives, we have no ground for using them in Bayesianinference.
In this paper, we provide a unifying view of Bayesian pseudocoresets and dataset distillation to takeadvantages of both approaches.
We first study various choicses of divergence measures being optimized in pseudocoreset learning.
While the existing work [19] minimizes the reverse KL divergencebetween the pseudocoreset posterior distribution and the full data posterior distribution, we showthat alternative divergence measures such as forward KL divergence and Wasserstein distance canalso be employed effectively.
Based on this perspective, we show that the existing dataset distillationalgorithms [36, 8] can be identified as approximations of the Bayesian pseudocoresets learned tominimize reverse KL and Wasserstein distance with specific choices of variational approximations forcoreset posteriors.
This connection justifies the heuristically chosen learning objectives of the datasetdistillation algorithms, and at the same time, provides a theoretical background for using the distilleddatasets obtained from such procedure for Bayesian inference.
Conversely, Bayesian pseudocoresetscan also benefit from this connection by borrowing various ideas and tricks used in the datasetdistillation algorithms to make them scale for complex tasks.
For instance, the variational posteriorswe identified by recasting dataset distillation algorithms as Bayesian pseudocoreset learning can beused for Bayesian pseudocoresets with various choices of divergence measures.
Also, we can adaptthe idea of reusing pre-computed optimization trajectories [8] which have already been shown towork at large scales.
We empirically compare the pseudocoreset algorithms based on various divergence measures onhigh-dimensional image classification with deep neural networks.
Our results demonstrate thatwe can efficiently and scalably construct Bayesian pseudocoresets with which MCMC algorithmssuch as Hamiltonian Monte-Carlo (HMC) [10, 22] or Stochastic Gradient Hamiltonian Monte-Carlo(SGHMC) [9] accurately approximate the full posterior.
Backgrounds2.1 Bayesian pseudocoresets
Let x = { x 1 , . . .
, x N } Nn =1 be observed data.
Given a probabilistic model index by a parameter θwith some prior distribution π 0 , we are interested in the posterior conditioned on the full data x ,
[Equation] [Equation]
where f ( x , θ )
:=
[ f ( x 1 , θ ) , . .
f ( x N , θ )] ⊤ , 1 N is the N -dimensional one vector, and Z ( x ) = 74 (cid:82) exp( 1 ⊤ n f ( x , θ ))
d θ is a partition function.
The posterior π x is usually intractable due to Z ( x ) , sowe often rely on approximate inference algorithms.
Bayesian pseudocoreset methods [19] aim toconstruct a synthetic dataset called a pseudocoreset u = { u m } Mm =1 with M ≪ N such that theposterior of θ conditioned on it approximates the original posterior π x .
The pseudocoreset posterioris written as 1 ,
[Equation]
where f ( u , θ ) , 1 M , and Z ( u ) are defined similarly.
Manousakas et al.
[19] suggests to learn u byminimizing the KL divergence between π u and π x :
[Equation]
and show that the gradient of this objective is computed as
[Equation]
The expectation over the pseudocoreset posterior π u in the above equation is further approximatedwith the Monte-Carlo estimation with an approximate posterior q u ≈ π u , for instance, from the
Laplace approximation.
2.2 Dataset Distillation
With a similar motivation as Bayesian pseudocoresets, dataset distillation methods construct a smallsynthetic dataset u , but from the perspective of matching optimal parameters.
That is, in datasetdistillation, we find u such that
[Equation]
with a loss function ℓ ( · , θ ) and a distance function d ( · , · ) comparing two parameters.
In general,optimizing d ( θ ∗ x , θ ∗ u ) is intractable since it is a bilevel optimization problem requiring θ ⋆ x and θ ∗ u , soexisting works relax the problem in several ways, such as kernel-based approaches [23, 24] or solvingonly for a short horizon in inner optimization [36, 34, 8].
Below, we briefely review two representativemethods — Dataset condensation (DC) [36] and Matching Training Trajectories (MTT) [8].
Dataset Condensation Instead of fully unrolling the inner optimization process, DC insteadcomputes the matching objective at each inner step, starting from some random parameter θ (
[Equation] [Equation]
where d cos denotes the cosine similarity loss between two gradients.
To implement this objective, webuild a trajectory of parameters starting from a random θ (0) with the current distilled dataset u .
Wethen compute the gradient matching objective for each layer at each step of the trajectory to update u .
Despite its appealing simplicity, this algorithm may suffer from short-horizon bias due to the limitedtrajectory length.
Matching Training Trajectories MTT extends the single step gradient matching in DC andconsiders longer inner optimization trajectories.
Starting from a randomly initialized parameter θ (0) ,
MTT takes multiple inner optimization steps with both u and x , and minimize the normalized L 2distance between two parameters at the end of the learning trajectories.
[Equation]
where d 2 ( · , · ) is the normalized L 2 distance between parameters.
Unlike DC, this algorithm startsfrom a common initial parameter θ (0) , keeps track of two separate optimization trajectories (one withu and another with x ), and the outer-update for u is done only after L x and L u inner steps.
To reducethe heavy computational cost for the inner update with the full dataset x , MTT employs mini-batch
SGD for the inner steps, in addition to saving and re-using precomputed SGD trajectories to trainu .
The use of precomputed SGD trajectories is similar to that of replay buffers for reinforcementlearning [17, 20, 1], and allows MTT to handle long inner-loop sequences.
While MTT significantlyimproves over DC, presumably due to its longer inner optimization horizon, the algorithm requireslarge memory for even a moderate number of inner-steps L u because it requires backpropagationthrough all inner steps with u .
On Divergence Measures for Bayesian Pseudocoresets
While the Bayesian pseudocoreset is originally obtained by minimizing the reverse KL divergence, inprinciple we can minimize an arbitrary divergence measure D ( · , · ) to achieve the goal:
[Equation]
In this section, we study the several choices for such divergence measures.
3.1 Reverse KL divergence
Manousakas et al.
[19] proposed the reverse KL divergence minimization with a Laplace approximation for the pseudocoreset posterior π u .
Here, we show that it is possible to recast DC as a reverse
KL divergence minimization with an alternative choice for the variational posterior.
Let θ ( t ) be aparameter in a learning trajectory with a loss function ℓ ( u , θ ) :
= − 1 ⊤ M f ( u , θ ) .
Provided that θ ( t ) issufficiently close to a local optimum, we can construct a naïve approximation of π u as,
[Equation]
Under this setting, we can actually show that a gradient descent for the reverse KL divergence
D KL [ π u ∥ π x ] is equivalent to a single step of gradient matching in DC when the parameter beingoptimized is close to a local optima.
We note that the Gaussian posterior is simple but commonlyused and its usefulness has been demonstrated, especially in several literatures [18], such as the
Bernstein-von Mises theorem [16, 27].
However, we also note that these Gaussian approximationsare necessary for the relation between the dataset distillation and Bayesian pseudocoresets; Gaussianapproximations are just special cases, with the benefit of simplifying the analysis.
Proposition 3.1.
Let q u ( θ ) set as Eq.
Assume θ
( t − 1) is close enough to a local optima, so thatwe have ∥ θ ( t − 1) − θ ( t ) ∥ ≪ 1 .
Then we have
[Equation] [Equation]
The proof is given in Appendix A.
In other words, when sufficiently close to a local optima, DC canbe interpreted as a special instance of Bayesian pseudocoreset via reverse KL minimization with Gaussian approximation for the pseudocoreset posterior π u .
Throughout the paper, we will refer tothe Bayesian pseudocoreset with reverse KL minimization as BPC-rKL.
To simply implement the Algorithm 1 in [19] for the image dataset, we approximated π u as a Gaussian distribution.
Pleaserefer to the appendix for the detailed algorithm.
3.2 Wasserstein distance
We can instead choose D ( · , · ) to be the Wasserstein distance between π u and π x .
The Wassersteindistance is generally intractable, but as we did earlier, if we approximate both π u and π x with
Gaussians, it can be a closed-form expression.
Specifically, let θ u
and θ x be two parameterssufficiently close to local optima.
Then we can construct a crude approximation for π u and π x as,
[Equation]
Then the Wasserstein distance between these two distributions are computed as
[Equation]
Now consider a single outer-step of the MTT.
Starting from a common initial θ 0 , the algorithm takesthe gradient steps both with x and u to get θ ( L x ) x
and θ ( L u ) u .
If we set q u ( θ ) =
N ( θ ; θ ( L u ) u , Σ) andq x ( θ ) =
N ( θ ; θ ( L x ) x , Σ) with a common covariance Σ , then the Wasserstein distance reduces to∥ θ ( L x ) x − θ ( L u ) u ∥ 22 , which is proportional to the L 2 objective being optimized in the MTT.
Similar tothe reverse KL, we refer to the Bayesian pseudocoreset with Wasserstein distance minimization as
BPC
3.3 Forward KL divergence
Finally, we consider an alternative Bayesian pseudocoreset algorithm based on the forward KLdivergence.
The forward KL minimization is known to encourage a model to cover the entire targetdistribution while the reverse KL minimization favors mode capturing models, indicating that theforward KL may be a better choice for the Bayesian pseudocoreset.
Under our setting, the forward
KL is computed as
[Equation] [Equation]
Then the gradient w.r.t.
the pseudocoreset u is computed as,
[Equation]
Again, we should approximate the expectation w.r.t.
π u and π x to obtain this gradient.
We introducesimilar variational posteriors,
[Equation]
The endpoint θ ( L u ) u is a function of u , so in principle, the expectation w.r.t.
q u involves unrollingthrough the parameter trajectories θ (0)
→ θ (1) u
→ · · · → θ ( L u ) u , as for the BPC-W.
To reduce memorycost due to this unrolling, we can simply block the gradient flow through θ ( L u ) to obtain
[Equation]
where sg ( · ) denotes the stop-grad operation.
We further approximate this via Monte-Carlo estimation,
[Equation]
with ε (1) x , . . .
, ε ( S ) x , ε (1) u , . .
. , ε ( S ) u i .
i .
d . ∼ N (0 , I ) .
The resulting algorithm starts from a common initialparameter θ (0) , keeps track of two parameter trajectories, and minimizes the difference in the loglikelihood 1 ⊤ M f ( u , θ ) evaluated at the Gaussian neighbors of the endpoints of the two trajectories.
Note that even if we have blocked the gradient flow through θ ( L u ) u , we still have the pseudocoreset uin the likelihood terms f ( u , θ ) , so the gradient descent gets meaningful learning signals.
On the otherhand, if we apply a similar trick to the BPC-W, the matching objective would be ∥ sg ( θ ( L u ) u ) − θ ( L x ) x ∥ ,which is just a constant w.r.t.
u .
Here, we can also borrow the idea from MTT, where we utilizethe set of expert trajectories to efficiently evaluate the parameter trajectory with x .
The resultingalgorithm which we call as BPC-fKL is summarized in Algorithm 1.
Relation to contrastive divergence BPC-fKL is closely related to contrastive divergence [13, 7],a learning algorithm to approximate maximum likelihood estimator.
Suppose we want to train amodel p θ ( x ) = 1 Z ( θ ) exp
( f θ ( x )) by maximizing the log-likelihood function of training examples{ x
[Equation]
[Equation]
Then the gradient of the log-likelihood is
[Equation] [Equation]
+ η ∇ 1 ⊤ M f A θ t − The contrastive divergence approximates this gradient by approximating the second term withsampling methods such as Langevin dynamics or HMC.
By doing this we can obtain the MLE forlog-likelihood without knowing the exact potential function.
Our proposed BPC-fKL performs thesame function as the contrastive divergence if the model parameter θ and training data x changeto the Bayesian pseudocoresets u and the parameters from the true posterior distribution whichare approximated by the points near the expert trajectories.
In this point of view, BPC-fKL is themaximum likelihood estimator that maximizes the log-likelihood of the parameters from the posteriordistribution of the original dataset.
Related works
Bayesian coresets As running algorithms such as MCMC and VI on large datasets is challengingdue to expensive computational cost, Bayesian coresets [14], methods that represent the entire datasetas a sparse and weighted sum of subsets, have been studied.
Campbell and Broderick [6] interpret thecoreset construction as a sparse approximation of vector sums and generalize it as a sparse regressionproblem in a Hilbert space.
Campbell and Broderick [5] demonstrates that Hilbert coresets scale thecoresets log-likelihood sub-optimally, and presents a modified algorithm.
As Hilbert coresets includethe choice of weighted L 2 inner product or finite-dimensional projection and it is difficult to choosethem optimally, another method that minimizes the KL divergence between the coreset posterior andtrue posterior has also been studied recently [4].
Bayesian coresets have the advantages of simpleand theoretically guaranteed quality of the posterior approximations.
However, as the dimensionsof the datasets increase, the KL divergences of the coresets consisting only of their subsets increasetogether, and there is also a problem of privacy, so it is difficult to apply them to the real-worldconditions.
Therefore, there is also a Bayesian pseudocoreset [19] consisting of synthetic data, but ithas been demonstrated experimentally only in relatively easy settings such as logistic regressions,and Bayesian coresets or pseudocoresets for large-scale and high-dimensional real data have not yetbeen studied.
Dataset distillation The goal of dataset distillation [28] is to create a small synthetic dataset thatallows the model to have similar test performance to the original large-scale dataset even if it istrained with that small synthetic dataset.
These synthetic datasets can be applicable to variousfields of machine learning, such as continual learning and neural architecture search.
However,dataset distillation typically involves a bilevel optimization structure in the learning process, whichsuffers from high computational costs or training instability.
So there are existing methods such asmatching one-step gradients between synthetic and real data [36, 34], solving using a closed formsolution of kernel ridge regression [23, 24], or reducing the normalized distance between parametersat the end of synthetic and real training trajectories [8].
Meanwhile, these methods focus only onhigh test accuracy, where uncertainty is not considered and do not include how much the posteriordistribution on the synthetic dataset matches the true posterior distribution.
Zhao and Bilen [35]match the distributions of embedding features for many random networks of synthetic and real data,
[Table]
[Figure] [Figure]
1.0 Frequency 0.
L o g a m p li t u d e
BPC-fKL BPC-W
BPC-rKL not posterior distributions. On the other hand, in this paper, we extend dataset distillation to theproblem of posterior distribution matching.
Experiments
5.1 Experimental setup
Datasets and models We use the CIFAR10 dataset [15] to generate Bayesian pseudocoresets, andevaluate on the test split of CIFAR10 in addition to the CIFAR10-C dataset [12], which imposesadditional uncertainty through image corruptions.
Following the experimental setup of previousworks [36, 34], we use a 3-layer ConvNet as the network architecture.
Evaluation We run HMC [22] and SGHMC [9] with the coresets obtained from Bayesian pseudocoreset construction methods, and evaluated the top-1 accuracy, negative log-likelihood (NLL) withrespect to the ground-truth labels.
Note that the coresets are small enough to run full-batch HMC,and SGHMC which originally subsamples mini-batches from the full data is not a favorable option inour situation.
Instead, at each iteration, while holding the batch (coreset) fixed, we apply randomdata augmentation for the batch.
As a result, the gradients computed from the randomly augmentedbatches are stochastic, and we run SGHMC with them.
To distinguish this setting from the typicalmini-batch SGHMC, we call what we run as Altered-SGHMC (A-SGHMC).
As our results were notsensitive to the choice of hyperparameters, we used a single set of hyperparameters which performedbest in the initial set of experiments.
Please refer to the appendix for detailed evaluation settingsincluding hyperparameters.
5.2 Main results
To validate each pseudocoreset method in various settings, we depict the HMC and A-SGHMC resultson CIFAR10 dataset in Table 1.
To evaluate the effectiveness of data augmentations in pseudocoresetstraining, we trained each pseudocoreset both with augmentations and without augmentations.
Also,through the A-SGHMC, we wanted to see the effectiveness of augmentation in test time.
Each rowrepresents the results for different pseudocoreset sizes, which are 1, 10 and 20 images per class.
Table 1 shows that all Bayesian pseudocoresets have higher accuracy and lower negative log-likelihoodthan those of the random coresets.
BPC-rKL is slightly worse than BPC-W and BPC-fKL, which weinterpret as the reverse KL typically suffers from the property that covers only one major mode sothat BPC-rKL posterior is somewhat sub-optimal.
In 1 image per class setting, BPC-W and BPC-fKLhave comparable results, and as the pseudocoreset size increases, BPC-fKL is better if there is noaugmentation in test time and BPC-W is better if not.
It seems that the strength of BPC-W with higherperformances with test time augmentations is due to the training method that exactly matches thepseudocoreset training trajectories and expert training trajectories that also be trained with the dataaugmentations.
However, the learning method of A-SGHMC is not an accurate Bayesian learning,and it is known that data augmentation causes cold posterior effect [30], the HMC results withoutaugmentations are preferred.
Fig.
1 shows some examples of Bayesian pseudocoresets trained with each method.
We can seethat BPC-fKL looks the most smooth and BPC-rKL is the most noisy.
To more quantitatively,we plot the log amplitude according to the diagonal components of each 2D Fourier transformedpseudocoresets.
Fig.
2 shows that BPC-rKL has the most high frequency noise which might disturbthe stable training.
This result is consistent with previous studies that CNNs are vulnerable to highfrequency noises [26, 25].
This may be also due to the reverse KL divergence that makes it difficultto escape the mode created by the initialized images and hard to cover the distribution of the entiredataset.
On the other hand, BPC-W and BPC-fKL seem to be learned as desired to include variousshapes of each class.
For the entire images of each pseudocoreset, please refer to the appendix.
5.3 Computational cost
In this section, we note that BPC-fKL significantly reduces the computational cost required forpseudocoreset training.
We evaluate computational cost for each pseudocoreset method on CIFAR10.
The computational cost is measured with fixed gradient steps with u , L u , of 30, on 32 cores of Intel
Xeon CPU Gold 5120 and 4 Tesla V100s.
For BPC-W, when the number of pseudocoresets perclass becomes large, it cannot be trained on single GPU due to the memory limits.
Therefore, weperformed parallel computations using 4 GPUs in all the methods for a fair comparison.
To comparethe training time, we measure a single iteration time averaged by 100 iterations, and repeat thisprocess three times to select the largest value among them.
Fig.
3a shows that the memory usagerequired for BPC-fKL compared to BPC-W is significantly reduced.
This is because BPC-W musthold all gradient flows in memory for trajectory matching, although BPC-fKL does not need to.
Asshown in Fig.
3b, BPC-fKL also has the advantage of training time.
The pseudocoreset training timeincreases linearly as the size of pseudocoresets increases, but the increasing rate of BPC-W is muchfaster than that of BPC-fKL.
Combining the two results and Table 1, the pseudocoreset training using
[Figure]
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 0.2 0.
A-SGHMC Acc.
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 1.9 2.
A-SGHMC NLL GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 0.2 0.
HMC Acc.
GN
SN
IN
DB
GB
MB
ZM SW FT
BS
ET
PIX
JPEG
SP 1.
HMC NLL forward KL divergence is not only comparable in terms of accuracy and NLL but also remarkablyefficient in both memory and training time, compared to BPC-W.
5.4 Out-of distribution dataset
Since one of the merits of Bayesian approach is Bayesian model averaging which enables robustnessto distributional shift and better calibration, we evaluate the performance of each pseudocoresetmethod on out-of distribution dataset, CIFAR10-C [12].
Fig.
4 shows the accuracy and negativelog-likelihood of each corrupted datasets for BPC-W and BPC-fKL.
Note that we optimize eachpseudocoreset with the clean CIFAR10 dataset and evaluate them on CIFAR10 with 14 corruptions.
For A-SGHMC results, we apply differentiable augmentations to both original dataset and coreset fortraining, but we do not use it for running HMC.
As shown in Fig.
4, for both HMC and A-SGHMC,
BPC-fKL achieves higher or comparable accuracy and lower negative log-likelihood than the BPC-
Wfor all the corruptions except for the frost corruption.
It suggests that the forward KL divergence isindeed effective divergence measure for Bayesian pseudocoreset in this experimental setup.
Conclusion
In this paper, we explored three divergence measures for Bayesian pseudocoresets: reverse KLD,
Wasserstein distance, and forward KLD.
We showed that existing dataset distillation methods canbe linked to the Bayesian pseudocoresets with reverse KLD and Wasserstein distance, and furtherproposed a novel algorithm for learning Bayesian pseudocoresets by minimizing forward KL divergence.
We empirically validated all three methods in terms of their ability to approximate theposterior distribution for real-world image datasets.
Bayesian pseudocoresets with both Wassersteindistance and forward KL divergence can approximate the true posterior well and BPC-fKL is moreeffective in terms of computational costs and robustness to the out-of distribution dataset.
Limitation Despite showing promising results for the first time on Bayesian pseudocoresets forreal datasets, the performance still underperforms stochastic gradient MCMC with entire dataset.
Thus, a promising future direction is to analyze whether such pseudocoresets are useful for stochasticgradient MCMC algorithms when they are used together with mini-batches of the original dataset.
Societal Impacts Our work is hardly likely to bring any negative societal impacts.
However, if theoriginal dataset has bias, it may be transferred to the pseudocoresets and should be used with care.
On the positive side, pseudocoresets can be utilized for downstream tasks without the original dataset,helping to increase data privacy.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 6.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix A.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include ourcode in supplementary material.(b)
Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5 and
Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 5.
We reported standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proofs
Proposition A.1.
Let q u ( θ ) set as Eq.
Assume θ
( t − 1) is close enough to a local optima, so thatwe have ∥ θ ( t − 1) − θ ( t ) ∥ ≪ 1 .
Then we have
[Equation]
Proof.
{+For notational simplicity, let θ 
+}
= θ ( t − 1) and θ = θ ( t ) .
We can reparameterize θ ∼ q u as
[Equation]
Assume that η and Σ are chosen such that ∥ η ∇ θ ℓ ( u , θ 0 ) − Σ 1 / 2 ε ∥ ≪ 1 .
Then we have
[Equation]
Similarly,
[Equation]
[Equation]
Note also that
[Equation]
Plugging this into the KL gradient, we get
[Equation]
B Experimental Details
Code is available at https://github.com/NeurIPS2022anonymous/bayesianpseudocoresets.
B.1 Hyperparameter settings
Training In Table 2, we enumerate the hyperparameters used for our results in Section 5.
Since weuse expert trajectories for all methods to train the Bayesian pseudocoresets, we refer to hyperparameters related to expert trajectories, such as the number of SGD steps or the maximum random startingpoints, described in [8].
We found that a slightly shorter expert training step is better for BPC-fKL, sowe used an expert step 1 epoch shorter than
BPC-W.
Another important hyperparameter for BPC-fKLis the inner SGD learning rate η .
For each setting, used the best learning rate from a hyperparametersweep over { 0 . 01 , 0 . 02 , 0 .
All other hyperparameters are same for all methods.
[Table]
Algorithm
Require:
Number of iteration N , initial sample distribution scale σ θ , initial momentum distribution scale σ r , number of leapfrog step m , step size ε ,
Require: Potential energy function U ( u , θ )
= − 1 ⊤ M f ( u , θ ) + λ
∥ θ ∥ 22 with a dataset u and the weight decay factor λ .
Initialize θ (1) ∼ N (0 , σ 2 θ ) .
for t = 1 , . . .
, N do
Resample momentum r ( t ) ∼ N (0 , σ 2 r ) .
( θ 0 , r 0 )
=
( θ ( t ) , r ( t ) ) .
r 0 ← r 0 − ε 2 ∇ U ( u , θ 0 ) for i = 1 , . . .
, m do
[Table]
θ
[Table]
+ εr i
[Table]
r i ← r i
[Table]
− ε ∇ U ( u , θ i ) end for r
Metropolis-Hastings correction: end if end for
Evaluation The evaluation methods we used are summarized in Algorithm 2 and Algorithm 3.
We sampled the momentum from a normal distribution with scale σ r only for initialization.
Duringleapfrog steps, we simulated the Hamiltonian dynamics as if it came from a standard Gaussian.
Asmentioned in the main text, the tendency did not significantly change depending on sampling hyperparameters.
Since our focus is on providing a fair comparison between each Bayesian pseudocoresetmethod rather than raw performance, we used a single set of hyperparameters to generate all results.
We summarize the hyperparmeters used for our evaluations in Table 3.
[Table]
[Table]
Require: Number of iteration N , initial sample distribution scale σ θ , initial momentum distribution scale σ r , number of leapfrog step m , step size ε , momentum decay factor α , noise scale T .
Require: Potential energy function U ( u , θ )
= − 1 ⊤ M f ( u , θ ) + λ
∥ θ ∥ 22 with a dataset u and the weight decay factor λ .
Require:
Differentiable augmentation function A used during the pseudocoreset training.
Initialize θ (1) ∼ N (0 , σ 2 θ ) .
Initialize momentum r (1) ∼ N (0 , σ 2 r ) .
for t = 1 , . . .
, N do ( θ 0 , r 0 )
=
( θ ( t ) , r ( t ) ) .
for i = 1 , . . .
, m do θ i ← θ i − 1 + εr i − 1 r i ← (1 − α ) r
i − 1 − ε ∇ U
( A ( u ) , θ i )
+ N (0 , 2 αT ) end for ( θ ( t +1) , r ( t +1) )
=
( θ m , r m ) end for
[Table]
B.2 Implementation details for BPC-rKL
To obtain a Bayesian pseudocoreset with reverse KL divergence by Algorithm 1 in [19]
[Table]
we need tosample from an approximated pseudocoreset posterior at each step through MCMC methods suchas Langevin dynamics or HMC.
To simply implement this, we also approximate the pseudocoresetposterior by a Gaussian distribution with mean of the end point of SGD training trajectories like
BPC-W or BPC-fKL.
In initial experiments, we tried using SGD training trajectories starting froman random initial point or a point on the expert trajectory, but we found that using the experttrajectory achieves better performance.
We provide a detailed description of the BPC-rKL algorithmin Algorithm 4.
C Additional ExperimentsC.1 Extending BPC-W to Gaussians with diagonal covariances
In Section 3, we approximated the pseudocoreset posterior and the original posterior to Gaussiandistributions with same covariances.
As an extension, we tried approximating the two distributionsusing Gaussian distributions with diagonal covariances.
The A-SHGMC results for these pseudocoresets are in Table 4.
We found that the results are comparable, and the additional expressivity ofa diagonal covariance did not further increase performance.
It seems to be because both posteriorswould be much more complicated to approximate with Gaussians with diagonal covariances.
Whileusing a more complex distribution family might improve performance, we use Gaussian distributionswith the same covariance in all dimensions to obtain BPC-W results throughout this paper.
C.2 Gaussian approximation in BPC-fKL
In this experiment, we investigate the effect of the hyperparameter of the Gaussian approximation tothe performance of BPC-fKL.
Firstly, we explore the number of Gaussian samples S and variancesΣ 1 / 2 u , Σ 1 / 2 x in Eq.
As shown in Fig.
5a, we plot the accuracy of HMC as the function of the numberof samples.
Even though the estimation becomes more accurate as the number of samples increases,in Fig.
5a, the number of samples do not significantly improve the performance of pseudocoresets.
Thus, we use 30 samples for all the experiments.
Fig.
5b, we show the accuracy with varying the
Require: Set of expert parameter trajectories { τ } trained with x , each parameter trajectory saves parameters at the end training epochs.
Require: updates training steps maximum start epoch number samples variance learning rate learning rate γ .
Require:
Differentiable augmentation function A (Optional).
Initialize the pseudocoreset u by randomly selecting a subset of size M from x . for k = 1 , . .
K do
Sample an expert trajectory τ = { θ ( r ) ∗ } Tr =0 .
Randomly choose an epoch to start r ≤ T + and initialize θ (0) u
= θ (0) x
= θ ( r ) ∗ .
for t = 1 , . . .
, L u do
Update the network parameter θ ( t ) u ← θ ( t − 1) u + η ∇ 1 ⊤ M f ( A ( u ) , θ ( t − 1) u ) .
end for
Sample random Gaussian noises { ε ( s ) u } Ss =1 i .
i .
d . ∼ N (0 , I ) .
Obtain a minibatch of B datapoints from the original dataset { x 1 , . . .
, x B } ⊂ x . for s = 1 , . . .
, S do g s ← (cid:16) f
( A ( x b ) , θ ( L u ) u
[Equation]
˜ g s ← (cid:16) f
( A ( u m ) , θ ( L u ) u
[Equation]
for m = 1 , . . .
, M do ˜ h m
[Figure]
s ← ∇ u f ( A ( u m ) , θ ( L u )
[Figure]
Σ
[Figure]
( s
[Figure]
(cid:80)
Ss
[Figure]
∇
[Figure]
f ( A ( u m )
[Figure]
θ ( L u
[Figure]
+ Σ
[Figure]
ε ( s ′ )
u
[Figure]
end for end for for m = 1 , . . .
, M do ˆ ∇ u
[Figure]
← −
[Figure]
end for for m = 1 , . . .
, M do u m ← u m − γ ˆ ∇ u m .
end for end for
[Figure]
variances.
The value of x-axis shows Σ 1 / 2 x ’s and Σ 1 / 2 u ’s are presented as colors.
As the graph shows,too small variances are not much different from using variance of 0, and when the both values are0.01 is the best and performance drops again for the variances larger than that.
So we used both Σ 1 / 2 xand Σ 1 / 2 u of 0.01.
C.3 Additional results on CIFAR
Table 5 shows additional results for the CIFAR10 dataset when the pseudocoreset size is larger.
Even in these cases, BPC-W and BPC-fKL effectively generate Bayesian pseudocoresets.
Moreover,compared to Table 1, the 10-ipc pseudocoreset trained with BPC-fKL outperforms the 100-ipc randomcoreset which has 10 times more images.
The overall BPC-fKL results demonstrate that the forward
[Table]
[Table]
KL divergence is effective for constructing the Bayesian pseudocoresets.
For evaluations, we usedsame hyperparameters as the case of ipc 20 as described in Table 2 and Table 3, except that σ r is0 .
01 , ε is 0 .
02 and λ is 0 .
Other coreset baselines and evaluation metrics
We compared our results with other coresetbaselines and other evaluation metrics for validating the quality of obtained posterior distributionsmore rigorously.
We added Herding [29] and K-center [31] as another coreset baslines and for otherevaluation metrics, we used expected calibration error (ECE) [21] and Brier score [2].
Herdingconstructs coresets by gathering samples close to the centers of the feature representations for eachclass and K-center constructs coresets by selecting multiple center points such that the distancebetween each data point is maximized while the distance between centers is minimized.
To obtainthe feature representations for both methods, we use a pre-trained ConvNet as in [13].
On the otherhand, the ECE and Brier scores are conventional metrics for evaluating posterior qualities.
Table 6shows the HMC results of various metrics for coresets and psuedocoresets with 10 ipc.
As [13]already shows that the coreset baselines underperform pseudocoresets for SGD, Table 6 also showsthat pseudocoresets are better for Bayesian inference tasks through various metrics.
C.4 Additional results on other datasets
In the main text, we trained Bayesian pseudocoresets only on the CIFAR10 dataset.
To validate howwell each method works on different datasets, we trained the pseudocoresets with a size of 1 imageper class on other datasets, CIFAR100 and ImageNet.
We can see that how well each method workswhen the number of classes is large with the CIFAR100 dataset and when the data dimension is largewith the ImageNet dataset.
The CIFAR100 dataset has the same image dimension as CIFAR10 buthas 100 classes and ImageNet has 128 × 128 data dimension.
For ImageNet, we use the same existingsubset of the entire dataset, ImageNette, which consists of 10 classes and an increased networkarchitecture.
Following previous work [8], we use a depth-5 ConvNet as the model architecture.
Asin the results on CIFAR10, Table 7 shows all three pseudocoresets are better than a random coreset.
Moreover, BPC-W and BPC-fKL outperform BPC-rKL, demonstrating that the proposed divergencemeasures are effective.
[Table]
C.5 Additional results on the synthetic dataset
To validate if the posterior distribution each algorithm learns as intended, we trained pseudocoresetson the synthetic dataset of samples from 10-dimensional multivariate Gaussian distribution, whoseposterior distribution is tractable.
Given data samples { x 1 , x 2 , ..., x 100 } i .
i .
d . ∼ N ( θ, Σ) , we trainedpseudocoresets { u m } Mm =1 of sizes M = { 5 , 10 , 15 , 20 } to have similar posterior distribution ofθ ∼ N ( θ 0 , Σ 0 ) with the true posterior which is tractable in this setting.
Since we know that the exactposterior distribution is given by N ( θ u , Σ u ) where
[Equation]
we validate each method by directly calculating the divergence measures between pseudocoresetsposteriors and true posterior.
Fig.
6 shows that all three methods work well in that all divergencesare well reduced even when trained with the algorithm with different divergence measure in thissimple synthetic setting.
Also, as expected, Fig.
7 shows divergences decrease as pseudocoreset sizesincrease.
D Example images of each Bayesian pseudocoreset
Fig.
8 are the example images of each pseudocoreset of 10 images per class.
[Figure]