Stochastic Multiple Target Sampling Gradient Descent
Abstract
Sampling from an unnormalized target distribution is an essential problem withmany applications in probabilistic inference.
Stein Variational Gradient Descent(SVGD) has been shown to be a powerful method that iteratively updates a set ofparticles to approximate the distribution of interest.
Furthermore, when analysingits asymptotic properties, SVGD reduces exactly to a single-objective optimizationproblem and can be viewed as a probabilistic version of this single-objective optimization problem.
A natural question then arises: “ Can we derive a probabilisticversion of the multi-objective optimization? ”.
To answer this question, we propose
Stochastic Multiple Target Sampling Gradient Descent (MT-SGD), enabling usto sample from multiple unnormalized target distributions.
Specifically, our MTSGD conducts a flow of intermediate distributions gradually orienting to multipletarget distributions, which allows the sampled particles to move to the joint highlikelihood region of the target distributions.
Interestingly, the asymptotic analysisshows that our approach reduces exactly to the multiple-gradient descent algorithmfor multi-objective optimization, as expected.
Finally, we conduct comprehensiveexperiments to demonstrate the merit of our approach to multi-task learning.
Introduction
Sampling from an unnormalized target distribution that we know the density function up to a scalingfactor is a pivotal problem with many applications in probabilistic inference [2, 18, 25].
For thispurpose, Markov chain Monte Carlo (MCMC) has been widely used to draw approximate posteriorsamples, but unfortunately, is often time-consuming and has difficulty accessing the convergence [14].
Targeting an efficient acceleration of MCMC, some stochastic variational particle-based approacheshave been proposed, notably Stochastic Langevin Gradient Descent [26] and Stein Variational
Gradient Descent (SVGD) [14].
Outstanding among them is SVGD, with a solid theoretical guaranteeof the convergence of the set of particles to the target distribution by maintaining a flow of distributions.
More specifically, SVGD starts from an arbitrary and easy-to-sample initial distribution and learnsthe subsequent distribution in the flow by push-forwarding the current one using a function T ( x ) =
x + ϵϕ ( x ) , where x ∈ R d , ϵ > 0 is the learning rate, and ϕ ∈ H dk with H k to be the Reproducing
Kernel Hilbert Space corresponding to a kernel k .
It is well-known that for the case of using Gaussian
RBF kernel, by letting the kernel width approach + ∞ , the update formula of SVGD at each stepasymptotically reduces to the typical gradient descent (GD) [14], showing the connection between aprobabilistic framework like SVGD and a single-objective optimization algorithm.
In other words,
SVGD can be viewed as a probabilistic version of the GD for single-objective optimization.
On the other side, multi-objective optimization (MOO) [7] aims to optimize a set of objectivefunctions and manifests itself in many real-world applications problems, such as in multi-tasklearning (MTL) [17, 24], natural language processing [1], and reinforcement learning [9, 22, 21].
Leveraging the above insights, it is natural to ask: “ Can we derive a probabilistic version of multiobjective optimization? ”.
By answering this question, we enable the application of the Bayesianinference framework to the tasks inherently fulfilled by the MOO framework.
Contribution.
In this paper, we provide an affirmative answer to that question.
In particular, we gobeyond the SVGD to propose Stochastic Multiple Target Sampling Gradient Descent (MT-SGD),enabling us to sample from multiple target distributions.
By considering the push-forward map
T ( x ) = x
+ ϵϕ ( x ) with ϕ ∈ H dk , we can find a closed-form for the optimal push-forward map
T ∗ pushing the current distribution on the flow simultaneously closer to all target distributions.
Similar to SVGD, in the case of using Gaussian RBF kernel, when the kernel width approaches + ∞ ,
MT-SGD reduces exactly to the multiple-gradient descent algorithm (MGDA)
[7] for multi-objectiveoptimization (MOO).
Our MT-SGD, therefore, can be considered as a probabilistic version of the GDmulti-objective optimization [7] as expected.
Additionally, in practice, we consider a flow of empirical distributions, in which, each distribution ispresented as a set of particles.
Our observations indicate that MT-SGD globally drives the particlesto close to all target distributions, leading them to diversify on the joint high-likelihood region forall distributions.
It is worth noting that, different from other multi-particle approaches [13, 15, 17]leading the particles to diversify on a Pareto front, our MT-SGD orients the particle to diversify onthe so-called Pareto common (i.e., the joint high-likelihood region for all distributions) (cf.
Section2.4 for more discussions).
We argue and empirically demonstrate that this characteristic is essentialfor the Bayesian setting, whose main goal is to estimate the ensemble accuracy and the uncertaintycalibration of a model.
In summary, we make the following contributions in this work:
• Propose a principled framework that incorporates the power of Stein Variational Gradient Descentinto multi-objective optimization.
Concretely, our method is motivated by the theoretical analysis ofSVGD, and we further derive the formulation that extends the original work and allows to samplefrom multiple unnormalize distributions.•
Demonstrate our algorithm is readily applicable in the context of multi-task learning.
The benefitsof MT-SGD are twofold: i) the trained network is optimal, which could not be improved in any taskwithout diminishing another, and ii) there is no need for predefined preference vectors as in previous works [13, 17], MT-SGD implicitly learns diverse models universally optimizing for all tasks.• Conduct comprehensive experiments to verify the behaviors of MT-SGD and demonstrate thesuperiority of MT-SGD to the baselines in a Bayesian setting, with higher ensemble performancesand significantly lower calibration errors.
Related works.
The work of [7] proposed a multi-gradient descent algorithm for multi-objectiveoptimization (MOO) which opens the door for the applications of MOO in machine learning and deeplearning.
Inspired by [7], MOO has been applied in multi-task learning (MTL) [17, 24], few-shotlearning [4, 28], and knowledge distillation [5, 8].
Specifically, in an earlier attempt at solving MTL,[24] viewed multi-task learning as a multi-objective optimization problem, where a task networkconsists of a shared feature extractor and a task-specific predictor.
In another study, [17] developeda gradient-based multi-objective MTL algorithm to find a set of solutions that satisfies the userpreferences.
Also follows the idea of learning neural networks conditioned on pre-defined preferencevectors, [13] proposed Pareto MTL, aiming to find a set of well-distributed Pareto solutions, whichcan represent different trade-offs among different tasks.
Recently, the work of [15] leveraged MOOwith SVGD [14] and Langevin dynamics [26] to diversify the solutions of MOO.
In another lineof work, [28] proposed a bi-level MOO that can be applied to few-shot learning.
Furthermore, asomewhat different result was proposed, [8] applied MOO to enable the knowledge distillation frommultiple teachers and find a better optimization direction in training the student network.
Outline.
The paper is organized as follows.
In Section 2, we first present our theoretical contributionby reviewing the formalism and providing the point of view adopted to generalize SVGD in thecontext of MOO.
Then, Section 3 introduces an algorithm to showcase the application of our proposedmethod in the multi-task learning scenario.
We report the results of extensive experimental studiesperformed on various datasets that demonstrate the behaviors and efficiency of MT-SGD in Section 4.
Finally, we conclude the paper in Section 5.
The complete proofs and experiment setups are deferredto the supplementary material.
Multi-Target Sampling Gradient Descent
We first briefly introduce the formulation of the multi-target sampling in Section 2.1.
Second, Section2.2 presents our theoretical development and shows how our proposed method is applicable to this problem.
Finally, we detail how to train the proposed method in Section 2.3 and highlight keydifferences between our method and related work in Section 2.4.
2.1 Problem Setting
Given a set of target distributions p 1: K ( θ ) :
= { p 1 ( θ ) , . . .
, p K ( θ ) } with parameter θ ∈ R d , we aimto find the optimal distribution q ∗ ∈ Q that simultaneously minimizes:
[Equation]
where D KL represents Kullback-Leibler divergence and Q is a family of distributions.
The optimization problem (OP) in (1) can be viewed as a multi-objective OP [7] on the probabilitydistribution space.
Let us denote H k by the Reproducing Kernel Hilbert Space (RKHS) associatedwith a positive semi-definite (p.s.d.) kernel k , and H dk by the d -dimensional vector function
f =[
f 1 , . .
f d ] , where f i ∈ H k .
Inspired by [14], we construct a flow of distributions q 0 , q 1 , ..., q Ldeparted from a simple distribution q 0 , that gradually move closer to all the target distributions.
Inparticular, at each step, assume that q is the current obtained distribution, and the goal is to learn atransformation T
= id + ϵϕ so that the feed-forward distribution q [ T ] =
T
# q moves closer to p 1: Ksimultaneously.
Here we use id to denote the identity operator, ϵ > 0 is a step size, and
ϕ ∈ H dk is avelocity field.
Particularly, the problem of finding the optimal transformation T for the current step isformulated as:
[Equation]
2.2 Our Theoretical Development
It is worth noting that the transformation T defined above is injective when ϵ is sufficiently small[14].
We consider each D KL (cid:0) q
[ T ] ∥
p i (cid:1) , i = 1 , ..., K as a function w.r.t.
ϵ , by applying the first-order
Taylor expansion at 0 , we have:
[Equation]
where lim ϵ → 0 O (cid:0) ϵ 2 (cid:1) /ϵ 2 = const .
Similar to [14], the gradient ∇ ϵ D KL (cid:0) q [ T ] ∥ p i (cid:1) | ϵ =0 can be calculated as provided in
[Equation]
where ψ i ( · ) =
E θ ∼ q [ k ( θ, · ) ∇ θ log p
i ( θ ) + ∇ θ k
( θ. · )] and ⟨· , ·⟩ H dk is the dot product in the RKHS.
This means that, for each target distribution p i , the steepest descent direction is ϕ ∗ i =
ψ i , in which the KL divergence of interest D KL (cid:0) q [ T ] ∥ p i (cid:1) gets decreased roughly by − ϵ ∥
ϕ ∗ i ∥ 2 H dk toward the target distribution p i .
However, this only guarantees a divergence reduction for a single target distribution p i itself.
Our next aim is hence to find a common direction ϕ ∗ to reduce the KL divergences w.r.t.
all target distributions, which is reflected in the following lemma, showing us how to combine the individual steepest descent direction ϕ ∗
i
=
ψ i to yield the optimal direction ϕ ∗ as summarized in Figure 1.
Lemma 1.
Let w ∗ be the optimal solution of the optimization problem w ∗ = argmin w ∈ ∆ K w T Uw andϕ ∗ =
(cid:80)
Ki
=1 w ∗
i ϕ ∗ i , where ∆ K = (cid:8) π ∈ R K + : ∥ π ∥ 1 = 1 (cid:9) and U ∈ R K × K with U ij = (cid:10) ϕ ∗ i , ϕ ∗ j (cid:11)
H dk ,then we have⟨ ϕ ∗ , ϕ ∗
i ⟩ H dk ≥ ∥ ϕ ∗ ∥ 2 H dk , i = 1 , ..., K.
Lemma 1 provides a common descent direction ϕ ∗ so that all KL divergences w.r.t.
the targetdistributions are consistently reduced by ϵ ∥ ϕ ∗ ∥
2 H dk roughly and Theorem 2 confirms this argument.
Theorem 2.
If there does not exist w ∈ ∆ K such that (cid:80) Ki =1 w i ϕ ∗
i
= 0 , given a sufficiently smallstep size ϵ , all KL divergences w.r.t.
the target distributions are strictly decreased by at least
A ∥ ϕ ∗ ∥ 2 H dk > 0 where A is a positive constant.
The next arising question is how to evaluate the matrix U ∈ R K × K with U ij = (cid:10)
ϕ ∗ i , ϕ ∗ j (cid:11)
H dk
forsolving the quadratic problem: min w ∈ ∆ K w T Uw .
To this end, using some well-known equalities inthe RKHS 1 , we arrive at the following formula:
[Equation]
where tr( · ) denotes the trace of a (square) matrix.
2.3 Algorithm for MT-SGD
For the implementation of MT-SGD, we consider q as a discrete distribution over a set of M ,( M ∈ N ∗ ) particles θ 1 , θ 2 , . .
, θ M ∼ q .
The formulation to evaluate U ij in Equation.
(3) becomes:
[Equation]
The optimal solution ϕ ∗ i then can be computed as:
[Equation]
The key steps of our MT-SGD are summarized in Algorithm 1, where the set of particles θ 1: M isupdated gradually to approach the multiple distributions p 1: K .
Furthermore, the update formulaconsists of two terms: (i) the first term (i.e., relevant to k ( θ j , · ) ∇ θ j log p
i ( θ j ) ) helps to push theparticles to the joint high-likelihood region for all distributions and (ii) the second term (i.e., relevantto ∇ θ j k ( θ j , · ) ) which is a repulsive term to push away the particles when they reach out each other.
Finally, we note that our proposed MT-SGD can be applied in the context where we know the targetdistributions up to a scaling factor (e.g., in the posterior inference).
Input: Multiple unnormalized target densities p 1: K .
Output: The optimal particles θ 1 , θ 2 , . . .
, θ M . 1: Initialize a set of particles θ 1 , θ 2 , . .
, θ M ∼ q 0 . 2: for t = 1 to L
do 3:
Form the matrix U ∈ R K × K with the element U ij computed as in Equation.
Solve the QP min w ∈ ∆ K w T Uw to find the optimal weights w ∗ ∈ ∆ K .
Compute the optimal direction ϕ ∗ ( · ) =
(cid:80) K i =1 w ∗
i ϕ ∗ i ( · ) , where ϕ ∗ i is defined in Equation.
Update θ i
=
θ i + ϵϕ ∗ ( θ i ) , i = 1 , ..., K .
7: end for 8: return θ 1 , θ 2 , . .
, θ M .
Analysis for the case of RBF kernel.
We now consider a radial basis-function (RBF) kernel ofbandwidth σ : k
( θ, θ ′ ) =
[Equation] [Equation]
and examine some asymptotic behaviors.
▶ General case: The elements of the matrix U become
[Equation]
▶ Single particle distribution q = δ θ : The elements of the matrix U become
[Equation]
and our formulation reduces exactly to MOO in [7].
▶
When σ → ∞ : The elements of the matrix U become
[Equation]
2.4 Comparison to MOO-SVGD and Other Works
The most closely related work to ours is MOO-SVGD [15].
In a nutshell, ours is principally differentfrom that work.
Figure 2 shows the fundamental difference between our MT-SGD and MOO-SVGD.
Our MT-SGD navigates the particles from one distribution to another distribution consecutively with atheoretical guarantee of globally getting closely to multiple target distributions, whereas MOO-SVGDuses the MOO [7] to update the particles individually and independently.
Additionally, MOO-SVGDemploys a repulsive term to encourage the particle diversity without any theoretical-guaranteedprinciple to control the repulsive term, hence it can force the particles to scatter on the multipledistributions.
Moreover, MOO-SVGD is not computationally efficient when the number of particlesis high because it requires solving an independent quadratic programming problem for each particle(cf.
Section 4.1.1 and Figure 3 for the experiment on a synthetic dataset).
[Figure]
Furthermore, it expects that our MT-SGD globally moves the set of particles to the joint highlikelihood region for all target distributions.
Therefore, we do not claim our MT-SGD as method todiversify the solution on a Pareto front for user preferences as in [15, 17].
Alternatively, our MT-SGDcan generate diverse particles on the so-called Pareto common (i.e., the joint high-likelihood regionfor all target distributions).
We argue and empirically demonstrate that by finding and diversifyingthe particles on Pareto common for the multiple posterior inferences, our MT-SGD can outperformthe baselines on Bayesian-inference metrics such as the ensemble accuracy and the calibration error.
Application to Multi-Task Learning
For multi-task learning, we assume to have K tasks {T i } Ki =1 and a training set D
= ( x i , y i 1 , ..., y iK ) } Ni =1 , where x i is a data example and y i 1 , ..., y iK are the labels for the tasks.
The model for each task θ j = (cid:2) α, β j (cid:3) , j = 1 , ..., K consists of the shared part
α and non-sharedpart β j targeting the task j .
The posterior p (cid:0) θ j | D (cid:1) for each task readsp [Equation] θ j | D (cid:1) ∝ p (cid:0) D | θ j (cid:1) p (cid:0) θ j (cid:1) ∝
[Equation] (cid:89) i = p (cid:0)
y ij | x i , θ j (cid:1)
∝
N (cid:89) i = exp (cid:8) − [Equation]
(cid:0) y
[Equation]
, x i ; θ j [Equation] where ℓ is a loss function and the predictive likelihood p ( y ij | x i , θ j ) ∝ exp (cid:8) − ℓ (cid:0) y ij , x i ; θ j (cid:1)(cid:9) isexamined.
Note that the prior p (cid:0) θ j (cid:1) here is retained from previous studies [13, 15], which is auniform and non-informative prior and can be treated as a constant term in our formulation.
For our approach, we maintain a set of models θ m = (cid:2) θ jm (cid:3) K j =1 with m = 1 , ..., M , where θ jm =(cid:2) α m , β jm (cid:3) .
At each iteration, given the non-shared parts (cid:2) β j (cid:3) K j =1 with β j = (cid:2) β jm (cid:3) M m =1 , we samplethe shared parts from the multiple distributions p (cid:0) α | β j , D (cid:1) , j = 1 , ..., K as
[Equation]
[Equation] We now apply our proposed MT-SGD to sample the shared parts [ α m ]
Mm =1 from the multiple distributions defined in (6) as
[Equation]
(7) where ϕ ∗ j ( α m )
=
1 M (cid:80) Mt =1 (cid:104)
k 1 ( α t , α m ) ∇ α t log p (cid:16) α t | β jt , D (cid:17) + ∇ α t k 1 ( α t , α m ) (cid:105) and w ∗ =[ w ∗ k
Kk =1 are the weights received from solving the quadratic programming problem.
Here wenote that ∇ α t log p (cid:16) α t | β jt , D (cid:17) can be estimated via the batch gradient of the loss using Equation(6).
Given the updated shared parts [ α m ]
Mm =1 , for each task j , we update the corresponding non-sharedparts (cid:2) β jm (cid:3) M m =1 by sampling
[Equation]
We now apply SVGD [14] to sample the non-shared parts (cid:2) β jm (cid:3) M m =1 for each task j from thedistribution defined in (8) as [Equation]
[Equation]
) where ψ ∗ j (cid:0) β jm (cid:1)
= 1 M (cid:80) Ma =1 (cid:104) k 2 (cid:0) β ja , β jm (cid:1) ∇ β ja log p (cid:0) β ja | α a , D (cid:1) + ∇ β ja k 2 (cid:0) β ja , β jm (cid:1)(cid:105) with whichthe term ∇ β ja log p (cid:0) β ja | α a , D (cid:1) can be estimated via the batch loss gradient using Equation (8).
Algorithm 2 summarizes the key steps of our multi-task MT-SGD.
Basically, we alternatively updatethe shared parts given the non-shared ones and vice versa.
Experiments
In this section, we verify our MT-SGD by evaluating its performance on both synthetic and real-worlddatasets.
For our experiments, we use the RBF kernel k
( θ, θ ′ )
=
exp (cid:110) − ∥ θ − θ ′
∥
22 / (cid:0) 2 σ 2 (cid:1)(cid:111) .
Thedetailed training and configuration are given in the supplementary material.
4.1 Experiments on Toy Datasets
4.1.1 Sampling from Multiple Distributions
We first qualitatively analyze the behavior of the proposed method on sampling from three targetdistributions.
Each target distribution is a mixture of two Gaussians as p i ( θ )
=
π
i 1 N ( θ | µ i 1 , Σ i 1 ) +π i 2 N ( θ | µ i 2 , Σ i 2 ) ( i = 1 , 2 , 3) where the mixing proportions π i 1 = 0 .
7 , ∀ i , π i 2 = 0 .
3 , ∀ i ,the means µ
=
[4 , − 4] T , µ 12 =
5] T , µ 21 = [ − 4 , 4] T , µ 22 =
[0 . 5 , 0] T , and µ 31 =[ − 3 , − 3] T , µ 32 =
[0 , 0] T , and the common covariance matrix Σ ij = (cid:20) 0 .
0 .(cid:21) , i = 1 , 2 , 3and j = 1 , 2 .
It can be seen from Figure 3 that there is a common high-density region spreadingaround the origin.
The fifty particles are drawn randomly in the space, and the initialization is retainedacross experiments for a fair comparison.
Figure 3 shows the updated particles by MOO-SVGD and MT-SGD at selected iterations, we observethat the particles from MOO-SVGD spread out and tend to characterize all the modes, some of themeven scattered along trajectories due to the conflict in optimizing multiple objectives.
By contrast,our method is able to find and cover the common high density region among target distributionswith well-distributed particles, which illustrates the basic principles of MT-SGD.
Additionally, atthe 1 , 000 -th step, the training time for ours is 0.23 min, whereas that for MOO-SVGD is 1.63 min.
The reason is that MOO-SVGD requires solving an independent quadratic programming problem foreach particle at each step.
4.1.2 Multi-objective Optimization
The previous experiment illustrates that MT-SGD can be used to sample from multiple targetdistributions, we next test our method on the other low-dimensional multi-objectives OP from[29].
In particular, we use the two objectives ZDT3, whose Pareto front consists of non-contiguousconvex parts, to show our method simultaneously minimizes both objective functions.
Graphically,the simulation results from Figure 4 show the difference in the convergence behaviors between
MOO-SVGD and MT-SGD: the solution set achieved by MOO-SVGD covers the entire Pareto front,while ours distributes and diversifies on the three middle curves (mostly concentrated in the middlecurve) which are the Pareto common having low values for two objective functions in ZDT3.
4.2 Experiments on Real Datasets4.2.1 Experiments on Multi-Fashion+Multi-MNIST Datasets
We apply the proposed MT-SGD method on multi-task learning, following Algorithm 2.
Our methodis validated on different benchmark datasets: (i) Multi-Fashion+MNIST [23], (ii) Multi-MNIST, and(iii)
Multi-Fashion.
Each of them consists of 120,000 training and 20,000 testing images generatedfrom MNIST [12] and FashionMNIST [27] by overlaying an image on top of another: one in thetop-left corner and one in the bottom-right corner.
Lenet [12] (22,350 params) is employed as thebackbone architecture and trained for 100 epochs with SGD in this experimental setup.
Baselines: In multi-task experiments, the introduced MT-SGD is compared with state-of-the-artbaselines including MGDA [24], Pareto MTL [13], MOO-SVGD [15].
We note that to reproduceresults for these baselines, we either use the author’s official implementation released on GitHub orask the authors for their codes.
For MOO-SVGD and Pareto MTL, the reported result is from theensemble prediction of five particle models.
Additionally, for linear scalarization and MGDA, wetrain five particle models independently with different initializations and then ensemble these models. [Figure]
We report the ensemble accuracy ( higher is better ) and the Brier score ( lower is better ) overindependent runs, as well as the standard deviation (the error bars and shaded areas in the figures).
Evaluation metrics: We compare MT-SGD against baselines regarding both average accuracy andpredictive uncertainty.
Besides the commonly used accuracy metric, we measure the quality anddiversity of the particle models by relying on two other popular Bayesian metrics: Brier score [3, 20]and expected calibration error (ECE) [6, 19].
From Figure
[Table]
we observe that MT-SGD consistently improves model performance across all tasksin both accuracy and Brier score by large margins, compared to existing techniques in the literature.
The network trained using linear scalarization, as expected, produces inferior ensemble results whileutilizing MOO techniques helps yield better performances.
Overall, our proposed method surpassesthe second-best baseline by at least 1% accuracy in any experiment.
Furthermore, Table 1 provides acomparison between these methods in terms of expected calibration error, in which MT-SGD alsoconsistently provides the lowest expected calibration error, illustrating our method’s ability to obtainwell-calibrated models (the accuracy is closely approximated by the produced confidence score).
It isalso worth noting that while Pareto MTL has higher accuracy, MOO-SVGD produces slightly bettercalibration estimation
[Table] [Table]
4.2.2 Experiment on CelebA Dataset
[Table] In this experiment [Table] we verify the significance of MT-SGD on a larger neural network: Resnet18 [10],which consists of 11.4M parameters.
We take the first 10 binary classification tasks and randomlyselect a subset of 40k images from the CelebA dataset [16].
Note that in this experiment, we consider
Single task, in which 10 models are trained separately and serves as a strong baseline.
[Table]
The performance comparison of the mentioned models in CelebA experiment is shown in Table 2.
Asclearly seen from the upper part of the table, MT-SGD performs best in all tasks, except in BL, where
MOO-SVGD is slightly better (86.4% vs 86.3%).
Moreover, our method matches or beats Single task-
the second-best baseline in all tasks.
Regarding the well-calibrated uncertainty estimates, ensemblelearning methods exhibit better results.
In particular, MT-SGD and MOO-SVGD provide the bestcalibration performances, which are 2% and 2 .
5% , respectively, which emphasizes the importance ofefficient ensemble learning for enhanced calibration.
Conclusion
In this paper, we propose Stochastic Multiple Target S ampling Gradient Descent (MT-SGD), allowingus to sample the particles from the joint high-likelihood of multiple target distributions.
Our MT-SGDis theoretically guaranteed to simultaneously reduce the divergences to the target distributions.
Interestingly, the asymptotic analysis of our MT-SGD reduces exactly to the multi-objective optimization.
We conduct comprehensive experiments to demonstrate that by driving the particles to the Paretocommon (the joint high-likelihood of multiple target distributions), our MT-SGD can outperformthe baselines on the ensemble accuracy and the well-known Bayesian metrics such as the expectedcalibration error and the Brier score. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
In the supplementary material,
we discuss the limitations of our proposed method.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We include
assumptions in all theoretical statements.
(b) Did you include complete proofs of all theoretical results?
[Yes] The main results and
lemmas appear in the main paper and their complete proof is put in the supplementary
material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] They will be
attached to the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] The training details are provided with the code, but the importantdetails should be in the main paper.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
In all experiments, we set the random seed as in
codebases of the related work and do not tune that specific parameter.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] The detailed training environment
will be attached in the supplementary material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the authors
[Table]
of the codes that are included in our implementations.
(b) Did you mention the license of the assets?
[No] All the assets we used are published
for research purposes.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
The tools used in our implementation are presented in our supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[No] The data we used are open to the public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data we used are open to the public.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]