Markov Chain Score Ascent: A Unifying Framework of Variational Inference with Markovian Gradients
Abstract
Minimizing the inclusive Kullback-Leibler (KL) divergence with stochasticgradient descent (SGD) is challenging since its gradient is defined as an integral over the posterior.
Recently, multiple methods have been proposed torun SGD with biased gradient estimates obtained from a Markov chain.
Thispaper provides the first non-asymptotic convergence analysis of these methodsby establishing their mixing rate and gradient variance.
To do this, we demonstrate that these methods–which we collectively refer to as Markov chain scoreascent (MCSA) methods–can be cast as special cases of the Markov chain gradient descent framework.
Furthermore, by leveraging this new understanding,we develop a novel MCSA scheme, parallel MCSA (pMCSA), that achieves atighter bound on the gradient variance.
We demonstrate that this improvedtheoretical result translates to superior empirical performance.
Introduction
Bayesian inference aims to analyze the posterior distribution of an unknown latent variable 𝐳from which data 𝐱 is observed.
By assuming a model 𝑝 (𝐱 | 𝐳) , the posterior 𝜋 (𝐳) is given by
Bayes’ rule such that 𝜋 (𝐳) ∝ 𝑝 (𝐱 | 𝐳) 𝑝 (𝐳) where 𝑝 (𝐳) represents our prior belief on 𝐳 .
Insteadof working directly with 𝜋 , variational inference (VI, Blei et al. 2017) seeks a variational approximation 𝑞 (𝐳; 𝛌) that is the most similar to 𝜋 according to a discrepancy measure 𝑑 (𝜋, 𝑞 (⋅; 𝛌)) .
The apparent importance of choosing the right discrepancy measure has led to a quest spanninga decade (Bamler et al. , 2017; Dieng et al. , 2017; Geffner & Domke, 2021b;
Hernandez-Lobatoet al. , 2016; Li & Turner, 2016; Regli & Silva, 2018; Ruiz & Titsias, 2019; Salimans et al. , 2015;
Wan et al. , 2020; Wang et al. , 2018; Zhang et al. , 2021).
So far, the exclusive (or reverse,backward) Kullback-Leibler (KL) divergence 𝑑 KL (𝑞 (⋅; 𝛌) ∥ 𝜋) has seen “exclusive” use, partlybecause it is defined as an integral over 𝑞 (𝐳; 𝛌) , which can be approximated efficiently.
Incontrast, the inclusive (or forward) KL is defined as an integral over 𝜋 as
[Equation]
Since our goal is to approximate 𝜋 with 𝑞 (⋅; 𝛌) but the inclusive KL involves an integral over 𝜋 ,we end up facing a chicken-and-egg problem.
Despite this challenge, the inclusive KL has consistently drawn attention due to its statistical properties, such as better uncertainty estimatesdue to its mass covering property (MacKay, 2001; Minka, 2005; Trippe & Turner, 2017).
Recently, Naesseth et al.
(2020); Ou & Song (2020) have respectively proposed Markovian scoreclimbing (MSC) and joint stochastic approximation (JSA).
These methods minimize the inclusive KL using stochastic gradient descent (SGD, Robbins & Monro 1951), where the gradientsare estimated using a Markov chain.
The Markov chain kernel 𝐾 𝛌 𝑡 (𝐳 𝑡 , ⋅) is 𝜋 -invariant (Robert& Casella, 2004) and is chosen such that it directly takes advantage of the current variational approximation 𝑞 (⋅; 𝛌 𝑡 ) .
Thus, the quality of the gradients improves over time as the KL divergencedecreases.
Still, the gradients are non-asymptotically biased and Markovian across adjacent iterations, which sharply contrasts MSC and JSA from classical black-box VI (Kucukelbir et al. ,2017; Ranganath et al. , 2014), where the gradients are unbiased and independent.
While Naesseth et al.
(2020) has shown the convergence of MSC through the work of Gu & Kong (1998), thisresult is only asymptotic and does not provide practical insight into the performance of MSC.
In this paper, we address these theoretical gaps by casting MSC and JSA into a general framework we call Markov chain score ascent (MCSA), which we show is a special case of Markovchain gradient descent (MCGD, Duchi et al. 2012).
This enables the application of the nonasymptotic convergence results of MCGD (Debavelaere et al. , 2021;
Doan et al. , 2020a,b; Duchiet al. , 2012; Karimi et al. , 2019; Sun et al. , 2018; Xiong et al. , 2021).
For MCGD methods, thefundamental properties affecting the convergence rate are the ergodic convergence rate ( 𝜌 ) ofthe MCMC kernel and the gradient variance ( 𝐺 ).
We analyze 𝜌 and 𝐺 of MSC and JSA, enablingtheir practical comparison given a fixed computational budget ( 𝑁 ).
Furthermore, based on therecent insight that the mixing rate does not affect the convergence rate of MCGD Doan et
al.(2020a,b), we propose a novel scheme, parallel MCSA (pMCSA), which achieves lower varianceby trading off the mixing rate.
We verify our theoretical analysis through numerical simulationsand compare MSC, JSA, and pMCSA on general Bayesian inference problems.
Our experimentsshow that our proposed method outperforms previous MCSA approaches.
Contribution Summary
Section 4: We provide the first non-asymptotic theoretical analysis of two recently pro-
posed inclusive KL minimization methods, MSC (Theorem 1) and JSA (Theorem 2) .2.
Section 3: To do this, we show that both methods can be viewed as what we call “Markovchain score ascent” (MCSA) methods, which are a special case of MCGD (Proposition 1) .3.
Section 5:
In light of this, we develop a novel MCSA method which we call parallel MCSA(pMCSA) that achieves lower gradient variance (Theorem 3) .4.
Section 6: We demonstrate that the improved theoretical performance of pMCSA translates to superior empirical performance across a variety of Bayesian inference tasks.
Background2.1 Inclusive Kullback-Leibler
Minimization
with
Stochastic
Gradients
VI with SGD The goal of VI is to find the optimal variational parameter 𝛌 identifying 𝑞 (⋅; 𝛌) ∈𝒬 that minimizes some discrepancy measure 𝐷 (𝜋, 𝑞 (⋅; 𝛌)) .
A typical way to perform VI is touse stochastic gradient descent (SGD, Robbins & Monro 1951), provided that unbiased gradientestimates of the optimization target 𝐠 (𝛌) are available such that we can repeat the update𝛌 𝑡 = 𝛌 𝑡−1 − 𝛾 𝑡 𝐠 (𝛌 𝑡−1 ) where 𝛾 1 , … , 𝛾 𝑇 is a step-size schedule.
Inclusive KL Minimization with SGD For inclusive KL minimization, 𝐠 should be set as
[Equation]
(𝛌; 𝘇) ] where ℍ [𝜋, 𝑞 (⋅; 𝛌)] is the cross-entropy between 𝜋 and 𝑞 (⋅; 𝛌) , which shows the connection withcross-entropy methods (de Boer et al. , 2005), and 𝐬 (𝛌; 𝐳) = ∇ 𝛌 log 𝑞 (𝐳; 𝛌) is known as the scoregradient .
Since inclusive KL minimization with SGD is equivalent to ascending towards thedirection of the score, Naesseth et al.
(2020) coined the term score climbing.
To better conformwith the optimization literature, we instead call this approach score ascent as in gradient ascent.
2.2 Markov Chain
Gradient
Descent
Overview of MCGD Markov chain gradient descent (MCGD, Duchi et al.
2012; Sun et al.2018) is a family of algorithms that optimize a function 𝑓 defined as 𝑓 (𝛌) = ∫ 𝑓 (𝛌, 𝛈) Π (𝑑𝛈)where Π (𝑑𝛈) is a probability measure.
MCGD repeats the steps
[Equation]
) where 𝑃 𝛌 𝑡−1 is a Π -invariant Markov chain kernel that may depend on 𝛌 𝑡−1 .
The noise of the gradient is Markovian and non-asymptotically biased, departing from vanilla SGD.
Non-asymptoticconvergence of this general algorithm has recently started to gather attention as by Debavelaereet al.
; Doan et al.
(2020a,b); Duchi et al.
Karimi et al.
; Sun et al.
Applications of MCGD MCGD encompasses an extensive range of problems, including distributed optimization (Ram et al. , 2009), reinforcement learning (Doan et al. , 2020a;
Tadić &
Doucet, 2017; Xiong et al. , 2021), and expectation-minimization (Karimi et al. , 2019), to name afew.
This paper extends this list with inclusive KL VI through the MCSA framework.
Markov Chain Score Ascent
First, we develop Markov chain score ascent (MCSA), a framework for inclusive KL minimization with MCGD.
This framework will establish the connection between MSC/JSA and MCGD.
3.1 Markov Chain Score Ascent as a Special Case of Markov Chain Gradient Descent
As shown in Equation (1), the basic ingredients of MCGD are the target function 𝑓 (𝛌, 𝜂) , thegradient estimator 𝐠 (𝛌, 𝜂) , and the Markov chain kernel 𝑃 𝛌 (𝜂, ⋅) .
Obtaining MCSA from MCGDboils down to designing 𝐠 and 𝑃 𝛌 such that 𝑓 (𝛌)
=
𝑑 KL (𝜋 ∥ 𝑞 (⋅; 𝛌)) .
The following propositionprovides sufficient conditions on 𝐠 and 𝑃 𝛌 to achieve this goal.
Proposition 1.
Let 𝛈 =
(𝐳 (1) , 𝐳 (2) , … , 𝐳 (𝑁) ) and a Markov chain kernel 𝑃 𝛌 (𝛈, ⋅) be Π -invariantwhere Π is defined as
[Equation]
Then, by defining the target function 𝑓 and the gradient estimator 𝐠 to be
[Equation]
where ℍ [ 𝜋 ] is the entropy of 𝜋 , MCGD results in inclusive KL minimization as
[Equation]
Proof.
See the full proof in Appendix D.
This simple connection between MCGD and VI paves the way toward the non-asymptotic analysis of JSA and MSC.
Note that 𝑁 here can be regarded as the computational budget of each
MCGD iteration since the cost of (i) generating the Markov chain samples 𝐳 (1) , … , 𝐳 (𝑁) and(ii)
computing the gradient 𝐠 will linearly increase with 𝑁 .
In addition, the MCGD framework often assumes 𝑃 to be geometrically ergodic.
An exceptionis the analysis of Debavelaere et al.
(2021) where they work with polynomially ergodic kernels.
Assumption 1.
(Markov chain kernel)
The Markov chain kernel 𝑃 is geometrically ergodic as
[Equation]
for some positive constant 𝐶 .
3.2 Non-Asymptotic Convergence of Markov Chain
Score Ascent
Non-Asymptotic Convergence Through Proposition 1 and Assumption 1 and some technical assumptions on the objective function, we can apply the existing convergence results of
MCGD to MCSA.
Table 1 provides a list of relevant results.
Apart from properties of the objective function (such as Lipschitz smoothness), the convergence rates are stated in terms of thegradient bound 𝐺 , kernel mixing rate 𝜌 , and the number of MCGD iterations 𝑇 .
We focus on 𝐺and 𝜌 as they are closely related to the design choices of different MCSA algorithms.
[Table]
Notation : 1 ℱ 𝑡 is the 𝜎 -field formed by all the iterates 𝛈 𝑡 , 𝛌 𝑡 up to the 𝑡 th MCGD iteration and ‖𝐱‖ ∗ is the dual norm such that ‖𝐱‖ ∗ = sup ‖𝐳‖≤1 ⟨𝐱,𝐳⟩ . 2 𝛽 𝑡 is the stepsize of the momentum.
23 𝐿 is the
Lipschitz smoothness constant.
3 𝜇 is the strong convexity constant.
Convergence and the Mixing Rate 𝜌 Duchi et al.
(2012) was the first to provide an analysisof the general MCGD setting.
Their convergence rate is dependent on the mixing rate throughthe 1/ log 𝜌 −1 term.
For MCSA, this results in an overly conservative rate since, on challengingproblems, mixing can be slow such that 𝜌 ≈ 1 .
Fortunately, Doan et al.
(2020a,b) have recentlyshown that it is possible to obtain a rate independent of the mixing rate 𝜌 .
For example, in theresult of Doan et al.
(2020b), the influence of 𝜌 decreases in a rate of 𝒪 ( 1 / 𝑇 2 ) .
This observationis critical since it implies that trading “lower gradient variance” with a “slower mixing rate” couldbe profitable .
We exploit this observation in our novel MCSA scheme in Section 5.
Gradient
Bound 𝐺
Except for Doan et al.
(2020b), most results assume that the gradient isbounded for ∀𝛈, 𝛌 as ‖𝐠(𝛌,𝛈)‖ < 𝐺 .
Admittedly, this condition is strong, but it is similar to thebounded variance assumption 𝔼[‖𝐠‖ 2 ] <
𝐺 2 used in vanilla SGD, which is also known to bestrong as it contradicts strong convexity (Nguyen et al. , 2018).
Nonetheless, assuming 𝐺 canhave practical benefits beyond theoretical settings.
For example, Geffner & Domke (2020) use 𝐺to compare the performance different VI gradient estimators.
In a similar spirit, we will obtainthe gradient bound 𝐺 of different MCSA algorithms and compare their theoretical performance.
Demystifying Prior Markov Chain Score Ascent Methods
In this section, we will show that MSC and JSA both qualify as MCSA methods.
Furthermore, weestablish (i) the mixing rate of their implicitly defined kernel 𝑃 and (ii) the upper bound on theirgradient variance.
This will provide insight into their practical non-asymptotic performance.
4.1 Technical Assumptions
To cast previous methods into MCSA, we need some technical assumptions.
Assumption 2.
(Bounded importance weight)
The importance weight ratio 𝑤 (𝐳)
= 𝜋 (𝐳)
/𝑞
(𝐳; 𝛌)is bounded by some finite constant as 𝑤 ∗ < ∞ for all 𝛌 ∈ Λ
such that 𝜌
=
(1 − 1/𝑤 ∗ ) < 1 .
This assumption is necessary to ensure Assumption 1, and can be practically ensured by using avariational family with heavy tails (Domke & Sheldon, 2018) or using a defensive mixture (Hesterberg, 1995; Holden et al. , 2009) as
[Equation]
where 0 < 𝑤 < 1 and 𝜈 (⋅) is a heavy tailed distribution such that sup 𝐳∈𝒵 𝜋 (𝐳) /𝜈
(𝐳) < ∞ .
Note that 𝑞 def.
is only used in the Markov chain kernels and 𝑞 (⋅; 𝛌 ∗ ) is still the output of the VIprocedure.
While these tricks help escape slowly mixing regions, this benefit quickly vanishesas we converge.
Therefore, ensuring Assumption 2 seems unnecessary in practice unless weabsolutely care about ergodicity ( e.g. adaptive MCMC, Brofos et al. 2022; Holden et al. 2009).
Model (Variational Family) Misspecification and 𝑤 ∗
Note that 𝑤 ∗ is bounded below exponentially by the inclusive KL as shown in Proposition 2.
Therefore, 𝑤 ∗ will be large (i) in theinitial steps of VI and (ii) under model (variational family) misspecification.
Assumption 3.
(Bounded Score)
The score gradient is bounded for ∀𝛌 ∈ Λ and ∀𝐳 ∈
𝒵 such that‖ 𝐬 (𝛌; 𝐳) ‖ ∗ ≤ 𝐿 for some finite constant 𝐿 > 0 .
Although this assumption is strong, it enables us to compare the gradient variance of MCSAmethods.
We empirically justify the bounds obtained using Assumption 3 in Section 6.2.
MSC (Algorithm 3 in Appendix B) is a simple instance of MCSA where 𝜂 𝑡 = 𝐳 𝑡 and 𝑃 𝛌 𝑡 = 𝐾 𝛌 𝑡 ,where 𝐾 𝛌 𝑡 is the conditional importance sampling (CIS) kernel (originally proposed by Andrieuet al.
where the proposals are generated from 𝑞 (⋅; 𝛌 𝑡 ) .
Although MSC uses only a singlesample for the Markov chain, the CIS kernel internally uses 𝑁 proposals to generate a singlesample.
Therefore, 𝑁 in MSC has a different meaning, but it still indicates the computationalbudget.
Theorem 1.
MSC (Naesseth et al. , 2020) is obtained by defining
[Equation]
with 𝛈 𝑡 = 𝐳 𝑡 where 𝐾 𝛌 (𝐳, ⋅) is the CIS kernel with 𝑞 def. (⋅; 𝛌) as its proposal distribution.
Then, 162 given Assumption 2 and 3, the mixing rate and the gradient bounds are given as [Equation] and 𝔼[‖𝐠(𝛌,𝝶 [Equation]
ℱ
[Equation]
[Equation]
𝐿
[Equation]
where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) .
Proof.
See the full proof in Appendix D.
Discussion Theorem 1 shows that the gradient variance of MSC is insensitive to 𝑁 .
Althoughthe mixing rate does improve with 𝑁 , when 𝑤 ∗ is large due to model misspecification and lack ofconvergence (see the discussion in Section 4.1), this will be marginal.
Overall, the performanceof MSC cannot be improved by increasing the computational budget 𝑁 .
Even under stationarity,the variance of 𝛈 is equal to that of a single posterior sample at best.
4.3 Joint Stochastic Approximation
JSA (Algorithm 4 in Appendix B) was proposed for deep generative models where the likelihoodfactorizes into each datapoint. Then, subsampling can be used through a random-scan versionof the independent Metropolis-Hastings (IMH, Hastings 1970) kernel.
Instead, we consider thegeneral version of JSA with the vanilla IMH kernel since it can be used with any type of likelihood.
At each MCGD step, JSA performs multiple Markov chain transitions and estimates thegradient by averaging all the intermediate states, which is closer to traditional MCMC estimation.
Independent Metropolis-Hastings Similarly to MSC, the IMH kernel in JSA generates pro-
179 posals from 𝑞 (⋅; 𝛌 𝑡 ) .
To show the geometric ergodicity of the implicit kernel 𝑃 , we utilize thegeometric convergence rate of IMH kernels provided by Mengersen & Tweedie (1996, Theorem 181 2.1) and Wang (2020).
Furthermore, to derive an upper bound on the gradient variance, we usethe exact 𝑛 -step marginal IMH kernel derived by Smith & Tierney (1996) as
[Equation]
where 𝑤 (𝐳) = 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) , 𝑥 ∨ 𝑦 = max (𝑥, 𝑦) , and for 𝑅 (𝑤) =
{ 𝐳 ′ ∣ 𝑤 (𝐳 ′ ) ≤ 𝑤 } ,𝑇 𝑛 (𝑤) = ∫ ∞ 𝑤 𝑛 𝑣
[Equation]
𝜆 𝑛−1 (𝑣
[Equation]
𝑑𝑣, and 𝜆(𝑤
[Equation]
= ∫ 𝑅(
[Equation]
[Equation]
[Equation]
Theorem 2.
JSA (Ou & Song, 2020) is obtained by defining𝑃 𝑘𝛌 (𝛈,𝑑𝛈 ′ ) =
𝐾 𝑁(𝑘−1)+1𝛌 (𝐳
[Equation]
,𝑑𝐳 ′
[Equation] [Equation] 𝐾 𝑁(𝑘− [Equation]
+
[Equation] 𝛌 (𝐳 ( [Equation] [Equation]
𝑁𝛌 (𝐳 ( [Equation]
𝑑𝐳
[Equation]
(𝑁) ) with 𝛈 𝑡 = (𝐳 (1)𝑡 , 𝐳 (2)𝑡 , … , 𝐳 (𝑁)𝑡 ) .
Then, given Assumption 2 and 3, the mixing rate and the gradientvariance bounds are
[Equation]
where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) and 𝐶 (𝜌, 𝑁) > 0 is a finite constant depending on 𝜌 and 𝑁 .
Proof.
See the full proof in Appendix D.
Discussion According to Theorem 2, JSA benefits from increasing 𝑁 both in terms of fastermixing rate and lower gradient variance.
However, similarly to MSC, under lack of convergenceand model misspecification (large 𝑤 ∗ ), the improvement becomes marginal.
More importantly,in the large 𝑤 ∗ regime, the variance reduction is limited by the constant 1/2 term .
This is because alarge 𝑤 ∗ results in more Metropolis-Hastings rejections, increasing the correlation between the𝑁 samples.
This inefficiency persists even under stationarity unless 𝜋 (⋅) ∈ 𝒬 and 𝑞 (⋅; 𝛌)
= 𝜋 (⋅) .
Parallel Markov Chain Score Ascent
Our analysis in Section 4 suggests that the statistical performance of both MSC and JSA areheavily affected by model specification and the state of convergence through 𝑤 ∗ .
This affectsboth the mixing rate 𝜌 and the gradient variance 𝐺 .
Furthermore, a large 𝑤 ∗ abolishes our abilityto counterbalance the inefficiency by increasing the computational budget 𝑁 .
However, 𝜌 and𝐺 do not impact convergence equally.
In particular, we noted in Section 3.1 that recent resultson MCGD suggest that MCSA depends more on the gradient variance than the mixing rate.
Weturn to leverage this understanding to overcome the limitations of previous MCSA methods.
5.1 Parallel Markov Chain Score Ascent
We propose a novel scheme, parallel Markov chain score ascent (pMCSA, Algorithm 5 in Appendix B), that embraces a slower mixing rate in order to consistently achieve an 𝒪 ( 1 / 𝑁 ) variancereduction, even on challenging problems with in a large 𝑤 ∗ ,
Algorithm Description Unlike JSA, that uses 𝑁 sequential Markov chain states, pMCSA operates 𝑁 parallel Markov chains.
To maintain a similar per-iteration cost with JSA, it performsonly a single Markov chain transition for each chain.
Since the chains are independent, the
Metropolis-Hastings rejections do not affect the variance of pMCSA.
Theorem 3.
pMCSA, our proposed scheme, is obtained by setting
[Equation]
with 𝛈 = (𝐳 (1) , 𝐳 (2) , … , 𝐳 (𝑁) ) .
Then, given Assumption 2 and 3, the mixing rate and the gradientvariance bounds are
[Equation]
where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) and 𝐶 is some positive constant depending on 𝑁 .
Proof.
See the full proof in Appendix D.
Discussion Unlike JSA and MSC, the variance reduction rate of pMCSA is independent of𝑤 ∗ .
Therefore, it should perform significantly better on challenging practical problems.
If weconsider the rate of Duchi et al.
(2012), the combined rate is constant with respect to 𝑁 since itcancels out.
In practice, however, we observe that increasing 𝑁 accelerates convergence quitedramatically.
Therefore, the mixing rate independent convergence rates by Doan et al.
(2020a,b)appears to better reflect practical performance.
This is because (i) the mixing rate 𝜌 is a conservative global bound and (ii) the mixing rate will improve naturally as MCSA converges.
5.2 Computational Cost Comparison
[Table]
The three schemes using the CIS and IMHkernels have different computational costs depending on 𝑁 as organized in Table 2.
Cost of Sampling Proposals For the CISkernel used by MSC, 𝑁 controls the number of internal proposals sampled from 𝑞 (⋅; 𝛌) .
For JSA and pMCSA, the IMH kernel onlyuses a single sample from 𝑞 (⋅; 𝛌) , but appliesthe kernel 𝑁 times.
On the other hand, pMCSA needs twice more evaluations of 𝑞 (⋅; 𝛌) .
However, this added cost is minimal since itis dominated by that of evaluating 𝑝 (𝐳, 𝐱) .
Cost of Estimating the Score When estimating the score, MSC computes ∇ 𝛌 log 𝑞 (𝐳; 𝛌) onlyonce, while JSA and our proposed scheme compute it 𝑁 times.
However, Naesseth et al.
(2020)also discuss a Rao-Blackwellized version of the CIS kernel, which also computes the score 𝑁times.
Lastly, notice that MCSA methods do not need to differentiate through the likelihood𝑝 (𝐳, 𝐱) , unlike ELBO maximization, making its per-iteration cost significantly cheaper.
[Figure] [Figure]
Evaluations
6.1 Experimental Setup
Implementation For the realistic experiments, we implemented MCSA methods on top ofthe
Turing (Ge et al. , 2018) probabilistic programming framework.
For the variational family, weuse diagonal multivariate Gaussians with the support transformation of Kucukelbir et al.
We use the ADAM optimizer by Kingma & Ba (2015) with a stepsize of 0.01 in all experiments.
The budget is set to 𝑁 = 10 for all experiments unless specified.
Baselines We compare (i) pMCSA (ours, Section 5), (ii) JSA (Ou & Song, 2020),(iii)
MSC (Naesseth et al. , 2020), (iv) MSC with with Rao-Blackwellization ( MSC-RB , Naesseth et al. 2020), and (v) evidence lower-bound maximization ( ELBO , Kucukelbir et al. 2017;
Ranganath et al. 2014) with the path derivative estimator (Roeder et al. , 2017).
6.2 Simulations
Setup First, we verify our theoretical analysis on multivariate Gaussians with full-rank covariances sampled from Wishart distribution with 𝜈 degrees of freedom (values of 𝜈 are in the figurecaptions).
This problem is challenging since an IMH (used by pMCSA, JSA) or CIS (used by
MSC, MSC-RB) kernel with a diagonal Gaussian proposal will mix slowly due to a large 𝑤 ∗ .
Gradient Variance We evaluate our theoretical analysis of the gradient variance.
The variance is estimated from 512 independent Markov chains using the parameters generated by themain MCSA procedure.
The estimated variances are shown in Figure 1.
We make the followingobservations: (i) pMCSA has the lowest variance overall, and it consistently benefits from increasing 𝑁 .
(ii) MSC does not benefit from increasing 𝑁 whatsoever.
(iii) JSA does not benefitfrom increasing 𝑁 until 𝑞 (⋅; 𝛌) has sufficiently converged (when 𝑤 ∗ has become small).
Theseresults confirm our theoretical results in Sections 4 and 5.
Robustness Against Optimizers Since the convergence of most sophisticated SGD optimizers has yet to be established for MCGD, we empirically investigate their effectiveness.
The results using SGD (Bottou et al. , 2018; Robbins & Monro, 1951), Momentum (Polyak, 1964), Nesterov (Nesterov, 1983), ADAM (Kingma & Ba, 2015), and varying stepsizes are shown in Figure 2.
Clearly, pMCSA successfully converges for the broadest variety of optimizer settings.
Overall,most MCSA methods seem to be the most stable with ADAM, which points out that establishingthe convergence of ADAM for MCGD will be a promising direction for future works.
[Table]
4: Test Log Predictive Density on Robust Gaussian Process Regression
[Table]
6.3 Bayesian Neural Network Regression
Setup For realistic experiments, we train Bayesian neural networks (BNN, Neal 1996) for regression.
We use datasets from the UCI repository (Dua & Graff, 2017) with 90% random traintest splits.
We run all methods with 𝑇 = 5 ⋅ 10 4 iterations.
For the model, we use the priors andforward propagation method of Hernandez-Lobato & Adams (2015) with a 50 -unit hidden layer(see Appendix C.1).
Results The results are shown in Table 3.
pMCSA achieves the best performance comparedto all other MCSA methods.
Also, its overall performance is comparable to exclusive KL minimization methods (ELBO) unlike other MCSA methods.
Furthermore, on airfoil and energy ,pMCSA improves over ELBO by 0.29 nat and 0.48 nat .
Even on gas where pMCSA did not beat
ELBO, its performance is comparable, and it dominates all other MCSA methods by roughly 0.4nat .
Additional experimental results can be found in Appendix E.
6.4 Robust Gaussian Process Regression
Setup We train Gaussian processes (GP) with a Student-T likelihood for robust regression.
Weuse datasets from the UCI repository (Dua & Graff, 2017) with 90% random train-test splits.
Weuse the Matérn 5/2 covariance kernel with automatic relevance determination (Neal, 1996) (see
Appendix C.2).
We run all methods with 𝑇 = 2⋅10 4 iterations.
For prediction, we use the modeof 𝑞 (⋅; 𝛌) for the hyperparameters and marginalize the latent function over 𝑞 (⋅; 𝛌) (Rasmussen &
Williams, 2006).
We consider ELBO with only 𝑁 = 1 since differentiating through the likelihoodmakes its per-iteration cost comparable to MCSA methods with 𝑁 = 10 .
Results The results are shown in Table 4.
Except for gas , pMCSA achieves better performancethan all other methods.
This suggests that, overall, the exclusive KL may be less effective for GPposteriors.
Although ELBO achieves the best performance on gas , pMCSA dominates other
MCSA methods.
Our encouraging regression results suggest that incorporating methods suchas inducing points (Snelson & Ghahramani, 2005) into MCSA may lead to an important newclass of GP models.
Additional experimental results can be found in Appendix E.2.
Related Works
Inclusive KL minimization Our MCSA framework generalizes MSC (Naesseth et al. ,
2020)and JSA Ou & Song (2020), which are inclusive KL minimization based on SGD and Markovchains.
Similar to MCSA is the method of Li et al.
However, the convergence of thismethod is not guaranteed since it uses short Markov chains, disqualifying for MCSA.
Othermethods based on biased gradients have been proposed by Bornschein & Bengio (2015);
Le et al.(2019), but these are specific for deep generative models.
On a different note, Jerfel et al.
(2021)use boosting instead of SGD to minimize the inclusive KL, which gradually builds a complexvariational approximation from a simple variational family.
Beyond the KL Divergence Discovering alternative divergences for VI has been an activeresearch area.
For example, the 𝜒 2 (Dieng et al. , 2017), 𝑓 (Wan et al. , 2020; Wang et al. , 2018),𝛼 (Hernandez-Lobato et al. , 2016; Li & Turner, 2016; Regli & Silva, 2018), reguarlized importance ratio (Bamler et al. , 2017) divergences have been studied for VI.
However, for gradientestimation these methods involve the importance ratio 𝑤 (𝐳)
= 𝜋 (𝐳) /𝑞
(𝐳) , which leads to significant variance and low signal-to-noise ratio under model misspecification (Bamler et al. , 2017;
Geffner & Domke, 2021a,c).
In contrast, under stationarity, the variance of pMCSA is 𝜎 2 /𝑁
( 𝜎 is the variance of the score over the posterior) regardless of model misspecification.
Meanwhile, Geffner & Domke (2021b); Ruiz & Titsias (2019); Salimans et al.
; Zhang et al.
(2021)construct implicit divergences formed by MCMC.
With the exception of Ruiz & Titsias (2019),most of these approaches aim to maximize auxiliary representations of the classic ELBO.
Therefore, their property is likely to be similar to the exclusive KL.
Adaptive MCMC and MCSA As pointed out by Ou & Song (2020), using 𝑞 (⋅; 𝛌) within the
MCMC kernel makes MCSA structurally equivalent to adaptive MCMC.
In particular, Andrieu& Thoms (2008); Brofos et al.
; Gabrié et al.
; Garthwaite et al.
(2016) discuss theuse of stochastic approximation in adaptive MCMC.
Also, Andrieu & Moulines (2006);
Brofoset al.
; Giordani & Kohn (2010); Habib & Barber (2019); Holden et al.
(2009); Keith et al.(2008); Neklyudov et al.
(2019) specifically discuss adapting the propsosal of IMH kernels, andsome of them use KL divergence minimization.
These methods focus on showing ergodicity thesamples ( 𝐳 (𝑛) in our notation) not the convergence of the variational approximation (or proposalin an adaptive MCMC context) 𝑞 (⋅; 𝛌) .
In this work, we focused on the convergence of 𝑞 (⋅; 𝛌) ,which could advance the adaptive MCMC side of the story.
Discussions
This paper presented a new theoretical framework for analyzing inclusive KL divergence minimization methods based on running SGD with Markov chains.
Furthermore, we proposedpMCSA, a new MCSA method that enjoys substantially low variance.
We have shown that thistheoretical improvement translates into better empirical performance.
Limitations Our work has two main limitations.
Firstly, since our work aims to understandexisting MCSA methods, it inherits the current limitations of MCSA methods.
For example,minibatch subsampling is challenging for models with non-factorizable likelihoods (Naessethet al. , 2020).
Secondly, our theoretical analysis in Section 4 requires Assumption 3, which isstrong, but required to connect with MCGD.
An important future direction would be to relaxthe assumptions needed by MCGD.
Towards Alternative Divergences
In Section 6, we have shown that minimizing the inclusive KL is competitive against minimizing the exclusive KL on general Bayesian inference problems.
Although Dhaka et al.
(2021) has shown that the inclusive KL fails on high-dimensionalproblems, this is only the case under the presence of strong correlations.
Before entirely ditchingthe inclusive KL, it is essential to ask, “how correlated posteriors really are in practice?”
Furthermore, the true performance of alternative divergences is often masked by the limitations ofthe inference procedure (Geffner & Domke, 2021a,c).
Given that pMCSA significantly advancesthe best-known performance of inclusive KL minimization, it is possible that similar improvements could be extracted from other divergences.
To conclude, our results motivate furtherdevelopment of better inference algorithms for alternative divergence measures.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect thepaper’s contributions and scope?
[Yes]
See the contribution summary in Section 1 .
(b) Did you describe the limitations of your work?
[Yes]
See Section 8 .
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms
to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
The key assumptions are stated in each proof.
(b) Did you include complete proofs of all theoretical results?
[Yes]
See Appendix D.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[Yes]
It is included in the supplementary material .
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
See Section 6.
Additional details can be found in the code in the supplementary ma-
terial.
(c) Did you report error bars (e.g., with respect to the random seed after running ex-
periments multiple times)?
[Yes]
See the text in Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes]
See Appendix A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new
as-
sets...
(a) If your work uses existing assets, did you cite the creators?
[No]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally iden-
tifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots,
if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation?
[N/A]
[Table] [Table]
B Pseudocodes
B.1 Markov Chain Kernels
Algorithm 1:
Input: previous sample 𝐳 𝑡−1 , previous parameter 𝛌 𝑡−1 , number of proposals 𝑁 𝐳 (0) =
𝐳 𝑡− 𝐳 (𝑖) ∼ 𝑞 def.
(𝐳; 𝛌 𝑡−1 ) for 𝑖 = 1, 2, … , 𝑁 ˜𝑤(𝐳 (𝑖) ) =
𝑝(𝐳 (𝑖) , 𝐱) / 𝑞 def.
(𝐳 (𝑖) ; 𝛌 𝑡−1 ) for 𝑖 = 0, 1, … , 𝑁 𝑤 (𝑖) =
˜𝑤(𝐳 (𝑖) )
∑ 𝑁𝑖=0 ˜𝑤(𝐳 (𝑖) ) for 𝑖 = 0, 1, … , 𝑁 𝐳 𝑡 ∼ Multinomial (𝑤 (0) , 𝑤 (1) , … , 𝑤 (𝑁) )
Algorithm 2:
Input: previous sample 𝐳 𝑡−1 , previous parameter 𝛌 𝑡−1 , 𝐳 ∗ ∼ 𝑞 def.
(𝐳; 𝛌 𝑡−1 ) 𝑤(𝐳) =
𝑝(𝐳, 𝐱)/𝑞 def.
(𝐳; 𝛌 𝑡−1 )
𝛼 = min (𝑤 (𝐳 ∗ )/𝑤 (𝐳 𝑡−1 ), 1)
𝑢 ∼ Uniform (0, 1) if u < 𝛼 then 𝐳 𝑡 = 𝐳 ∗ else 𝐳 𝑡 = 𝐳 𝑡− end
Input: MCMC kernel 𝐾 𝛌 (𝐳, ⋅) , initial sample 𝐳 0 , initial parameter 𝛌 0 , number of iterations 𝑇 , stepsize schedule 𝛾 𝑡 for 𝑡 = 1, 2, … , 𝑇 do 𝐳 𝑡 ∼ 𝐾 𝛌 𝑡−1 (𝐳 𝑡−1 , ⋅) 𝐠
(𝛌) =
𝐬 (𝛌; 𝐳 𝑡 ) 𝛌 𝑡 = 𝛌 𝑡−1 + 𝛾 𝑡 𝐠 (𝛌 𝑡−1 ) end
Input: MCMC kernel 𝐾 𝛌 (𝐳, ⋅) , initial sample 𝐳 (𝑁)0 , initial parameter 𝛌 0 , number of iterations 𝑇 , stepsize schedule 𝛾 𝑡 for 𝑡 = 1, 2, … , 𝑇 do 𝐳 (0)𝑡 = 𝐳 (𝑁)𝑡− for 𝑛 = 1, 2, … , 𝑁 do 𝐳 (𝑛)𝑡 ∼ 𝐾 𝛌 𝑡−1 (𝐳 (𝑛−1)𝑡 , ⋅) end 𝐠 (𝛌) = 1 𝑁 ∑ 𝑁 𝑛=1 𝐬 (𝛌; 𝐳 (𝑛)𝑡 ) 𝛌 𝑡 = 𝛌 𝑡−1 + 𝛾 𝑡 𝐠 (𝛌 𝑡−1 ) end
Input: MCMC kernel 𝐾 𝛌 (𝐳, ⋅) , initial samples 𝐳 (1)0 , … , 𝐳 (𝑁)0 , initial parameter 𝛌 0 , number of iterations 𝑇 , stepsize schedule 𝛾 𝑡 for 𝑡 = 1, 2, … , 𝑇 do for 𝑛 = 1, 2, … , 𝑁 do 𝐳 (𝑛)𝑡 ∼ 𝐾 𝛌 𝑡−1 (𝐳 (𝑛)𝑡−1 , ⋅) end 𝐠 (𝛌) = 1 𝑁 ∑ 𝑁 𝑛=1 𝐬 (𝛌; 𝐳 (𝑛)𝑡 )
𝛌 𝑡 = 𝛌 𝑡−1 + 𝛾 𝑡 𝐠 (𝛌 𝑡−1 ) end
C Probabilistic Models Used in the Experiments
C.1 Bayesian Neural Network Regression
We use the BNN model of Hernandez-Lobato & Adams (2015) defined as𝜆 −1 ∼ inverse - gamma (𝛼 = 6, 𝛽 = 6) 𝛾 −1 ∼ inverse - gamma (𝛼 = 6, 𝛽 = 6) 𝐖 1 ∼ 𝒩 ( 0 , 𝜆 −1 𝐈) 𝐳 = ReLU (𝐖 1 𝐱 𝑖 )
[Equation]
where 𝐱 𝑖 and 𝑦 𝑖 are the feature vector and target value of the 𝑖 th datapoint.
Given the variational distribution of 𝜆 −1 , 𝛾 −1 , 𝐖 1 , 𝐖 2 , we use the same posterior predictive approximationof Hernandez-Lobato & Adams (2015).
We apply z-standardization (whitening) to the features𝐱 𝑖 and the target values 𝑦 𝑖 , and unwhiten the predictive distribution.
C.2 Robust Gaussian Process Logistic Regression
We perform robust Gaussian process regression by using a student-t prior with a latent Gaussianprocess prior.
The model is defined as
[Equation]
The covariance 𝚺 is computed using a kernel 𝑘 (⋅, ⋅)
such that [𝚺] 𝑖,𝑗 = 𝑘 (𝐱 𝑖 , 𝐱 𝑗 ) where 𝐱 𝑖 and𝐱 𝑗 are data points in the dataset.
For the kernel, we use the Matern 5/2 kernel with automaticrelevance determination (Neal, 1996) defined as𝑘
[Equation]
, 𝐱
[Equation]
[Equation] [Equation] [Equation] where 𝑟 = 𝐷 ∑
[Equation]
ℓ 2𝑖 where 𝐷 is the number of dimensions.
The jitter term 𝛿 is used for numerical stability.
We set asmall value of 𝛿 = 1 × 10 −6 .
D Proofs
Proposition 1.
Let 𝛈 =
(𝐳 (1) , 𝐳 (2) , … , 𝐳 (𝑁) ) and a Markov chain kernel 𝑃 𝛌 (𝛈, ⋅) be Π -invariantwhere Π is defined as
[Equation]
Then, by defining the target function 𝑓 and the gradient estimator 𝐠 to be
[Equation]
and
[Equation]
∑
[Equation]
𝛌
[Equation]
where ℍ [ 𝜋 ] is the entropy of 𝜋 , MCGD results in inclusive KL minimization as
[Equation]
and 𝔼 Π [ 𝐠 (𝛌, 𝝶) ]
= ∇ 𝛌 𝑑 KL (𝜋 ∥ 𝑞 (⋅
[Equation]
𝛌)).
Proof of Proposition 1.
For notational convenience, we define the shorthand
[Equation]
Then,
[Equation]
ℍ
[Equation]
𝜋
[Equation]
Pulled out constant
[Equation]
∑ 𝑛
[Equation]
[Equation]
[Equation]
log 𝑞 ( 𝐳 (𝑛) ; 𝛌 ) ) 𝜋 (𝐳 (1∶𝑁) ) 𝑑𝐳 (1∶𝑁) } − ℍ [ 𝜋 ] Swapped integral and sum = 1 𝑁 𝑁 ∑ 𝑛= ∫ ( − log 𝑞 ( 𝐳 (𝑛) ; 𝛌 ) ) 𝜋 (𝐳 (𝑛) ) 𝑑𝐳 (𝑛) − ℍ [ 𝜋 ]
Marginalized 𝐳 (𝑚) for all 𝑚 ≠ 𝑛 =
[Equation]
Definition of entropy ℍ [𝜋]
[Equation]
Definition of 𝑑 KL
=
[Equation]
For 𝔼 Π [ 𝐠 (𝛌, 𝝶) ] , note that
[Equation]
Therefore, it suffices to show that
[Equation]
Equation (4)
[Equation]
Leibniz derivative rule
[Equation]
Equation (5)
Proposition 2.
The maximum importance weight 𝑤 ∗ =
sup 𝐳 𝑤 (𝐳) =
sup 𝐳 𝜋 (𝐳) /𝑞
(𝐳; 𝛌) isbounded below exponentially by the KL divergence as
[Equation]
Proof of Proposition 2.
[Equation]
Lemma 1.
For the probability measures 𝑝 1 , … , 𝑝 𝑁 and 𝑞 1 , … , 𝑞 𝑁 defined on a measurable space( X , 𝒜) and an arbitrary set 𝐴 ∈ 𝒜 ,
[Equation]
Proof of Lemma 1.
By using the following shorthand notations
[Equation]
the result follows from induction as
[Equation]
1 ) − 𝑞 1 (𝑑𝑥 1 ) || | ∫ 𝐴 𝑁−1 𝑝 (2∶𝑁) (𝑑𝑥 (2∶𝑁) )
+ ∫ 𝐴 𝑞 1 (𝑑𝑥 1 ) | | | ∫ 𝐴 𝑁− 𝑝 (2∶𝑁) (𝑑𝑥 (2∶𝑁) )
− 𝑞 (2∶𝑁) (𝑑𝑥 (2∶𝑁) ) | | |
Triangle inequality ≤ || | ∫ 𝐴 𝑝 1 (𝑑𝑥 1 ) − 𝑞 1 (𝑑𝑥 1 ) || | + || | ∫ 𝐴 𝑁−1 𝑝 (2∶𝑁) (𝑑𝑥 (2∶𝑁) )
− 𝑞 (2∶𝑁) (𝑑𝑥 (2∶𝑁) ) || |.
Applied 𝑝 𝑛 (𝐴) , 𝑞 𝑛 (𝐴) ≤
Theorem 1.
MSC (Naesseth et al. , 2020) is obtained by defining
[Equation]
with 𝛈 𝑡 = 𝐳 𝑡 where 𝐾 𝛌 (𝐳, ⋅) is the CIS kernel with 𝑞 def. (⋅; 𝛌) as its proposal distribution.
Then, 617 given Assumption 2 and 3, the mixing rate and the gradient bounds are given as
[Equation]
𝑘 and 𝔼[‖𝐠
[Equation] 𝛌,𝝶)‖ 2 ∗ ||ℱ 𝑡 ] ≤ 𝐿 2 , where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) .
Proof of Theorem 1.
MSC is described in Algorithm 3.
At each iteration, it performs a single
MCMC transition with the CIS kernel where it internally uses 𝑁 proposals.
Ergodicity of the Markov Chain
The ergodic convergence rate of 𝑃 𝛌 is equal to that of 𝐾 𝛌 , the
CIS kernel proposed by Naesseth et al.
Although not mentioned by Naesseth et al.
(2020),this kernel has been previously proposed as the iterated sequential importance resampling (iSIR) by Andrieu et al.
(2018) with its corresponding geometric convergence rate as
[Equation]
Bound on the Gradient Variance The bound on the gradient variance is straightforwardgiven Assumption 3.
For simplicity, we denote the rejection state as 𝐳 (1) = 𝐳 𝑡−1 .
Then,𝔼
[Equation]
Andrieu et al. , 2018)
[Equation]
Assumption
[Equation]
The sum of weights is= 𝐿 2 .
Lemma 2.
For 𝑤 ∗ = sup 𝐳 𝑤 (𝐳) , 𝜆 (⋅) in Equation (3) is bounded as
[Equation]
Proof of Lemma 2.
The proof can be found in the proof of Theorem 3 of Smith & Tierney (1996).
Lemma 3.
For 𝑤 ∗ = sup 𝐳 𝑤 (𝐳) , 𝑇 𝑛 (⋅) in Equation (3) is bounded as
[Equation]
Proof of Lemma 3.
[Equation]
Pulled out constant
[Equation]
Solved indefinite integral
=
[Equation]
This upper bound is in general difficult to improve unless we impose stronger assumptions on𝜋 and 𝑞 .
Lemma 4.
For a positive test function 𝑓 ∶ 𝒵 → ℝ + , the estimate of a 𝜋 -invariant independent
Metropolis-Hastings kernel with a proposal 𝑞 is bounded as
[Equation]
where 𝑤 (𝐳) = 𝜋 (𝐳) /𝑞 (𝐳) and 𝑤 ∗ = sup 𝐳 𝑤 (𝐳) .
Proof of Lemma 4.
[Equation]
Equation
[Equation] [Equation]
Lemma
[Equation]
Pulled out constant
[Equation]
Definition of 𝑤 (𝐳)
[Equation]
Lemma
[Equation]
Theorem 2.
JSA (Ou & Song, 2020) is obtained by defining
[Equation]
with 𝛈 𝑡 = (𝐳 (1)𝑡 , 𝐳 (2)𝑡 , … , 𝐳 (𝑁)𝑡 ) .
Then, given Assumption 2 and 3, the mixing rate and the gradientvariance bounds are
[Equation]
where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) and 𝐶 (𝜌, 𝑁) > 0 is a finite constant depending on 𝜌 and 𝑁 .
Proof of Theorem 2.
JSA is described in Algorithm 4.
At each iteration, it performs 𝑁 MCMCtransitions, and uses the 𝑁 samples to estimate the gradient.
Ergodicity of the Markov Chain
The state transitions of the Markov chain samples 𝐳 (1∶𝑁)are visualized as
[Equation]
𝑡
[Equation]
⋮ ⋮
[Equation]
[Table]
where 𝐾 𝛌 (𝐳, ⋅) is an IMH kernel.
Therefore, the 𝑛 -step transition kernel for the vector of the
Markov-chain samples 𝛈
= 𝐳 (1∶𝑁) is represented as
[Equation]
Now, the convergence in total variation 𝑑 TV (⋅, ⋅) can be shown to decrease geometrically as
[Equation]
TV (𝑃 𝑘𝛌 (𝛈, ⋅) , Π)
[Equation]
Definition of 𝑑 TV
[Equation]
Lemma
[Equation]
Definition of 𝑑 TV
[Equation]
Geometric ergodicity
[Equation]
Solved sum
[Equation]
Although the constant depends on 𝜌 and 𝑁 , the kernel 𝑃 is geometrically ergodic and converges𝑁 times faster than the base kernel 𝐾 .
Bound on the Gradient Variance To analyze the variance of the gradient, we require detailed information about the 𝑛 -step marginal transition kernel, which is unavailable in generalfor most MCMC kernels.
Fortunately, specifically for the IMH kernel, Smith & Tierney (1996) have shown that the 𝑛 -step marginal IMH kernel is given as Equation (2).
From this, we showthat
[Equation]
Triangle inequality
[Equation]
Linearity of expectation
[Equation]
Lemma
[Equation]
𝑛
[Equation]
Assumption
[Equation]
Moved constant forward
[Equation]
Solved sum
[Equation]
Laurent series expansion at 𝑤 ∗ → ∞
[Equation]
The Laurent approximation becomes exact as 𝑤 ∗ → ∞ , which is useful as a consequence of
Proposition 2.
Theorem 3.
pMCSA, our proposed scheme, is obtained by setting
[Equation]
with 𝛈 = (𝐳 (1) , 𝐳 (2) , … , 𝐳 (𝑁) ) .
Then, given Assumption 2 and 3, the mixing rate and the gradientvariance bounds are
[Equation]
where 𝑤 ∗ = sup 𝐳 𝜋 (𝐳) /𝑞 def.
(𝐳; 𝛌) and 𝐶 is some positive constant depending on 𝑁 .
Proof of Theorem 3.
Our proposed scheme, pMCSA, is described in Algorithm 4.
At each iteration, our scheme performs a single MCMC transition for each of the 𝑁 samples, or chains, toestimate the gradient.
Similarly to JSA, we use the IMH kernel 𝐾 𝛌 .
Ergodicity of the Markov Chain Since our kernel operates the same MCMC kernel 𝐾 𝛌 foreach of the 𝑁 parallel Markov chains, the 𝑛 -step marginal kernel 𝑃 𝛌 can be represented as
[Equation]
Then, the convergence in total variation 𝑑 TV (⋅, ⋅) can be shown to decrease geometrically as
[Equation]
Definition of 𝑑 TV
[Equation]
Lemma
[Equation]
Definition of TV distance
[Equation]
Geometric ergodicity
[Equation]
Solved sum .
Bound on the Gradient Variance The bound on the gradient variance can be derived insimilar manner to JSA as
[Equation]
Triangle inequality
[Equation]
Linearity of expectation
[Equation]
Lemma
[Equation]
Assumption
[Equation]
Moved constant forward
[Equation]
Solved sum
E Additional Experimental Results
E.1 Bayesian Neural Network Regression
T e s t
R M S
E 1.
Iteration 1 10k20k30k40k50k pMCSAJSAMSCMSC -RBELBO N=
ELBO N=
T e s t
L P D −4.
−3. −3.
−2. −2.
Iteration 1 10k20k30k40k50k 29 min.
T e s t
L P D −4.
−3. −3.
−2. −2.
[Table]
Wall Clock Time (min.)
[Table]
(a) yacht
T e s t
R M S
E 5.
Iteration 1 10k20k30k40k50k pMCSA
JSAMSCMSC-RB
ELBO N=
 ELBO N=
T e s t
L P D −3.
−3. −3.
−3.
Iteration 1 10k20k30k40k50k 68 min.
T e s
[Table]
P D −3.
−3.
−3. −3.
[Table]
Wall Clock Time (min.)
(b) concrete
T e s t
R M S
E 2.
Iteration 1 10k20k30k40k50k pMCSAJSAMSCMSC -RBELBO N=
ELBO N=
T e s t
L P D −3.
−2.
−2. −2.
−2.
Iteration 1 10k20k30k40k50k 88 min.
T e s t
L P D −3.
−2.
−2. −2.
Wall
[Figure]
MSCMSC
[Figure]
RB
[Figure]
t
[Figure]
P D −
[Figure]
T e s t
R M S
E 0.
Iteration 1 10k20k30k40k50k pMCSAJSAMSCMSC -RBELBO N=
ELBO N=
T e s t
L P D −1.
−1. −1.
−0. −0.
Iteration 1 10k20k30k40k50k 101 min.
T e s t
L P D −1.
−1. −1.
−0. −0.
Wall Clock Time (min.)
(a) wine
T e s t
R M S
E 3.
Iteration 1 10k20k30k40k50k pMCSAJSAMSCMSC -RBELBO N=
ELBO N=
T e s t
L P D −3.
−3.
−2. −2.
Iteration 1 10k20k30k40k50k 41 min.
T e s t
L P D −3.
−3.
−2. −2.
Wall Clock Time (min.)
(b) boston
T e s t
R M S
E 0.
Iteration 1 10k20k30k40k50k pMCSAJSAMSCMSC -RBELBO N=
ELBO N=
T e s t
L P D −3.
−2. −2.
−1. −1.
Iteration 1 10k20k30k40k50k 289 min.
T e s t
L P D −3.
−2. −2.
−1. −1.
Wall Clock Time (min.)
(c) sml
T e s
[Figure]
Gaussian Process Regression
T
[Figure]
R
[Figure]
E
[Figure]
Iteration
[Figure]
pMCSAJSAMSCMSC -RBELBO
T e s t
L P D −
[Table]
−
[Table]
−3.
[Table]
−3.
[Table]
Iteration
[Table]
[Figure]
[Table]
[Figure]
T e s t
L P D
[Table]
[Figure]
[Table]
[Figure]
T
[Figure]
s t
R
[Figure]
S
E
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
pMCSAJSAMSCMSC -RB ELBO
T
[Table]
s t
L P D
[Table]
Iteration
[Table]
T e s t
L P D −
[Table]
−
[Table]
[Table] [Figure]
[Table]
[Figure]
T
[Figure]
[Table]
R M S
E
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Iteration 5k 10k 15k 20k pMCSAJSAMSCMSC -RBELBO
T e s t
L P D −4.
[Table]
[Figure]
[Table]
Iteration
[Table]
T e s t
L P D
[Table]
[Figure]
[Table]
T e s t
R M S
E 0.
[Figure] [Figure]
[Figure]
Iteration 5k 10k 15k 20k
[Figure] [Figure]
[Table]