Local-Global MCMC kernels: the best of both worlds
Anonymous Author(s)
Abstract
Recent works leveraging learning to enhance sampling have shown promisingresults, in particular by designing effective non-local moves and global proposals.
However, learning accuracy is inevitably limited in regions where little datais available such as in the tails of distributions as well as in high-dimensionalproblems.
In the present paper we study an Explore-Exploit Markov chain Monte
Carlo strategy ( Ex 2 MCMC ) that combines local and global samplers showingthat it enjoys the advantages of both approaches.
We prove V -uniform geometricergodicity of Ex 2 MCMC without requiring a uniform adaptation of the globalsampler to the target distribution.
We also compute explicit bounds on the mixingrate of the Explore-Exploit strategy under realistic conditions.
Moreover, we alsoanalyze an adaptive version of the strategy ( FlEx 2 MCMC ) where a normalizingflow is trained while sampling to serve as a proposal for global moves.
We illustratethe efficiency of Ex 2 MCMC and its adaptive version on classical sampling benchmarks as well as in sampling high-dimensional distributions defined by Generative
Adversarial Networks seen as Energy Based Models.
Introduction
We consider the setting where a target distribution π on a measurable space ( X , X ) is known upto a normalizing constant and one tries to estimate the expectations of some function f : X → Rwith respect to π .
Examples include the extraction of Bayesian statistics from posterior distributionsderived from observations as well as the computation of observables of a physical system x ∈ X underthe Boltzmann distribution with non-normalized density π ( x ) =
e − βU ( x ) for the energy function Uat the inverse temperature β .
A common strategy to tackle this estimation is to resort to Markov chain Monte Carlo algorithms(MCMCs).
The MCMC approach aims to simulate a realization of a time-homogeneous Markovchain { Y n , n ∈ N } such that the distribution of the n -th iterate Y n with n → ∞ is arbitrarily closeto π , regardless of the initial distribution of Y 0 .
In particular, the Metropolis-Hastings kernel (MH) isthe cornerstone of MCMC simulations, with a number of successful variants following the processof a proposal step followed by an accept/reject step (see e.g. [62]).
In large dimensions, proposaldistributions are typically chosen to generate local moves that depend on the last state of the chain inorder to guarantee an admissible acceptance rate.
However, local samplers suffer from long mixingtimes as exploration is inherently slow, and mode switching, when there is more than one, can beextremely infrequent.
On the other hand, independent proposals are able to generate more global updates, but they aredifficult to design.
Developments in deep generative modelling, in particular versatile autoregressiveand normalising flows [39, 37, 20, 55], spurred efforts to use learned probabilistic models to improvethe exploration ability of MCMC kernels.
Among a rapidly growing body of work, references include[36, 2, 53, 25, 33].
While these works show that global moves in a number of practical problems canbe successfully informed by machine learning models, it remains the case that the acceptance rate of independent proposals decreases dramatically with dimensions – except in the unrealistic case thatthey perfectly reproduce the target.
This is a well-known problem in the MCMC literature [12, 71, 1],and it was recently noted that deep learning-based suggestions are no exception in works focusing onphysical systems [19, 46].
In this paper we focus on the benefits of combining local and global samplers.
Intuitively, localsteps interleaved between global updates from an independent proposal (learned or not) increaseaccuracy by allowing accurate sampling in tails that are not usually well handled by the independentproposal.
Also, mixing time is usually improved by the local-global combination, which preventslong chains of consecutive rejections.
Here we focus on a global kernel of type iterative-samplingimportance resampling ( i-SIR ) [73, 4, 5].
This kernel uses multiple proposals in each iteration to takefull advantage of modern parallel computing architectures.
For local samplers, we consider commontechniques such as Metropolis Adjusted Langevin (MALA) and Hamiltonian Monte Carlo (HMC).
We call this combination strategy Explore-Exploit MCMC ( Ex 2 MCMC ) in the following.
Contributions The main contributions of the paper are as follows:
• We provide theoretical bounds on the accuracy and convergence speed of Ex 2 MCMC strategies.
In particular, we prove V -uniform geometric convergence of Ex 2 MCMC under assumptions muchmilder than those required to prove uniform geometric ergodicity of the global sampler i-SIR alone.•
We provide convergence guarantees for an adaptive version of the strategy, called FlEx 2 MCMC ,which involves learning an efficient proposal while sampling, as in adaptive MCMC.• We perform a numerical evaluation of Ex 2 MCMC and FlEx 2 MCMC for various sampling problems, including sampling GANs as energy-based models.
The results clearly show the advantages
of the combined approaches
compared to purely local or purely global MCMC methods.
Notations Denote N ∗ =
N \ { 0 } .
For a measurable function f : X (cid:55)→ R , we define | f | ∞ =
sup x ∈ X | f ( x ) | and π ( f ) :
= (cid:82) X
f ( x ) π (d x ) .
For a function V : X (cid:55)→ [1, ∞ ) we introduce the
V - norm of two probability measures ξ and ξ ′ on ( X , X ) , ∥ ξ − ξ ′
∥
V := sup | f ( x ) |≤ V ( x )
| ξ ( f ) − ξ ′
( f ) | .
If V ≡ 1 , ∥ · ∥ 1 is equal to the total variation distance (denoted ∥ · ∥ TV ).
Explore-Exploit Samplers
Suppose we are given a target distribution π on a measurable space ( X , X ) that is known only up toa normalizing constant.
We will often assume that X
=
R d or a subset thereof.
Two related problemsare sampling from π and estimating integrals of a function f : X (cid:55)→ R w.r.t.
π , i.e., π ( f ) .
Amongthe many methods devoted to solving these problems, there is a popular family of techniques basedon Importance Sampling (IS) and relying on independent proposals, see e.g. [1, 74].
We first give abrief overview of IS, to describe the global sampler i-SIR .
We recall ergodicity results for the latterbefore investigating the Explore-Exploit sampling strategy which couples the global sampler with alocal kernel.
Then we present the main theoretical result of the paper on the ergodicity of the coupledstrategy.
2.1 From Importance Sampling to i-SIR
The primary purpose of IS is to approximate integrals of the form π ( f ) .
Its main instrument is a(known) proposal distribution , which we denote by λ (d x ) .
To describe the algorithm, we assumethat π (d x ) =
w ( x ) λ (d x ) /λ ( w ) .
In this formula, w ( x ) is the importance weight function assumedto be known and positive, i.e., w ( x ) > 0 for all x ∈ X , and λ ( w ) is the normalizing constant ofthe distribution π .
Typically λ ( w ) is unknown.
If we assume that π and λ have positive densitiesw.r.t.
a common dominant measure, denoted also by π and λ respectively, then the self-normalizedimportance sampling (SNIS, see [61]) estimator of π ( f ) is given by
[Equation]
where X 1: N i.i.d.
∼ λ , and ω iN = w ( X i ) /
(cid:80) Nj =1 w ( X j ) are the self-normalized importance weights.
Note that computing ω
i N does not require the knowledge of λ ( w ) .
The main problem in thepractical applications of IS is the choice of the proposal distribution λ .
The representationπ (d x ) = w ( x ) λ (d x ) /λ ( w ) implies that the support of λ covers the support of π .
At the sametime, too large variance of λ is obviously detrimental to the quality of (1).
This suggests adaptive importance sampling techniques (discussed in [16]), which involve learning the proposal λ to improvethe quality of (1).
We return to this idea in section 3.
1 Procedure i-SIR ( Y k , λ ) :
Input :
; proposal
Output :
Set X 1 k
+
=
Y k , draw X 2: N k +1 ∼ λ ; for i
∈
[ N ] do compute the normalized weights ω
i , k +1 = w ( X ik +1 ) / (cid:80)
Nℓ =1 w ( X ℓk +1 ) ;
Draw the proposal index
I k +1 ∼ Cat( ω 1, k +1 , . .
. , ω N , k +1 ) ;
Set Y k +1 :
=
X I k
+1 k +1 .
1 Procedure Ex 2 MCMC (
Draw Y k +1 ∼ R
( Z k +1 , · ) .
IS -based techniques can also be used to draw an (approximate) sample from π .
For instance,
Sampling Importance Resampling (SIR, [68]) follows the steps:
Draw X 1: N i.i.d.
∼ λ ;2.
Compute the self-normalized importance weights ω iN = w ( X i ) / (cid:80)
Nℓ =1 w ( X ℓ ) , i ∈ { 1, . . .
, N } ;3.
Select M samples Y 1: M from the set X 1: N choosing X
i with probability ω iN with replacement.
The drawback of the procedure is that it is only asymptotically valid with N → ∞ .
Alternatively,
SIR can be repeated to define a Markov Chain as in iterated SIR ( i-SIR ), proposed in [73] andalso studied in [4, 43, 42, 5].
At each iteration of i-SIR described in Algorithm 1, a candidate pool
X 2: N k +1 is sampled from the proposal and the next state Y k +1 is choosen among the candidates and theprevious state
X 1 k +
=
Y k according to the importance weights.
i-SIR shares similarities with the
Multiple-try Metropolis (MTM) algorithm [44], but is computationally simpler and exhibits morefavorable mixing properties; see Appendix A.1.
The Markov chain { Y k , k ∈ N } generated by i-SIRhas the following Markov kernel
[Equation]
Interpreting i-SIR as a systematic-scan two-stage Gibbs sampler (see Appendix A.2 for more details),it follows easily that the Markov kernel P N is reversible w.r.t.
the target π , Harris recurrent andergodic (see Theorem 5).
Provided also that | w | ∞ < ∞ , it was shown in [5] that the Markov kernel
P N is uniformly geometrically ergodic.
Namely, for any initial distribution ξ on ( X , X ) and k ∈ N ,∥ ξ P
[Equation]
with
[Equation]
We provide a simple direct proof of (2) in Appendix B.1.
Yet, note that the bound (2) reliessignificantly on the restrictive condition that weights are uniformly bounded | w | ∞ < ∞ .
Moreover,even when this condition is satisfied, the rate κ N can be close to 1 when the dimension d is large.
We illustrate this phenomenon on a Gaussian target in Appendix E.2 Figure 7 with an experiment thatalso contrasts the degradation as dimension grows of the purely global sampler with the robustness ofthe local-global kernels analyzed in the next section.
2.2 Coupling with local kernels: Ex 2 MCMC
After each i-SIR step, we apply a local MCMC kernel R (rejuvenation kernel), with an invariant distribution π .
We call this startegy Ex 2 MCMC because it combines steps of exploration by i-SIR and steps of exploitation by the local MCMC moves.
The resulting algorithm, formulated in
Algorithm 2, defines a Markov chain { Y j , j ∈ N } with Markov kernel K N ( x , · ) =
P N R ( x , · ) = 117 (cid:82)
P N ( x , d y ) R ( y , · ) .
We now present the main theoretical result of this paper on the properties of Ex 2 MCMC .
Underrather weak conditions, provided that R is geometrically regular (see [21, Chapter 14]), it is possibleto establish that Ex 2 MCMC remains V -uniformly geometrically ergodic even if the weight functionw ( x ) is unbounded.
Definition 1 ( V -Geometric Ergodicity) .
A Markov kernel Q with invariant probability measure π is
V -geometrically ergodic if there exist constants ρ ∈ (0, 1) and M < ∞ such that
, for all x ∈ X andk ∈ N , ∥ Q k ( x , · ) − π ∥
V ≤ M { V ( x ) + π ( V ) } ρ k .
In particular, V -geometric ergodicity ensures that the distribution of the k -th iterate of a Markov chainconverges geometrically fast to the invariant probability in V -norm, for all starting points x ∈ X .
Here the dependence on the initial state x appears on the right-hand side only in V ( x ) .
Denoteby Var λ
[ w ] = (cid:82) { w ( x ) − λ ( w ) } 2 λ (d x ) the variance of the importance weight functions under theproposal distribution and consider the following assumptions:
A1.
(i) R has π as its unique invariant distribution; (ii) There exists a function V : X → [1, ∞ ) , suchthat for all r ≥ r R > 1 there exist
λ R , r ∈ [0, 1) , b R , r < ∞ , such that R V ( x ) ≤ λ R , r V ( x ) + b R , r 1 V r ,where V r = { x : V ( x ) ≤ r } ;A2.
(i)
For all r ≥ r R , w ∞ , r := sup x
∈ V r { w ( x ) /λ ( w ) } < ∞ and (ii) Var λ
[ w ] / { λ ( w ) } 2 < ∞ .
A 1-(ii) states that R satisfies a Foster-Lyapunov drift condition for V .
This condition is fulfilled bymost classical MCMC kernels - like Metropolis-Adjusted Langevin (MALA) algorithm or Hamiltonian Monte Carlo (HMC), typically under tail conditions for the target distribution; see [63, 22],and [21, Chapter 2] with the references therein.
A 2-(i) states that the (normalized) importance weightsw ( · ) /λ ( w ) are upper bounded on level sets of V r .
This is a mild condition: if X = R d , and V isnorm-like, then the level sets V r are compact and w ( · ) is bounded on V r as soon as π and λ arepositive and continuous.
A 2-(ii) states that the variance of the importance weights is bounded; notethat this variance is also equal to the χ 2 -distance between the proposal and the target distributionswhich plays a key role in the non-asymptotic analysis of the performance of IS methods [1, 70].
Theorem 2.
Assume A 1 and A 2.
Then, for all x ∈ X and k ∈ N ,
[Equation]
where the constant c K N , ˜ κ K N ∈ [0, 1) are given in the proof.
In addition, c K N = c K ∞ + O ( N − 1 )and ˜ κ K ∞ =
˜ κ K N + O ( N − 1 ) with explicit expressions provided in (13) .
The proof of Theorem 2 is provided in Appendix B.2.
We stress that in many situations, the mixingrate ˜ κ K N of the Ex 2 MCMC Markov Kernel K N is significantly better than the corresponding mixingrate of the local kernel R , provided N is large enough.
This is due to the fact that assumptions A 1 and
A 2 do not require to identify the small sets of the rejuvenation kernel R (see [21, Definition 9.3.5]).
At the same time, the quantitative bounds on the mixing rates relies on the constants appearing in thesmall set condition, see [21, Theorem 19.4.1].
Focusing on MALA (see, e.g. [66]) as the rejuvenationkernel R we detail bounds in Appendix C and prove in Theorem 20 that the ratio of mixing times of
K N is typically very favorable compared to MALA provided that N is large enough.
Adaptive version: FlEx 2 MCMC
The performance of proposal-based samplers depends on the distribution of importance weightswhich is related to the similarity of the proposal and target distributions 2 .
Therefore, yet anotherstrategy to improve sampling performance is to select the proposal distribution λ from a family ofparameterized distributions { λ θ } and fit the parameter θ ∈ Θ
= R q to the target π , for example, byminimizing a Kullback-Leibler divergence (KL) [57, 2, 50] or matching moments [59].
In adaptive
MCMCs , parameter adaptation is performed along the MCMC run [6, 9, 64].
In this section wepropose an adaptive version of Ex 2 MCMC , which we call FlEx 2 MCMC .
Normalizing flow proposal.
A flexible way to parameterize proposal distributions is to combinea tractable distribution φ with an invertible parameterized transformation.
Let T : X (cid:55)→ X be a
C 1 diffeomorphism.
We denote by T # φ the push-forward of φ under T , that is, the distributionof X = T ( Z ) with Z ∼ φ .
Assuming that φ has a p.d.f.
(also denoted φ ), the correspondingpush-forward density (w.r.t.
the Lebesgue measure) is given by λ T ( y ) =
φ (cid:0) T − 1 ( y ) (cid:1) J T − 1 ( y ) , where
J T denotes the Jacobian determinant of T .
The parameterized family of diffeomorphisms { T θ } θ ∈ Θdefines a family of distributions { λ T θ } θ ∈ Θ , denoted for simplicity as { λ θ } θ ∈ Θ .
This construction iscalled a normalizing flow (NF) and a great deal of work has been devoted to ways of parameterizinginvertible flows T θ with neural networks; see [40, 55] for reviews.
Simultaneous learning and sampling.
As with adaptive MCMC methods, the parameters ofa NF proposal are learned for the global proposal during sampling, see also [25].
We workwith M copies of the Markov chains { ( Y k
[ j ], X 1: N k
[ j ]) } k ∈ N ∗ indexed by j ∈ { 1, . . . , M } .
Ateach step k ∈ N ∗ , each copy is sampled as in Ex 2 MCMC using the NF proposal, independently from the other copies, but conditionally to the the current value of the parametersθ k − 1 .
We then adapt the parameters by taking steps of gradient descent on a convex combination of the forward KL, KL( π || λ θ )
= (cid:82) π
( x ) log( π ( x ) /λ θ ( x ))
d x and the backward KL
KL( λ θ || π ) =
(cid:82) λ θ ( x ) log( π ( x ) /λ θ ( x ))
d x = (cid:82) φ ( z ) log w θ ◦ T θ ( z )d z .
Let { γ k , k ∈ N } be asequence of nonnegative stepsizes and { α k , k ∈ N } be a nondecreasing sequence in [0, 1] withα ∞ =
lim k →∞ α k .
The update rule is θ k
= θ k − 1 + γ k M − 1 (cid:80) Mj
=1 H ( θ k − 1 , X 1: N k
[ j ], Z 2: N k
[ j ])where H ( θ , x 1: N , z 2: N ) =
α k H f
( θ , x 1: N ) + (1 − α k )
H b ( θ , z 2: N ) with
[Equation]
f
[Equation]
θ ,
[Equation] [Equation]
Note that we use a Rao-Blackwellized estimator of the gradient of the forward KL (4) where wefully recycle all the N candidates sampled at each iteration of i-SIR .
The quality of this estimator isexpected to improve along the iterations k of the algorithm as the variance of importance weightsdecreases as the proposal improves.
Note also that using only gradients from the backward KL (5) isprone to mode-collapse [57, 54, 50, 25], hence the need for also using gradients from the forward KL
H f
( θ , x 1: N ) , which requires the simultaneous sampling from π .
See also Appendix E.5 for furtherdiscussions.
The FlEx 2 MCMC algorithm is summarized in Algorithm 3.
Since the parameters of the Markov kernel θ k are updated using samples X 1: N k from the chain,(( Y k , X 1: N k ))
k ∈ N is no longer Markovian.
This type of problems has been considered in [48, 13, 30, 7]and to prove convergence of the strategy we need to strengthen assumptions compared to the previoussection.
A3.
There exists a function W : X → R + such that φ ( W 2 ) =
(cid:82) W 2 ( z ) φ (d z ) < ∞ , and aconstant L < ∞ such that, for all θ , θ ′ ∈ Θ and z ∈ X , ∥∇ θ log π ◦ T θ ( z ) − ∇ θ log π ◦ T θ ′ ( z )
∥ ≤
L ∥ θ − θ ′ ∥
W ( z ) and ∥∇ θ log J T
θ ( z ) − ∇ θ log J T θ ′
( z ) ∥ ≤ L ∥
θ − θ ′ ∥
W ( z ) .
A4.
(i)
For all d ≥ d R , w ∞ , d = sup θ ∈ Θ sup x ∈ V d w θ ( x ) /λ θ ( w θ ) < ∞ and (ii) sup θ ∈ Θ Var φ ( w θ ◦
T θ ) / { λ θ ( w θ ) } 2 < ∞ .
A 3 is a continuity condition on the NF push-forward density w.r.t.
its parameters θ .
A 4 implies thatthe Markov kernel K N , θ = P N , θ R satisfies a drift and minorization condition uniform in θ .
Theorem 3 (simplified) .
Assume
A 1- A
3- A 4 and that (cid:80) ∞
k =0 γ k = ∞ , (cid:80) ∞
k =0 γ 2 k < ∞ andlim k →∞ α k = α ∞ .
Then, w.p. 1, the sequence { θ k , k ∈ N } converges to the set { θ ∈ Θ, 0 =α ∞ ∇ KL( π || λ θ )
+ (1 − α ∞ ) ∇ KL( λ θ || π ) } .
Theorem 3 proves the convergence of the learning of parameters θ to a stationary point of theloss.
The proof is postponed to Appendix D. Note that once the proposal learning has converged,
FlEx 2 MCMC boils back to Ex 2 MCMC with a fixed learned proposal.
Our experiments show thatadaptivity can significantly speed up mixing for i-SIR , especially for distributions with complexgeometries and that the addition of a rejuvenation kernel further improves samples quality.
Related Work
The possibility to parametrize very flexible probabilistic models with neural networks thanks todeep learning has rekindled interest in adapting MCMC kernels; see e.g. [72, 36, 2, 53, 33].
Whilesignificant performance gain were found in problems of moderate dimensions, these learning-based
Input
: M ]
Output
] 1 for j ∈
[ M ] do
Y k
+ j ] =
Ex 2 MCMC ( Y k , T θ k # φ , R )
Draw Z [1 : M ] ∼ φ .
4 Update θ k
= θ k − 1 + γ k M − 1 (cid:80) Mj
=1 H ( θ k − 1 , X 1: N k
[ j ], Z 2: N k
[ j ]) methods were found to suffer from increasing dimensions as fitting models accurately becomes moredifficult [19, 46].
Similarly to FlEx 2 MCMC , a few work proposed adaptive algorithms that alternatesbetween global and local MCMC moves to ensure ergodicity without requiring a perfect learningof the proposal[59, 25].
More precisely, [59] focused on multimodal distributions and analysed amode jumping algorithm using proposals parametrized as mixture of simple distributions.
While [25],closer to this work, introduced a combination of a local and a global sampler leveraging normalizingflows with a more classical choice for the global sampler: independent Metropolis-Hasting (IMH)instead of i-SIR .
The present work builds on these previous propositions of combinations of localand global sampler by clarifying the reasons of their effectiveness through entirely novel detailedmathematical and empirical analyses.
We chose to focus on i-SIR with an adaptive proposal asthe global sampler since (i) the learning component allows to tackle high-dimensional targets, (ii)theoretical guarantees can be obtained for i-SIR whereas IMH is more difficult to analyze, (iii)
IMHand i-SIR (as a multiple-try MCMC) are expected to have similar performances for comparablecomputational budget [11] but IMH is sequential where i-SIR can be parallelized by increasing thenumber N of proposals per iteration.
Another line of work exploits both normalizing flows and common local MCMC kernels for sampling[57, 36, 54, 77], yet following the different paradigm of using the flow as a reparametrization map,a method sometimes referred to as neural transport: the flow T is trained to transport a simpledistribution φ near π , which is equivalent to bringing T − 1 # π (the pushforward of the original targetdistribution π by the inverse flow T − 1 ) close to φ .
If φ is simple enough to be efficiently sampledby local samplers, the hope is that local samplers can also obtain high-quality samples of T − 1 # π –samples which can be transported back through T to obtain samples of π .
This method attempts toreparametrize the space to disentangle problematic geometries for local kernels.
Yet, it is unclear whatwill happen in the tails of the distribution for which the flow is likely poorly learned.
Furthermore,in order to derive an ergodicity theory for these transported samplers, [57] necessitated substantialconstraints on maps (see section 2.2.2.).
Numerical experiments
5.1 Synthetic examples
Multimodal distributions.
Let us start with a toy example highlighting differences between purelyglobal i-SIR , purely local MALA and Ex 2 MCMC combining both.
We consider sampling froma mixture of 3 equally weighted Gaussians in dimension d = 2 .
In Figure 1a, we compare singlechains produced by each algorithms.
The global proposal is a wide Gaussian, with pools of N = 3candidate.
The MALA stepsize is chosen to reach a target acceptance rate of ∼ 0.67 .
This simpleexperiment illustrates the drawbacks of both approaches: i-SIR samples reach all the modes of thetarget, but the chains often get stuck for several steps hindering variability.
MALA allows for betterlocal exploration of each particular mode, yet it fails to cover all the target support.
Meanwhile,
Ex 2 MCMC retains the benefits of both methods, combining the i-SIR -based global exploration with
MALA-based local exploration.
In larger dimensions, an adaptive proposal is necessary.
In Appendix E.5 we show that
FlEx 2 MCMC can mix between modes of a 50 d Gaussian mixture, provided that the rough location of all the modes is known and used to initialize walkers.
We also stress the robustness of theon-the-fly training exploiting running MCMC chains to evaluate the forward KL term of the loss.
To illustrate further the performance of the combined kernel, we keep the 2 d target mixture modelyet assigning the uneven weights (2 / 3, 1 / 6, 1 / 6) to the 3 modes.
We start M chains drawing from
[Figure]
the initial distribution ξ ∼ N (0, 4 I d ) and use the same hyper-parameters as above.
In Figure 1bwe provide a simple illustration to the statement (2) and Theorem 2, namely we compare the targetdensity to the instantaneous distributions for each sampler propagating ξ during burn-in steps.
As
MALA does not mix easily between modes, the different statistical weights of the different modescan hardly be rendered in few iterations and KL and TV distances stalls after a few iterations.
i-SIRcan visit the different modes, yet it does not necessarily move at each step which slows down itscovering of the modes full support, which again shows in the speed of decrease of the TV and KL.
Overcoming both of these shortcomings, Ex 2 MCMC instantaneous density comes much closer tothe target.
Finally, Figure 1c evaluates the same metrics yet for the density estimate obtained withsingle chain samples after burn-in.
Results demonstrate once again the superiority of Ex 2 MCMC .
Further details on these experiments can be found in Appendix E.3.
Distributions with complex geometry.
Next, we turn to highly anisotropic distributions in highdimensions.
Following [52] and [32], we consider the funnel and the banana-shape distributions.
We remind densities in Appendix E.6 along with providing experiments details.
For d ∈ [10; 200] ,we run i-SIR , MALA, Ex 2 MCMC , FlEx 2 MCMC , adaptive i-SIR (using the same proposal as
FlEx 2 MCMC , but without interleaved local steps) and the versatile sampler NUTS [35] as a baseline.
Here the parameter adaptation for FlEx 2 MCMC is performed in a pre-run and parameters are frozenbefore sampling.
For the adaptive samplers, a simple RealNVP-based normalizing flow [20] is usedsuch that total running times, including training, are comparable with NUTS.
For Ex 2 MCMC andi-SIR the global proposal is a wide Gaussian with a pool of N = 2000 candidates drawn at eachiteration.
For MALA we tune the step size in order to keep acceptance rate approximately at0.5 .
We report the average sliced TV distance and ESS in Figure 2 (see Appendix E.1 for metricsdefinition).
In most cases, FlEx 2 MCMC is the most reliable algorithm.
The only exception is atvery high dimension for the banana where NUTS performs the best: in this case, tuning the flow tolearn tails in high-dimension faithfully was costly such that we proceeded to an early stopping tomaintain comparability with the baseline.
Remarkably, FlEx 2 MCMC compensates significantly forthe imperfect flow training, improving over adaptive- i-SIR , but NUTS eventually performs better.
Conversely, for the funnel, most of the improvement comes from well-trained proposal flow, leadingto similar behaviors of adaptive i-SIR and FlEx 2 MCMC , while both algorithms clearly outperforms
NUTS in terms of metrics.
5.2 Sampling from GANs as Energy-based models (EBMs)
Generative adversarial networks (GANs [27]) are a class of generative models defined by a pair of agenerator network G and a discriminator network D .
The generator G takes a latent variable z from aprior density p
( z ) , z ∈ R d , and generates an observation
G ( z ) ∈ R D in the observation space.
Thediscriminator takes a sample in the observation space and aims to discriminate between true examplesand false examples produced by the generator.
Recently, it has been advocated to consider GANs as
Energy-Based Models (EBMs) [75, 17].
Following [17], we consider the EBM model induced by the
GAN in latent space.
Recall that an EBM is defined by a Boltzmann distribution p ( z ) =
e − E
( z ) / Z ,z ∈ R d , where E ( z ) is the energy function and Z is the normalizing constant.
Note that Wasserstein
[Figure]
4 x 1        xi-SIR
[Figure]
4 x
[Figure]
4 x
[Figure] [Figure]
GANs also allow for an energy-based interpretation (see [17]), although the interpretation of thediscriminator in this case is different.
The energy function is given by
[Equation]
for the vanilla Jensen-Shannon and Wasserstein GANs, respectively.
Here logit( y ), y ∈ (0, 1) is theinverse of the sigmoid function and p 0 ( z ) =
N (0, I d ) .
MNIST results.
We consider a simple Jensen-Shannon GAN model trained on the MNIST datasetwith latent space dimension d = 2 .
We compare samples obtained by i-SIR , MALA, and Ex 2 MCMCfrom the energy-based model associated with E JS ( z ) , see (6).
We use a wide normal distributionas the global proposal for i-SIR and Ex 2 MCMC , and pools of candidates at each iteration N = 10 .
The step-size of MALA is tuned to keep an acceptance rate ∼ 0.5 .
We visualize chains of 100 stepsin the latent space obtained with each method in Figure 3.
Note that the poor agreement between theproposal and the landscape makes it difficult for i-SIR to accept from the proposal and for MALAto explore many modes of the latent distribution, as shown in Figure 3.
Ex 2 MCMC combineseffectively global and local moves, encouraging better diversity associated with a better mixingtime.
The images corresponding to the sampled latent space locations are displayed in Figure 4and reflect the diversity issue of MALA and i-SIR .
Further details and experiments are provided in
Appendix E.7.1, including similar results for WGAN-GP [31] and the associated EBM E W ( z ) .
Cifar- 10 results.
We consider two popular architectures trained on Cifar-10, DC-GAN [60] and
SN-GAN [49].
In both cases the dimension of the latent space equals d = 128 .
Together with thenon-trivial geometry of the corresponding energy landscapes, the large dimension makes samplingwith NUTS unfeasible in terms of computational time.
We perform sampling from mentioned
GANs as energy-based models using i-SIR , MALA, Ex 2 MCMC , and FlEx 2 MCMC .
In i-SIRand Ex 2 MCMC we use the prior p
( z ) as a global proposal with a pool of N = 10 candidates.
[Figure] [Figure]
For FlEx 2 MCMC we perform training and sampling simultaneously.
Implementation details areprovided in Appendix E.7.2.
To evaluate sampling quality, we report the values of the energy function
E ( z ) , averaged over 500 independent runs of each sampler.
We also visualize the inception score (IS)dynamics calculated over 10000 independent trajectories.
We present the results in Figure 5 togetherwith the images produced by each sampler.
Note that Ex 2 MCMC and FlEx 2 MCMC reach low levelof energies faster than other methods, and reach high IS samples in a limited number of iterations.
Visualizations indicate that MALA is unlikely to escape the mode of the distribution p ( z ) it startedfrom, while i-SIR and Ex 2 MCMC / FlEx 2 MCMC better explores the target support.
However, globalmove appear to become more rare after some number of iterations for Ex 2 MCMC / FlEx 2 MCMC ,which then exploit a particular mode with MALA steps.
We here hit the following limitation:i-SIR remains at relatively high-energies, failing to explore well modes basins but still acceptingglobal moves, while Ex 2 MCMC / FlEx 2 MCMC explores well modes basins but eventually remainstrapped.
We predict that improving further the quality of the FlEx 2 MCMC proposal by scaling thenormalizing flow architecture would allow for more global moves.
See Appendix E.7.2 for additionalexperiments (including ones with SN-GAN), FID dynamics, and visualizations.
Conclusions and further research directions
The present paper examines the benefits of combining local and global samplers.
From a theoreticalpoint of view, we show that global samplers are more robust when coupled with local samplers.
Namely, a V -geometric ergodicity is obtained for the Ex 2 MCMC kernel under minimal assumptions.
Meanwhile, the global samplers drives exploration when properly adjusted.
Therefore, we alsodescribe the adaptive version FlEx 2 MCMC of the strategy involving the learning of a global proposalparametrized by a normalizing flow.
We also check for the learning convergence along the adaptive
MCMC run.
Finally, a series of numerical experiments confirms the superiority of the strategy,including the high-dimensional examples.
While the startegy was described and analyzed for thei-SIR global kernel, we note that it would be possible to extend the theory to other independentglobal samplers.
We expect that the benefit of the combination would remain.
Further studies of
FlEx 2 MCMC , in particular the derivation of its mixing rate, is an interesting direction for futurework.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] The paper suggests novel MCMC technique and is validated on artificial
and standard datasets.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
and Section 3 in the main text.
(b) Did you include complete proofs of all theoretical results?
[Yes] Yes, the proofs of
Section 2 and Section 3 are provided in Appendix B and Appendix D.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] Code to
reproduce experiments is attached to the supplement.
Due to size constraints, we are
not available to present all the pre-trained GANs models for the section Section 5, but
we intend to do so when possible.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] The hyperparameters are provided in the supplement paper.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Partially yes, but not for all experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We provide this information in the
supplement paper.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We use only the
common knowledge datasets.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A i-SIR Algorithm
A.1 i-SIR and Multiple-try Metropolis (MTM) algorithm
In the MTM algorithm, N i.i.d.sample proposals { X ik +1 } Ni =1 are drawn from a kernel T ( y , · ) in eachiteration.
In a second step, a sample Y ∗ k +1 is selected with probability proportional to the weights (theexact expression of the weighting weights differs from ours, but this does not change the complexityof the algorithm).
In a third step, N − 1 i.i.d.
proposals are drawn from the kernel T ( Y ∗ k +1 , · ) and itis assumed that the move is Y k
+
=
Y ∗ k +1 with an generalised M-H ratio, see [44, eq. 3].
This step isbypassed in i-SIR , reducing the computational complexity by a factor of 2 .
A.2 i-SIR as a systematic scan two-stage Gibbs sampler
We analyze a slightly modified version of the i-SIR algorithm, with an extra randomization of thestate position.
The k -th iteration is defined as follows.
Given a state Y k ∈ X ,
(i) draw I k +1 ∈ { 1, . . .
, N } uniformly at random and set
X I k
+1 k
+
=
Y k ;
(ii) draw X 1: N \{ I k +1 } k +
independently
from
the proposal distribution λ
(iii) compute, for i ∈ { 1, . .
. , N }
, the normalized importance weights
ω iN , k +
[Equation]
w ( X ik +1 ) /
N (cid:88) ℓ = w (
[Equation]
+1 )
; (iv) select Y k +1 from the set X 1: N k +1 by choosing X ik +1 with probability ω iN , k +1 .
Thus, compared to the simplified i-SIR algorithm given in the introduction, the state is inserteduniformly at random into the list of candidates instead of being inserted at the first position.
Of course,this change has no impact as long as we are interested in integrating functions that are permutationinvariant with respect to candidates, which is the case throughout our work.
Still, this randomizationmakes the analysis much more transparent.
In what follows, we show that i-SIR can be interpreted as a systematic-scan two-stage Gibbs samplersampling, which alternately samples from the full conditionals of the extended target φ N , whichis carefully defined below in terms of the state and candidate pool.
Here we essentially follow thework of [73, 4, 5].
This is formalized by a dual representation of φ N , presented below in Theorem 4,which provides the two complete conditionals in question.
We introduce the Markov kernel
[Equation]
on X × X (cid:15) N , which probabilistically describes the candidate selection operation in i-SIR .
Note that by construction, for each y ∈ X , ℓ ∈ { 1, . .
. , N } and nonnegative measurable function h :
X → R + ,
[Equation]
Using the kernel Λ N we may now define properly the extended target φ N as the probability law
[Equation]
on ( X N +1 , X (cid:15) ( N +1) ) .
Note that since for every A ∈ X , φ N (1 A × X ) =
π ( A ) , the target π coincideswith the marginal of φ N with respect to the state.
Moreover, it is easily seen that Λ N provides theconditional distribution, under φ N , of the candidate pool given the state.
hand using π (d y )
δ y (d x i ) =
w ( x i ) λ
(d x i )
δ x i (d y ) /λ ( w ) , the marginal distributionπ N of φ N with respect to x 1: N is given by
[Equation]
where we have set
[Equation]
It is interesting to note that the marginal π N has a probability density function, proportional toΓ N 1 X ( x 1: N )
=
(cid:80) Ni =1 w ( x i ) /N , with respect to the product measure λ (cid:15) N .
Using (7), we immediately obtain the following result.
Theorem 4 (duality of extended target) .
For every N ∈ N ∗ ,
[Equation]
Using this dual representation of φ N , i-SIR can be interpreted as a two-stage Gibbs sampler.
Giventhe state Y k , N candidates X 1: N k +1 are sampled from Λ N ( Y k , · ) .
In a second step, the next state Y k +1is sampled given the current candidates from Π N ( X 1: N k +1 , · ) .
The two-stages Gibbs sampler generatesa Markov chain (( Y k , X 1: N k ))
k ∈ N with Markov kernel
[Equation]
Note that the Markov kernel P N ( y , x 1: N , · ) does not depend on x 1: N , which means that only thestate Y k needs to be stored from one iteration to another.
Given a distribution ξ on ( X n +1 , X ⊗ ( n +1) ) ,we denote by P ξ the distribution of the canonical Markov chain (( Y k , X 1: N k ))
k ∈ N with kernel P N .
With these notations, for any nonnegative measurable function f : X n +1 → R , we get, for k ∈ N ∗ ,
E ξ (cid:2) f ( Y k , X 1: N k
[Equation]
The systematic scan two-stages Gibbs sampler is one of the MCMC algorithm structures that hasgiven rise to many works.
We summarize in the theorem below the important properties of thissampler; see [45], [61, Chapter 9], [3] and the references therein.
Theorem 5.
Assume that for any y ∈ X , w ( y ) > 0 .
Then,
• The Markov kernel P N is Harris recurrent and ergodic with unique invariant distribution φ N .•
The Markov kernel P N is reversible w.r
t. π , Harris recurrent and ergodic.
The proof follows from [61, Theorem 9.6, Lemma 9.11].
The following theorem establishes theunbiasedness of the estimator Π N f
( X 1: N ) under φ N .
Theorem 6.
For every N ∈ N ∗ and π -integrable function f ,
[Equation]
Proof.
Using (7) we get
[Equation] [Equation]
ℓ = w ( x ℓ )Π
N f ( x 1: N )
N (cid:89) j
= λ (d x j ) =
Nλ ( w ) (cid:90) N (cid:88) i =
w ( x i ) f
( x i )
N (cid:89) j
=
λ
[Equation]
x
[Equation] [Equation]
π ( f ), and the first identity follows.
The second identity stems from the fact that the function Π N f ( x 1: N ) isinvariant under permutation.
B Proofs of Section 2B.1 Uniform geometric ergodicity of the i-SIR Markov kernel
Here we provide a simple direct proof of the bound (2).
We preface the proof by a technical lemma.
Lemma 7.
Let Y 1: M be M independent random variables, satisfying E
[ Y i ] = 1 , and Var[ Y i ] < ∞for i ∈ { 1, . . .
, M } .
Then, for S M = (cid:80) Mi =1 Y i and a , b >
E (cid:104) ( a + bS M ) − 1 (cid:105) ≤ ( a + bM/ 2) − 1 + (4 /a )
Var[ S M ] /M 2 .
Proof.
Let K ≥ 0 .
Then we get
[Equation]
and in particular, E [( a + bS M ) − 1 ] ≤ ( a + bK ) − 1 + a − 1 P ( S M < K ) .
By Markov’s inequality,
P ( S M <
[Equation]
In particular, for K = M/ 2 , we have P ( S M < K ) ≤ 4 Var[ S M ] /M 2 .
Proof of (2) .
For ( x , A ) ∈ X × X , we get
P N ( x , A ) = (cid:90)
δ x (d x 1 )
N (cid:88) i =
w ( x i ) (cid:80) Nj =1 w ( x j )
A
( x i )
N (cid:89) j
= λ (d x j ) = (cid:90) w ( x ) w ( x ) + (cid:80)
Nj =2 w ( x j )1 A ( x )
N (cid:89) j
= λ (d x j ) + (cid:90)
[Equation]
Here in (a) we used Fubini’s theorem together with w ( x ) λ (d x ) =
π (d x ) λ ( w ) .
Finally, since thefunction f : z (cid:55)→ ( z + a ) − 1 is convex on R + and a > 0 , we get for i ∈ { 2, . . .
, N } ,(cid:90) λ ( w )
[Equation]
With the bound above we obtain the inequality
[Equation]
This means that the whole space X is (1, ϵ N π ) -small
(see [21, Definition 9.3.5]).
Since P N ( x , · ) andπ are probability measures, (9) implies
[Equation]
The statement follows from [21, Theorem 18.2.4] applied with m = 1 .
B.2 Proof of Theorem
We preface the proof with some preparatory lemmas.
Lemma 8.
Let K ⊂ X , such that w ∞ , K := sup x ∈ K { w ( x ) /λ ( w ) } < ∞ and π ( K ) > 0 .
Then, for all( x , A ) ∈ K × X , we get that
[Equation] [Equation]
, with ϵ N , K = ( N − 1) π ( K ) /
[2 w ∞ , K + N − 2] and π K ( A )
=
π ( A ∩ K ) /π
( K ) .
Note that if the weight function w is upper semi-continuous, then for any compact K , w ∞ , K =sup x ∈ K w ( x ) < ∞ .
Moreover, lim N →∞ ϵ N , K = π ( K ) .
Proof.
Let ( x , A ) ∈
X × X .
Then, using the lower bound (8), we obtain
P N ( x , A ) ≥
N
[Equation]
where the last inequality follows from Jensen’s inequality and the convexity of the function
z (cid:55)→( z + a ) − 1 on R + .
We conclude by noting that
[Equation]
Lemma 9.
Assume A 1.
Then for all x ∈ X , any function V : X → [1, ∞ ) with π ( V ) < ∞ ,λ ( V ) < ∞ , and N ≥ 3 , it holds that
[Equation]
where b P N is given in (12) .
Note that
[Equation] [Equation]
Proof.
Note first that
[Equation]
where we have set
[Equation]
Since the function z (cid:55)→ z/ ( z + a ) is concave on R + for a > 0 , we have
[Equation]
The bound above implies that, with renormalization,
[Equation]
Applying now Lemma 7 with a = π ( V ) /λ
( V ) , b = 1 , M = N − 2 , and Y j = w ( x j ) /λ ( w ) , weobtain that
[Equation]
Combining the bounds above yields (10) with
[Equation]
Lemma 10.
Let P be a Markov kernel on ( X , X ) , γ be a probability measure on ( X , X ) , and ϵ > 0 .
Let C ∈ X be an (1, ϵγ ) -small set for P .
Then for arbitrary Markov kernel Q on ( X , X ) , the set C isan (1, ϵγ Q ) -small set for P Q , where γ Q ( A ) =
(cid:82) γ (d y ) Q ( y , A ) for A ∈ X .
Proof.
Let ( x , A ) ∈ C
× X .
Then it holds
[Equation]
Lemma 11.
Let P and Q be two irreducible Markov kernels with π as their unique invariantdistribution.
Let V : X → [1, ∞ ) be a measurable function.
Suppose that there exist λ Q ∈ [0, 1)and
b P , b Q ∈ R + such, that P V ( x ) ≤ V ( x ) + b P and Q V ( x ) ≤ λ Q V ( x ) + b Q .
Let r 0 ≥ 1 .
Also assume that for all r ≥ r 0 , there exist ϵ r > 0 and a probability measure γ r such that for all( x , A ) ∈ V r × X , P( x , A ) ≥ ϵ r γ r ( A ) , where V r = { x ∈ X : V ( x ) ≤ r } .
Define K = P Q andλ K = λ Q , b K
= b P + b Q .
Then,
K V ( x ) ≤ λ K V ( x ) + b K and, for all x ∈ V r , K( x , A ) ≥ ϵ r γ Q , r ( A ), where γ Q , r ( A ) = (cid:82) γ r (d y ) Q ( y , A ) .
Moreover, let r ≥ r 0 be such that λ K + 2 b K
(1 + r ) < 1 .
Then, for any x ∈ X and k ∈ N ,
[Equation]
k K , where
[Equation]
Proof.
By Lemma 10, it holds that for any ( x , A ) ∈
V r × X , K( x , A ) ≥ ϵ r γ Q , r ( A ) .
Moreover, forany x ∈ X , K V ( x ) =
P Q V ( x ) ≤ λ Q P V ( x ) + b Q ≤ λ Q V ( x ) + b Q + b P .
The proof is completedwith [21, Theorem 19.4.1].
Proof of Theorem 2.
The proof consists of the 3 main steps:
Lemma 8 implies that for all r ≥ r R , the level sets V r for the Markov kernel P N are (1, ϵ r , N γ r ) -
small for the Markov kernel P N , where
[Equation]
w ∞ , r + N − 2], and γ r ( A ) =
(cid:82)
π V r (d y ) R ( y , A ) with π V r ( B ) =
π ( B ∩ V r ) /π
( V r ) , for any B ∈ X .2.
Lemma 9 implies that for all x ∈ X , P N V ( x ) ≤ V ( x ) + b P N , where b P N is given in (12).3.
We finally show (see Lemma 11) that the Markov kernel K N also satisfies a Foster-Lyapunovcondition with the same drift function V as R , that is, K N V ≤ λ R V + b K N with b K N =
b R + b P N .
We conclude by using Lemma 11.
We choose r N = r R ∨{ 4 b K N / (1 − λ R ) − 1 } .
Then λ R +2 b K N / (1+r N ) ≤ (1 + λ R ) / 2 < 1 , and Lemma 11 implies (3) with
[Equation]
Set b K ∞ =
lim N →∞ b K N =
b R + b P ∞ , where b P ∞ is defined in (11), r ∞ =
r R ∨ [4 b K ∞ / (1 − λ R ) − 1]and ϵ ∞ =
π ( V r ∞ ) .
With these notations, we have
[Equation]
C Metropolis-Adjusted Langevin rejunevation kernel
This section addresses the convergence of the Metropolis Adjusted Langevin algorithm (MALA) forsampling from a positive target probability density π on ( R d , B ( R d )) , where B ( R d ) is the Borel σfield of R d endowed with the Euclidean topology.
For simplicity, let U = − log
π be the associatedpotential function.
MALA is a Markov chain Monte Carlo (MCMC) method based on Langevindiffusion associated with π :
[Equation]
(14) where ( B t ) t ≥ 0 is a d -dimensional Brownian motion.
It is known that under mild conditions thisdiffusion admits a strong solution ( X ( x ) t ) t ≥ 0 for any starting point x ∈ R d and defines a Markovsemigroup ( P t ) t ≥ 0 for any t ≥ 0 , x ∈ R d and A ∈ B ( R d ) by P t ( x , A ) =
PP ( X ( x ) t ∈ A ) .
Moreover, this Markov semigroup admits π as its unique stationary measure, is ergodic and even
V -uniformly geometrically ergodic with additional assumptions on U (see [65, 47]).
However,sampling a path solution of (14) is a real challenge in most cases, and discretizations are used insteadto obtain a Markov chain with similar long-term behaviour.
Here we consider the Euler-Maruyamadiscretization, which is given by (14), defined for all k ≥ 0 by
[Equation]
15) where γ is the step size of the discretization and.
{ Z k , k ∈ N ∗ } is a i.i.d.
sequence of d -dimensionalstandard Gaussian random variables.
This algorithm was proposed by [24, 56] and later studiedby [28, 29, 51, 65].
According to [65], this algorithm is called the Unadjusted Langevin algorithm(ULA).
A drawback of this method is that even if the Markov chain { Y k , k ∈ N } has a uniquestationary distribution π γ and is ergodic (which is guaranteed under mild assumptions about U ), π γ isdifferent from π most of the time.
To solve this problem, in [67, 65] it is proposed to use the Markovkernel associated with the recursion defined by the Euler-Maruyama discretization (15) as a proposalkernel in a Metropolis-Hastings algorithm that defines a new Markov chain { X k , k ∈ N } by:
[Equation]
(16) where ˜ Y k
+
=
X k − γ ∇ U
( X k ) +
√ 2 γZ k +1 , { U k , k ∈ N ∗ } is a sequence of i.i.d.
uniform randomvariables on [0, 1] and α γ : R 2 d → [0, 1] is the usual Metropolis acceptance ratio.
This algorithmis called Metropolis Adjusted Langevin Algorithm (MALA) and has since been used in manyapplications.
Denote by r γ the proposal transition density associated to the Euler-Maruyama discretization (15)with stepsize γ > 0 , i.e. , for any x , y ∈ R d ,
[Equation]
Then, the Markov kernel R γ of the MALA algorithm (16) is given for γ > 0 , x ∈ R d , and A ∈ B ( R d )by
[Equation]
It is well-known, see e.g. [65], that for any γ > 0 , R γ is reversible with respect to π and π -irreducible.
H1.
The function U : R d → R is three times continuously differentiable.
In addition, ∇ U (0) = 0 andthere exists L ≥ 0 and M ≥ 0 such that sup x ∈ R d ∥ D 2 U ( x ) ∥ ≤ L such that sup x
∈
R d ∥ D 3 U ( x ) ∥ ≤ M .
The condition ∇ U (0) = 0 is satisfied (up to a translation) as soon as U has a local minimum, whichis the case when lim ∥ x ∥→ + ∞ U ( x ) =
+ ∞ , since U is continuous.
H2.
There exist m > 0 and K ≥ 0 such that for any x , y ∈ R d , ∥ x ∥ ≥ K and ∥ y ∥ = 1 ,
D 2 U ( x ) { y } ⊗ 2 ≥ m .
Note that under H 1 and H 2, for any x , y ∈ R d , ∥ y ∥
= 1 , it holds that
[Equation]
In the case K = 0 , H 2 amounts to U being strongly convex and the convexity constant being equal tom .
However, if K > 0 , H 2 is a slight strengthening of the condition of strong convexity at infinityconsidered in [18, 23]: there is m ′ > 0 and K ′ ≥
such that for each x , y ∈ R d , ∥ x − y ∥ ≥ K ′
[Equation]
Indeed, if (18) holds for any x , y ∈ R d that ∥ x ∥ ∨ ∥ y ∥ ≥ K ′ instead of ∥ x − y ∥ ≥ K ′ , then a simplecalculation implies that H 2 holds with m ← m ′ and K ← K ′
+ 1 .
Finally, while the condition (18)holds for x , y ∈ R d , ∥ x − y ∥ ≥ K ′ , is weaker than H 2, it may be more convenient in many situationsto check whether the latter holds.
Lemma 12.
Assume H 1 and H 2 hold.
The function U satisfies for any x ∈ R d ,
[Equation]
, with ˜ K = 2 K (1 + L / m ) and ˜ C
=
L ˜ K 2 .
Note that under H 1 and H 2, m ≤ L .
Define for any η > 0 , V η : R d
→
[1, + ∞ ) for any x ∈ R d by
[Equation]
The analysis of MALA is naturally related to the study of the ULA algorithm.
More precisely, sincefor any x ∈ R d and A ∈ B ( R d ) , the Markov kernel corresponding to ULA (15) is given by
[Equation]
To show that MALA satisfies a Lyapunov condition, we first state a drift condition for the ULAalgorithm.
Proposition 13.
Assume H 1 and H 2 and let ¯ γ ∈ (cid:0) 0, m / (4 L 2 ) (cid:3) .
Then, for any γ ∈ (0, ¯ γ ] , x ∈ R d ,
[Equation]
where V ¯ η is defined in (19) ,
¯ η = m / 16 , K U =
max(˜ K , 4 (cid:112) d/ m ) , ˜ K is defined in Lemma 12 and
[Equation]
η ¯ γd ) .
Proof.
The proof follows from [22, Proposition 6].
We now introduce for ¯ γ > 0 the auxiliary constant
[Equation]
For ¯ γ ∈ (cid:0) 0, m 3 / (4 L 4 ) (cid:3) , we also define C 2,¯ γ as
[Equation]
Using Proposition 13, we state a drift condition for the MALA kernel R γ .
Proposition 14.
Assume H 1 and H 2.
Then, there exist Γ > 0 (given in (21) )
such that for any¯ γ ∈ (0, Γ] , γ ∈ (0, ¯ γ ] and x ∈ R d ,
[Equation]
where V ¯ η is defined by (19) , R γ is the Markov kernel of MALA defined by (17) , ¯ η = m / 16 , ϖ =¯ η m ( K M ) 2 / 16 , and
[Equation]
where K U , b U¯ γ are defined in Proposition 13, and ˜ K is defined in Lemma 12.
Proof.
The proof follows from [22, Proposition 7].
Quantitative bound on the mixing rate of the MALA sampler requires also the minorization conditionfor the MALA kernel.
The result below is due to [22, Proposition 12].
Proposition 15.
Assume H 1 and H 2.
Then for any K ≥ 0 there exists ˜Γ K > 0 (given in (22) below),such that for any x , y ∈ R d ,
∥ x ∥ ∨ ∥ y ∥ ≤ K , and γ ∈ (0, ˜Γ K ] we have
[Equation]
, where
[Equation]
where C 1,˜Γ 1 / 2 is defined in (20) , ˜ K is defined in Lemma 12, and Φ ( · ) is the cumulative distributionfunction of the Gaussian distribution with zero mean an unit variance on R .
It is interesting to note that γ is the discretization step of the underlying Langevin diffusion.
Wehave to iterate the kernel 1 /γ times for the diffusion to progress by one time unit.
Combining
Proposition 14 and Proposition 15 yields the following ergodicity result in V ¯ η -norm.
Theorem 16.
Assume H 1 and H 2.
Then, there exist ¯ Γ > 0 (defined in (23) below), such that for anyγ ∈ (cid:0) 0, ¯Γ (cid:3) , there exist C ¯Γ ≥ 0 and ρ ¯Γ ∈ [0, 1) (given in (23) ) satisfying for any x ∈ R d ,
[Equation]
where ¯ η = m / 16 ,
[Equation]
(23) and ϖ is given in Proposition
[Equation]
Proof.
The proof follows from [22, Theorem 2].
For completeness we repeat here the main steps ofthe proof.
Proposition 14 shows that there exist Γ > 0 (given in (21)) such that for any ¯ γ ∈ (0, Γ] ,γ ∈ (0, ¯ γ ] and x ∈ R d ,
[Equation]
where the constants ϖ and b M¯ γ are given in Proposition 14.
Hence, setting λ =
e − ϖ < 1 , we obtainby induction that
[Equation]
Now we set M ¯ γ and K ¯ γ as in (23).
Then Proposition 15 implies that for any ¯ γ ∈ (cid:16) 0, ˜Γ K Γ (cid:105) , anyx , y ∈ { V ¯ η ( · ) ≤ M ¯ γ } , and γ ∈ (0, ¯ γ ] ,
[Equation]
Now it remains to combine both statements with ¯
γ = Γ ∧ ˜Γ K Γ and apply [21, Theorem 19.4.1] to the
Markov kernel R ⌈ 1 /γ ⌉ γ .
Comparison with Ex 2 MCMC kernel.
Based on the results above, we first state the quantitativemixing rate bounds for Ex 2 MCMC algorithm with the MALA kernel R ⌈ 1 /γ ⌉ γ (iterated ⌈ 1 /γ ⌉ times)applied as rejuvenation kernel.
The corresponding Markov kernel writes for x ∈ R d and A ∈ B ( R d )as
[Equation]
where R γ ( x , A ) is defined in (17).
Note also that, for r ≥ 1 , and V ¯ η defined in (19), the level sets
[Equation]
The result above allows to state the following ergodicity result for K N , γ kernel.
Theorem 17.
Assume H 1, H 2, and A 1, A 2 with V ¯ η defined in (19) .
Then there exist ¯Γ (defined in(23) ), such that for any γ ∈ (cid:0) 0, ¯Γ (cid:3) , x ∈ R d , and k ∈ N ,
[Equation]
where V ¯ η is defined in (19) , and the constants c N , ˜ κ N ∈ [0, 1) are given by
[Equation]
and λ is defined in (23) .
Proof.
The proof follows from the combination of Theorem 2 and Proposition 14.
To derive the geometric ergodicity rates in Theorem 17, it is not required to identify the smallsets of the MALA rejuvenation kernel R γ .
The only quantity of interest is the Foster-Lyapunovdrift condition satisfied by R ⌈ 1 /γ ⌉ γ .
Theorem 16 implies that the rate of convergence of MALA isγ log ρ ¯Γ .
The following statement allows to quantify the improvement in the convergence rate of
K N , γ compared to R ⌈ 1 /γ
⌉ γ .
Following [58], we consider the relative improvement of the mixing timeof the considered Markov kernels.
To introduce formally the mixing time, we need an auxiliarydefinition of the V -Dobrushin coefficient.
We refer the reader to [21, Section 18.3] for more detailedexposition.
Recall that M 1, V ( X ) is a set of probability measures on ( X , X ) , such that ξ ( V ) < ∞ .
Definition 18 ( V -Dobrushin coefficient) .
Let V : X (cid:55)→ [1; + ∞ ) be a measurable function, and
Q be a Markov kernel on ( X , X ) , such that ξ ( V ) < ∞ implies ξ Q ( V ) < ∞ for any measureξ ∈ M 1, V ( X ) .
Then the V -Dobrushin coefficient of the Markov kernel Q , is defined by
[Equation]
It is known (see e.g. [21, Theorem 18.4.1]), that V -geometric ergodicity of the Markov kernel Q (see
Definition 1) is equivalent to the fact, that
[Equation]
ε for some m ∈ N ∗ and 0 < ε < 1 .
Definition 19.
Let Q be V -geometrically ergodic Markov kernel.
Then the corresponding mixingtime t mix ∈ N ∗ is defined as
[Equation]
Note that if Q is V -geometrically ergodic with factor 0 < ρ < 1 given in Definition 1 , its mixingtime t mix is bounded as t mix ≤ (log(1 /ρ ))
− 1 log(4 M ) .
Now we compare the mixing time of K N , γ , which is inversely proportional to log(1 / ˜ κ N ) , to themixing time of R ⌈ 1 /γ ⌉ γ , which is inversely proportional to log(1 /ρ ¯Γ ) .
Theorem 20.
Assume H
1- H 2 and A 1- A 2 with V ¯ η .
Then there exist ¯Γ (defined in (23) ), such that forany γ ∈ (cid:0) 0, ¯Γ (cid:3) , it holds that
[Equation]
where λ , ¯ λ , and ¯ b M¯Γ are defined in (23) , ε ( · ) is defined in (22) , andr ∞
= 1 ∨ { 4 b ∞ / (1 − λ ) − 1 } ,
[Equation]
Proof.
The proof follows by combining the expressions (23) and (24).
The ratio log(1 − 2 − 1 ε ( K ¯Γ )) / log(1 − ϵ ∞ ) is extremely small in most settings.
This explains theobserved behavior: the mixing time of the Ex2MCMC kernel is much smaller than the mixing time ofthe MALA algorithm, which we observe in practice in all the examples we discuss.
The difference iseven more spectacular when the dimension increases.
To illustrate this phenomenon, we consider thefollowing numerical scenario for (25).
We assume that H 1- H 2 holds with m = 0.1, M = 2.0, L = 1.0 ,and K = 5.0 . One can evaluate that even for d = 2 the respective value K ¯Γ ≈ 10 3 . We now show, howthe bound for K ¯Γ scales with the dimension d .
The respective plot for d ∈ [2; 100] is given in Figure 6.
It implies that K ¯Γ grows as √ d .
At the same time, the standard bound Φ ( − x ) ≤ exp {− x 2 / 2 } ,valid for x ≥ 0 , yields that ε ( K ¯Γ ) / 2 ≤ exp {− (3 / 2) ( L + 1) K 2¯Γ } .
At the same time, ϵ ∞ typicallydoes not decrease with the growth of d due to the construction of r ∞ .
Hence, the ratio (25) decreasesexponentially with the growth of d in our model scenario.
D Proof of Theorem 3
The proof relies on results of stochastic approximation with Markovian dynamics; see [7, 8].
Forreader’s convenience, before going into the details, we give an outline of the proof.
The motivation ofsuch algorithms is to find the roots of the function h :
Θ → R q , Θ ⊂ R q
[Equation]
for families of functions { H ( θ , u , e ) :
Θ × U × E → Θ } , a family of probability distributions{ ρ θ , θ ∈ Θ of ( E , E ) and a probability distribution µ on a space ( U , U ) .
These roots are not availableanalytically and a way of finding them numerically consists of considering the controlled Markov chain on (cid:8) (Θ × U ) N , ( B (Θ) ⊗ U ) ⊗ N (cid:9) initialized at some ( θ 0 , U 0 )
=
( ϑ , u ) ∈
Θ × U and definedrecursively for a sequence of stepsize { γ i , i ∈ N } by
[Equation]
where { P θ , θ ∈ Θ } is a family of Markov kernels such that for each θ ∈ Θ , µP θ = µ .
The rationalefor this recursion goes as follows.
Let us first rewrite the Robbins-Monro recursion
[Equation]
where ξ i +
=
H ( θ i , U
i +1 , E i +1 ) is referred to as the "noise".
Therefore, { θ i } is a noisy versionof the sequence { ¯ θ i } defined as ¯ θ i +
=
¯ θ i + γ
i +1 h
(¯ θ i ) .
The convergence of such sequences hasbeen studied by many authors, starting with [48] under various conditions.
A crucial step of suchconvergence analysis consists of assuming that the sequence { θ i } remains bounded with probability 1in a compact set of Θ .
This problem has traditionally can be circumvented by means of modificationsof the recursion.
Indeed, one of the major difficulties specific to the Markovian dynamic scenario isthat { θ i } governs the ergodicity of the controlled Markov chain { U i } and that stability properties of{ θ i } require "good" ergodicity properties which might vanish whenever { θ i } approaches ∂ Θ oftenaway from the roots of h ( θ ) , resulting in instability.
Most existing results rely on modifications of theupdates designed to ensure a form of ergodicity of { ξ i } which in turn ensures that { θ i } inherits thestability properties of { ¯ θ i } ; see e.g. [7, 10] and the discussion in [8, Section 3].
We follow here [10].
Let {R i } be a sequence of compact subsets of Θ and consider the recursion:
[Equation]
where, denoting F i = σ ( U 0 , θ j , j ≤ i ) , θ proj i +1 is a random variable measurable w.r.t F i ∨ σ ( θ ∗ i +1 ) .
Most common practical projection mechanisms include θ proj
i +
=
θ i , ’rejecting’ an update outside thecurrent feasible set, and θ proj
i +1 =
Proj
R i +
( θ ∗ i +1 ) , where Proj is a measurable mapping Θ \ R
i +1 →
R i +1 .
In words, the expanding projections approach only ensures that θ i is in a feasible set R i butdoes not involve potentially harmful ‘restarts’ as is the case with the adaptive reprojection strategyof [7].
We use the results in [10] to show that the SA { θ i } ’stays away’ from ∂ Θ with probabilityone for any initialization ( θ 0 , u ) ∈ R
× U under appropriate conditions on { H ( θ , u , e ), ( θ , u , e ) ∈Θ × U × E } , { P θ , θ ∈ Θ } and {R i } .
We denote throughout the probability distribution associatedto the process ( θ i , U i )
i ≥ 0 defined in Algorithm 1.1 and starting at ( θ 0 , U 0 )
≡ ( θ , u ) ∈
Θ × U as
P θ , u ( · ) and the associated expectation as E θ , u [ · ] .
The approach developed in [10] relies on theexistence of a Lyapunov function w : Θ → [0, ∞ ) for the recursion on θ and the subsequent proofthat { w ( θ i ) } is P θ , u -a.s.
under some adequate level.
For any M > 0 , we define the level sets
W M := { θ ∈ Θ : w ( θ ) ≤ M } .
Consider the following assumptions:
SA1.
There exists a continuously differentiable function w : Θ → [0, ∞ ) such that
(i)
For all θ , θ ′ ∈ Θ ,
[Equation]
(ii) the projection sets are increasing subsets of Θ , that is, R i ⊂ R i +1 for all i ≥ 0 , and
[Equation]
(iii) there exists a constant M 0 > 0 such that for any θ ∈ W cM 0 ∩ ˆΘ
[Equation] [Equation]
(iv) the family of random variables (cid:110)
θ proj
i
[Equation]
i ≥ 1 satisfies for all i ≥ 1 whenever θ ∗ i / ∈
R
i θ proj i ∈
R i and w
[Equation]
proj
i
[Equation]
(v) there exists constants c ∈ [0, ∞ ) and a non-decreasing sequence of constants ζ i ∈ [1, ∞ )satisfying sup θ ∈R i |∇ w ( θ ) | ≤ cζ i for all i ≥ 0 .
Hereafter, we denote the ’centred’ version of ¯ H ( θ , u , e ) :
= H ( θ , u , e ) − h ( θ ) .
For the stabilityresults, we shall introduce the following general condition on the noise sequence.
In general terms, itis related to the rate at which { θ i } may approach ∂ ˆΘ in relation to the growth of ∥ H ( θ , u , e ) ∥ andthe loss of ergodicity of P θ .
SA2.
For any ( θ , u ) ∈ R 0 × U it holds that
(i) P θ , u (lim i →∞ γ
i +1 ∥∇ w ( θ i ) ∥ ·
∥ H ( θ i , U
i +1 , E i +1 )
∥ = 0)
= 1 ,(ii) E θ , u (cid:2)(cid:80) ∞ i =
γ 2 i +1 ∥
H ( θ i , U
i +1 , E i +1 )
∥ 2 (cid:3) < ∞ ,(iii) E θ , u (cid:104) sup k ≥
(cid:12)(cid:12)(cid:12)(cid:80) ki =
γ i +1 (cid:10) ∇ w ( θ i ),
¯ H
( θ i , U
i +1 , E i +1 ) (cid:11)(cid:12)(cid:12)(cid:12)(cid:105) < ∞ .
(iv) lim θ → ∂ ˆΘ w ( θ )
= ∞
Theorem 21.
Assume SA 1- SA 2.
Then, for any ( θ , u ) ∈ R 0 × U
[Equation]
Proof.
The proof is a simple adaptation of [10, Theorem 2.5].
The condition lim θ → ∂ ˆΘ w ( θ ) = ∞ is weakened in [10, Section 2.2].
Verifiable conditions implying
SA 2 are given in [10, Section 3, Condition 3.1].
They are summarized in the next assumption.
In theassumptions below, it is implicitly assumed that SA 1 holds with constants ( ζ i ) i ≥ 0 .
We denote ˜ H
( θ , u )
=
(cid:82) ¯ H ( θ , u , e ) ρ (d e ) and we consider the following assumptions:
SA3.
For all θ ∈ ˆΘ , the solution g θ : U → Θ to the Poisson equation
g θ ( u ) − P θ g θ ( u ) ≡ ˜ H
( θ , u )exists and for all i ≥
the step size Γ i +1 is independent of F i and U i +1 .
Moreover, there exist ameasurable function V : U → [1, ∞ ) and constants c < ∞ , β H , β g ∈ [0, 1 / 2] and α g , α H , α V ∈[0, ∞ ) such that for all ( θ , u ) ∈ R 0 × U
[Equation]
[Equation]
[Equation]
[Equation]
(vi) (cid:80) ∞
i =1 γ i +1 γ i ζ α H + α g +( β H + β g
α V i
[Equation]
(vii) (cid:80) ∞ | γ i +1 − γ i | ζ
+ α g + β g
α V
i
< ∞ .
[Equation]
For geometrically ergodic Markov chain, these conditions may be shown to boil down to "uniform-inθ " geometric ergodicity conditions and "smoothness" of the mapping θ (cid:55)→ P θ .
MC1.
For any r ∈ (0, 1] and any θ ∈ ˆΘ , there exist constants M θ , r ∈ [0, ∞ ) and ρ θ , r ∈ (0, 1) , such that for any function ∥ f
∥ V r < ∞
[Equation]
for all k ≥ 0 and all u ∈ U .
Moreover, it holds that sup θ ∈R i M θ , r ≤ c r ζ α M i andsup θ ∈R i (1 − ρ θ , r ) − 1 ≤ c r ζ
α
ρ
i .
MC2.
For any θ , θ ′ ∈ ˆΘ , there exist a constant D θ , θ ′ , r ∈ [0, ∞ ) and a constant β D ∈ (0, ∞ ) independent of θ , θ ′ and r such that for any function ∥
f
∥ V r < ∞
[Equation]
Moreover, sup ( θ , θ
[Equation]
) ∈R 2 i D θ , θ ′ , r ≤ c Dr ζ α D i for some constant c Dr ∈ [0, ∞ ) depending only on r ∈(0, 1]
MC3.
SA 3-(i) and (ii) hold with constants α H , β H and α V , and there exist constants c < ∞ , α ∆ ∈ [0, ∞ ) and β ∆ > 0 such that
[Equation]
Up to this point, we have only considered the stability of the stochastic approximation process withexpanding projections.
Indeed, after showing the stability we know that the projections can occuronly finitely often (almost surely), and the noise sequence can typically be controlled.
Given this,the stochastic approximation literature provides several alternatives to show the convergence; see[41, 15].
We formulate below a convergence result following from [7].
SA4.
The set Θ ⊂ R d is open, the mean field h :
Θ → R d is continuous, and there exists acontinuously differentiable function ˆ w : Θ → [0, ∞ ) such that
(i) there exists a constant M 0 > 0 such that
[Equation]
(ii) there exists M 1 ∈ ( M 0 , ∞ ]
such that { θ ∈ Θ : ˆ w ( θ ) ≤ M 1 } is compact.(iii) for all θ ∈ Θ \ L , the inner product ⟨∇
ˆ w ( θ ), h ( θ ) ⟩ < 0 and the closure of ˆ w ( L ) has an emptyinterior.
Theorem 22.
Assume SA 4 holds, and let K ⊂ Θ be a compact set intersecting L , that is, K ∩ L ̸ = 907 ∅ .
Suppose that ( γ i ) i ≥ 1 is a sequence of non-negative real numbers satisfying lim
i →∞ γ i
= 0and (cid:80) ∞
i =1 γ i = ∞ .
Consider the sequence ( θ i ) i ≥ 0 taking values in Θ and defined through therecursion θ i
= θ i − 1 + γ i h
( θ i − 1 )
+ γ
i ε i for all i ≥ 1 , where ( ε i )
i ≥ 1 take values in R d .
Ifthere exists an integer i
such that { θ i } i ≥ i 0 ⊂ K and lim m →∞ sup n ≥ m | (cid:80) ni
= m γ i ε i | = 0 , thenlim n →∞ inf x ∈L∩K
∥
θ n − x ∥ = 0.
We have now all the necessary elements to prove Theorem 3.
For simplicity, we set α k = α ∞ forany k ∈ N and γ k = 1 / (1 + k ) ι where ι ∈ (1 / 2, 1] .
In this case, the state space is U = X M and
E =
Z ( N − 1) · M , U k
=
( Y k
[ j ]) Mj
=1 , E k =
( Z 2: N k [ j ]) Nj =2 .
With u = ( y [ j ]) Mj
=1 and e = ( z 2: N
[ j ]) Mj
=1 ,
H ( θ , u , e ) is given by
[Equation]
where H f and H b are defined respectively in (4) and (5).
In this case, the Markov kernel P θ is givenfor any nonnegative function f ,
[Equation]
and K θ , N is defined in (2.2) with λ ← λ θ and w ← w θ .
By construction, for any θ ∈ Θ , P θ has aunique stationary distribution which is given by µ = π ⊗ M .
Using Theorem 6, and, for all θ ∈ Θ ,
[Equation]
we get that
[Equation]
Recall that Θ = R q .
To check SA 1, we set
[Equation]
) , for θ ∈ Θ. and for i ∈ N , ζ i = log( i + 1) .
The subset R i is a ball centered at 0 and of radius r i where r i ischosen so that sup ∥
θ ∥≤ r i ∇ w ( θ ) ∥ ≤ cζ i (such r i exists using A 3).
It is easily checked that SA 1 issatisfied thanks to A 3 (note in particular that ∇ w is globally Lipshitz under the stated conditions).
Conditions SA 3-(v)-(vi)-(vii) are automatically satisfied.
We consider the drift function for the Markov kernel P θ , N
V ( y [1], . . .
, y
[Equation]
where V is the drift function in A 1.
MC 1 follows from Theorem 2 under A 4.
It is important to notethat it is essential to have explicit controls on the drift and reduction conditions here.
Conditions MC2 and MC 2 follow from A 3.
The precise tuning of constants is done along the same lines as [10,
Section 5.3].
E Numerical experiments
E.1 Metrics
ESTV To compute Empirical sliced total variation distance (ESTV), we perform 25 random onedimensional projections and then perform Kernel Density Estimation there for reference and producedsamples.
We then take the TV-distance between two distributions over 1 D grids of 1000 points.
We consider the value averaged over the projections to show the divergence between the MCMCdistribution and the reference distribution.
EMD We compute the EMD as the transport cost between sample and reference points in L 2 usingthe algorithm proposed in [14].
Then we report the EMD rescaled by the target dimension d .
ESS ESS (effective sample size) measures how many independent samples from target yield(approximately)
the same variance for estimating the mean of some function.
The closer ESS isto 1 , the better is the sampler.
Following [26], we compute ESS component-wise for multivariatedistributions.
Namely, given a sample { Y t } M t =1 , Y t ∈ R d of size M , for i = 1, . . .
, d , we compute
[Equation]
Here ρ ( i ) k
= Cov ( Y t , i , Y t + k , i )
Var ( Y t , i ) is the autocorrelation at lag k for i − th component.
We replace ρ ( i ) k by itssample counterpart (cid:98) ρ
( i ) k , an report ESS =
d − 1 (cid:80) di =1 (cid:100)
ESS i , where
[Equation]
E.2 Unimodal Gaussian target and impact of dimension
With the simple experiment presented on Figure 7, we illustrate the sensitivity of the purely globali-SIR to the match between the proposal and target, which typically worsens with dimension.
Namely,the rate κ N can be close to 1 when the dimension d is large, even when the restrictive condition thatweights are uniformly bounded | w | ∞ < ∞ is satisfied.
[Figure]
300 dimension 0.
[Figure]
E SS i-SIR
Ex 2 MCMC
[Figure]
To illustrate this phenomenon, we consider a simple problem of sampling from the standard normaldistribution N (0, I d ) with the proposal N (0, 2 I d ) in increasing dimensions d up to 300 .
Resultsvisualized in Figure 7 show that the performance of vanilla i-SIR quickly deteriorates as mostproposals get rejected.
This problem can be tackled by using the Explore-Exploit strategy couplingi-SIR with local MCMC steps to define a new sampler.
This simple experiment previously considers
Ex 2 MCMC with MALA applied as R .
E.3 Mixtures of Gaussians
Equally weighted Gaussians in two dimension The target density is
[Equation]
Here we choose σ = 1 , β i = 1 / 3 , and µ i , i ∈ { 1, 2, 3, } as vertices of an equilateral triangle withside length 4 √ 3 and center (0, 0) .
The contour representation of (26) can be found in Figure 1a.
Wecompare 3 sampling strategies:
• i-SIR algorithm with N = 3 particles and N (0, 4 I) proposal distribution;• MALA with step size γ = 0.5 , tuned to obtain acceptance rate 0.67 ;• Ex 2 MCMC algorithm with the same parameters as i-SIR and 3 consecutive MALA steps withγ = 0.5 as rejuvenations.
We generate 100 observations within each sampler and represent them in Figure 1a.
For the MALAsampler, we generate 300 samples and select every 3 th to maintain compatibility with the Ex 2 MCMCsetup.
Note that in this example, the variance of the global proposals in i-SIR should be relativelylarge to cover well all modes of the (26) mixture.
However, since the modes are narrow, the stepsize of MALA cannot be very large to obtain a sensible acceptance rate.
Therefore, Figure 1a showsthe drawbacks of the two approaches: i-SIR covers all modes of the target, but the chain often getsstuck at a certain point, which affects the variability of the samples.
MALA allows a better localexploration of each mode, but does not cover the whole support of the target.
The Ex 2 MCMC algorithm combines the advantages of both methods by combining i-SIR -based global explorationwith MALA -based local exploration.
Now, the mixture model of (26) is modified with the weights parameters β
= ( β 1 , β 2 , β 3 ) =(2 / 3, 1 / 6, 1 / 6) and same values of µ i and σ .
To compare the quality of the methods, we perform thefollowing procedure
• starting with the initial distribution N (0, 4 I) , we generate the trajectory ( X 1 , . . .
, X n ) for differentvalues of n ∈ [25, 800] for each of the compared methods ( i-SIR MALA, Ex 2 MCMC ).
Samplerhyperparameters are the same as above, and the burn-in period equals 50 ;• We perform the kernel density estimate (KDE) (cid:98) p n based on the observations ( X 1 , . . .
, X n ) ,and compute the total variation distance between (cid:98) p n and the target density p β , and the forwardKL( (cid:98) p n || p β ) .
Then we average the results over 100 independent runs of each sampler.
Now we use the same values for the means and covariances but set the mixing weights to β =( β 1 , β 2 , β 3 ) =
To compare the different sampling methods, we perform the followingprocedure.
• starting from the initial distribution N (0, 4 I) , we generate the trajectory ( X 1 , . . .
, X n ) for differentvalues of n ∈ [25, 800] for each of the compared methods ( i-SIR MALA, Ex 2 MCMC ).
Thehyperparameters of the sampler are the same as above, and the burn-in period is 50 ;• We perform kernel density estimation (KDE) (cid:98) p n based on the observations ( X 1 , . . .
, X n ) andcalculate the total variation distance between (cid:98) p n and the target density p β , as well as the forwardvalue KL
( (cid:98) p n || p β ) .
We then average the results over 100 independent runs of each sampler.
The results for each sampler are given in Figure 1c, Figure 8b.
We also provide a simple illustrationto the statements of (2) and Theorem 2.
Starting from the initial distribution ξ ∼ N (0, 4 I) , wedraw 500 independent chains of length 50 for each of the compared methods.
Using these 500observations, we create a KDE (cid:98) p n for the density corresponding to the distribution of ξ Q n fordifferent n ∈ { 5, . .
, 50 } and Q corresponding to i-SIR MALA or Ex 2 MCMC ˙
Then we calculatethe total variation distance between (cid:98) p n and the target density p β .
Corresponding plots can be foundin Figure 1b, Figure 8a.
Note that Ex 2 MCMC significantly outperforms the results of both MALAand i-SIR ˙
Indeed, the inhomogeneous mixture model is a complicated target for the Langevin-basedmethods.
The trajectories generated by MALA tend to remain in a single mode of mixture (26),which reduces the reliability of the estimates and requires the generation of long trajectories even ford = 2 .
At the same time, it is difficult for i-SIR type methods without local exploration trajectoriesto quickly cover all the modes.
E.4 Normalizing flow RealNVP
We use the RealNVP architecture ([20]) for our experiments with adaptive MCMC.
The key elementof RealNVP is a coupling layer, defined as a transformation f : R D → R D :
[Equation]
where s and t are some functions from R D to R D .
Thus, it is clear that the Jacobian of such atransformation is a triangular matrix with nonzero diagonal terms.
We use fully connected neuralnetworks to parameterize the functions s and t .
In all experiments with normalizing flows, we use the optimizer Adam ([38]) with β 1 = 0.9, β 2 =0.999 and weight decay 0.01 to avoid overfitting.
E.5 High-dimensional multi-modal distribution
In an additional experiment we consider a high-dimensional toy target distribution: a Gaussian mixturesimilar as Appendix E.3 above in 50 d .
Modes are equally weighted, isotropic and well-separated.
A purely local sampler would not mix between modes, as in the 2 d case.
A unimodal Gaussianproposal also fails in large dimension because of the concentration of the target measure in a smallfraction of the proposal’s bulk.
Hence we only examine the performance of FlEx 2 MCMC .
We setthe number of proposals per iterations to N = 20 .
[Figure]
All panels are
Using a normalizing flow proposal distribution initialized as an isotropic Gaussian covering the 3 modes (top
), training with backward KL loss only still typically leads to mode collapse on one of the modes (bottom left).
Running instead the simulataneous training and sampling of
FlEx 2 MCMC with the mixture of backward and forward KL loss can lead to successful mixing between distant modes (top and bottom right), yet at the condition that chains are initialized such that all modes can be reached by the local-rejuvenation kernel- which is here enforced by an initialization as random draws of the initial proposal.
Conversely, if all the chains are initialized in a single mode, the forward-KL estimated with states visited by the chains will not prevent a mode collapse (top and bottom center panels).
Using a RealNVP flow, we compare in Figure 9 the different outcomes depending on the choices ofinitialization of the MCMC walkers and training loss.
Training the proposal offline through uniquelythe backward KL (i.e. α = 0 in the combinaison of KL losses) is typically unstable in this multimodalcase and the network collapse on the first detected mode.
Successful backward-KL training isprobably possible, yet at the cost of designing a proper annealing schedule of the target distributionas in [76].
Resorting instead to a loss involving the forward KL ( α = 0.9 in this experiment), mixingbetween the well separated modes in high-dimension is possible, provided that chain initializationensures that all modes can be reached by the local kernel.
To summarize, the choice of loss composition depends on the information a priori available on theconsidered target distribution.
If rough location of modes is available - as it might be the case inchemistry applications where isomers of interest are known but sampling is necessary for relative freeenergy calculations - relying on the forward KL to draw the proposal to the modes is a simple andefficient strategy.
Conversely, if little is known, there is no free lunch with the local-global kernelsand an annealing might be necessary to train the global proposal, possibly using only the backward
KL loss.
E.6 Distributions with complex geometry
In this section, we study the sampling quality from high-dimensional distributions, whose densitylevels have high curvature (Banana shaped and Funnel distributions, details below).
With such distributions, standard MCMC algorithms like MALA or i-SIR , fail to explore fully the densitysupport.
The corresponding densities are given for x ∈ R d by
[Equation]
where Z is a normalizing constant.
We set a = 2 , b = 0.5 for funnel and a = 5 , b = 0.02 for bananashape distributions, respectively.
For MALA we use an adaptive step size tuning strategy to maintainacceptance rate approximately 0.5 .
For i-SIR and Ex 2 MCMC algorithms we use wide Gaussianglobal proposal N (0, σ 2 p I) with σ 2 p = 4 for Funnel and σ 2 p = 9 for Banana-shape distribution.
For FlEx 2 MCMC use a simple RealNVP-based normalizing flow [20] with 4 hidden layers.
Notethat for p f ( x ) the energy landscape in the region with x 1 < 0 is steep, so the distributions (27) arehard to capture, especially when the dimension d is large.
Moreover, due to the complex geometryof the distribution support, we cannot hope that local samplers (MALA) or global samplers ( i-SIR )alone will give good results.
In this example, we want to compare FlEx 2 MCMC with i-SIR MALAand the HMC-based NUTS sampler [35].
We also add a vanilla version of the Ex 2 MCMC algorithmto the comparison.
To generate the ground-truth samples, we use the explicit reparametrisation of (27).
Indeed, given a random vector ( Z 1 , . . .
, Z d ) ∼ N (0, I) , we consider its transformation ( X 1 , . . .
, X d )under the formulas
[Equation]
It is easy to check that ( X 1 , . . .
, X d ) follows the density p f ( x ), x ∈ R d .
Similarly, for d = 2 kconsider the transformation
[Equation]
Then ( Y 1 , . .
, Y d ) follows the density p b ( x ), x ∈ R d .
We provide the average computation timefor NUTS, adaptive i-SIR and FlEx 2 MCMC algorithms in Table 1 and Table 2 for the Funneland Banana-shape distributions, respectively, averaged over 50 runs.
Note that different runs of
NUTS algorithm yields high variance of the running time, especially for the Funnel distribution anddimensions d ≥ 50 .
We give the computation time for the above algorithms and additional implementation details in
Appendix E.6.
The implementation of FlEx 2 MCMC is based on the use of 5 MALA steps asrejuvenation steps.
Method d =d =d =d =d =
NUTS 33.4 ± 8.
41.1 ± 12.
61.6 ± 30.
82.3 ± 73.
88.4 ± 59.
Adaptive i-SIR 38.1 ± 3.
39.4 ± 2.
45.3 ± 2.
59.8 ± 0.
80.4 ± 0.
FlEx 2 MCMC 46.8 ± 3.
48.2 ± 2.
54.2 ± 2.
68.8 ± 0.
89.5 ± 0.
Method d =d =d =d =
[Table]
4 x 1        xi-SIR samples    4 x 1        x
MALA samples    4 x 1        x
Ex 2 MCMC samples
[Figure]
E.7 GANs as energy-based models
E.7.1 MNIST results
For this example, we consider both the Wasserstein GAN (WGAN) setup with energy function
E W ( z ) and the classical Jensen-Shannon GAN with energy function E JS ( z ) .
In both cases, we usefully connected networks with 3 convolutional layers for discriminator and 3 linear + 3 convolutionallayers for generator.
For WGAN training, we use gradient penalty regularisation, following [31].
We provide additional visualisations of the latent space and samples along a given trajectory for
Jensen-Shannon GAN in Appendix E.7.1 and for Wasserstein GAN in Appendix E.7.1.
Samplinghyperparameters are summarized in Table 3.
For fair comparison, we take each 3 -rd sample producedby the MALA, when running this algorithm separately.
Both for WGAN-GP and vanilla GANexperiments we apply i-SIR and Ex 2 MCMC with wide Gaussian global proposal N (0, σ 2 p ) .
Theparticular values of σ 2 p are specified in Table 3.
E.7.2 Cifar- 10 results
We consider two popular GAN architectures, DC-GAN [60] and SN-GAN [49].
Below we providethe details on experimental setup and evaluation for both of the models.
E.8 Training and sampling details
For DC-GAN and SN-GAN experiments, we took the implementation and training script of themodels from Mimicry repository https://github.com/kwotsin/mimicry .
Both models weretrained on a single GPU GeForce GTX 1060 for approximately 20 hours.
[Table]
4 x 1        xi-SIR samples    4 x 1        x
MALA samples    4 x 1        x
Ex 2 MCMC samples
[Figure]
Both for DC-GAN and SN-GAN, the latent dimension is equal to d = 128 .
Following [17], for bothmodels we consider sampling from the latent spatial distribution
[Equation]
where logit( y ) =
log ( y/ (1 − y ))
y ∈ (0, 1) is the inverse of the sigmoid function and p 0 ( z ) =
N (0, I) .
Evaluation protocol We perform n = 100 iterations of the algorithms
MALA, i-SIR Ex 2 MCMCand FlEx 2 MCMC ˙
For both the vanilla Ex 2 MCMC algorithm (Algorithm 2) and FlEx 2 MCMC weuse the Markov kernel (17), which corresponds to 3 MALA steps, as the rejuvenation kernel.
Thestep size γ given for the algorithm Ex 2 MCMC corresponds to its rejuvenation kernel MALA.
Formore experimental details, see Table 4.
For i-SIR and Ex 2 MCMC algorithms we use N (0, σ 2 p I)with σ 2 p = 1 as a global proposal distribution.
We run M = 500 independent chains for each of the above MCMC algorithms.
Then, for thej − th iteration, we compute the average value of the energy function E ( z ) averaged over M chains.
Hyperparameters are specified in Table 4.
Energy profiles for different algorithms for DC-GAN and
SN-GAN are provided in Figure 17 and Figure 14, respectively.
Note that in both cases Ex 2 MCMC or
FlEx 2 MCMC algorithms yields lower energy samples.
We visualize 10 randomly chosen trajectoriesobtained with each sampling methods in Figure 15-Figure 16 for SN-GAN and Figure 18-Figure 19for DC-GAN, respectively.
For each trajectory we visualize every 10 -th sample.
Both architecturesindicate the same findings:
MALA typically is not available to escape the mode of the correspondingtarget density p ( z ) during one particular run.
i-SIR travels well across the support of p ( z ) , yetthe corresponding energy values are higher then the ones of Ex 2 MCMC or FlEx 2 MCMC .
Somei-SIR trajectories can get trapped in one particular image due to the absence of local explorationmoves.
At the same time, Ex 2 MCMC as illustrated in Figure 16-16a and Figure 19-19a, can bothexploit the particular mode of the distribution and perform global moves over the support of p ( z ) .
Of course, these global moves are more likely to occur during the first sampling iterations.
For the
DC-GAN architecture, we provide also the dynamics of FID (Frechet Inception Distance, [34]),and IS (Inception Score, [69]) values computed over 10000 independent trajectories.
We plot themetrics in Figure 13a and Figure 13b.
Metrics illustrate the image quality improvement achieved by
FlEx 2 MCMC and Ex 2 MCMC algorithms.
[Figure] [Figure]
i
[Figure] [Figure]
# iterations
MALA step size γ
[Equation]
, N
[Equation]
MALA steps
SNGAN
[Figure]
DCGAN
[Figure]
[Figure] [Figure]
[Figure] [Figure]