Efﬁcient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods
Abstract
Normalizing ﬂows can generate complex target distributions and thus show promise in many applications in Bayesian statistics as an alternative or complement to MCMC for sampling posteriors.
Since no data set from the target posterior distribution is available beforehand, the ﬂow is typically trained using the reverse Kullback-Leibler (KL) divergence that only requires samples from a base distribution.
This strategy may perform poorly when the posterior is complicated and hard to sample with an untrained normalizing ﬂow.
Here we explore a distinct training strategy, using the direct KL divergence as loss, in which samples from the posterior are generated by (i) assisting a local MCMC algorithm on the posterior with a normalizing ﬂow to accelerate its mixing rate and (ii) using the data generated this way to train the ﬂow.
The method only requires a limited amount of a priori input about the posterior, and can be used to estimate the evidence required for model validation, as we illustrate on examples.
1. Introduction
Given a model with continuous parameters θ ∈ Θ ⊆ R d a prior on these parameters in the form of a probability density function ρ o ( θ ) and a set of observational data D giving the likelihood L ( θ ) for the model parameter θ , Bayes formula asserts that the posterior distribution of the parameters has probability density
[Equation]
where the normalization factor Z ∗ = (cid:82) Θ L ( θ )
ρ o ( θ ) dθ is the unknown evidence.
A primary aim of Bayesian inference is to sample this posterior to identify which parameters best explain the data given the model.
In addition one is typically interested in estimating Z ∗ since it allows for model validation, comparison, and selection.
Markov Chain Monte Carlo (MCMC) algorithms (Liu, 2008) are nowadays the methods of choice to sample complex posterior distributions.
MCMC methods generate a sequence of conﬁgurations over which the time average of any suitable observable converges towards its ensemble average over some target distribution, here the posterior.
This is achieved by proposing new samples from a proposal density that is easy to sample, then accepting or rejecting them using a criterion that guarantees that the transition kernel of the chain is in detailed balance with respect to the posterior density: a popular choice is Metropolis-Hastings criterion.
MCMC methods, however, suffer from two problems.
First, mixing may be slow when the posterior density ρ ∗ is multimodal, which can occur when the likelihood is non-logconcave (Fong et al., 2019).
This is because proposal distributions using local dynamics like the popular Metropolis adjusted Langevin algorithm (MALA) (Roberts & Tweedie, 1996) are inefﬁcient at making the chain transition from one mode to another, whereas uninformed non-local proposal distributions lead to high rejection rates.
The second issue with MCMC algorithms is that they provide no efﬁcient way to estimate the evidence Z ∗ : to this end, they need to be combined with other techniques such as thermodynamic integration or replica exchange, or traded for other techniques such as annealed importance sampling (Neal, 2001), nested sampling (Skilling, 2006), or the descent/ascent nonequilibrium estimator proposed in (Rotskoff & Vanden-Eijnden, 2019) and recently explored in (Thin et al., 2021).
Here, we employ a data-driven approach to aid designing a fast-mixing transition kernel along the lines of mode-hoping algorithms proposed by (Tjelmeland & Hegstad, 2001; Andricioaei et al., 2001; Sminchisescu & Welling, 2017) and learning based algorithms of (Levy et al., 2018; Titsias, 2017; Song et al., 2017).
Normalizing ﬂows (Tabak & Vanden-Eijnden, 2010; Tabak & Turner, 2013; Papamakarios et al., 2021) are especially promising in this context: these maps can approximate the posterior density ρ ∗ as the pushforward of a simple base density ρ B (e.g. the prior density ρ o ) by an invertible map T : Θ → Θ .
Their use for
Bayesian inference, as opposed to density estimation (Song et al., 2017; 2021), was ﬁrst advocated in Rezende & Mohamed (2015).
Since a representative training set of samples from the posterior density is typically unavailable beforehand, these authors proposed to use the reverse KullbackLeibler (KL) divergence of the posterior ρ ∗ from the push forward of the base ρ B , since this divergence can be expressed as an expectation over samples generated from ρ B , consistent with the variational inference framework (Jordan et al., 1998; Blei et al., 2017).
This procedure has the potential drawback that training the map is hard if the posterior differs signiﬁcantly from the initial pushforward (Hartnett &
Mohseni, 2020), as it may lead to “mode collapse.”
Annealing of ρ ∗ during training was shown to reduce this issue (Wu et al., 2019; Nicoli et al., 2020).
Building on works using normalizing ﬂows with MCMC (Albergo et al., 2019; No´e et al., 2019; Gabri´e et al., 2021)
here we explore an alternative strategy that blends sampling and learning where we (i) assist a MCMC algorithm with a normalizing ﬂow to accelerate mixing and (ii) use the generated data to train the ﬂow on the direct KL divergence.
2. Posterior Sampling and Model Validation with Normalizing Flows
A normalizing ﬂow (NF) is an invertible map T that pushes forward a simple base density ρ B (typically a Gaussian with unit variance, though we could also take ρ B = ρ o ) towards a target distribution, here the posterior density ρ ∗ .
An ideal map T ∗ (with inverse ¯ T ∗ ) is such that if θ B is drawn from ρ B then T ∗ ( θ B ) is a sample from ρ ∗ .
Of course, in practice, we have no access to this exact T ∗ , but if we have an approximation T of T ∗ , it still assists sampling ρ ∗ .
Denote by ˆ ρ the push-forward of ρ B under the map T ,
[Equation]
As long as ˆ ρ and ρ ∗ are either both positive or both zero at any point θ ∈ Θ , we can use a Metropolis-Hasting MCMC algorithm to sample from ρ ∗ using ˆ ρ as a transition kernel: a proposed conﬁguration θ (cid:48)
=
T ( θ B ) from a given conﬁguration θ is accepted with probability
[Equation]
This procedure is equivalent to using the transition kernel
[Equation]
where r ( θ ) = (cid:82) Θ acc ( θ, θ (cid:48) )
ˆ ρ ( θ (cid:48) ) dθ (cid:48) .
Since π T ( θ, θ (cid:48) ) is irreducible and aperiodic under the aforementioned conditions on ρ ∗ and ˆ ρ , its associated chain is ergodic with respect
1: S AMPLE T RAIN ( U ∗ , T , { θ i (0) } ni =1 , τ , k max , k Lang , (cid:15) ) 2: Inputs: U ∗ target potential, T initial map, { θ i (0) } ni = initial chains, τ > 0 time step, k max ∈ N total duration, k Lang ∈ N number of Langevin steps per NF resampling step, (cid:15) > 0 map training time step 3: k =4: while k < k max do 5: for i = 1 , . . .
, n do 6: if k mod k Lang + 1 = 0 then 7: θ (cid:48)
B , i ∼ ρ B 8: θ (cid:48)
i =
T ( θ (cid:48) B ,i ) { push-forward via T } 9: θ i ( k + 1)
= θ (cid:48)
i with prob acc
( θ i ( k ) , θ (cid:48)
i ) , otherwise θ i ( k + 1)
=
θ i ( k ) { resampling step } 10: else 11: θ (cid:48)
i
=
θ i ( k ) − τ ∇ U ∗ ( θ i ( k ))+ √ 2 τ η i with η i ∼
N (0 d , I d ) { discretized Langevin step } 12: θ i ( k + 1) = θ (cid:48)
i with MALA acceptance prob or
ULA, otherwise θ i ( k + 1)
=
θ i ( k ) 13: k ← k +14:
L [ T ] = − 1 n (cid:80) ni
=1 log ˆ ρ
( θ i ( k + 1)) 15:
T ← T − (cid:15) ∇L [ T ] { Update the map } 16: return: { θ i ( k ) } k max ,n k =0 ,i =1 , T to ρ ∗ (Meyn & Tweedie, 2012). In addition the evidence is given by [Equation]
For the scheme to be efﬁcient, two conditions are required.
First the parametrization of the map T must allow for easy evaluation of the density ˆ ρ which requires easily estimable
Jacobian determinants and inverses.
This issue has been one of the main foci in the normalizing ﬂow literature (Papamakarios et al., 2021) and is for instance solved using coupling layers (Dinh et al., 2015; 2017).
Second, as shown by formula (3) the proposal density ˆ ρ must produce samples with statistical weights comparable to the posterior density ρ ∗ to ensure appreciable acceptance rates.
This requires training the map T to resemble the optimal T ∗ .
3. Compounding Local MCMCs and Generative Sampling
In Algorithm 1, we present a concurrent sampling/training strategy that synergistically uses T to improve sampling from ρ ∗ and samples obtained from ρ ∗ to train T .
Let us describe the different components of the scheme.
Sampling.
The sampling component of Algorithm 1 alternates between steps of the Metropolis-Hasting procedure using a NF as discussed in Section 2 and steps of a local [Figure]
MCMC sampler (here MALA) (line 11), using as potential
[Equation]
Strictly speaking, the second transition kernel does not need to be local, it should, however, have satisfactory acceptance rates early in the training procedure to provide initial data to start up the optimization of T .
From a random initialization
T , the parametrized density ˆ ρ has initially little overlap with the posterior ρ ∗ and the moves proposed by the NF have a high probability to be rejected.
However, thanks to the data generated by the local sampler, the training of T can be initiated.
As training goes on, more and more moves proposed by the NF can be accepted.
It is crucial to notice that these moves, generated by pushing forward independent draws from the base distribution
ρ B ( θ ) , are non-local and easily mix between modes.
Training.
A standard way of training T is to minimize the
Kullback-Leibler divergence from ˆ ρ to ρ ∗ .
Since we do not have access to samples from ρ ∗ , here we use instead
[Equation]
[Figure] where ρ k denotes the density of the MCMC after k ∈ N steps and C k = (cid:82) Θ log ρ k ( θ ) ρ k ( θ ) dθ is a constant irrelevant for the optimization of T .
In practice, we run n walkers in parallel in the chain: denoting their positions at iteration k by { θ i ( k ) } ni =1 , we use the following estimator of (7) (minus the constant) as objective for training:
[Equation]
The training component of Algorithm 1 uses stochastic gradient descent on this loss function to update the parameters of the normalizing ﬂow (line 15).
Note that { θ i ( k ) } ni =1 are not perfect samples from ρ ∗ to start with, but their quality increases with the number of iterations of the MCMC.
Note also that this training strategy is different from the one proposed in (Rezende & Mohamed, 2015), which uses instead the reverse KL divergence of ˆ ρ from ρ ∗ : this objective could be combined with the one in (8).
Here we will stick to using (8) as loss, using approximate input from ρ ∗ through the MCMC samples initialized as explained next. M ¨ uller et al.
(2019) used the same forward KL divergence for training yet using a reweighing of samples from ˆ ρ for its estimation, which may be imprecise if the initial pushforward density bears little overlap with the target ρ ∗ .
Initialization.
To initialize MCMC chains, we assume that we have initial data lying in each of the important modes of the posterior ρ ∗ , but require no additional information.
That is, we take θ i (0)
=
θ i , where the θ i are initially located in these modes but not necessarily drawn from ρ ∗ .
We stress that the method therefore applies in situations where these modes have been located beforehand, for example by doing gradient descent on L ( θ )
ρ o ( θ ) from random points in Θ .
Architecture details of Algorithm 1 are deferred to Appendix A, and questions of convergence to Appendix B. − 2 v5 .7 .10 . K − 2 v5 .7 .10 .
{+K − 2 v    φ− 2 v    φ− 2 v3 .3 .4 .4 .
 .l n P − 2 v3 .3 .4 .4 .
 .l+}
{+n P 10 K   φ 10 K   φ 10 K 3 .3 .
+}
.l
{+n P 10 K 3 .3 .4 .
+}
.l
n P 0 .2 .5 .
{+φ 0 3 .3 .4 .4 .
+}
.l
n P 0 .2 .5 .
{+φ 0 3 .3 .4 .4 .
+}
.l n P 4. Numerical Experiments 4.1. Sampling Mixture of Gaussians in High-dimension As a ﬁrst test case of Algorithm 1, we sample a Gaussian mixture with 2 components in 10 dimensions and estimate the relative statistical weights of its two modes.
The bottom row of Fig.
1 shows 2 d projections of the trajectories of representative chains (in black) from initializations in each of the modes (red stars) as the NF learns to model the target density (blue contours).
Running ﬁrst locally under the
Langevin sampler, the chains progressively mix between modes and grasp the difference of statistical weights also captured by the ﬁnal map T .
Quantitatively, the acceptance rate of moves proposed by the NF reaches ∼ 80% at the end of training (Fig.
1 top row).
The estimator of the relative statistical weights of each modes (the right mode A is twice as likely as the left mode B) using Eq.
(5) also converges to the exact value within a small statistical error (Fig.
1 middle row).
4.2. Radial Velocity of an Exoplanet
Next we apply our method to the Bayesian sampling of radial velocity parameters in a model close to the one studied by Price-Whelan et al.
(2017) for a star-exoplanet system.
The model has 4 parameters: an offset velocity v 0 , an amplitude K , a period P and a phase φ 0 .
Introducing θ = ( v 0 , K, φ 0 , ln P ) ∈
Θ ⊂ R 4 , the radial velocity is
[Equation]
with Ω = 2 π/P .
From a set of observations D = { v k , t k } Nk
=1 , the goal is to sample the posterior distribution over θ .
Following (Price-Whelan et al., 2017), we assume a Gaussian likelihood L ( θ ) =
N ( v k ; v ( t k ; θ ) , σ 2obs ) , with known variance σ 2obs , and the prior distributions
[Equation]
We sample N = 6 noisy observations at different times t k (red diamonds in Fig. 2) from a ground-truth radial velocity with parameters θ 0 (dashed blue line).
Using one iteration of the accept-reject Joker algorithm (Price-Whelan et al., 2017) with 10 3 samples from the prior distributions we obtainsets of likely parameters (corresponding to the gray lines in
Fig. 2), which we will use as starting points for the MCMC chains in Algorithm 1.
Note that to ensure a minimum acceptance rate of ∼ 1% the Joker samples priors of P and φ 0 only, and computes the maximum likelihood value for the “linear parameters” K and v 0 .
We assess the quality of sampling after 10 4 iterations of
Algorithm 1 on Fig. 3, looking at all the possible 2d projections of space of parameters.
The samples from Algorithm 1 (top row, in blue, ﬁnal acceptance rate of 60% , see Fig. 4) are generally covering well the modes of the marginal posterior distribution (second row), far beyond the initial samples (top row, in pink), at the exception of a light mode in the neighborhood of v 0 = 0 and φ 0 = 0 in which no chain was initialized.
This is an illustration of the need for prior knowledge of the rough location of a basin of interest to successful sample it with the proposed method.
For comparison, we also report samples accepted by the Joker algorithm from an initial draw of 10 6 samples (top row, in green).
Because of the maximum likelihood step along K and v 0 mentioned above, the posterior is not appropriately sampled along these two dimensions by this strategy.
5. Conclusion
Our results show that normalizing ﬂows can be exploited to augment MCMC schemes used in Bayesian inference with a nonlocal transport component that signiﬁcantly enhances mixing.
By design, the method blends MCMC with an optimization component similar to that used in variational inference (Blei et al., 2017), and it would be interesting to investigate further how both approaches compare on challenging applications with complex posteriors on high dimensional parameter spaces.
A. Computational details
All codes are made available through the Github repository anonymous (to be disclosed after double-blind review).
A.1. Architectures
We parametrize the map T as a RealNVP (Dinh et al., 2017), for which inverse and Jacobian determinant can be computed efﬁciently.
Its building block is an invertible afﬁne coupling layer updating half of the variables,
[Equation]
11) where s ( · ) and t ( · ) are learnable neural networks from R d/ 2 → R d/ 2 , (cid:12) denotes component-wise multiplication (Hadamard product), and k indexes the depth of the network.
In our experiments we use fully connected networks with depth 3, hidden layers of width 100 with ReLU activations.
The parameters of these networks are initialized with random Gaussian values of small variance so that the RealNVP implements a map T close to the identity at initialization.
A.2. Mixture of Gaussians experiment
Target and hyperparameters.
The target distribution is a mixture of Gaussian in dimension d = 10 with two components:
[Equation]
with weights w A = 2 / 3 , w B = 1 / 3 and centroids with non-zero coordinates only in the ﬁrst two dimensions θ A1 , 2 = (8 , 3) and θ B1 , 2 = ( − 2 , 3) .
Fig.
1 displays density slices in the ( θ 1 , θ 2 ) -plane.
We use a RealNVP network with 6 pairs of coupling layers and a standard normal distribution as a base ρ B .
A set ofindependent walkers were initialized in equal shares in modes A and B of the target.
For the optimization, we compute gradients using batches of 1000 samples corresponding to 10 consecutive repeated sampling steps of Algorithm 1 (with τ = 0 .
005 and k Lang = 1 ).
We run 4000 parameters updates using Adam with a learning rate of 0.005.
Computing log-evidence differences.
Interpreting ρ ∗ ( θ ) as a posterior distribution over a set of parameters Θ , we would like to estimate the relative evidence for modes A and B .
Denoting by A and B two sets of conﬁgurations in Θ corresponding to each mode, the difference between their log-evidence is given by
[Equation]
Once the normalizing ﬂow has been learned to assist sampling, it can be used to approximate Eq. 5.
Drawing { θ i } ni =1 from ˆ ρ
we have the Monte Carlo estimate ˆ Z ∗ ,
A of E ∗ ( 1 A ) :
[Equation]
14) with the unnormalized weights ˆ w i =
L ( θ i ) ρ
o ( θ i ) /
ˆ ρ ( θ i ) , taking the form of an importance sampling estimator.
The quality of the estimator can be monitored using an estimate of the effective sample size to adjust the variance estimate from the empirical variance
[Equation]
Finally the unknown Z ∗ cancels out in the log-evidence difference:
[Equation]
[Equation]
In the middle right panel of Fig.
1 we report the performance of this estimator using the sets
A = { θ : (cid:107) θ − θ A (cid:107) 2 ≤ 5 } and B = { θ : (cid:107) θ − θ B (cid:107) 2 ≤ 5 } , and n = 10 5 .
The estimator convergence to the exact value ln 2 / 3 − ln 1 / 3 =
ln 2 as the quality of ˆ ρ increases over training.
[Figure]
A.3. Radial velocity experiment
Setup.
The parameters of the prior are set to the values ln P min = 3 , ln P max = 5 , σ obs = 1 . 8 , σ K = 3 , µ K = 5 and σ v 0 = 1 .
The RealNVP used to learn the posterior distribution has 6 pairs of coupling layers.
The base distribution ρ B is a standard normal distribution as in the previous experiment, but the NF is learned on a whitened representation of the parameters θ .
Concretely, using the 11 initial samples from the Joker, we compute the empirical mean ˆ θ and empirical covariance ˆΣ =
(cid:80) ni
=
( θ i − ˆ θ )( θ i − ˆ θ ) (cid:62) /n .
The latter admits an eigenvalue decomposition ˆΣ =
ODO (cid:62) , with D the diagonal matrix of eigenvalues and O the orthogonal eigenvectors basis.
Using W = OD − 1 / 2 O (cid:62) , we train the normalize ﬂow to model the distribution of the whitened parameters θ W = W ( θ − ˆ θ ) .
Training.
A set of 110 independent walkers were initialized using 10 copies of each of 11 set of parameters retained from an initial iteration of the Joker algorithm from 10 3 draws of φ 0 and ln P .
Here again we use 5 consecutive repeated sampling steps of Algorithm 1 (with τ = 5 e − 6 and k Lang = 1 ) to compute gradients using batches of 550 samples.
We run
Adam for 10 4 iterations with a learning rate of 0.001.
Fig.
4 reports the evolution of the approximate KL divergence L nk along training (left) and the acceptance rates of the non-local moves proposed by the NF (middle).
We also plot the projected trajectories of 11 chains started with the initial
Joker samples updated with combined sampling component of Algorithm 1 using the ﬁnal learned T .
During the short chains that included 10 non-local resampling steps, the walkers can be seen to jump back and forth between two of the modes of the marginalized density (underlying in blue in the plot).
However mixing is not well assisted with the mode around φ 0 (see discussion in the main text as well).
B. Continuous limit of the MCMC scheme B.1. Chapman-Kolmogorov equation
Written in terms of the densities ρ ∗ and ˆ ρ (assumed to be ﬁxed for now) the transition kernel in (4) reads
[Equation]
where
[Equation]
Denoting as { ρ k ( θ ) } k ∈ N the updated probability density of the walker in the Markov chain associated with the kernel π T ( θ, θ (cid:48) ) alone, this density satisﬁes the Chapman-Kolmogorov equation
[Equation]
Using the explicit form of π T ( θ, θ (cid:48) ) in (17), after some simple reorganization this equation can be written as
[Equation]
where we deﬁned
[Equation]
Note that if we had ˆ ρ =
ρ ∗ , then R ( θ, θ (cid:48) )
= 1 and (20) would reach equilibrium in one step, ρ k +
=
ρ ∗ whatever ρ k .
B.2. Continuous limit
To take the continuous limit of (20), we modify this equation in a way that the update of the density is only partial.
Speciﬁcally, denoting ρ t the value of the density at time t ≥ 0 , we turn this equation into
[Equation]
[Equation]
where α > 0 and τ > 0 are parameters.
This will allow us to make the MCMC resampling updates on par with those of
MALA, using τ > 0 as timestep in both.
Subtracting ρ t ( θ ) from both sides of (22), dividing by τ , and letting τ → 0 gives
[Equation]
We can now add the Langevin terms that arise in the continuous limit of the compounded MCMC scheme that we use, to arrive at
[Equation]
24) where α > 0 measures the separation of time scale between the Langevin and the resampling terms.
Written in term of g t = ρ t /ρ ∗ and ˆ g t =
ˆ ρ t /ρ ∗ (now also allowed to vary with time)
Eq.
(24) reads
[Equation] [Equation]
B.3. Convergence rate
Consider the Pearson χ 2 -divergence of ρ t with respect to ρ ∗ deﬁned
[Equation]
Assuming that D 0 < ∞ and using (25) we deduce that D t satisﬁes dD t
[Equation]
[Equation]
where we used ( −∇ U ∗ · ∇ g t + ∆ g t ) ρ ∗ = ∇ · ( ρ ∗ ∇ g t ) to reexpress the ﬁrst integral in the second equality.
If we denote ˆ
G t = inf θ ∈ Θ ˆ g t ( θ ) ∈
[0 , 1] , (27) implies
[Equation]
) where we used the normalization conditions (cid:82) Θ
g t ( θ ) ρ ∗ ( θ ) dθ =
(cid:82) Θ ˆ ρ ( θ ) dθ = 1 .
As a result, using Gronwall inequality we deduce
[Equation]
This equation indicates that D t → 0 as t → ∞ as long as (cid:82) t 0 ˆ G s ds → ∞ .
That is, convergence can only fail if ˆ G t
= o ( t − 1 ) as t → ∞ , and it is guaranteed otherwise.
Convergence is also exponential asymptotically, as long as ˆ G t remains bounded away from 0 as t → ∞ .
To get a more explicit convergence rate, let us analyze (29) in two subcases.
First let us assume that the map is not trained, i.e. ˆ g t ( θ )
=
ˆ g ( θ ) is ﬁxed, and denote ˆ G = inf θ
∈ Θ
ˆ g ( θ ) ∈
In this case, (29) reduces to
[Equation]
Note that this bound is only nontrivial if ˆ G > 0 .
Even if that is the case, the rate in (30) can be pretty poor if ˆ G is very small (e.g exponentially small in the input dimension d ), which is to be expected if the map is not trained.
The best case scenario is of course the idealized situation when ˆ G = 1 , which requires that ˆ g
= 1 (i.e. ˆ ρ = ρ ∗ ) because of the normalization conditions (cid:82) Θ ˆ g [Equation] θ
[Equation] dθ [Equation] = 1 : this case is the continuous equivalent of the one step convergence of the discrete MCMC scheme with resampling from ρ ∗ .
Second let us assume that ˆ g t =
g t , that is the trained distribution instantaneously follows the walkers distribution at all times.
In this case, (29) reduces to
[Equation]
31) where we denote
[Equation]
[Equation]
To make this bound explicit, let us consider the evolution of G t .
Denoting θ t = argmin θ ∈ Θ g t ( θ ) so that G t =
g t ( θ t ) , and using min( g t ( θ t ) , g t ( θ (cid:48) )) =
g t ( θ t ) =
G t , ∇ g t ( θ t ) = 0 , and ∆ g t ( θ t ) ≥ 0 by deﬁnition of θ t , from (25) we have
[Equation]
(33) where we used again the normalization conditions (cid:82) Θ g
[Equation]
[Equation]
Θ
[Equation]
θ
[Equation]
dθ
[Equation]
= 1 .
Eq.
(33) implies that
[Equation]
[Equation]
which after integration gives
[Equation] [Equation]
This means that we have
[Equation] [Equation]
Inserting this equation in (31) and performing the integral explicitly gives
[Equation]
[Equation]
This bound is only nontrivial if G 0 ∈ (0 , 1] .