M ULTISCALE I NVERTIBLE G ENERATIVE N ETWORKS FOR H IGH -D IMENSIONAL B AYESIAN I NFERENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
High-dimensional Bayesian inference problems cast a long-standing challenge in generating samples, especially when the posterior has multiple modes.
For a wide class of Bayesian inference problems equipped with the multiscale structure that low-dimensional (coarse-scale) surrogate can approximate the original highdimensional (ﬁne-scale) problem well, we propose to train a Multiscale Invertible Generative Network (MsIGN) for sample generation.
A novel prior conditioning layer is designed to bridge networks at different resolutions, enabling coarse-toﬁne multi-stage training.
Jeffreys divergence is adopted as the training objective to avoid mode dropping.
On two high-dimensional Bayesian inverse problems, MsIGN approximates the posterior accurately and clearly captures multiple modes, showing superior performance compared with previous deep generative network approaches.
On the natural image synthesis task, MsIGN achieves the superior performance in bits-per-dimension compared with our baseline models and yields great interpret-ability of its neurons in intermediate layers.
I NTRODUCTION
Bayesian inference provides a powerful framework to blend prior knowledge, data generation process and (possibly small) data for statistical inference.
With some prior knowledge ⇢ (distribution) for the quantity of interest x 2 R d , and some (noisy) measurement y 2 R d y , it casts on x a posterior q
[Equation]
where L ( y | x ) = N ( y  F ( x ); 0 ,  " ) .
(1) where L ( y | x ) is the likelihood that compares the data y with system prediction F ( x ) from the candidate x , here F denotes the forward process. We can use different distributions to model the mismatch " = y  F ( x ) , and for illustration simplicity, we assume Gaussian in Equation 1. For example, Bayesian deep learning generates model predicted logits F ( x ) from model parameters x , and compares it with discrete labels y through binomial or multinomial distribution.
Sampling or inferring from q is a long-standing challenge, especially for high-dimensional (high- d ) cases.
An arbitrary high- d posterior can have its importance regions (also called “modes”) anywhere in the high- d space, and ﬁnding these modes requires computational cost that grows exponentially with the dimension d .
This intrinsic difﬁculty is the consequence of “the curse of dimensionality”, which all existing Bayesian inference methods suffer from, e.g., MCMC-based methods (Neal et al., 2011; Welling & Teh, 2011; Cui et al., 2016), SVGD-type methods (Liu & Wang, 2016;
Chen et al., 2018; 2019a), and generative modeling (Morzfeld et al., 2012; Parno et al., 2016; Hou et al., 2019).
In this paper, we focus on Bayesian inference problems with multiscale structure and exploit this structure to sample from a high- d posterior.
While the original problem has a high spatial resolution (ﬁne-scale), its low resolution (coarse-scale) analogy is computationally attractive because it lies in a low-dimension (low- d ) space.
A problem has the multiscale structure if such coarse-scale low- d surrogate exists and gives good approximation to the ﬁne-scale high- d problem, see Section 2.1.
Such multiscale property is very common in high- d Bayesian inference problems.
For example, inferring 3-D permeability ﬁeld of subsurface at the scale of meters is a reasonable approximation of itself at the scale of centimeters, while the problem dimension is 10 6 -times fewer.
We propose a Multiscale Invertible Generative Network (MsIGN) to sample from high- d Bayesian inference problems with multiscale structure.
MsIGN is a ﬂow-based generative network that can both generate samples and give density evaluation. It consists of multiple scales that recursively lifts up samples to a ﬁner-scale (higher-resolution), except that the coarsest scale directly samples from a low- d (low resolution) distribution.
At each scale, a ﬁxed prior conditioning layer combines coarse-scale samples with some random noise according to the prior to enhance the resolution, and then an invertible ﬂow modiﬁes the samples for better accuracy, see Figure 1.
The architecture of
MsIGN makes it fully invertible between the ﬁnal sample and random noise at all scales.
MsIGN undergoes a multi-stage training that learns a hierarchy of distributions with dimensions growing from the lowest to the highest (the target posterior).
Each stage gives a good initialization to the next stage thanks to the multiscale property.
To capture multiple modes, we choose Jeffreys divergence D J ( p k q ) as the training objective at each stage, which is deﬁned as
[Equation]
Jeffreys divergence removes bad local minima of single-sided Kullback-Leibler (KL) divergence to avoid mode missing.
We build an unbiased estimation of it by leveraging prior conditioning layer in importance sampling.
Proper loss function and good initialization from multi-stage training solve the non-convex optimization stably and capture multi-modes of the high- d distribution.
In summary, we claim four contributions in this work.
First, we propose a Multiscale Invertible deep Generative Network (MsIGN) with a novel prior conditioning layer, which can be trained in a coarse-to-ﬁne scale manner.
Second, Jeffreys divergence is used as the objective function to avoid mode collapse, and is estimated by importance sampling based on the prior conditioning layer.
Third, when applied to two Bayesian inverse problems, MsIGN clearly captures multiple modes in the high- d posterior and approximates the posterior accurately, demonstrating its superior performance compared with previous methods via the generative modeling approach.
Fourth, we also apply MsIGN to image synthesis tasks, where it achieves superior performance in bits-perdimension among our baseline models, like Glow (Kingma & Dhariwal, 2018), FFJORD (Grathwohl et al., 2018), Flow++ (Ho et al., 2019), i-ResNet (Behrmann et al., 2019), and Residual Flow (Chen et al., 2019b). MsIGN also yields great interpret-ability of its neurons in intermediate layers.
M ETHOLOGY
We will abbreviate q ( x | y ) in Equation 1 as q ( x ) for simplicity in the following context, because y only plays the role of deﬁning the target distribution q ( x ) in MsIGN. In Section 2.1, we discuss the multiscale structure in detail of the posterior q ( x ) and derive a scale decoupling that can be utilized to divide and conquer the high- d challenge of Bayesian inference.
As a ﬂow-based generative model like in Dinh et al.
(2016), MsIGN models a bijective that maps Gaussian noise z to a sample x whose distribution is denoted as p ✓ ( x ) , where ✓ is the network parameters. MsIGN allows fast generation of samples x and density evaluation p ✓ ( x ) , so we train our working distribution p ✓ ( x ) to approximate the target distribution q ( x ) .
We present the architecture of MsIGN in Section 2.2 and the training algorithm in Section 2.3.
2. M ULTISCALE S TRUCTURE AND S CALE D ECOUPLING
We say a Bayesian inference problem has multiscale structure if the associated coarse-scale likelihood L c approximates the original likelihood L
well:
[Equation] where L c ( y | x c ) : = N ( y  F c ( x c ) [Equation]
0 ,  " ) . [Equation]
Here x c 2 R d c is a coarse-scale version of the ﬁne-scale quantity x 2 R d ( d c < d ), given by a deterministic pooling operator A : x c =
A ( x ) . The map F c : R d c !
R d y is a forward process that gives system prediction based on the coarse-scale information x c . A popular case of the multiscale structure is when A is the average pooling operator, and F ( x ) ⇡ F c ( x c ) , meaning that the system prediction mainly depends on the lower-resolution information x c .
Equation 3 motivates us to deﬁne a surrogate distribution ˜ q ( x ) / ⇢ ( x ) L c ( y |A ( x )) that approximates the target posterior q ( x ) well 1 : [Equation]
[Equation]
We also notice that the prior ⇢ allows an exact scale decoupling. To generate a sample x from ⇢ , one can ﬁrst sample its coarse-scale version x c = A ( x ) , and then replenish missing ﬁne-scale details without changing the coarse-scale structure by sampling from the conditional distribution ⇢ ( x | x c ) = ⇢ ( x |A ( x ) = x c ) .
Using ⇢ c to denote the distribution of x c = A ( x ) , the conditional probability calculation summarizes this scale decoupling process as ⇢ ( x ) = ⇢ ( x | x c ) ⇢ c ( x c ) . Combining the scale effect in the likelihood and the scale decoupling in the prior, we decouple the surrogate ˜ q ( x ) = ⇢ ( x ) L c ( y |A
( x )) into the prior conditional distribution ⇢ ( x | x c ) and a coarse-scale posterior, deﬁned as q c ( x c ) := ⇢ c ( x c ) L ( y | x c ) . The decoupling goes as
[Equation] The prior conditional distribution ⇢ ( x | x c ) bridges the coarse-scale posterior q c ( x c ) and the surrogate ˜ q ( x ) , which in turn approximates the original ﬁne-scale posterior q ( x ) .
Parno et al.
(2016) proposed a similar scale decoupling relation, and we leave the discussion and comparison to Appendix A.
Figure 1 shows the integrated sampling strategy.
To sample an x from q , we start with an x c from q c .
The prior conditioning layer then performs random upsampling from the prior conditional distribution ⇢ ( ·| x c ) , and the output will be a sample ˜ x of the surrogate ˜ q .
Due to the approximation ˜ q ⇡ q from Equation 4, we stack multiple invertible blocks for the invertible ﬂow F to modify the sample ˜ x ⇠ ˜ q to a sample x ⇠ q : x = F (˜ x ) . F is initialized as an identity map in training.
Finally, to obtain the x c from q c , we apply the above procedure recursively until the dimension of the coarsest scale is small enough so that q c can be easily sampled by a standard method.
2. M ULTISCALE I NVERTIBLE G ENERATIVE N ETWORK : A RCHITECTURE
Our proposed MsIGN has multiple levels to recursively apply the above strategy.
We denote L the number of levels, x l 2 R d l the sample at level l , and A l : R d l ! R d l  1 the pooling operator from level l to l  1 : x l  1 =
A l ( x l ) .
Following the idea in Section 2.1, we can deﬁne the l -th level target q l ( x l ) and surrogate ˜ q l (˜ x l ) , and the last-level target q L is our original target q in Equation 1.
The l -th level of MsIGN uses a prior conditioning layer PC l and an inverse transform F l to capture q l .
Prior conditioning layer.
The prior conditioning layer PC l at level l lifts a coarse-scale sample x l  1 2 R d l  1 up to a random ﬁne-scale one x l 2 R d l following the conditional distribution ⇢ ( x l | x l  1 ) .
The difference in dimension is compensated by a Gaussian noise z l 2 R d l  d l  1 , which is the source of randomness: x l = PC l ( x l  1 , z l ) .
PC l depends only on the prior conditional distribution ⇢ ( x l | x l  1 ) , and thus can be pre-computed independently for different levels regardless of the likelihood L .
When the prior is Gaussian and the pooling operators are linear (e.g., average pooling), the prior conditional distribution is still Gaussian with moments speciﬁed as follows.
Lemma 2.1 Suppose that ⇢ ( x l ) =
N ( x l ; 0 , ⌃ l ) , and A l ( x l ) =
A l x l for some A l 2 R d l  1 ⇥ d l , then with U l  1 := ⌃ l A Tl ( A l ⌃ l A Tl )  1 and ⌃ l
| l  1 :=
⌃ l  ⌃ l A Tl ( A l ⌃ l A Tl )  1 A l ⌃ l , we have ⇢ ( x l | x l  1 = A l x l ) = N ( x l ; U l  1 x l  1 , ⌃ l | l  1 ) .
With the Cholesky decomposition (or eigen-decomposition) ⌃ l | l  1 = B l B Tl , we design the prior conditioning layer PC l as below, which is invertible between x l and ( x l  1 , z l ) :
[Equation]
We refer readers to Appendix B for proof of Lemma 2.1 and the invertibility in Equation 6.
When the prior is non-Gaussian or the pooling operators are nonlinear, there exists a nonlinear invertible prior conditioning operator x l = PC l ( x l  1 , z l ) such that x l follows the prior conditional distribution ⇢ ( x l | x l  1 ) given x l  1 and z l ⇠ N ( 0 , I d l  d l  1 ) .
We can pre-train an invertible network to approximate this sampling process, and ﬁx it as the prior conditioning layer.
Invertible ﬂow.
The invertible ﬂow F l at level l modiﬁes the surrogate ˜ q l towards the target q l .
The more accurate the multiscale structure in Equation 3 is, the better ˜ q l approximates q l , and the closer
F l is to the identity map.
Therefore, we parameterize F l by some ﬂow-based generative model and initialize it as an identity map.
In practice, we utilize the invertible block of Glow (Kingma &
Dhariwal, 2018), which consists of actnorm, invertible 1 ⇥ 1 convolution, and afﬁne coupling layer, and stack several blocks as the inverse ﬂow F l in MsIGN.
Overall model.
MsIGN is a bijective map between random noise inputs at different scales { z l } Ll = and the ﬁnest-scale sample x L . The forward direction of MsIGN maps { z l } Ll =1 to x L as below:
[Equation]
As a ﬂow-based generative model, sample generation as in Equation 7 and density evaluation p ✓ ( x ) by the change-of-variable rule is accessible and fast for MsIGN. In scenarios when certain bound needs enforcing to the output, we can append element-wise output activations at the end of MsIGN. For example, image synthesis can use the sigmoid function so that pixel values lie in [0 , 1] . Such activations should be bijective to keep the invertible relation between random noise to the sample.
2. M ULTISCALE I NVERTIBLE G ENERATIVE N ETWORK : T RAINING
Since the prior conditioning layer PC is pre-computed and the output activation G is ﬁxed, only the inverse ﬂow F contains trainable parameters in MsIGN.
We train MsIGN with the following strategy so that the distribution p ✓ of its output samples, where ✓ is the network parameter, can approximate the target distribution q deﬁned in Equation 1 well.
Multi-stage training and interpret-ability.
The multiscale strategy in construction of MsIGN enables a coarse-to-ﬁne multi-stage training.
At stage l , we target at capturing q l , and only train invertible ﬂows before or at this level: F l 0 , l 0  l .
Equation 4 implies that q l can be well approximated by the surogate ˜ q l , which is the conditional upsampling from q l  1 as in Equation 5.
So we use ˜ q l to initialize our model by setting F l 0 , l 0 < l as the trained model at stage l  1 and setting F l as the identity map.
Our experiments demonstrate such multi-stage strategy signiﬁcantly stabilizes training and improves ﬁnal performance.
Figure 1 and Equation 7 imply that intermediate activations, i.e., ˜ x l and x l , who are samples of predeﬁned posterior distributions at the coarse scales (see Equation 5), are semantically meaningful and interpret-able.
This is different from Glow (Kingma & Dhariwal, 2018), whose intermediate activations are not interpret-able due to the loss of spatial relation.
Jeffreys divergence and importance sampling with the surrogate.
The KL divergence is easy to compute, and thus is widely used as the training objective.
However, its landscape could admit local minima that don’t favor the optimization.
Nielsen & Nock (2009) suggests that D KL ( p ✓ k q ) is zero-forcing, meaning that it enforces p ✓ be small whenever q is small.
As a consequence, mode missing can still be a local minimum, see Appendix C.
Therefore, we turn to the Jeffreys divergence deﬁned in Equation 2 which penalizes mode missing much and can remove such local minima.
Estimating the Jeffreys divergence requires computing an expectation with respect to the target q , which is normally prohibited.
Since MsIGN constructs a good approximation ˜ q to q , and ˜ q can be constructed from coarser levels in multi-stage training, we do importance sampling with the surrogate ˜ q for the Jeffreys diveregence and its derivative (see Appendix D for detailed derivation):
[Equation]
With the derivative estimate given above, we optimize the Jeffreys divergence by stochastic gradient descent. We remark that @ log p ✓ ( x ) / @ ✓ is computed by the backward propagation of MsIGN.
R ELATED W ORK
Invertible generative models (Deco & Brauer, 1995) are powerful exact likelihood models with efﬁcient sampling and inference.
They have achieved great success in natural image synthesis, see, e.g., Dinh et al.
; Kingma & Dhariwal (2018); Grathwohl et al.
; Ho et al.
; Chen et al.
(2019b), and variational inference in providing a tight evidence lower bound (ELBO), see, e.g, Rezende & Mohamed (2015).
In this paper, we propose a new multiscale invertible generative network (MsIGN) structure, which utilizes the invertible block in Glow (Kingma & Dhariwal, 2018) as building piece for the invertible ﬂow at each scale.
The Glow block can be replaced by any other invertible blocks, without any algorithmic changes.
Different from Glow, different scales of MsIGN can be trained separately, and thus features in its intermediate layers can be interpreted as lowresolution approximation of the ﬁnal high-resolution output.
This novel multiscale structure enables better explain-ability of its hidden neurons and makes training much more stable.
Different from the image synthesis task where large amount of samples from target distribution are available, in Bayesian inference problems only an unnormalized density is available and i.i.d.
samples from the posterior are the target.
This paper’s main goal is to train MsIGN to approximate certain high- d Bayesian posteriors.
Various kinds of parametric distributions have been proposed to approximate posteriors before, such as polynomials (El Moselhy & Marzouk, 2012), non-invertible generative networks (Feng et al., 2017;
Hou et al., 2019), invertible networks (Rezende & Mohamed, 2015; Ardizzone et al., 2018; Kruse et al., 2019) and certain implicit maps (Chorin & Tu, 2009; Morzfeld et al., 2012).
Generative modeling approach has the advantage that i.i.d.
samples can be efﬁciently obtained by evaluating the model in the inference stage.
However, due to the tricky non-convex optimization problem, this approach for both invertible (Chorin & Tu, 2009; Kruse et al., 2019) and non-invertible (Hou et al., 2019) generative models becomes increasingly challenging as the dimension grows.
To overcome this difﬁculty, we propose (1) to use the Jeffreys divergence as loss function, which has fewer shallow local minima and better landscape compared with the commonly-used KL divergence (see Appendix C for a concrete example), and (2) to train MsIGN in a coarse-to-ﬁne manner with coarse-scale solution serving as an initialization to ﬁne-scale optimization problem.
In Kruse et al.
(2019), authors list some recent models for low- d inverse problems.
We remark that their formulation of posterior assumes no observation or model error in Equation 1, and is different from ours. See Appendix J for detailed discussion and experimental comparison.
Other than the generative modeling, various Markov Chain Monte Carlo (MCMC) methods have been the most popular in Bayesian inference, see, e.g., Beskos et al.
; Neal et al.
Welling & Teh (2011)
; Chen et al.
Cui et al.
Particle-optimization-based sampling is a recently developed effective sampling technique with Stein variational gradient descent (SVGD)
(Liu & Wang, 2016)) and many related works, e.g., Liu (2017); Liu & Zhu (2018); Chen et al.
(2018; 2019a).
The intrinsic difﬁculty of Bayesian inference displays itself as highly correlated samples, leading to undesired low sample efﬁciency, especially in high- d cases.
The multiscale structure and multi-stage strategy proposed in this paper can also beneﬁt these particle-based methods, as we can observe that they beneﬁt the amortized-SVGD (Feng et al., 2017; Hou et al., 2019) in Section 4.1.3.
We leave a more thorough study of this topic as a future work.
Works in Parno et al.
; Matthies et al.
(2016) utilize the multiscale structure in Bayesian inference and build generative models with polynomials.
They suffer from exponential growth of parameter number for high- d polynomial basis.
The Markov property (Spantini et al., 2018) is used to alleviate this exponential growth.
Different from these works, we leverage the great capacity of invertible generative networks to parametrize the high- d distribution, and we design novel network architecture to make use of the multiscale structure.
The multiscale structure is a more general structure than commonly-used intrinsic low- d structure (Spantini, 2017; Cui et al., 2016; Chen et al., 2019a), which assumes that the density of high- d posterior concentrates in a low- d subspace. In the image synthesis task, this multiscale idea incorporates with various generative models. For example, Denton et al. ; Odena et al.
Karras et al. ; Xu et al. (2018) uses it in generative adversarial networks (GANs) to grow a high-resolution image from low-resolution ones. But the lack of invertibility in these models makes it difﬁcult for them to apply to Bayesian inference problems.
Invertible generative models like Dinh et al.
; Kingma & Dhariwal (2018);
Ardizzone et al.
(2019) adopted this multiscale idea, but their multiscale strategy is not in the spatial sense: the intermediate neurons are not semantically interpret-able, as we show in Figure 6.
We study two high- d Bayesian inverse problems (BIPs) known to have at least two equally important modes in Section 4.1 as test beds for distribution approximation and multi-mode capture: one with true samples available in Section 4.1.1; one without true samples but close to real-world applications in subsurface ﬂow in Section 4.1.2.
We also report the ablation study of MsIGN in Section 4.1.3.
In addition, we apply MsIGN to the image synthesis task to benchmark with ﬂow-based generative models and demonstrate its interpret-ability in Section 4.2.
We adopt the invertible block in Glow (Kingma & Dhariwal, 2018) as the building piece, and stack several of them to build our invertible ﬂow F .
We utilize average pooling with kernel size 2 and stride 2 as our pooling operator A .
4. B AYESIAN I NVERSE P ROBLEMS
Sample x of our target posterior distribution q is a vector on a 2-D uniform 64 ⇥ 64 lattice, which means the problem dimension d is 4096 .
Every x is equivalent to a piece-wise constant function on the unit disk: x ( s ) for s 2 ⌦ =
[0 , 1] 2 , and we don’t distinguish between them thereafter.
We place a centered Gaussian with a Laplacian-type covariance as the prior: N  0 ,  2 (   )  1  ↵  , which is very popular in geophysics and electric tomography.
See Appendix E for problem settings in detail.
The key to guarantee the multi-modality of our posteriors is the symmetry.
Combining properties of the prior deﬁned above and the likelihood deﬁned afterwards, the posterior is mirror-symmetric: q ( x ( s 1 , s 2 ))
=
q ( x ( s 1 , 1  s 2 )) .
We carefully select the prior and the likelihood so that our posterior q has at least two modes. They are mirror-symmetric to each other and possess equal importance.
As in Figure 1, we plan to learn our 4096 -D posteriors at the end of L = 6 levels, and set problem dimension at each level as d l = 2 l ⇤ 2 l = 4 l .
The training follows our multi-stage strategy, and the ﬁrst stage l = 1 is initialized by minimizing the Jeffreys divergence without importance sampling, because samples to q 1 is available since d 1 = 4 is relatively small.
See Appendix E for details.
We compare MsIGN with representatives of major approaches: amortized-SVGD (short as ASVGD)
(Feng et al., 2017) and Hamilton Monte Carlo (short as HMC) (Neal et al., 2011), for high- d BIPs, see our discussion in Section 3.
We measure the computational cost by the number of forward simulations (nFSs), because running the forward simulation F occupies most training time, especially in Section 4.1.2.
We budget a same nFS for all methods for fair comparison.
4.1. S YNTHETIC B AYESIAN I NVERSE P ROBLEMS
This problem allows access to ground-truth samples so the comparison is clear and solid.
The forward process is given by F ( x ) = h ' , x [-(cid:105) 
-]i 2 = (
R ⌦ ' ( s ) x
( s )d s ) 2 , where ' ( s ) = sin( ⇡ s 1 ) sin(2 ⇡ s 2 ) .
Together with the prior, our posterior can be factorized into one-dimensional sub-distributions, namely q ( x ) =
Q dk =
1 q k ( h w k , x i ) for some orthonormal basis { w k } dk =1 .
This property gives us access to true samples via inversion cumulative function sampling along each direction w k .
Furthermore, these 1-D sub-distributions are all single modal except that there’s one, denoted as q k ⇤ , with two symmetric modes.
In other words, the marginal distribution along w k ⇤ is double-model and the rest are uni-model.
This conﬁrms our construction of two equally important modes.
See Appendix E for more details in problem settings.
The computation budget is ﬁxed at 8 ⇥ 10 5 nFSs.
Multi-mode capture.
To visualize mode capture, we plot the marginal distribution of generated samples along the critical direction w k ⇤ , which by construction is the source of double-modality of the posterior.
The (visually) worst one in three independent experiments is shown in Figure 2(a).
[Table]
Distribution approximation.
To measure distribution approximation, we report the error of mean, variance and correlation at or between all sub-distributions, as well as the Jeffreys divergence.
Thanks to the factorization property, we compare the mean, variance and correlation estimate with theoretical groundtruths, and report the root mean square of error at all dimensions in Figure 2(b).
For MsIGN and A-SVGD that gives access to not only samples but also density, we also report the Monte Carlo estimates of the Jeffreys divergence with the target posterior in Table 1.
We can see that MsIGN has superior accuracy in approximating the target distribution.
4.1. E LLIPTIC B AYESIAN I NVERSE P ROBLEMS
This problem originates from geophysics and ﬂuid dynamics.
The forward model is given by linear measurement of the solution to an elliptic partial differential equation associated with x .
We deﬁne
[Equation]
where ' k are ﬁxed measurement functions, and u ( s ) is the solution of
[Equation]
with boundary condition
[Equation]
s ) [Equation]
This model appears frequently in real applications.
For example, x, u can be seen as permeability ﬁeld and pressure in geophysics.
However, there is no known access to true samples of q .
Again the trick of symmetry introduced in Section 4.1 and explained in Appendix E guarantees at least two equally important modes in the posterior.
We put a 5 ⇥ 10 5 -nFS budget on our computation cost.
Multi-mode capture.
Due to lack of true samples, we check the marginal distribution of the posterior along eigen-vectors of the prior, and pick a particular one to demonstrate that we can capture double modes in Figure 3(a).
We also conﬁrm the capture of multiple modes by embedding samples by Principle Component Analysis (PCA) to a 2-D space.
We report the clustering (by K-means) result and means of each cluster in Figure 3(b), where we can see that A-SVGD failed to capture the two symmetric modes, while MsIGN has a more balanced capture of the symmetric posterior.
4.1. A BLATION S TUDY OF A RCHITECTURE D ESIGN AND T RAINING S TRATEGY
We run extensive experiments to study the effectiveness of the network architecture and training stragtegy of MsIGN, see Figure 4. We refer to Appendix G for details in setting and more results.
Network architecture.
We replace the prior conditioning layer by two direct alternatives: a stochastic nearest-neighbor upsampling layer (model denoted as “MsIGN-SNN”), or the split and squeeze layer in Glow design (now the model is essentially Glow, so we also denote it as “Glow”). Figure 4(a) shows that the prior conditioning layer design is crucial to the performance of MsIGN on both problems, because neither “MsIGN-SNN” nor “Glow” has a successful mode capture.
Training strategy. We study the effectiveness of the Jeffreys divergence objective and multi-stage training.
We try substituting the Jeffreys divergence objective (no extra marks) with the KL divergence (model denoted with a string “-KL”) or kernelized Stein discrepancy (which resumes A-SVGD algorithm, model denoted with a string “-AS”), and switching between multi-stage (no extra marks) or single-stage training (model denoted with a string “-S”).
We remark that single-stage training using Jeffreys divergence is infeasible because of the difﬁculty to estimate D KL ( q k p ✓ ) . Figure 4(b) and (c) show that, all models trained in the single-stage manner (“MsIGN-KL-S”, “MsIGN-AS-S”) will face mode collapse.
We also observe that our multi-stage training strategy can beneﬁt training with other objectives, see “MsIGN-KL” and “MsIGN-AS”. We also notice that the Jeffreys divergence leads to a more balanced samples for these symmetric problems, especially for the complicated elliptic BIP in Section 4.1.2.
4. I MAGE S YNTHESIS T ASK
We train our MsIGN architecture with maximum likelihood estimation to benchmark with other ﬂow-based generative models.
The prior conditional distribution ⇢ ( x | x c ) is modeled by a simple Gaussian with a scalar matrix as its covariance and is learned from a training set.
We refer readers to Appendix H for more experimental details, and to Appendix I for additional results.
We report the bits-per-dimension value with our baseline models of ﬂow-based generative networks in Table 2.
Our MsIGN is superior in number and also is more efﬁcient in parameter size: for example, MsIGN uses 24 .
4% fewer parameters than Glow for CelebA 64, and uses 37 .
4% fewer parameters than Residual Flow for ImageNet 64.
In Figure 5, we show synthesized images of MsIGN from CelebA 64 dataset, and linear interpolation of real images in the latent feature space.
In Figure 6, we visualize internal activations at checkpoints of the invertible ﬂow at different scales which demonstrates the interpret-ability of MsIGN
[Table]
[Table]
C ONCLUSION
For high-dimensional Bayesian inference problems with multiscale structure, we propose Multiscale Invertible Generative Networks (MsIGN) and associated training algorithms to approximate the high-dimensional posterior.
In this paper, we demonstrate the capability of this approach in highdimensional (up to 4096 dimensions)
Bayesian inference problems with spatial multiscale structure, leaving several important directions as future work.
The network architecture also achieves the state-of-the-art performance in various image synthesis tasks.
We plan to apply this methodology to other Bayesian inference problems, for example, Bayesian deep learning with multiscale structure in model width or depth (e.g., Chang et al.
; Haber et al.
and data assimilation problem with multiscale structure in the temporal variation (e.g., Giles (2008)).
We also plan to develop some theoretical guarantee of the posterior approximation performance for MsIGN.