Bayesian Inference in Physics-Driven Problems with Adversarial Priors
Abstract
Generative adversarial networks (GANs) have found multiple applications in the solution of inverse problems in science and engineering.
These applications are driven by the ability of these networks to learn complex distributions and to map the original feature space to a low-dimensional latent space.
In this manuscript we consider the use of GANs as priors in physics-driven Bayesian inference problems.
Within this approach the posterior distribution is learnt by mapping the problem to the latent space of the GAN and then using an HMC sampler for efﬁcient sampling.
We apply this approach to solving linear and nonlinear inverse problems, including an example with experimental data acquired from an application in biophysical imaging.
Furthermore, we analyze the weak convergence of the approximate prior to the true prior and elucidate its dependence on the capacity of the network and the number of training samples.
Introduction
Bayesian inference is often used to quantify the uncertainty in inferring parameters and ﬁelds for problems that are constrained by physical principles [1, 2, 3].
Within this approach, Bayes rule is used to assert that the conditional distribution of the inferred parameters is proportional to the product of the likelihood of the measurement and the prior distribution of the parameters.
Thereafter, any point estimate for the posterior-distribution can be evaluated by sampling from this distribution.
There are two challenges in applying Bayesian inference to practical problems.
The ﬁrst is the so-called curse of dimensionality which restricts its application to problems with small number of independent parameters (in the tens).
The second is the difﬁculty associated with constructing complex prior distributions that need to be learned from data.
Generative adversarial networks (GANs) [4] are type of deep generative models which can address both the challenges described above.
Through adversarial learning, the generative component of a
GAN learns complex distributions from i.i d. samples and can therefore be used to represent the prior distribution of parameters.
Further, a GAN derives its efﬁciency by mapping the feature space to a lower-dimensional latent space.
In doing so it also provides a lower-dimensional representation for the parameters to be inferred, thus making the inference problem tractable.
Signiﬁcant recent work has been done on using GANs for solving inverse problems.
This includes using GANs for solving problems in compressed sensing [5], to learn regularizers [6], to learn the posterior distribution in a linear inverse problem [7], and as priors in linear inverse problems arising in computer vision and physics [8].
In this manuscript we describe two new developments along these lines.
We focus on the case where a Wasserstein GAN (WGAN) [9, 10] is used to represent the prior density in Bayesian inference.
Our contributions are: (1) a proof of convergence of the prior density learned by the WGAN to the underlying true distribution with increasing model capacity and training data.
This result provides the foundation for the use of WGAN in Bayesian inference to estimate the posterior density.
The application of this approach to non-linear physics-informed inverse problems in heat conduction and elasticity.
For the inverse elasticity problem we present results on experimental data.
Problem formulation
We consider the problem of inferring a ﬁeld x ∈ Ω x ⊂ R N x from the measurement
y ∈
Ω
y ⊂ R N y .
Let f : Ω x (cid:55)→ Ω y be the forward map i.e., y = f
( x ) , which is assumed to be known either exactly or approximately.
In practice, the measurements may be corrupted by noise, ˆ y =
f
( x ) + η , where ˆ y is the noisy measurement and η is the noise.
The well-posedness of f does not guarantee the well-posedness of its inverse, making the inference of x from ˆ y challenging.
Bayesian inference provides a way to tackle the lack of well-posedness.
We assume a prior distribution p X on x .
Let p l ( y | x ) be the likelihood of y given an instance of x .
Then, we get the posterior distribution of x given ˆ y by Bayes’ rule
[Equation]
where p η is the model distribution for noise, and Q is the prior-predictive distribution of x .
The posterior encapsulates the uncertainty in x given the measured data.
However, the dimension N x of x is typically very large.
This makes the estimation of statistical quantities computationally challenging.
This problem can be resolved by mapping the x to a latent space variable z ∈ R N z sampled from a distribution p Z , with N z (cid:28) N x .
We achieve this by training a WGAN with a generator g N θ ( z ; θ ) and discriminator d N φ ( x ; φ ) , where θ ∈ R N θ , φ ∈ R N φ are the trainable network parameters.
The training makes use of a dataset S
= { x (1) , · · · , x ( N ) } of N i.i.d.
samples from the true distribution p X to learn the true distribution.
Speciﬁcally, by assuming that a) the discriminator is smooth, and b) the set of derivatives of the discriminator with respect to its parameters form a dense subset of the space of bounded continuous functions in the limit of inﬁnite capacity, we prove the following statistical error estimate for any F ∈ C b (Ω x ) in Appendix A,
[Equation]
where p NZ is the distribution of z N :
=
( z (1) , ..., z ( N ) ) while C and α are positive constants depending on F .
The result above demonstrates how the point estimate
[Equation]
] can be approximated by a ﬁnite sum of samples drawn from the latent space and pushed through the generator.
Under the additional assumption that f and p η are continuous, we can show that in the limit of inﬁnite capacity ( N φ → ∞ ) and with inﬁnite sampling ( N → ∞ ), we have (see Appendix B)
[Equation]
for every F ∈ C b (Ω X ) , where g ∞ is the generator obtained in the limit and z is drawn from the posterior distribution of the latent space
[Equation]
Thus our approach to estimating any point estimate for the posterior involves: (a) training the generator of the GAN with i.i.d.
samples drawn from p X , (b) training an MCMC algorithm using (4) to generate samples from p post Z , and (c) using these samples in the RHS of (3) to approximate the desired point estimate.
We note that all these calculations are restricted to the latent space and the signiﬁcantly smaller dimension of z makes the computation of various statistical quantities tractable.
Results
Inverse heat conduction - inferring thermal conductivity We consider a non-linear coefﬁcient inversion problem arising in thermal imaging.
The forward model is given by the steady-state heat conduction equation (see Appendix C.1).
The goal of the coefﬁcient inversion problem is to recover [Figure] the thermal conductivity x ( s ) at each location given the noisy (and potentially partial) measurement of temperature y
( s ) .
We consider two distinct instances of the dataset S : one is obtained by sampling from an underlying parametric description, and the other is not.
The parametric dataset allows the computation of the true posterior statistics, thereby enabling the veriﬁcation of estimated statistics.
For the parametric dataset, we consider conductivity with a background of 1, and a rectangular sub-domain where it varies linearly from 50 units on the left edge to 100 units on the right edge.
The coordinates of the the lower left and upper right corners of the rectangle parameterize this distribution.
The dataset
S contains 10,000 images generated by sampling each parameter from a uniform distribution.
For the non-parametric case, we consider MNIST dataset.
Realizations from the true prior density as well as the density learned by the WGAN are shown in the ﬁrst two panels of Figure 1.
In the right panel of Figure 1 we have shown the true thermal conductivity, the measured temperature (with and without noise), the ﬁelds inferred using our method (MAP, mean and standard deviation) and the noisy temperature.
For the parametric case, the true mean and SD are determined by Monte Carlo sampling in the parametric space.
We observe that (a) the MAP is close to the true distribution in every case, even in the presence of signiﬁcant noise; (b) the estimated mean and SD are close to the true values for the case where the latter can be determined; (c) the SD is large in regions where the conductivity has sharp gradients; (d) even when a large amount of measured signal is occluded (last row), the strong prior for the problem is able to accurately recover the true conductivity.
Inverse Radon transform We demonstrate the efﬁcacy of the proposed Bayesian inference strategy in the context of inverse Radon transforms.
Given an input image x ∈ R N × N , the forward maps leads to an output y ∈ R N × N comprising N 2 one-dimensional Radon transforms, which are essentially integrals of the image density along parametrized straight lines passing though the image (see Appendix C.2).
For the current problem, the inferred ﬁeld is generated using the modiﬁed SheppLogan head phantom [11], which is composed of a union of ten ellipses with distinct constant relative densities.
The total density at any point in the phantom ranges from 0 (air cavity) to 1 (bone).
We generated a data set S of 10,000 phantom images of dimension 128 × 128 by randomly perturbing the axes lengths, axis inclination and densities of each ellipse, and further rotating and linearly shifting the whole phantom.
Care was taken to ensure that the perturbed images did not violate the underlying spatial topology of the phantom, and that the density at each point stayed within the valid range.
A few samples from this data set are shown in the the left panel of Figure 2.
The output of the forward map, also called a sinogram, is depicted in the second column of the right panel in
Figure 2 for the phantom in column one of the same panel.
A WGAN is trained using the data set
S to approximate the prior of the inferred ﬁeld.
The realizations from the learned prior shown in the second panel of Figure 2, once again demonstrate the strength of GANs in capturing the target distribution.
We choose a sample from the true prior (not in the training set) and add Gaussian noise to the measurement.
We consider three different noise levels, as shown in each row of the right panel in Figure 2.
For each case, we recover the distribution of the likely phantom image using the MCMC approach, with the MAP estimate, mean and pixel-wise standard deviations shown in the last three columns of the left panel in Figure 2.
Even for the highest noise level in the measurement, we get an excellent recovery of the phantom, with a relatively low variance concentrated along the boundaries of the ellipses.
Elasticity imaging We provide results for an imaging problem involving experimental measurements.
Elasticity imaging is a technique of inferring mechanical properties of tissue from displacement data collected via different medical imaging modalities [12].
The forward problem is given by elasticity problem which solves for the displacement ﬁeld of an incompressible linear elastic solid (see
Appendix C.3.
We are interested in recovering the shear modulus ﬁeld x , given a noisy observation of displacement ﬁeld y .
For this study, we used experimental data obtained from a physical phantom experiment [13].
The phantom was manufactured from a mixture of gelatin, agar, and oil and contained a spherical inclusion with an elevated shear modulus compared to the background.
The phantom was subjected to uniaxial loading and the interior deformation was measured using ultrasound.
The sample set S contained 3,000 images of elliptical inclusions centered around different locations inside discretized into 56 2 grid points.
The ratio of the shear modulus of inclusion to that of the background was varied between 1:1 and 8:1 to account for a wide range of possibilities.
A WGAN was trained using this sample set and the learned GAN prior was used in Bayesian inference in conjunction with the experimentally measured displacement ﬁeld (shown in the leftmost panel of
Figure 3).
The reconstruction results are shown in the middle panel of Figure 3 and reveal the circular inclusion.
These results compare very well with the true physical quantities for the inclusion (see
Appendix D).
Since the measurement data for this study was obtained experimentally, we do not know the true measurement noise.
Given this, we tested the robustness of our algorithm with different assumed values of measurement noise.
As shown in the Figure 3, the algorithm produces consistent reconstruction results across different noise values, with elevated pixel-wise standard deviation for the case where higher value of noise is assumed.
[Figure]
Conclusions
We have demonstrated how WGANs can be used to learn the prior in a range of Bayesian inference problems.
Since GANs inherently map the inferred ﬁeld to a latent space with a signiﬁcantly lower dimension, we are able to alleviate the curse of dimensionality, while being able to capture complex distributions.
Coupled with an HMC algorithm to determine the posterior, we have presented three physics-driven problems to show the robustness of our method to quantify the uncertainty in determining the underlying inferred ﬁeld, particularly when the measurement with relatively high noise, which might even be partially occluded. Acknowledgements
The support from ARO grant W911NF2010050 is acknowledged.
A Error estimate for WGAN predictions
Let the generator of the WGAN be g N θ
( z ; θ ) , where
[Equation]
Similarly, the discriminator d N φ ( x ; φ ) is the function
[Equation]
Consider the loss function being optimized during the WGAN training
[Equation]
where { x (1) , x (2) , ..., x ( N ) } and { z (1) , z (2) , ..., z ( N ) } are a particular set of N i.i.d.
realizations of x ∼ p X and z ∼ p Z , respectively.
Denote the joint probability distribution of x N :=
( x (1) , ..., x ( N ) ) , z N :=
( z (1) , ..., z ( N ) ) and ( x N , z N ) as p
[Equation]
i
[Equation]
respectively, where p X,Z is the joint distribution of ( x , z ) .
Deﬁne the optimal parameter values
[Equation]
for a ﬁxed N and a ﬁxed architecture of the WGAN.
Then we have the following result for convergence of the WGAN.
Theorem A.1.
Consider the loss function (7) being optimized to train the WGAN, with (8) denoting the optimal parameters for a particular number of samples N and a particular ﬁnite network architecture.
Let us assume the following conditions are true:
A
The domain Ω X is closed and bounded.
A2 The samples for x and z are chosen independently, with p X,Z = p X p Z .
A3 The discriminator is a C 1 function of it inputs and parameters φ .
A4 Let the derivates of the discriminator with respect to its parameters, denoted by
[Equation]
We assume that given F ∈ C b (Ω X ) , there exists α > 0 and integers (cid:101) N, (cid:101) N φ > 0 such that
for all N ≥ (cid:101) N and N φ ≥ (cid:101)
N φ
[Equation]
for
[Equation] [Equation]
where (cid:101) N, (cid:101) N φ and the constant C F depend only on F and the activation function used in the networks.
Then, given an F ∈ C b (Ω X ) and the corresponding (cid:101) N, (cid:101) N φ , α and C F from assumption A4, the following error estimate holds true
[Equation]
for N ≥ (cid:101) N , N φ ≥ (cid:101) N φ .
Proof.
Since θ ∗ ,N θ N , φ ∗ ,N φ N satisfy (8), the necessary optimality conditions are
[Equation]
Using (7) in (13) and the notation introduced in (9)
[Equation]
we get
[Equation]
Given F ∈ C b (Ω X ) , we use (14) and Assumption A4 to get
[Equation]
for N ≥ (cid:101) N, N φ ≥ (cid:101) N
φ , where (cid:101) N, (cid:101) N
φ , α and k (cid:48) are as deﬁned in Assumption A4.
Using assumption
A2, we can split the joint probability distribution of ( x N , z N ) as
[Equation]
Since F is bounded, we have
[Equation]
Taking expectation on both sides of (15) with respect to p NX,Z and using the linearity of expectation and the standard Monte Carlo error estimate, we have for N ≥ (cid:101) N, N φ ≥ (cid:101) N φ
[Equation]
Remark A.1.
We make a few comments about the assumptions of the above theorem.
• Assumption A1 is meaningful since the training data is typically scaled to a suitable hypercube before being used with the network.
• Assumptions A1 and A3 ensure that w N φ k ∈ C b (Ω X ) .
Note that A3 can be guaranteed if we take a C 1 activation function in the discriminator.
• The space C b (Ω X ) is separable, i.e., it is guaranteed to contain a countable dense subset.
Assumption A4 implies that
in the limit of inﬁnite capacity ( N φ → ∞ ) and with sufﬁcient number of samples ( N → ∞ ), the set { w N φ k } converges to such a countable dense subset of C b (Ω X ) .
Note that this is indeed a subset due to Assumption A3.
• Another implicit implication of Assumption A4 is that the trained WGAN steers away from modal collapse.
B Convergence to the Bayesian posterior
Let us assume that in the limit of inﬁnite capacity ( N φ → ∞ ) and inﬁnite sampling ( N → ∞ ) , the generator g N θ ( z ; θ ∗ ,N θ N ) converges to g ∞ ( z ) .
Then in this limit, the result of Theorem A.1 implies that
[Equation]
for any F ∈ C b (Ω X ) .
Thus, the approximate prior weakly converges to the true prior.
Furthermore, let us assume that the forward map f noise density p η are both continuous functions.
Consider the function
[Equation]
Then clearly G ∈ C b (Ω X ) .
Substituting (18) in (17), and using (1) gives us
[Equation]
where z is sampled from the posterior distribution of the latent space (4).
In other words, we obtain an expression for evaluating the statistics with respect to the posterior distribution.
Since (19) holds for any F ∈ C b (Ω X ) , we get the weak convergence of the posterior distribution.
C Forward models
The various forward models considered in this work are described below.
C.1 Heat conduction
Consider the following steady-state heat conduction equation
[Equation]
[Equation]
(21) where x ( s ) , y ( s ) , and b ( s ) denote thermal conductivity, temperature, and the heat source respectively.
The goal of coefﬁcient inversion problem is to recover the thermal conductivity at each location given the noisy (and potentially partial) measurement of temperature.
For our simulations, we take the source term to be constant, with b ( s ) = 100 .
C.2 Radon transform
The forward map for this problem is given by one-dimensional Radon transforms, which are integrals of the phantom density along straight lines passing through the phantom image.
Given an input x image of size N × N , the forward map evaluates the line integral of N equally spaced lines though the image inclined at an angle ψ for N equally spaced angles in (0 ◦ , 180 ◦ ) ,
[Equation]
) where (cid:96) t i ,ψ j is the line through the image inclined at an angle ψ j and at a signed-distance of t i from the center of the image.
C.3 Elasticity imaging
Consider the following partial differential equation,
[Equation]
where, σ = 2 x ( ∇ s y + ( ∇ · y ) 1 ) for plane-stress incompressible linear elastic solid, x ∈ R N × N is the discretized shear modulus and y ∈ R N × N × 2 is the discretized displacement ﬁeld.
We are interested in recovering the shear modulus ﬁeld x , given a noisy observation of displacement ﬁeld y .
D Errors in the Elasticity Imaging Reconstructions
The important physical parameters of the inclusion and their reconstructed values are:
Horizontal dimension: true = 10 mm, MAP-based reconstruction = 10.22 ± 0.49 mm, meanbased reconstruction = 10.06 ± 0.27 mm. 2.
vertical dimension: true = 10 mm, MAP-based reconstruction=10.30 ± 0.35 mm, mean-based reconstruction=10.30 ± 0.35 mm. 3.
Mean true shear modulus: true = 10.7 kPa, MAP-based reconstruction = 11.05 ± 1.32 kPa, mean-based reconstruction=10.94 ± 0.62 kPa