Original: Efﬁcient posterior inference & generalization in physics-based Bayesian inference with conditional GANs
Revised: Efﬁcient posterior inference & generalization in physics-based Bayesian inference with conditional GANs

Original: Abstract
Revised: Abstract

Original: In this work, we propose a conditional generative adversarial network (cGAN)to sample from the posterior of physics-based Bayesian inference problems.
Revised: In this work, we propose a conditional generative adversarial network (cGAN) to sample from the posterior of physics-based Bayesian inference problems.

Original: Weutilize a U-Net architecture for the generator and inject the latent variable usingconditional instance normalization.
Revised: We utilize a U-Net architecture for the generator and inject the latent variable using conditional instance normalization.

Original: We solve the inverse heat conduction problemand demonstrate how the proposed strategy effectively quantiﬁes the uncertaintyin the inferred ﬁeld.
Revised: We solve the inverse heat conduction problem and demonstrate how the proposed strategy effectively quantiﬁes the uncertainty in the inferred ﬁeld.

Original: We also show that the structure of the generator promotesgeneralizability due to the local-nature of the learned inverse map.
Revised: We also show that the structure of the generator promotes generalizability due to the local-nature of the learned inverse map.

Original:  Introduction
Revised:  Introduction

Original: Inverse problems are ubiquitous in science and engineering, in areas such as computerized tomography[1], seismology [2, 3], climate-modeling [4, 5], and astronomy [6].
Revised: Inverse problems are ubiquitous in science and engineering in areas such as computerized tomography [1], seismology [2, 3], climate-modeling [4, 5] and astronomy [6].

Original: Unlike the forward/direct problem,the inverse problem can be challenging to solve, and mighteven lack well-possednessin the senseof Hadarmard [7].
Revised: Unlike the forward/direct problem, the inverse problem can be challenging to solve, and might lack well-possedness [7].

Original: Bayesian inference provides a meaningful strategy to overcome many of thechallenges by posing the problem in a stochastic framework.
Revised: Bayesian inference provides a meaningful strategy to overcome these challenges by posing the problem in a stochastic framework.

Original: Speciﬁcally, Bayesian inferencecombines the prior knowledge and data likelihood to formulate an expression for the posterior.
Revised: It combines prior knowledge and data likelihood to formulate an expression for the posterior distribution.

Original: Knowing the posterior, or being able to sample from it, is useful in evaluating statistical estimatesand quantifying the reliability of the inferred ﬁeld for a given measurement.
Revised: Knowing the posterior is useful in evaluating statistical estimates and quantifying the reliability of the inferred ﬁeld for a given measurement.

Original: Over the last few years, several deep learning-based strategies have been developed to solve inverseproblems.
Revised: Over the last few years several deep learning-based strategies have been developed to solve inverse problems.

Original: Popular among these are generative adversarial networks (GANs), which are usefulin approximating the underlying distribution characterized by data [8, 9, 10, 11, 12].
Revised: Popular among these are generative adversarial networks (GANs), which are useful in approximating the underlying distribution characterized by data [8, 9, 10, 11, 12].

Original: In [13], aconditional generative adversarial network (cGAN) was developed to solve large-scale medicalimaging inverse problems in the Bayesian framework.
Revised: In [13], a conditional generative adversarial network (cGAN) was developed to solve large-scale medical imaging inverse problems in the Bayesian framework.

Original: In particular, a U-Net architecture withconvolution-residual blocks was used to construct the GAN generator.
Revised: In particular, a U-Net architecture with convolution-residual blocks was used to construct the GAN generator.

Original: Such cGAN models havebeen recently extended to solve nearshore bathymetry problems in [14].
Revised: Such cGAN models have been recently extended to solve nearshore bathymetry problems in [14].

Original: Additionally, a GAN-basedapproach was proposed in [15] to approximate many-to-many image mappings arising in computervision.
Revised: Additionally, a GAN-based approach was proposed in [15] to approximate many-to-many image mappings arising in computer vision.

Original: A U-Net architecture was used for the generator and the latent information was injected atvarious levels of the U-Net using conditional instance normalization [16].
Revised: A U-Net architecture was used for the generator and the latent information was injected at various levels of the U-Net using conditional instance normalization [16].

Original: In our work, we combine the two distinct but related themes from [13] and [15].
Revised: In our work, we combine the two distinct but related themes from [13] and [15].

Original: We designcGANs to learn the posterior in physics-based Bayesian problems, where the generator uses the U-Net architecture and conditional instance normalization.
Revised: We design cGANs to learn the posterior in physics-based Bayesian problems, where the generator uses the U-Net architecture and conditional instance normalization.

Original: The inverse heat conduction problem isconsidered, with the cGANused to infer the thermal conductivity for the steady-state problem, andthe initial condition for the unsteady problem.
Revised: We consider the inverse heat conduction, and use the cGAN to infer the thermal conductivity for the steady-state problem, and the initial condition for the unsteady problem.

Original: Additionally, we demonstrate how the architecture ofthe generator plays a crucial role in promoting generalizability for the proposed approach.
Revised: Additionally, we demonstrate how the architecture of the generator plays a crucial role in promoting generalizability for the proposed approach.

Original:  Problem formulation
Revised:  Problem formulation

Original: Consider the forward/direct problem given by y = f
Revised: Consider the forward/direct problem given by y = f

Original: ( x ) , where x ∈ Ω X is the input while y ∈Ω
Revised: ( x ) , where x ∈ Ω X is the input while y ∈

Original: Yis the response.
Revised: Ω Y is the response.

Original: A typical example is determining the temperature ﬁeld in a solid body given theheat source, boundary conditions and thermal conductivity.
Revised: A typical example is determining the temperature ﬁeld, given the heat source, boundary conditions and thermal conductivity.

Original: In this case, f is the solution map forthe underlying PDE.
Revised: In this case, f is the solution map for the underlying PDE.

Original: The associated inverse problem corresponds to inferring the heat source or thethermal conductivity ﬁeld, given the (possibly noisy) temperature ﬁeld; that is, inferring x given y .
Revised: The associated inverse problem is to infer the heat source or the thermal conductivity ﬁeld, given the noisy temperature ﬁeld; that is, inferring x given y .

Original: We solve the inverse problem in a Bayesian setting, by assuming that the inferred ﬁeld is sampledaccording to the distribution x ∼ p x .
Revised: We solve the inverse problem in a Bayesian setting, by assuming that the inferred ﬁeld is sampled according to the distribution x ∼ p x .

Original: Then the measurements are sampled using the distributiony ∼ p
Revised: Then the measurements are sampled using the distribution y ∼ p

Original: y induced by the forward map f .
Revised: y induced by the forward map f .

Original: We are interested in generating samples from the posteriorp x | y for a given the measurement y .
Revised: We are interested in generating samples from the posterior p x | y for a given the measurement y .

Original: In particular, we evaluate statistics in terms of this posterior.

Original: Following the strategy proposed in [13], we train a conditional Wassesrtein GAN to achieve this goal.
Revised: Following the strategy proposed in [13], we train a conditional Wassesrtein GAN to achieve this goal.

Original: We assume access to M realizations { x ( i ) } Mi =1 from p x , and construct the dataset of pairwise samples
Revised: We assume access to M realizations { x ( i ) } Mi =1 from p x , and construct the dataset of pairwise samples

Original: S =
Revised: S =

Original: { ( x ( i ) , y ( i ) ) }
Revised: { ( x ( i ) , y ( i ) ) }

Original: Mi =1 by solving the forward model, exactly or approximately.
Revised: Mi =1 by solving the forward model, exactly or approximately.

Original: Note that the samplesin S represent realizations from the joint distribution p xy .
Revised: Note that the samples in S represent realizations from the joint distribution p xy .

Original: Let z ∈ Ω Z ⊂ R N Z , be the latentvariable of the GAN distributed according to a distribution p z which is easy to sample from.
Revised: Let z ∈ Ω Z ⊂ R N Z , be the latent variable of the GAN distributed according to a distribution p z which is easy to sample from.

Original: Wedeﬁne the GAN generator as g : Ω Z × Ω Y (cid:55)→ Ω X , which generates samples from the induceddistribution p g x .
Revised: We deﬁne the GAN generator as g : Ω Z × Ω Y (cid:55)→ Ω X , which generates samples from the induced distribution p g x .

Original: Further, for a given y , we denote the induced posterior distribution as p g x | y .
Revised: Further, for a given y , we denote the induced posterior distribution as p g x | y .

Original: Thecritic d : Ω X × Ω Y (cid:55)→ R , is constructed to distinguish between true samples ( x , y ) ∼ p xy and fakesamples ( x g , y ) , where x g ∼ p
Revised: The critic d : Ω X × Ω Y (cid:55)→ R , is constructed to distinguish between true samples ( x , y ) ∼ p xy and fake samples ( x g , y ) , where x g ∼ p

Original: g x | y .
Revised: g x | y .

Original: The loss function of the WGAN is given by
Revised: The loss function of the WGAN is given by

Original: [Equation]
Revised: [Equation]

Original: In the equation above, we begin with the deﬁnition of the loss term (also how it is constructed inpractise), and derive the equality using the deﬁnition of the joint distribution and that of p g x .
Revised: In the equation above, we begin with the deﬁnition of the loss term (also how it is constructed in practise), and derive the equality using the deﬁnition of the joint distribution and that of p g x .

Original: Thisrelation is used to derive the weak equivalence between p x | y and p g x | y below.
Revised: This relation is used to derive the weak equivalence between p x | y and p g x | y below.

Original: The WGAN is trainedby solving the following min-max problem
Revised: The WGAN is trained by solving the following min-max problem

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to minimizing the Wasserstein-1 distance between p x | y and p g x | y under theassumption that d is 1-Lipschitz [13].
Revised: which is equivalent to minimizing the Wasserstein-1 distance between p x | y and p g x | y under the assumption that d is 1-Lipschitz [13].

Original: The Lipschitz constraint on the critic can be weakly imposedusing a gradient penalty term [17] while training d .
Revised: The Lipschitz constraint on the critic can be weakly imposed using a gradient penalty term [17] while training d .

Original: Further, convergence in the Wasserstein-1 metricis equivalent to the weak convergence of the two measures, i.e.,
Revised: Further, convergence in the Wasserstein-1 metric is equivalent to the weak convergence of the two measures, i.e.,

Original: [Equation]
Revised: [Equation]

Original: This implies that for a given measurement y , any statistic of the posterior distribution can be evaluatedby sampling z from p z and pushing it through the trained generator g ∗ ( · , y ) .
Revised: This implies that for a given measurement y , any statistic of the posterior distribution can be evaluated by sampling z from p z and pushing it through the trained generator g ∗ ( · , y ) .

Original: The proposed algorithm is: (a) Generate or collect { x ( i ) }
Revised: The proposed algorithm is: (a) Generate or collect { x ( i ) }

Original: Mi =1 sampled from p x .
Revised: Mi =1 sampled from p x .

Original: (b) For each x ( i )solve the forward model to determine y ( i ) .
Revised: (b) For each x ( i ) solve the forward model to determine y ( i ) .

Original: (c) Train the WGAN using the set of pairwise samples
Revised: (c) Train the WGAN using the set of pairwise samples

Original: S =
Revised: S =

Original: { ( x ( i ) , y ( i ) ) }
Revised: { ( x ( i ) , y ( i ) ) }

Original: Mi =1 .
Revised: Mi =1 .

Original: (d) For a given measurement y , compute the desired statistics of the posteriorby sampling z from p z and pushing it through the trained generator g ∗ ( · , y ) .
Revised: (d) For a given measurement y , compute the desired statistics of the posterior by sampling z from p z and pushing it through the trained generator g ∗ ( · , y ) .

Original:  Numerical results
Revised:  Numerical results

Original: We consider the two-dimensional heat conduction problem.
Revised: We consider the two-dimensional heat conduction problem.

Original: Given a noisy measurement of thetemperature ﬁeld, we infer the thermal conductivity ﬁeld for the steady-state problem, and the initialcondition for the unsteady problem (see Appendix B).
Revised: Given a noisy measurement of the temperature ﬁeld, we infer the thermal conductivity ﬁeld for the steady-state problem, and the initial condition for the unsteady problem (see Appendix B).

Original: The cGANs comprise of generators with a
Revised: The cGANs comprise of generators with a

Original: U-Net architecture.
Revised: U-Net architecture.

Original: The latent variable z is injected at various levels of the U-Net using conditionalinstance normalization, instead of concatenating it with the input y .
Revised: The latent variable z is injected at various levels of the U-Net using conditional instance normalization, instead of concatenating it with the input y .

Original: This approach gives us theﬂexibility of choosing the latent dimension N Z independent of the dimension of y .
Revised: This approach gives us the ﬂexibility of choosing the latent dimension N Z independent of the dimension of y .

Original: Details of thecGAN architectures and the corresponding hyper-parameters can be found in Appendix A.
Revised: Details of the cGAN architectures and the corresponding hyper-parameters can be found in Appendix A.

Original: The meanand standard deviation of the inferred ﬁeld (for a given y ) are computed using 200 samples from thethe latent space.
Revised: The mean and standard deviation of the inferred ﬁeld (for a given y ) are computed using 200 samples from the the latent space.

Original: 3.1 Inferring the thermal conductivity
Revised: 3.1 Inferring the thermal conductivity

Original: We solve the non-linear coefﬁcient inversion problem for steady-state heat conduction, where inferthe conductivity ﬁeld κ given a noisy measurement of the temperature ﬁeld (cid:101) u
Revised: We solve the non-linear coefﬁcient inversion problem for steady-state heat conduction, where we infer the conductivity ﬁeld κ given a noisy measurement of the temperature ﬁeld (cid:101) u

Original: (see Appendix B.1).
Revised: (see Appendix B.1).

Original: The cGAN is trained on a dataset of 8000 samples of discrete conductivity x and measurementy pairs evaluated on a 64 × 64 Cartesian grid.
Revised: The cGAN is trained on a dataset of 8000 samples of discrete conductivity x and measurement y pairs evaluated on a 64 × 64 Cartesian grid.

Original: The conductivity ﬁeld is taken as random circularinclusion with a uniform background value of unity.
Revised: The conductivity ﬁeld is taken as random circular inclusion with a uniform background value of unity.

Original: We test the performance of the trained generatoron three different datasets.
Revised: We test the performance of the trained generator on three different datasets.

Original: The ﬁrst dataset consists of circular inclusions (like the training set),the second and third sets are out-of-distribution (OOD) and consist of elliptical inclusions and twocircular inlcusions, respectively.
Revised: The ﬁrst dataset consists of circular inclusions (like the training set), the second and third sets are out-of-distribution (OOD) and consist of elliptical inclusions and two circular inlcusions, respectively.

Original: In Figure 1(a)-(f), we show the predictions of our trained generatorfor examples from each of these test sets.
Revised: In Figure 1(a)-(f), we show the predictions of our trained generator for examples from each of these test sets.

Original: In general, we notice that the trained network is ableto reliably predict the position as well as orientation of the inclusions in all the three scenarios,including the OOD examples.
Revised: In general, we notice that the trained network is able to reliably predict the position as well as orientation of the inclusions in all the three scenarios, including the OOD examples.

Original: For samples with a single circular inclusion, the mean is very closeto the target.
Revised: For samples with a single circular inclusion, the mean is very close to the target.

Original: The uncertainty in prediction increases near the corners of the square domain, as onewould expect, in the SD plots of Figures 1(b) and (d).
Revised: The uncertainty in prediction increases near the corners of the square domain, as one would expect, in the SD plots of Figures 1(b) and (d).

Original: Interestingly, in Figure 1(d), it appears as if thegenerator is attempting to combine two circles together to approximate the elongated shape of theellipse.
Revised: Interestingly, in Figure 1(d), it appears as if the generator is attempting to combine two circles together to approximate the elongated shape of the ellipse.

Original: In the case of two circles seen in Figures 1(e) and (f), the size of the predominant circles arepredicted accurately, while the smaller circles exhibit higher standard deviation.
Revised: In the case of two circles seen in Figures 1(e) and (f), the size of the predominant circles are predicted accurately, while the smaller circles exhibit higher standard deviation.

Original: This indicates thatthe generator is less certain about the placement of an additional inclusion.
Revised: This indicates that the generator is less certain about the placement of an additional inclusion.

Original: Overall, we observe thatthe generator generalizes reasonably well to the OOD datasets.
Revised: Overall, we observe that the generator generalizes reasonably well to the OOD datasets.

Original: 3.2 Inferring the initial condition
Revised: 3.2 Inferring the initial condition

Original: We consider the time-dependent heat conduction equation without a source term and a constantconductivity ﬁeld (see Appendix B.2).
Revised: We consider the time-dependent heat conduction equation without a source term and a constant conductivity ﬁeld (see Appendix B.2).

Original: Given a noisy temperature ﬁeld at time t = 1 , we wish toinfer the initial condition.
Revised: Given a noisy temperature ﬁeld at time t = 1 , we wish to infer the initial condition.

Original: This is a severely ill-posed problem as signiﬁcant information is lostvia the diffusion process when moving forward in time.
Revised: This is a severely ill-posed problem as signiﬁcant information is lost via the diffusion process when moving forward in time.

Original: We denote the discretized initial and ﬁnaltemperature ﬁeld as x and y , respectively, evaluated on a 28 × 28 Cartesian grid.
Revised: We denote the discretized initial and ﬁnal temperature ﬁeld as x and y , respectively, evaluated on a 28 × 28 Cartesian grid.

Original: The trainingdataset is generated using linearly scaled MNIST [18] handwritten digits as the initial condition,with the training set consisting of 8000 samples.
Revised: The training dataset is generated using linearly scaled MNIST [18] handwritten digits as the initial condition, with the training set consisting of 8000 samples.

Original: In Figure 1(g)-(i), we present the results with thetrained generator for measurements corresponding to test MNIST digits.
Revised: In Figure 1(g)-(i), we present the results with the trained generator for measurements corresponding to test MNIST digits.

Original: As can be observed, themean correctly captures the target initial condition of the samples.
Revised: As can be observed, the mean correctly captures the target initial condition of the samples.

Original: The SD is the highest at theboundaries of the digits, indicating a higher uncertainty in predicting the sharp transition region ofinitial temperature ﬁeld.
Revised: The SD is the highest at the boundaries of the digits, indicating a higher uncertainty in predicting the sharp transition region of initial temperature ﬁeld.

Original: In Figure 1(j)-(l), we show the results when the the target initial condition ischosen from the notMNIST dataset (OOD dataset).
Revised: In Figure 1(j)-(l), we show the results when the the target initial condition is chosen from the notMNIST dataset (OOD dataset).

Original: We observe that the generator struggles to capturehigh-temperature regions closer to the boundary and the broader zones in the interior.
Revised: We observe that the generator struggles to capture high-temperature regions closer to the boundary and the broader zones in the interior.

Original: Note that
Revised: Note that

Original: MNIST digits in the training set are more spatially centered and have narrower features.
Revised: MNIST digits in the training set are more spatially centered and have narrower features.

Original: Nonetheless,the generator is able to visibly capture the underlying notMINST characters, once again indicatingthat it has the capacity to generalize beyond the training distribution.
Revised: Nonetheless, the generator is able to visibly capture the underlying notMINST characters, once again indicating that it has the capacity to generalize beyond the training distribution.

Original: Further, we note that thestandard deviation peaks not only at the characters’ boundaries, but also in the interior regions wherethe generator failed to predict the higher-values of initial condition ﬁeld.
Revised: Further, we note that the standard deviation peaks not only at the characters’ boundaries, but also in the interior regions where the generator failed to predict the higher-values of initial condition ﬁeld.

Original: 3.3 Generalizability
Revised: 3.3 Generalizability

Original: The numerical results presented above indicate that the trained cGANs have the ability to performwell on measurements sampled from a test distribution that is out-of-distribution.
Revised: The numerical results presented above indicate that the trained cGANs have the ability to perform well on measurements sampled from a test distribution that is out-of-distribution.

Original: To understandthis, we ﬁrst examine the locality of the inverse map learned by the generator.
Revised: To understand this, we ﬁrst examine the locality of the inverse map learned by the generator.

Original: We consider the GANgenerator trained in Section 3.2, and compute the gradient of the k -th pixel of the prediction x withrespect to the network input y .
Revised: We consider the GAN generator trained in Section 3.2, and compute the gradient of the k -th pixel of the prediction x with respect to the network input y .

Original: We evaluate the magnitude of gradient and average over 10 distinctsamples of y and 10 realizations of z , i.e., 
Revised: We evaluate the magnitude of gradient and average over 10 distinct samples of y and 10 realizations of z , i.e., grad k =

Original: [Equation]
Revised: [Equation]

Original: The averaged gradients for a few pixels k are shown in Figure 2.
Revised: The averaged gradients for a few pixels k are shown in Figure 2.

Original: Note that the gradient for each pixelis concentrated in the neighbourhood of the corresponding pixel in y .
Revised: Note that the gradient for each pixel is concentrated in the neighbourhood of the corresponding pixel in y .

Original: In other words, the domain ofinﬂuence of k -th pixel of x is a neighbourhood of the k -th pixel of y , but not all of y .
Revised: In other words, the domain of inﬂuence of k -th pixel of x is a neighbourhood of the k -th pixel of y , but not all of y .

Original: The locality ofthe generator is not unexpected, since most of the operations in the U-Net architecture (convolutions,transpose-convolutions, up-/down-sampling, etc) are local in nature.
Revised: The locality of the generator is not unexpected, since most of the operations in the U-Net architecture (convolutions, transpose-convolutions, up-/down-sampling, etc) are local in nature.

Original: Although we do also implement a conditional instance normalization in the intermediate layers, it does not seem to substantially alterthe local inﬂuence y on the prediction.
Revised: Although we do also implement a conditional instance normalization in the intermediate layers, it does not seem to substantially alter the local inﬂuence y on the prediction.

Original: Recognizing that (a) the inverse operator learned by the generator is spatially local, and (b) the GANis trained with data (circular inclusions and MNIST digits) that contains instances of x with sharpbinary contrast, we anticipate that it will generalize well to instances where the distribution of therecovered parameter consists of samples with sharp binary contrast.
Revised: Recognizing that (a) the inverse operator learned by the generator is spatially local, and (b) the GAN is trained with data (circular inclusions and MNIST digits) that contains instances of x with sharp binary contrast, we anticipate that it will generalize well to instances where the distribution of the recovered parameter consists of samples with sharp binary contrast.

Original: This would be independent of theglobal form of the spatial distribution of x .
Revised: This would be independent of the global form of the spatial distribution of x .

Original: This is what is observed in Figure 1, where GAN trainedon instances of single circular inclusions generalizes to multiple circular inclusions and ellipticalinclusion, and a GAN trained on MNIST digits generalizes to notMNIST characters.
Revised: This is what is observed in Figure 1, where GAN trained on instances of single circular inclusions generalizes to multiple circular inclusions and elliptical inclusion, and a GAN trained on MNIST digits generalizes to notMNIST characters.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have proposed training cGANs to solve physics-based Bayesian inference problems.
Revised: In this work, we have proposed training cGANs to solve physics-based Bayesian inference problems.

Original: The efﬁcacy of this strategy has been shown by solving the inverse head conduction model in twodimensions.
Revised: The efﬁcacy of this strategy has been shown by solving the inverse head conduction model in twodimensions.

Original: The generator of the GAN has a special architecture which i) injects latent informationusing conditional instance normalization, giving us the ﬂexibility to choose an arbitrary latentdimension, and ii) uses a U-Net architecture which promotes the learning of a local inverse mapping.
Revised: The generator of the GAN has a special architecture which i) injects latent information using conditional instance normalization, giving us the ﬂexibility to choose an arbitrary latent dimension, and ii) uses a U-Net architecture which promotes the learning of a local inverse mapping.

Original: The local-nature of the learned map promotes the generalizability of the cGAN, which is demonstratedthrough numerical experiments.
Revised: The local-nature of the learned map promotes the generalizability of the cGAN, which is demonstrated through numerical experiments.

Original: Future work will consider additional physics-based problems andexplore a mathematical explanation connecting the network’s architecture and the locality of the“true” inverse map to the expected generalizability.Checklist For all authors...
Revised: Future work will consider additional physics-based problems and explore a mathematical explanation connecting the network’s architecture and the locality of the “true” inverse map to the expected generalizability.

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: Yes]

Original: b

Original: ) Did you describe the limitations of your work?

Original: [Yes] In Sections and 3.1 and 3.2, wemention that the proposed cGAN may fail to capture some features of the inferred ﬁeldif they were not present in the training samples.

Original: In other words, while the networks arecapable of generalizing, there are limits to this generalization

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do not

Original: foresee our work to have potential negative societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] If
Revised: Acknowledgments and Disclosure of Funding

Original: you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: The authors acknowledge support from ARO grant W911NF2010050, NRL grant N00173-19-P-1119, and AIER (USC), and resources from the Center for Advanced Research Computing (USC).

Original: [Yes] We have notincluded the codes as they are still in development.

Original: However, we have provided explicitdetails of the network architecture and the various hyper-parameters used to train them.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] These are provided in Appendix A.

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type

Original: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We made use of the

Original: MNIST and notMNIST datasets, and cited the

Original: sources

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: If you used

Original: crowdsourcing

Original: or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: A WGAN architecture
Revised: A WGAN architecture

Original: The cGANs in this work were trained using the following loss function 
Revised: The cGANs in this work were trained using the following loss function L

Original: [Equation]
Revised: [Equation]

Original: which has been augmented with a gradient penalty term.
Revised: which has been augmented with a gradient penalty term.

Original: Here, U (0 , 1) denotes the uniform distribution on [0 , 1] , ∂ 1 d ( ., . )
Revised: Here, U (0 , 1) denotes the uniform distribution on [0 , 1] , ∂ 1 d ( ., . )

Original: denotes the derivative with resespect to the ﬁrst argument, and
Revised: denotes the derivative with resespect to the ﬁrst argument, and

Original: [Equation]
Revised: [Equation]

Original: For all experiments, we set the gradient penalty parameter as λ = 10 .
Revised: For all experiments, we set the gradient penalty parameter as λ = 10 .

Original: All networks are trained on TensorFlow [19].
Revised: All networks are trained on TensorFlow.

Original: Each network took less than 20 hours to train on asingle NVIDIA Tesla K40 GPU.
Revised: Each network took less than 20 hours to train on a single NVIDIA Tesla K40 GPU.

Original: To describe the architecture of the generator and discriminator, we ﬁrst introduce the followingnotations:
Revised: To describe the architecture of the generator and discriminator, we ﬁrst introduce the following notations:

Original: • norm denotes the type of normalization used.
Revised: • norm denotes the type of normalization used.

Original: This can take the value cin for condition instance normalization using the latent variable or ln for layer normalization
Revised: This can take the value cin for condition instance normalization using the latent variable or ln for layer normalization.

Original: Note that cin implicitly implies that the latent variable is also an input to the particular layer using the normalization.
Revised: Note that cin implicitly implies that the latent variable is also an input to the particular layer using the normalization.

Original: • Conv(k,n) denotes a 2D convolution with k ﬁlters of size n and stride 1.
Revised: • Conv(k,n) denotes a 2D convolution with k ﬁlters of size n and stride 1.

Original: If n > 1 , reﬂectivepadding of width 1 is applied in the spatial dimensions of the input before applying the convolution.
Revised: If n > 1 , reﬂective padding of width 1 is applied in the spatial dimensions of the input before applying the convolution.

Original: • LReLU denotes the leaky ReLU activation with parameter 0.1.
Revised: • LReLU denotes the leaky ReLU activation with parameter 0.1.

Original: • Res(norm) denotes a residual block whose action on an input X with k channels is given 
Revised: • Res(norm) denotes a residual block whose action on an input X with k channels is given by: X

Original: by:
Revised: [Equation]

Original: X [-→ norm → Conv(k, → X
-]
Revised: X [-→ norm → Conv(k, → X
-][Equation] Conv

Original: X → norm → LReLU → Conv

Original: [Equation]
Revised: [Equation]

Original: k,
Revised: k

Original: [Equation]
Revised: [Equation]

Original: → norm → LReLU → Conv(k,3) → 
Revised: → [Equation] → LReLU → Conv(k,3) → X

Original: [-X
-]
Revised: [-X
-]X

Original: X1 + X2 → Block_Output If the norm argument is absent, then no normalization is applied in the residual block.
Revised: + X2 → Block_Output If the norm argument is absent, then no normalization is applied in the residual block.

Original: • Down(k) denotes a down-sampling block which applies Conv(k,3) and LReLU operationsto the input, and then reduces the spatial resolution by a factor of 2. • CC(X_old) denote the the concatenation of X_old to the input along the channel direction.
Revised: • Down(k) denotes a down-sampling block which applies Conv(k,3) and LReLU operations to the input, and then reduces the spatial resolution by a factor of 2. • CC(X_old) denote the the concatenation of X_old to the input along the channel direction.

Original: This helps implement skip connections.
Revised: This helps implement skip connections.

Original: • Up(k,X_old) denotes an up-sampling block which ﬁrst applies CC( X_old) , Conv(k,3)and LReLU operations to the input, and then increases the spatial resolution by a factor of 2 using nearest neighbour interpolation.
Revised: • Up(k,X_old) denotes an up-sampling block which ﬁrst applies CC( X_old) , Conv(k,3) and LReLU operations to the input, and then increases the spatial resolution by a factor of 2 using nearest neighbour interpolation.

Original: If the argument X_old is absent, then no concatenation is applied.
Revised: If the argument X_old is absent, then no concatenation is applied.

Original: • Dense(k) denotes a fully connected layer of width k.
Revised: • Dense(k) denotes a fully connected layer of width k.

Original: Based on the above notations, we now describe the U-Net generator architecture with a sigmoidoutput function, and the corresponding critic.
Revised: Based on the above notations, we now describe the U-Net generator architecture with a sigmoid output function, and the corresponding critic.

Original: We use l 2 regularization of the network weights with10 − 3 as the regularization parameter.

Original: The training is performed using the Adam optimizer withβ 1 = 0 .
Revised: The training is performed using the Adam optimizer with β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 9 and learning-rates 10 − 3 .The
Revised: 9 and learning-rates 10 − 3 .The

Original: remaining hyper-parameters are listed in Table 1.
Revised: remaining hyper-parameters are listed in Table 1.

Original: Architecture of cGAN for inferring conductivity
Revised: Architecture of cGAN for inferring conductivity

Original: Generator architecture:
Revised: Generator architecture:

Original: X → Conv [Equation] 3) → LReLU → Res → X1 → Down(64) → Res(cin)→ X2 → Down( →
Revised: X → Conv(32,3) → LReLU → Res → X1 → Down(64) → Res(cin)

Original: Res(cin)

Original: →  X3 → Down(256) → Res(cin) → X4 → Res(cin) → X5 → Up( →Res→ X6 → Up(
Revised: → X2 → Down(128) → Res(cin) → X3 → Down(256) → Res(cin) → X4 → Res(cin) → X5 → Up(256) → Res → X6 → Up(128,X3)

Original: X3)

Original: → Res (cin)
Revised: → Res(cin)

Original: → X7 →

Original: Up

Original: 64,X → Res(cin) → X8 → CC(X → 
Revised: → X7 → Up(64,X2) → Res(cin)

Original: Conv( 3) → LReLU → Res (cin
Revised: → X8 → CC(X1) → Conv(32,3) → LReLU → Res(cin)

Original: ) → Conv(1,3)
Revised: → Conv(1,3)

Original: → LReLU → Sigmoid → Output
Revised: → LReLU → Sigmoid → Output

Original: Critic architecture:[X;Y] → Down(32) → 
Revised: Critic architecture: [X;Y] → Down(32) → Res → Down(64) → Res(ln) →

Original: Res → Down(64) → Res(ln) →Down(128) → Res(ln)→ Down(256) → Res(ln) → Flatten → Dense(128) → LReLU → ln → Dense(1)[Table] →
Revised: Down(128) → Res(ln) → Down(256) → Res(ln) → Flatten → Dense(128) → LReLU → ln → Dense(1)

Original: [Table]
Revised: [Table]

Original: Architecture of cGAN for inferring initial temperature ﬁeld
Revised: Architecture of cGAN for inferring initial temperature ﬁeld

Original: Generator architecture:
Revised: Generator architecture:

Original: X → Conv(32,3) → LReLU → Res → X1 → Down(64) → Res(cin)
Revised: X → Conv(32,3) → LReLU → Res → X1 → Down(64) → Res(cin)

Original: → X2 → Down(128)→ Res(cin) → X3 → Res(cin) → X4 → Up(128) → Res → X5 → Up(64,X2) → Res(cin)→ X6 → CC(X1) → Conv(32,3) → LReLU → Res(cin) 
Revised: → X2 → Down(128) → Res(cin) → X3 → Res(cin) → X4 → Up(128) → Res → X5 → Up(64,X2) → Res(cin) → X6 → CC(X1) → Conv(32,3) → LReLU → Res(cin) → Conv(1,3) → LReLU → Sigmoid →

Original: → Conv(1,3)
Revised: [Table]

Original: → LReLU → Sigmoid→ Output Critic architecture:[X;Y] → Down(32) → 
Revised: Critic architecture: [X;Y] → Down(32) → Res → Down(64) → Res(ln) →

Original: Res → Down(64) → Res(ln) → Down(128) → Res(ln) → Flatten→ Dense(128) → LReLU → ln → Dense(1) → Output
Revised: Down(128) → Res(ln) → Flatten → Dense(128) → LReLU → ln → Dense(1) → Output

Original: B Heat conduction model and dataset generation
Revised: B Heat conduction model and dataset generation

Original: Consider the two-dimensional time-dependent heat conduction problem on a bounded domain Ω ⊂ R 2with Dirichlet boundary conditions∂u ( s , t ) ∂t − ∇ · ( κ ( s ) ∇ u ( s , t ))
Revised: Consider the two-dimensional time-dependent heat conduction problem on a bounded domain Ω ⊂ Rwith Dirichlet boundary conditions ∂u ( s , t ) ∂t − ∇ · ( κ ( s ) ∇ u ( s , t ))

Original: =
Revised: =

Original: b ( s ) , ∀ ( s , t ) ∈
Revised: b ( s ) , ∀ ( s , t ) ∈

Original: Ω × (0 , T ) (5) u
Revised: Ω × (0 , T ) (5) u

Original: ( s , 0)
Revised: ( s , 0)

Original: = m ( s ) , ∀ s ∈ Ω (6) u ( s , t ) = 0 , ∀ ( s , t ) ∈
Revised: = m ( s ) , ∀ s ∈ Ω (6) u ( s , t ) = 0 , ∀ ( s , t ) ∈

Original: ∂ Ω × (0 , T ) .
Revised: ∂ Ω × (0 , T ) .

Original: Here u denotes the temperature ﬁeld, κ denotes the spatially-varying conductivity ﬁeld, b denotesthe heat source and m denotes the initial temperature ﬁeld.
Revised: Here u denotes the temperature ﬁeld, κ denotes the spatially-varying conductivity ﬁeld, b denotes the heat source and m denotes the initial temperature ﬁeld.

Original: The steady-state problem corresponds todropping the temporal dependence and the initial condition (6).
Revised: The steady-state problem corresponds to dropping the temporal dependence and the initial condition (6).

Original: B.1 Datasets for inferring κ
Revised: B.1 Datasets for inferring κ

Original: We solve the non-linear coefﬁcient inversion problem for the steady-state heat conduction problemposed on Ω = (0 , 1) 2 for a constant heat source b ≡ 10 .
Revised: We solve the non-linear coefﬁcient inversion problem for the steady-state heat conduction problem posed on Ω = (0 , 1) 2 for a constant heat source b ≡ 10 .

Original: We assume that the measured temperatureﬁeld is corrupted by noise, i.e., (cid:101) u
Revised: We assume that the measured temperature ﬁeld is corrupted by noise, i.e., (cid:101) u

Original: = u + η , where η is modelled as an uncorrelated Gaussian noisewith zero mean and a covariance matrix given by σ 2 I .
Revised: = u + η , where η is modelled as an uncorrelated Gaussian noise with zero mean and a covariance matrix given by σ 2 I .

Original: Here σ is taken to be 2.5% of the maximumvalue of u on the entire dataset.
Revised: Here σ is taken to be 2.5% of the maximum value of u on the entire dataset.

Original: To generate the training data for the GAN, the forward steady-state problem is solved using the
Revised: To generate the training data for the GAN, the forward steady-state problem is solved using the

Original: Bubnov-Galerkin method with triangular ﬁrst-order Lagrangian shape functions.
Revised: Bubnov-Galerkin method with triangular ﬁrst-order Lagrangian shape functions.

Original: The conductivityﬁeld is constructed as having a circular inclusion on a constant background of value unity.
Revised: The conductivity ﬁeld is constructed as having a circular inclusion on a constant background of value unity.

Original: Theposition of the circle ( s c 1 , s c 2 ) , the radius r and the conductivity value κ c inside the circle were chosenfrom uniform distributions
Revised: The position of the circle ( s c 1 , s c 2 ) , the radius r and the conductivity value κ c inside the circle were chosen from uniform distributions

Original: [Equation]
Revised: [Equation]

Original: The vectors x and y of each training sample denote the values of the discretized κ and noisytemperature ﬁelds (cid:101) u , respectively, interpolated on a 64 × 64 Cartesian grid.
Revised: The vectors x and y of each training sample denote the values of the discretized κ and noisy temperature ﬁelds (cid:101) u , respectively, interpolated on a 64 × 64 Cartesian grid.

Original: We additionally construct two OOD test datasets; one with an ellipse and the other having two circles.
Revised: We additionally construct two OOD test datasets; one with an ellipse and the other having two circles.

Original: The ellipses are parameterized by their center coordinates ( s e 1 , s e 2 ) , aspect ratio r e , angle of tilt θ , andconductivity κes e 1 ∼ U (0 , 1) , [Equation] ∼ U (0 , 1) ,
Revised: The ellipses are parameterized by their center coordinates ( s e 1 , s e 2 ) , aspect ratio r e , angle of tilt θ , and conductivity κ e s

Original: [Equation]
Revised: [Equation]

Original: ∼ U (0 . 3 ,
Revised: ∼ U (

Original: [Equation]
Revised: [Equation]

Original: , θ

Original: [Equation]
Revised: [Equation]

Original: U (
Revised: U (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  For the double circles, we have their centers ( ( s a 1 , s a 2 ) and ( s b 1 , s b 2 ) ), radii ( r a and r b ), and conductivityvalues ( κ a and κ b ) sampled froms a 1 ∼ U (0 , 1) , s a 2 ∼ U (0 , 1) , r a ∼ U (0 . 05 , 0 . 25) , κ a ∼ U (2 , 5) , s b 1 ∼ U (0 , 1) , s b 2 ∼ U (0 , 1) , r b ∼ U (0 . 05 , 0 . 25) , κ b ∼ U (2 , 5) .
Revised: r e ∼ U (0 . 3 , 1) , θ ∼ U (0 , π ) , κ e ∼ U (2 , 10) . For the double circles, we have their centers ( ( s a 1 , s a 2 ) and ( s b 1 , s b 2 ) ), radii ( r a and r b ), and conductivity values ( κ a and κ b ) sampled from [Equation] [Equation]

Revised: [Equation]

Original: B.2 Datasets for inferring the initial condition
Revised: B.2 Datasets for inferring the initial condition

Original: We consider the time-dependent heat equation on the domain Ω
Revised: We consider the time-dependent heat equation on the domain Ω

Original: = (0 , 2 π ) 2 with b ≡ 0 and κ ≡ 0 . 2 .
Revised: = (0 , 2 π ) 2 with b ≡ 0 and κ ≡ 0 . 2 .

Original: We assume that the measured temperature ﬁeld at t =
Revised: We assume that the measured temperature ﬁeld at t =

Original: T = 1 is corrupted by an uncorrelated Gaussiannoise with zero mean and identity covariance matrix.
Revised: T = 1 is corrupted by an uncorrelated Gaussian noise with zero mean and identity covariance matrix.

Original: Given such a measurement, we wish to inferthe initial temperature ﬁeld m .
Revised: Given such a measurement, we wish to infer the initial temperature ﬁeld m .

Original: To construct training dataset, we consider the MNIST handwritten digits [18] and linearly scalethe image intensity to lie between 0 and 4 units.
Revised: To construct training dataset, we consider the MNIST handwritten digits [18] and linearly scale the image intensity to lie between 0 and 4 units.

Original: This forms the discrete initial temperature ﬁeldrepresented on a 28 × 28 Cartesian grid.
Revised: This forms the discrete initial temperature ﬁeld represented on a 28 × 28 Cartesian grid.

Original: The corresponding ﬁnal temperature ﬁeld on the same gridis obtained by solving (5)-(7) using a central-space-backward-time ﬁnite difference scheme.
Revised: The corresponding ﬁnal temperature ﬁeld on the same grid is obtained by solving (5)-(7) using a central-space-backward-time ﬁnite difference scheme.

Original: Thevectors x and y of each training sample denote the discretized initial and (noisy) ﬁnal temperatureﬁelds, respectively.
Revised: The vectors x and y of each training sample denote the discretized initial and (noisy) ﬁnal temperature ﬁelds, respectively.

Original: We create an additional OOD test dataset, where the initial condition is represented using thenotMNIST 1 dataset with the same linear scaling of the image intensity.
Revised: We create an additional OOD test dataset, where the initial condition is represented using the notMNIST 1 dataset with the same linear scaling of the image intensity.