Original: Analyzing Inverse Problems with Invertible Neural Networks
Revised: Analyzing Inverse Problems with Invertible Neural Networks

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: For many applications, in particular in natural science, the task is to determine hidden system parameters from a set of measurements.
Revised: For many applications, in particular in natural science, the task is to determine hidden system parameters from a set of measurements.

Original: Often, the forward process from parameter- to measurement-space is well-deﬁned, whereas the inverse problem is ambiguous: multiple parameter sets can result in the same measurement.
Revised: Often, the forward process from parameter- to measurement-space is well-deﬁned, whereas the inverse problem is ambiguous: multiple parameter sets can result in the same measurement.

Original: To fully characterize this ambiguity, the full posterior parameter distribution , conditioned on an observed measurement, has to be determined.
Revised: To fully characterize this ambiguity, the full posterior parameter distribution , conditioned on an observed measurement, has to be determined.

Original: We argue that a particular class of neural networks is well suited for this task – so-called Invertible Neural Networks (INNs).
Revised: We argue that a particular class of neural networks is well suited for this task – so-called Invertible Neural Networks (INNs).

Original: Unlike classical neural networks, which attempt to solve the ambiguous inverse problem directly, INNs focus on learning the forward process, using additional latent output variables to capture the information otherwise lost.
Revised: Unlike classical neural networks, which attempt to solve the ambiguous inverse problem directly, INNs focus on learning the forward process, using additional latent output variables to capture the information otherwise lost.

Original: Due to invertibility, a model of the corresponding inverse process is learned implicitly.
Revised: Due to invertibility, a model of the corresponding inverse process is learned implicitly.

Original: Given a speciﬁc measurement and the distribution of the latent variables, the inverse pass of the INN provides the full posterior over parameter space.
Revised: Given a speciﬁc measurement and the distribution of the latent variables, the inverse pass of the INN provides the full posterior over parameter space.

Original: We prove theoretically and verify experimentally, on artiﬁcial data and real-world problems from medicine and astrophysics, that
Revised: We prove theoretically and verify experimentally, on artiﬁcial data and real-world problems from medicine and astrophysics, that

Original: INNs are a powerful analysis tool to ﬁnd multi-modalities in parameter space, uncover parameter correlations, and identify unrecoverable parameters.
Revised: INNs are a powerful analysis tool to ﬁnd multi-modalities in parameter space, uncover parameter correlations, and identify unrecoverable parameters.

Original:  Introduction
Revised:  Introduction

Original: When analyzing complex physical systems, a common problem is that the system parameters of interest cannot be measured directly.
Revised: When analyzing complex physical systems, a common problem is that the system parameters of interest cannot be measured directly.

Original: For many of these systems, scientists have developed sophisticated theories on how measurable quantities y arise from the hidden parameters x .
Revised: For many of these systems, scientists have developed sophisticated theories on how measurable quantities y arise from the hidden parameters x .

Original: We will call such mappings the forward process .
Revised: We will call such mappings the forward process .

Original: However, the inverse process is required to infer the hidden states of a system from measurements.
Revised: However, the inverse process is required to infer the hidden states of a system from measurements.

Original: Unfortunately, the inverse is often both intractable and ill-posed, since crucial information is lost in the forward process.
Revised: Unfortunately, the inverse is often both intractable and ill-posed, since crucial information is lost in the forward process.

Original: To fully assess the diversity of possible inverse solutions for a given measurement, an inverse solver should be able to estimate the complete posterior of the parameters, conditioned on an observation.
Revised: To fully assess the diversity of possible inverse solutions for a given measurement, an inverse solver should be able to estimate the complete posterior of the parameters, conditioned on an observation.

Original: This makes it possible to quantify uncertainty, reveal multi-modal distributions, and identify degenerate and unrecoverable parameters – all highly relevant for applications in natural science.
Revised: This makes it possible to quantify uncertainty, reveal multi-modal distributions, and identify degenerate and unrecoverable parameters – all highly relevant for applications in natural science.

Original: In this paper, we ask if invertible neural networks (INNs) are a suitable model class for this task.
Revised: In this paper, we ask if invertible neural networks (INNs) are a suitable model class for this task.

Original: INNs are characterized by three properties: (i) The mapping from inputs to outputs is bijective, i.e. its inverse exists, (ii) both forward and inverse mapping are eﬃciently computable, and (iii) both mappings have a tractable Jacobian, which allows explicit computation of posterior probabilities.
Revised: INNs are characterized by three properties: (i) The mapping from inputs to outputs is bijective, i.e. its inverse exists, (ii) both forward and inverse mapping are eﬃciently computable, and (iii) both mappings have a tractable Jacobian, which allows explicit computation of posterior probabilities.

Original: Networks that are invertible by construction oﬀer a unique opportunity: We can train them on the well-understood forward process x → y and get the inverse y
Revised: Networks that are invertible by construction oﬀer a unique opportunity: We can train them on the well-understood forward process x → y and get the inverse y

Original: → x for free by running them backwards at prediction time.
Revised: → x for free by running them backwards at prediction time.

Original: To counteract the inherent information loss of the forward process, we introduce additional latent output variables z , which capture the information about x that is not contained in y .
Revised: To counteract the inherent information loss of the forward process, we introduce additional latent output variables z , which capture the information about x that is not contained in y .

Original: Thus, our INN learns to associate hidden parameter values x with unique pairs [ y , z ] of measurements and latent variables. 
Revised: Thus, our INN learns to associate hidden parameter values x with unique pairs [ y , z ] of measurements and latent variables. [Figure] x

Original: Forward training learns the mapping [ y , z ] 
Revised: INN y z forward (simulation): x → y inverse (sampling): [ y , z ] → x U [Figure] S

Revised: [Figure]

Revised: U S L

Revised: [Figure]

Revised: NN y

Revised: [Figure]

Revised: (simulation)

Revised: [Figure]

Revised: x

Revised: [Figure]

Revised: y

Revised: [Figure]

Revised: (prediction)

Revised: [Figure]

Revised: y

Revised: [Figure]

Revised: Forward training learns the mapping [ y , z ]

Original: = f
Revised: = f

Original: ( x ) and implicitly determines its inverse x = f − 1 ( y , z )
Revised: ( x ) and implicitly determines its inverse x = f − 1 ( y , z )

Original: =
Revised: =

Original: g ( y , z ) .
Revised: g ( y , z ) .

Original: Additionally, we make sure that the density p ( z ) of the latent variables is shaped as a Gaussian distribution.
Revised: Additionally, we make sure that the density p ( z ) of the latent variables is shaped as a Gaussian distribution.

Original: Thus, the INN represents the desired [Figure]
Revised: Thus, the INN represents the desired posterior by deterministic function

Original: x INN y z forward (simulation): x → y inverse (sampling): [ y , z ] → x U S L
Revised: = that transforms

Original: S L U S L x (Bayesian) NN y forward (simulation): x → y inverse (prediction): y → x [Figure] posterior p ( x | y ) by a deterministic function
Revised: “pushes” the known distribution

Original: = g (
Revised: ) to on

Original: , z ) that transforms (

Original: ) the known distribution p ( z ) to x -space, conditional on y .

Original: Compared to standard approaches (see Fig. 1, left ), INNs circumvent a fundamental diﬃculty of learning inverse problems:
Revised: Compared to standard approaches (see Fig. 1, left ), INNs circumvent a fundamental diﬃculty of learning inverse problems:

Original: Deﬁning a sensible supervised loss for direct posterior learning is problematic since it requires prior knowledge about that posterior’s behavior, constituting a kind of hen-end-egg problem.
Revised: Deﬁning a sensible supervised loss for direct posterior learning is problematic since it requires prior knowledge about that posterior’s behavior, constituting a kind of hen-end-egg problem.

Original: If the loss does not match the possibly complicated (e.g. multi-modal) shape of the posterior, learning will converge to incorrect or misleading solutions.
Revised: If the loss does not match the possibly complicated (e.g. multi-modal) shape of the posterior, learning will converge to incorrect or misleading solutions.

Original: Since the forward process is usually much simpler and better understood, forward training diminishes this diﬃculty.
Revised: Since the forward process is usually much simpler and better understood, forward training diminishes this diﬃculty.

Original: Speciﬁcally, we make the following contributions:
Revised: Speciﬁcally, we make the following contributions:

Original: • We show that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically in the asymptotic limit of zero loss, and practically on synthetic and real-world data from astrophysics and medicine.
Revised: • We show that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically in the asymptotic limit of zero loss, and practically on synthetic and real-world data from astrophysics and medicine.

Original: • The architectural restrictions imposed by invertibility do not seem to have detrimental eﬀects on our network’s representational power.
Revised: • The architectural restrictions imposed by invertibility do not seem to have detrimental eﬀects on our network’s representational power.

Original: • While forward training is suﬃcient in the asymptotic limit, we ﬁnd that a combination with unsupervised backward training improves results on ﬁnite training sets.
Revised: • While forward training is suﬃcient in the asymptotic limit, we ﬁnd that a combination with unsupervised backward training improves results on ﬁnite training sets.

Original: • In our applications, our approach to learning the posterior compares favourably to approximate Bayesian computation (ABC) and conditional VAEs.
Revised: • In our applications, our approach to learning the posterior compares favourably to approximate Bayesian computation (ABC) and conditional VAEs.

Original: This enables identifying unrecoverable parameters, parameter correlations and multimodalities.
Revised: This enables identifying unrecoverable parameters, parameter correlations and multimodalities.

Original:  Related work
Revised:  Related work

Original: Modeling the conditional posterior of an inverse process is a classical statistical task that can in principle be solved by Bayesian methods.
Revised: Modeling the conditional posterior of an inverse process is a classical statistical task that can in principle be solved by Bayesian methods.

Original: Unfortunately, exact Bayesian treatment of real-world problems is usually intractable.
Revised: Unfortunately, exact Bayesian treatment of real-world problems is usually intractable.

Original: The most common (but expensive) solution is to resort to sampling, typically by a variant of Markov Chain Monte Carlo (Robert and
Revised: The most common (but expensive) solution is to resort to sampling, typically by a variant of Markov Chain Monte Carlo (Robert and

Original: Casella (2004); Gamerman and Lopes (2006)).
Revised: Casella, 2004; Gamerman and Lopes, 2006).

Original: If a model y = s ( x ) for the forward process is available, approximate Bayesian computation (ABC) is often preferred, which embeds the forward model in a rejection sampling scheme for the posterior p ( x | y )
Revised: If a model y = s ( x ) for the forward process is available, approximate Bayesian computation (ABC) is often preferred, which embeds the forward model in a rejection sampling scheme for the posterior p ( x | y )

Original: (Sunnåker et al.
Revised: (Sunnåker et al., 2013;

Original: (2013); Lintusaari et al. Wilkinson (2013)).
Revised: Lintusaari et al., 2017; Wilkinson, 2013).

Original: Variational methods oﬀer a more eﬃcient alternative, approximating the posterior by an optimally chosen member of a tractable distribution family (Blei et al.
Revised: Variational methods oﬀer a more eﬃcient alternative, approximating the posterior by an optimally chosen member of a tractable distribution family (Blei et al., 2017).

Original: Neural networks can be trained to predict accurate suﬃcient statistics for parametric posteriors (Papamakarios and Murray (2016); Siddharth et al. (2017)), or can be designed to learn a mean-ﬁeld distribution for the network’s weights via dropout variational inference (Gal andGhahramani (2015);
Revised: Neural networks can be trained to predict accurate suﬃcient statistics for parametric posteriors (Papamakarios and Murray, 2016; Siddharth et al., 2017), or can be designed to learn a mean-ﬁeld distribution for the network’s weights via dropout variational inference (Gal and

Original:  Kingma et al.
Revised: Ghahramani, 2015; Kingma et al., 2015).

Original: Both ideas can be combined (Kendall and Gal (2017)) to diﬀerentiate between data-related and model-related uncertainty.
Revised: Both ideas can be combined (Kendall and Gal, 2017) to diﬀerentiate between data-related and model-related uncertainty.

Original: However, the restriction to limited distribution families fails if the true distribution is too complex (e.g. when it requires multiple modes to represent ambiguous or degenerate solutions) and essentially counters the ability of neural networks to act as universal approximators.
Revised: However, the restriction to limited distribution families fails if the true distribution is too complex (e.g. when it requires multiple modes to represent ambiguous or degenerate solutions) and essentially counters the ability of neural networks to act as universal approximators.

Original: Conditional GANs (cGAN; Mirza and Osindero (2014); Isola et al. (2017)) overcome this restriction in principle, but often lack satisfactory diversity in practice (Zhu et al. (2017b)).
Revised: Conditional GANs (cGANs; Mirza and Osindero, 2014; Isola et al., 2017) overcome this restriction in principle, but often lack satisfactory diversity in practice (Zhu et al., 2017b).

Original: For our tasks, conditional variational autoencoders (cVAEs; Sohn et al. (2015)) perform better than cGANs, and are also conceptually closer to our approach (see appendix Sec. 2), and hence serve as a baseline in our experiments.
Revised: For our tasks, conditional variational autoencoders (cVAEs; Sohn et al., 2015) perform better than cGANs, and are also conceptually closer to our approach (see appendix Sec. 2), and hence serve as a baseline in our experiments.

Original: Normalizing ﬂows , introduced to the ﬁeld of neural networks by Rippel and Adams (2013); Rezende and Mohamed (2015), have the potential to solve these problems.
Revised: Generative modeling via learning of a non-linear transformation between the data distribution and a simple prior distribution (Deco and Brauer, 1995; Hyvärinen and Pajunen, 1999) has the potential to solve these problems.

Original: They gradually transform normal densities into target densities and rely on bijectivity to guarantee the validity of these mappings. Thus, they possess property (i) of an INN, and are usually designed to fulﬁll requirement (iii) as well.
Revised: Today, this approach is often formulated as a normalizing ﬂow (Tabak et al., 2010; Tabak and Turner, 2013), which gradually transforms a normal density into the desired data density and relies on bijectivity to ensure the mapping’s validity. These ideas were applied to neural networks by Deco and Brauer (1995); Rippel and Adams (2013);

Original: Besides variants of the original architectures (Tomczak and Welling (2016); Berg et al.
Revised: Rezende and Mohamed (2015) and reﬁned by Tomczak and Welling (2016); Berg et al.

Original: ; Trippe and Turner (2018)), the most common realizations use auto-regressive ﬂows , where the density is decomposed according to the Bayesian chain rule (Kingma et al. ; Huang et al.
Revised: ; Trippe and Turner (2018), Today, the most common realizations use auto-regressive ﬂows , where the density is decomposed according to the Bayesian chain rule (Kingma et al., 2016; Huang et al., 2018;

Original: ; Germain et al.
Revised: Germain et al., 2015;

Original: ; Papamakarios et al.
Revised: Papamakarios et al., 2017;

Original: Oord et al. ; Kolesnikov and Lampert (2017); Salimans et al. ; Uria et al.
Revised: Oord et al., 2016; Kolesnikov and Lampert, 2017; Salimans et al., 2017; Uria et al., 2016).

Original: These networks successfully learned unconditional generative distributions for artiﬁcial data and standard image sets (e.g. MNIST, CelebA, LSUN bedrooms), and some encouraging results for conditional modeling exist as well (Oord et al. ; Salimans et al.
Revised: These networks successfully learned unconditional generative distributions for artiﬁcial data and standard image sets (e.g. MNIST, CelebA, LSUN bedrooms), and some encouraging results for conditional modeling exist as well (Oord et al., 2016; Salimans et al., 2017;

Original: ; Papamakarios et al. ; Uria et al.
Revised: Papamakarios et al., 2017; Uria et al., 2016).

Revised: These normalizing ﬂows possess property (i) of an INN, and are usually designed to fulﬁll requirement (iii) as well.

Original: Although ﬂow-based networks are invertible in principle,  actual computation of their inverse is too costly to be practical, i.e.  property (ii) is not fulﬁlled.
Revised: In other words, ﬂow-based networks are invertible in principle, but the actual computation of their inverse is too costly to be practical, i.e. INN property (ii) is not fulﬁlled.

Original: This precludes the possibility of bi-directional or cyclic training, which has been shown to be very beneﬁcial in generative adversarial nets and auto-encoders (Zhu et al. (2017a) ; Dumoulin et al. ; Donahue et al.
Revised: This precludes the possibility of bi-directional or cyclic training, which has been shown to be very beneﬁcial in generative adversarial nets and auto-encoders (Zhu et al., 2017a; Dumoulin et al., 2016; Donahue et al., 2017; Teng et al., 2018).

Original: ; Teng et al.

Original: In fact, optimization for cycle consistency forces such models to converge to invertible architectures, making fully invertible networks a natural choice.
Revised: In fact, optimization for cycle consistency forces such models to converge to invertible architectures, making fully invertible networks a natural choice.

Original: True INNs can be built using coupling layers , as introduced in the NICE (Dinh et al. and
Revised: True INNs can be built using coupling layers , as introduced in the NICE (Dinh et al., 2014) and

Original: RealNVP (Dinh et al. architectures.
Revised: RealNVP (Dinh et al., 2016) architectures.

Original: Despite their simple design and training, these networks were rarely studied:Danihelka et al.
Revised: Despite their simple design and training, these networks were rarely studied:

Original: ; Grover et al.

Original: (2017) showed that minimization of an adversarial loss is superior to maximum likelihood training in RealNVPs,

Original: Gomez et al.
Revised: Gomez et al.

Original: (2017) used a NICE-like design as a memory-eﬃcient alternative to residual networks, Jacobsen et al.
Revised: (2017) used a NICE-like design as a memory-eﬃcient alternative to residual networks, Jacobsen et al.

Revised: (2018) demonstrated that the lack of information reduction from input to representation does not cause overﬁtting, and Schirrmeister et al.

Revised: (2018) trained such a network as an adverserial autoencoder.

Original: (2018) demonstrated that the lack of information reduction from input to representation does not cause overﬁtting, and the  Glow architecture by Kingma and Dhariwal (2018) incorporates invertible 1x1 convolutions into RealNVPs to achieve impressive image manipulations.
Revised: Danihelka et al. (2017) showed that minimization of an adversarial loss is superior to maximum likelihood training in RealNVPs, whereas the Flow-GAN of Grover et al. (2017) performs even better using bidirectional training, a combination of maximum likelihood and adverserial loss. The Glow architecture by Kingma and Dhariwal (2018) incorporates invertible 1x1 convolutions into RealNVPs to achieve impressive image manipulations.

Original: This line of research inspired us to extend RealNVPs for the task of computing posteriors in real-world inverse problems from natural and life sciences.
Revised: This line of research inspired us to extend RealNVPs for the task of computing posteriors in real-world inverse problems from natural and life sciences.

Original:  Methods 3. Problem specification
Revised:  Methods 3. Problem specification

Original: We consider a common scenario in natural and life sciences: Researchers are interested in a set of variables x ∈ R D describing some phenomenon of interest, but only variables y ∈ R M can actually be observed, for which the theory of the respective research ﬁeld provides a model y
Revised: We consider a common scenario in natural and life sciences: Researchers are interested in a set of variables x ∈ R D describing some phenomenon of interest, but only variables y ∈ R M can actually be observed, for which the theory of the respective research ﬁeld provides a model y

Original: = s ( x ) for the forward process.
Revised: = s ( x ) for the forward process.

Original: Since the transformation from x to y incurs an information loss, the intrinsic dimension m of y is in general smaller than D , even if the nominal dimensions satisfy M > D .
Revised: Since the transformation from x to y incurs an information loss, the intrinsic dimension m of y is in general smaller than D , even if the nominal dimensions satisfy M > D .

Original: Hence we want to express the inverse model as a conditional probability p ( x | y ) , but its mathematical derivation from the forward model is intractable in the applications we are going to address.
Revised: Hence we want to express the inverse model as a conditional probability p ( x | y ) , but its mathematical derivation from the forward model is intractable in the applications we are going to address.

Original: We aim at approximating p ( x | y ) by a tractable model q ( x | y ) , taking advantage of the possibility to create an arbitrary amount of training data { ( x i , y i ) } Ni =1 from the known forward model s ( x ) and a suitable prior p ( x ) .
Revised: We aim at approximating p ( x | y ) by a tractable model q ( x | y ) , taking advantage of the possibility to create an arbitrary amount of training data { ( x i , y i ) } Ni =1 from the known forward model s ( x ) and a suitable prior p ( x ) .

Original: While this would allow for training of a standard regression model, we want to approximate the full posterior probability.
Revised: While this would allow for training of a standard regression model, we want to approximate the full posterior probability.

Original: To this end, we introduce a latent random variable z ∈ R K drawn from a multi-variate standard normal distribution and reparametrize q ( x | y ) in terms of a deterministic function g of y and z , represented by a neural network with parameters θ :[Equation] with
Revised: To this end, we introduce a latent random variable z ∈ R K drawn from a multi-variate standard normal distribution and reparametrize q ( x | y ) in terms of a deterministic function g of y and z , represented by a neural network with parameters θ :

Original: [Equation]
Revised: [Equation]

Original: Note that we distinguish between hidden parameters x representing unobservable real-world properties and latent variables z carrying information intrinsic to our model. 
Revised: Note that we distinguish between hidden parameters x representing unobservable real-world properties and latent variables z carrying information intrinsic to our model. Choosing a

Revised: Gaussian prior for z poses no additional limitation, as proven by the theory of non-linear independent component analysis (Hyvärinen and Pajunen, 1999).

Original: In contrast to standard methodology, we propose to learn the model g
Revised: In contrast to standard methodology, we propose to learn the model g

Original: ( y , z ; θ ) of the inverse process jointly with a model f ( x ; θ ) approximating the known forward process s ( x ) :[ y , z [Equation] f ( x ;
Revised: ( y , z ; θ ) of the inverse process jointly with a model f ( x ; θ ) approximating the known forward process s ( x ) :

Original: [Equation]
Revised: [Equation]

Original: withf y (
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Functions f and g share the same parameters θ and are implemented by a single invertible neural network .
Revised: Functions f and g share the same parameters θ and are implemented by a single invertible neural network .

Original: Our experiments show that joint bi-directional training of f and g avoids many complications arising in e.g. cVAEs or Bayesian neural networks, which have to learn the forward process implicitly.
Revised: Our experiments show that joint bi-directional training of f and g avoids many complications arising in e.g. cVAEs or Bayesian neural networks, which have to learn the forward process implicitly.

Original: The relation f = g − 1 is enforced by the invertible network architecture, provided that the nominal and intrinsic dimensions of both sides match.
Revised: The relation f = g − 1 is enforced by the invertible network architecture, provided that the nominal and intrinsic dimensions of both sides match.

Original: When m ≤ M denotes the intrinsic dimension of y , the latent variable z must have dimension K
Revised: When m ≤ M denotes the intrinsic dimension of y , the latent variable z must have dimension K

Original: = D − m , assuming that the intrinsic dimension of x equals its nominal dimension D .
Revised: = D − m , assuming that the intrinsic dimension of x equals its nominal dimension D .

Original: If the resulting nominal output dimension M + K exceeds D , we augment the input with a vector x 0 ∈ R M + K − D of zeros and replace x with the concatenation [ x , x 0 ] everywhere.
Revised: If the resulting nominal output dimension M + K exceeds D , we augment the input with a vector x 0 ∈ R M + K − D of zeros and replace x with the concatenation [ x , x 0 ] everywhere.

Original: Combining these deﬁnitions, our network expresses q ( x | y ) as
Revised: Combining these deﬁnitions, our network expresses q ( x | y ) as

Original: [Equation]
Revised: [Equation]

Original: with Jacobian determinant J x .
Revised: with Jacobian determinant J x .

Original: When using coupling layers, according to Dinh et al.
Revised: When using coupling layers, according to Dinh et al.

Original: (2016), computation of J x is simple, as each transformation has a triangular Jacobian matrix.
Revised: (2016), computation of J x is simple, as each transformation has a triangular Jacobian matrix.

Original: 3. Invertible architecture
Revised: 3. Invertible architecture

Original: To create a fully invertible neural network, we follow the architecture proposed by Dinh et al.
Revised: To create a fully invertible neural network, we follow the architecture proposed by Dinh et al.

Original: The basic unit of this network is a reversible block consisting of two complementary aﬃne coupling layers.
Revised: The basic unit of this network is a reversible block consisting of two complementary aﬃne coupling layers.

Original: Hereby, the block’s input vector u is split into two halves, u 1 and u 2 , which are transformed by an aﬃne function with coeﬃcients exp( s i ) and t i ( i ∈ { 1 , 2 } ), using element-wise multiplication ( [Equation] ) and addition:
Revised: Hereby, the block’s input vector u is split into two halves, u 1 and u 2 , which are transformed by an aﬃne function with coeﬃcients exp( s i ) and t i ( i ∈ { 1 , 2 } ), using element-wise multiplication ( (cid:12) ) and addition:

Original: [Equation]
Revised: [Equation]

Original: Given the output v = [ v 1 , v 2 ] , these expressions are trivially invertible: 
Revised: Given the output v = [ v 1 , v 2 ] , these expressions are trivially invertible: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Importantly, the mappings s i and t i can be arbitrarily complicated functions of v 1 and u 2 and need not themselves be invertible.
Revised: Importantly, the mappings s i and t i can be arbitrarily complicated functions of v 1 and u 2 and need not themselves be invertible.

Original: In our implementation, they are realized by a succession of several fully connected layers with leaky ReLU activations.
Revised: In our implementation, they are realized by a succession of several fully connected layers with leaky ReLU activations.

Original: A deep invertible network is composed of a sequence of these reversible blocks.
Revised: A deep invertible network is composed of a sequence of these reversible blocks.

Original: To increase model capacity, we apply a few simple extensions to this basic architecture.
Revised: To increase model capacity, we apply a few simple extensions to this basic architecture.

Original: Firstly, if the dimension D isrelatively small, but a complex transformation has to be learned, we ﬁnd it advantageous to pad both the in- and output of the network with an equal number of zeros.
Revised: Firstly, if the dimension D is small, but a complex transformation has to be learned, we ﬁnd it advantageous to pad both the in- and output of the network with an equal number of zeros.

Original: This does not change the intrinsic dimensions of in- and output, but enables the network’s interior layers to embed the data into a larger representation space in a more ﬂexible manner.
Revised: This does not change the intrinsic dimensions of in- and output, but enables the network’s interior layers to embed the data into a larger representation space in a more ﬂexible manner.

Original: Secondly, we insert permutation layers between reversible blocks, which shuﬄe the elements of the subsequent layer’s input in a randomized, but ﬁxed, way.
Revised: Secondly, we insert permutation layers between reversible blocks, which shuﬄe the elements of the subsequent layer’s input in a randomized, but ﬁxed, way.

Original: This causes the splits u
Revised: This causes the splits u

Original: = [ u 1 , u 2 ] to vary between layers and enhances interaction among the individual variables.
Revised: = [ u 1 , u 2 ] to vary between layers and enhances interaction among the individual variables.

Original: Kingma and Dhariwal (2018) use a similar architecture with learned permutations.
Revised: Kingma and Dhariwal (2018) use a similar architecture with learned permutations.

Original: 3. Bi-directional training
Revised: 3. Bi-directional training

Original: Invertible networks oﬀer the opportunity to simultaneously optimize for losses on both the inand output domains to make training more eﬀective.
Revised: Invertible networks oﬀer the opportunity to simultaneously optimize for losses on both the inand output domains (Grover et al., 2017), which allows for more eﬀective training.

Original: Hereby, we perform forward and backward iterations in an alternating fashion, accumulating gradients from both directions before performing a parameter update.
Revised: Hereby, we perform forward and backward iterations in an alternating fashion, accumulating gradients from both directions before performing a parameter update.

Original: For the forward iteration, we penalize deviations between simulation outcomes y i
Revised: For the forward iteration, we penalize deviations between simulation outcomes y i

Original: =
Revised: =

Original: s ( x i ) and network predictions
Revised: s ( x i ) and network predictions

Original: f y ( x i ) with a loss L y (cid:0)
Revised: f y ( x i ) with a loss L y (cid:0)

Original: y i , f y ( x i ) (cid:1) .
Revised: y i , f y ( x i ) (cid:1) .

Original: Depending on the problem, L y can be any supervised loss, e.g. squared loss for regression or cross-entropy for classiﬁcation.
Revised: Depending on the problem, L y can be any supervised loss, e.g. squared loss for regression or cross-entropy for classiﬁcation.

Original: The loss for latent variables penalizes the mismatch between the joint distribution of network outputs q (cid:0) y
Revised: The loss for latent variables penalizes the mismatch between the joint distribution of network outputs q (cid:0) y

Original: =
Revised: =

Original: f y ( x ) , z
Revised: f y ( x ) , z

Original: = f z ( x ) (cid:1) =
Revised: = f z ( x ) (cid:1) =

Original: p ( x ) / | J yz | and the product of marginal distributions of simulation outcomes p (cid:0) y
Revised: p ( x ) / | J yz | and the product of marginal distributions of simulation outcomes p (cid:0) y

Original: = s ( x ) (cid:1)
Revised: = s ( x ) (cid:1)

Original: =
Revised: =

Original: p ( x ) /
Revised: p ( x ) /

Original: | J s | and latents p ( z ) as L z (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) .
Revised: | J s | and latents p ( z ) as L z (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) .

Original: We block the gradients of L z with respect to y to make sure that the resulting updates only aﬀect the predictions of z and do not worsen the predictions of y .
Revised: We block the gradients of L z with respect to y to ensure the resulting updates only aﬀect the predictions of z and do not worsen the predictions of y .

Original: Thus, L z enforces two things: ﬁrstly, the generated z must follow the desired normal distribution p ( z ) ; secondly, y and z must be independent, and not encode the same information twice.
Revised: Thus, L z enforces two things: ﬁrstly, the generated z must follow the desired normal distribution p ( z ) ; secondly, y and z must be independent upon convergence (i.e. p ( z | y ) = p ( z ) ), and not encode the same information twice.

Original: Since L z is implemented as Maximum Mean Discrepancy D (see Sec. 3.4), which only requires samples from the distributions to be compared, the Jacobian determinants J yz and J s do not have to be known explicitly.
Revised: As L z is implemented by Maximum Mean Discrepancy D (Sec. 3.4), which only requires samples from the distributions to be compared, the Jacobian determinants J yz and J s do not have to be known explicitly.

Original: In  Sec. 1 of the appendix, we prove the following theorem:
Revised: In appendix Sec. 1, we prove the following theorem:

Original: Theorem:
Revised: Theorem:

Original: If an INN f ( x ) =
Revised: If an INN f ( x ) =

Original: [ y , z ] is trained as proposed, and both the supervised loss
Revised: [ y , z ] is trained as proposed, and both the supervised loss

Original: L y =
Revised: L y =

Original: E
Revised: E

Original: [( y − f y ( x )) 2 ] and the unsupervised loss L z = D (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) reach zero, sampling according to Eq.
Revised: [( y − f y ( x )) 2 ] and the unsupervised loss L z = D (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) reach zero, sampling according to Eq.

Original: 1 with g = f − 1 returns the true posterior p ( x |ˆ y  ) for any measurementˆ y  .
Revised: 1 with g = f − 1 returns the true posterior p ( x | y ∗ ) for any measurement y ∗ .

Original: Although L y and L z are suﬃcient asymptotically, a small amount of residual dependency between y and z remains after a ﬁnite amount of training.
Revised: Although L y and L z are suﬃcient asymptotically, a small amount of residual dependency between y and z remains after a ﬁnite amount of training.

Original: This causes q ( x | y ) to deviate from the true posterior p ( x | y ) .
Revised: This causes q ( x | y ) to deviate from the true posterior p ( x | y ) .

Original: To speed up convergence, we also deﬁne a loss L x on the input side, implemented again by MMD.
Revised: To speed up convergence, we also deﬁne a loss L x on the input side, implemented again by MMD.

Original: It matches the distribution of backward predictions q ( x ) =
Revised: It matches the distribution of backward predictions q ( x ) =

Original: p (cid:0) y =
Revised: p (cid:0) y =

Original: f y ( x ) (cid:1) p (cid:0) z
Revised: f y ( x ) (cid:1) p (cid:0) z

Original: = f z ( x ) (cid:1) /
Revised: = f z ( x ) (cid:1) /

Original: |
Revised: |

Original: J x | against the prior data distribution p ( x ) through
Revised: J x | against the prior data distribution p ( x ) through

Original: L x (cid:0) p ( x ) , q ( x ) (cid:1) .
Revised: L x (cid:0) p ( x ) , q ( x ) (cid:1) .

Original: In the appendix, Sec. 1, we prove that L x is guaranteed to be zero when the forward losses L y and L z have converged to zero.
Revised: In the appendix, Sec. 1, we prove that L x is guaranteed to be zero when the forward losses L y and L z have converged to zero.

Original: Thus, incorporating L x does not alter the optimum, but improves convergence in practice.
Revised: Thus, incorporating L x does not alter the optimum, but improves convergence in practice.

Revised: Finally, if we use padding on either network side, loss terms are needed to ensure no information is encoded in the additional dimensions.

Revised: We a) use a squared loss to keep those values close to zero and b) in an additional inverse training pass, overwrite the padding dimensions with noise of the same amplitude and minimize a reconstruction loss, which forces these dimensions to be ignored.

Original: 3. Maximum mean discrepancy
Revised: 3. Maximum mean discrepancy

Original: Maximum Mean Discrepancy (MMD) is a kernel-based method for comparison of two probability distributions that are only accessible through samples (Gretton et al.
Revised: Maximum Mean Discrepancy (MMD) is a kernel-based method for comparison of two probability distributions that are only accessible through samples (Gretton et al., 2012).

Original: While a trainable discriminator loss is often preferred for this task in high-dimensional problems, especially in GAN-based image generation, MMD also works well, is easier to use and much cheaper, and leads to more stable training (Tolstikhin et al.
Revised: While a trainable discriminator loss is often preferred for this task in high-dimensional problems, especially in GAN-based image generation, MMD also works well, is easier to use and much cheaper, and leads to more stable training (Tolstikhin et al., 2017).

Original: The method requires a kernel function as a design parameter, and we found that kernels with heavier tails than Gaussian are needed to get meaningful gradients for outliers.
Revised: The method requires a kernel function as a design parameter, and we found that kernels with heavier tails than Gaussian are needed to get meaningful gradients for outliers.

Original: We achieved best results with the Inverse Multiquadratic k ( x , x (cid:48) )
Revised: We achieved best results with the Inverse Multiquadratic k ( x , x (cid:48) )

Original: = α/ ( α + (cid:107)  x − x (cid:48)  (cid:107) 22 ) , reconﬁrming the suggestion from Tolstikhin et al.(2017).
Revised: = 1 / (1 + (cid:107) ( x − x (cid:48) ) /h (cid:107) 22 ) , reconﬁrming the suggestion from Tolstikhin et al.

Revised: Since the magnitude of the MMD depends on the kernel choice, the relative weights of the losses L x , L y , L z are adjusted as hyperparameters, such that their eﬀect is about equal.

Original:  Experiments
Revised:  Experiments

Original: We ﬁrst demonstrate the capabilities of INNs on two well-behaved synthetic problems and then show results for two real-world applications from the ﬁelds of medicine and astrophysics.
Revised: We ﬁrst demonstrate the capabilities of INNs on two well-behaved synthetic problems and then show results for two real-world applications from the ﬁelds of medicine and astrophysics.

Original: Additional details on the datasets and network architectures are provided in the appendix. 
Revised: Additional details on the datasets and network architectures are provided in the appendix. [Figure]

Original: 4. Artificial data
Revised: 4. Artificial data

Original: Gaussian mixture model: To test basic viability of INNs for inverse problems, we train them on a standard 8-component Gaussian mixture model p ( x ) .
Revised: Gaussian mixture model: To test basic viability of INNs for inverse problems, we train them on a standard 8-component Gaussian mixture model p ( x ) .

Original: The forward process is very simple:
Revised: The forward process is very simple:

Original: The ﬁrst four mixture components (clockwise) are assigned label y = red , the next two get label y
Revised: The ﬁrst four mixture components (clockwise) are assigned label y = red , the next two get label y

Original: = blue , and the ﬁnal two are labeled y
Revised: = blue , and the ﬁnal two are labeled y

Original: = green and y
Revised: = green and y

Original: = purple (Fig. 2).
Revised: = purple (Fig. 2).

Original: The true inverse posteriors p ( x | y  ) consist of the mixture components corresponding to the given label.
Revised: The true inverse posteriors p ( x | y ∗ ) consist of the mixture components corresponding to the given one-hot-encoded label y ∗ .

Revised: We train the INN to directly regress one-hot vectors y using a squared loss L y , so we can provide plain one-hot vectors y ∗ to the inverse network when sampling p ( x | y ∗ ) .

Original: We observe the following: (i) The INN learns very accurate approximations of the posteriors and does not suﬀer from mode collapse.
Revised: We observe the following: (i) The INN learns very accurate approximations of the posteriors and does not suﬀer from mode collapse.

Original: (ii)
Revised: (ii)

Original: The coupling block architecture does not reduce the network’s representational power – results are similar to standard networks of comparable size (see appendix Sec. 2).
Revised: The coupling block architecture does not reduce the network’s representational power – results are similar to standard networks of comparable size (see appendix Sec. 2).

Original: (iii) Bidirectional training works best, whereas forward training alone (using only L y and L z ) captures the conditional relationships properly, but 
Revised: (iii) Bidirectional training works best, whereas forward training alone (using only L y and L z ) captures the conditional relationships properly, but places too much mass in unpopulated regions of x -space.

Original: [Figure]
Revised: Conversely, pure inverse training (just L x ) learns the correct x -distribution, but loses all conditioning information.

Original: Inverse kinematics: For a task with a more complex and continuous forward process, we simulate a simple inverse kinematics problem in 2D space:
Revised: Inverse kinematics: For a task with a more complex and continuous forward process, we simulate a simple inverse kinematics problem in 2D space:

Original: An articulated arm moves vertically along a rail and rotates at three joints.
Revised: An articulated arm moves vertically along a rail and rotates at three joints.

Original: These four degrees of freedom constitute the parameters x .
Revised: These four degrees of freedom constitute the parameters x .

Original: Their priors are given by a normal distribution, which favors a pose with 180 ◦ angles and centered origin.
Revised: Their priors are given by a normal distribution, which favors a pose with 180 ◦ angles and centered origin.

Original: The forward process is to ﬁnd the coordinates of the end point y , given a conﬁguration x .
Revised: The forward process is to calculate the coordinates of the end point y , given a conﬁguration x .

Original: The inverse problem asks for the posterior probability over all possible inputs x that place the arm’s end point at a given y position.
Revised: The inverse problem asks for the posterior distribution over all possible inputs x that place the arm’s end point at a given y position.

Original: An example for a ﬁxed y  is shown in Fig.
Revised: An example for a ﬁxed y ∗ is shown in Fig.

Original: 3, where we compare our INN to a conditional variational autoencoder (cVAE), as presented by Sohn et al. (2015).
Revised: 3, where we compare our INN to a conditional VAE (see appendix Fig. 7 for conceptual comparison of architectures). Adding Inverse Autoregressive

Original: The conceptual diﬀerence between our method and the cVAE is highlighted in the appendix, Fig.
Revised: Flow (IAF, Kingma et al., 2016) does not improve cVAE performance in this case (see appendix, Table 2).

Original: Thechosen y  is a diﬃcult example, because it isan unlikely end point position according to the prior p ( x ), it is outside the prior’s 97% quantile (Fig. 7, right ), so that p ( x | y ) considerably deviates from p ( x ) .
Revised: The y ∗ chosen in Fig. 3 is a hard example, as it is unlikely under the prior p ( x ) (Fig. 3, right ) and has a strongly bi-modal posterior p ( x | y ∗ ) .

Original: In this case, due to the computationally cheap forward process, we can use approximate
Revised: In this case, due to the computationally cheap forward process, we can use approximate

Original: Bayesian computation (ABC, see appendix Sec. 7) to compute the ground truth posterior.
Revised: Bayesian computation (ABC, see appendix Sec. 7) to sample from the ground truth posterior.

Original: Compared to ground truth, we ﬁnd that both INN and cVAE recover the two symmetric modes well.
Revised: Compared to ground truth, we ﬁnd that both INN and cVAE recover the two symmetric modes well.

Original: However, the true end points of x -samples produced by the cVAE tend to miss the target y  by a wider margin.
Revised: However, the true end points of x -samples produced by the cVAE tend to miss the target y ∗ by a wider margin.

Original: This is because the forward process x → y is only learned implicitly in the cVAE training scheme.
Revised: This is because the forward process x → y is only learned implicitly during cVAE training.

Original: See appendix for quantitative analysis and details.
Revised: See appendix for quantitative analysis and details.

Original: 4. Real-world applications
Revised: 4. Real-world applications

Original: After demonstrating the viability on synthetic data, we apply our method to two real world problems from medicine and astronomy.
Revised: After demonstrating the viability on synthetic data, we apply our method to two real world problems from medicine and astronomy.

Original: While we focus on the medical task in the following, the astronomy application is shown in Fig.
Revised: While we focus on the medical task in the following, the astronomy application is shown in Fig.

Original: [Table]

Original: In medical science [Table] the functional state of biological tissue is of interest for many applications [Table]
Revised: In medical science, the functional state of biological tissue is of interest for many applications.

Original: such as tumor detection. Tumors are expected to show changes in oxygen saturation s O(Hanahan and Weinberg (2011)).
Revised: Tumors, for example, are expected to show changes in oxygen saturation s O 2 (Hanahan and Weinberg, 2011).

Original: Such changes  inﬂuence the reﬂectance of the tissue, which can be measured by multispectral cameras (Lu and Fei (2014)).
Revised: Such changes cannot be measured directly, but inﬂuence the reﬂectance of the tissue, which can be measured by multispectral cameras (Lu and Fei, 2014).

Original: We can simulate these measurements from a tissue model  involving s O 2 , blood volume fraction v hb , scattering magnitude a mie , anisotropy g and tissue layer thickness d
Revised: Since ground truth data can not be obtained from living tissue, we create training data by simulating observed spectra y from a tissue model x involving s O 2 , blood volume fraction v hb , scattering magnitude a mie , anisotropy g and tissue layer thickness d

Original: (Wirkert et al. (2016)).
Revised: (Wirkert et al., 2016).

Original: While these simulations can determine the spectrum ( y ) for a given tissue, inverting the measurements to recover the underlying functional properties ( x ) is an active ﬁeld of research. State-of-the-art approaches (Wirkert et al. (2016; 2017); Claridge and Hidovic-Rowe (2013)) learn the inverse process directly, but cannot express the estimate’s uncertainty, which can be vital for an accurate diagnosis.
Revised: This model constitutes the forward process, and traditional methods to learn point estimates of the inverse (Wirkert et al., 2016; 2017; Claridge and Hidovic-Rowe, 2013) are already suﬃciently reliable to be used in clinical trials. However, these methods can not adequately express uncertainty and ambiguity, which may be vital for an accurate diagnosis.

Original: Competitors.
Revised: Competitors.

Original: We train an INN for this problem, along with two ablations (only forward or only inverse training), as well as a cVAE  and a network using the method of Kendall and Gal (2017), with dropout sampling and additional aleatoric error terms for each parameter.
Revised: We train an INN for this problem, along with two ablations (as in Fig. 2), as well as a cVAE with and without IAF (Kingma et al., 2016) and a network using the method of Kendall and Gal (2017), with dropout sampling and additional aleatoric error terms for each parameter.

Original: The latter also provides a point-estimate baseline (classical NN) when used without dropout and error terms, which matches the current state-of-the-art results in Wirkert et al.
Revised: The latter also provides a point-estimate baseline (classical NN) when used without dropout and error terms, which matches the current state-of-the-art results in Wirkert et al.

Original: Finally, we compare to ABC, approximating p ( x | y  ) with the 256 nearest neighbors to y  .
Revised: Finally, we compare to ABC, approximating p ( x | y ∗ ) with the 256 samples closest to y ∗ .

Original: Note that with enough samples ABC produces the true posterior.
Revised: Note that with enough samples ABC would produce the true posterior.

Original: We performed 50simulations to generate samples for ABC at test time, taking one week on a GPU, but still measure inconsistencies in the posteriors.
Revised: We performed 50 000 simulations to generate samples for ABC at test time, taking one week on a GPU, but still measure inconsistencies in the posteriors.

Original: The learning-based methods can use the training set of 15 000 samples generated oﬄine, and can be trained within minutes.
Revised: The learning-based methods are trained within minutes, on a training set of 15 000 samples generated oﬄine.

Original: Error measures.
Revised: Error measures.

Original: We are interested in both the accuracy (point estimates), and the shape of the posterior distributions.
Revised: We are interested in both the accuracy (point estimates), and the shape of the posterior distributions.

Original: For point estimates ˆ x , i.e. MAP estimates, we compute the deviation from ground-truth values x ∗ in terms of the RMSE over test set observations y ∗ ,
Revised: For point estimates ˆ x , i.e. MAP estimates, we compute the deviation from ground-truth values x ∗ in terms of the RMSE over test set observations y ∗ ,

Original: RMSE =
Revised: RMSE =

Original: (cid:112) E y ∗ [ (cid:107) ˆ x − x ∗ (cid:107) 2 ] .
Revised: (cid:112) E y ∗ [ (cid:107) ˆ x − x ∗ (cid:107) 2 ] .

Original: The scores are reported both for the main parameter of interest s O 2 , and the parameter subspace of s O 2 , v hb , a mie , which we found to be the only recoverable parameters.
Revised: The scores are reported both for the main parameter of interest s O 2 , and the parameter subspace of s O 2 , v hb , a mie , which we found to be the only recoverable parameters.

Original: Furthermore, we check the re-simulation error:
Revised: Furthermore, we check the re-simulation error:

Original: We apply the simulation s ( x ) to the point estimate, and compare the simulation outcome to the conditioning y ∗ .
Revised: We apply the simulation s (ˆ x ) to the point estimate, and compare the simulation outcome to the conditioning y ∗ .

Original: To evaluate the shape of the posteriors, we compute the calibration error for the sampling-based methods, based on the fraction of ground truth inliers q inl.
Revised: To evaluate the shape of the posteriors, we compute the calibration error for the sampling-based methods, based on the fraction of ground truth inliers α inl.

Original: for a given conﬁdence interval q .
Revised: for corresponding α -conﬁdence-region of the marginal posteriors of x .

Original: The reported error is the median of | q inl.
Revised: The reported error is the median of | α inl.

Original: − q | over all q .
Revised: − α | over all α .

Original: All values are computed over 5000 test-set observations y ∗ , or 1000 observations in the case of re-simulation error.
Revised: All values are computed over 5000 test-set observations y ∗ , or 1000 observations in the case of re-simulation error.

Original: Each posterior uses 4096 samples, or 256 for ABC, and MAP estimates are found using the mean-shift algorithm.
Revised: Each posterior uses 4096 samples, or 256 for ABC; all MAP estimates are found using the mean-shift algorithm.

Original: Quantitative results.
Revised: Quantitative results.

Original: Evaluation results for all methods are presented in Table 1.
Revised: Evaluation results for all methods are presented in Table 1.

Original: The
Revised: The

Original: INN matches or outperforms other methods in terms of point estimate error.
Revised: INN matches or outperforms other methods in terms of point estimate error.

Original: Its accuracy deteriorates slightly when trained without L x , and entirely when trained without the conditioning losses L y and L z , just as in Fig.
Revised: Its accuracy deteriorates slightly when trained without L x , and entirely when trained without the conditioning losses L y and L z , just as in Fig.

Original: The calibration error, which summarizes the correctness of the  posterior distribution in one number, is as follows for the individual methods: INN 0.34%;
Revised: For our purpose, the calibration error is the most important metric, as it summarizes the correctness of the whole posterior distribution [Table] I NN c VA E - I

Original: Dropout 1.91%; cVAE 2.19%; ABC 0.90%.
Revised: A F 1 s O D r o p o u t 0 .0 . 1 v hb 2k 4k a mie 0 .

Original: The INN has a big lead over cVAE and Dropout, andeven over ABC. See calibration curves in appendix, Fig. 10.
Revised: [Figure] in one number (see appendix Fig. 11). Here, the INN has a big lead over cVAE(-IAF) and

Revised: Dropout, and even over ABC due to the low ABC sample count.

Original: Qualitative results.
Revised: Qualitative results.

Original: Fig.
Revised: Fig.

Original: 4 shows generated parameter distributions for one ﬁxed measurement y  , comparing the INN to cVAE, Dropout sampling and ABC.
Revised: 4 shows generated parameter distributions for one ﬁxed measurement y ∗ , comparing the INN to cVAE-IAF, Dropout sampling and ABC.

Original: The three former methods 
Revised: The three former methods use a sample count of 160 000 to produce smooth curves.

Original: A B C
Revised: Due to the sparse posteriors of 256 samples in the case of ABC, kernel density estimation was applied to its results, with a bandwidth of σ = 0 .

Original: I NN c VA E 1 s O
Revised: The results produced by the INN provide relevant insights:

Original: D r o p o u t 0 .0 .
Revised: First, we ﬁnd that the posteriors for layer thickness d and anisotropy g match the shape of their priors, i.e. y holds no information about these parameters – they are unrecoverable. This ﬁnding is supported by the ABC results, whereas the other two methods misleadingly suggest a roughly Gaussian posterior.

Original: 1 v hb 2k 4k a mie [-0 .0 .1 . 
 d 0 .0 .-]
Revised: Second, we ﬁnd that the sampled distributions for the blood volume fraction v hb and scattering amplitude a mie [-0 .0 .1 . 
 d 0 .0 .-]are strongly correlated (rightmost plot).

Original: 9 g CorrelationMatrix − 1 .− 0 .− 0 .− 0 .0 .0 .0 .0 .1 .
Revised: This phenomenon is not an analysis artifact, but has a sound physical explanation: As blood volume fraction increases, more light is absorbed inside the tissue. For the sensor to

Original: IonizingLuminosity
Revised: IonizingLuminosity

Original: Ionizing Emission Rate
Revised: Ionizing Emission Rate

Original: CloudDensity
Revised: CloudDensity

Original: ExpansionVelocity
Revised: ExpansionVelocity

Original: Age of the Cluster
Revised: Age of the Cluster

Original: CorrelationMatrix − 1 .− 0 .0 .0 .1 .
Revised: CorrelationMatrixof x − 1 .− 0 .0 .0 .1 .

Revised: are inferred from multispectral measurements y .

Revised: We train an INN on simulated data, and show the sampled posterior of 5 parameters for one y ∗ (colors as in Fig.

Revised: 4, second row).

Original: The peculiar shape of the prior is due to the dynamic nature of the simulations.
Revised: The peculiar shape of the prior is due to the dynamic nature of these simulations.

Original: We multi-modal posteriors as well asstrong correlations use a sample count of 160 000 to produce smooth curves. Due to the sparse posteriors of 256 samples in the case of ABC, kernel density estimation was applied to its results, with a bandwidth of σ = 0 . The results produced by the INN provide relevant insights: First, we ﬁnd that the posteriors for layer thickness d and anisotropy g match the shape of their priors, i.e. y holds no information about these parameters – they are unrecoverable. This ﬁnding is supported by the ABC results, whereas the other two methods misleadingly suggest a roughly Gaussian posterior.
Revised: We include this application as a real-world example for the INN’s ability to recover multiple posterior modes, and strong correlations in p ( x | y ∗ ) , see details in appendix, Sec. 5.

Original: Second, we ﬁnd that the sampled distributions for the blood volume fraction v hb and scattering amplitude a mie are strongly correlated (rightmost plot).

Original: This phenomenon is not an analysis artifact, but has a sound physical explanation: As blood volume fraction increases, more light is absorbed inside the tissue.

Original: For the sensor to record the same intensities y  as before, scattering must be increased accordingly.
Revised: record the same intensities y ∗ as before, scattering must be increased accordingly.

Original: In Fig. 9 in the appendix, we show how the INN is applied to real multispectral images.
Revised: In Fig.in the appendix, we show how the INN is applied to real multispectral images.

Original:  Conclusion
Revised:  Conclusion

Original: We have shown that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically and practically on problems from medicine and astrophysics.
Revised: We have shown that the full posterior of an inverse problem can be estimated with invertible networks, both theoretically and practically on problems from medicine and astrophysics.

Original: We share the excitement of the application experts to develop INNs as a generic tool, helping them to better interpret the data and models, and to improvethe experimental setups.
Revised: We share the excitement of the application experts to develop INNs as a generic tool, helping them to better interpret their data and models, and to improve experimental setups.

Original:  In future work, we plan to apply our procedure to forward processes that cannot be fully encoded in a neural network, by incorporating domain knowledge into the method.
Revised: As a side eﬀect, our results conﬁrm the ﬁndings of others, that the restriction to coupling layers does not noticeably reduce the expressive power of the network. In future work, we plan to systematically analyze the properties of diﬀerent invertible architectures and of more ﬂexible, cycle-loss based models in this setup. We are also interested in how our method can be scaled up to higher dimensionalities, where MMD becomes less eﬀective.

Original:  Proof of correctness of generated posteriors
Revised:  Proof of correctness of generated posteriors

Original: Lemma:
Revised: Lemma:

Original: If some bijective function f : x → z transforms a probability density p X ( x ) to p Z ( z ) , then the inverse function f − 1 transforms p Z ( z ) back to p X ( x ) .
Revised: If some bijective function f : x → z transforms a probability density p X ( x ) to p Z ( z ) , then the inverse function f − 1 transforms p Z ( z ) back to p X ( x ) .

Original: Proof: We denote the probability density obtained through the reverse transformation as p ∗ X ( x ) .
Revised: Proof: We denote the probability density obtained through the reverse transformation as p ∗ X ( x ) .

Original: Therefore, we have to show that p ∗ X ( x ) =
Revised: Therefore, we have to show that p ∗ X ( x ) =

Original: p X ( x ) .
Revised: p X ( x ) .

Original: For the forward direction, via the change-of-variables formula, we have
Revised: For the forward direction, via the change-of-variables formula, we have

Original: [Equation]
Revised: [Equation]

Original: with the Jacobian ∂ z f − 1 ≡ ∂f − 1 i /∂z j .
Revised: with the Jacobian ∂ z f − 1 ≡ ∂f − 1 i /∂z j .

Original: For the reverse transformation, we have
Revised: For the reverse transformation, we have

Original: [Equation]
Revised: [Equation]

Original: We can substitute p Z from Eq.
Revised: We can substitute p Z from Eq.

Original: 6 and obtain
Revised: 6 and obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In Eq.
Revised: In Eq.

Original: 9, the Jacobians cancel out due to the inverse function theorem, i.e. the Jacobian ∂ z
Revised: 9, the Jacobians cancel out due to the inverse function theorem, i.e. the Jacobian ∂ z

Original: ( f − 1 ) is the matrix inverse of ∂ x f .
Revised: ( f − 1 ) is the matrix inverse of ∂ x f .

Original: Theorem:
Revised: Theorem:

Original: If an INN f ( x ) =
Revised: If an INN f ( x ) =

Original: [ y , z ] is trained as proposed, and both the supervised loss
Revised: [ y , z ] is trained as proposed, and both the supervised loss

Original: L y =
Revised: L y =

Original: E
Revised: E

Original: [( y − f y ( x )) 2 ] and the unsupervised loss L z = D (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) reach zero, sampling according to Eq.
Revised: [( y − f y ( x )) 2 ] and the unsupervised loss L z = D (cid:0) q ( y , z ) , p ( y ) p ( z ) (cid:1) reach zero, sampling according to Eq.

Original: 1 with g = f − 1 returns the true posterior p ( x |ˆ y  ) for any measurementˆ y  .
Revised: 1 with g = f − 1 returns the true posterior p ( x | y ∗ ) for any measurement y ∗ .

Original: Proof: We denote the chosen latent distribution as p Z ( z ) , the distribution of observations as p Y ( y ) , and the joint distribution of network outputs as q ( y , z ) .
Revised: Proof: We denote the chosen latent distribution as p Z ( z ) , the distribution of observations as p Y ( y ) , and the joint distribution of network outputs as q ( y , z ) .

Original: As shown by Gretton et al.
Revised: As shown by Gretton et al.

Original: (2012), if the MMD loss converges to 0, the network outputs follow the prescribed distribution:
Revised: (2012), if the MMD loss converges to 0, the network outputs follow the prescribed distribution:

Original: [Equation]
Revised: [Equation]

Original: Suppose we take a posterior conditioned on a ﬁxedˆ y  , i.e. p ( x |ˆ y  ) , and transform it using the forward pass of our perfectly converged INN.
Revised: Suppose we take a posterior conditioned on a ﬁxed y ∗ , i.e. p ( x | y ∗ ) , and transform it using the forward pass of our perfectly converged INN.

Original: From this we obtain an output distributionˆ q  ( y , z ) .
Revised: From this we obtain an output distribution q ∗ ( y , z ) .

Original: Because L y = 0 , we know that the output distribution of y (marginalized over z ) must beˆ q  ( y ) =
Revised: Because L y = 0 , we know that the output distribution of y (marginalized over z ) must be q ∗ ( y ) =

Original: δ ( y −ˆ y  ) .
Revised: δ ( y − y ∗ ) .

Original: Also, because of the independence between z and y in the output, the distribution of z -outputs is stillˆ q  ( z ) =
Revised: Also, because of the independence between z and y in the output, the distribution of z -outputs is still q ∗ ( z ) =

Original: p Z ( z ) .
Revised: p Z ( z ) .

Original: So the joint distribution of outputs is
Revised: So the joint distribution of outputs is

Original: [Equation]
Revised: [Equation]

Original: When we invert the network, and repeatedly inputˆ y  while sampling z ∼ p
Revised: When we invert the network, and repeatedly input y ∗ while sampling z ∼ p

Original: Z ( z ) , this is the same as sampling [ y , z ] from theˆ q  ( y , z ) above.
Revised: Z ( z ) , this is the same as sampling [ y , z ] from the q ∗ ( y , z ) above.

Original: Using the Lemma from above, we know that the inverted network will output samples from p ( x |ˆ y  ) .
Revised: Using the Lemma from above, we know that the inverted network will output samples from p ( x | y ∗ ) .

Original: Corollary:
Revised: Corollary:

Original: If the conditions of the theorem above are fulﬁlled, the unsupervised reverse loss L x = D (cid:0) q ( x ) , p X ( x ) (cid:1) between the marginalized outputs of the inverted network, q ( x ) , and the prior data distribution, p X ( x ) , will also be 0.
Revised: If the conditions of the theorem above are fulﬁlled, the unsupervised reverse loss L x = D (cid:0) q ( x ) , p X ( x ) (cid:1) between the marginalized outputs of the inverted network, q ( x ) , and the prior data distribution, p X ( x ) , will also be 0.

Original: This justiﬁes using the loss on the prior to speed up convergence, without altering the ﬁnal results.
Revised: This justiﬁes using the loss on the prior to speed up convergence, without altering the ﬁnal results.

Original: Proof: Due to the theorem, the estimated posteriors generated by the INN are correct, i.e. q ( x |ˆ y  ) =
Revised: Proof: Due to the theorem, the estimated posteriors generated by the INN are correct, i.e. q ( x | y ∗ ) =

Original: p ( x |ˆ y  ) .
Revised: p ( x | y ∗ ) .

Original: If they are marginalized over observationsˆ y  from the training data, then q ( x ) will be equal to p X ( x ) by deﬁnition.
Revised: If they are marginalized over observations y ∗ from the training data, then q ( x ) will be equal to p X ( x ) by deﬁnition.

Original: As shown by Gretton et al.
Revised: As shown by Gretton et al.

Original: (2012), this is equivalent to L x = 0 .
Revised: (2012), this is equivalent to L x = 0 .

Original:  Artificial data – Gaussian mixture
Revised:  Artificial data – Gaussian mixture

Original: In Sec. 4.1, we demonstrate that the proposed INN can approximate the true posteriors very well and is not hindered by the required coupling block architecture.
Revised: In Sec. 4.1, we demonstrate that the proposed INN can approximate the true posteriors very well and is not hindered by the required coupling block architecture.

Original: Here we show how some existing methods do on the same task, using neural networks of similar size as the INN.
Revised: Here we show how some existing methods do on the same task, using neural networks of similar size as the INN.

Original: [Figure]
Revised: [Figure]

Original: cGAN
Revised: cGAN

Original: Training a conditional GAN of network size comparable to the INN (counting only the generator) and only two noise dimensions turned out to be challenging.
Revised: Training a conditional GAN of network size comparable to the INN (counting only the generator) and only two noise dimensions turned out to be challenging.

Original: Even with additional pre-training to avoid mode collapse, the individual modes belonging to one label are reduced to nearly one-dimensional structures.
Revised: Even with additional pre-training to avoid mode collapse, the individual modes belonging to one label are reduced to nearly one-dimensional structures.

Original: Larger cGAN
Revised: Larger cGAN

Original: In order to match the results of the INN, we trained a more complex cGAN with 2M parameters instead of the previous 10K, and a latent dimension of 128, instead of 2.
Revised: In order to match the results of the INN, we trained a more complex cGAN with 2M parameters instead of the previous 10K, and a latent dimension of 128, instead of 2.

Original: To prevent mode collapse, we introduced an additional regularization: an extra loss term forces the variance of generator outputs to match the variance of the training data prior.
Revised: To prevent mode collapse, we introduced an additional regularization: an extra loss term forces the variance of generator outputs to match the variance of the training data prior.

Original: With these changes, the cGAN can be seen to recover the posteriors reasonably well.
Revised: With these changes, the cGAN can be seen to recover the posteriors reasonably well.

Original: Generator + MMD Another option is to keep the cGAN generator the same size as our
Revised: Generator + MMD Another option is to keep the cGAN generator the same size as our

Original: INN, but replace the discriminator with an MMD loss (cf.
Revised: INN, but replace the discriminator with an MMD loss (cf.

Original: Sec. 3.4).
Revised: Sec. 3.4).

Original: This loss receives a concatenation of the generator output x and the label y
Revised: This loss receives a concatenation of the generator output x and the label y

Original: it was supplied with, and compares these batch-wise with the concatenation of ground truth ( x , y ) -pairs.
Revised: it was supplied with, and compares these batch-wise with the concatenation of ground truth ( x , y ) -pairs.

Original: Note that in contrast to this, the corresponding MMD loss of the INN only receives x , and no information about y .
Revised: Note that in contrast to this, the corresponding MMD loss of the INN only receives x , and no information about y .

Original: For this small toy problem, we ﬁnd that the hand-crafted MMD loss dramatically improves results compared to a learned discriminator.
Revised: For this small toy problem, we ﬁnd that the hand-crafted MMD loss dramatically improves results compared to the smaller learned discriminator.

Original: cVAE
Revised: cVAE

Original: We also compare with a conditional Variational Autoencoder ofthe same size.
Revised: We also compare to a conditional Variational Autoencoder of same total size as the

Original:  There is some similarity between the training setup of our method (Fig. 7, right ) and that of a cVAE (Fig. 7, left ), as the forward and inverse pass of an INN can also be seen as an encoder-decoder pair.
Revised: INN. There is some similarity between the training setup of our method (Fig. 7, right ) and Invertible [Figure]

Revised: x

Revised: INN y z forward (simulation): x → y inverse (sampling): [ y , z ] → x

Revised: U S L

Revised: S L

Revised: U S L x y z

Revised: IAF Encoder

Revised: Decoder [ x , y ] → µ z , σ z → z [ z , y ] → x r e c o n s t r u c t i o n l o ss

Revised: E L B

Revised: O l o ss that of cVAE (Fig. 7, left ), as the forward and inverse pass of an INN can also be seen as an encoder-decoder pair.

Original: The main diﬀerences are that the cVAE learns the relationship x
Revised: The main diﬀerences are that the cVAE learns the relationship x

Original: → y only indirectly, since there is no explicit loss for it, and that the INN requires no reconstruction loss, since it is bijective by construction.
Revised: → y only indirectly, since there is no explicit loss for it, and that the INN requires no reconstruction loss, since it is bijective by construction.

Original:  On the Gaussian mixture toy problem, the trained cVAE generates correct posteriors for the uni-modal labels, but introduces too much variance in the multi-modal cases.
Revised: cVAE-IAF We adapt the cVAE to use Inverse Autoregressive Flow (Kingma et al., 2016) between the encoder and decoder. On the Gaussian mixture toy problem, the trained cVAE-IAF generates correct posteriors on par with our INN (see Fig. 6).

Original: Dropout sampling The method of dropout sampling with learned error terms is by construction not able to produce multi-modal outputs, and therefore fails on this task.
Revised: Dropout sampling The method of dropout sampling with learned error terms is by construction not able to produce multi-modal outputs, and therefore fails on this task.

Original: 2. Latent space analysis
Revised: 2. Latent space analysis

Original: To analyze how the latent space  is structured for this task, we choose a ﬁxed label y  and sample z from a dense grid.
Revised: To analyze how the latent space of our INN is structured for this task, we choose a ﬁxed label y ∗ and sample z from a dense grid.

Original: For each z , we compute x through our inverse network and colorize this point in latent ( z ) space according to the distance from the closest mode in 
Revised: For each z , we compute x through our inverse network and colorize this point in latent ( z ) space according to the distance from the closest mode in x -space.

Original: Invertible Neural Network Conditional Variational Autoencoder x INN y z forward (simulation): x → y inverse (sampling): [ y , z ] → x
Revised: We can see that our network learns to shape the latent space such that each mode receives the expected fraction of samples (Fig.

Original: U S L S L U S L x y z
Revised: to used not relate those

Original: Encoder Decoder [ x , y ] → µ z , σ z [ z , y ] → x r e c o n s t r u c t i o n l o ss
Revised: Fig.

Original: E L B O l o ss
Revised: x label y

Original:  Artificial data – inverse kinematics
Revised:  Artificial data – inverse kinematics

Original: A short video demonstrating the structure of our INN’s latent space can be found under https://gfycat.com/SoggyCleanHog , for a slightly diﬀerent arm setup.
Revised: A short video demonstrating the structure of our INN’s latent space can be found under https://gfycat.com/SoggyCleanHog , for a slightly diﬀerent arm setup.

Original: To judge the quality of posteriors, we quantify both the re-simulation error, as in Sec. 4. of the paper, as well as the reduced χ 2 red.

Original: value of sampled x  , which corresponds to the normalized expected log-likelihood of the generated samples.
Revised: The dataset is constucted using gaussian priors x i ∼ N (0 , σ i ) , with σ 1 = 0 . 25 and σ 2 = σ 3 = σ 4 = 0 . 5 ∧ = 28 .

Original: (The reduced χ 2 red.
Revised: 65 ◦ .

Original:  is deﬁned as χ 2 red. =
Revised: The forward process is given by

Original: χ 2 /ν , with ν denoting the degrees of freedom).
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 13) with i ∈ { 1  ,  2  ,  3 , 4 } for each x -dimension, and σ i , µ i of the prior.
Revised: with the arm lenghts l 1 = 0 . 5 , l 2 = 0 . 5 , l 3 = 1 . 0 .

Original: With the expectation overy ∗ from the test set, the ground truth value of χred. is 1.
Revised: To judge the quality of posteriors, we quantify both the re-simulation error and the calibration error over the test set, as in Sec. 4.2 of the paper.

Original: We ﬁnd that the re-simulation error of the INN is dramatically lower than that of the cVAE, with slightly better quality of the prior, conﬁrming the observations in Fig.
Revised: Because of the cheap simulation, we average the re-simulation error over the whole posterior, and not only the MAP estimate. In Table 2, we ﬁnd that the INN has a clear advantage in both metrics, conﬁrming the observations from Fig.

Original: [Table]
Revised: [Table]

Original:  Multispectral measurements of biological tissue
Revised:  Multispectral measurements of biological tissue

Original: The following ﬁgure shows the results when the INN trained in Sec.
Revised: The following ﬁgure shows the results when the INN trained in Sec.

Original: 4.2 is applied pixel-wise to multispectral endoscopic footage.
Revised: 4.2 is applied pixel-wise to multispectral endoscopic footage.

Original: In addition to estimating the oxygenation s O 2 , we measure the uncertainty in the form of the 68% conﬁdence interval. 
Revised: In addition to estimating the oxygenation s O 2 , we measure the uncertainty in the form of the 68% conﬁdence interval. [Figure] [Figure] Est

Original: [Figure]
Revised: [Figure]

Original: The clips ( arrows ) on the connecting tissue cause lower oxygenation (blue) in the small intestine. Uncertainty is low in crucial areas and high only at some edges and specularities.
Revised: uncertainty 0 .0 .0 .0 .0 .0 .0 .0 .0 . [Figure]

Original:  Star cluster spectral data
Revised:  Star cluster spectral data

Original: Star clusters are born from a large reservoir of gas and dust that permeates the galaxy, the interstellar medium.
Revised: Star clusters are born from a large reservoir of gas and dust that permeates the Galaxy, the interstellar medium (ISM).

Revised: The densest parts of the ISM are called molecular clouds, and star formation occurs in regions that become unstable under their own weight.

Original: The process is governed by the complex interplay of competing physical agents such as gravity, turbulence, magnetic ﬁelds, and radiation; with stellar feedback playing a decisive regulatory role S. Klessen and C. O. Glover (2016).
Revised: The process is governed by the complex interplay of competing physical agents such as gravity, turbulence, magnetic ﬁelds, and radiation; with stellar feedback playing a decisive regulatory role (S. Klessen and C. O. Glover, 2016).

Original: To study this, astronomers measure emission lines from chemical elements such as hydrogen or oxygen
Revised: To characterize the impact of the energy and momentum input from young star clusters on the dynamical evolution of the ISM, astronomers frequently study emission lines from chemical elements such as hydrogen or oxygen.

Original: Baldwin et al. ; Kewley et al. Their relative intensities depend on the ionization potential, the spectrum of the ionizing radiation, the gas density, and the absolute intensity of the radiation Pellegrini et al.
Revised: These lines are produced when gas is ionized by stellar radiation, and their relative intensities depend on the ionization potential of the chemical species, the spectrum of the ionizing radiation, the gas density as well as the 3D geometry of the cloud, and the absolute intensity of the radiation (Pellegrini et al., 2011). Key diagnostic tools are the so-called BPT diagrams (after Baldwin et al., 1981) emission of ionized hydrogen, H+ , to normalize the recombination lines of O++ , O+ and S+ (see also Kewley et al., 2013).

Original: We investigate the dynamical feedback of young star clusters on their parental environment in a 1D model  Rahner et al.
Revised: We investigate the dynamical feedback of young star clusters on their parental cloud using the WARPFIELD 1D model developed by Rahner et al.

Original: (2017) which describes the complete temporal evolution of the system and allows us to generate synthetic emission line maps.
Revised: It follows the entire temporal evolution of the system until the cloud is destroyed, which could take several stellar populations to happen. At each timestep we employ radiative transfer calculations (Reissl et al., 2016) to generate synthetic emission line maps which we use to train the neural network.

Original: Similar to the medical application, thereverse of this mapping  is highly degenerate and ill-posed.
Revised: Similar to the medical application from Section 4.2, the mapping from simulated observations to underlying physical parameters (such as cloud and cluster mass, and total age of the system) is highly degenerate and ill-posed.

Original: We train our forward model to predict the observable quantities y (e.g. emission line ratios) from composite simulation outputs x (e.g. ionizing luminosity and emission rate, cloud density, expansion velocity, and age of the youngest cluster in the system).
Revised: As an intermediary step, we therefore train our forward model to predict the observable quantities y (emission line ratios) from composite simulation outputs x (such as ionizing luminosity and emission rate, cloud density, expansion velocity, and age of the youngest cluster in the system, which in the case of multiple stellar populations could be considerably smaller than the total age).

Original: Fig.shows the resulting marginal posteriors for these ﬁve parameters given  one speciﬁc y .
Revised: Using the inverse of our trained model for a given set of observations y ∗ , we can obtain a distribution over the unobservable properties x of the system. Results for one speciﬁc y are shown in Fig.

Original: Note that our network recovers a decidedly multimodal distribution of x that visibly deviates from the prior p ( x ) .
Revised: Note that our network recovers a decidedly multimodal distribution of x that visibly deviates from the prior p ( x ) .

Original: Note also the strong correlations in the system.
Revised: Note also the strong correlations in the system.

Original: For example, the measurements y  investigated may correspond to a young cluster with large expansion velocity, or to an older system that expands slowly.
Revised: For example, the measurements y ∗ investigated may correspond to a young cluster with large expansion velocity, or to an older system that expands slowly.

Original: Finding these ambiguities in p ( x | y  ) and identifying degeneracies in the underlying model are pivotal aspects of astrophysical research, and a method to eﬀectively approximate full posterior distributions has the potential to lead to a major breakthrough in this ﬁeld.
Revised: Finding these ambiguities in p ( x | y ∗ ) and identifying degeneracies in the underlying model are pivotal aspects of astrophysical research, and a method to eﬀectively approximate full posterior distributions has the potential to lead to a major breakthrough in this ﬁeld.

Original:  Calibration curve for tissue parameter estimation
Revised:  Calibration curve for tissue parameter estimation

Original: In Sec.
Revised: In Sec.

Original: 4.2, we report the median calibration error for each method.
Revised: 4.2, we report the median calibration error for each method.

Original: The following ﬁgure plots the calibration error, q inliers − q , against the level of conﬁdence q .
Revised: The following ﬁgure plots the calibration error, q inliers − q , against the level of conﬁdence q .

Original: Negative values mean that a model is overconﬁdent, while positive values say the opposite.
Revised: Negative values mean that a model is overconﬁdent, while positive values say the opposite.

Original: [Figure]
Revised: [Figure]

Original:  Approximate Bayesian computation (ABC)
Revised:  Approximate Bayesian computation (ABC)

Original: While there is a whole ﬁeld of research concerned with ABC approaches and their eﬃciencyaccuracy tradeoﬀs, our use of the method here is limited to the essential principle of rejection sampling.
Revised: While there is a whole ﬁeld of research concerned with ABC approaches and their eﬃciencyaccuracy tradeoﬀs, our use of the method here is limited to the essential principle of rejection sampling.

Original: When we require N samples of x from the posterior p ( x | y ∗ ) conditioned on some y ∗ , there are two basic ways to obtain them:
Revised: When we require N samples of x from the posterior p ( x | y ∗ ) conditioned on some y ∗ , there are two basic ways to obtain them:

Original: Threshold: We set an acceptance threshold (cid:15) , repeatedly draw x -samples from the prior, compute the corresponding y -values (via simulation) and keep those where dist ( y , y ∗ ) < (cid:15) , until we have accepted N samples.
Revised: Threshold: We set an acceptance threshold (cid:15) , repeatedly draw x -samples from the prior, compute the corresponding y -values (via simulation) and keep those where dist ( y , y ∗ ) < (cid:15) , until we have accepted N samples.

Original: The smaller we want (cid:15) , the more simulations have to be run, which is why we use this approach only for the experiment in Sec.
Revised: The smaller we want (cid:15) , the more simulations have to be run, which is why we use this approach only for the experiment in Sec.

Original: 4.1, where we can aﬀord to run the forward process millions or even billions of times.
Revised: 4.1, where we can aﬀord to run the forward process millions or even billions of times.

Original: Quantile: Alternatively, we choose what quantile q of samples shall be accepted, and then run exactly N/q simulations.
Revised: Quantile: Alternatively, we choose what quantile q of samples shall be accepted, and then run exactly N/q simulations.

Original: All sampled pairs ( x , y ) are sorted by dist ( y , y ∗ ) and
Revised: All sampled pairs ( x , y ) are sorted by dist ( y , y ∗ ) and

Original: the
Revised: the

Original: N closest to y ∗ form the posterior.
Revised: N closest to y ∗ form the posterior.

Original: This allows for a more predictable runtime when the simulations are costly, as in the medical application in Sec. 4.2.
Revised: This allows for a more predictable runtime when the simulations are costly, as in the medical application in Sec. 4.2.

Original:  Details of datasets and network architectures
Revised:  Details of datasets and network architectures

Original: Table 3 summarizes the datasets used throughout the paper.
Revised: Table 3 summarizes the datasets used throughout the paper.

Original: The architecture details are given in the following.
Revised: The architecture details are given in the following.

Original: [Table]
Revised: [Table]

Original: see also
Revised: see also

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Wirkert et al.
Revised: Wirkert et al.

Original: [Table]

Original: Pellegrini et al.
Revised: Pellegrini et al.

Original: 8. Artificial data – Gaussian mixture
Revised: 8. Artificial data – Gaussian mixture

Original: INN: 3 invertible blocks, 3 fully connected layers per aﬃne coeﬃcient function with ReLU activation functions in the intermediate layers, zero padding to a nominal dimension of 16,
Revised: INN: 3 invertible blocks, 3 fully connected layers per aﬃne coeﬃcient function with ReLU activation functions in the intermediate layers, zero padding to a nominal dimension of 16,

Original: Adam optimizer, decaying learning rate from 10 − 3 to 10 − 5 , batch size 200.
Revised: Adam optimizer, decaying learning rate from 10 − 3 to 10 − 5 , batch size 200.

Original: The inverse multiquadratic kernel was used for MMD, with α = 0 .
Revised: The inverse multiquadratic kernel was used for MMD, with h = 0 .

Original: 2 in both x - and z -space.
Revised: 2 in both x - and z -space.

Original: Dropout sampling: 6 fully connected layers with ReLU activations, Adam optimizer, learning rate decay from 10 − 3 to 10 − 5 , batch size 200, dropout probability p = 0 . 2 .
Revised: Dropout sampling: 6 fully connected layers with ReLU activations, Adam optimizer, learning rate decay from 10 − 3 to 10 − 5 , batch size 200, dropout probability p = 0 . 2 .

Original: cGAN: 6 fully connected layers for the generator and 8 for the discriminator, all with leaky
Revised: cGAN: 6 fully connected layers for the generator and 8 for the discriminator, all with leaky

Original: ReLU activations.
Revised: ReLU activations.

Original: Adam was used for the generator, SGD for the discriminator, learning rates decaying from 2 · 10 − 3 to 2 · 10 − 6 , batch size 256.
Revised: Adam was used for the generator, SGD for the discriminator, learning rates decaying from 2 · 10 − 3 to 2 · 10 − 6 , batch size 256.

Original: Initially 100 iterations training with
Revised: Initially 100 iterations training with

Original: L = 1 N (cid:80) i (cid:107) g
Revised: L = 1 N (cid:80) i (cid:107) g

Original: ( z i , y i ) − x i (cid:107) 22 , to separate the diﬀerently labeled modes, followed by pure GAN training.
Revised: ( z i , y i ) − x i (cid:107) 22 , to separate the diﬀerently labeled modes, followed by pure GAN training.

Original: Larger cGAN: 2 fully connected layers with 1024 neurons each for discriminator and generator, batch size 512, Adam optimizer with learning rate 8 · 10 − 4 for the generator, SGD with learning rate 1 . 2 · 10 − 3 and momentum 0.05 for the discriminator, 1 . 6 · 10 − 3 weight decay for both, 0.25 dropout probabiliy for the generator at training and test time.
Revised: Larger cGAN: 2 fully connected layers with 1024 neurons each for discriminator and generator, batch size 512, Adam optimizer with learning rate 8 · 10 − 4 for the generator, SGD with learning rate 1 . 2 · 10 − 3 and momentum 0.05 for the discriminator, 1 . 6 · 10 − 3 weight decay for both, 0.25 dropout probabiliy for the generator at training and test time.

Original: Equal weighting of discriminator loss and penalty of output variance L
Revised: Equal weighting of discriminator loss and penalty of output variance L

Original: =
Revised: =

Original: ( Var i [ g ( z i , y i )]
Revised: ( Var i [ g ( z i , y i )]

Original: − Var i [ x i ])
Revised: − Var i [ x i ])

Original: Generator with MMD: 8 fully connected layers with leaky ReLU activations, Adam optimizer, decaying learning rate from 10 − 3 to 10 − 6 , batch size 256.
Revised: Generator with MMD: 8 fully connected layers with leaky ReLU activations, Adam optimizer, decaying learning rate from 10 − 3 to 10 − 6 , batch size 256.

Original: Inverse multiquadratic kernel, α = 0 .
Revised: Inverse multiquadratic kernel, h = 0 .

Original: cVAE: 3 fully connected layers each for encoder and decoder, ReLU activations, learning rate 2 · 10 − 2 , decay to 2 .
Revised: cVAE: 3 fully connected layers each for encoder and decoder, ReLU activations, learning rate 2 · 10 − 2 , decay to 2 .

Original: 5 · 10 − 5 , Adam optimizer, batch size 25, reconstruction loss weighted 50:1 versus KL divergence loss.
Revised: 5 · 10 − 5 , Adam optimizer, batch size 25, reconstruction loss weighted 50:1 versus KL divergence loss.

Original: 8. Artificial data – inverse kinematics
Revised: 8. Artificial data – inverse kinematics

Original: INN: 6 aﬃne coupling blocks with 3 fully connected layers each and leaky ReLU activations
Revised: INN: 6 aﬃne coupling blocks with 3 fully connected layers each and leaky ReLU activations

Original: Adam optimizer,
Revised: Adam optimizer

Original: decaying learning rate from 10 − 2 to 10 −
Revised: decaying learning rate from 10 − 2 to 10 −

Original: multiquadratic kernel with α = 1 . 
Revised: multiquadratic kernel with h = 1 . 2 .

Original: cVAE: 4 fully connected layers each for encoder and decoder
Revised: cVAE: 4 fully connected layers each for encoder and decoder

Original: , ReLU activations, learning rate 5 · 10 − 3 , decay to 1 . 6 · 10 − 5 , Adam optimizer, batch size 250, reconstruction loss weighted 15:1 versus KL divergence loss.
Revised: , ReLU activations, learning rate 5 · 10 − 3 , decay to 1 . 6 · 10 − 5 , Adam optimizer, batch size 250, reconstruction loss weighted 15:1 versus KL divergence loss.

Original: 8. Functional parameter estimation from multispectral tissue images
Revised: 8. Functional parameter estimation from multispectral tissue images

Original: INN: 3 invertible blocks, 4 fully connected layers per aﬃne coeﬃcient function with leaky
Revised: INN: 3 invertible blocks, 4 fully connected layers per aﬃne coeﬃcient function with leaky

Original: ReLUs in the intermediate layers, zero padding to double the original width.
Revised: ReLUs in the intermediate layers, zero padding to double the original width.

Original: Adam optimizer, learning rate decay from 2 · 10 − 3 to 2 · 10 − 5 , batch size 200.
Revised: Adam optimizer, learning rate decay from 2 · 10 − 3 to 2 · 10 − 5 , batch size 200.

Original: Inverse multiquadratic kernel with α = 1 , weighted MMD terms by observation distance with decaying γ = 0 .
Revised: Inverse multiquadratic kernel with h = 1 , weighted MMD terms by observation distance with decaying γ = 0 .

Original: 2 to 0.
Revised: 2 to 0.

Original: Dropout sampling/point estimate: 8 fully connected layers, ReLU activations, Adam with decaying learning rate from 10 − 2 to 10 − 5 , batch size 100, dropout probability p = 0 .
Revised: Dropout sampling/point estimate: 8 fully connected layers, ReLU activations, Adam with decaying learning rate from 10 − 2 to 10 − 5 , batch size 100, dropout probability p = 0 .

Original: cVAE: 4 fully connected layers each for encoder and decoder, ReLU activations, learning rate 10 − 3 , decay to 3 . 2 · 10 − 6 , Adam optimizer, batch size 25, reconstruction loss weighted 10 3 :1 versus KL divergence loss.
Revised: cVAE: 4 fully connected layers each for encoder and decoder, ReLU activations, learning rate 10 − 3 , decay to 3 . 2 · 10 − 6 , Adam optimizer, batch size 25, reconstruction loss weighted 10 3 :1 versus KL divergence loss.

Original: 8. Impact of star clusters on the dynamical evolution of the galactic gas
Revised: 8. Impact of star clusters on the dynamical evolution of the galactic gas

Original: INN: 5 invertible blocks, 4 fully connected layers per aﬃne coeﬃcient function with leaky ReLUs in the intermediate layers, no additional zero padding.
Revised: INN: 5 invertible blocks, 4 fully connected layers per aﬃne coeﬃcient function with leaky ReLUs in the intermediate layers, no additional zero padding.

Original: Adam optimizer with decaying learning rate from 2 · 10 − 3 to 1 . 5 · 10 − 6 , batch size 500.
Revised: Adam optimizer with decaying learning rate from 2 · 10 − 3 to 1 . 5 · 10 − 6 , batch size 500.

Original: Kernel for latent space: k ( z , z (cid:48) )
Revised: Kernel for latent space: k ( z , z (cid:48) )

Original: =
Revised: =

Original: exp( − α (cid:107) z − z (cid:48)  (cid:107) 2 ) with α = 0 .
Revised: exp( −(cid:107) ( z − z (cid:48) ) /h (cid:107) 2 ) with h = 7 . 1 .

Original: Kernel for x -space: k ( x , x (cid:48) )
Revised: Kernel for x -space: k ( x , x (cid:48) )

Original: = −(cid:107) x − x (cid:48) (cid:107) 1 / 4 1 / 2 .
Revised: = −(cid:107) x − x (cid:48) (cid:107) 1 / 4 1 / 2 .

Original: Due to the complex nature of the prior distributions, this was the kernel found to capture the details correctly, whereas the peak of the inverse multiquadratic kernel was too broad for this purpose.
Revised: Due to the complex nature of the prior distributions, this was the kernel found to capture the details correctly, whereas the peak of the inverse multiquadratic kernel was too broad for this purpose.