Original: A PPROXIMATE P ROBABILISTIC I NFERENCE WITH C OMPOSED F LOWS
Revised: A PPROXIMATE P ROBABILISTIC I NFERENCE WITH C OMPOSED F LOWS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the problem of probabilistic inference on the joint distribution deﬁned by a normalizing ﬂow model
Revised: We study the problem of probabilistic inference on the joint distribution deﬁned by a normalizing ﬂow model.

Original: Given a pre-trained ﬂow model p ( x ) , we wish to estimate p ( x 2 | x 1 ) for some arbitrary partitioning of the variables x
Revised: Given a pre-trained ﬂow model p ( x ) , we wish to estimate p ( x 2 | x 1 ) for some arbitrary partitioning of the variables x

Original: = ( x 1 , x 2 ) .
Revised: = ( x 1 , x 2 ) .

Original: We ﬁrst show that this task is computationally hard for a large class of ﬂow models.
Revised: We ﬁrst show that this task is computationally hard for a large class of ﬂow models.

Original: Motivated by this hardness result , we propose a framework for approximate probabilistic inference.
Revised: Motivated by this hardness result, we propose a framework for approximate probabilistic inference.

Original: Speciﬁcally, our method trains a new generative model with the property that its composition with the given model approximates the target conditional distribution.
Revised: Speciﬁcally, our method trains a new generative model with the property that its composition with the given model approximates the target conditional distribution.

Original: By parametrizing this new distribution as another ﬂow model, we can efﬁciently train it using variational inference and also handle conditioning under arbitrary differentiable transformations.
Revised: By parametrizing this new distribution as another ﬂow model, we can efﬁciently train it using variational inference and also handle conditioning under arbitrary differentiable transformations.

Original:  We  experimentally demonstrate that our approach outperforms Langevin Dynamics in terms of sample quality while requiring much fewer parameters and training time compared to regular variational inference
Revised: Since the resulting approximate posterior remains a ﬂow, it offers exact likelihood evaluation, inversion, and efﬁcient sampling. We provide an extensive empirical evidence showcasing the ﬂexibility of our method on a variety of inference tasks with applications to inverse problems. We also experimentally demonstrate that our approach is comparable to simple MCMC baselines in terms of sample quality.

Original: We further validate the ﬂexibility of  our method on a variety of inference tasks with applications to inverse problems
Revised: Further, we explain the failure of naively applying variational inference and show that our method does not suffer from the same issue.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative modeling has seen an unprecedented growth in the recent years.
Revised: Generative modeling has seen an unprecedented growth in the recent years.

Original: Building on the success of deep learning, deep generative models have shown impressive ability to model complex distributions in a variety of domains and modalities.
Revised: Building on the success of deep learning, deep generative models have shown impressive ability to model complex distributions in a variety of domains and modalities.

Original: Among them, normalizing ﬂow models (see Papamakarios et al.
Revised: Among them, normalizing ﬂow models (see Papamakarios et al.

Original: and references therein) stand out due to their computational ﬂexibility, as they offer efﬁcient sampling, likelihood evaluation, and inversion.
Revised: and references therein) stand out due to their computational ﬂexibility, as they offer efﬁcient sampling, likelihood evaluation, and inversion.

Original: While other types of models currently outperform ﬂow models in terms of likelihood and sample quality, ﬂow models have the advantage that they are relatively easy to train using maximum likelihood and do not suffer from issues that other models possess (e.g. mode collapse for GANs, posterior collapse for VAEs, slow sampling for autoregressive models).
Revised: While other types of models currently outperform ﬂow models in terms of likelihood and sample quality, ﬂow models have the advantage that they are relatively easy to train using maximum likelihood and do not suffer from issues that other models possess (e.g. mode collapse for GANs, posterior collapse for VAEs, slow sampling for autoregressive models).

Original: These characteristics make normalizing ﬂows attractive for a variety of downstream tasks, including density estimation, inverse problems, semi-supervised learning, reinforcement learning, and audio synthesis (Ho et al., 2019;
Revised: These characteristics make normalizing ﬂows attractive for a variety of downstream tasks, including density estimation, inverse problems, semi-supervised learning, reinforcement learning, and audio synthesis (Ho et al., 2019;

Original: Asim et al., 2019;
Revised: Asim et al., 2019;

Original: Atanov et al., 2019;
Revised: Atanov et al., 2019;

Original: Ward et al., 2019; Oord et al., 2018).
Revised: Ward et al., 2019; Oord et al., 2018).

Original: Even with such computational ﬂexibility, how to perform efﬁcient probabilistic inference on a ﬂow model still remains largely unknown.
Revised: Even with such computational ﬂexibility, how to perform efﬁcient probabilistic inference on a ﬂow model still remains largely unknown.

Original: This question is becoming increasingly important as generative models increase in size and the computational resources necessary to train them from scratch are out of reach for many researchers and practitioners 1 .
Revised: This question is becoming increasingly important as generative models increase in size and the computational resources necessary to train them from scratch are out of reach for many researchers and practitioners 1 .

Original: If it was possible to perform probabilistic inference on ﬂow models, we could re-purpose these powerful pre-trained generators for numerous custom tasks.
Revised: If it was possible to perform probabilistic inference on ﬂow models, we could re-purpose these powerful pre-trained generators for numerous custom tasks.

Original: This is the central question we study in this paper: One wishes to estimate the conditional distribution p ( x 2 | x 1 ) from a given ﬂow model p ( x ) for some partitioning of variables x
Revised: This is the central question we study in this paper: One wishes to estimate the conditional distribution p ( x 2 | x 1 ) from a given ﬂow model p ( x ) for some partitioning of variables x

Original: = ( x 1 , x 2 ) .
Revised: = ( x 1 , x 2 ) .

Original: Existing methods for this task largely fall under two categories: Markov Chain Monte Carlo (MCMC) and variational inference (VI).
Revised: Existing methods for this task largely fall under two categories: Markov Chain Monte Carlo (MCMC) and variational inference (VI).

Original: While MCMC methods can perform exact conditional sampling in theory, they often have prohibitively long mixing time for complex high-dimensional distributions and also do not provide likelihoods.
Revised: While MCMC methods can perform exact conditional sampling in theory, they often have prohibitively long mixing time for complex high-dimensional distributions and also do not provide likelihoods.

Original: On the other hand, VI allows for approximate likelihood evaluation under the variational posterior and fast sampling, but at a lower sample quality compared to MCMC counterparts.
Revised: On the other hand, VI allows for approximate likelihood evaluation under the variational posterior and fast sampling, but at a lower sample quality compared to MCMC counterparts.

Original: We propose a novel method that leverages a powerful pre-trained ﬂow model by constructing carefully designed latent codes to generate conditional samples via variational inference.
Revised: We propose a novel method that leverages a powerful pre-trained ﬂow model by constructing carefully designed latent codes to generate conditional samples via variational inference.

Original: While this procedure is intractable for latent variable models in general, the invertibility of the pre-trained model gives us a tractable algorithm for learning a distribution in the latent space whose samples approximately match the true conditional when fed into the pre-trained model.
Revised: While this procedure is intractable for latent variable models in general, the invertibility of the pre-trained model gives us a tractable algorithm for learning a distribution in the latent space whose samples approximately match the true conditional when fed into the pre-trained model.

Original: Our contributions:
Revised: Our contributions:

Original: • We start with an interesting theoretical hardness result.
Revised: • We start with an interesting theoretical hardness result.

Original: We show that even though ﬂow models are designed to provide efﬁcient inversion and sampling, even approximate sampling from the exact conditional distribution is provably computationally intractable for a wide class of ﬂow models.
Revised: We show that even though ﬂow models are designed to provide efﬁcient inversion and sampling, even approximate sampling from the exact conditional distribution is provably computationally intractable for a wide class of ﬂow models.

Original: This motivates our approach of smoothing the observation.
Revised: This motivates our approach of smoothing the observation.

Original: • We develop a method to estimate the target conditional distribution by composing a second generative model (which we call the pre-generator ) with the given model.
Revised: • We develop a method to estimate the target conditional distribution by composing a second ﬂow model (which we call the pre-generator ) with the given model.

Original: Speciﬁcally, we show how to train the pre-generator to yield structured noise so that the composed model approximately matches the target conditional distribution.
Revised: In particular, this parametrization allows us to employ variational inference and avoid unstable adversarial training as was explored in existing work. • The resulting approximate posterior retains the computational ﬂexibility of a ﬂow model and can be used on a wide variety of downstream tasks that require fast sampling, exact likelihood evaluation, or inversion. Compared to MCMC methods, it has the beneﬁt of being able to generate samples that are guaranteed to be i.i.d.

Original: • We experimentally show that our approach is competitive with MCMC methods in terms ofspeed and various sample quality metrics such as Frechet Inception Distance (Heusel et al., 2017).
Revised: • We experimentally show that our approach is comparable to simple MCMC baselines in terms of sample quality metrics such as Frechet Inception Distance (Heusel et al., 2017).

Original: We also demonstrate that it achieves superior conditional likelihood estimation performance compared to regular variational inference.
Revised: We also demonstrate that it achieves superior conditional likelihood estimation performance compared to regular variational inference.

Original: • We extend and validate our method for conditioning under arbitrary differentiable transformations with applications to inverse problems.
Revised: • We extend and validate our method for conditioning under arbitrary differentiable transformations with applications to inverse problems.

Original: We qualitatively demonstrate its ﬂexibility on various complex inference tasks.
Revised: We qualitatively demonstrate its ﬂexibility on various complex inference tasks.

Original:  B ACKGROUND 2. N ORMALIZING F LOWS
Revised:  B ACKGROUND 2. N ORMALIZING F LOWS

Original: Normalizing ﬂow models (also known as invertible generative models) represent complex probability distributions by transforming a simple input noise z (typically standard Gaussian) through a differentiable bijection f : R d → R d .
Revised: Normalizing ﬂow models (also known as invertible generative models) represent complex probability distributions by transforming a simple input noise z (typically standard Gaussian) through a differentiable bijection f : R d → R d .

Original: Since f is invertible, change of variables formula allows us to compute the probability density of x = f ( z ) :
Revised: Since f is invertible, change of variables formula allows us to compute the probability density of x = f ( z ) :

Original: [Equation]
Revised: [Equation]

Original: where df − 1 d x denotes the Jacobian of the inverse transformation f − 1 : x (cid:55)→ z .
Revised: where df − 1 d x denotes the Jacobian of the inverse transformation f − 1 : x (cid:55)→ z .

Original: Flow models are explicitly designed so that the above expression can be easily computed, including the log-determinant term.
Revised: Flow models are explicitly designed so that the above expression can be easily computed, including the log-determinant term.

Original: This tractability allows them to be directly trained with maximum likelihood objective on data.
Revised: This tractability allows them to be directly trained with maximum likelihood objective on data.

Original: Starting from the early works of Dinh et al.
Revised: Starting from the early works of Dinh et al.

Original: and Rezende & Mohamed (2015), there has been extensive research on invertible architectures for generative modeling.
Revised: and Rezende & Mohamed (2015), there has been extensive research on invertible architectures for generative modeling.

Original: Many of them work by composing a series of invertible layers, such as in RealNVP (Dinh et al., 2016), IAF (Kingma et al., 2016), Glow (Kingma & Dhariwal, 2018), invertible ResNet (Behrmann et al., 2019), and Neural
Revised: Many of them work by composing a series of invertible layers, such as in RealNVP (Dinh et al., 2016), IAF (Kingma et al., 2016), Glow (Kingma & Dhariwal, 2018), invertible ResNet (Behrmann et al., 2019), and Neural

Original: Spline Flows (Durkan et al., 2019).
Revised: Spline Flows (Durkan et al., 2019).

Original: One of the simplest invertible layer construction is additive coupling layer introduced by Dinh et al.
Revised: One of the simplest invertible layer construction is additive coupling layer introduced by Dinh et al.

Original: (2015), which served as a basis for many other subsequently proposed models mentioned above.
Revised: (2015), which served as a basis for many other subsequently proposed models mentioned above.

Original: In an additive coupling layer, the input variable is partitioned as x = ( x 1 , x 2 ) ∈ R d 1 × R d 2 .
Revised: In an additive coupling layer, the input variable is partitioned as x = ( x 1 , x 2 ) ∈ R d 1 × R d 2 .

Original: The layer is parametrized by a neural network g ( x 1 ) :
Revised: The layer is parametrized by a neural network g ( x 1 ) :

Original: R d 1 → R d 2 used to additively transform x 2 .
Revised: R d 1 → R d 2 used to additively transform x 2 .

Original: Thus the layer’s output y
Revised: Thus the layer’s output y

Original: =
Revised: =

Original: ( y 1 , y 2 ) ∈ R
Revised: ( y 1 , y 2 ) ∈ R

Original: d 1 × R d 2 and its inverse can be computed as follows:
Revised: d 1 × R d 2 and its inverse can be computed as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original:  Notably, the determinant of the Jacobian of this transformation is always 1 for any mapping g .
Revised: z f x Notably, the determinant of the Jacobian of this transformation is always 1 for any mapping g .

Original: 2. V ARIATIONAL I NFERENCE
Revised: 2. V ARIATIONAL I NFERENCE

Original: Variational inference (VI) is a family of techniques for estimating the conditional distribution of unobserved variables given observed ones (Jordan et al., 1999;
Revised: Variational inference (VI) is a family of techniques for estimating the conditional distribution of unobserved variables given observed ones (Jordan et al., 1999;

Original: Wainwright et al., 2008; Blei et al., 2017).
Revised: Wainwright et al., 2008; Blei et al., 2017).

Original: At its core, VI tries to ﬁnd a tractable approximation of the true conditional density by minimizing the KL divergence between them over a set of distributions called the variational family.[Figure]
Revised: At its core, VI tries to ﬁnd a tractable approximation of the true conditional density by minimizing the KL divergence between them over a set of distributions called the variational family.

Original: f x Within our context of probabilistic inference from a joint distribution p ( x 1 , x 2 ) , the KL minimization problem we solve is the following:
Revised: Within our context of probabilistic inference from a joint distribution p ( x 1 , x 2 ) , the KL minimization problem we solve is the following:

Original: [Equation]
Revised: [Equation]

Original: where x ∗ 1 is some ﬁxed observation and the variational family Q is chosen such that efﬁcient sampling and likelihood evaluation are possible for all q ∈ Q .
Revised: where x ∗ 1 is some ﬁxed observation and the variational family Q is chosen such that efﬁcient sampling and likelihood evaluation are possible for all q ∈ Q .

Original: Note that q is tied to a particular x ∗ 1 and is not shared across different observations.
Revised: Note that q is tied to a particular x ∗ 1 and is not shared across different observations.

Original: While the conditional density p ( x 2 | x 1 = x ∗ 1 ) is generally intractable to compute, we can efﬁciently evaluate the joint density p ( x 1 , x 2 ) when it is given as a ﬂow model.
Revised: While the conditional density p ( x 2 | x 1 = x ∗ 1 ) is generally intractable to compute, we can efﬁciently evaluate the joint density p ( x 1 , x 2 ) when it is given as a ﬂow model.

Original: Thus, we instead perform variational inference on the conditional joint distribution:
Revised: Thus, we instead perform variational inference on the conditional joint distribution:

Original: [Equation]
Revised: [Equation]

Original:  which is obtained by dropping the log p ( x 1 = x ∗ 1 ) term that is constant w.r.t.
Revised: = x ∗ 1 )) = min q ∈Q E x 2 ∼ q [log q ( x 2 ) − log p ( x 1 = x ∗ 1 , x 2 )] , which is obtained by dropping the log p ( x 1 = x ∗ 1 ) term that is constant w.r.t.

Original: q .
Revised: q .

Original: We note that this procedure is intractable for other types of latent variable models such as VAE or GAN.
Revised: We note that this procedure is intractable for other types of latent variable models such as VAE or GAN.

Original:  H ARDNESS OF C ONDITIONAL S AMPLING ON F LOWS
Revised:  H ARDNESS OF C ONDITIONAL S AMPLING ON F LOWS

Original: Before we present our method, we ﬁrst establish a hardness result for exact conditional sampling for ﬂow models that use additive coupling layer.
Revised: Before we present our method, we ﬁrst establish a hardness result for exact conditional sampling for ﬂow models that use additive coupling layer.

Original: Speciﬁcally, if an algorithm is able to efﬁciently sample from the conditional distribution of a ﬂow model with additive coupling layers, then this algorithm can be used to solve NP-complete problems efﬁciently.
Revised: Speciﬁcally, if an algorithm is able to efﬁciently sample from the conditional distribution of a ﬂow model with additive coupling layers, then this algorithm can be used to solve NP-complete problems efﬁciently.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Informal)
Revised: (Informal)

Original: Suppose there is an efﬁcient algorithm that can draw samples from the conditional distribution p ( x 2 | x 1 ) of a normalizing ﬂow model p ( x 1 , x 2 ) implemented with additive coupling layers as deﬁned in Dinh et al. (2015).
Revised: Suppose there is an efﬁcient algorithm that can draw samples from the conditional distribution p ( x 2 | x 1 ) of a normalizing ﬂow model p ( x 1 , x 2 ) implemented with additive coupling layers as deﬁned in Dinh et al. (2015).

Original: Then RP = NP .
Revised: Then RP = NP .

Original: Moreover, even approximating the true conditional distribution remains hard as long as we require that the conditioning is exact.
Revised: Moreover, even approximating the true conditional distribution remains hard as long as we require that the conditioning is exact.

Original: The formal statement of the above theorem and its corollary to the approximate case can be found in Appendix A.
Revised: The formal statement of the above theorem and its corollary to the approximate case can be found in Appendix A.

Original: Importantly, this result implies that allowing approximate sampling does not affect the hardness of the problem, as long as the hard constraint of exact conditioning ( x 1 = x ∗ 1 ) is there.
Revised: Importantly, this result implies that allowing approximate sampling does not affect the hardness of the problem, as long as the hard constraint of exact conditioning ( x 1 = x ∗ 1 ) is there.

Original: Thus we are motivated to instead consider approximate conditioning where the conditioned variable x 1 is not required to match the given observation exactly.
Revised: Thus we are motivated to instead consider approximate conditioning where the conditioned variable x 1 is not required to match the given observation exactly.

Original: We also note that the class of ﬂow models that include additive coupling layers encompasses a large number of existing models (e.g. most of the models in Section 2.1).
Revised: We also note that the class of ﬂow models that include additive coupling layers encompasses a large number of existing models (e.g. most of the models in Section 2.1).

Original: Thus our hardness result applies to a variety of ﬂow models used in practice.
Revised: Thus our hardness result applies to a variety of ﬂow models used in practice.

Original:  A PPROXIMATE P ROBABILISTIC I NFERENCE WITH C OMPOSED F LOWS
Revised:  A PPROXIMATE P ROBABILISTIC I NFERENCE WITH C OMPOSED F LOWS

Original: Notation.
Revised: Notation.

Original: Let p f ( x ) denote the base model , a ﬁxed ﬂow model deﬁned by the invertible mapping f : z (cid:55)→ x .
Revised: Let p f ( x ) denote the base model , a ﬁxed ﬂow model deﬁned by the invertible mapping f : z (cid:55)→ x .

Original: The pre-generator p ˆ f ( z ) is similarly deﬁned by the invertible mapping ˆ f : (cid:15) (cid:55)→ z and represents a distribution in the latent space of the base model.
Revised: The pre-generator p ˆ f ( z ) is similarly deﬁned by the invertible mapping ˆ f : (cid:15) (cid:55)→ z and represents a distribution in the latent space of the base model.

Original: We assume that all ﬂow models use standard Gaussian prior, i.e. z ∼ N ( 0 , I d ) → x = f ( z ) and (cid:15) ∼ N ( 0 , I d ) →
Revised: We assume that all ﬂow models use standard Gaussian prior, i.e. z ∼ N ( 0 , I d ) → x = f ( z ) and (cid:15) ∼ N ( 0 , I d ) →

Original: z = ˆ f ( (cid:15) ) .
Revised: z = ˆ f ( (cid:15) ) .

Original: By composing the base model and the pre-generator, we obtain the composed model p f ◦ ˆ f ( x ) whose samples are generated via (cid:15) ∼ N ( 0 , I d ) → x = f ( ˆ f ( (cid:15) )) .
Revised: By composing the base model and the pre-generator, we obtain the composed model p f ◦ ˆ f ( x ) whose samples are generated via (cid:15) ∼ N ( 0 , I d ) → x = f ( ˆ f ( (cid:15) )) .

Original: Figure 1 details this sampling procedure.
Revised: Figure 1 details this sampling procedure.

Revised: [Figure]

Original: Our method.
Revised: Our method.

Original: Following the principle of leveraging the favorable geometry of the latent space commonly employed in VI and MCMC literature, we also perform our inference in the latent space of the base model.
Revised: Following the principle of leveraging the favorable geometry of the latent space commonly employed in VI and MCMC literature, we also perform our inference in the latent space of the base model.

Original: Because the base model maps Gaussian noise to samples, the pre-generator can simply focus on putting probability mass in the regions of the latent space corresponding to the conditional samples we wish to model.
Revised: Because the base model maps Gaussian noise to samples, the pre-generator can simply focus on putting probability mass in the regions of the latent space corresponding to the conditional samples we wish to model.

Original: This is in contrast with regular variational inference, which essentially trains an entirely new generative model from scratch to approximate the target conditional density.[Figure] [Equation] [Figure]
Revised: This is in contrast with regular variational inference, which essentially trains an entirely new generative model from scratch to approximate the target conditional density.

Original: [Equation]

Original: [Figure] [Equation] [Figure] Motivated by this observation and the hardness of conditional sampling, we propose to perform variational inference in the latent space on a smoothed version of the problem with Gaussian smoothing.
Revised: Motivated by this observation and the hardness of conditional sampling, we propose to perform variational inference in the latent space on a smoothed version of the problem with Gaussian smoothing.

Original: Speciﬁcally, we create a smoothed variable ˜ x 1 distributed according to p σ (˜ x 1 | x 1 ) =
Revised: Speciﬁcally, we create a smoothed variable ˜ x 1 distributed according to p σ (˜ x 1 | x 1 ) =

Original: N (˜ x 1 ; x 1 , σ 2 I d ) and condition on ˜ x 1 = x ∗ 1 instead.
Revised: N (˜ x 1 ; x 1 , σ 2 I d ) and condition on ˜ x 1 = x ∗ 1 instead.

Original: Here, σ is the smoothing parameter that controls the tightness of this relaxation.
Revised: Here, σ is the smoothing parameter that controls the tightness of this relaxation.

Original: We have experimented with non-Gaussian smoothing kernels such as the one implicitly deﬁned by the perceptual distance measure LPIPS (Zhang et al., 2018).
Revised: We have experimented with non-Gaussian smoothing kernels such as the one implicitly deﬁned by the perceptual distance measure LPIPS (Zhang et al., 2018).

Original: While we chose to use Gaussian smoothing because our preliminary results showed no appreciable difference, we note that a more extensive tuning of the smoothing kernel may lead to a better sample quality, which we leave for future work.
Revised: While we chose to use Gaussian smoothing because our preliminary results showed no appreciable difference, we note that a more extensive tuning of the smoothing kernel may lead to a better sample quality, which we leave for future work.

Original: The objective we  minimize is the KL divergence between our variational distribution and the smoothed conditional density:
Revised: Thus the objective we would like to minimize is the KL divergence between our variational distribution and the smoothed conditional density:

Original: [Equation]
Revised: [Equation]

Original: For comparison, a direct application of variational inference in the image space (which we refer to as
Revised: For comparison, a direct application of variational inference in the image space (which we refer to as

Original: Ambient VI ) would minimize the following objective:
Revised: Ambient VI ) would minimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: where we write p g to denote the variational distribution that directly models f ( x | ˜
Revised: where we write p g to denote the variational distribution that directly models f ( x | ˜

Original: y 
Revised: y = y ∗ ) in the ambient (image) space, parametrized by an invertible mapping g : z (cid:55)→ x .

Original: [Equation]
Revised: Intractability of equation 1.

Original: y
Revised: Note that in our setting, we cannot directly approximate p f ( x 2 | ˜ x 1 = x ∗ 1 ) because we only have access to the joint distribution p f ( x 1 , x 2 ) through the base model. Fortunately, the VI loss for approximating the joint conditional p f ( x | ˜ x 1 = x ∗ 1 ) is an upper bound to eq.

Original: [Equation]
Revised: [Equation]

Original: ) in the ambient (image) space, parametrized by an invertible mapping g : z (cid:55)→ x . Note that in our setting, we cannot directly model p f ( x 2 | ˜ x 1 = x ∗ 1 ) because we only have access to the joint distribution p f ( x 1 , x 2 ) through the base model.
Revised: which we prove in Appendix B.2.

Original: Thus we instead approximate the joint distribution conditioned on our observation, i.e. p f ( x | ˜ x 1 = x ∗ 1 ) , using the variational posterior p f ◦ ˆ f ( x ) .
Revised: Thus we are justiﬁed in our use of the joint VI loss instead of eq. Moreover, this concern vanishes for the general case where we observe a transformation of x , as discussed below.

Original: Figures 2a and 2b show the graphical models describing this formulation.
Revised: Figures 2a and 2b show the graphical models reﬂecting this formulation.

Original: Conditioning under differentiable transformations.
Revised: Conditioning under differentiable transformations.

Original: The ﬂexibility of VI allows us to easily extend our method to conditioning under an arbitrary transformation.
Revised: The ﬂexibility of VI allows us to easily extend our method to conditioning under an arbitrary transformation.

Original: Concretely, let T ( x ) be a differentiable function and y ∗ be some ﬁxed observation in the range of T .
Revised: Concretely, let T ( x ) be a differentiable function and y ∗ be some ﬁxed observation in the range of T .

Original: We now observe y = T ( x ) instead, so we similarly deﬁne a smoothed variable ˜
Revised: We now observe y = T ( x ) instead, so we similarly deﬁne a smoothed variable ˜

Original: y distributed according to p σ (˜ y | y )
Revised: y distributed according to p σ (˜ y | y )

Original: =
Revised: =

Original: N (˜ y ; y , σ 2 I d ) .
Revised: N (˜ y ; y , σ 2 I d ) .

Original: We estimate the conditional density p f ( x | ˜ y
Revised: We estimate the conditional density p f ( x | ˜ y

Original: = y ∗ ) by minimizing the following objective:
Revised: = y ∗ ) by minimizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where p f ( z ) denotes the prior distribution of the base model, i.e. N ( 0 , I d ) .
Revised: where p f ( z ) denotes the prior distribution of the base model, i.e. N ( 0 , I d ) .

Original: We provide the derivation of eq.
Revised: We provide the derivation of eq.

Original: (3) in Appendix B. See Figure 2c for a comparison to Equation (1).
Revised: (3) in Appendix B.1. See Figure 2c for a comparison to Equation (1).

Original: This loss function offers an intuitive interpretation.
Revised: This loss function offers an intuitive interpretation.

Original: The ﬁrst term tries to keep the learned distribution p f ◦ ˆ f close to the base distribution by pushing p ˆ f to match the prior of the base model, while the second term tries to match the observation y ∗ .
Revised: The ﬁrst term tries to keep the learned distribution p f ◦ ˆ f close to the base distribution by pushing p ˆ f to match the prior of the base model, while the second term tries to match the observation y ∗ .

Original: Moreover, the above expectation can be rewritten in terms of (cid:15) , which allows us to employ the reparametrization trick to obtain a low-variance gradient estimator for training:
Revised: Moreover, the above expectation can be rewritten in terms of (cid:15) , which allows us to employ the reparametrization trick to obtain a low-variance gradient estimator for training:

Original: [Equation]
Revised: [Equation]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Conditional Generative Models.
Revised: Conditional Generative Models.

Original: There has been a large amount of work on conditional generative modeling, with varying levels of ﬂexibility for what can be conditioned on.
Revised: There has been a large amount of work on conditional generative modeling, with varying levels of ﬂexibility for what can be conditioned on.

Original: In the simplest case, a ﬁxed set of observed variables can be directly fed into the model as an auxiliary conditioning input (Mirza & Osindero, 2014; Sohn et al., 2015; Ardizzone et al., 2019).
Revised: In the simplest case, a ﬁxed set of observed variables can be directly fed into the model as an auxiliary conditioning input (Mirza & Osindero, 2014; Sohn et al., 2015; Ardizzone et al., 2019).

Original: Some recent works proposed to extend existing models to support conditioning on arbitrary subsets of variables (Ivanov et al., 2018;
Revised: Some recent works proposed to extend existing models to support conditioning on arbitrary subsets of variables (Ivanov et al., 2018;

Original: Belghazi et al., 2019; Li et al., 2019).
Revised: Belghazi et al., 2019; Li et al., 2019).

Original: This is a much harder task as there are exponentially many subsets of variables that can be conditioned on.
Revised: This is a much harder task as there are exponentially many subsets of variables that can be conditioned on.

Original: More relevant to our setting is (Engel et al., 2017), which studied conditional sampling from noninvertible latent variable generators such as VAE and GAN.
Revised: More relevant to our setting is (Engel et al., 2017), which studied conditional sampling from noninvertible latent variable generators such as VAE and GAN.

Original: It proposes to adversarially train a
Revised: It proposes to adversarially train a

Original: GAN-style generator within the latent space of a pre-trained latent variable model, thereby avoiding the issue of intractability of variational inference for non-invertible models.
Revised: GAN-style generator within the latent space of a pre-trained latent variable model, thereby avoiding the issue of intractability of variational inference for non-invertible models.

Original: Due to adversarial training and the lack of invertibility of the base model, however, the learned conditional sampler loses the computational ﬂexibility of our method.
Revised: Due to adversarial training and the lack of invertibility of the base model, however, the learned conditional sampler loses the computational ﬂexibility of our method.

Original: We highlight several reasons why one might prefer our approach over the above methods: (1) The data used to train the given model may not be available, and only the generative model itself is made public.
Revised: We highlight several reasons why one might prefer our approach over the above methods: (1) The data used to train the given model may not be available, and only the generative model itself is made public.

Original: The given model is too costly to train from scratch.
Revised: The given model is too costly to train from scratch.

Original: (3) We wish to perform custom downstream tasks (e.g. compression, inversion) that are difﬁcult to do with other parametrizations of the posterior (4) We need to condition on a transformation of the variables, instead of just a subset of them.
Revised: (3) We wish to perform custom downstream tasks (e.g. compression, inversion) that are difﬁcult to do with other parametrizations of the posterior (4) We need to condition on a transformation of the variables, instead of just a subset of them.

Original: (5) We want to get some insight on the distribution deﬁned by the given model.
Revised: (5) We want to get some insight on the distribution deﬁned by the given model.

Original: Markov Chain Monte Carlo Methods.
Revised: Markov Chain Monte Carlo Methods.

Original: When one is only concerned with generating samples,
Revised: When one is only concerned with generating samples,

Original: MCMC techniques offer a promising alternative.
Revised: MCMC techniques offer a promising alternative.

Original: Unlike variational inference and conditional models,
Revised: Unlike variational inference and conditional models,

Original: MCMC methods come with asymptotic guarantees to generate correct samples.
Revised: MCMC methods come with asymptotic guarantees to generate correct samples.

Original: Though directly applying MCMC methods on complex high-dimensional posteriors parametrized by a neural network often comes with many challenges in practice (Papamarkou et al., 2019), methods based on Langevin
Revised: Though directly applying MCMC methods on complex high-dimensional posteriors parametrized by a neural network often comes with many challenges in practice (Papamarkou et al., 2019), methods based on Langevin

Original: Monte Carlo have shown promising results (Neal et al., 2011;
Revised: Monte Carlo have shown promising results (Neal et al., 2011;

Original: Welling & Teh, 2011; Song & Ermon, 2019).
Revised: Welling & Teh, 2011; Song & Ermon, 2019).

Original: Moreover, recent works by Parno & Marzouk (2018) and Hoffman et al.
Revised: Moreover, recent works by Parno & Marzouk (2018) and Hoffman et al.

Original: (2019) also leveraged the favorable geometry of the latent space of a ﬂow model to improve mixing of MCMC chains.
Revised: (2019) also leveraged the favorable geometry of the latent space of a ﬂow model to improve mixing of MCMC chains.

Original: This idea was later adapted by Nijkamp et al.
Revised: This idea was later adapted by Nijkamp et al.

Original: (2020) in the context of training energy-based models.
Revised: (2020) in the context of training energy-based models.

Original: While our VI-based method lacks asymptotic guarantees, we hope to show that it is empirically competitive with simple MCMC techniques.
Revised: While our VI-based method lacks asymptotic guarantees, we hope to show that it is empirically competitive with simple MCMC techniques.

Original: For this purpose, we consider two baselines: Langevin dynamics and PL-MCMC (Cannella et al., 2020).
Revised: For this purpose, we consider two baselines: Langevin dynamics and PL-MCMC (Cannella et al., 2020).

Original: PL-MCMC is particularly relevant as it also tackles the task of conditional sampling from a ﬂow model.
Revised: PL-MCMC is particularly relevant as it also tackles the task of conditional sampling from a ﬂow model.

Original: Unlike our method, it constructs a Markov chain in the latent space and uses Metropolis-Hastings to sample from the target conditional distribution with asymptotic guarantees.
Revised: Unlike our method, it constructs a Markov chain in the latent space and uses Metropolis-Hastings to sample from the target conditional distribution with asymptotic guarantees.

Original: Inverse Problems with Generative Prior.
Revised: Inverse Problems with Generative Prior.

Original: In a linear inverse problem, a vector x ∈ R d generates a set of measurements y ∗ =
Revised: In a linear inverse problem, a vector x ∈ R d generates a set of measurements y ∗ =

Original: A x ∈ R m , where the number of measurements is much smaller than the dimension: m (cid:28) d .
Revised: A x ∈ R m , where the number of measurements is much smaller than the dimension: m (cid:28) d .

Original: The goal is to reconstruct the vector x from y ∗ .
Revised: The goal is to reconstruct the vector x from y ∗ .

Original: While in general there are (potentially inﬁnitely) many possible values of x that agree with the given measurements, it is possible to identify a unique solution when there is an additional structural assumption on x .
Revised: While in general there are (potentially inﬁnitely) many possible values of x that agree with the given measurements, it is possible to identify a unique solution when there is an additional structural assumption on x .

Original: Classically, the simplifying structure was that x is sparse, and there has been extensive work in this setting (Tibshirani, 1996; Candes et al., 2006;
Revised: Classically, the simplifying structure was that x is sparse, and there has been extensive work in this setting (Tibshirani, 1996; Candes et al., 2006;

Original: Donoho et al., 2006;
Revised: Donoho et al., 2006;

Original: Bickel et al., 2009; Baraniuk, 2007).
Revised: Bickel et al., 2009; Baraniuk, 2007).

Original: Recent work has considered alternative structures, such as the vector x coming from a generative model.
Revised: Recent work has considered alternative structures, such as the vector x coming from a generative model.

Original: Starting with Bora et al.
Revised: Starting with Bora et al.

Original: (2017), there has been extensive work in this setting as well (Grover & Ermon, 2019; Mardani et al., 2018; Heckel & Hand, 2019; Mixon & Villar, 2018;
Revised: (2017), there has been extensive work in this setting as well (Grover & Ermon, 2019; Mardani et al., 2018; Heckel & Hand, 2019; Mixon & Villar, 2018;

Original: Pandit et al., 2019).
Revised: Pandit et al., 2019).

Original: In particular, Asim et al.
Revised: In particular, Asim et al.

Original: (2019) and Shamshad et al.
Revised: (2019) and Shamshad et al.

Original: (2019) utilized ﬂow models to solve inverse problems such as compressive sensing, image deblurring, and image inpainting.
Revised: (2019) utilized ﬂow models to solve inverse problems such as compressive sensing, image deblurring, and image inpainting.

Original: It is important to note that the above approaches focus on recovering a single point estimate that best matches the measurements.
Revised: It is important to note that the above approaches focus on recovering a single point estimate that best matches the measurements.

Original: However, there can be many inputs that ﬁt the measurements and thus uncertainty in the reconstruction.
Revised: However, there can be many inputs that ﬁt the measurements and thus uncertainty in the reconstruction.

Original: Due to this shortcoming, several recent works focused on recovering the distribution of x conditioned on the measurements (Tonolini et al., 2019; Zhang &
Revised: Due to this shortcoming, several recent works focused on recovering the distribution of x conditioned on the measurements (Tonolini et al., 2019; Zhang &

Original: Jin, 2019; Adler & Öktem, 2018; 2019).
Revised: Jin, 2019; Adler & Öktem, 2018; 2019).

Original: We note that our approach differs from these, since they are learning-based methods that require access to the training data.
Revised: We note that our approach differs from these, since they are learning-based methods that require access to the training data.

Original: On the contrary, our work attempts to perform conditional sampling using a given pre-trained generative model, leveraging all that previous computation to solve a conditional task with reconstruction diversity.
Revised: On the contrary, our work attempts to perform conditional sampling using a given pre-trained generative model, leveraging all that previous computation to solve a conditional task with reconstruction diversity.

Original:  Q UANTITATIVE E XPERIMENTS
Revised:  Q UANTITATIVE E XPERIMENTS

Original: We validate the efﬁcacy of our proposed method in terms of both sample quality and likelihood on various inference tasks against three baselines: Ambient VI (as deﬁned by the loss in Equation (2)),
Revised: We validate the efﬁcacy of our proposed method in terms of both sample quality and likelihood on various inference tasks against three baselines: Ambient VI (as deﬁned by the loss in Equation (2)),

Original: Langevin Dynamics, and PL-MCMC.
Revised: Langevin Dynamics, and PL-MCMC.

Original: We also conduct our experiments on three different datasets (MNIST, CIFAR-10, and CelebA-HQ) to ensure that our method works across a range of settings.
Revised: We also conduct our experiments on three different datasets (MNIST, CIFAR-10, and CelebA-HQ) to ensure that our method works across a range of settings.

Original: We report four different sample quality metrics: Frechet Inception Distance (FID), Learned Perceptual
Revised: We report four different sample quality metrics: Frechet Inception Distance (FID), Learned Perceptual

Original: Image Patch Similarity (LPIPS), and Inception Score (IS) for CIFAR-10 (Heusel et al., 2017; Zhang et al., 2018; Salimans et al., 2016).
Revised: Image Patch Similarity (LPIPS), and Inception Score (IS) for CIFAR-10 (Heusel et al., 2017; Zhang et al., 2018; Salimans et al., 2016).

Original: While not strictly a measure of perceptual similarity, the average mean squared error (MSE) is also reported for completeness.
Revised: While not strictly a measure of perceptual similarity, the average mean squared error (MSE) is also reported for completeness.

Original: For all our experiments, we use the multiscale RealNVP architecture (Dinh et al., 2016) for both the base model and the pre-generator.
Revised: For all our experiments, we use the multiscale RealNVP architecture (Dinh et al., 2016) for both the base model and the pre-generator.

Original: We use Adam optimizer (Kingma & Ba, 2014) to optimize the weights of the pre-generator using the loss deﬁned in Equation (3).
Revised: We use Adam optimizer (Kingma & Ba, 2014) to optimize the weights of the pre-generator using the loss deﬁned in Equation (3).

Original: The images used to generate observations were taken from the test set and were not used to train the base models.
Revised: The images used to generate observations were taken from the test set and were not used to train the base models.

Original: We refer the reader to Appendix C for model hyperparameters and other details of our experiment setup.
Revised: We refer the reader to Appendix C for model hyperparameters and other details of our experiment setup.

Original: [Figure]
Revised: [Figure]

Original: 6. I MAGE I NPAINTING
Revised: 6. I MAGE I NPAINTING

Original: We perform inpainting tasks using our approach, where we sample missing pixels conditioned on the visible ones.
Revised: We perform inpainting tasks using our approach, where we sample missing pixels conditioned on the visible ones.

Original: We consider three different conditioning schemes: the bottom half (MNIST), the upper half (CelebA-HQ), and randomly chosen subpixels (CIFAR-10).
Revised: We consider three different conditioning schemes: the bottom half (MNIST), the upper half (CelebA-HQ), and randomly chosen subpixels (CIFAR-10).

Original: For MNIST, we use the smoothing parameter value of σ = 0 .
Revised: For MNIST, we use the smoothing parameter value of σ = 0 .

Original: 1 and for CIFAR-10 and CelebA-HQ, we use σ = 0 .
Revised: 1 and for CIFAR-10 and CelebA-HQ, we use σ = 0 .

Original: In Figure 3 we see that our approach produces natural and diverse samples for the missing part of the image.
Revised: In Figure 3 we see that our approach produces natural and diverse samples for the missing part of the image.

Original: The empirical pixelwise variance (normalized and averaged over the color channels) also conﬁrms that, while the observation is not perfectly matched, most of the high-variance regions are in the unobserved parts as we expect.
Revised: The empirical pixelwise variance (normalized and averaged over the color channels) also conﬁrms that, while the observation is not perfectly matched, most of the high-variance regions are in the unobserved parts as we expect.

Original: We also quantitatively evaluate the quality of the generated samples using widely used sample quality metrics, as shown in Table 1.
Revised: We also quantitatively evaluate the quality of the generated samples using widely used sample quality metrics, as shown in Table 1.

Original: As we can see, our method outperforms the baseline methods on most of the metrics.
Revised: As we can see, our method outperforms the baseline methods on most of the metrics.

Original: Note that PL-MCMC results for CIFAR-10 and CelebA-HQ are omitted because it was prohibitively slow to run for hundreds of images, as each MCMC chain required over 20, proposals.
Revised: Note that PL-MCMC results for CIFAR-10 and CelebA-HQ are omitted because it was prohibitively slow to run for hundreds of images, as each MCMC chain required over 20, proposals.

Original: Cannella et al.
Revised: Cannella et al.

Original: (2020) also report using 25,000 proposals for their experiments. 
Revised: (2020) also report using 25,000 proposals for their experiments. [Table]

Original: 6. L IKELIHOOD E STIMATION
Revised: L IKELIHOOD E STIMATION

Original: Next, we evaluate our method on the task of conditional likelihood estimation.
Revised: Next, we evaluate our method on the task of conditional likelihood estimation.

Original: By varying the size of the pre-generator, we also test the parameter efﬁciency of our method in comparison to Ambient VI.
Revised: By varying the size of the pre-generator, we also test the parameter efﬁciency of our method in comparison to Ambient VI.

Original: Results are shown in Table 2; we see that our method is able to produce reasonable samples using

Original: [Table]

Original: [Figure] [Table] [Figure] only about 1% of the base model’s parameters, conﬁrming the effectiveness of inference in the latent space.
Revised: Results are shown in Table 2; we see that our method is able to produce reasonable samples using only about 1% of the base model’s parameters, conﬁrming the effectiveness of inference in the latent space.

Original: [Table]
Revised: [Table]

Original:  Q UALITATIVE E XPERIMENTS
Revised:  Q UALITATIVE E XPERIMENTS

Original: Extracting Class-conditional Models:
Revised: Extracting Class-conditional Models:

Original: Here we present an interesting application of conditioning under a complex transformation T .
Revised: Here we present an interesting application of conditioning under a complex transformation T .

Original: Since the only requirement for T is that it must be differentiable, we can set it to be another neural network.
Revised: Since the only requirement for T is that it must be differentiable, we can set it to be another neural network.

Original: For example, if T is a pre-trained binary classiﬁer for a speciﬁc attribute, we can extract a model conditioned on the presence (or the absence) of that attribute from an unconditional base model.
Revised: For example, if T is a pre-trained binary classiﬁer for a speciﬁc attribute, we can extract a model conditioned on the presence (or the absence) of that attribute from an unconditional base model.

Original: We test this idea on the MNIST dataset.
Revised: We test this idea on the MNIST dataset.

Original: We trained 10 binary classiﬁers on the MNIST dataset, one for each digit k = 0 , . . .
Revised: We trained 10 binary classiﬁers on the MNIST dataset, one for each digit k = 0 , . . .

Original: , 9 , to predict whether the given image is k or not.
Revised: , 9 , to predict whether the given image is k or not.

Original: By setting T to be each of those classiﬁers, we were able to extract the class-conditional model p
Revised: By setting T to be each of those classiﬁers, we were able to extract the class-conditional model p

Original: f
Revised: f

Original: ( x | Label ( x ) ≈ k ) .
Revised: ( x | Label ( x ) ≈ k ) .

Original: See
Revised: See

Original: Figure 4a for samples generated from the extracted models.
Revised: Figure 4a for samples generated from the extracted models.

Original: Inverse Problems: We additionally test the applicability of our method to linear inverse problems.
Revised: Inverse Problems: We additionally test the applicability of our method to linear inverse problems.

Original: In Figure 5, we show the conditional samples obtained by our method on three different tasks: image colorization, super-resolution ( 2 × ), and compressed sensing with 500 random Gaussian measurements (for reference, CIFAR-10 images have 3072 dimensions).
Revised: In Figure 5, we show the conditional samples obtained by our method on three different tasks: image colorization, super-resolution ( 2 × ), and compressed sensing with 500 random Gaussian measurements (for reference, CIFAR-10 images have 3072 dimensions).

Original: We notice that the generated samples look natural, even when they do not match the original input perfectly, again showing our method’s capability to generate semantically meaningful conditional samples and also provide sample diversity.
Revised: We notice that the generated samples look natural, even when they do not match the original input perfectly, again showing our method’s capability to generate semantically meaningful conditional samples and also provide sample diversity.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Figure]

Original:  C o m p [Equation]
Revised: S R ( 2x ) C o l o r i z a t

Original: e d
Revised: i o n

Original: [Equation]
Revised: C o m p re ss e d

Original:  e n s i n g
Revised: S e n s i n g

Original: 7. W HY A MBIENT VI F AILS
Revised: 7. W HY A MBIENT VI F AILS

Original: From Table 1, notice that Ambient VI achieves signiﬁcantly worse sample quality compared to other methods.
Revised: From Table 1, notice that Ambient VI achieves signiﬁcantly worse sample quality compared to other methods.

Original: The low-quality samples from the image inpainting task in Figure 4b further conﬁrm that
Revised: The low-quality samples from the image inpainting task in Figure 4b further conﬁrm that

Original: Ambient VI is unable to produce good conditional samples, even though the observation is matched well.
Revised: Ambient VI is unable to produce good conditional samples, even though the observation is matched well.

Original: This may seem initially surprising but is a natural consequence of the VI objective.
Revised: This may seem initially surprising but is a natural consequence of the VI objective.

Original: Recall that our loss function decomposes into two terms: the KL term and the reconstruction term.
Revised: Recall that our loss function decomposes into two terms: the KL term and the reconstruction term.

Original: [Equation]
Revised: [Equation]

Original: If we alternatively derive the loss for Ambient VI, we arrive at an analogous objective:
Revised: If we alternatively derive the loss for Ambient VI, we arrive at an analogous objective:

Original: [Equation]
Revised: [Equation]

Original: While these two loss functions seem like simple reparametrizations of each other via f , they behave very differently during optimization due to the KL term.
Revised: While these two loss functions seem like simple reparametrizations of each other via f , they behave very differently during optimization due to the KL term.

Original: Notice that for both loss functions, the ﬁrst term is the reverse KL divergence between the variational distribution and the base distribution.
Revised: Notice that for both loss functions, the ﬁrst term is the reverse KL divergence between the variational distribution and the base distribution.

Original: Because reverse KL divergence places no penalty whenever p ˆ f or p g is zero regardless of p f , minimizing the reverse KL is known to have a mode-seeking behavior where variational distribution ﬁts a single mode of p f and ignores the rest of the support of p f
Revised: Because reverse KL divergence places no penalty whenever p ˆ f or p g is zero regardless of p f , minimizing the reverse KL is known to have a mode-seeking behavior where variational distribution ﬁts a single mode of p f and ignores the rest of the support of p f

Original: (Murphy, 2013, Chapter 21.2.2).
Revised: (Murphy, 2013, Chapter 21.2.2).

Original: In contrast, minimizing the forward KL has a zero-avoiding behavior and tries to cover all of p f ’s support.
Revised: In contrast, minimizing the forward KL has a zero-avoiding behavior and tries to cover all of p f ’s support.

Original: For our method, this is not a problem because the prior distribution of the base model p f ( z ) is a standard Gaussian and hence unimodal.
Revised: For our method, this is not a problem because the prior distribution of the base model p f ( z ) is a standard Gaussian and hence unimodal.

Original: However, for Ambient VI, p f ( x ) is the base model itself and is highly multimodal.
Revised: However, for Ambient VI, p f ( x ) is the base model itself and is highly multimodal.

Original: This can be empirically seen by visualizing the landscape of log p f ( x ) projected onto a random 2D subspace.
Revised: This can be empirically seen by visualizing the landscape of log p f ( x ) projected onto a random 2D subspace.

Original: In Figure 6, we clearly see that p f ( x ) has numerous local maxima.
Revised: In Figure 6, we clearly see that p f ( x ) has numerous local maxima.

Original: For Ambient VI, the variational distribution collapses into one of these modes.
Revised: For Ambient VI, the variational distribution collapses into one of these modes.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a new inference algorithm for distributions parametrized by normalizing ﬂow models.
Revised: We proposed a new inference algorithm for distributions parametrized by normalizing ﬂow models.

Original: The need for approximate inference is motivated by our theoretical hardness result for exact inference, which is surprising given that it applies to invertible models.
Revised: The need for approximate inference is motivated by our theoretical hardness result for exact inference, which is surprising given that it applies to invertible models.

Original: We also presented a detailed empirical evaluation of our method with both quantitative and qualitative results on a wide range of tasks and datasets.
Revised: We also presented a detailed empirical evaluation of our method with both quantitative and qualitative results on a wide range of tasks and datasets.

Original: There are numerous directions for further research.
Revised: There are numerous directions for further research.

Original: For example, our method can be possibly extended to other latent-variable generators such as VAEs and GANs (Kingma & Welling, 2013; Goodfellow et al., 2014).
Revised: For example, our method can be possibly extended to other latent-variable generators such as VAEs and GANs (Kingma & Welling, 2013; Goodfellow et al., 2014).

Original: A signiﬁcant limitation of our work is that we need to re-train the pre-generator for each observation.
Revised: A signiﬁcant limitation of our work is that we need to re-train the pre-generator for each observation.

Original: It may be possible to avoid retraining by amortizing the pre-generator over all possible observations.
Revised: It may be possible to avoid retraining by amortizing the pre-generator over all possible observations.

Original: Studying the various trade-offs resulting from such scheme would be an interesting result similar in spirit to Cremer et al.
Revised: Studying the various trade-offs resulting from such scheme would be an interesting result similar in spirit to Cremer et al.

Original: Overall, we believe that the idea of a pre-generator creating structured noise is a useful and general method for leveraging pre-trained generators to solve new generative problems.
Revised: Overall, we believe that the idea of a pre-generator creating structured noise is a useful and general method for leveraging pre-trained generators to solve new generative problems.

Original: A P ROOF OF H ARDNESS R ESULTS A. P RELIMINARIES
Revised: A P ROOF OF H ARDNESS R ESULTS A. P RELIMINARIES

Original: A Boolean variable is a variable that takes a value in {− 1 , 1 } .
Revised: A Boolean variable is a variable that takes a value in {− 1 , 1 } .

Original: A literal is a Boolean variable x i or its negation ¬ x i .
Revised: A literal is a Boolean variable x i or its negation ¬ x i .

Original: A clause is set of literals combined with the OR operator, e.g., x 1 ∨ ¬ x 2 ∨ x 3 .
Revised: A clause is set of literals combined with the OR operator, e.g., x 1 ∨ ¬ x 2 ∨ x 3 .

Original: A conjunctive normal form formula is a set of clauses joined by the AND operator, e.g., ( x∨ ¬ x 2 ∨ x 3 ) ∧ ( x 1 ∨ ¬ x 3 ∨ x 4 ) .
Revised: A conjunctive normal form formula is a set of clauses joined by the AND operator, e.g., ( x∨ ¬ x 2 ∨ x 3 ) ∧ ( x 1 ∨ ¬ x 3 ∨ x 4 ) .

Original: A satisfying assignment is an assignment to the variables such that the Boolean formula is true.
Revised: A satisfying assignment is an assignment to the variables such that the Boolean formula is true.

Original: The 3-SAT problem is the problem of deciding if a conjunctive normal form formula with three literals per clause has a satisfying assignment.
Revised: The 3-SAT problem is the problem of deciding if a conjunctive normal form formula with three literals per clause has a satisfying assignment.

Original: We will show that conditional sampling from ﬂow models allows us to solve the 3-SAT problem.
Revised: We will show that conditional sampling from ﬂow models allows us to solve the 3-SAT problem.

Original: We ignore the issue of representing samples from the conditional distribution with a ﬁnite number of bits.
Revised: We ignore the issue of representing samples from the conditional distribution with a ﬁnite number of bits.

Original: However the reduction is still valid if the samples are truncated to a constant number of bits.
Revised: However the reduction is still valid if the samples are truncated to a constant number of bits.

Original: A. D ESIGN OF THE A DDITIVE C OUPLING N ETWORK
Revised: A. D ESIGN OF THE A DDITIVE C OUPLING N ETWORK

Original: Given a conjunctive normal form with m clauses, we design a ReLU neural network with 3 hidden layers such that the output is 0 if the input is far from a satisfying assignment, and the output is about a large number M if the input is close to a satisfying assignment.
Revised: Given a conjunctive normal form with m clauses, we design a ReLU neural network with 3 hidden layers such that the output is 0 if the input is far from a satisfying assignment, and the output is about a large number M if the input is close to a satisfying assignment.

Original: We will deﬁne the following scalar function
Revised: We will deﬁne the following scalar function

Original: [Equation]
Revised: [Equation]

Original: This function is 1 if the input is 1 , 0 if the input x has | x − 1 | ≥ ε and is a linear interpolation on ( − ε, 1 + ε ) .
Revised: This function is 1 if the input is 1 , 0 if the input x has | x − 1 | ≥ ε and is a linear interpolation on ( − ε, 1 + ε ) .

Original: Note that it can be implemented by a hidden layer of a neural network and a linear transform, which can be absorbed in the following hidden layer.
Revised: Note that it can be implemented by a hidden layer of a neural network and a linear transform, which can be absorbed in the following hidden layer.

Original: See Figure 7 for a plot of this function.
Revised: See Figure 7 for a plot of this function.

Original: For each variable x i , we create a transformed variable ˜
Revised: For each variable x i , we create a transformed variable ˜

Original: x
Revised: x

Original: i by applying ˜ x i =
Revised: i by applying ˜ x i =

Original: δ ε ( x i ) − δ ε ( − x i ) .
Revised: δ ε ( x i ) − δ ε ( − x i ) .

Original: Note that this function is 0 on ( −∞ , − 1 − ε ] ∪ [ − 1 + ε, 1 − ε ] ∪ [1 + ε, ∞ ) , − 1 at x i = − 1 , 1 at x i = 1 , and a smooth interpolation on the remaining values in the domain.
Revised: Note that this function is 0 on ( −∞ , − 1 − ε ] ∪ [ − 1 + ε, 1 − ε ] ∪ [1 + ε, ∞ ) , − 1 at x i = − 1 , 1 at x i = 1 , and a smooth interpolation on the remaining values in the domain.

Original: Every clause has at most 8 satisfying assignments.
Revised: Every clause has at most 8 satisfying assignments.

Original: For each satisfying assignment we will create a neuron with the following process: (1) get the relevant transformed values
Revised: For each satisfying assignment we will create a neuron with the following process: (1) get the relevant transformed values

Original: ˜ x i , ˜ x j , ˜ x k , (2) multiply each variable by 1 / 3 if it is equal to 1 in the satisfying assignment and − 1 / 3 if it is equal to − 1 in the satisfying assignment, (3) sum the scaled variables, (4) apply the δ ε function to the sum.
Revised: ˜ x i , ˜ x j , ˜ x k , (2) multiply each variable by 1 / 3 if it is equal to 1 in the satisfying assignment and − 1 / 3 if it is equal to − 1 in the satisfying assignment, (3) sum the scaled variables, (4) apply the δ ε function to the sum.

Original: We will then sum all the neurons corresponding to a satisfying assignment for clause C j to get the value c j .
Revised: We will then sum all the neurons corresponding to a satisfying assignment for clause C j to get the value c j .

Original: The ﬁnal output is the value M × ReLU( (cid:80)
Revised: The ﬁnal output is the value M × ReLU( (cid:80)

Original: j c j − ( m − 1)) , where M is a large scalar.
Revised: j c j − ( m − 1)) , where M is a large scalar.

Original: We say that an input to the neural network x corresponds to a Boolean assignment x (cid:48) ∈ {− 1 , 1 } d if for every x i we have | x i − x (cid:48)
Revised: We say that an input to the neural network x corresponds to a Boolean assignment x (cid:48) ∈ {− 1 , 1 } d if for every x i we have | x i − x (cid:48)

Original: i | < ε .
Revised: i | < ε .

Original: For ε < 1 / 3 , if the input does not correspond to a satisfying assignment of the given formula, then at least one of the values
Revised: For ε < 1 / 3 , if the input does not correspond to a satisfying assignment of the given formula, then at least one of the values

Original: c j is 0 .
Revised: c j is 0 .

Original: The remaining values of c j are at most 1 , so the sum in the output is at most ( m − 1) , thus the sum is at most zero, so the ﬁnal output is 0 .
Revised: The remaining values of c j are at most 1 , so the sum in the output is at most ( m − 1) , thus the sum is at most zero, so the ﬁnal output is 0 .

Original: However, if the input is a satisfying assignment, then every value of c j = 1 , so the output is M .
Revised: However, if the input is a satisfying assignment, then every value of c j = 1 , so the output is M .

Original: A. G ENERATING SAT S OLUTIONS FROM THE C ONDITIONAL D ISTRIBUTION
Revised: A. G ENERATING SAT S OLUTIONS FROM THE C ONDITIONAL D ISTRIBUTION

Original: Our ﬂow model will take in Gaussian noise x 1 , . . .
Revised: Our ﬂow model will take in Gaussian noise x 1 , . . .

Original: , x d , z ∼ N (0 , 1) .
Revised: , x d , z ∼ N (0 , 1) .

Original: The values x 1 , . .
Revised: The values x 1 , . .

Original: , x d will be passed through to the output.
Revised: , x d will be passed through to the output.

Original: The output variable y will be z + f M ( x 1 , . . .
Revised: The output variable y will be z + f M ( x 1 , . . .

Original: , x d ) , where f M is the neural network described in the previous section, and M is the parameter in the output to be decided later.
Revised: , x d ) , where f M is the neural network described in the previous section, and M is the parameter in the output to be decided later.

Original: Let A be all the valid satisfying assignments to the given formula.
Revised: Let A be all the valid satisfying assignments to the given formula.

Original: For each assignment a , we will deﬁne X a to be the region X a = { x ∈ R
Revised: For each assignment a , we will deﬁne X a to be the region X a = { x ∈ R

Original: d : (cid:107) a − x (cid:107) ∞ ≤ ε } , where as above ε is some constant less than 1 / 3 .
Revised: d : (cid:107) a − x (cid:107) ∞ ≤ ε } , where as above ε is some constant less than 1 / 3 .

Original: Let X
Revised: Let X

Original: A = (cid:83) a ∈
Revised: A = (cid:83) a ∈

Original: A X a .
Revised: A X a .

Original: Given an element x ∈ X a , we can recreate the corresponding satisfying assignment a .
Revised: Given an element x ∈ X a , we can recreate the corresponding satisfying assignment a .

Original: Thus if we have an element of X A , we can certify that there is a satisfying assignment.
Revised: Thus if we have an element of X A , we can certify that there is a satisfying assignment.

Original: We will show that the distribution conditioned on y =
Revised: We will show that the distribution conditioned on y =

Original: M can generate satisfying assignments with high probability.
Revised: M can generate satisfying assignments with high probability.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: If we can show that p ( y = M, X A ) (cid:28) p ( y = M, X A ) , then we have that the generated samples are with high probability satisfying assignments.
Revised: If we can show that p ( y = M, X A ) (cid:28) p ( y = M, X A ) , then we have that the generated samples are with high probability satisfying assignments.

Original: Note that,
Revised: Note that,

Original: [Equation]
Revised: [Equation]

Original: Also notice that if x ∈ X A , then f M ( x ) = 0 .
Revised: Also notice that if x ∈ X A , then f M ( x ) = 0 .

Original: Thus y ∼ N (0 , 1) and P ( y =
Revised: Thus y ∼ N (0 , 1) and P ( y =

Original: M | X A ) =
Revised: M | X A ) =

Original: Θ(exp( − M 2 / 2)) .
Revised: Θ(exp( − M 2 / 2)) .

Original: Now consider any satisfying assignment x a .
Revised: Now consider any satisfying assignment x a .

Original: Let X (cid:48) a be the region X (cid:48) a = { x ∈ R d : (cid:107) a − x (cid:107) ∞
Revised: Let X (cid:48) a be the region X (cid:48) a = { x ∈ R d : (cid:107) a − x (cid:107) ∞

Original: ≤ 12 m } .
Revised: ≤ 12 m } .

Original: Note that for every x in this region we have f M ( x ) ≥ M/ 2 .
Revised: Note that for every x in this region we have f M ( x ) ≥ M/ 2 .

Original: Additionally, we have that
Revised: Additionally, we have that

Original: P ( X (cid:48) a ) =
Revised: P ( X (cid:48) a ) =

Original: Θ( m ) − d .
Revised: Θ( m ) − d .

Original: Thus for any x ∈ X (cid:48) a , we have p ( Y = M | x ) (cid:38) exp( − M 2 / 8) .
Revised: Thus for any x ∈ X (cid:48) a , we have p ( Y = M | x ) (cid:38) exp( − M 2 / 8) .

Original: We can conclude that
Revised: We can conclude that

Original: [Equation]
Revised: [Equation]

Original: For M = O ( √ d log m ) , we have that p ( y = M, X A ) is exponentially smaller than p ( y = M, X A ) .
Revised: For M = O ( √ d log m ) , we have that p ( y = M, X A ) is exponentially smaller than p ( y = M, X A ) .

Original: This implies that sampling from the distribution conditioned on y =
Revised: This implies that sampling from the distribution conditioned on y =

Original: M will return a satisfying assignment with high probability.
Revised: M will return a satisfying assignment with high probability.

Original: A. H ARDNESS OF A PPROXIMATE S AMPLING
Revised: A. H ARDNESS OF A PPROXIMATE S AMPLING

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: The complexity class RP is the class of decision problems with efﬁcient random algorithms that (1) output YES with probability 1 / 2 if the true answer is YES and (2) output NO with probability 1 if the true answer is NO.
Revised: The complexity class RP is the class of decision problems with efﬁcient random algorithms that (1) output YES with probability 1 / 2 if the true answer is YES and (2) output NO with probability 1 if the true answer is NO.

Original: It is widely believed that RP is a strict subset of NP .
Revised: It is widely believed that RP is a strict subset of NP .

Original: A simple extension of the above theorem shows that even approximately matching the true conditional distribution in terms of TV distance is computationally hard.
Revised: A simple extension of the above theorem shows that even approximately matching the true conditional distribution in terms of TV distance is computationally hard.

Original: The total variation (TV) distance is deﬁned as d TV ( p, q ) =
Revised: The total variation (TV) distance is deﬁned as d TV ( p, q ) =

Original: sup E | p ( E )
Revised: sup E | p ( E )

Original: − q ( E ) | ≤ 1 , where E is an event.
Revised: − q ( E ) | ≤ 1 , where E is an event.

Original: The below corollary shows that it is hard to conditionally sample from a distribution that is even slightly bounded away from 1 .
Revised: The below corollary shows that it is hard to conditionally sample from a distribution that is even slightly bounded away from 1 .

Original: Corollary 3.
Revised: Corollary 3.

Original: The conditional sampling problem remains hard even if we only require the algorithm to sample from a distribution q such that d TV ( p ( · | x = x ∗ ) , q ) ≤ 1 − 1 / poly( d ) , where d is the dimension of the distribution.
Revised: The conditional sampling problem remains hard even if we only require the algorithm to sample from a distribution q such that d TV ( p ( · | x = x ∗ ) , q ) ≤ 1 − 1 / poly( d ) , where d is the dimension of the distribution.

Original: We show that the problem is still hard even if we require the algorithm to sample from a distribution q such that d TV ( p ( x | y =
Revised: We show that the problem is still hard even if we require the algorithm to sample from a distribution q such that d TV ( p ( x | y =

Original: y ∗ ) , q ) ≥ 1 / poly( d ) .
Revised: y ∗ ) , q ) ≥ 1 / poly( d ) .

Original: Consider the event
Revised: Consider the event

Original: X
Revised: X

Original: A from above.
Revised: A from above.

Original: We saw that p ( X A | y
Revised: We saw that p ( X A | y

Original: =
Revised: =

Original: M ) ≥ 1 − exp( − Ω( d )) .
Revised: M ) ≥ 1 − exp( − Ω( d )) .

Original: We have that d
Revised: We have that d

Original: TV ( p ( · | y = M ) , q ) ≥ 1 − exp( − Ω( d ) − q ( X A )) .
Revised: TV ( p ( · | y = M ) , q ) ≥ 1 − exp( − Ω( d ) − q ( X A )) .

Original: Suppose that the distribution q has q ( X A ) ≥ 1 / poly( d ) .
Revised: Suppose that the distribution q has q ( X A ) ≥ 1 / poly( d ) .

Original: Then by sampling a polynomial number of times from q we sample an element of X A , which allows us to ﬁnd a satisfying assignment.
Revised: Then by sampling a polynomial number of times from q we sample an element of X A , which allows us to ﬁnd a satisfying assignment.

Original: Thus if we can efﬁciently create such a distribution, we would be able to efﬁciently solve SAT and RP = NP.
Revised: Thus if we can efﬁciently create such a distribution, we would be able to efﬁciently solve SAT and RP = NP.

Original: As we are assuming this is false, we must have q ( X A ) ≤ 1 / poly( d ) , which implies d TV ( p ( · | y = M ) , q ) ≥ 1 − 1 / poly( d ) .
Revised: As we are assuming this is false, we must have q ( X A ) ≤ 1 / poly( d ) , which implies d TV ( p ( · | y = M ) , q ) ≥ 1 − 1 / poly( d ) .

Original: B  D ERIVATION OF E QUATION (3)
Revised: B M ISSING D ERIVATIONS B. D ERIVATION OF E QUATION (3)

Original: Here we present a detailed derivation of Equation (3).
Revised: Here we present a detailed derivation of Equation (3).

Original: Note that this equality is true up to a constant w.r.t.
Revised: Note that this equality is true up to a constant w.r.t.

Original: ˆ f [Equation] which is ﬁne as we use this as the optimization objective.
Revised: ˆ f , which is ﬁne as we use this as the optimization objective.

Original: [Equation]
Revised: [Equation]

Original: In ( A ), we drop the log p f
Revised: In ( A ), we drop the log p f

Original: (˜ y
Revised: (˜ y

Original: =
Revised: =

Original: y ∗ ) term as it is constant w.r.t.
Revised: y ∗ ) term as it is constant w.r.t.

Original: ˆ f .
Revised: ˆ f .

Original: In ( B ), we use the conditional independence ˜
Revised: In ( B ), we use the conditional independence ˜

Original: y ⊥⊥ x | y .
Revised: y ⊥⊥ x | y .

Original: In ( C ), we use the invariance of KL divergence under invertible transformation to rewrite the KL divergence in terms of z .
Revised: In ( C ), we use the invariance of KL divergence under invertible transformation to rewrite the KL divergence in terms of z .

Original: C E XPERIMENT D ETAILS C. O UR A LGORITHM
Revised: B. J OINT VI VS . M ARGINAL VI

Original: 1: Input :
Revised: We also provide a justiﬁcation for using the joint VI loss as discussed in section 4.

Original: : observation we conditioning 2: for i = num steps do 3: for j m do (cid:46) generate m latent codes from p z ) 4: Sample (cid:15) ( j
Revised: Speciﬁcally, we show that the joint VI loss is an upper bound to the intractable marginal VI loss in eq.

Original: [Table]
Revised: (1), which we wish to minimize.

Original: ∼ N ( 0 , I d ) 5: z ( j ) ← [Table] f ( (cid:15) ( j )  ) 6: end for 7:
Revised: For notational brevity, we write q ( x ) and q ( z ) to denote the variational posterior p f ◦ ˆ f ( x ) and the pre-generator p ˆ f ( z ) , respectively.

Original: L ←
Revised: Then, (Joint KL)

Original: [Equation]
Revised: [Equation]

Original: m (cid:80)
Revised: + E q ( x 2 ) [ D KL ( q ( x 1 | x 2 ) (cid:107) p f ( x 1 | ˜ x 1 = x ∗ 1 , x 2 ))] ≥ D KL ( q ( x 2 ) (cid:107) p f

Original: [Equation]
Revised: ( x 2 | ˜ x 1 = x ∗ 1 )) =

Original: 8: θ ← θ − ∇ θ L (cid:46) gradient step 9: end for [Figure]
Revised: (Marginal KL) , where the last inequality is due to the nonnegativity of KL. Note that equality holds when D KL ( q ( x 1 | x 2 ) (cid:107) p f

Original: [Figure] − 10 − 8 − 6 − 4 − 2log 10 σ 0 .0 .0 .1 .1 .1 .
Revised: ( x 1 | ˜ x 1 = x ∗ 1 , x 2 )) = 0 , i.e. when our variational posterior matches the true conditional.

Original: M S E o f o b s e r v e d p a rt
Revised: C E XPERIMENT D ETAILS

Original: [Figure]
Revised: C.

Revised: O UR A LGORITHM

Original: C. E FFECTS OF THE SMOOTHING PARAMETER
Revised: C. E FFECTS OF THE SMOOTHING PARAMETER

Original: The choice of variance for Gaussian smoothing is an important hyperparameter, so we provide some empirical analysis of the effects of σ .
Revised: The choice of variance for Gaussian smoothing is an important hyperparameter, so we provide some empirical analysis of the effects of σ .

Original: As shown in Figure 8a, large values of σ cause the samples to ignore the observation, whereas small values lead to unnatural samples as the learned distribution becomes more degenerate.
Revised: As shown in Figure 8a, large values of σ cause the samples to ignore the observation, whereas small values lead to unnatural samples as the learned distribution becomes more degenerate.

Original: Visually, we achieve essentially negligible variance on the observed portion past σ = 0 .
Revised: Visually, we achieve essentially negligible variance on the observed [Table] [Table] [Table]

Original:  01 , but at the slight degradation in the sample quality.
Revised: portion past σ = 0 . 01 , but at the slight degradation in the sample quality.

Original: In Figure 8b, we also notice that the difference between the true observation ( x ∗ 1 ) and generated observation ( ˜ x 1 ) stops improving past σ = 1e − 4 .
Revised: In Figure 8b, we also notice that the difference between the true observation ( x ∗ 1 ) and generated observation ( ˜ x 1 ) stops improving past σ = 1e − 4 .

Original: We tried annealing σ from a large value to a small positive target value to see if that would help improve the sample quality at very small values of σ , but noticed no appreciable difference.
Revised: We tried annealing σ from a large value to a small positive target value to see if that would help improve the sample quality at very small values of σ , but noticed no appreciable difference.

Original: In practice, we recommend using the largest possible σ that produces observations that are within the (task-speciﬁc acceptable range of the true observation.
Revised: In practice, we recommend using the largest possible σ that produces observations that are within the (task-speciﬁc) acceptable range of the true observation.

Original: C. H YPERPARAMETERS : B ASE M ODEL AND P RE - GENERATOR
Revised: C. H YPERPARAMETERS : B ASE M ODEL AND P RE - GENERATOR

Original: See Table 3 and Table 4 for the hyperparameters used to deﬁne the network architectures train them.
Revised: See Table 3 and Table 4 for the hyperparameters used to deﬁne the network architectures train them.

Original: For the color datasets CIFAR-10 and CelebA-HQ, we used 5-bit pixel quantization following Kingma & Dhariwal (2018).
Revised: For the color datasets CIFAR-10 and CelebA-HQ, we used 5-bit pixel quantization following Kingma & Dhariwal (2018).

Original: Additionally for CelebA-HQ, we used the same train-test split (27,000/3,000) of
Revised: Additionally for CelebA-HQ, we used the same train-test split (27,000/3,000) of

Original: Kingma & Dhariwal (2018) and resized the images to 64 × 64 resolution.
Revised: Kingma & Dhariwal (2018) and resized the images to 64 × 64 resolution.

Original: Uncurated samples from the base models are included for reference in Figure 
Revised: Uncurated samples from the base models are included for reference in Figure 9.

Original: [Table]
Revised: [Table]

Revised: Observed

Revised: [Figure]

Revised: − 10 − 8 − 6 − 4 − 2log 10 σ 0 .0 .0 .0 .0 .0 .

Revised: M S E o f o b s e r v e d p a rt

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: C. H YPERPARAMETERS : I MAGE I NPAINTING
Revised: C. H YPERPARAMETERS : I MAGE I NPAINTING

Original: We randomly chose 900/500/300 images from MNIST/CIFAR-10/CelebA-HQ test sets, applied masks deﬁned in Section 6. [Table] and generated samples conditioned on the remaining parts.
Revised: We randomly chose 900/500/300 images from MNIST/CIFAR-10/CelebA-HQ test sets, applied masks deﬁned in Section 6.1, and generated samples conditioned on the remaining parts.

Original: FID and other sample quality metrics were computed using 32 conditional samples per test image for VI 
Revised: FID and other sample quality metrics were computed using 32 conditional samples per test image for VI methods (ours and Ambient VI), 8 for Langevin Dynamics, and 6 for PL-MCMC. We note that using more samples for VI methods do not unfairly affect the result of sample quality evaluation, i

Original: [Table]
Revised: [Table]

Original: methods (ours and Ambient VI), 8 for Langevin Dynamics, and 6 for PL-MCMC. We note that using more samples for VI methods do not unfairly affect the result of sample quality evaluation, i.e. there was no appreciable difference when using 8 vs. 32 samples to compute FID.
Revised: e. there was no appreciable difference when using 8 vs. 32 samples to compute FID.

Original: We used more samples simply because it is much cheaper for VI methods to generate samples compared to MCMC methods.
Revised: We used more samples simply because it is much cheaper for VI methods to generate samples compared to MCMC methods.

Original: For VI Methods (Ours & Ambient VI)
Revised: For VI Methods (Ours & Ambient VI)

Original: • Learning rate: 1e − 3 for MNIST; 5e − 4 for the others • Number of training steps: 2000 for CelebA-HQ; 1000 for the others
Revised: • Learning rate: 1e − 3 for MNIST; 5e − 4 for the others • Number of training steps: 2000 for CelebA-HQ

Original: For Langevin Dynamics
Revised: [Table]

Original: • Learning rate: 5e − 4 for all datasets • Length of chain: 1000 for CIFAR-10; 2000 for the others
Revised: Learning rate: 5e − 4 for all datasets • Length of chain: 1000 for CIFAR-10; 2000 for the others

Original: For PL-MCMC
Revised: [Table]

Original: • Learning rate: 5e − 4 • Length of chain: 2000 for MNIST • σ a = 1e − 3 , σ p = 0 .
Revised: • Learning rate: 5e − 4 • Length of chain: 2000 for MNIST • σ a = 1e − 3 , σ p = 0 .

Original: C. H YPERPARAMETERS : I NVERSE P ROBLEMS
Revised: C. H YPERPARAMETERS : I NVERSE P ROBLEMS

Original: Colorization Compressed Sensing Compressed Sensing Super-resolution
Revised: [Table]

Original: Learning rate 5e −5e −5e −5e − 4 σ 0. Dataset CelebA-HQ CelebA-HQ
Revised: Engel et al

Original: CIFARCIFAR-

Original: Batch size

Original: Number

Original: [Table]
Revised: [Table]

Original: Engel et al.

Original: ; Parno & Marzouk (2018);
Revised: (2017); Parno & Marzouk (2018)

Original: [Table]
Revised: [Table]

Revised: Hoffman et al.

Revised: ; Nijkamp et al.